Nanoindentation experiment is an important means to study micro mechanical properties of rock. Up to now only a few studies have discussed the correlation between the micro mechanical properties of rock and various macroscopic strengths of the rock. Firstly, the nanoindentation experiments on four different kinds of marble were carried out by the continuous stiffness measurement technique to obtain the micro mechanical parameters of dolomite and calcite. Secondly, the microscopic data scale was upgraded by Mori-Tanaka method to obtain the homogenized elastic modulus and Poisson’s ratio. Finally, the correlation between the microscopic parameters and the macroscopic mechanical experiment results was analyzed, and the applicability of predicting macroscopic properties of rock using nanoindentation data was discussed. The results show that: (1) The elastic modulus of dolomite in marble is 122.5 GPa, and the hardness is 5.4 GPa. The elastic modulus of calcite in marble is 70.3 GPa, and the hardness is 2.3 GPa. The strength and deformation properties of calcite are relatively poorer compared to those of dolomite. (2) For dolomite and calcite in marble, an indention depth of approximately 800 nm is recommended for determining the elastic modulus and hardness with the continuous stiffness measurement technique. (3) The data dispersion at the grain boundary points is larger than that at the inner points of the grain, and the hardness can better reflect the defect effect of grain boundary than elastic modulus does. (4) The homogenized results obtained by Mori-Tanaka method are somewhat reliable for predicting macroscopic elastic modulus and Poisson’s ratio. (5) Nanoindentation data can reflect the influence of mineral types on rock strength. To accurately predict uniaxial compressive strength, tensile strength, and fracture toughness, it is necessary to consider other strength factors, such as rock texture and structure. These results demonstrate the application of nanoindentation experiment in rock materials and provide a reference for predicting macroscopic strength using micro mechanical parameters.