We take red sandstone as the research object and apply the freeze-thaw cycles, CT scans and mechanical properties experiments. We use image processing technology combined with genetic algorithm optimization model to achieve the denoise, enhancement, segmentation and three-dimensional reconstruction of CT scan images after 0, 5, 10, 20, and 40 freeze-thaw cycles. With the damage identification and comparative study of the same object across scales, we established a prediction formula of elastic modulus deterioration based on mesoscopic damage. Therefore, the macroscopic mechanical behavior of freeze-thaw red sandstones can be interpreted from the physical nature of the material meso-structure. The results show that genetic algorithm based on image maximum entropy can quickly and accurately select the threshold for image segmentation, and achieve the recognition of matrix and defects in rock meso-structure. With the increase of freezing and thawing cycles, the porosity of rock increases, and the fractal dimension of pore decreases. On the meso-scale, the evolution shows that the pores expand and the number increases, but the structural complexity decreases. The macroscopic and mesoscopic damage variables defined by the traditional methods are based on the effective bearing area and elastic modulus, and they fail to fully consider the damage physical mechanism and the internal structure information of the material. The damage evolution curves are different. Based on the two physical mechanisms, we define the meso-damage variable and the macro-damage variable that considers the natural rock damage, which achieves the combination of macroscopic and mesoscopic damages. Finally, according to the relationship between meso-structure evolution and macroscopic mechanical response in the process of freeze-thaw cycles, we propose a prediction formula of elastic modulus degradation, and analyze the different dominant roles of pore size and pore structure morphology through the damage process. We interpret the mechanical mechanism of macroscopic sandstone freeze-thaw damage based on the meso-structure physical mechanisms.