Under repeated loading and unloading, the mechanical response of brittle rock exhibits pronounced hysteresis characteristic which is related to its fundamental damage behavior. Quantitative analysis based on 4D X-ray Computed Tomography (XCT), and Digital Volume Correlation (DVC) techniques were conducted on a fine-grained granite to get insights of damage process and strain filed evolution along loading/unloading and unloading/reloading hysteresis loops. Based on DVC analysis, the bulk axial, volumetric and von Mises strains were calculated. Then, the evolution of strain fields and strain population distributions were analyzed to investigate the evolving strain process and identified the relationship between bulk strain and local strain distribution. Finally, the mechanism of hysteresis loop behavior was investigated by analyzing the interaction between volumetric strain and voids evolution, and the strain concentration zones along the failure faults. The results showed that the bulk volumetric and lateral strains could better reflect the damage evolution compared to axial strain fine-grained granite. The dilation developed at the onset of loading and expanded with loading. During unloading, the dilation volume was increased slightly, and a non-uniform strain release process was observed: the strain release of contraction volumes was prior to the dilation volumes. Further damage was found in unloading/reloading hysteresis. Despite the tensile mechanism dominated the damage evolution pattern approaching to the failure, the shear-tensile mechanism was also involved in the process of damage evolution and final failure. Based on analysis, it is speculated that the development of damage unlocked the behavior of hysteresis: the difference in between sliding, and reverse-sliding of shear-tensile fractures was considered as the primary mechanism of the non-uniform strain release path. The dynamic interaction between opening of perpendicular voids and the reverse-sliding of shear fractures was considered as the damage mechanism during unloading.