The aging brain undergoes structural changes even in very healthy individuals. Quantifying these changes could help disentangle pathologic changes from those associated with the normal human aging process. Using longitudinal magnetic resonance imaging (MRI) data from 227 carefully selected healthy human cohort with age ranging from 50 to 80 years old at baseline scan, we quantified age-related volumetric changes in the brain of healthy human older adults. Longitudinally, the rates of tissue loss in total gray matter (GM) and white matter (WM) were 2497.5 and 2579.8 mm³ per year, respectively. Across the whole brain, the rates of GM decline varied with regions in the frontal and parietal lobes having faster rates of decline, whereas some regions in the occipital and temporal lobes appeared relatively preserved. In contrast, cross-sectional changes were mainly observed in the temporal-occipital regions. Similar longitudinal atrophic changes were also observed in subcortical regions including thalamus, hippocampus, putamen, and caudate, whereas the pallidum showed an increasing volume with age. Overall, regions maturing late in development (frontal, parietal) are more vulnerable to longitudinal decline, whereas those that fully mature in the early stage (temporal, occipital) are mainly affected by cross-sectional changes in healthy older cohort. This may suggest that, for a successful healthy aging, the former needs to be maximally developed at an earlier age to compensate for the longitudinal decline later in life and the latter to remain relatively preserved even in old age, consistent with both concepts of reserve and brain maintenance.