Background: Increased leisure-time physical activity (LTPA) is linked with decreased mortality risk, while also with increased left ventricular mass, which may induce left ventricular hypertrophy (LVH). We investigated whether LVH modifies the association between higher LTPA and lower mortality risk in population at high cardiovascular risk. Methods: In a prospective national cohort, we used the left ventricular mass/body surface area (LVM/BSA) method to define LVH. Baseline LTPA was self-reported and divided into: low (500 metabolic equivalent of task [MET]) min/week), moderate (500–1999 MET min/week) and high (\ge2000 MET-min/week). Analyses of the dose-response relationship between LTPA and left ventricular mass were performed using restricted cubic spline regression. A multivariate adjusted Cox proportional hazards regression analysis was used to estimate hazard ratios (HRs). Results: A total of 163,006 participants (55.3% females, mean [standard deviation] age, 62.4 [7.4] years) were included. During a median of 4.8 years of follow-up, 6586 (4.0%) died from all causes and 3024 (1.9%) from cardiovascular causes. Multivariate adjusted Cox proportional hazards regression analyses revealed that moderate and high LTPA were linked with less cardiovascular and all-cause mortality risk than low LTPA in the absence of LVH. In those with LVH, the association of high (0.83, 0.69–0.99) or moderate (0.72, 0.56–0.91) LTPA with cardiovascular mortality risk persisted. For all-cause mortality risk, this association was only significant in high LTPA (0.73, 0.61–0.86), while marginal in moderate LTPA (0.96, 0.84 to 1.08). Overall, the correlation patterns between LTPA and mortality risk appears distinct between those with LVH and those without LVH; the modification of LVH was not significant regarding mortality risk among the high cardiovascular risk population (all-cause: p-value for interaction = 0.074; cardiovascular cause: p-value for interaction = 0.581), except in females regarding all-cause mortality risk (p-value for interaction = 0.006). Conclusions: The association between higher LTPA and lower mortality risk was not modified by LVH in high cardiovascular risk population. However, the presence of LVH altered this association in females regarding the all-cause mortality risk.