Background: we aimed to assess both cross-sectional and temporal links between measured hearing impairment and self-perceived hearing handicap, and health outcomes. Methods: in total, 811 Blue Mountains Hearing Study participants (Sydney, Australia) aged ≥55 years were examined twice (1997-99 and 2002-04). Hearing levels were measured with pure-tone audiometry. The shortened version of the hearing handicap inventory (HHIE-S) was administered, scores ≥8 defined hearing handicap. Results: baseline hearing impairment was strongly associated with 7 of the 10 HHIE-S questions, 5 years later. Individuals with and without hearing impairment at baseline reported that they felt embarrassed and/or frustrated by their hearing problem, and that it hampered their personal/social life, multivariable-adjusted OR: 11.5 (CI: 3.5-38.1), OR: 6.3 (CI: 2.5-15.7) and OR: 6.0 (CI: 2.1-17.5), respectively, 5 years later. Hearing-impaired, compared with non-hearing-impaired adults had a significantly higher risk of developing moderate or severe hearing handicap, OR: 3.35 (CI: 1.91-5.90) and OR: 6.60 (CI: 1.45-30.00), respectively. Cross-sectionally (at wave 2), hearing handicap increased the odds of depressive symptoms and low self-rated health by 80 and 46%, respectively. Conclusion: older, hearing-impaired adults were significantly more likely to experience emotional distress and social engagement restrictions (self-perceived hearing handicap) directly due to their hearing impairment.