Face recognition is normally a rapid and effortless process enabling individuals to identify others who are familiar to them. Underlying this remarkable, but perhaps overlooked, aspect of visual function is a network of interconnected brain regions, which process the raw visual information provided by the visual system. As a result, face recognition is vulnerable to both ocular disease and neurological impairments. This article reviews current evidence about these causes of impaired face perception and outlines the clinical implications for affected patients.