Visual scanning of faces was studied during categorization of expression and gender with the aim of revealing possible differences in the perceptual mechanisms that mediate analysis of these facial properties. A more distributed scanning pattern, with increased fixation in the lower face, was observed in the expression compared to the gender task. Distribution of fixations across the upper and lower face also varied depending on the specific gender and expression of the viewed faces, with female faces and faces showing anger attracting more fixations to the eye region. Variations reflecting an interaction between gender and expression were also observed. However, the nature of these modulations suggests a differential influence of perceptual interactions and social/affective value on eye movement measures and on pupil size. Collectively, these results show that the visual inspection of faces is determined in a complex manner by the specific demands of different categorization tasks and by the perceptual and social/affective properties of the emotional expressions shown by either male or female faces.