Recent evidence with a spatial interference paradigm has shown that arrows and eye gaze yield opposite congruency effects, arrow target eliciting faster responses when their direction is congruent with their position (standard congruency effect), and gaze producing faster reaction times for incongruent conditions (reversed congruency effect). But in ecological contexts eye gaze tend to be more perceptually complex (i.e., embedded in the whole face) than simple arrows. The present study aimed to replicate this dissociation using whole faces and a comparable non-social target, formed by arrows embedded in a colored geometric background. Whereas the reversed congruency effect with gaze was replicated, the standard spatial interference with arrows was surprisingly absent. A similar outcome appeared when the contrast between the arrows and the task-irrelevant background increased. The results confirm the robustness of the reversed congruency effect with eyes, regardless of whether they are presented alone or within a face. In addition, and importantly, the unexpected absence of the spatial conflict with complex arrow targets seems to be a consequence of higher figure-ground segregation demands, which extend the processing of the task-relevant spatial dimension and, in turn, cause the decay of the location code. This pattern of results, and the provided interpretation, can explain previous unexplained findings in the spatial interference literature.