A primate infant’s experience during early life is heavily influenced by characteristics of its biotic environment, including characteristics of its mother and its broader social environment. For example, infant baboons are more likely to die if their mothers experienced high levels of early life adversity. These intergenerational effects may be mediated by differences in the social environment experienced by infants born to mothers that experienced high levels of early adversity. Adult males play an especially important role in infants’ experience by providing protection and a zone of relative safety in which an infant can develop. Here we present the most detailed analysis to date of the determinants of the immediate adult male social environment that infants experience and the influences of adult males on a wide range of infant behaviors. We show that the average number of adult males within 5m of infant baboons is significantly repeatable over time (R = 0.15) and that this repeatability is partially explained by the levels of early life adversity experienced by the infant’s mother. We also show that the number of adult males in close proximity to an infant predicts a wide range of fundamental infant behavioral traits, including the mother-infant spatial relationship, infant activity budgets, and the frequency of both positive and negative social interactions with non-mothers. Our results are consistent with the possibility that the effects of maternal early life adversity can be transmitted, in part, via differences in the early life social environments that infants of high-adversity mothers experience.