I don't know if it is the biggest factor but it is a relevant factor.
@MatureDJ has talked about how attending an all-male high school impacted him socially. The fact that
@MatureDJ is discussing this at age 55+ shows how relevant a childhood/adolescence experience is. I have met males who went to all-male high schools and they felt it took years to unwind some of the damage from that experience.
As a man in my early 40s right now, I am still feeling the consequences of multiple relocations from childhood and in my 20s. That has had a lifelong impact on my ability to get dates from social circle connections, meaning that I had to make more effort with other forms of arranging dates (co-ed sports leagues, traditional daygame, nightlife venues, and swipe apps).
In addition, my parents were a terrible married couple with an unhealthy relational dynamic. If my parents had been healthier, it actually might not have served me well because I would have had more blue pill fantasies. My mother, who mainly raised me, raised me a 'nice guy' so I had to unwind the damage of being raised as a 'nice guy'. 'Nice guys' finish last.