You body (men and women) changes dramatically in your mid 30s. I remember when I turned 34... suddenly I found myself putting on weight without eating more, and workouts became more difficult.
The difference between most men and most women is that men have the discipline to push past discomfort and find workout regimes that work with you changing body. Men typically also have the discipline to change their eating habits.
But really... lets be fair... there are plenty of fat @ss men that look like crap. I think we notice it more with women because a woman's looks is very important.