I have found the opposite. When I was at my peak in terms of Physique, Women who were girlfriend, Fiances or Wives either threw themselves at me or at minimum heavily flirted with me to the point that if I was slightly more persistent, I could of fvcked.
Just off the top of my head I remember meeting a Woman who was in her mid 20's from Washington D.C. visiting NYC in a bar, she was engaged but still flirted with me. I grabbed up on her ass and she was still loving it. If I pushed for it, I could of slept with her that night before she had to go back to Washington. If I remember correctly she was in Law and very attractive.