I would say YES to the OP. Even celebs and really hot guys have limited "market penetration." Even Mic Jagger sang he "can't get no satisfaction."
No man is desired by all women. The hotter/more famous/richer you are, the greater your market share will be. But it will never come close to being 100%. Even the hunkiest celebs have been turned down by many, many women who just weren't interested.
Some women will totally fall in love with you and MOST women won't give two sh*ts about you.
Men are much more homogenous in their desires. Whereas most men will find, say, Scarlett Johanson or Megan Fox highly desirable, women are much more varied in their "types." I've known women who thought a young Ozzie Osbourne was sexy as hell and others who thought he was ridiculous and gross.
This is why you need to expose yourself to as many women as possible (no pun intended). If 5% of women, let's say, will fall in love with you, and you only meet 10 women, you're fukked. But if you meet 100 women, then you've got 5 who love you.
Here's an article that's quite illuminating, as well:
http://blog.okcupid.com/index.php/the-mathematics-of-beauty/
I imagine it would be quite similiar for women. ie., the more disagreement there is about your attractiveness, the more the women who find you attractive will find you REALLY attractive.