Let's think about this with science.
A guy who's relatively short (5'4") is still over 84 percent of the height of a tall man (6'4"). Fifteen percent difference. We're not talking night and day, here.
Also, evolutionary biology/psychology is one of my interests (and was my major in college) and it is a proven scientific fact (im too lazy to cite this, but there was a link to this on a college website for anyone that wants to search the forums that doesn't believe me), that for most women, the most attractive body type for a man is:
about average height (5'8" to 6'0")
broad shoulders
muscular chest and muscular arms, but not as muscular as a bodybuilder's physique.
Think Eminem, and Tupac. Women love those guys and their bodies.
Average heights are average for a reason, and that's because their naturally selected for. If only 6'+ guys were considered attractive, and were the only ones getting laid, then the average dude's height would be over 6'
FWIW I'm 5'10", and in my personal experience, most guys seem taller than me. I live in Colorado, and I dont know if it's something in the water but most guys out here seem to be 6'-6'2"
BUT: I don't really feel short. When I'm around white people they're tend to think I'm a little short, but when I'm around hispanics or native americans (my races), they think I'm tall. It's never bothered me or my game.
I've actually talked to quite a few women about this, and most women said as far as height is concerned, all that really matters is that the man is taller than them, but as David DeAngelo says, attraction is not a choice, and it's not an unbreakable rule by any means