Yes, that was precisely the reason! Just like horse-racing uses small men as jockeys (less mass to be accelerated and burdened on the horse), a smaller human is less deadweight (i.e., in aerospace, anything that is meant to return to base is considered deadweight, as opposed to disposable weight) as well as requiring a smaller capsule and less oxygen.
The earliest NASA astronauts had a height limit of 5'11" - and a weight that matches that height on a wirey military physique. Think about that - today's hypergamous chick would reject EVERY ONE OF THE "RIGHT STUFF" MEN!
. It wasn't until the 2nd group of astronauts that a Chad would be in the group - the ShortCel (Gus Grissom) was in the 1st group (and commander for the ill-fated Apollo 1 mission, but look how Ed White (center) mogged him (aside from the beginning hairline retreat).
View attachment 11100