This. Hitting the gym just makes you more productive in many areas of life. With work, with women, on the field/court when playing weekend sports. You just feel great.
And true, certain girls will screw a guy on his physique alone. Heck Id say many girls get reeled in that way. I noticed an increase in female attention when I started working out and also when I grew out my hair. Definitely helps.
Plus leaning out and putting on muscle ends up helping your face too. As you lean out, usually your face will become my masculine and aesthetic. Depending on how you shed fat, you may lose some face fat and get that sexy jawline girls like.
To directly answer the OPs question, no you dont need to be really buff. Id say as long as you have a soccer player or swimmers build, girls will be wanting the body. Chicks seem to like that lean look with some muscle, but not too much buffness. Think Cristiano Ronaldo, Didier Drogba, or Michael Phelps.