Personal experience: I have kinda big arms, shoulders and back. From what I have been told, that has always been a good thing those areas are what they grab when we are first starting to get to know each other (flirting). I do have abs but I am not super shredded, single digit body fat - and yes, I have been told it is nice when it comes to sexi time. They do find it appealing and grab/run their hands over.
The moral of the story is you should be in shape / look to get into shape if you aren't. In addition to hitting the gym / taking up sport or however you want to do it, wear flattering clothes - not just colors that work with your skin tone, but also fit right. I grew up during the period where baggy was in and some of those pictures look ****in ridiculous looking back now. Most of my clothes now are fitted (my suits) - for tshirts and sport shirts I usually get the 'slim fit' as opposed to regular. It makes a difference the way it clings to you - if you have muscles they show and it is a turn on to women. The oblivious, likely fat **** guys that say being in shape doesn't matter --- you agree a woman with a killer body looks much hotter in tight, form fitting clothes? Leggings/Athleisure Wear, dresses to the club...if this is the case why wouldn't the opposite be true?