Yes
Sure as hell matters. It ain't everything, but it matters. Find Pook's skinny guy thread and read it, and go to the HF forum. Lots of good stuff there.
It's definitely more healthy to be on the thin side, than on the heavy side, provided you're not dangerously thin.
The changes to my workout regimen that blasted me through my most difficult plateau were:
1) Work out less often.
2) Use very heavy weight.
3) Keep constantly feeding myself: power drinks with lots of proteing and complex carbs when I wake up before breakfast, between regular (relatively small and frequent) meals, and before I crash at night, if I don't eat a proper meal then.
Regarding the above:
1) Working out more than once a week for any given body part resulted in more muscle breakdown than buildup. Muscle is built between the workout, and the muscles need to recover well, lest you get injured or break them down. At my age, it takes about a week. Perhaps less if you're a teen or in your early 20s, but I'd still say give 5 days rest per bodypart.
2) One must lift heavy weight to activate mainly the "fast twitch" muscles that make for bulk. I do 1-2 rep sets with nearly max weight often. Or 3-2-1 descending reps (switching arms with dumbells) to effectively get 6 reps of volume with 3 rep weight (or 4-3-2-1, 4-2-1, whatever happens). I prefer dumbells where possible, because they balance your left & right sides, favoring neither, and free weights work more stabilizing muscles than machine exercises, so you get a more comprehensive workout.
During my periods of greatest progress, I did "reverse pyramid" workouts: warm up, then the heavy set (1 - 3 rep weight) first, follwed by a 6-8 rep per exercise set, a protein/carb drink, a warmdown (drumming a bit, for me), and stretching, which elongates the muscles to allow yet more growth.
3) Any longer periods of hunger allow muscle breakdown to take place. As long as you're sore, there's muscle repair ( = building ) to be done. Keep feeding it.
You don't have to work out a real lot. Just make it count when you do.