It's all about harmony in your face...
Either it's part of a complete "look" or it's just something that looks out of place! Ideally your facial hair style matches your hairstyle and vice versa, can go so far as dressing style.
If you go out in a worn (but clean, well fitting, basically tailored to 'look' worn), low waist jeans with a CK shirt, somewhat longer and unorganised hair (not messy, actually nicely cut and groomed but made to look messy) and then wear some stubble along with that i can guarantee some looks!
I've went out with this style before and i've gotten a lot of comments on it. Most of them said that i looked "rough" but in a very good way (like a take me to the back of your car and rip off my clothes sort of way ;-) ).
Any kind of facial hairs that are longer style i don't really think look very good, unless you want to look like a rockstar or something. Then i suggest some pretty extreme dress style and long hair to match. But all in all unless you really ARE a rockstar, that look is pretty much out right now.
Very short cut, discrete beards can look very good, especially if you mix it with a "slick" look. Though they ALWAYS end up looking more or less ghey and make you look very "pretty"... if you get my drift.
So all in all, i'd have to say go clean or stubbed for a change of scenery, but facial hair mostly does more bad than good on your look. Beards are out, and the ones who are more or less "in" make you look gay... so yeah ;-)