Most of the swimmers I knew in high school got their bodies from just swimming. I bet Olympic level swimmers probably do some weight training too, but not at the level of bodybuilders.
Swimming is great. It gives you a cardio workout and a muscular workout too. I powerlift, and I use it as my primary cardio exercise because I find it's one of the few cardio exercises that don't devour my strength.
So, yes, you can get a good body from swimming and a good diet. In fact, it's an excellent way to get the lean, kinda muscular look chicks seem to really dig.