mate I think the first thing you need to look at is the frequency of your workouts. Sounds like you're already on the right lines by cutting your workout in half, but be careful about upping the tempo back to what you were doing before...the workout regime for someone who sits at a desk all day is different from someone who does manual labour.
Depending on your split, 4 days a week is generally enough even if you had a desk job, 5 starts pushing towards overtraining. Your body only makes gains while it is resting, obviously your rest time is being cut down because of the physical nature of your job. I would look at cutting down to 3 workouts per week as a maximum. Keep the time down, 45-60 min in the gym is fine, no more: do 2 or 3 sets of 6-8 reps on an exercise then move on. If you discipline yourself to only allow 3 sets (after warm up) on an exercise then psychologically you push yourself harder because you don't want to leave anything in the tank.
No need to quit the job - working out should fit around your lifestyle not dominate it. The work you are doing shifting the water barrels sounds like good cardio to me, it will get your heartrate up and keep you aerobically fit, also it is going to keep away the fat, so the good news is you are going to be more fit than you ever would be before.