I don't know if the boss is really a jerk here. A little strong arm-ish maybe, but I mean if they're putting together a golf outing, and planning around the people who accepted, and you waited 6 weeks to back out ... yeah, I'd be a little annoyed too if I was him.
My advise would be to not take the free time off and go to the golf thing and next time don't accept things on a whim if you hate them.
But no, you didn't damage your reputation, yet
