I think planning work events, which don't involve a guest, are a stupid idea outside of regular work hours and an employee should be able to pull the plug anytime they want. But in your situation were your boss really wants you there just go, whatever, I know it sucks.
Thank god our golf tourneys are always on working Fridays
.