I wouldn't worry that you are not there for Crankworx or Pemberton - there are tons of awesome things to do in Whistler, and personally I think those events just create crowds that I'd like to avoid.
I've done a few outings with this group:
http://tagwhistler.com/ They have pretty much all the activities you would want, ziplines, ropes course, white water rafting, ATV, etc. and I had a good experience both times.