A quick correction to your back of the envelop calculations, (which are good).
You're using percentage points instead of percent. So if you assume a 2.4% growth rate in a BAU scenario, then if you substract 4% from that you'd have an annual growth rate of 2.3. It's not a huge difference but over time it will add up. Just not nearly as severe as your scenario.
I'm using modelling results from Mark Jaccard's CIMS model and a report found here:
http://www.davidsuzuki.org/files/rep..._green_eng.pdf
The model predicts that at a carbon tax of $75 per tonne with the revenue being recycled into reduced payroll and income taxes the effect would be a 0.18% reduction on baseline GDP of $1.7 trillion. In the order of 3 billion by 2020.