One other thing is to be aware of the difference between current expenses and capital expenses. For your example of creating a website, say if you buy a computer to work on the website. That would be a capital expense, and you cannot deduct 100% of the cost off your income. Instead you have to depreciate it according to the Capital Cost Allowance rules.
But if you pay a monthly fee to host the website, that seems reasonable that is a current expense.
This CRA guide has a lot of info and examples about business expenses:
http://www.cra-arc.gc.ca/E/pub/tg/t4002/t4002-e.html
Also, though you don't have to register for GST, you might want to. If you voluntarily register, you will have to charge your client GST and remit it to CRA. But then you can also claim any GST on expenses and get that money back.
Finally, this is just my understanding of things, if anything is wrong, hopefully someone can correct me. I imagine any decent accountant could help significantly.