It's an individual choice/decision (and not a static decision either).
I've lived in homes that I owned from a 600 sq ft condo, a mobile home, a bungalow, and now to a massive two storey. Those decisions were based on finances versus wants/needs at the time. I think the key factor is - regardless of what a person chooses, if they invest carefully and wisely, it can pay significant financial dividends (as well as personal).
We have become home people now, after spending years out and about exploring our local, national and international world - and our home is now an extension of that lifestyle.
That said, we are considering moving again, and are seriously considering a move like Sliver made - simply for some convenience/practical purposes. I kind of doubt we will ultimately downsize though - I think it truly is hard to go back once one has become accustomed to a huge home.
Great advice here, btw. Good luck in your decision