Haha so much hate for TO! Lots of good suggestions already by people, there is a ton to do in Toronto (big reason why I moved here from Vancouver)! I also live downtown so feel free to PM me for more specific requests - I got you!
If you and the family are good with walking, Toronto is a fantastic city to walk in. A nice walk down Queen Street westward from say University down all the way is fun as you pass by different neighborhoods. Trinity Bellwoods Park (Queen Street, between Ossington and Bathurt) is very nice too and you and the family can find a spot and enjoy the people and sounds and have a nice picnic or something. Good thing is we don't have homeless people there unlike other cities and their parks!
Also, since you're downtown, take the TTC (so, Subway and streetcars). That's the fastest, cheapest and most convenient way to get around downtown (take subway to get to the closest stop to wherever you want to go and streetcars to take you east/west). Uber in downtown traffic will be tiring.
Finally, go to
https://www.toronto.com/events/ and filter for the dates you are in town. Toronto is amazing in the summers for activities and festivals. There's always something happening!
Edit to add that Toronto has done a really good job to make sure that many of the weekend festivities are free or very low cost specifically for families, which helps in today's economy!