Server discussion... - Page 4 - Calgarypuck Forums - The Unofficial Calgary Flames Fan Community

teamchachi · 03-20-2009, 12:27 AM

Quote:

Originally Posted by photon

A page served is one page, regardless of how many posts are in the page. The software will request the entire page at once, render it, and serve it.

So more posts per page should actually decrease the hits to the server, although at the same time it will increase the processing and bandwidth used per page, so it might be a wash in the end.

Exactly. Even if we could "optimize" this way, we'd only be increasing the efficiency but 10% tops.

vBulletin (and all the other forums) are CPU hogs by their very nature. They contain a lot of dynamic data that exists in many tables that contain hundreds of thousands (and millions!) of records. Every single page view requires the execution of a ton of PHP code, lots of looping, and 30+ database queries.

We've done everything that we can to reduce the load on the server by offloading all of the static content to a content distribution network. The only thing that the server does now is generate the actual HTML for the page. All the CSS, JS, and images come from the CDN.

We've done everything possible to optimize Apache2, PHP, MySQL, and Linux to squeeze every last drop of performance out of the hardware.

As long as we're just serving a couple of pages per second we're fine. Thats what the site does at 10 am on a typical morning. The CPU just bounces along at 10-20%. Increase that number to 5-7 pages per second and now we're at 50%. Now increase it to 10+ pages/second and we're maxed out.

Bobblehead · 03-23-2009, 02:16 PM

Is there still some "tweaking" going on?

I just wonder because on my work computer I run firefox with the NoScript extension. I have for a long time. But I noticed some time last week that suddenly some of the functionality of the board stopped working unless I permitted simplecdn.net to execute. Things like the format buttons, or some of the user-profile tabs. And I have never had to do that before, so I was just curious if that was a tweak one of you did, or if NoScript changed something (NoScript updates pretty regularly).

So I guess I was just curious.

photon · 03-24-2009, 07:05 AM

Yeah, I moved a bunch of the scripts over to simplecdn to help alleviate the load on the server a few days ago.

teamchachi · 04-17-2009, 08:18 AM

I imagine many of you noticed that the site maxed out last night after the overtime goal by Havlat.

To put things in perspective:

We peak out at about 8-10 hits/second during a busy lunch hour or just after a regular season game. That puts the server load average between 1 and 2.

Last night we maxed out at 44.69 hits/second which put the server load up over 20. In our case, anything over 2 means we're overloaded. So a 20 means that we had 18 processes waiting in a queue waiting to use a CPU.

photon · 04-17-2009, 08:29 AM

Nice.

Thanks for watching that teamchachi too I'm currently in Mexico so can't keep an eye on it.

teamchachi · 04-17-2009, 09:39 AM

I experimented last night with scaling back the number of Apache processes and it appears to have been successful. I put it down to 30 (from 50) and that keeps the server load around 3.5 even when all 30 are getting hammered. Loading a page takes 2-3x longer than normal, but at least the site is still usable. That's better than having the server load at 20 and *nobody* getting in due to the "server is too busy" message.

teamchachi · 04-17-2009, 11:49 AM

Quote:

Originally Posted by teamchachi

I experimented last night with scaling back the number of Apache processes and it appears to have been successful. I put it down to 30 (from 50) and that keeps the server load around 3.5 even when all 30 are getting hammered. Loading a page takes 2-3x longer than normal, but at least the site is still usable. That's better than having the server load at 20 and *nobody* getting in due to the "server is too busy" message.

I'm hoping that this holds up under an onslaught. I guess we'll see on Saturday night...

teamchachi · 04-18-2009, 10:53 PM

Well that worked out nicely. 30 Apache threads and the server peaked out at 3.7 with no noticeable slowness. Now if we could just score more than 2 goals...

JiriHrdina · 04-18-2009, 11:17 PM

Yeah the site seemed fine to me tonight. Nice job Chachi.

Hemi-Cuda · 04-19-2009, 12:32 AM

good work, i feel a lot better tonight after being able to vent on CP than having to store it up in a ball and stew about it on Friday

teamchachi · 04-19-2009, 01:29 PM

To be fair, Photon has done most of the grunt work. Offloading all the static content to a content distribution network has been the biggest factor in our ability to scale.

We're still limited by the size of our internet connection, but now we're just using it for HTML instead of HTML, CSS, Javascript, and tons of images.

Tweaking the # of server processes really just limits how many people can actively connect to the server at any given time. It is a form of throttling which ensures that there is a enough CPU to go around...

teamchachi · 09-18-2009, 08:23 AM

Well, the new hosting provider is doing its job. We got up to 4 Mbit/s outbound last night after Fleury's goal and we frequently hit 2 Mbit/s throughout the night.

Unfortunately, removing the bandwidth bottleneck means that we have a new bottleneck: CPU. The load hit 6.0 last night...

photon · 09-18-2009, 10:58 AM

Yeah, almost 1000 people on at the end of the game, where 5-600 was typical last year.

I guess we'll see if that continues.

I had moved all the images off of remote hosting as part of trying to figure out the forum problems for some people, so I will have to move them all back to help alleviate the load again.

Dion · 09-18-2009, 11:03 AM

Bingo · 09-18-2009, 12:00 PM

was quick for me ...

great job guys.

Azure · 09-18-2009, 05:14 PM

I had some problems, but it apparently had to do with OpenDNS.

Photon do you have any idea how much CP is increasing in amount of hits from month to month?

flip · 09-18-2009, 07:18 PM

Quote:

Originally Posted by Azure

I had some problems, but it apparently had to do with OpenDNS.

Photon do you have any idea how much CP is increasing in amount of hits from month to month?

I haven't been following this discussion at all but would a site like this help?

photon · 09-18-2009, 07:42 PM

Quote:

Originally Posted by Azure

I had some problems, but it apparently had to do with OpenDNS.

Photon do you have any idea how much CP is increasing in amount of hits from month to month?

We do keep stats on hits and visits and pageviews. It's tough to gauge because the amounts vary wildly from one month to the next depending on what's going on, so you almost have to a year over year comparison for individual months to get an idea.

But yeah if I look back over the 3 years since I've been collecting data, it's a pretty substantial increase.

LockedOut · 09-21-2009, 10:27 PM

Can we get rid of guests to speed up the site if it helps. At least during the game. It's pretty slow and this during just an exhibition game. As I type this, in OT guests outnumber us by about 2 to 1.

Azure · 09-21-2009, 10:33 PM

yeah, during the games the server is extremely slow. hard to read anything.

and its just preseason.