Server discussion...

[photon]

2 3ghz Xeon processors with mirrored 10k SCSI drives.

[teamchachi]

Originally Posted by photon

2 3ghz Xeon processors with mirrored 10k SCSI drives.

They're single core Socket 604 Xeons, but I believe the board will accomodate 2 dual cores. The only problem is the upgrade price. This is a 3 year old server and the s604 dual core processors are still going for $1400 *EACH*! So upgrading from 2 to 4 cores is looking like a $2800 hit for what amounts to really just an incremental upgrade. 2 more cores won't allow this box to scale *that* much because we'll start being limited by disk IO before we can max out 4 cores.

Ouch!

For $2800 we could buy an entirely new quad core Xeon server with cash to spare!

[ken0042]

What if I could get my hands on 2 more of those processors? The place I just left was getting rid of Dell Precision 470s- they are about that age but I'd have to check what socket type they are. It would mean calling in a favour- so when you say a slight increase would it be worth it for a couple of hundred bucks?

Or would we be just better off using a Precision 470 as a second server?

[Madman]

Thanks for the replies guys.

It just seems to me, (not being an IT guy), that leaving the server cost and maintenance to a dedicated hosting/data centre would be a better option.

Not saying you guys don't know what you are doing, or didn't buy the right stuff by any means - but the buying power of larger companies and rotation of hardware would seem to play in your favour - not to mention a larger support and knowledge base as well as possible bandwidth increases.

[photon]

Originally Posted by ken0042

What if I could get my hands on 2 more of those processors? The place I just left was getting rid of Dell Precision 470s- they are about that age but I'd have to check what socket type they are. It would mean calling in a favour- so when you say a slight increase would it be worth it for a couple of hundred bucks?

Or would we be just better off using a Precision 470 as a second server?

Well the server only has 2 sockets so 2 more of the same processors won't help, they'd have to be dual core processors to make any difference.. but it would make a difference and be worth a few hundred bucks for sure.

We can't really use it as a server though, most colocation facilities need rack-mounted servers, the ones that have shelves for non rack mount charge extra per month (like $50 or more).

[teamchachi]

Originally Posted by Madman

Thanks for the replies guys.

It just seems to me, (not being an IT guy), that leaving the server cost and maintenance to a dedicated hosting/data centre would be a better option.

Not saying you guys don't know what you are doing, or didn't buy the right stuff by any means - but the buying power of larger companies and rotation of hardware would seem to play in your favour - not to mention a larger support and knowledge base as well as possible bandwidth increases.

This really comes down to $. We can easily build up the infrastructure to handle the growing membership and the increased server load that goes with it. We've experienced a great deal of growth in members and daily page views since the server was put into production in 2006. Owning our own server hardware is still probably the cheapest way to go, and we're definitely on a budget. I don't know how much $ the site generates, but I don't imagine anyone is going to be retiring anytime soon! :-)

The current server is still more than capable of handling the load 98% of the time. So do we have the political will to spend a bunch of additional money so that they server doesn't slow to a crawl at the trade deadline, beginning of free agency, and after a few playoff games? Thats a tough question to answer.

[cal_guy]

I've done some research and apparantly there were only 5 dual-core cpu that supported by the server, the Dual Core Xeon 2.8 Ghz which were designed for dual processor configurations, and the Xeon 7020, 7030, 7040 and 7041 cpu which are designed for multiprocessor configuration. Amazing all 5 are still apparantely still available from Intel although obviously you have to get them from a major distributed like Techdata and there probably insanely expensive. There also available on ebay and the 2.8 ghz version is "reasonable", with the cheapest one going for $125. Obviously no point unless you replace both CPU.

Of course talk of replacing CPU might premature since the server has a lot of room to add memory and as photon said memory is cheap, would adding memory be effective?

[photon]

Not really, it's CPU that's getting pegged it seems.. and if we alleviate that I'm a bit concerned that we'd just saturate i/o then, and that's even harder to fix.

[teamchachi]

Originally Posted by cal_guy

I've done some research and apparantly there were only 5 dual-core cpu that supported by the server, the Dual Core Xeon 2.8 Ghz which were designed for dual processor configurations, and the Xeon 7020, 7030, 7040 and 7041 cpu which are designed for multiprocessor configuration. Amazing all 5 are still apparantely still available from Intel although obviously you have to get them from a major distributed like Techdata and there probably insanely expensive. There also available on ebay and the 2.8 ghz version is "reasonable", with the cheapest one going for $125. Obviously no point unless you replace both CPU.

Of course talk of replacing CPU might premature since the server has a lot of room to add memory and as photon said memory is cheap, would adding memory be effective?

We've got lots of memory. We've still got 40% of the memory free even when the CPUs are pinned.

There are two big problems that are inherent to virtually all web forums:
1) Every single page view results in a dynamically generated page. When your database contains close to 2M posts (and growing), this is expensive on the CPU side.

2) The developers insist on tracking sessions even when visitors are unauthenticated. If they didn't, then we could setup a reverse proxy cache and serve pages out of the cache to unauthenticated users. That would dramatically reduce the load on the server.

We're doing everything that we can to make the generation of pages as efficient as possible. All the internal VB caching is turned on, and we're run xcache to do PHP op code caching.

What it really comes down to is horsepower. If you want to run a big forum with a lot of posts and capacity for a lot of concurrent visitors then you need some serious horsepower. We had serious horsepower. We no longer have serious horsepower.

The server that we currently have was good bang for the buck 3 years ago. It was a relatively powerful server in 2006. And it was more than adequate for most of the peaks that we were experiencing in 2006. Unfortunately (or fortunately depending on your perspective), the load on the server has increased and now its not entirely up to the task during peak periods.

All vBulletin forums come to a point when they are forced to go the multi-server route. I think we're pretty much there. To facilitate further growth we really need to be considering the following:

1) At least a 10mbps internet connection.
2) Converting the existing server to MySQL only.
3) Procuring a 4-8 core xeon server with 4+ GB RAM that will run Apache 2.2 or Lighttpd only. My server OS preference is FreeBSD 7.1 for this configuration.

[teamchachi]

Couldn't sleep last night so did some benchmarking using Apache Bench. I tested the follow pages.

10 KB static html page:
3000 requests per second. It could go a lot higher if we tweaked apache. Minimal CPU required. Obviously we would run out of internet bandwidth before we ever approached these numbers in the real world.

Site Home Page: http://www.calgarypuck.com/
Current hardware tops out at 800 requests/second. This is dynamically generated, but by Wordpress which is far more efficient than vBulletin. It could probably go higher if we tweaked the maximum # of clients in Apache. CPU is around 60% Obviously we would run out of bandwidth before we ever approached these numbers in the real world.

Main forum index: http://forum.calgarypuck.com/forumdisplay.php?f=6
The current hardware can dynamically generate this page at a rate of 14 per second. CPUs are 100%. That is only 840 pages per minute.

Typical Thread Page http://forum.calgarypuck.com/showthread.php?t=71407
The current hardware can dynamically generate this page at a rate of 12.5 per second. CPUs are 100%. That is only 750 pages per minute.


This really just shows what a pig vBulletin is (its not a secret - there's an entire forum on vbulletin.com that is dedicated to improving vBulletin performance). Unfortunately, there really aren't any better forum options that I'm aware of.

So its not really a surprise that 1000+ concurrent users aren't happy with the site performance when the server is maxing out generating 750-850 pages per minute!

Assuming that 1000 people each want a new page every 30 seconds, we need to get our capacity up to *at least* 2000 pages/minute. With an averge html page size of 15 KB, that comes out to about 500 KB/s which would require a 4mbps internet connection.

Having said that, we should also probably be taking future growth into account as well, otherwise we'll be in this position again in a year. Our current dual Xeon processors are roughly equivalent to a current single Xeon. So an 8 core Xeon box should have approximately 8-10x the processing capacity that we have now. Being conservative, increasing CPU performance by even a factor of 5 should increase the server performance to around 70 pages/second which is 4200 pages/minute.

Anyway, thought it might be interesting to throw some numbers out...

[Gozer]

Damn, now that's insomnia.

[Burninator]

How much horsepower does painting a racing stripe or flames onto the server add?

[teamchachi]

Originally Posted by Burninator

How much horsepower does painting a racing stripe or flames onto the server add?

Not enough I'm afraid. But I like your style.

[teamchachi]

To put things in a bit further perspective: Most forums of this size are running a multi-server setup. Typically they have one server running Apache and another running the database. All things being equal, we've actually done very well to keep everything on one server. However, we're rapidly approaching the breaking point and the playoffs may not be a lot of fun around here unless we can find a way to kick up the performance a few notches...

[Madman]

Originally Posted by teamchachi

To put things in a bit further perspective: Most forums of this size are running a multi-server setup. Typically they have one server running Apache and another running the database. All things being equal, we've actually done very well to keep everything on one server. However, we're rapidly approaching the breaking point and the playoffs may not be a lot of fun around here unless we can find a way to kick up the performance a few notches...

Could the current server be split off to run one of the two parts? Or would it just become a bottleneck?

Did you already mention that and I missed it?

[teamchachi]

Originally Posted by Madman

Could the current server be split off to run one of the two parts? Or would it just become a bottleneck?

Did you already mention that and I missed it?

Yes, the current server is still very serviceable and would make an excellent database server for the next couple of years. What we really need to do is offload Apache onto another server if we want to be able to handle the peaks.

[teamchachi]

Let me be clear: I'm not saying that "the sky is falling" or trying to hold anybody's feet to the fire. The server was an excellent acquisition and has served CP well. It isn't (likely) to spontaneously combust any time soon. I'm just saying that the current server hardware and bandwidth aren't adequate to handle periods of peak usage.

Its up to the CP decision makers to take that information, look at the budget, and then decide what to do with it.

[Azure]

I'm sure Bingo has seen this.

I wonder what he has to say about this.....

[Bobblehead]

OK, those pages served per minute, how many posts per page does that assume?

I know you can adjust the # of posts per page. If the database hit is the bottleneck then perhaps limit the number of posts per page (although I don't think many people have changed this setting, so there may not be much to gain). On the other hand, if the rendering is the hog then perhaps increasing the default posts per page may be a benefit.

[photon]

A page served is one page, regardless of how many posts are in the page. The software will request the entire page at once, render it, and serve it.

So more posts per page should actually decrease the hits to the server, although at the same time it will increase the processing and bandwidth used per page, so it might be a wash in the end.