Server discussion...

[Azure]

I had some problems today getting on....obviously it was slow when the Jokinen deal went down, but even after that, a few problems.

I know you posted in the other threads about upgrades, or what the options would be, but those posts are going to be lost in all the other threads.

Would member donations help? What would it take to keep the server running as fast as possible for 10,000 users?

Just throwing out some ideas to generate discussion here.

[photon]

For 10,000 users? We're talking multiple front end servers and probably multiple database servers as well... that's RedFlagDeals level traffic, we're not there yet.

There's really two separate issues at work here.

First is the server itself. Up to a certain level the server itself was actually working very well, during the Leopold deal the server was working hard but not really in trouble. The problem is when you go from 20 requests per second to 100's of requests per second. That stresses every aspect of the server, and basically overwhelms them all, be it processor, disk input/output, or memory.

We can do some things to help with this. Adding memory is something we've been thinking about, and it might be worth doing to see how it helps; memory is relatively cheap. Processor upgrades are possible, but are not cheap, the cost would almost be the same as the server cost in the first place strangely enough! Disk upgrades would also be more expensive and would require a new enclosure for them.. any additional space we take up would increase the monthly cost (double it or more).

There are some more software things we can do to try and help with this, and we'll probably look at doing them by the draft too (install lighttpd or a different http server than Apache for those curious to reduce memory and CPU footprint).

The other factor is bandwidth. During the Leopold deal, the server wasn't really the problem, it was the bandwidth. Everyone's waiting for the data to make its way through the pipe while the server is also waiting.

This is pure $$ to upgrade, and it's not cheap.

However we've done one thing to help.. the reason we could even set a new record for users, and that the board was more usable than last year, was because of that. We offloaded the image files onto a content deliver network; so all the image files, avatars, and CSS files get served from somewhere else, which reduces the load on the server drastically. At one point the load on the image server was 2 times our total bandwidth connection for the server. So that's good, that helped significantly.

The other thing which I will do is ask for a temporary increase in our bandwidth for one day. The host seemed amenable to this. This will help a lot as well. I wasn't able to get this in place until well after the Jokinen deal today, but next time I'll know to do it in advance.

So anyway that's a summary.

The thing is anything we do such as going to the next increment of bandwidth, or adding another server (thus requiring more space at the facility) results in a very significant increase in monthly cost. So even if we added a 2nd server with a fund raising drive, that's not the most significant cost.

I'm being vague about the numbers because a) in some cases I don't know them and b) they're Bingo's #'s to share or not.

As far as donations and such, those tend to ebb and flow while monthly costs don't

There's always options such as having a monthly membership for additional benefits, but I don't think Bingo is warm to that idea, as that can separate the user base, the donators feel better than those who don't, or a special access area for donators, or whatever. Keeping it open and equal has its merits.

[Azure]

Yeah, I think everyone would agree with keeping it open and equal.

But, I also think that there are a lot of people on here that would be willing to donate in order to help out with the costs. Maybe put up a 'donate' ad or something on the side? I'm sure a lot of people simply forget about what it takes to keep this place running behind the scenes.

Making it easy for people to donate might help.

As for everything else you said.....I'll be the first to admit that I know nothing when it comes to servers, but I also think it helps if the staff is open about it. If people see what is required, and what it'll take to upgrade in the future, they'll be willing to contribute.

[teamchachi]

Originally Posted by Azure

I had some problems today getting on....obviously it was slow when the Jokinen deal went down, but even after that, a few problems.

I know you posted in the other threads about upgrades, or what the options would be, but those posts are going to be lost in all the other threads.

Would member donations help? What would it take to keep the server running as fast as possible for 10,000 users?

Just throwing out some ideas to generate discussion here.

Capacity planning for CP is really tricky. 99% of the time the server just loafs along doing very little. The server and colocation situation that we have is more than adequate virtually all the time.

The main problem comes on March 5th and July 1st. Traffic spikes to *crazy* levels. We're talking 3 to 4 times the load that you would see even after a big win or a playoff game.

There are lots of things that we could do to handle the increased load, but they cost a lot of money and would be totally wasted 363 days per year.

We would be looking at multiple servers, more rack space, and a significantly more expensive bandwidth bill.

Simply throwing hardware at it is a small part of the equation. Servers are relatively inexpensive. Bandwidth and decent colocation space are not. We would be talking about increasing monthly expenses by 300-400% just so that the servers are more responsive on 2 days out of the year.

[Azure]

Yeah, interesting.....I didn't realize that.

I guess the one day bandwidth idea would probably work great. Even if it costs a bit....I'm sure its not THAT much to increase your bandwidth for one day. Especially if the guys who look after the server realize that we only have a few days out of the year where traffic spikes like that.

I still think we should have a clearly visible donation button though. Doesn't matter if there are only small hardware upgrades. We can still help pay for everything.

[ken0042]

Originally Posted by Azure

I'm sure its not THAT much to increase your bandwidth for one day.

I'm not sure it's that simple. It might be more like you having a 125 amp electrical coming into your house, and asking Enmax if 2 days out of the year you could have 500 amp service.

[photon]

Originally Posted by Azure

Making it easy for people to donate might help.

You mean the big donate button on the front page of the site?

http://www.calgarypuck.com/donate/

[teamchachi]

Originally Posted by Azure

Yeah, interesting.....I didn't realize that.

I guess the one day bandwidth idea would probably work great. Even if it costs a bit....I'm sure its not THAT much to increase your bandwidth for one day. Especially if the guys who look after the server realize that we only have a few days out of the year where traffic spikes like that.

I still think we should have a clearly visible donation button though. Doesn't matter if there are only small hardware upgrades. We can still help pay for everything.

Its always a tug of war between bandwidth and server power. You bump up the bandwidth and now the server can't keep up. You upgrade the servers and then you have unutilized processing power even with a saturated internet connection.

The current server is way more than adequate 363 days per year. However, today we temporarily tripled the bandwidth and the additional volume of requests almost maxed out the server CPU. Dynamically generating all the forum pages from the database requires a lot of processing power.

I do think that a donation button is a good idea.

Ideally, a Calgary based company would donate some rack space and bandwidth for the server. That would eliminate the monthly colocation costs and would bring the server closer to home. Then all we would need to worry about is one time server purchase/upgrade expenses whenever additional capacity is required.

I'd love to have a caching proxy server sitting in front of the web server. And then move the database to a separate server as well. But that likely isn't going to happen if we're having to pay significant monthly expenses for rack space and bandwidth...

[photon]

Originally Posted by Azure

Yeah, interesting.....I didn't realize that.

I guess the one day bandwidth idea would probably work great. Even if it costs a bit....I'm sure its not THAT much to increase your bandwidth for one day. Especially if the guys who look after the server realize that we only have a few days out of the year where traffic spikes like that.

Originally Posted by ken0042

I'm not sure it's that simple. It might be more like you having a 125 amp electrical coming into your house, and asking Enmax if 2 days out of the year you could have 500 amp service.

It's kind of in the middle.

Technology wise we could increase the bandwidth to the server instantly by a factor of 100 with the flip of a switch... the hosting facility has the capacity easily to handle that, and the hardware they use to allocate and manage the bandwidth to each of their clients is built for just such a thing.

So doing it is pretty easy.. click calgarypuck in some management software and use a dropdown to select the faster speed and click ok.

The difficulty is in how a company operates..

Using the popular tube theory of Internets, if bandwidth is water, the size of the tube dictates how much water you can get. The way the industry works is you can either get water at VERY high speeds but pay per gallon, or you can get water at a lower speed (narrow tube) but not have to worry about how much water you get, unlimited water, as much as you can suck through your tube.

What we need is the ability to switch back and forth when we need.. of course this works well for the customer, difficult for the water provider since everyone's going to game the system to maximize the benefit and minimize their cost.

So from a business point of view having a temporary increase in bandwidth for 2 days out of the year just isn't something that they handle.. there's no sku for it, no billing procedure, no price, no technical procedure, no way to make sure it gets returned to normal levels later, etc etc. So in that respect it's difficult to get done.

That said they were open to doing it for us today, so hopefully we'll get the same response on July 1 and relax that portion of the limits.

I know people are very willing to contribute financially, but you don't want to go to that well too many times otherwise it starts to dry up.

[photon]

Originally Posted by teamchachi

Ideally, a Calgary based company would donate some rack space and bandwidth for the server. That would eliminate the monthly colocation costs and would bring the server closer to home.

Yeah that'd be awesome.

When we got the server I hit up all the guys in town that I could think of and none were interested unfortunately.

And that can get a wee bit touchy if the company in question becomes a topic of discussion among the forum, or some other kind of conflict of interest happens.. Editorial freedom is probably important to Bingo I probably already spend more time than is necessary thinking about how some content might be viewed by the object of our adoration.

[teamchachi]

Originally Posted by ken0042

I'm not sure it's that simple. It might be more like you having a 125 amp electrical coming into your house, and asking Enmax if 2 days out of the year you could have 500 amp service.

Sort of. Some hosting companies do provide something called "burstable" bandwidth which will allow the customer to exceed their contracted bandwidth allotment during unusual peak periods. However this doesn't apply to us in our current situation.

http://en.wikipedia.org/wiki/Burstable_billing

[teamchachi]

Originally Posted by photon

Using the popular tube theory of Internets, if bandwidth is water, the size of the tube dictates how much water you can get. The way the industry works is you can either get water at VERY high speeds but pay per gallon, or you can get water at a lower speed (narrow tube) but not have to worry about how much water you get, unlimited water, as much as you can suck through your tube.

Going with your example:

Lets say we do go for a *huge* pipe (which costs a lot more money). Most of the time, all we would have is a little trickle of water and our server has no problem drinking it all up. It rarely breaks a sweat.

Today, it would have been like putting a garden hose in the server's mouth and then turning it on full blast. There comes a point where the server can only handle so many requests per second. We reached that point a few times today even with the modest increase in bandwidth.

So really it comes down to this: The current server and bandwidth allotment is more than adequate 99% of the time. Is it worth increasing monthly hardware and hosting costs by a factor of 3-4 so that things don't slow down 1% of the time?

[photon]

If I'm the one hitting refresh to find out what everyone else thinks of Jokinen, the answer is YES!

[Azure]

Originally Posted by photon

You mean the big donate button on the front page of the site?

http://www.calgarypuck.com/donate/

Baah.

I never visit the front page.

[Azure]

Interesting take on the whole thing.

About the 'donate' button, maybe put one on the 'main' forum page too? I'm sure lots of people just type in forum.calgarypuck.com when they visit the site.

At least I do.

[Bobblehead]

This is a pretty common issue.

I'm wrestling with my database right now - 9 months of the years it is underutilized, the other 3 it has issues. And even then, in those 3 months the issues only happen for about 3 hours in the day.

Do you drop $100K in licensing/maintenance costs for a few hours a day for a few months?

Fast, good or cheap - pick 2.

[teamchachi]

The real issue here is return on investment (ROI). If this was an ecommerce site then you could do the math and figure out if there was a business case for doing a major upgrade. However, given the nature of the site, upgrading to handle peak traffic that happens for 5-10 hours a year just doesn't make sense.

Now, if the site was lagging on a daily or weekly basis then it would be a totally different issue...

[JiriHrdina]

I compare the performance yesterday of the server to what we saw elsewhere and I think we have to be pleased. I would suggest that if the Jokinen deal didn't go down we might have stayed up the entire time. So really there are only a few type of events that will cause us to crash
- winning the big shiny thing
- a major deal on trade deadline
- a major signing on July 1st

I guess I'm in-line with Chachi - doesn't seem like enough reason to pile more money into thing just to deal with what are very rare situation.

[Azure]

Yeah, but if we do win the big shiny thing.....I want to be able to browse CP.

[photon]

You will be able to!

Just not right away

You should be out on the Red Mile anyway at that point.