Ridiculous long load times during games

[photon]

Originally Posted by frinkprof

Photon, I will agree that some of the responses do have an edge to them, but I don't think that you should take it as a commentary on the work you're doing and your efforts. Correct me if I'm wrong, but I do sense a bit of sensitivity that way.

It's not sensitivity to my efforts, it's sensitivity to outright condescension; that someone would feel the need to give me unsolicited opinions and advice about my choices.

If you're walking down the street (not you specifically, the royal you), do you walk up to someone a few houses down and out of the blue tell them that their choice of house design was poor, or that their job doesn't pay them enough, or that the way they run their life is ill advised? Of course not, I wouldn't give that kind of advice unless it was solicited or if I knew the person extremely well, otherwise it'd be pretty condescending.

Or in the other case just outright whining, everyone's sensitive to that.

Originally Posted by frinkprof

I can appreciate that some of the features were disabled on a temporary basis only, but many of them are disabled often (busy times of year such as trade deadline, etc.), nevermind that the crashes and load times that spur these measures happen frequently in the first place. The loading and crashes are the main source of most people's frustration, I'm sure.

It's a very common question in scaling infrastructure; how far to build out for. You can spent to the moon and make sure your servers can handle every eventuality, but much more common is to build for the 95% of the time and just accept that there'll be some times when service will be degraded. Especially when the service is free and not mission critical.

The more recent pauses and load times are different, have a different cause, and not what we want.

Originally Posted by frinkprof

To clarify with the post history searches, I mean the search that is performed when you click on a user's name and the option to view that user's past posts. It used to return, I believe, 5 times the number of results than they do now. Noticed that change a couple weeks ago maybe.

Ah ok, yeah I was reviewing some settings and notice that value was unreasonably high. There's no use in returning huge numbers of posts in such a search; searching by clicking on a user's name will be because you're trying to find a very recent post for that user. No need to have 500 results, even 100 is tons.

If one wants to go back further looking for a post for a user, they're probably looking for something specific so the advanced search feature can be used for that.

Originally Posted by frinkprof

Also, I did indeed PM Bingo with my post above, although I do have to echo the concerns of others above that the owner doesn't peruse the Help/Suggestions section of the forum, especially in light of a persistent technical issue such as this one. Surely you shouldn't be left to answer for everything. I haven't received a reply yet.

Bingo wouldn't have anything more to add to the technical issues, he's already asked me what I need and I've told him it's all already in place and I'm answering for it because I'm the best one to answer for it.

Originally Posted by frinkprof

I'm sure the new servers will be great and all, but that it has gotten to this point (and likely will again), raises some questions.

Well the answer to that question is pretty easy, until sureLoss posted this thread (and other feedback) and I started to do some investigations, I didn't know the performance issues were as significant as they were, so it simply wasn't high on my priority list.

We don't have a dedicated operations team doing load testing and constant monitoring analysis and making 5 year plans for scaling requirements (which is what's required to not get to this point), so there's going to be times when I'm responding instead of being pro-active. At this level, feeling entitled to not getting to this situation isn't reasonable IMO.

I would cut my rate to $125 an hour though if people want to hire me part time to be more pro-active

[MrMastodonFarm]

Originally Posted by photon

We don't have a dedicated operations team doing load testing and constant monitoring analysis and making 5 year plans for scaling requirements (which is what's required to not get to this point), so there's going to be times when I'm responding instead of being pro-active. At this level, feeling entitled to not getting to this situation isn't reasonable IMO.

I would cut my rate to $125 an hour though if people want to hire me part time to be more pro-active

So has Calgarypuck grown beyond what one person can do in his home?

[Minnie]

I know I don't understand enough of the jargon/programming but I do miss a lot of the things that used to be here when I found CP 2 years ago. Removing so much of it doesn't seem to be the issue which irks slightly because I enjoyed those features.

It's been working decent since I posted last, until tonight when I got quite a lot of hang time and then a timed out message. Seems ok now but it farted earlier.

I know there's an upgrade out there as one of my other v-b forums utilized it. It was horrible and it has pretty much killed the forum. They've "downgraded" back to the old version but it's nearly a ghost town there. I rather hope you don't do that particular upgrade because if you think there's bitching now, there will be bitching on LSD if you do the upgrade.

[Skaloper]

I am glad that Photon is working on finding a solution. Once I read up a bit on how v bulletin works I can hopefully be of some help.

Sent from my SGH-I337M using Tapatalk

[JiriHrdina]

Originally Posted by Minnie

I know I don't understand enough of the jargon/programming but I do miss a lot of the things that used to be here when I found CP 2 years ago. Removing so much of it doesn't seem to be the issue which irks slightly because I enjoyed those features.
.

What features are you referring to?

[photon]

Originally Posted by MrMastodonFarm

So has Calgarypuck grown beyond what one person can do in his home?

That depends entirely on one's expectations. I'd say no, since the root cause of the current performance issues don't appear to be growth and my expectations are for best effort and decent performance most of the time.

If one's expectations are that there's never any performance issues ever, guaranteed sub 250ms response times 100% of the time, then it grew beyond that the day it opened.

EDIT: That said I'd be glad to have other people involved.

[photon]

Originally Posted by Minnie

I know there's an upgrade out there as one of my other v-b forums utilized it. It was horrible and it has pretty much killed the forum.

Yeah there's no way I'd upgrade to vBulletin 4 or 5, I hate them. XenForo is the upgrade target.

Originally Posted by Skaloper

I am glad that Photon is working on finding a solution. Once I read up a bit on how v bulletin works I can hopefully be of some help.

Sent from my SGH-I337M using Tapatalk

Haha or maybe read up on XenForo, as that's what I'd want to upgrade to.

[photon]

And one other update, spoke with Vox (where the servers are hosted) and gave him some of the results I'm seeing from running disk benchmarks and he's of the opinion that we shouldn't be seeing the slowdowns we are, so it might be an actual technical problem underneath the servers, so they're looking into that now. That doesn't change the plan to migrate to the new servers.

[MrMastodonFarm]

Thanks again for all the hard work and updates, Photon.

You're the man, never forget it.

[CroFlames]

The updates are appreciated.

[Dion]

Somebody give Photon the winning number to Lotto 6/49. The updates and long hours behind the scenes are greatly appreciated.

[mikephoen]

Photon:

Since (allegedly) Bingo never checks the Help/Suggestions forum anyway, you should set up a paypal donate button that goes directly to you.

[photon]

So got an update from Vox while I was YOLO rebooting the MySQL server (and then had to have Vox's guys actually reboot it for me when it didn't reboot, thanks!)... Apparently there was a network issue that likely was impacting the database server's connection to the storage, and it would have caused random slowdowns and intermittant problems, just like we were seeing.

They think it should be resolved, so will keep an eye on it but hopefully things will improve. If things look stable I'll try and remember to turn some things back on in the morning.

[getbak]

Originally Posted by photon

So got an update from Vox while I was YOLO rebooting the MySQL server and then had to have Vox's guys actually reboot it for me... Apparently there was a network issue that likely was impacting the database server's connection to the storage, and it would have caused random slowdowns and intermittant problems, just like we were seeing.

They think it might be resolved, so will keep an eye on it but hopefully things will improve. If things look stable I'll try and remember to turn some things back on in the morning.

[CroFlames]

I wasn't able to log in for a while at all, but now all seems to be fine. Good work.

[Street Pharmacist]

Working like a charm now!

[zamler]

Site was not responding for about 60 seconds a few minutes ago. My guess is the problem is not a network one but a CPU cycles one, meaning the processor is being overwhelmed (or the server is running out of memory) so request take a very long time to complete.

Mind you this is just a guess but I've seen very similar behaviour to what this forum is experiencing and that was the cause.

[Dion]

Originally Posted by photon

So got an update from Vox while I was YOLO rebooting the MySQL server (and then had to have Vox's guys actually reboot it for me when it didn't reboot, thanks!)... Apparently there was a network issue that likely was impacting the database server's connection to the storage, and it would have caused random slowdowns and intermittant problems, just like we were seeing.

They think it should be resolved, so will keep an eye on it but hopefully things will improve. If things look stable I'll try and remember to turn some things back on in the morning.

Seems to be working fine on this end.

[photon]

Originally Posted by zamler

Site was not responding for about 60 seconds a few minutes ago. My guess is the problem is not a network one but a CPU cycles one, meaning the processor is being overwhelmed (or the server is running out of memory) so request take a very long time to complete.

Mind you this is just a guess but I've seen very similar behaviour to what this forum is experiencing and that was the cause.

That's what's weird about it, when the pauses happen the CPUs are doing nothing, and the memory usage isn't anything different than normal, but the IO utilization for the database server is 100%. If I did some small IO tests at that time I'd get wildly varying throughput, but often in the KB/s rather than MB/s which is very very bad. Then things would smooth out and the disk IO would go down to its normal usage level and transfer rates would go back up to normal.

The machines are VMs and the disks are on a shared storage kind of thing so network problems made/make sense why the transfer rates would be wildly variable.

[photon]

How has it been tonight?