Ridiculous long load times during games

[sureLoss]

Website is extremely slow during games. Getting worse every game.

[Jbo]

Agreed. Snails pace.

[photon]

I wasn't around to take a look at things but Vox says that something was using up 100% of the database server for a while which is unusual.

I have a couple of new VMs almost ready to host the site but just haven't had any spare time to finish the setup and do some testing... it should alleviate things quite a bit. If anyone knows someone trustworthy who would volunteer their time to help out with finishing setting up the new servers and migrating over let me know. Not sure how to vet someone though, considering there's a lot of personal information in the database.

[sworkhard]

Originally Posted by photon

I wasn't around to take a look at things but Vox says that something was using up 100% of the database server for a while which is unusual.

I have a couple of new VMs almost ready to host the site but just haven't had any spare time to finish the setup and do some testing... it should alleviate things quite a bit. If anyone knows someone trustworthy who would volunteer their time to help out with finishing setting up the new servers and migrating over let me know. Not sure how to vet someone though, considering there's a lot of personal information in the database.

Sent you a PM. I can probably help you out in the second half of next week if nobody else is able to before then.

[photon]

Replied.

How was it tonight? There's over 1000 on after the game and it seems good.

I hopefully made Thanks a little better, and removed some more very large threads that were really old but still being viewed by search engines which drags things down.

[photon]

Is there a link to finding user thanks I'm not seeing? There's a thanks search that's running that's eating up a lot of database resources, it's almost like a search engine as it seems to come and go... Man this plugin is bad.

[psyang]

Originally Posted by photon

Is there a link to finding user thanks I'm not seeing? There's a thanks search that's running that's eating up a lot of database resources, it's almost like a search engine as it seems to come and go... Man this plugin is bad.

I found the link to find all thanks commented out in the html. Just tried browsing to it (after I read your post above - the original wasn't me) and it works.

It's conceivable that one or more people have that link bookmarked. Might be a good idea to actually disable the link completely instead of just disabling access to the link through the ui.

[photon]

Yeah, and the search engines probably follow it too even if it is commented out. Good find, thanks!

[Da_Chief]

Site has been super slow all day.

[SportsJunky]

Is the recent increase in duplicate posts connected to the long load times I wonder?

[photon]

Probably. There's a couple of different things going on I think. One is I used something to log incoming requests and I think a lot of the load is due to search engines.

I watched the top IP addresses making requests to the forum for a while and a number of the top IPs belong to search engines I've never heard of nor care about. Yandex or Omgili or ahrefs.

I added a few of those to the robots.txt to hopefully stop them from indexing the site, search engine bots have a different pattern of usage, they dig up really old threads and stuff, so it really impacts the site much more than a typical user who usually stays in the current threads.

I also noticed a lot of activity from a couple of Netdna IP addresses, but the user agents showed regular stuff like Firefox or an iPad or whatever, and the IP addresses correlated to a number of actual users, so I wonder if it's some kind of proxy or anonymizer. Users using proxies to register is kind of suspicious and I usually deny the registration but this one is big enough with enough "legit" users I wonder if I should talk to a few of the users to find out what's going on there. I don't want to block something that's legit traffic use.

Anyway, blocked a few of the current top offending search engines so hopefully that'll improve over time, and I've seen a few lists for common search engines to block so I might expand that list at the web server level.

EDIT: Bing is pretty aggressive too

The second thing that seems to happen is every so often, on no schedule I can figure out, all the web server processes just go away... or rather the master process just stops making new ones. So things'll not respond for a while, 15-30 seconds, then it'll start again and of course there's a mini-rush which makes things slower as well, then it's all fine again. Have never been able to figure out what causes it.

EDIT: I blocked Yandex and ahrefs bots at the web server level and already am seeing a pretty decent decrease in incoming bot requests.

[ResAlien]

It's been pretty bad the last couple of days with time outs and the forum not loading at all. I'm mostly on Chrome mobile. The delays have been intermittent it seems, no real rhyme or reason.

[photon]

I also found a weird thing in the logs and it may have been related to some logging that was installed but wasn't working, I've removed that. Just making a note here to remind myself to check the logs after a few days to see if it's resolved what I was seeing.

[Minnie]

Originally Posted by ResAlien

It's been pretty bad the last couple of days with time outs and the forum not loading at all. I'm mostly on Chrome mobile. The delays have been intermittent it seems, no real rhyme or reason.

Also having these issues for days now. Doesn't seem to matter which browser or device I'm on.

[Joborule]

Ya, it's really randomized. Just occured to me now.

[sun]

Load times are brutal right now on a Thursday morning.

Glad to see that photon is on the case!

[photon]

Yeah it just happened again for me.

Another candidate, though I'm not sure if it's a watched threads or just a plain search that's causing it, but searching might be the issue.

It might be worth a trail to turn searching off for a few days and see if that makes things substantially better or not, just to confirm that that's the cause.

[photon]

Might not be search, just happened to me again and this time I didn't see any searches in the slow query log, nothing obvious, but the database server was basically using 100% of its disk IO.

[White Out 403]

Load times are pretty bad right now

[V]

Man, Bingo must be choked, he finally turns this site into a real revenue machine, and now he can't get it to run properly. He obviously pays photon too much.