Calgarypuck Forums - The Unofficial Calgary Flames Fan Community
Old 07-03-2017, 04:12 PM   #1
dobbles
addition by subtraction
 
Join Date: Feb 2008
Location: Tulsa, OK
Exp:
Default Need to wipe CentOS web server - questions inside!

Hey guys, I know there's a lot of people here much smarter than me, so looking for a bit of help...

Long story short, the website I run was compromised. We switched over to a backup version of the site and have been running that since. I have a fairly high level of confidence that the production server has been cleaned up, but in an overabundance of caution, I want to wipe it just to be sure.

This is a dedicated machine we run through BlueHost. (yeah I know they suck...) Their people are telling me that I have 2 slightly different options: a total factory reset type option and a 're-imaging.' I had initially asked about the total reset, but we need to maintain the DNS nameserver records that are tied to that account. Because I don't have the ability to coordinate with BH when the reset may occur (I submit a ticket and wait 24-48 hours apparently), I am wary of this option because of the downtime that it will cause. So the tech support guy mentioned a re-image option that would maintain the DNS but still 'reset' the system.

I am trying to Google things, but thought there might be some folks here that could offer advice. I have SSH access to the server and could probably do things myself, but don't want to make things even worse by flying blind.

So can anyone provide me more information on the types of wipes they may be talking about or what my options are? Any overall advice on how to best reset this thing. I am happy to provide more details as needed.
dobbles is offline   Reply With Quote
Old 07-03-2017, 04:30 PM   #2
Rathji
Franchise Player
 
Rathji's Avatar
 
Join Date: Nov 2006
Location: Supporting Urban Sprawl
Exp:
Default

Why don't you switch Name Servers while the process is happening?

Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
Rathji is offline   Reply With Quote
The Following User Says Thank You to Rathji For This Useful Post:
Old 07-03-2017, 04:35 PM   #3
dobbles
addition by subtraction
 
Join Date: Feb 2008
Location: Tulsa, OK
Exp:
Default

I unfortunately don't have the ability to change to different nameservers. The domain recently came under control of the corporate IT department. And I also cannot transfer control between my different BH servers as each has a separate domain tied to the account.

Does that make sense?
dobbles is offline   Reply With Quote
Old 07-03-2017, 04:37 PM   #4
Rathji
Franchise Player
 
Rathji's Avatar
 
Join Date: Nov 2006
Location: Supporting Urban Sprawl
Exp:
Default

You don't have control of the domain registrar? Can't you request the change to be made?

Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
Rathji is offline   Reply With Quote
The Following User Says Thank You to Rathji For This Useful Post:
Old 07-03-2017, 04:59 PM   #5
dobbles
addition by subtraction
 
Join Date: Feb 2008
Location: Tulsa, OK
Exp:
Default

Our company was acquired recently and we had to transfer control of the domain to our new owner. We arent really sure if and when we can get them to give is access or make the change. Was trying to avoid that headache if possible.
dobbles is offline   Reply With Quote
The Following User Says Thank You to dobbles For This Useful Post:
Old 07-03-2017, 05:33 PM   #6
photon
The new goggles also do nothing.
 
photon's Avatar
 
Join Date: Oct 2001
Location: Calgary
Exp:
Default

So I take it to mean that a total reset involves a new IP address from BlueHost (meaning you'd have to change the DNS records for the domain to point to the new server), while a reimaging means you're essentially keeping the same server and they'll just wipe it and reload the bare OS on it?

Is it possible for them to bring up a new server, then you could set it up as necessary, then they could just switch the IPs so the new server has the IP of the old server? Probably not as they probably don't have that kind of functionality in their system but is worth asking.

I guess the question is how much downtime is tolerable.. Does the website content change due to user input? I.e. in an ideal world if you had DNS control, would you still need downtime to migrate a database or set of files over (disable the production site, move the critical files over, then point the DNS at the new server and wait until the DNS change migrates)?

If some downtime is tolerable, then you can minimize it by scripting or automating as much as possible.. start with a blank install of CentOS in a VM and then do everything you need to do to get it running.. make copies of config files so you can copy them into place, create a shell script to run all the commands, etc. Still downtime, but minimizing it.
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
photon is offline   Reply With Quote
The Following User Says Thank You to photon For This Useful Post:
Old 07-04-2017, 06:03 AM   #7
rbochan
Scoring Winger
 
rbochan's Avatar
 
Join Date: May 2008
Location: Syracuse, NY
Exp:
Default

Quote:
Originally Posted by dobbles View Post
...The domain recently came under control of the corporate IT department...
Sound like they should be dealing with the entire situation then.
__________________
...Rob
The American Dream isn't an SUV and a house in the suburbs;
it's Don't Tread On Me.
rbochan is offline   Reply With Quote
The Following 2 Users Say Thank You to rbochan For This Useful Post:
Old 07-04-2017, 03:06 PM   #8
Rathji
Franchise Player
 
Rathji's Avatar
 
Join Date: Nov 2006
Location: Supporting Urban Sprawl
Exp:
Default

Quote:
Originally Posted by rbochan View Post
Sound like they should be dealing with the entire situation then.
Yep.

Either they give you the access to do your job, or they handle it.

Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
Rathji is offline   Reply With Quote
The Following User Says Thank You to Rathji For This Useful Post:
Old 07-04-2017, 03:16 PM   #9
psicodude
Powerplay Quarterback
 
psicodude's Avatar
 
Join Date: Nov 2006
Location: Calgary
Exp:
Default

I'm not a BlueHost or CentOS expert, so I am speaking in generalities here.

A "re-image typically means they are going to start over with the OS and hand the server back to you in the same state as when you first purchased it. IP addresses, DNS, settings, admin access, etc should be left in tact, but you will have a brand new, blank OS likely including Apache or whatever. At this point, you load your website back up and away you go.

Like others have said though, there is no way to avoid some level of downtime in this scenario. If everything goes perfectly, you may only be down a couple of hours. But nothing ever goes perfectly in IT.

What sort of compromise was it? Was it just your website and code that were infected? It's fairly tough for CentOS to be compromised, is why I'm asking.
psicodude is offline   Reply With Quote
The Following 2 Users Say Thank You to psicodude For This Useful Post:
Old 07-04-2017, 05:55 PM   #10
photon
The new goggles also do nothing.
 
photon's Avatar
 
Join Date: Oct 2001
Location: Calgary
Exp:
Default

That's true, usually it only goes so far as the directories the websites are in or stuff that Apache/nginx/whatever has access to. Unless Apache is running as root.
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
photon is offline   Reply With Quote
The Following User Says Thank You to photon For This Useful Post:
Old 07-04-2017, 11:28 PM   #11
dobbles
addition by subtraction
 
Join Date: Feb 2008
Location: Tulsa, OK
Exp:
Default

Thanks for the feedback so far guys.

Re: letting IT handle it - I am trying to make sure I look good for the new company and can handle it myself and not have to pass the buck if possible.

Re: downtime - If DNS can stay up, downtime is not an issue as the server has not been doing anything publicly facing since the incident. I can Do all the work of migrating files and databases while its still hidden and then switch the A record back when ready. Regarding the DNS downtime specifically, my worry isn't so much the amount of time, but mainly that I have no idea of when that time would be. If they are doing it late on a Friday night, I am sure management would not blink an eye if it took hours but if all of a sudden BH decides to work the ticket on a Tuesday afternoon, sales would freak out on me. However, because of DNS propagation, the time could quickly blow up to unacceptable levels regardless.

Re: the hack, and perhaps it would have just been easier to give this info initially so you guys could properly advise - from what we have found via log files, it appears that they brute forced a WP admin account and then used that access to escalate their privileges further. I would have to pull the link from my work machine as there is a pretty good article on how it works that I found. The main part of the attack is they overwrite all JS files with malicious code. However, the one worrying thing that I recall (and I would have to consult screenshots to he 100% sure) is that they were able to create bogus FTP accounts. That was the one part I couldn't figure out as that seems as it would require system access and not what they could gain coming through WP and the file/db side. You guys may know the answer though.

found the article about the hack: https://blog.sucuri.net/2017/04/word...ipt-files.html

edit: also, to save some googling, whats the best way to check what user is running apache? it is not something I knowingly changed, so whatever default BH set me up with is what was used.
dobbles is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT -6. The time now is 08:53 AM.

Calgary Flames
2016-17




Powered by vBulletin® Version 3.8.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright Calgarypuck 2016