Need to wipe CentOS web server - questions inside!

[dobbles]

Hey guys, I know there's a lot of people here much smarter than me, so looking for a bit of help...

Long story short, the website I run was compromised. We switched over to a backup version of the site and have been running that since. I have a fairly high level of confidence that the production server has been cleaned up, but in an overabundance of caution, I want to wipe it just to be sure.

This is a dedicated machine we run through BlueHost. (yeah I know they suck...) Their people are telling me that I have 2 slightly different options: a total factory reset type option and a 're-imaging.' I had initially asked about the total reset, but we need to maintain the DNS nameserver records that are tied to that account. Because I don't have the ability to coordinate with BH when the reset may occur (I submit a ticket and wait 24-48 hours apparently), I am wary of this option because of the downtime that it will cause. So the tech support guy mentioned a re-image option that would maintain the DNS but still 'reset' the system.

I am trying to Google things, but thought there might be some folks here that could offer advice. I have SSH access to the server and could probably do things myself, but don't want to make things even worse by flying blind.

So can anyone provide me more information on the types of wipes they may be talking about or what my options are? Any overall advice on how to best reset this thing. I am happy to provide more details as needed.

[Rathji]

Why don't you switch Name Servers while the process is happening?

Sent from my Nexus 6 using Tapatalk

[dobbles]

I unfortunately don't have the ability to change to different nameservers. The domain recently came under control of the corporate IT department. And I also cannot transfer control between my different BH servers as each has a separate domain tied to the account.

Does that make sense?

[Rathji]

You don't have control of the domain registrar? Can't you request the change to be made?

Sent from my Nexus 6 using Tapatalk

[dobbles]

Our company was acquired recently and we had to transfer control of the domain to our new owner. We arent really sure if and when we can get them to give is access or make the change. Was trying to avoid that headache if possible.

[photon]

So I take it to mean that a total reset involves a new IP address from BlueHost (meaning you'd have to change the DNS records for the domain to point to the new server), while a reimaging means you're essentially keeping the same server and they'll just wipe it and reload the bare OS on it?

Is it possible for them to bring up a new server, then you could set it up as necessary, then they could just switch the IPs so the new server has the IP of the old server? Probably not as they probably don't have that kind of functionality in their system but is worth asking.

I guess the question is how much downtime is tolerable.. Does the website content change due to user input? I.e. in an ideal world if you had DNS control, would you still need downtime to migrate a database or set of files over (disable the production site, move the critical files over, then point the DNS at the new server and wait until the DNS change migrates)?

If some downtime is tolerable, then you can minimize it by scripting or automating as much as possible.. start with a blank install of CentOS in a VM and then do everything you need to do to get it running.. make copies of config files so you can copy them into place, create a shell script to run all the commands, etc. Still downtime, but minimizing it.

[rbochan]

Originally Posted by dobbles

...The domain recently came under control of the corporate IT department...

Sound like they should be dealing with the entire situation then.

[Rathji]

Originally Posted by rbochan

Sound like they should be dealing with the entire situation then.

Yep.

Either they give you the access to do your job, or they handle it.

Sent from my Nexus 6 using Tapatalk

[psicodude]

I'm not a BlueHost or CentOS expert, so I am speaking in generalities here.

A "re-image typically means they are going to start over with the OS and hand the server back to you in the same state as when you first purchased it. IP addresses, DNS, settings, admin access, etc should be left in tact, but you will have a brand new, blank OS likely including Apache or whatever. At this point, you load your website back up and away you go.

Like others have said though, there is no way to avoid some level of downtime in this scenario. If everything goes perfectly, you may only be down a couple of hours. But nothing ever goes perfectly in IT.

What sort of compromise was it? Was it just your website and code that were infected? It's fairly tough for CentOS to be compromised, is why I'm asking.

[photon]

That's true, usually it only goes so far as the directories the websites are in or stuff that Apache/nginx/whatever has access to. Unless Apache is running as root.

[dobbles]

Thanks for the feedback so far guys.

Re: letting IT handle it - I am trying to make sure I look good for the new company and can handle it myself and not have to pass the buck if possible.

Re: downtime - If DNS can stay up, downtime is not an issue as the server has not been doing anything publicly facing since the incident. I can Do all the work of migrating files and databases while its still hidden and then switch the A record back when ready. Regarding the DNS downtime specifically, my worry isn't so much the amount of time, but mainly that I have no idea of when that time would be. If they are doing it late on a Friday night, I am sure management would not blink an eye if it took hours but if all of a sudden BH decides to work the ticket on a Tuesday afternoon, sales would freak out on me. However, because of DNS propagation, the time could quickly blow up to unacceptable levels regardless.

Re: the hack, and perhaps it would have just been easier to give this info initially so you guys could properly advise - from what we have found via log files, it appears that they brute forced a WP admin account and then used that access to escalate their privileges further. I would have to pull the link from my work machine as there is a pretty good article on how it works that I found. The main part of the attack is they overwrite all JS files with malicious code. However, the one worrying thing that I recall (and I would have to consult screenshots to he 100% sure) is that they were able to create bogus FTP accounts. That was the one part I couldn't figure out as that seems as it would require system access and not what they could gain coming through WP and the file/db side. You guys may know the answer though.

found the article about the hack: https://blog.sucuri.net/2017/04/word...ipt-files.html

edit: also, to save some googling, whats the best way to check what user is running apache? it is not something I knowingly changed, so whatever default BH set me up with is what was used.

[dobbles]

Update, so I figured out why I was getting the cold shoulder from everyone and having to fend for myself on this.... turns out I was on the list to get downsized. Half my dept got the axe not too long after my last post.

Good news is I had a new job in about a week and 15 days from getting let go I had already started the new gig!

[photon]

Haha congratz!