07-03-2017, 04:12 PM
|
#1
|
addition by subtraction
Join Date: Feb 2008
Location: Tulsa, OK
|
Need to wipe CentOS web server - questions inside!
Hey guys, I know there's a lot of people here much smarter than me, so looking for a bit of help...
Long story short, the website I run was compromised. We switched over to a backup version of the site and have been running that since. I have a fairly high level of confidence that the production server has been cleaned up, but in an overabundance of caution, I want to wipe it just to be sure.
This is a dedicated machine we run through BlueHost. (yeah I know they suck...) Their people are telling me that I have 2 slightly different options: a total factory reset type option and a 're-imaging.' I had initially asked about the total reset, but we need to maintain the DNS nameserver records that are tied to that account. Because I don't have the ability to coordinate with BH when the reset may occur (I submit a ticket and wait 24-48 hours apparently), I am wary of this option because of the downtime that it will cause. So the tech support guy mentioned a re-image option that would maintain the DNS but still 'reset' the system.
I am trying to Google things, but thought there might be some folks here that could offer advice. I have SSH access to the server and could probably do things myself, but don't want to make things even worse by flying blind.
So can anyone provide me more information on the types of wipes they may be talking about or what my options are? Any overall advice on how to best reset this thing. I am happy to provide more details as needed.
|
|
|
07-03-2017, 04:30 PM
|
#2
|
Franchise Player
Join Date: Nov 2006
Location: Supporting Urban Sprawl
|
Why don't you switch Name Servers while the process is happening?
Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
|
|
|
The Following User Says Thank You to Rathji For This Useful Post:
|
|
07-03-2017, 04:35 PM
|
#3
|
addition by subtraction
Join Date: Feb 2008
Location: Tulsa, OK
|
I unfortunately don't have the ability to change to different nameservers. The domain recently came under control of the corporate IT department. And I also cannot transfer control between my different BH servers as each has a separate domain tied to the account.
Does that make sense?
|
|
|
07-03-2017, 04:37 PM
|
#4
|
Franchise Player
Join Date: Nov 2006
Location: Supporting Urban Sprawl
|
You don't have control of the domain registrar? Can't you request the change to be made?
Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
|
|
|
The Following User Says Thank You to Rathji For This Useful Post:
|
|
07-03-2017, 04:59 PM
|
#5
|
addition by subtraction
Join Date: Feb 2008
Location: Tulsa, OK
|
Our company was acquired recently and we had to transfer control of the domain to our new owner. We arent really sure if and when we can get them to give is access or make the change. Was trying to avoid that headache if possible.
|
|
|
The Following User Says Thank You to dobbles For This Useful Post:
|
|
07-03-2017, 05:33 PM
|
#6
|
The new goggles also do nothing.
Join Date: Oct 2001
Location: Calgary
|
So I take it to mean that a total reset involves a new IP address from BlueHost (meaning you'd have to change the DNS records for the domain to point to the new server), while a reimaging means you're essentially keeping the same server and they'll just wipe it and reload the bare OS on it?
Is it possible for them to bring up a new server, then you could set it up as necessary, then they could just switch the IPs so the new server has the IP of the old server? Probably not as they probably don't have that kind of functionality in their system but is worth asking.
I guess the question is how much downtime is tolerable.. Does the website content change due to user input? I.e. in an ideal world if you had DNS control, would you still need downtime to migrate a database or set of files over (disable the production site, move the critical files over, then point the DNS at the new server and wait until the DNS change migrates)?
If some downtime is tolerable, then you can minimize it by scripting or automating as much as possible.. start with a blank install of CentOS in a VM and then do everything you need to do to get it running.. make copies of config files so you can copy them into place, create a shell script to run all the commands, etc. Still downtime, but minimizing it.
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
|
|
|
The Following User Says Thank You to photon For This Useful Post:
|
|
07-04-2017, 06:03 AM
|
#7
|
Scoring Winger
Join Date: May 2008
Location: Syracuse, NY
|
Quote:
Originally Posted by dobbles
...The domain recently came under control of the corporate IT department...
|
Sound like they should be dealing with the entire situation then.
__________________
...Rob
The American Dream isn't an SUV and a house in the suburbs;
it's Don't Tread On Me.
|
|
|
The Following 2 Users Say Thank You to rbochan For This Useful Post:
|
|
07-04-2017, 03:06 PM
|
#8
|
Franchise Player
Join Date: Nov 2006
Location: Supporting Urban Sprawl
|
Quote:
Originally Posted by rbochan
Sound like they should be dealing with the entire situation then.
|
Yep.
Either they give you the access to do your job, or they handle it.
Sent from my Nexus 6 using Tapatalk
__________________
"Wake up, Luigi! The only time plumbers sleep on the job is when we're working by the hour."
|
|
|
The Following User Says Thank You to Rathji For This Useful Post:
|
|
07-04-2017, 03:16 PM
|
#9
|
First Line Centre
Join Date: Nov 2006
Location: Calgary
|
I'm not a BlueHost or CentOS expert, so I am speaking in generalities here.
A "re-image typically means they are going to start over with the OS and hand the server back to you in the same state as when you first purchased it. IP addresses, DNS, settings, admin access, etc should be left in tact, but you will have a brand new, blank OS likely including Apache or whatever. At this point, you load your website back up and away you go.
Like others have said though, there is no way to avoid some level of downtime in this scenario. If everything goes perfectly, you may only be down a couple of hours. But nothing ever goes perfectly in IT.
What sort of compromise was it? Was it just your website and code that were infected? It's fairly tough for CentOS to be compromised, is why I'm asking.
|
|
|
The Following 2 Users Say Thank You to psicodude For This Useful Post:
|
|
07-04-2017, 05:55 PM
|
#10
|
The new goggles also do nothing.
Join Date: Oct 2001
Location: Calgary
|
That's true, usually it only goes so far as the directories the websites are in or stuff that Apache/nginx/whatever has access to. Unless Apache is running as root.
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
|
|
|
The Following User Says Thank You to photon For This Useful Post:
|
|
07-04-2017, 11:28 PM
|
#11
|
addition by subtraction
Join Date: Feb 2008
Location: Tulsa, OK
|
Thanks for the feedback so far guys.
Re: letting IT handle it - I am trying to make sure I look good for the new company and can handle it myself and not have to pass the buck if possible.
Re: downtime - If DNS can stay up, downtime is not an issue as the server has not been doing anything publicly facing since the incident. I can Do all the work of migrating files and databases while its still hidden and then switch the A record back when ready. Regarding the DNS downtime specifically, my worry isn't so much the amount of time, but mainly that I have no idea of when that time would be. If they are doing it late on a Friday night, I am sure management would not blink an eye if it took hours but if all of a sudden BH decides to work the ticket on a Tuesday afternoon, sales would freak out on me. However, because of DNS propagation, the time could quickly blow up to unacceptable levels regardless.
Re: the hack, and perhaps it would have just been easier to give this info initially so you guys could properly advise - from what we have found via log files, it appears that they brute forced a WP admin account and then used that access to escalate their privileges further. I would have to pull the link from my work machine as there is a pretty good article on how it works that I found. The main part of the attack is they overwrite all JS files with malicious code. However, the one worrying thing that I recall (and I would have to consult screenshots to he 100% sure) is that they were able to create bogus FTP accounts. That was the one part I couldn't figure out as that seems as it would require system access and not what they could gain coming through WP and the file/db side. You guys may know the answer though.
found the article about the hack: https://blog.sucuri.net/2017/04/word...ipt-files.html
edit: also, to save some googling, whats the best way to check what user is running apache? it is not something I knowingly changed, so whatever default BH set me up with is what was used.
|
|
|
09-06-2017, 08:53 PM
|
#12
|
addition by subtraction
Join Date: Feb 2008
Location: Tulsa, OK
|
Update, so I figured out why I was getting the cold shoulder from everyone and having to fend for myself on this.... turns out I was on the list to get downsized. Half my dept got the axe not too long after my last post.
Good news is I had a new job in about a week and 15 days from getting let go I had already started the new gig!
|
|
|
09-06-2017, 09:07 PM
|
#13
|
The new goggles also do nothing.
Join Date: Oct 2001
Location: Calgary
|
Haha congratz!
__________________
Uncertainty is an uncomfortable position.
But certainty is an absurd one.
|
|
|
Posting Rules
|
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts
HTML code is Off
|
|
|
All times are GMT -6. The time now is 05:21 AM.
|
|