Account Suspensions and how to deal with them

[twinscythe12332]

Hey All,

I've been trying to help out someone. Their website got hacked, a particularly nasty page was uploaded as the prominent display, and he told the hosting company to restore everything back to how it was. They have done so... but within a few hours, his site was taken down and replaced by an account suspended notification. Other than the obvious fact that he wouldn't want to tell the world he had hacked his own site, as well as the confirmation of the hosting company that they had put his site back to normal, I'd like to know what sort of leg he has to stand on with regards to taking the matter further.
I'd especially like some advice on how to resolve it as quickly as possible. I've been in discussions with their 24 hour support team, and they've pretty much acknowledged that there is a major problem. Further feedback should have been supplied by this morning, but it seems they aren't quite ready to give out any details.
I don't want to go name dropping yet, because I feel I'd like to give them 24 hours and the benefit of the doubt before raging.
I'm looking for any similar experiences and how you interacted with the hosting company to get your site back up and running.

Thanks

[Dave A]

It's not enough just to restore a backup - you should also change all the usernames and passwords with admin, ftp or database access privileges. Wouldn't hurt to force a change of password on any hosted email accounts while you're about it.

If it's a script flaw that's enabling the hacking, taking the site down until the flaw is resolved probably is a good idea.

I would expect the hosting company would take a close look at the log files to identify the source of the problem (provided you're on supported hosting, of course).

[twinscythe12332]

The support guy we spoke to pointed towards a possible security flaw, but he wasn't entirely certain. The support team has said they will be emailing him stuff from the technical support team, but he still hasn't received anything. He has been dealing with this group for years, and this is the first time they've ever been slow to respond. This makes me think it's a little more than simply an outdated version of joomla!
I'll hopefully be able to get something going this evening or tomorrow, especially the password changes like you have said.

[Dave A]

This makes me think it's a little more than simply an outdated version of joomla!

Just how outdated?

[SilverNodashi]

Hey All,

I've been trying to help out someone. Their website got hacked, a particularly nasty page was uploaded as the prominent display, and he told the hosting company to restore everything back to how it was. They have done so... but within a few hours, his site was taken down and replaced by an account suspended notification. Other than the obvious fact that he wouldn't want to tell the world he had hacked his own site, as well as the confirmation of the hosting company that they had put his site back to normal, I'd like to know what sort of leg he has to stand on with regards to taking the matter further.
I'd especially like some advice on how to resolve it as quickly as possible. I've been in discussions with their 24 hour support team, and they've pretty much acknowledged that there is a major problem. Further feedback should have been supplied by this morning, but it seems they aren't quite ready to give out any details.
I don't want to go name dropping yet, because I feel I'd like to give them 24 hours and the benefit of the doubt before raging.
I'm looking for any similar experiences and how you interacted with the hosting company to get your site back up and running.

Thanks

Website security is your, or your client's responsibility - it's like a credit card. The bank has vaults, keycard access to their premises, security guards, armed patrol & response, etc. They give you a credit card with a PIN and signature - this is your responsibility, and no amount of security they apply on their side can actually keep your card safe if you type your PIN in full view of other people.


The same goes with website hosting. The ISP / hosting company will have firewalls and many other security measures in place to protect their network, servers, data on the servers, etc. But, if you have an insecure password, or use the same username & password all over the internet, or have outdated & insecure scripts on your website then it's your own fault if it gets hacked. This may sound harsh, but you need to take responsibility for your own property.


You should ask for a backup of the website, and any related logs and scan thoroughly through them.
Change ALL passwords.
don't use the same username & password as being used on forums / blogs / twitter / facebook / etc.
don't use a recognizable username, i.e. something that could tie to the owner or company.
user strong passwords. Check this out: http://howsecureismypassword.net/
Follow the developer's recommended security measures.
google and see if other people has recommendations for additional security measures to take.
Make a backup of your website, at least once a week and keep 2 or 3 different copies (i.e. 3 weeks in a row) in-case you need to go back to a previous version, after making changes which could have led to the hacking attempt.

Just cause our neighborhood has armed partol, security booms, CCTV on the major corners and even neighbor watch doesn't mean I can leave my gate & doors wide open. When someone decides to walk in and rob us, it's my own fault.

[twinscythe12332]

I'm with you on the reasons for the site being hacked SoftDux, 100%, as well as where the blame lies.
Imagine that the Neighbourhood patrol has boarded up the house, lost the spare key, denies access to the CCTV footage and has arrested you as the criminal. That's the situation he now faces. If we manage to get the site back up and running, I'm locking it down tighter than a nun. I'm also in the process of training up the poor oke. His dev uploaded the site, explained nothing and just left him to the wolves. By the end of this, I hope to have him a little more tech and web savvy.

[SilverNodashi]

I'm with you on the reasons for the site being hacked SoftDux, 100%, as well as where the blame lies.
Imagine that the Neighbourhood patrol has boarded up the house, lost the spare key, denies access to the CCTV footage and has arrested you as the criminal.

Surely you should be able to identify yourself as a resident?


But, most hosting providers aren't very helpful in this case since they just presume clients know-it-all.

He should still however get a backup of the game

[twinscythe12332]

We've identified ourselves as a resident (this metaphor gets more elaborate as it goes on, doesn't it =P), so much so that at least half of the support center know us by now. His backup was apparently "corrupted", even though it worked perfectly fine the last time. Ah, the fickleness of 1's and 0's...Anyway, it looks like the environment will need to be reset, emails lost and site files deleted. We're holding off on this until we can locate a pproper backup.

Another analogy for this: I've been called to a train wreck. There are no spare coaches, and I have to put the train together again... or build a new one.

[twinscythe12332]

The hosting company came to the party. They acknowledged that he wasn't the hacker, but the hacked. They cleaned up his site, removed any of the scripts that were causing issues and let the site run. All the necessary passwords have been changed (and even some that weren't entirely necessary), and the site is live again. The scariest thing were some of the passwords used by the developer... I'll leave that to your imagination. I've been teaching him how to make backups, what a safer password is, and to try not to use the same password everywhere. I think this was a bit of a scare, and that's often enough to get people on board with the security mindset.

[SilverNodashi]

This thread has made me think about our own strategies and I would like some input to see if we can improve it to a point where both us and the client is happy about the outcome.


As a side note: One of our datacenters in the USA will send out an email to me if they find anything suspicious on our servers (like reported spam of phishing sites, etc) and literally nullroute the offending IP address of the server withing 24hours of sending out the email Regardless of which day of the year they send the mail - public holiday or not. I haven't received anything over Christmas / New Years yet, so I don't know what would have happened...... Anyhow once the IP has been nullrouted (i.e. it's totally unaccessible on the network or internet they fine me $1000USD to have it de-listed and give me 2hours to fix the problem. The whole server then gets decommissioned 36hours later. No questions asked, no backups kept (they delete that as well).


So, now we have the takedown notice, and 24hours to respond.

We then immediately send out an email with a full report giving the client upto 1hour less than when they will nullroute the IP to get it fixed - just for some leeway. An automated script will suspend the account at that time, unless we hear back from the client at which point we disable the automated suspension.

We kindly also tell the client what recommended security steps to take and most of the times the client resolves the issue right away. Just about every case which I can remember involved an out of date insecure web script or poor admin password.


The most recent incident was were a client had an old Wordpress website, which was a demo for one of their clients but never used that somehow produced an email email everytime a certain page was visited, dumping 35MB's worth of data (at which point the server cuts it off) to themselves, and generated 113GB's worth of emails. It took us a few hours to find this one, purely cause it took us about 3hours to contain the sudden rush of mail. The script literally sent out an error email every second, but back to the local user. At first we thought the server was under DDOS attack but couldn't see anything on the routers or firewalls.

I have chosen not to phone the clients, purely cause one client kept a tech on the line for 7 hours during that time forcing the tech to tell the client over the phone how to fix a coding error on his website. Oh, and off course blaming us for having insecure servers. The client in question had many websites, all of which were hacked and phishing sites uploaded to. Only his accounts were hacked. For this reason we don't phone clients for this kind of thing anymore.


Internet Solutions actually told me over the phone one day I have 1hour to have a solution fixed



So my question is: how much grace do you give a client, who's website(s) is directly affecting you server, network and other clients?
How long do you allow his hacked account to deny service to other clients on the same server, affecting their productivity and business as well?

[Dave A]

I have no problem with the hosting company suspending the account until the problem is resolved.

Where things get tricky (and you've said this before, Rudi) - often the client doesn't have the skills to identify what is causing the problem, or to resolve the problem, and at times doesn't have access to the tools needed either.

Just how far do you go as the hosting company to help resolve the problem?

Well, if the client is on "supported hosting", I think you need to do what it takes to secure the site. Not patch the code, perhaps, but identify the user profile that's been hacked / shut down email sending / change insecure folder write and execute permissions...

I had a redundant sub-account hacked earlier this year on an overseas server. The hosting company identified the hacked profile, disabled the user, and sent me an email telling me what they had done. In this case I simply deleted the user profile as the site had been moved elsewhere.

Obviously there are a number of potential scenarios. I suggest as a bare minimum, the host should bundle the site into a backup that is made available to the client to download. Just deleting the whole account (even if the onus is on the client to maintain their own backups) simply isn't on i.m.o.

[SilverNodashi]

I have no problem with the hosting company suspending the account until the problem is resolved.

Where things get tricky (and you've said this before, Rudi) - often the client doesn't have the skills to identify what is causing the problem, or to resolve the problem, and at times doesn't have access to the tools needed either.

So, how would you suggest we rather approach this problem? i.e. what would you rather expect your host do for you, or try and do for you if you're in this situation?

And I'm seriously looking for some useful input in this regard. let's help make life easier together.

Just how far do you go as the hosting company to help resolve the problem?

I was up till 2am this morning helping a client with exactly this kind of situation. She notified me yesterday morning that one of her sites got hacked. I got to bed after 2am - my family got about zero time with me. supper was behind the laptop. And this is not an isolated case, this is how it goes.

But, are (hosts) responsible for client's hacked websites, due to their lack of knowledge in this field?


the hosts who don't help their clients are labeled as bad hosts - the internet is scattered with stories about hosts who shutdown clients sites with no help at all.



On a different note, just like you I run a business and I need to pay someone to support the clients. Support, especially in our industry doesn't make any money at all. It costs money to have the techs, office furniture, PC's, phones, etc. And while one can argue that if there was no support, the equipment will fail and the clients will move on, one can also argue that if the client knew how to operate the equipment properly support would be less of a burden. Surely if everyone who wanted to have a website actually spend some time to learn how the internet works, how websites work, how email works, etc their own work would have been more productive?
There's a reason why internet in our country is so expensive - clients think that the ISP should be their general IT support department as well, all for R15/pm.

Well, if the client is on "supported hosting", I think you need to do what it takes to secure the site. Not patch the code, perhaps, but identify the user profile that's been hacked / shut down email sending / change insecure folder write and execute permissions...

But where do you draw the line? Would you pay more for a "full support hosting account", than a no "support hosting account"? Be honest. Would you rather pay R20/pm, or R200pm for a hosting account?


How many people do you think would really add a monitory value to support?

I had a redundant sub-account hacked earlier this year on an overseas server. The hosting company identified the hacked profile, disabled the user, and sent me an email telling me what they had done. In this case I simply deleted the user profile as the site had been moved elsewhere.

We do this as well, but in your case you're probably on a VPS or dedicated server and could take action yourself. Most websites on the internet is on a shared server and the client doesn't have direct access to disable, or delete a user account like this.

their account is either active ( and causing havoc on the internet) or disabled.

Obviously there are a number of potential scenarios. I suggest as a bare minimum, the host should bundle the site into a backup that is made available to the client to download. Just deleting the whole account (even if the onus is on the client to maintain their own backups) simply isn't on i.m.o.

We give the clients the option to download a backup of the website. But how do you download a 8GB account?



I'm very open for suggestions, and discussion on this one. But I sometimes also think that clients need to remember that the supplier is also just another person, not a robot.

[Dave A]

But where do you draw the line? Would you pay more for a "full support hosting account", than a no "support hosting account"? Be honest. Would you rather pay R20/pm, or R200pm for a hosting account?

I've always paid the extra for a "supported hosting" package. Generally the cost isn't that much extra - at least once we get past the oversellers which should be avoided anyway the moment you realise you've got a site with some traction.

Let's talk minimum standards for a moment.

I suggest, even if you are an overseller doing $5 per month hosting, the very least you can do instead of just deleting an account is backup the account for transfer and store the file before shutting the account down. Better still, add a db and file backup as options for the client to download.

The next thing is at least take a look at the log files for anything obvious. Most shared hosting web site owners haven't got a clue what they're seeing when they look at a log file - if they've even got access to them. If it's a high volume exploit, the script call should stand out and you know where to start looking. From there you can at least point the client in the right direction.