The first thing we need to understand is what is being attacked; as the post subject implies, it is primarily Joomla being attacked as the software has had a series of 9 vulnerabilities released since the 1st of September of which a number of more in depth attacks have formed around. The intended purpose of most of these attacks is to taint web sites with injected javascript, that code takes advantage of a number of client side browser vulnerabilities that if not patched or stopped by an antivirus can cause further issues for web site visitors.

Now, at a glance you might be thinking that if someone fails to patch web site software then it is there own problem, how does this affect me? That is where the dl() function comes into play, the dl() function is essentially a dynamic loader for php modules or 3rd party extensions. To simplify this a bit, the dl() function when enabled allows anyone to add extensible features onto php, generally these are all well to do features but if someone so desires they can create a dynamic loader module with malicious intent.

The scenario we are looking at is that attackers have gained entry to vulnerable web sites, primarily through joomla then they upload a series of malicious scripts including a dynamic loadable module for php that once enabled through dl() has the ability to inject javascript code into pages. The code usually finds itself placed before the body tags and executes its payload on a visitors first visit to a site, a cookie is then set that expires every 2hours then the payload executes itself again on a new visit.

This attack though had far reaching implications, only affected 4 servers on our network (denver, dantooine, alderaan, chewbacca) of which only about half the sites on the given servers or in some cases less were being tainted by the attack. As alarming as this situation is, we need to stress that no content was actually modified on sites except the joomla sites themselves that were compromised.

The way we have come to deal with this situation is a layered approach, we have first and foremost made increased efforts to identify compromised sites on our servers and suspend/remove them. The next step was to cut off the enabling function of the attack, which is the dl() function. This function was actually something we used to disable on servers for its malicious implications but over time that procedure was phased out in the interest of allowing users to install custom dynamic loadable modules from their home directories such as ioncube. However, now that ioncube is standard server-wide on all servers, there is little in the way of other commonly installed packages that depend on dl(), php.net has even went as far as to declare dl() deprecated as of php 5.3.

With dl() disabled on servers, the effects were immediate and all reports of tainted sites stopped, now when I say stopped I do not just mean that that lightly. We literally sat around all evening bashing the f5 key on our keyboards trying to get the javacode injections to reappear on sites, between myself, Bill and Dick we must have done over 6 hours of combined keyboard kungfu in this effort. It was with great relief that we were not seeing anymore reports or issues ourselves first hand but it was still not quite enough to actually be confident that we had done enough.

We are continuing to be extra vigilant with compromise assessment on the servers to prevent any further malicious content from being injected into sites, in addition to this we have on some servers started to use suPHP as a basis for new php security standards. Essentially, by using suPHP we enforce php code to run as the user who executed it instead of as the web server but it goes beyond that by enforcing strict permissions on content and not allowing anything to run above mode 755 (such as world writable data) and also making sure that executed content is owned by the user. This might seem problematic however since the code is now executing as the user, there is no longer a need for data to be set to mode 777 (world writable) or its ownership set as the web server user, which reduces support issues and vastly increases security. The suPHP changes are something we have only rolled out to about 6 servers so far but the support issues it has generated are minimal for the advantages it provides, in the future we will be looking to roll this change out to more servers on a slow but steady basis.

That is where we are at, if you have any questions or concerns regarding this blog or the topics discussed please feel free to comment or head to the TCH forums for further dialog.

Here at TCH we operate a network that is dominated by Linux, to say we took this matter very seriously would be an understatement. It was decided after evaluating the threat this vulnerability poses to our network, dedicated servers, and shared/reseller clients, that waiting any longer on an upstream update was not reasonable. Originally there was an estimate of Saturday 1900GMT for upstream updates but this fell through prompting us to take action. In addition to a lack of a reliable upstream update for this issue is the fact that this vulnerability is being actively exploited in the wild with publicly available attack code on many security and underground web sites.

At this moment, we are rolling out to all Linux servers on our network an updated kernel version that will close this vulnerability while maintaining version compatibility with future upstream software updates. This effort in retaining version support will allow our dedicated clients in addition to our own support team to resume normal update practices with tools such as ‘yum’ or ‘apt-get’ and not have to worry about conflicting versions against our in-house kernel update.

Please do not be alarmed if you experience an outage temporarily on dedicated, shared or reseller servers, we thank everyone for understanding the urgency of this matter and if you have any questions or comments please feel free to submit a help desk ticket at https://www.tchhelp.com.

UPDATE:Aug 18, 2009
We will be conducting reboots again this evening to push out a revised version of last nights kernel that corrects issues with r1backup agent, local firewall services and the network driver on certain servers. In addition, this new kernel revision is binary compatible with CentOS/RHEL 4 kernels being that it was built off the same kernel source tree as the standard kernels.

Update: Authorize.net is now back online. You will be able to make payments. We will update you if this or any other changes arise.

We are sorry for any inconvenience this causes you.

At this time we do not have a estimated time for when normal network conditions will return.

We will keep this site updated as this unplanned outage moves along.

We are very sorry for this issue and are working to correct the networking issues as quickly as possible.

Thank you for your understanding during this outage.

TotalChoice Hosting

April 24, 2009 4:24 PM
All services have now been returned to normal. We had a core router crash due to a failed supervisor card, however it did not switch over smoothly. The router was brought back online, switched to the backup supervisor card, and then a replacement card was installed.

We apologize for the inconvenience and thank you for your patience.

As a Team Mom/Mother of three little leaguers and an avid Astros’ fan, I know this gnawing urge for a WIN all to well. I have seen the hurt of defeat in a child’s eyes and the smile of victory on their faces. I’ve watched as they have picked themselves up out of the dust just as the Umpire shouts out “Safe”. I’ve followed the ball as it flew into outfield and fell perfectly into the Left fielders glove, when they were only inches away from second base. They have learned to overcome the losses and bask in the glory of the game that is Baseball.

When they are not on the field themselves, we are perched in the stands of  Minute Maid Park, the home of Houston Astros’. We are a loyal Astros’ family, and attend as many games as we can. Each year we spend Mother’s and Father’s day with these guys. Today, April 6, 2009, is Opening Day of the 2009 Series. The Astros’ will go head to head with The Chicago Cubs.

Roy Oswalt, who has been with the Astros’ since 2001, will be pitching the opening game. This year’s 40 man roster will consist of new and old faces of the organization. Lance Berkman and Hunter Pence, my favorites, will be with us once again and are currently part of the active roster. We are all looking forward to another exciting season.

So whether it be a Little League field or Minute Maid Park, as they say “Let me root, root, root for the home team”.

PLAY BALL!!!

If you have been unable to change your password, please try the change again using 6-8 characters with the first character being a lower case letter.

Thank you,
Dick DeVance
General Manager
TotalChoice Hosting, L.L.C.

Here at TCH we take backups very seriously and when I say that I do not say it lightly, there is no single more important aspect of our management regime than our backup infrastructure. I am going to explain a bit about the extent to which we go through to protect the data you host with TCH.

The first layer of protection we use is raid 1 mirroring on all our shared, reseller and operations servers (help desk, dns servers etc..), this allows for the servers to maintain an identical copy of the system so that in the event of a disk failure the server can continue operating with no adverse effects. There is a catch here though, that is the fact that software support for raid cards in Linux and even under Windows is severely lacking in the capacity of failure notification, which means when a disk fails there is no industry standard method for alerting someone that there is a problem. At TCH, we have developed an in-house software solution that works with our two preferred raid hardware vendors (AMC 3ware and ARECA raid) to the extent that when there is a disk failure in a raid array it captures information on the failure then sends e-mail alerts to management blackberry pagers and to our help desk ensuring that problems are identified and maintenance is immediately scheduled.

Using raid however reliable it may be, is still not impervious to data loss, which brings us to our next level of protection. All our servers are setup with a spare hard disk and configured to take weekly backups of all user data and server configurations. Although these backups have proven to be extremely reliable they are not always ideal as they can be up to a week old depending on the situation where they are required. We look at these backups as a strictly first-line restore point in the event of a failure that allows us to restore accounts in a fashion that is application compatible with the cPanel interface which makes sure accounts function properly and consistently and overcome the data gaps using our CDP solution below.

Finally, we have our gigabit network enabled continuous data protection (CDP) which runs on absolutely every server that retains client and mission critical data. This is a solution we maintain on network accessible storage (NAS) devices that we have build in house, they contain hardware raid across 16 hard disks with redundant power supplies and a capacity of between 6-13tb of space. The continuous data protection (CDP) is a low-level software run on servers with minimal load impact as it does not read the file system but rather the disk in a raw block-by-block method. This allows CDP to identify differences on the disk quickly and backup only those areas of the disk that have changed since the last backup run (incremental backups). These backups are captured on a 12h schedule of every day, 365 days a year and saved to the NAS devices in a “snapshot” capacity. The snapshots make it possible to recover data as it was 12h ago or 5 days ago or anywhere in between – we save copies of the data in every state from every backup run, we do not overwrite backups!. We can further leverage this backup solution as it is a backup of the servers entire hard disk once you pancake together all the snapshots, so in the event of a catastrophic failure we can take the CDP backup image and restore an entire hard disk in a single swift action. You can also leverage this solution from inside cPanel with the R1Soft Backup feature that allows you to restore from the CDP backup images any data you require just as we would without having to request support, although we are always more than happy to help you with any data recovery needs you may have, so do not hesitate.

I hope you have enjoyed learning a bit more about how we protect the data you host with TCH and understand that there is never any substitute for a well planned and executed backup solution. Before I sign off, let me remind you to take the time and consider the data you store at home or work and ask, do you have a backup solution? If not consider storing some of your more important data on your TCH account so in the event of a failure you can rest assured knowing that TCH has you covered.

I personally think it is a conspiracy between my wife and my doctor to torture me, although they claimed to only be concerned with my health. Despite my paranoia,  I decided they had a legitimate point and sadly drank my last Dew a few weeks ago.

Well it only took a couple a of days before the crankiness set in and the throbbing pain of what my doctor so lovingly called “caffeine withdrawal headaches”. Caffeine headaches? Really? The paranoid side of me started to wonder what they really put in Mountain Dew? I shook off that crazy thought and eventually came to my senses, put away my tinfoil hat, and begin to look for an alternative fix.

As I frantically rummaged through the kitchen, I came across an old dusty coffee pot hidden in the back of the pantry.  Eureka, a source of caffeine with no sugar, I can’t possible go wrong here!  I never have been a big coffee drinker but that was about to change!  Since we only break out the coffee pot for company, it took a bit more rummaging before I was able to find what I needed.  One stale can of coffee-it will have to do for now, paper filters, and one coffee cup from the “good dishes” I am not allowed to use. I never understood why we needed a set of “good dishes” just for company, but I digress.

Intent on curing my headache with as much caffeine as possible, I quickly begin the task of setting up the coffee pot. Within a few minutes the kitchen was filled with the smell of fresh brewed coffee. Stale coffee or not, the aroma was overwhelmingly intoxicating, which only intensified my craving. While it only took a few minutes to brew-it felt like an eternity- with every drip of the coffee pot slowly torturing me. I wonder what Attorney General Eric Holder would have to say about that?

The coffee pot then began to gurgle as if it was dying, signaling me that I could finally have a hot cup of caffeine coffee. It wasn’t the greatest cup of coffee, it was very bitter tasting but it was serving its purpose and it didn’t take long for the effects of the caffeine to kick in. As started on my second cup,  I could feel my heart rate increase and the throbbing in my head begin to fade away. I had soon emptied the entire pot(8 cups)and happily found the way to my office.

I am sure all that coffee was not good for me, but I was feeling so much better that it didn’t really matter. It also made for a very productive day-I think-as I bounced off the office walls from my caffeine high. I was now talking nonstop as I was fielding the phone calls that seemed to come in one after another. I do apologize to the brave souls that called me that day, you must have thought I was crazy and I  hope you can forgive me for rambling on. I can’t say that I didn’t enjoy this day and the energetic feeling the coffee provided, but I knew it was going to wear off soon-resulting in that typical caffeine crash.

This all occurred about 6 weeks ago and I am still starting my day with a pot of coffee and  just a bit more self control.  It doesn’t seem very wise to replace one vice with another, so I have been considering cutting back on the coffee too.  Although, I doubt that will happen any time soon.  Some would say I just have an addictive nature but I prefer to think that if you enjoy something-then enjoy it as often as you can.

I need a refill…

One of the great benefits of hosting with TotalChoice Hosting, is being a true part of our family.  I can not express how many times a confused client has approached me inquiring about an account upgrade.  Just recently I have been helping a client with a large amount of shared hosting sites make some choices on how to upgrade.

Mike Ford, owner of newyorkmustangs.com , has been a happy TCH client for over 5 years.  During this time Mike has grown from one shared hosting account to over ten shared accounts.  Each time adding a new account with a smile. Mike was ready to order his 11th account, however this time, he approached TCH asking about his options.

TotalChoice offers many upgrade options.  I explained to Mike that moving into a reseller account would be the next logical step.  This would allow Mike to host unlimited domains within the allotted disk and bandwidth space.  I explained that a reseller account was still on a shared server. However, Mike really wanted to have full control.  He wanted to add as many domains to his account as humanly possible.  I explained to Mike, that a fully managed dedicated server was the way to go.

After a few days of back and forth questions, Mike looked around and shopped the competitors.  In the end, Mike choose to host his new dedicated server at TotalChoice Hosting.

Thank you Mike! I know you will be pleased with your new server!

Check out Mike Ford’s website at

http://www.newyorkmustangs.com

Happy Hosting!