You don't have to outrun a lion - it's enough you outrun the guy running next to you.
Funny enough, the same stands for securing your IT infrastructure - if you are in the "low hanging fruit" category, you get owned for sure - possibly before you even notice anything shady going on behind your shiny website. When you raise the bar a bit and step out of the damned circle, most of the attackers will give up on you and move to find some other target that is easier to compromise.Of course that doesn't work for determined attackers that want YOU and nobody else, but that's a story for another time.
What's that smell?
It's a smell of FAIL my friend...
Just recently I was helping two of my friends and doing some forensics on their servers (or rather on what was left out of them) after they noticed something strange was going on. Long story short, the key part is that the attackers owned those boxes for months before they were discovered. They got in via path of the least resistance - badly written PHP web apps (there's so many of them!), dropped c99 or similar shell and owned the box to their liking.
In general, we suck really bad if it takes us months to detect such hacks.
Here come the benefits of scale
Wherever and whenever I look at any shared hosting providers, dedicated servers and alike, their default configuration is wide open by default. As long as the box is on-line and Nagios doesn't report issues, nobody is actually checking what's going on that box. Basically operators don't care - they provide functionality and they charge you for it. Oh yes, that's exactly what they do - charge you first and then provide a ton of stuff you don't need and don't use - unless you are an attacker that is :-)
Plenty of dangerous PHP functions enabled, dumb/bad configuration of network services and often the networks itself, total lack of monitoring (except for Nagios)... and all of that provided by default, just in case a customer comes back and says 'oh, that breaks functionality I need'; all because that would mean they (operators) have to go back and spend some time on enabling it later. Sure, it's easier to blame it on the "bad hackers in my interwebz" - great business model guys! I believe, that if you build the security into your system from the start, your TCO will be lower than going with defaults (loss of clients due to compromise, cos of bringing system back in service, etc) but that's a business decision of course.
Default configs are similar to default passwords.
Improving security posture
If you are on a shared hosting platform, there's not much you can do really. It's a
shared host, so you (or rather the operator) has to find the common denominator - something that will satisfy everybody using this particular host. It's about finding the weakest link and bringing everybody else down to the same level - not good.
If you go with VPS or dedicated server, you can change a lot and it won't cost you a lot of money. Simple things can improve your posture and make it much harder for the attackers to run loose on your servers. Here are just three things you can do for free...
Egress filtering
Do you have an outbound firewall policy set to DROP by default? Can you imagine that in datacenter environment? Can it work well or will be a huge PITA?
Yup, easily doable and not that painful if you think about it. If we consider Linux, you can use
iptables for that and I guess you already do have an
iptables firewall of some sort that filters inbound packets. Let's extend it a bit - example below is for a simple web server:
# fail close - just in case
/sbin/iptables -P OUTPUT DROP
/sbin/iptables -F OUTPUT
# allow responses - majority of traffic comes here so it's a first rule
/sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# allow from self to self
/sbin/iptables -A OUTPUT -d 127.0.0.1 -j ACCEPT
# allow DNS servers listed in /etc/resolv.conf
for DNS in `grep "^nameserver" /etc/resolv.conf`; do
if [ $DNS != "nameserver" ]; then
/sbin/iptables -A OUTPUT -p udp --dport 53 -d $DNS -j ACCEPT
fi
done
# allow SMTP out to email admins
/sbin/iptables -A OUTPUT -p tcp --dport 25 -d $DOMAIN_MX -j ACCEPT
# allow NTP outbound, local NTP is nice to have!
/sbin/iptables -A OUTPUT -p udp --dport 123 -d $LOCAL_NTP -j ACCEPT
# allow connections to our Linux repository mirror for updates
/sbin/iptables -A OUTPUT -p tcp -d $LINUX_REPO -j ACCEPT
# generic log and drop all
# /etc/syslog.conf => kern.=debug /var/log/firewall
/sbin/iptables -A OUTPUT -j LOG --log-level 7 --log-prefix "FW-DROP: "
/sbin/iptables -A OUTPUT -j REJECT --reject-with icmp-host-prohibited
Simple? Yes!
Does it raise the bar? Yes!
Do I have to write IP addresses everywhere? No - iptables will resolve hostnames used in rules and I've noticed that for example if my
$LINUX_REPO has several IP addresses, iptables actually created an entry for each of them.
But I can't do anything else! That's exactly the point - you shouldn't do anything else on a web server, unless there is a justified need for that (say access SQL database on another host, etc).
Wrapping up - all your web traffic (responses from web server and other services hosted here) will go into state matching rule, then you care for DNS, access to your own MX (only this one unless you have very good reason to do otherwise), NTP and distro updates are really nice to have, then drop all the other traffic. You could add rules for remote (off-site) logging, so you know when something tries to call out/pops your box.
Now, when an attacker drops his php shell he is pretty much very limited (no call back home, no portscans, no IRC bots, etc), unless he escalates access to root, but hey - how about a network based firewall implementing above?
Server hardening
Wow, you could write a book on that, but let's stick to the basics:
Install only the software you really really need (do you need that gcc and all dev libraries to run your web server) - remove what was installed and is not needed - you can always put it back if you need it later!
Turn off all services that shouldn't be running - my rule of thumb is to bring the system to the point, where I can run it entirely without any firewall, because there is no services to hide.
Keep your software updated - cron is your friend (to see what updates are available)
Kernel hardening -
SELinux and
Grsecurity (+RBAC) seem to be the key candidates here. Yes, that can take a lot of time to set up, but in most cases it's well worth it.
Just try to imagine how annoying it has to be for an attacker to own the box via web app, get root via local privilege escalation and not be able to install his rootkit (and hide) because kernel is monolithic (no loadable modules support) and has grsecurty baked in, with IP logging on resource overstep and other nice features it offers.
BTW here's the funny note left in one of the toolkits I lifted from one of friend's servers - what you make out of it is up to you. Oh... and credit to Ingo Molnár for his exploits and awesome comments in their source code ;-)
Logging and monitoring
Best things are free right? How about using syslog that comes with the system to send the logs off-site? Make a small box somewhere and simply pump it all out, so you have an off-site record in case of unwanted guests showing up.
Not enough bandwidth you say? There's an app for that - pump logs out via OpenVPN using LZO compression with or without encryption (hint: you can set the cipher to none) and as my test show, this can drop your logging bandwidth by around 80% and on top of that you can do traffic shaping in OpenVPN itself.
Now, having logs and not looking at them is a waste of resources, unless you are "checkbox security" organization and need it for compliance on paper... Depending on your pocket condition, you can use simple scripts to get what you need or get some free tools that sift through and visualize large amounts of data. For example
Splunk has a free edition (up to 500MB raw log input per day) and there is many other (mostly paid for) products that you could use. Even "cloud based" services like
Loggly (also offers free developer account) are available these days - simply pick something that works for you.
It is not a rocket science - it's really about common sense, so calm down and carry on.
UPDATE:
As
@denishowe pointed out "it seems we need a checklist for dumb providers with the list of things to disable and another checklist for dumb users, so they can enable what they really need" - yes, that might just work :-)
