Fixing HAProxy configuration in pfSense

Some time ago I was experimenting with pfSense and HAProxy to deploy both as firewall and load balancer for one of the websites I was working on at the time. The key incentive was that pfSense is great BSD based firewall distribution with amazing features offered out of the box, and if that was not enough, you can install additional packages to add features you need.

One of those packages is HAProxy (proxy/load balancer) and both work together very nicely but...

Problem

I have installed pfSense with HAProxy several times, more than several in fact... and every time I did it, the configuration file generated using HAProxy web configurator (integrated with pfSense interface) was broken. First row in the table showing defined backends was empty -  the configuration file itself had just variables but no values. The second entry was just fine... Obviously HAProxy refused to start.

Quick fix

Just so I remember next time what I did. Get shell on pfSense console or install file manager package and edit /usr/local/pkg/haproxy.inc to add the line highlighted below:

Not a rocket science, just shift() the first (empty) backend definition and let the script do the rest.

Note

Manually fixing config file is pointless because new config is generated every time HAProxy (re)starts and the code above is used to generate it.

Raspberry Pi meets Edimax EW-7811Un wireless adapter

This post contains my notes - what I did to make it work properly, so next time I build the system, I have a step by step guide. In case you lived under the rock for the last months and don't know what Raspberry Pi is, you should visit www.raspberrypi.org now.

I bought Edimax EW-7811Un adapter for my Pi - small factor, b/g/n type, so why not... especially when vendor says it provides Linux drivers (wohoo!). Sadly as it turns out, compiling drivers on x86 or x64 and ARM architectures can be totally different experience - especially with Raspberry Pi being in it's software infancy. In all seriousness - Pi is for those that like to solve problems (at least at the current stage), but the community works hard to catch up very quickly - great stuff!

Logfile tail the web way

Recently I needed something like web based equivalent of tail -f and tail -n commands, so I could display running tail or last N lines from specific log file. To avoid reinventing the wheel I started looking at previous works on-line and found some interesting bits here and there - one of the most useful being AJAX Logfile Tailer & Viewer, so I based my work on this one.

The trick is, that as far as it does exactly what I needed, this solution requires web server with PHP... and installing web server (not to mention PHP) is not really what I want on my logserver.

Mojolicious to the rescue!

Mojolicious is a very powerful Perl web framework that comes without bloat (almost unheard of these days!) - all you need is standard Perl interpreter and core Perl modules as they come preinstalled with your Linux distro and you can install Mojolicious - no other dependencies. On Debian systems installation is as simple as

apt-get install libmojolicious-perl

and we're up and running. Writing Mojolicious::Lite app is really simple and the best part is that it comes with it's own, built in web server (operating in several different modes if needed). Sounds like nice way to go - no dedicated web server on the machine, self-contained application, etc. One more thing - writing, testing and deploying the whole code to actual machine took less than 10 minutes!

Secure backup of untrusted remote hosts

I didn't blog for a long time, so it will be a long post caused by some nightmares I had about not doing proper backups on some of my hosts.

Servers - all those small and big machines most of the geeks own, run or operate. As VPS pricing drops, we see more and more of those low-end, resource strapped servers. Organic growth usually means you start with empty server, some kind of definition what it will be doing and... from there it just goes downhill. How do you backup such VPS? Here is something I use myself.

My backup requirements

• Automated - it has to run without supervision in roughly regular time intervals, if it's not automated it will never be done (read no backup)
• Off-site - in case I loose the whole machine for some reason (because RAID is not backup and what fire doesn't destroy, water poured by firemen will)
• No Cross-Backups - because they require trust relationship between machines and if you think about using cheap VPS'es for cross-backups, remember that you get what you pay for!
• Automatically delete old backups - to save space, (my) time and money
• Append only - machine can only write data to its own, designated backup volume but can not delete or modify other volumes (accidents and rogue users do happen)
• Confidentiality - no unauthorized access backed up data
• Availability - storage volume has to be highly available so I can not only write to it knowing it's there, but also access backups when I need them
• Access controls - ability to define granular access rules and enforce append-only usage
• Economy - it has to have reasonable cost

How To Outrun A Lion?

You don't have to outrun a lion - it's enough you outrun the guy running next to you.

Funny enough, the same stands for securing your IT infrastructure - if you are in the "low hanging fruit" category, you get owned for sure - possibly before you even notice anything shady going on behind your shiny website. When you raise the bar a bit and step out of the damned circle, most of the attackers will give up on you and move to find some other target that is easier to compromise.Of course that doesn't work for determined attackers that want YOU and nobody else, but that's a story for another time.

What's that smell?

It's a smell of FAIL my friend...

Just recently I was helping two of my friends and doing some forensics on their servers (or rather on what was left out of them) after they noticed something strange was going on.  Long story short, the key part is that the attackers owned those boxes for months before they were discovered. They got in via path of the least resistance - badly written PHP web apps (there's so many of them!), dropped c99 or similar shell and owned the box to their liking.

In general, we suck really bad if it takes us months to detect such hacks.

Here come the benefits of scale

Wherever and whenever I look at any shared hosting providers, dedicated servers and alike, their default configuration is wide open by default. As long as the box is on-line and Nagios doesn't report issues, nobody is actually checking what's going on that box. Basically operators don't care - they provide functionality and they charge you for it. Oh yes, that's exactly what they do - charge you first and then provide a ton of stuff you don't need and don't use - unless you are an attacker that is :-)


Plenty of dangerous PHP functions enabled, dumb/bad configuration of network services and often the networks itself, total lack of monitoring (except for Nagios)... and all of that provided by default, just in case a customer comes back and says 'oh, that breaks functionality I need'; all because that would mean they (operators) have to go back and spend some time on enabling it later. Sure, it's easier to blame it on the "bad hackers in my interwebz" - great business model guys! I believe, that if you build the security into your system from the start, your TCO will be lower than going with defaults (loss of clients due to compromise, cos of bringing system back in service, etc) but that's a business decision of course.

Default configs are similar to default passwords.

Improving security posture

If you are on a shared hosting platform, there's not much you can do really. It's a shared host, so you (or rather the operator) has to find the common denominator - something that will satisfy everybody using this particular host. It's about finding the weakest link and bringing everybody else down to the same level - not good.

If you go with VPS or dedicated server, you can change a lot and it won't cost you a lot of money. Simple things can improve your posture and make it much harder for the attackers to run loose on your servers. Here are just three things you can do for free...

Egress filtering

Do you have an outbound firewall policy set to DROP by default? Can you imagine that in datacenter environment? Can it work well or will be a huge PITA?

Yup, easily doable and not that painful if you think about it. If we consider Linux, you can use iptables for that and I guess you already do have an iptables firewall of some sort that filters inbound packets. Let's extend it a bit - example below is for a simple web server:

# fail close - just in case
/sbin/iptables -P OUTPUT DROP
/sbin/iptables -F OUTPUT
# allow responses - majority of traffic comes here so it's a first rule
/sbin/iptables -A OUTPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
# allow from self to self
/sbin/iptables -A OUTPUT -d 127.0.0.1 -j ACCEPT
# allow DNS servers listed in /etc/resolv.conf
for DNS in `grep "^nameserver" /etc/resolv.conf`; do
        if [ $DNS != "nameserver" ]; then
                /sbin/iptables -A OUTPUT -p udp --dport 53 -d $DNS -j ACCEPT
        fi
done
# allow SMTP out to email admins
/sbin/iptables -A OUTPUT -p tcp --dport 25 -d $DOMAIN_MX -j ACCEPT
# allow NTP outbound, local NTP is nice to have!
/sbin/iptables -A OUTPUT -p udp --dport 123 -d $LOCAL_NTP -j ACCEPT
# allow connections to our Linux repository mirror for updates
/sbin/iptables -A OUTPUT -p tcp -d $LINUX_REPO -j ACCEPT
# generic log and drop all
# /etc/syslog.conf => kern.=debug     /var/log/firewall
/sbin/iptables -A OUTPUT -j LOG --log-level 7  --log-prefix "FW-DROP: "
/sbin/iptables -A OUTPUT -j REJECT --reject-with icmp-host-prohibited

Simple? Yes!
Does it raise the bar? Yes!
Do I have to write IP addresses everywhere? No - iptables will resolve hostnames used in rules and I've noticed that for example if my $LINUX_REPO has several IP addresses, iptables actually created an entry for each of them.
But I can't do anything else! That's exactly the point - you shouldn't do anything else on a web server, unless there is a justified need for that (say access SQL database on another host, etc).

Wrapping up - all your web traffic (responses from web server and other services hosted here) will go into state matching rule, then you care for DNS, access to your own MX (only this one unless you have very good reason to do otherwise), NTP and distro updates are really nice to have, then drop all the other traffic. You could add rules for remote (off-site) logging, so you know when something tries to call out/pops your box.

Now, when an attacker drops his php shell he is pretty much very limited (no call back home, no portscans, no IRC bots, etc), unless he escalates access to root, but hey - how about a network based firewall implementing above?

Server hardening

Wow, you could write a book on that, but let's stick to the basics:
Install only the software you really really need (do you need that gcc and all dev libraries to run your web server) - remove what was installed and is not needed - you can always put it back if you need it later!
Turn off all services that shouldn't be running - my rule of thumb is to bring the system to the point, where I can run it entirely without any firewall, because there is no services to hide.
Keep your software updated - cron is your friend (to see what updates are available)
Kernel hardening - SELinux and Grsecurity (+RBAC) seem to be the key candidates here. Yes, that can take a lot of time to set up, but in most cases it's well worth it.

Just try to imagine how annoying it has to be for an attacker to own the box via web app, get root via local privilege escalation and not be able to install his rootkit (and hide) because kernel is monolithic (no loadable modules support) and has grsecurty baked in, with IP logging on resource overstep and other nice features it offers.

BTW here's the funny note left in one of the toolkits I lifted from one of friend's servers - what you make out of it is up to you. Oh... and credit to Ingo Molnár for his exploits and awesome comments in their source code ;-)

Logging and monitoring

Best things are free right? How about using syslog that comes with the system to send the logs off-site? Make a small box somewhere and simply pump it all out, so you have an off-site record in case of unwanted guests showing up.
Not enough bandwidth you say? There's an app for that - pump logs out via OpenVPN using LZO compression with or without encryption (hint: you can set the cipher to none) and as my test show, this can drop your logging bandwidth by around 80% and on top of that you can do traffic shaping in OpenVPN itself.

Now, having logs and not looking at them is a waste of resources, unless you are "checkbox security" organization and need it for compliance on paper... Depending on your pocket condition, you can use simple scripts to get what you need or get some free tools that sift through and visualize large amounts of data. For example Splunk has a free edition (up to 500MB raw log input per day) and there is many other (mostly paid for) products that you could use. Even "cloud based" services like Loggly (also offers free developer account) are available these days - simply pick something that works for you.


It is not a rocket science - it's really about common sense, so calm down and carry on.

UPDATE:
As @denishowe pointed out "it seems we need a checklist for dumb providers with the list of things to disable and another checklist for dumb users, so they can enable what they really need" - yes, that might just work :-)

Building cheap console server

This time from the department of almost wasted time...

We all know that serial ports come very handy when you need to (re)configure something like a switch/server/firewall or similar device. In theory you can do that over TCP/IP nowadays with one hint - you need to have connectivity. All would be ok if not the fact that those very switches/firewalls you want to reconfigure actually provide the connectivity you need :-)

The Idea

Now... why spend hundreds of pounds/dollars on off-the shelf kit? Sure, it's cool, properly built and works unless you mess it up, but where's the fun part?! Today I needed a very very quick and cheap solution, so:

1. SheevaPlug - £114.00
2. 13-port USB hub - £19.99
3. USB-serial dongles (pl2303) - £14.99 each

This way I have fully networked console server with 4 ports just under £200 - acceptable, especially when the whole thing is running off DHCP and calls home via OpenVPN - very easy to deploy!

Tricky bits

Generic Sheeva has one USB host port and hub has 13 of them - I want to send it off to remote location and have somebody plug it in and not mess up what's where. Trick is to write appropriate udev rules to detect adapters and give them ttyUSBn names according to physical port on the hub.

All would be fine and easy if it worked as documented - sadly it doesn't. First problem was that ATTRS{devpath} (as returned by udevadm info --attribute-walk -n /dev/ttyUSBn that allows to distinguish usb ports) was used by rule in tests but wasn't propagated properly on none of my Debian or Ubuntu boxes. Then I tried to match KERNELS for parent devices - nope... if you go too far up the tree it doesn't see s**t :-/


The Solution

Finally I got the working rule set - long story short, here it is:

KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.1:1.0",   NAME="ttyUSB0"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.1:1.0", NAME="ttyUSB1"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.2:1.0", NAME="ttyUSB2"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.3:1.0", NAME="ttyUSB3"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.4:1.0", NAME="ttyUSB4"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.4:1.0", NAME="ttyUSB5"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.3:1.0", NAME="ttyUSB6"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.2:1.0", NAME="ttyUSB7"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.1:1.0", NAME="ttyUSB8"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.4:1.0", NAME="ttyUSB9"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.3:1.0", NAME="ttyUSB10"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.2:1.0", NAME="ttyUSB11"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.1:1.0", NAME="ttyUSB12"

I had to use KERNELS match as above to have variables seen by the rule. I still don't know (and at this moment don't care any more) why it didn't work as documented...

The bottom line is that it works, it can be done way cheaper than commercial solutions, literally at the fraction of cost - if you don't mind the spider-ish look of it :-)


Update:
Hat tip to @herkii for pointing out another approach.

Making new friends with kippo

Less than two weeks ago I've sent a tweet asking for honeypot recommendations. I wanted to play a bit with something new, something I never did before, mostly because I never had time for it (right, like I have it now). Anyway, thanks to all the great people that replied to my tweet I've learned a lot and found some great software. Now it's time to give something back to the community.

Kippo - simply amazing

First honeypot I've reached for was kippo. It is a medium interaction SSH honeypot designed to log brute force attacks and log the whole session as it goes - including timings, typos, etc. The magic sauce is that you can play the session back (with typos!) and see what the attackers are made of. Believe me - playing back those session is totally amazing! Some samples are available on project's page.
There are also other features to like, like trapping sessions and not disconnecting them even if bad guys do logout, logging ssh client used (very easy to tell scanning bots apart from real people), quite nice interaction and most of all easy way to extend your honeypot it with your own commands.


Installing kippo



For the base platform I used one of my Debian hosts and started with
kippo 0.4. It was good to see how to run it, but options are limited,
so go full steam ahead and get SVN version - it is well worth it!


By default kippo runs on port 2222 but I wanted it on port 22 as normal
SSH would be (running as unprivileged user), so I've set it up on one
of my unused IP addresses - the setup was very easy.



Before you grab the latest version from SVN repo, you should install
required python packages (dependencies will be pulled in
automagically):

apt-get install python-twisted

svn checkout http://kippo.googlecode.com/svn/trunk/ kippo-read-only

Main benefit of SVN version is that it can use MySQL to log events
(alongside the regular log file) and that it can actually bind to given
IP address - version 0.4 binds to all available addresses which is a
bummer for me when I want to spawn totally fake host and have normal
ssh working as well.



Honeypots - rule #1



DO NOT run honeypot as root!



Remember that honeypots are software components, they may (and most
likely do) have their own bugs. Of course you have to be root
to bind to port <1024, or do you?



Configuration



Couldn't be easier... create unprivileged, regular user  account to run
your honeypot (I called it honeytrap), create your own kippo.cfg using
kippo.cfg.dist as template, set MySQL parameters, honeypot hostname
(attackers will see it after they log in), IP address to bind to and
port. If you don't want to use MySQL - your call... it may come very
handy for reporting. That's it - you are ready to go.



Now the trick is to get it running on port 22. There is obviously more
than one way to do it. If you have only one IP address available, you
should most likely go to kippo's Wiki page that describes how to make
kippo reachable through port 22 but if you
have spare IP address... =B-]



Now, how do I bind to port 22 as regular user? Somebody must have solved that problem before, right?
Sure, and they even created a package that solves this issue! It's called authbind and it's amazingly easy to use.

apt-get  install authbind

touch /etc/authbind/byport/22

chown honeytrap:honeytrap /etc/authbind/byport/22

chmod +x /etc/authbind/byport/22

Authbind works almost like sudo, except for the ports - not the
commands. In kippo's directory you will find start.sh script - add
authbind in front of the startup command and you are good to go.



Ready, set, go!



Ok - don't do my mistake... test your install - ssh into the honeypit and
see if you can log in. The root password is in kippo.cfg. Testing setup
is important - if sql database is gone, then you won't log in and in
kippo.log it will say that the root password was incorrect, when in
fact the problem is disconnected sql log backend.



Observations



Kippo is really great tool to learn what the bad guys are up to. If
they add user, they can log in as that user later. If they change root
password, it will be there for them when they return. The best part is
that of course you can see those passwords and suddenly you will have
new 'accounts' added and new, correct and active at the same time root
passwords (yes, more than one correct password!).

I've spent some time
watching the sessions recorded so far - there's so much to see, laugh
and cry, but I'll leave that for another post...

BTW, I blame Andrew (@Infosanity) for all of that - he got me back to honeypots topic, then other great tweeps came back with advice (much appreciated), so make sure to visit his blog ;-)