ctrl-alt-del.cc

Some time ago I was experimenting with pfSense and HAProxy to deploy both as firewall and load balancer for one of the websites I was working on at the time. The key incentive was that pfSense is great BSD based firewall distribution with amazing features offered out of the box, and if that was not enough, you can install additional packages to add features you need.
One of those packages is HAProxy (proxy/load balancer) and both work together very nicely but...

Problem
I have installed pfSense with HAProxy several times, more than several in fact... and every time I did it, the configuration file generated using HAProxy web configurator (integrated with pfSense interface) was broken. First row in the table showing defined backends was empty -  the configuration file itself had just variables but no values. The second entry was just fine... Obviously HAProxy refused to start.

Quick fix
Just so I remember next time what I did. Get shell on pfSense console or install file manager package and edit /usr/local/pkg/haproxy.inc to add the line highlighted below:

Not a rocket science, just shift() the first (empty) backend definition and let the script do the rest.

Note
Manually fixing config file is pointless because new config is generated every time HAProxy (re)starts and the code above is used to generate it.

This post contains my notes - what I did to make it work properly, so next time I build the system, I have a step by step guide. In case you lived under the rock for the last months and don't know what Raspberry Pi is, you should visit www.raspberrypi.org now.

I bought Edimax EW-7811Un adapter for my Pi - small factor, b/g/n type, so why not... especially when vendor says it provides Linux drivers (wohoo!). Sadly as it turns out, compiling drivers on x86 or x64 and ARM architectures can be totally different experience - especially with Raspberry Pi being in it's software infancy. In all seriousness - Pi is for those that like to solve problems (at least at the current stage), but the community works hard to catch up very quickly - great stuff!

Recently I needed something like web based equivalent of tail -f and tail -n commands, so I could display running tail or last N lines from specific log file. To avoid reinventing the wheel I started looking at previous works on-line and found some interesting bits here and there - one of the most useful being AJAX Logfile Tailer & Viewer, so I based my work on this one.

The trick is, that as far as it does exactly what I needed, this solution requires web server with PHP... and installing web server (not to mention PHP) is not really what I want on my logserver.

Mojolicious to the rescue!

Mojolicious is a very powerful Perl web framework that comes without bloat (almost unheard of these days!) - all you need is standard Perl interpreter and core Perl modules as they come preinstalled with your Linux distro and you can install Mojolicious - no other dependencies. On Debian systems installation is as simple as

apt-get install libmojolicious-perl

and we're up and running. Writing Mojolicious::Lite app is really simple and the best part is that it comes with it's own, built in web server (operating in several different modes if needed). Sounds like nice way to go - no dedicated web server on the machine, self-contained application, etc. One more thing - writing, testing and deploying the whole code to actual machine took less than 10 minutes!

I didn't blog for a long time, so it will be a long post caused by some nightmares I had about not doing proper backups on some of my hosts.

Servers - all those small and big machines most of the geeks own, run or operate. As VPS pricing drops, we see more and more of those low-end, resource strapped servers. Organic growth usually means you start with empty server, some kind of definition what it will be doing and... from there it just goes downhill. How do you backup such VPS? Here is something I use myself.

My backup requirements

• Automated - it has to run without supervision in roughly regular time intervals, if it's not automated it will never be done (read no backup)
• Off-site - in case I loose the whole machine for some reason (because RAID is not backup and what fire doesn't destroy, water poured by firemen will)
• No Cross-Backups - because they require trust relationship between machines and if you think about using cheap VPS'es for cross-backups, remember that you get what you pay for!
• Automatically delete old backups - to save space, (my) time and money
• Append only - machine can only write data to its own, designated backup volume but can not delete or modify other volumes (accidents and rogue users do happen)
• Confidentiality - no unauthorized access backed up data
• Availability - storage volume has to be highly available so I can not only write to it knowing it's there, but also access backups when I need them
• Access controls - ability to define granular access rules and enforce append-only usage
• Economy - it has to have reasonable cost

You don't have to outrun a lion - it's enough you outrun the guy running next to you.

Funny enough, the same stands for securing your IT infrastructure - if you are in the "low hanging fruit" category, you get owned for sure - possibly before you even notice anything shady going on behind your shiny website. When you raise the bar a bit and step out of the damned circle, most of the attackers will give up on you and move to find some other target that is easier to compromise.Of course that doesn't work for determined attackers that want YOU and nobody else, but that's a story for another time.

What's that smell?

It's a smell of FAIL my friend...

Just recently I was helping two of my friends and doing some forensics on their servers (or rather on what was left out of them) after they noticed something strange was going on.  Long story short, the key part is that the attackers owned those boxes for months before they were discovered. They got in via path of the least resistance - badly written PHP web apps (there's so many of them!), dropped c99 or similar shell and owned the box to their liking.

In general, we suck really bad if it takes us months to detect such hacks.

Here come the benefits of scale

Wherever and whenever I look at any shared hosting providers, dedicated servers and alike, their default configuration is wide open by default. As long as the box is on-line and Nagios doesn't report issues, nobody is actually checking what's going on that box. Basically operators don't care - they provide functionality and they charge you for it. Oh yes, that's exactly what they do - charge you first and then provide a ton of stuff you don't need and don't use - unless you are an attacker that is :-)