Fixing HAProxy configuration in pfSense

Some time ago I was experimenting with pfSense and HAProxy to deploy both as firewall and load balancer for one of the websites I was working on at the time. The key incentive was that pfSense is great BSD based firewall distribution with amazing features offered out of the box, and if that was not enough, you can install additional packages to add features you need.

One of those packages is HAProxy (proxy/load balancer) and both work together very nicely but...

Problem

I have installed pfSense with HAProxy several times, more than several in fact... and every time I did it, the configuration file generated using HAProxy web configurator (integrated with pfSense interface) was broken. First row in the table showing defined backends was empty -  the configuration file itself had just variables but no values. The second entry was just fine... Obviously HAProxy refused to start.

Quick fix

Just so I remember next time what I did. Get shell on pfSense console or install file manager package and edit /usr/local/pkg/haproxy.inc to add the line highlighted below:

Not a rocket science, just shift() the first (empty) backend definition and let the script do the rest.

Note

Manually fixing config file is pointless because new config is generated every time HAProxy (re)starts and the code above is used to generate it.

Logfile tail the web way

Recently I needed something like web based equivalent of tail -f and tail -n commands, so I could display running tail or last N lines from specific log file. To avoid reinventing the wheel I started looking at previous works on-line and found some interesting bits here and there - one of the most useful being AJAX Logfile Tailer & Viewer, so I based my work on this one.

The trick is, that as far as it does exactly what I needed, this solution requires web server with PHP... and installing web server (not to mention PHP) is not really what I want on my logserver.

Mojolicious to the rescue!

Mojolicious is a very powerful Perl web framework that comes without bloat (almost unheard of these days!) - all you need is standard Perl interpreter and core Perl modules as they come preinstalled with your Linux distro and you can install Mojolicious - no other dependencies. On Debian systems installation is as simple as

apt-get install libmojolicious-perl

and we're up and running. Writing Mojolicious::Lite app is really simple and the best part is that it comes with it's own, built in web server (operating in several different modes if needed). Sounds like nice way to go - no dedicated web server on the machine, self-contained application, etc. One more thing - writing, testing and deploying the whole code to actual machine took less than 10 minutes!

Secure backup of untrusted remote hosts

I didn't blog for a long time, so it will be a long post caused by some nightmares I had about not doing proper backups on some of my hosts.

Servers - all those small and big machines most of the geeks own, run or operate. As VPS pricing drops, we see more and more of those low-end, resource strapped servers. Organic growth usually means you start with empty server, some kind of definition what it will be doing and... from there it just goes downhill. How do you backup such VPS? Here is something I use myself.

My backup requirements

• Automated - it has to run without supervision in roughly regular time intervals, if it's not automated it will never be done (read no backup)
• Off-site - in case I loose the whole machine for some reason (because RAID is not backup and what fire doesn't destroy, water poured by firemen will)
• No Cross-Backups - because they require trust relationship between machines and if you think about using cheap VPS'es for cross-backups, remember that you get what you pay for!
• Automatically delete old backups - to save space, (my) time and money
• Append only - machine can only write data to its own, designated backup volume but can not delete or modify other volumes (accidents and rogue users do happen)
• Confidentiality - no unauthorized access backed up data
• Availability - storage volume has to be highly available so I can not only write to it knowing it's there, but also access backups when I need them
• Access controls - ability to define granular access rules and enforce append-only usage
• Economy - it has to have reasonable cost

Building cheap console server

This time from the department of almost wasted time...

We all know that serial ports come very handy when you need to (re)configure something like a switch/server/firewall or similar device. In theory you can do that over TCP/IP nowadays with one hint - you need to have connectivity. All would be ok if not the fact that those very switches/firewalls you want to reconfigure actually provide the connectivity you need :-)

The Idea

Now... why spend hundreds of pounds/dollars on off-the shelf kit? Sure, it's cool, properly built and works unless you mess it up, but where's the fun part?! Today I needed a very very quick and cheap solution, so:

1. SheevaPlug - £114.00
2. 13-port USB hub - £19.99
3. USB-serial dongles (pl2303) - £14.99 each

This way I have fully networked console server with 4 ports just under £200 - acceptable, especially when the whole thing is running off DHCP and calls home via OpenVPN - very easy to deploy!

Tricky bits

Generic Sheeva has one USB host port and hub has 13 of them - I want to send it off to remote location and have somebody plug it in and not mess up what's where. Trick is to write appropriate udev rules to detect adapters and give them ttyUSBn names according to physical port on the hub.

All would be fine and easy if it worked as documented - sadly it doesn't. First problem was that ATTRS{devpath} (as returned by udevadm info --attribute-walk -n /dev/ttyUSBn that allows to distinguish usb ports) was used by rule in tests but wasn't propagated properly on none of my Debian or Ubuntu boxes. Then I tried to match KERNELS for parent devices - nope... if you go too far up the tree it doesn't see s**t :-/


The Solution

Finally I got the working rule set - long story short, here it is:

KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.1:1.0",   NAME="ttyUSB0"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.1:1.0", NAME="ttyUSB1"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.2:1.0", NAME="ttyUSB2"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.3:1.0", NAME="ttyUSB3"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.4.4:1.0", NAME="ttyUSB4"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.4:1.0", NAME="ttyUSB5"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.3:1.0", NAME="ttyUSB6"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.2:1.0", NAME="ttyUSB7"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.3.1:1.0", NAME="ttyUSB8"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.4:1.0", NAME="ttyUSB9"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.3:1.0", NAME="ttyUSB10"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.2:1.0", NAME="ttyUSB11"
KERNEL=="ttyUSB*", SUBSYSTEM=="tty", DRIVERS=="pl2303", KERNELS=="1-1.2.1:1.0", NAME="ttyUSB12"

I had to use KERNELS match as above to have variables seen by the rule. I still don't know (and at this moment don't care any more) why it didn't work as documented...

The bottom line is that it works, it can be done way cheaper than commercial solutions, literally at the fraction of cost - if you don't mind the spider-ish look of it :-)


Update:
Hat tip to @herkii for pointing out another approach.

AirView2 Spectrum Analyzer

Recently I had some serious problems with wi-fi at home - especially one of the laptops was dropping off and couldn't come back. Quick survey using Kismet and other tools to scan what's flying around has proven that my network is in less populated part of the spectrum (at least here) but still, problems are getting worse and worse.

I was fully aware of Wi-Spy by Metageek, seen it in action previously but never had a chance to buy one. Part of the decision was the price back then, maybe now it would be another game, but anyway - I got myself another device, made by well known wi-fi vendor Ubiquiti and it's called AirView2.

What's so special about this one? Why it's better than Wi-Spy?

First of all I didn't say it's better. It's different, woks with Linux, Mac OS X and Windows, has a nice price tag and does pretty much the same as Wi-Spy. Let's have a closer look then, shall we?


It's different

Well, obviously it is... it comes from different vendor... and this post is not a sales pitch - it's just what I've experienced myself. On a bit more serious note, it's smaller than all the Wi-Spy models I've seen so far. Smaller is good, right? Yes - takes up less space, No - easier to loose (looks almost like USB stick).

Works with Linux, Mac and Windows

Yes, it does... better or worse but it does and it's not a matter of hardware or bundled software, but clearly it depends on the host OS and Java. You got it right - Java!
Software is written in Java to be really cross platform, but those that are not Java developers but use it a bit know quite well what a pain in the rear Java can be. Same is here - Ubiquiti warns about compatibility issues, there are long posts on the forums why this particular version of AirView software doesn't work (mostly on Mac OS X) and how to fix it, etc.

Windows - OK, even inside VM with USB passed through to the guest VM (tested VirtualBox, VMWare Workstation and Fusion - all with Windows 7 and latest Java). It was all very slow, loosing connection with the device and re-initializing it all the time, but worked. In native mode with Windows 7 on bare metal box worked like a charm (tested on a netbook PC).

Linux - didn't try, not enough time - sorry.

Mac OS X - yeah... that sucked! If you have the latest patches installed most likely the software will hang on detecting the device. Of course the reason is Java + OS X (I'm on 10.6.2 as of now with Java 1.6.0_17 in 64-bit mode).

java.lang.UnsatisfiedLinkError: /Library/Java/Extensions/librxtxSerial.jnilib:  no suitable image found.  Did find:  /Library/Java/Extensions/librxtxSerial.jnilib: no matching architecture in universal wrapper thrown while loading gnu.io.RXTXCommDriver

Exception in thread "AirViewer-Initializer" java.lang.UnsatisfiedLinkError: /Library/Java/Extensions/librxtxSerial.jnilib:  no suitable image found.  Did find:  /Library/Java/Extensions/librxtxSerial.jnilib: no matching architecture in universal wrapper

WTF?! File not found... but found? Never mind - luckily the solution is very simple - AirView comes with it's own version of librxtxSerial.jnilib so the one that came with OS X needs to be disabled temporarily and problem will go away. That can be done very easily with one command in the terminal:

mv /Library/Java/Extensions/librxtxSerial.jnilib{,-disabled}

That's it, now it works :-)

Price tag

This argument is obviously quite important. Is it that much cheaper? I'm not so sure... of course you can get the basic Wi-Spy for about £65+VAT so even if AirView2 would be equal to it in hardware terms, it would cost a bit more - £69+VAT... There is one catch to it though - AirView2 comes in several versions. I bought the AirView2-EXT for £64+VAT and this one has MMCX connector for external antenna (because it doesn't have a built-in one) and with clip-on omnidirectional antenna (~3-5dBi I guess) in the package. To buy Wi-Spy with RP-SMA connector you would have to spend at least £120+VAT which is almost double the price of AirView2-EXT.

HINT: Wi-Spy with RP-SMA is 2nd generation - faster with better scan resolution than the first generation of the device. So far I didn't have enough time to grab the full spec for AirView2 and compare them side by side - that would be very interesting (it's already on my TODO list).

Does it do the same stuff?

I would say YES based on what I can see, but as most of those devices are SDRs (Software Defined Radio), they can do all the software allows them to do and I didn't have a chance to compare recent version of Wi-Spy software to the AirView one, so please take my words here with a grain of salt and look for other sources to confirm that.

Conclusions

The device worked for me like a charm - it turned out that the signal from my AP was attenuated by temporary objects that came in the way (books - whole piles of them) and as the amount of networks around at least doubled in the last 12 months, somebody put up some very messy device that is transmitting all the time with a very wide signal, exactly in the area of channel I was on - so here come the interference!
Quick look at the graphs and it was clear, that simple channel change should cut down on the interference and moving books a bit will improve signal strength in a place where this unlucky laptop is used most of the time - it worked very well, no more problems!

Looking at the bottom line, for me that's a money very well spent! I was asking myself a question 'how often I will use this thing' and now I really appreciate the power of seeing something that Kismet and similar tools won't see.

Using spectrum analyzer like AirView or Wi-Spy (doesn't really matter which one - pick one that suits your needs) is like reading between the lines - there is a lot of valuable information out there... if only you can see it!

DIY cloud computing - it is easier than you may think!

It seems that the weather forecast for the Internet is a bit "cloudy" nowadays and it will stay this way at least for some time. The "clouds" are a very hot topic right now and more and more companies try to get on the bandwagon as soon as possible - some just run tests while others go into production. You can run "your own" cloud environment for peanuts, the costs are so marginal that it made me laugh when I got my last bill from Amazon AWS, but nevertheless it doesn't always calculate to run your stuff on commercial cloud, especially if you have hardware at hand. The DIY approach is easier than it seems to be. Here is how I've built my own, small "cloud" to solve a problem I was facing at work. It's not a rocket science, it's not full blown management system with hundreds of machines... it works for me and I believe anyone can build similar system - hopefully much better than I did with mine.

Staying away from terminology like HPC/cluster/cloud/grid and meanings of those I use the term "cloud" because I think it's the closest to what I've got now in my prototype - it's still work in progress and it gets even more "cloudy" or change shape otherwise. There won't be any code this time - maybe when I finish it properly and have some proper performance stats - so far it's just a running and usable PoC I describe here :-)

Tuning Nagios for running off CF Card

As a follow up to my previous post I've run my Nagios installation on Soekris net4801 implementing the advice I've given you in my last post (focusing on slow I/O when writing to CF Card), describing the platform and what can be done with it. The changes in system behavior are huge - in a positive way of course.

First of all the system is not so overloaded now and I guess I could double the amount of tests run on this platform without getting into trouble like before. At the moment this system is monitoring 36 machines with 86 services in total. Some time ago I had to stop adding and literally remove some less important tests, because most of the time I was getting false positives - usually warnings, with comment that the plugin has timed out. So how big is the difference?