DIY cloud computing - it is easier than you may think!

It seems that the weather forecast for the Internet is a bit "cloudy" nowadays and it will stay this way at least for some time. The "clouds" are a very hot topic right now and more and more companies try to get on the bandwagon as soon as possible - some just run tests while others go into production. You can run "your own" cloud environment for peanuts, the costs are so marginal that it made me laugh when I got my last bill from Amazon AWS, but nevertheless it doesn't always calculate to run your stuff on commercial cloud, especially if you have hardware at hand. The DIY approach is easier than it seems to be. Here is how I've built my own, small "cloud" to solve a problem I was facing at work. It's not a rocket science, it's not full blown management system with hundreds of machines... it works for me and I believe anyone can build similar system - hopefully much better than I did with mine.

Staying away from terminology like HPC/cluster/cloud/grid and meanings of those I use the term "cloud" because I think it's the closest to what I've got now in my prototype - it's still work in progress and it gets even more "cloudy" or change shape otherwise. There won't be any code this time - maybe when I finish it properly and have some proper performance stats - so far it's just a running and usable PoC I describe here :-)

Is that me or is the Internet down? Ahhh... Google Mail is!

As we all know Google had a rough day yesterday - massive "outage" on one of their products... actually a key product - Google Mail. Both, Gmail and Google Apps users (including paid ones) had problems accessing their mail via web interface, but ONLY via web interface. SMTP/POP/IMAP all seemed to work - at least for me - alongside calendars, docs, etc.

Leaving the media hype surrounding this situation (and people saying that google is evil) there is a few things to keep in mind... The situation that happened yesterday leaves no doubt that even the best brains and the best people money can buy, they DO MAKE MISTAKES - as we all do. It happened to Google this time, it may happen to your company tomorrow (or has already happened but you don't want to talk about it).
Yes - I got my part of 'grief and complaints' from my own users, it's perfectly normal situation I would say, so I wasn't even annoyed. I've called Google Support Team, got connected immediately to a very nice guy that confirmed that they have a wide-spread problem running for about 20 minutes now... and that all engineers are already on it (sounds like 'all hands on deck') and offered a callback when it is sorted. As a matter of fact, about 30-45 minutes later webmail access was quite slow but was working again.

Now let's wrap it up:

1. they had a problem - big deal, who hasn't?
   
2. they admitted it - there was no fooling around
3. they got really good response time and fixed the problem
4. ... and I would say they will have no problems keeping up to their SLAs (only one access channel was down - webmail, IMAP/POP/SMTP worked for me all the time), so can we say that Google Mail was down? Not really!
   

So if you are crying/moaning about the situation yesterday, then think again and get over it - it could be much worse! To get you in a better mood - think about calling one of those big telecoms and speaking to 5+ consultants before they put you through to the right department, where you will hear it's a problem on your side, not theirs. How does that sound? Did I hear "8 phone calls, hours wasted on the phone and then 2 weeks to get it fixed"?

SANS Security 504 - 19 Feb - 23 Apr, Mentor Session, London, UK

December has just passed by and left very nice memories of SANS London 2008 conference - very extensive training, new friends and more than anything else - great fun all week long!

To stay in the good mood - I have the pleasure to announce, that I will be mentoring Security 504: Hacker Techniques, Exploits and Incident Handling during Mentor Sessions in London, UK. We will start on Thursday, 23 Feb 2009 and will meet weekly, every Thursday evening, till 23 Apr 2009. We will of course finish with Capture the Flag game (wohooo - that will be fun!) at the end of the course. If you would be interested in participating in the course, please contact me or SANS Institute directly.

You can find more information about my mentor SEC 504 session, it's content and requirements at the SANS website.

Backing up MySQL with daily snapshots and Bacula

I have decided that from time to time I will be putting here some scrips I find useful. As most sysadmins I am lazy and proud of it - that is exactly what makes me write more and more useful scripts. Following some slightly twisted logic if you are lazy sysadmin as well, you might find those useful (as well). Some code may and will be trivial, but still useful - that is the goal!

Today nothing fancy - just (another) database snapshot script that uses mysqldump to do the job and Bacula to get the daily backups automated.

Tuning Nagios for running off CF Card

As a follow up to my previous post I've run my Nagios installation on Soekris net4801 implementing the advice I've given you in my last post (focusing on slow I/O when writing to CF Card), describing the platform and what can be done with it. The changes in system behavior are huge - in a positive way of course.

First of all the system is not so overloaded now and I guess I could double the amount of tests run on this platform without getting into trouble like before. At the moment this system is monitoring 36 machines with 86 services in total. Some time ago I had to stop adding and literally remove some less important tests, because most of the time I was getting false positives - usually warnings, with comment that the plugin has timed out. So how big is the difference?

Soekris net4801 as Nagios powered network monitor

Some time ago (rather long long time ago) we have decided to purchase some small device to turn it into very portable server, that we could send to one of our friends to host. The whole purpose was to get Nagios on it and to monitor our sites from outside of our networks. To some people it may sound crazy, but it makes kind of sense - how many times you have heard from someone "it works on my computer"? Too many times?

The goal is to know when my (and possibly why) visitors/customers can't reach my servers and to be able to diagnose if that is local to some location or network part or it affects wider audience. Up to some point remote sensor answers that question - at least from a perspective of his particular location.

After looking around the net we've decided to get one of those famous Soekris kits.

Was it a good choice as a hardware platform? How will it scale when the amount of monitored systems will reach certain level? Let's see where it got us so far as the system is live for about a year now.

Migration finished

For quite a long time I was investigating different blogging platforms and finally I've made my choice - move away from WordPress to Movable Type. My requirements were rather simple - host entries, be able to publish code snippets, maybe some photos from time to time... I prefer static HTML instead of tons of PHP scripts running to display any single page. I don't expect being 'slashdotted' :-) I just want to keep it simple (K.I.S.S.).