Sunday, 4 March 2012

Logfile tail the web way

Recently I needed something like web based equivalent of tail -f and tail -n commands, so I could display running tail or last N lines from specific log file. To avoid reinventing the wheel I started looking at previous works on-line and found some interesting bits here and there - one of the most useful being AJAX Logfile Tailer & Viewer, so I based my work on this one.

The trick is, that as far as it does exactly what I needed, this solution requires web server with PHP... and installing web server (not to mention PHP) is not really what I want on my logserver.

Mojolicious to the rescue!

Mojolicious is a very powerful Perl web framework that comes without bloat (almost unheard of these days!) - all you need is standard Perl interpreter and core Perl modules as they come preinstalled with your Linux distro and you can install Mojolicious - no other dependencies. On Debian systems installation is as simple as

apt-get install libmojolicious-perl

and we're up and running. Writing Mojolicious::Lite app is really simple and the best part is that it comes with it's own, built in web server (operating in several different modes if needed). Sounds like nice way to go - no dedicated web server on the machine, self-contained application, etc. One more thing - writing, testing and deploying the whole code to actual machine took less than 10 minutes!

Implementation details

I decided to take HTML and JavaScript elements from the AJAX Logfile Tailer & Viewer as they seemed to do just what I need and because JavaScript is just not my cup of tea so certainly, I wouldn't write it myself.

All of the code is written as Mojolicious::Lite app, with HTML and JavaScript stored as embedded templates (see DATA section of the script), so all I need to run it is Mojolicious and the script itself - nice, portable solution with low memory footprint when running. Yes, I could use Web Sockets, Comet or any similar technology (Mojolicious supports those out of the box anyway) but I didn't have time to play with it right then - I needed something that will work.

Note to all Perl purists - I know you won't like the code because I call external (system) tail command to get log lines, but I didn't have time and honestly was too lazy to write it in pure Perl - will fix that in v2.0.

To keep code listing short, I'll put placeholders for HTML and Javascript elements.

use strict;
use warnings;
use Mojolicious::Lite;
use HTML::Entities;

# logfile we want to see
my $logfile = '/var/log/syslog';

# Route requests to templates in DATA section
get '/' => 'index';
get '/js/ajax.js' => 'ajax';
get '/js/logtail.js' => 'logtail';

# RESTful interface - fixed tail size
get '/logdata' => sub {
    my $self    = shift;
    open (IN, "tail -40 $logfile |");
    chomp(my @log = (<IN>));
    close (IN);

    map { $_ = encode_entities($_) } @log;

    $self->render(text => join("\n", reverse @log));

# variable tail size
get '/tail-n/:N' => sub {
    my $self = shift;
    my $N = $self->param('N');
    if ($N =~ /\D/) {
        # command injection attempt?

        $self->render(text => "Y U NO GIVE UP, NICE TRY!");
    } else {
        open (IN, "tail -$N $logfile |");
        chomp(my @log = (<IN>));
        close (IN);
        map { $_ = encode_entities($_) } @log;
        $self->render(text => join("<br/>", reverse @log));

# cookie encryption passphrase  - no use here but if missing it produces warning :-)

@@ index.html.ep
<!-- here goes all the index.html contents -->

@@ ajax.js.ep
<!-- yes, you guessed it -->

@@ logtail.js.ep
/* an ajax log file tailer / viewer
copyright 2007 john minnihan.

Released under these terms
1. This script, associated functions and HTML code ("the code") may be used by you ("the recipient") for any purpose.
2. This code may be modified in any way deemed useful by the recipient.
3. This code may be used in derivative works of any kind, anywhere, by the recipient.
4. Your use of the code indicates your acceptance of these terms.
5. This notice must be kept intact with any use of the code to provide attribution.
<!-- original disclaimer, the rest is as above -->

That's it! Keep in mind that you have to customize a bit logtail.js.ep part - function getLog has url variable you need to point to /logdata provided by our script. You can also specify how often the AJAX call will be made to fetch log data - this is done in startTail function. I use 2000ms value and it's well enough, if not too often anyway - tune it so you won't get more than 40 lines in the log during this time... or tune for the maximum smoke - your call.

How it works?

Built-in web server will respond to all paths defined with get '<path>' statement. Those that are routed to templates, will respond with templates (which can have dynamic content as well but that's out of scope here). Those with defined subroutines will get the code executed - no magic here.

Index page pulls in two JavaScript files (all template based), logtail.js requests data from first subroutine responsible for '/logdata' and this one is refreshed as per timer in startTail function.

Second subroutine is used to display static log chunk that won't refresh itself automatically - in case you are debugging something, the last thing you want are disappearing logs. This one is manually called by the user as http://scrpt_url/tail-n/<lines to display>. Just in case someone had the idea to run script as root (command injection could be deadly!) the script will terminate if provided number of lines contains non-digits.

Running the app

You can run it in many ways, but for small deployments (like mine) this is entirely enough:

./ daemon

This will start listener on port 3000 (default, can be changed with command line parameter).

Security warning

Logs can contain data that is not safe to be displayed via web interface as-is - think of XSS for example. At best, you will get popup, at worst... well, much worse. This is why I've added encode_entities() from HTML::Entities to the script - current version escapes at least the basic elements but you can decide which ones you want to encode - see module documentation for details.


Big thank you goes to Sebastian Riedel (@kriah) for his work on Mojolicious which simply rocks and John Minnihan who wrote the HTML and JavaScript I used... as well as and many others that gave me some ideas but the approach they proposed was sadly not acceptable in my usage scenario.