Alright, must be time for another geek post.

I’m obsessed with stats, anything and everything, if you can report and graph on it, I’m interested. Recently I’ve been experimenting with different server monitoring tools, having mainly been a Cacti user for local networks I wanted to get away from having to rely on a funky perl script via SNMP to get some specific stats on multiple servers from the one location. After much research, Munin seemed to be the most appropriate option, so I gave it a go.

There is a Munin package available for Debian Etch, so installation was simple and it detected what other services I was running and enabled the appropriate plugins for this. This went mostly ok, there were still some bits and pieces left over from Postfix which I was no longer using even though it thought I had, and the Sendmail stats weren’t generating any information. It was easy enough to remove the Postfix plugin, and it turns out the error with Sendmail stats isn’t actually an issue with Munin, it’s just that the plugin for Munin uses the “mailstats” command which doesn’t appear to be configured correctly out of the box for Etch. I won’t go into detail, but if you’re having this issue, the fix is simple and described here.

Example memory usage on one server over nearly 24 hours.

Everything was going pretty smoothy, except Apache monitoring, which was one of the main things I wanted to keep an eye on. The built in Apache stats didn’t work straight out of the box, as they required the server-status module to be loaded along with the libwww-perl package, which I hadn’t done, but even then they were still lacking. I did some searching and found this blog post that looked to display pretty much exactly what I was after. Unfortunately most of the links to software were dead and the ones I found weren’t appropriate for Debian Etch.

Apache Watch example from Freshnet

After a lot of trouble shooting and searching and this very helpful wiki post, I got a completely workable solution. If you’re interested in this, follow the steps below. I haven’t included the actual install/configure of Munin, but this will take care of making sure the default Apache plugins work correctly, along with the extra third party mod_watch and associated apache_watch_ munin plugins.

So first up, to reiterate, if you don’t already have a working Munin, don’t follow these steps. I’m assuming you already have that part done, you’re just having issues with the Apache stats, or want extra Apache stats. If you have a working Munin, continue on:

First up, we need the Apache2 Dev packages to compile the mod_watch plugin from source. Along with this we’ll grab libwww-perl, as we need this to grab the stats.

apt-get install apache2-dev libcrypt-ssleay-perl libwww-perl

Download and extract the apache_watch module. The original website for this software is here, however for what ever reason it is not available from there anymore.

cd /root
wget http://rand.stubbedtoe.co.nz/binarys/mod_watch-4.3_apache22_mod.tar.gz
tar -xvzf mod_watch-4.3_apache22_mod.tar.gz
cd mod_watch-4.3_apache22_mod

Change the Makefile.dso to our needs for Debian Etch

nano Makefile.dso

You should only have to change two things, the APXS location, and STATEDIR:

# The location of apxs utility.
#
#APXS=/home/apache2/bin/apxs
APXS=/usr/bin/apxs2

#
# The location of apachectl utility to stop/start/restart targets.
#
APACHECTL=apache2ctl

#
# Where the scripts should live
#
SCRIPTDIR=/usr/local/sbin

#
# Where to store the weenie files.
#
STATEDIR=/usr/lib/apache2/modules/mod_watch/

#
# Define to use unsigned long long counters.
#
#BIG=-DUSE_OCTET_COUNTER_64

#
# Extras
#
DEF=$(BIG) -DSTATEDIR='\"$(STATEDIR)\"'
INC=
LIB=
~~~~~~~~~

Now build the package and copy it to the correct place

make -f Makefile.dso build
make -f Makefile.dso install
cp mod_watch.c /usr/share/munin/plugins/
chmod 755 /usr/share/munin/plugins/mod_watch.c

Now we need to make Apache2 load the module. There are a few ways to do this, but I wanted to stick with the default way rather than editing the standard apache2.conf. First up we need to create a couple of new files in the mods-available directory.

nano /etc/apache2/mods-available/watch.load

This only needs one line of code:

LoadModule watch_module /usr/lib/apache2/modules/mod_watch.so

Now the conf file. We create this for consistences sake, however I leave everything commented out. This is so we have more control over where this information is going to be displayed, if we didn’t want it on the default website.

nano /etc/apache2/mods-available/watch.conf

And fill it with this:

#<ifmodule mod_watch.c>
# Allows the URL used to query virtual host data:
#
# http://www.snert.com/watch-info
#
#    <location /watch-info>
#    SetHandler watch-info
#    Order allow,deny
#    Allow from 127.0.0.1
#    </location>

# Intended for debugging and analysis of shared memory
# hash table and weenie files:
#
# http://127.0.0.1/watch-table
#

#    <location /watch-table>
#    SetHandler watch-table
#    Order allow,deny
#    Allow from 127.0.0.1
#    </location>

#    <location /watch-list>
#    SetHandler watch-list
#    Order allow,deny
#    Allow from 127.0.0.1
#    </location>
#</ifmodule>

We need to comment out the standard .conf files for both server-status and server-info handlers:

nano /etc/apache2/mods-available/status.conf

#<ifmodule mod_status.c>
#
# Allow server status reports generated by mod_status,
# with the URL of http://servername/server-status
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
#<location /server-status>
#    SetHandler server-status
#    Order deny,allow
#    Deny from all
#    Allow from localhost ip6-localhost
#    Allow from .example.com
#</location>
#</ifmodule>

And info.conf

nano /etc/apache2/mods-available/info.conf

#<ifmodule mod_info.c>
#
# Allow remote server configuration reports, with the URL of
#  http://servername/server-info (requires that mod_info.c be loaded).
# Uncomment and change the ".example.com" to allow
# access from other hosts.
#
#<location /server-info>
#    SetHandler server-info
#    Order deny,allow
#    Deny from all
#    Allow from localhost ip6-localhost
#    Allow from .example.com
#</location>
#</ifmodule>

By default all of the Apache related Munin scripts access the information they require via http://localhost/ – You can change this in the configuration of the Munin plugins, but I won’t go into detail here. If you do do this, you’ll have to make sure the appropriate vhost’s config file has the following configuration added to it, but for this example we’ll be adding it to the “default” website in Debian Etch.

nano /etc/apache2/sites-available/default

Add the following code at the very bottom, just above the </VirtualHost>. If you want to view this information from your own computer, replace your.ip.address with your IP. This step is helpful for trouble shooting. Remember to remove it afterwards if you don’t have a static IP address, or if it is largely shared.

#Munin Stats
<ifmodule mod_status.c>
    <location /server-status>
        SetHandler server-status
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 your.ip.address
    </location>
</ifmodule>

<ifmodule mod_info.c>
<location /server-info>
    SetHandler server-info
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 your.ip.address
</location>
</ifmodule>

<ifmodule mod_watch.c>
    <location /watch-info>
    SetHandler watch-info
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 your.ip.address
    </location>

    <location /watch-table>
    SetHandler watch-table
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 your.ip.address
    </location>

    <location /watch-list>
    SetHandler watch-list
        Order deny,allow
        Deny from all
        Allow from 127.0.0.1 your.ip.address
    </location>
</ifmodule>
#End Munin Stats

Now we need to enable all the appropriate Apache2 modules, and restart Apache:

a2enmod watch info status
/etc/init.d/apache2 restart

If everything has worked, you should be able to execute the following with out wget throwing any 404 or 403 errors. Likewise if you chose to add your IP address, you should be able to access from your web browser, however if the wget still fails, then the plugins will not worth either:

wget http://your.server.ip/server-status
wget http://your.server.ip/server-info
wget http://your.server.ip/watch-info
wget http://your.server.ip/watch-table
wget http://your.server.ip/watch-list

If all of those work fine and the pages that are downloaded have content in them, you can proceed to the next step. If they haven’t, double check your config or post a comment and I’ll see if I can help you out.

Now we create the apache_watch_ Munin plugin file.

/usr/share/munin/plugins/apache_watch_

And fill it with this:

#!/usr/bin/perl
#
# Parameters supported:
#
#     config
#     autoconf
#
# Configurable variables
#
#     url      - Override default status-url
#
# Must be symlinked to what the graph should monitor. Run with --suggest
# to see valid targets - or just run munin-node-configure --shell
#
# Written by Björn Ruberg 2006-2007
#
# Magic markers:
#%# family=auto
#%# capabilities=autoconf suggest

my $ret = undef;
if (!eval "require LWP::UserAgent;") {
  $ret = "LWP::UserAgent not found";
}

# watch-list exists on localhost
# watch-info does not

my %plugs = (
         'bytes'     => 'Input/output (bytes)',
         'requests'  => 'Requests',
         'documents' => 'Documents served',
            );

my $URL = exists $ENV{'url'} ? $ENV{'url'} : "http://localhost:%d/watch-list";
my @PORTS = exists $ENV{'ports'} ? split(' ', $ENV{'ports'}) : (80);
my $type = "throughput";

if (exists $ARGV[0] and $ARGV[0] eq "autoconf") {
  if ($ret) {
    print "no ($ret)\n";
    exit 1;
  }
  my $ua = LWP::UserAgent->new (timeout => 30);
  my @badports;

  foreach my $port (@PORTS) {
    my $url = sprintf $URL, $port;
    my $response = $ua->request (HTTP::Request->new('GET', $url));
    push @badports, $port unless $response->is_success;
  }

  if (@badports) {
    print "no (no mod_watch exists on ports @badports)\n";
    exit 1;
  } else {
    print "yes\n";
    exit 0;
  }
}

if (exists $ARGV[0] and $ARGV[0] eq "suggest") {
  while (my ($key, undef) = each %plugs) {
    print "$key\n";
  }
  exit 0;
}

my @servers = ();
my @data;
foreach my $port (@PORTS) {
  my $ua = LWP::UserAgent->new (timeout => 30);
  my $url = sprintf $URL, $port;
  my $response = $ua->request (HTTP::Request->new ('GET', $url));
  foreach my $string (split (/\n/, $response->content)) {
    my ($server, undef, $ifInOctets, $ifOutOctets, $ifRequests,
        $ifDocuments) = split (/\s/, $string, 6);
    push @servers, $server unless $server eq "SERVER";
    push @data, "$server $ifInOctets $ifOutOctets $ifRequests $ifDocuments"
      unless $server eq "SERVER";
  }
}

# From here and out, the plugin must be run with a symlinked service.
my $check = join ("|", keys %plugs);
die ("Plugin must be symlinked to aspect to be monitored")
  unless $0 =~ /\_($check)$/;

my $action = $1;

if (exists $ARGV[0] and $ARGV[0] eq "config") {
  print "graph_title Apache $plugs{$action}\n";
  print "graph_args --base 1000 -l 0\n";
  print "graph_category apache\n";
  print "graph_vlabel activity\n";
  my $i = 0;
  foreach my $server (sort (@servers)) {
    (my $txtserver = $server) =~ s/(-|\.)/\_/g;
    my $draw = ($i==0) ? 'AREA' : 'STACK';
    if ($action eq "bytes") {
      print "${txtserver}.label $server\n";
      print "${txtserver}.draw $draw\n";
      print "${txtserver}.type COUNTER\n";
    } else {
      print "${txtserver}.label $server\n";
      print "${txtserver}.draw $draw\n";
      print "${txtserver}.type COUNTER\n";
    }
    $i++;
  }
  exit 0;
}

foreach my $string (sort (@data)) {
  my ($server, $ifInOctets, $ifOutOctets, $ifRequests, $ifDocuments) =
    split (/\s/, $string);
  (my $txtserver = $server) =~ s/(-|\.)/\_/g;
  if ($action eq "documents") {
    print "${txtserver}.value $ifDocuments\n";
  } elsif ($action eq "requests") {
    print "${txtserver}.value $ifRequests\n";
  } elsif ($action eq "bytes") {
    print "${txtserver}.value " . ($ifInOctets + $ifOutOctets) . "\n";
  }
}

Now we just need to do some chmod, create some symlinks, restart Munin and we’re there!

chmod 755 /usr/share/munin/plugins/apache_watch_
ln -s /usr/share/munin/plugins/apache_watch_ /etc/munin/plugins/apache_watch_bytes
ln -s /usr/share/munin/plugins/apache_watch_ /etc/munin/plugins/apache_watch_documents
ln -s /usr/share/munin/plugins/apache_watch_ /etc/munin/plugins/apache_watch_requests
cd /usr/share/munin/plugins/
./apache_watch_ autoconf

If the last command returns “yes.” – Then you are safe to restart Munin and config the individual stats.

/etc/init.d/munin-node restart
cd /usr/share/munin/plugins/
munin-run apache_watch_bytes config
munin-run apache_watch_documents config
munin-run apache_watch_requests config

You should now start seeing Apache VHost graphs propagate in your Munin stats.