In this guide, I will show you a quick and dirty way to get open source syslog monitoring done with Opsview that can be applied to Nagios also.
This guide will use the built-in Linux syslog server ‘rsyslog’, and a great plugin called ‘check_logfiles’ that will run on pretty much any Nagios variant.
Step 1: The basics
So what you need to do is quite simple – you send your syslog’s to the syslog server (Opsview, in this example) – and then you run a plugin / script against the central log file, and check for just ‘HostX’ sys logs, and just for certain patterns against HostX. i.e. you send your routers syslogs to Opsview, and you want to check for the word ‘CRITICAL’ in that routers log files – as below:
Step 2: Setting up the syslog server
Firstly, you need to configure rsyslog on the Opsview server. Edit the file rsyslog.conf and uncomment the section at the top:
# Provides UDP syslog reception $ModLoad imudp $UDPServerRun 514
As above, we are going to be using UDP in this example (I’ve found that my Draytek doesn’t like playing with TCP syslog servers, hmm!). Next, give it a restart – “service rsyslog restart” and that’s that configured.
Finally, open up a port on the firewall using a rule such as:
iptables -I INPUT 3 -p udp --dport 514 -j ACCEPT
You can add a “-s” rule if your only allowing a certain subnet, also.
Step 3: Configure the syslog client
On the network device(s), you want to configure it to send the syslogs to the Opsview server. In this example, we are using a Draytek router and as such configure it as below:
As above, we are pointing the router to send syslogs to the Opsview server (192.168.0.15) at port 514 (the one that was allowed through the firewall earlier)
That’s all that is required in honesty – you can tweak the different messages to your preference as above, but in this example we’ll leave everything ticked bar VPN and Call Log.
Step 4: Check logs are received
This is simple, run a command such as:
tail –f /var/log/messages | grep hostname
..where ‘hostname’ is your devices name, i.e. smnet_router as above:
[root@localhost sam]# tail -f /var/log/messages | grep smnet_router Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080] Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080] Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080] Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080] Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080] Jul 29 11:17:39 192.168.0.254 smnet_router: [FW_QoS][Class1][@S:R=13:1, 192.168.0.12:5356->192.168.0.25:8080]
As you can see above, the router is now logging to the Opsview syslog server – terrific!
Step 5: Begin analysis
Next, you can begin to carve up your log files and analyse them using a multitude of methods such as “check_logs” or “check_logfiles” (one of the better open source ones around).
I wont bore you with the specifics, however you can download the package from the website here: http://labs.consol.de/lang/en/nagios/check_logfiles/
You then “configure, make, make install” as you would normally, and then run “check_logfiles” against your newly gathered log files (read the documentation provided in the link for more information).
Essentially, once you’ve built the plugin – copy it and the .pl version to /usr/local/nagios/libexec – and then chmod +x it and also chown nagios:nagios it:
chmod +x check_logfiles* chown nagios:nagios check_logfiles*
This ensures it is now available via the Opsview GUI to be used. You can then configure it however you want. The official blurb says:
“For the most simple applications it is sufficient to call check_logfile with command line parameters. More complex scan jobs can be described with a config file.”
So basically, run something similar to:
check_logfiles –tag smnet_router –logfile=/var/log/messages –criticalpattern=”CRITICAL”
Where –tag hostname is your device, i.e. smnet_router; and “CRITICAL” is the string you want to alert on. In the example below, we have chosen a fairly common string just to demonstrate the plugin is working:
[nagios@localhost libexec]$ ./check_logfiles --tag=smnet_router --logfile=/var/log/messages --criticalpattern="FW_Session" CRITICAL - (494 errors in check_logfiles.protocol-2013-07-29-11-23-11) - Jul 29 11:23:10 192.168.0.254 smnet_router: [FW_Session][Pass][320/60000][@S:R=13:1, 192.168.0.12:5409->192.168.0.16:9890] ...|smnet_router_lines=3382 smnet_router_warnings=0 smnet_router_criticals=494 smnet_router_unknowns=0 [nagios@localhost libexec]$
Step 6: Bring it all together
Finally, you can take the plugin and tests from the CLI and add them into your Opsview GUI as below:
You can even potentially replace “–tag=smnet_router” with “–tag=$HOSTADDRESS$”, so that you can apply the Opsview methodology of ‘create once, apply many times’ – this saves you having to create 1000 service checks for 1000 devices, to check for the word ‘CRITICAL’ for example.
Step 7: Good luck!
So there you have it – a quick and dirty way to monitor syslogs with Opsview in an open source way. Enjoy!
Notes
Whilst I remember, I did have to modify permissions on /var/log/messages to allow the nagios user to view it. You could potentially look at using rsyslog to log remote logs to a different file that nagios has read/execute on – for security reasons.
