Add event handler in nagios

It'll be cool if nagios is going to handle some of your tasks while an incident happens on your application servers. Nagios has feature of eventhandler. Let say nagios send a warning/critical alert for high memory on an application server and you need a stats from logs like number of request per second in last 10 mins. Network transactions on api server in last 10 mins. Need to restart a service and send a mail regarding the same. Kind of any task after triggering an alert or state of change in nagios service.

You need below thing to be pre-configured in your environment

• Nagios monitoring with Nrpe
• Nrpe client compiled with –enable-command-args
• Nagios Services would be configured with Nrpe
• Shell Script for task needs to be done after an incident/alert received 

I expect nagios 3.0 is running in your environment which monitors your hosts and services

This example is set with client server is on ubuntu and nagios 4.1.1 compiled version is monitoring

To allow nrpe to add arguments compile Nrpe with –enable-command-args on client/application server

#wget http://sourceforge.net/projects/nagios/files/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz

#tar -zxvf nrpe-2.15.tar.gz; cd nrpe-2.15

#/configure --with-ssl=/usr/bin/openssl --enable-command-args --with-ssl-lib=/usr/lib/x86_64-linux-gnu && make all && make install-plugin && make install-daemon && make install-daemon-config

Add command in client $NRPE_HOME/etc/nrpe.cfg (Here $NRPE_HOME=/usr/local/nagios)

command[check_mem_rep]=/usr/local/nagios/libexec/check_mem.sh -w 72 -c 75

command[api-rep-gen]=/usr/local/nagios/libexec/run-api.sh $ARG1$ $ARG2$ $ARG2$  

On Nagios server, add a command in $NAGIOS_HOME/etc/objects/commands.cfg

define command{

command_name my_eventhandler

command_line /usr/local/nagios/libexec/check_nrpe -H $HOSTADDRESS$ -c api-rep-gen -a '$ARG1$ $ARG2$ $ARG3$'

}

define service{

use prod-service

host_name kings-api-server-5

service_description run-api-report

check_command check_nrpe!check_mem_rep

max_check_attempts 1

notifications_enabled 1

event_handler my_eventhandler!$SERVICESTATE$ $SERVICESTATETYPE$ $SERVICEATTEMPT$

}

Linux Boot-Process

Linux Boot Process

• Power on
• Power will go through SMPS to main board.
• On main Board BIOS chip / ROM will wake up first
• Then BIOS will loaded in memory. It is also called as boot strapping.
• Now BIOS will take H/W inventory. Like keyboard, monitor, Display card etc.
• Bios will go to CMOS for bootable device list and sequence
• Bios has only target to load Kernel in RAM
• Bios will now load first sector of bootable device i.e. HDD in RAM. Also known as MBR

- MBR : Master boot Record

• Cylinder 0, Head 0 and Sector 1.
• In fdisk -l we have seen size of sector is 512 bytes. So MBR is of 512 bytes
• MBR is devided in 3 parts : Boot Sector ( BS ), Partition Table ( PT ) and Magic number.

0x228

<=========x=================== 512 bytes ========================>

| |

<-----446 bytes ---->| <--PT -64 bytes->| < Magic no 2 bytes--->

BS | PT | Magic #

| |

<============================ 512 bytes ========================>

M B R

• Boot sector is of 446 bytes which actual contains boot loader of all OS loaded in Hard Disk. On 0x228 byte of BS there is a tiny set of binaries which will help Bios to redirect to Linux loader
• Partition Table contains information about partition like primary, extended, size, format etc. It size 64 bytes.
• Magic number is 2 bytes which is like a signal that all is OK. This is also called as MBR signature
• Continue to step 8.......... Bios come to MBR where 1^st stage boot loader resides which has the target to load 2^nd stage boot loader i.e. GRUB in older version of linux LILO. 1^st stage boot loader has the information about GRUB.
• BIOS can understand binary. Grub configuration is also in MBR. BIOS will take CHS number of splash image from GRUB configuration and load in in RAM.
• BIOS will also read default kernel and time out from GRUB configuration. It will then display GRUB message on default selected kernel till time out seconds.
• After time out Bios will then read CHS number of kernel i.e. vmluniz-2.6.18..... . Now Bios will pull the kernel in Ram from specified CHS and hand over all the controls to Kernel i.e. vmluniz-2.6.18.....
• Now Kernel is king of all and he is going to call Process. Kernel makes another call through Bios for initrd image. Now initrd .img will extrract all drivers to load real

What is initrd : initrd maens Initial Ram Disk to mount a temporary file system in Ram

• To see what is in initrd mount initrd.img to one directory

# cp /boot/initrd-2.4.20-8.img /tmp - Let's play with a safe copy

# cd /tmp

# mv initrd-2.4.20-8.img initrd-2.4.20-8.img.gz --> gunzip need extn

# gunzip initrd-2.4.20-8.img.gz

# mount -t sysfs -o loop initrd-2.4.20-8.img /win

# cd /win

# ls

• Initrd contains driver of ext3, jdb etc
• Kernel will now mount “ / “ with the help of ext driver. Now he can access hard disk partition
• Kernel call its 1^st process /sbin/init

Init Tab

# Author:       Miquel van Smoorenburg, 
#               Modified for RHS Linux by Marc Ewing and Donnie Barnes

id:3:initdefault:

# System initialization.
si::sysinit:/etc/rc.d/rc.sysinit

l0:0:wait:/etc/rc.d/rc 0
l1:1:wait:/etc/rc.d/rc 1
l2:2:wait:/etc/rc.d/rc 2
l3:3:wait:/etc/rc.d/rc 3
l4:4:wait:/etc/rc.d/rc 4
l5:5:wait:/etc/rc.d/rc 5
l6:6:wait:/etc/rc.d/rc 6

# Things to run in every runlevel.  This line is only in Red Hat 7.X Used to flush disk buffers.
ud::once:/sbin/update

# Trap CTRL-ALT-DELETE
ca::ctrlaltdel:/sbin/shutdown -t3 -r now

# When our UPS tells us power has failed, schedule a shutdown for 2 minutes from now.
pf::powerfail:/sbin/shutdown -f -h +2 "Power Failure; System Shutting Down"

# If power was restored before the shutdown kicked in, cancel it.
pr:12345:powerokwait:/sbin/shutdown -c "Power Restored; Shutdown Canceled"

# Run gettys in standard runlevels
1:2345:respawn:/sbin/mingetty tty1
2:2345:respawn:/sbin/mingetty tty2
3:2345:respawn:/sbin/mingetty tty3
4:2345:respawn:/sbin/mingetty tty4
5:2345:respawn:/sbin/mingetty tty5
6:2345:respawn:/sbin/mingetty tty6

# Run xdm in runlevel 5
x:5:respawn:/etc/X11/prefdm -nodaemon

Init is the 1^st process called by Kernel

Init works as per /etc/inittab file which is specified above. In redhat linux it seems like this

First it will check run level In the above example it is 3 i.e. command line full user mode with networking

Then it will call /etc/rc.sysinit in a different sub-shell.

rc.sysinit will

• Run initlog
• Sets minimal Path
• Check for network configuration and sets hostname as per file
• It calls /etc/init.d/functions this script will call mainly 17 functios

1. daemon - Start a program
2. killproc - Stop a program
3. pidofproc - Find the process ID of a program
4. status - Gets the status of a process (running, dead, locked)
5. echo_success - Echo OK to the screen in proper colors.
6. echo_failure - Echo FAILED to the screen in proper colors.
7. echo_passed - Echo PASSED to the screen in proper colors.
8. success - Log to the system logger that something succeeded and echo_success if bootup is verbose
9. failure - Log that something failed and echo_failure if verbose bootup.
10. passed - Log that something passed and echo_passed if verbose bootup.
11. action - Run some action, log output to the system logger.
12. confirm - Confirm whether we want to run this service.

• /sbin/loglevel will sets the log level

0 - The system is unusable

1 - Action must be taken immediately

2 - Critical condition

3 - Error condition

4 - Warning condition

5 - Normal but significant condition

6 – Informational

6 - Debug message

• Configures Kernel Parameter
• Welcome Message “Welcome To CentOS...”
• Map the KEYTABLES
• Raid Setup
• Mounts filesystem from /etc/fstab
• Check mouting options fro fsck and dumping options
• Checks quotas, sets quota parameter
• Mounts all file system like except NFS and /proc which is already mounted
• Re configures system files like keynoard, network
• Start up swap
• Initialised serial ports
• Run dmesg and dumps to /var/log/message

• As specified in runlevel id it is 3 so /etc/rc script will run all the script from /etc/rc3.d in a different sub-shell
  • See the correct run level
  • checks previous run level
  • Checks run level in and see whether rc<run-level>.d exist or not
  • /etc/rc script will kill all services which start with letter 'k' and starts all scripts which starts with 's'

• /etc/rc.d/rc.local script will run after starting of all services. Even we can also add scripts/commands to run in start up
• Then As per inittab ctrl-alt-tab trap will get set
• Now ups consideration will get set
• 6 min-getties will come out in Ram if runlevel 3 is there and if run level 5 is there then it will run UI scripts from /etc/X11/prefdm
• A getty is is a program that opens a tty port, prompts for a login name,

and runs the /bin/login command. It is normally invoked by init.

• The mingetty daemon is used to listen for virtual consoles

(like the 6 that run by default with your keyboard and monitor)

and cannot be used to handle serial lines

• The first mingetty [awake one] loads device driver /dev/tty1

• execs /etc/issue
• mingetty then executes the /bin/login process

and is then put to sleep state by init

• /bin/login displays login. You login!
• /bin/login execs /usr/bin/passwd which challenges for passwd from

user which then does authentication and authorization using PAM

• init then puts the login daemon into sleep state, wakes up

mingetty which now takes over

• mingetty checks for .hushlogin in $HOME/$USER
• If $HOME/.hushlogin DOES NOT EXISTS

then it does the foll :

• execs lastlog* -u $USER using /var/log/lastlog
• execs /etc/motd
• checks users mail, if any, in /var/spool/mail/$USER
• init then wakes /bin/login process
• mingetty then goes into zombie state and is killed by init
• login then loads /bin/bash as a monitored child process
• /bin/bash takes over and login goes into sleep state
• Execs /etc/profile Sets system-wide ENV variables Global Profile
• exec /etc/inputrc Sets keyb mappings [See stty -a]

Show Terminal characteristics

• stty -echo --> Terminal Characteristics
• eg /etc/inputrc See man bash - /bell-style

============

set bell-style [audible] [none] [visible]

set disable-completion [off] on

• exec /etc/profile.d/*.sh [13 shell scripts are executed]

1. colorls.sh

2. glib2.sh

3. gnome-ssh-askpass.sh

4. krb5.sh

5. lam.sh

6. lang.sh

7. less.sh

8. mc.sh

9. pvm.sh

10. qt.sh

11. vim.sh

12. which-2.sh

13. xpvm.sh

• Execs /etc/bashrc - Sets system-wide [Global] aliases/shell functions

sets the users and root's final global umask

• execs user's personal .bash_profile
• execs .bashrc Run after every command you execute
• .bash_history
• .bash_logout
• Finally, at last, the comforting shell login prompt bash !!

Install Cacti-0.8.8b on Ubuntu 12.04

It`s really difficult to find any single and straight document on installation of Cacti-0.8.8b on Ubuntu 12.04,  which includes "plugin management" in built. So below post is created will help in installing Cacti with plugin management

Step 1

Upgrade and install Cacti dependent applications

#apt-get update

#apt-get install elinks

#apt-get install python-software-properties

#apt-get install snmpd snmp mysql-server apache2 libapache2-mod-php5 php5-mysql php5-cli php5-snmp

If you have not set up MySQL, you will be prompted for a root user password. Make your selection and confirm the password to continue.

Step 2

Add repository to install latest version of Cacti ( i.e. Cacti-0.8.8b)

#add-apt-repository ppa:paul-climbing/cacti

Edit below lines from newly added repository

#vim /etc/apt/sources.list.d/paul-climbing-cacti-precise.list

change word “precise” to “quantal” in both line.As cacti does not need to change between most Ubuntu releases

deb http://ppa.launchpad.net/paul-climbing/cacti/ubuntu precise main

deb-src http://ppa.launchpad.net/paul-climbing/cacti/ubuntu precise main

to

deb http://ppa.launchpad.net/paul-climbing/cacti/ubuntu quantal main

deb-src http://ppa.launchpad.net/paul-climbing/cacti/ubuntu quantal main

Step 3

Update repository and install cacti

#apt-get update

#apt-get install cacti

Next, press “Return” or “Enter” to acknowledge a configuration change in php.

The initial configuration of Cacti also happens during installation. There are a few questions you need to answer. Select “Apache2” from the list of webservers.

After Cacti configures apache, the installation sets up a MySQL account for the application. Select “Yes” to allow a generic database configuration.

Provide the password for the administration of the Cacti database that you set up during the MySQL configuration.

Next, it asks for a password for Cacti to use with the database. This is an internal password that you should not ever have to use, so it is okay if you just press “Enter” to create a random password.

As installation completes one apache error will be seen. To overcome this error take backup of /etc/apache2/conf.d/cacti.conf and replace with below configuration

_____________________________________________________________________________

Alias /cacti /usr/share/cacti/site

<Directory /usr/share/cacti/site>

Options +FollowSymLinks

AllowOverride None

order allow,deny

allow from all

AddType application/x-httpd-php .php

<IfModule mod_php5.c>

php_flag magic_quotes_gpc Off

php_flag short_open_tag On

php_flag register_globals Off

php_flag register_argc_argv On

php_flag track_vars On

# this setting is necessary for some locales

php_value mbstring.func_overload 0

php_value include_path .

</IfModule>

DirectoryIndex index.php

</Directory>

Restart Apache

#/etc/init.d/apache2 restart

Install Cacti-spine

#apt-get install cacti-spine

Step 4

Make Below changes in SNMP

#vim /etc/snmp/snmpd.conf

Open the file and find the section labeled Agent Behaviour. Comment out the line that specifies 127.0.0.1 as the agent address by placing a # in front of it. Uncomment the other line that defines the agentAddress as all interfaces.Also uncomment rocommunity and place address of your network. The Agent Behavior section should now resmble the following:

# Listen for connections from the local system only

#agentAddress udp:127.0.0.1:161

# Listen for connections on all interfaces (both IPv4 *and* IPv6)

agentAddress udp:161,udp6:[::1]:161

rocommunity secret 172.16.0.0/24

Restart SNMPD

#/etc/init.d/snmpd restart

Step 5

Configure Cacti with web interface.

The rest of the configuration will be done through a web browser. Open your web browser and navigate to your server ip address or domain name with “/cacti” on the end.
http://<server-ip>/cacti
The first page you will see is an introduction to the Cacti software. Click “Next >>” when you are finished reading. Click "Next >>" again on the following page since this is a new installation.

The next page shows the application paths of the “helper” applications that Cacti
uses to operate. All of the applications should be green and marked with “[FOUND]”. Click “Finish” to continue.

Next, you’ll be asked to enter the Cacti user name and password. These are not the passwords you entered during installation. Instead, enter the following default values.
Username : admin
password : admin
You’ll be prompted to enter a new password for administrating Cacti. Choose a password and click “Save”.

You are now on your Cacti page.

Version will be 0.8.8b ( On left hand upper side corner)
In configuration you could see Plugin Management




Short summary of steps
Update/Upgrade repository and Ubuntu packages
apt-get update
apt-get upgreade
Install Dependency applications
apt-get install elinks
apt-get install python-software-properties
apt-get install snmpd snmp mysql-server apache2 libapache2-mod-php5 php5-mysql php5-cli php5-snmp
Install private repository to install Cacti
add-apt-repository ppa:paul-climbing/cacti
vim paul-climbing-cacti-precise.list
apt-get update
apt-get install cacti
Correct Apache settings and restart apache
cp /etc/apache2/conf.d/cacti.conf /root/
echo > /etc/apache2/conf.d/cacti.conf
vim /etc/apache2/conf.d/cacti.conf (Edit file as mentioned above)
/etc/init.d/apache2 restart
Install cacti-spine
apt-get install cacti-spine
Configure SNMP
vim /etc/snmp/snmpd.conf  ( Edit file as mentioned above )
/etc/init.d/snmpd restart
Configure Cacti first-time confiuration with Web-UI