Syslog TACACS+ Rancid

ScreenHunter_02 Dec. 15 13.21

ScreenHunter_01 Dec. 15 13.21

Syslog

1. Go to /etc/rc.conf
10.25.x.x -replace for your syslog server IP.
2. go to /etc/syslog.conf and comment everything  what was there originally.
Add your devices at the bottom
3. Create the log files in /var/log/
e.x
touch /var/log/RT01-MUN01.log
chmod 600 /var/log/RT01-MUN01.log
4. /etc/rc.d/syslogd restart
ps ax | grep syslogd
NMS# netstat -an | grep 514
udp4       0      0 *.1514                 *.*
udp4       0      0 *.514                  *.*
udp6       0      0 *.514                  *.*

 

To  archive the Syslog Logs and control the size of the logs use Newsyslog

/etc/newsyslog.conf

600 -permissions to file

7-Number of days to store the archives

20M size of the archive

$W4D6 – $W4- Forth days of the week at 6Am

Z-Archive

ScreenHunter_07 Dec. 15 14.28

******************************************************************************************

 

 

“Zbxlog”

Zbxlog is a standalone daemon written on Perl that receives all the syslog messages forwarded to it by syslog daemon (native syslogd doesn’t do it properly and has to be replaced by Rsyslogd) and put it into Zabbix’s mySQL DB so that you can see those messages in Zabbix WEB interface separately for every device and work with them in a more convenient way.

Zbxlog originally was written for Linux and has been adopted for FreeBSD and Zabbix v2.0.6

 

Original thread on Zabbix forum started by the developer with latest versions of Zbxlog can be found here: https://www.zabbix.com/forum/showthread.php?t=19180&highlight=syslog

 

1.  Checking Perl dependencies

Zbxlog is dependent on the following Perl modules:

POSIX

NetAddr::IP::Util

IO::Socket6

IO::Socket::INET6

Time::Local

DBI

DBD::mysql

Data::Dumper

 

First step to perform before the installation is to check those dependencies if they are installed or not.

Just issue the commands below one by one and notice if you have man pages for every module (case sensitive)

perldoc POSIX

perldoc NetAddr::IP::Util

perldoc IO::Socket6

perldoc IO::Socket::INET6

perldoc Time::Local

perldoc DBI

perldoc DBD::mysql

perldoc Data::Dumper

 

If not, you have to install modules that are missing by issuing cd /<path>/<module name>/ && make install clean

For FreeBSD 8.3 the paths for installation are below:

POSIX – usually installed with Perl itself, called ‘in core’

/usr/ports/net-mgmt/p5-NetAddr-IP

/usr/ports/net/p5-Socket6

/usr/ports/net/p5-IO-Socket-INET6

/usr/ports/devel/p5-Time-Local

/usr/ports/databases/p5-DBI

/usr/ports/databases/p5-DBD-mysql

/usr/ports/devel/p5-Data-Dumper

 

2. Making Zbxlog a service

Issue echozbxlog_enable=”YES” >> /etc/rc.conf so that Zbxlog starts automatically after every reload.

 

3. Copying files

Place the Zbxlog files onto FTP server and copy them into appropriate directories

zbxlog.pl -> /usr/local/sbin/zbxlog.pl && chmod 774 zbxlog.pl

zbxlog.conf.example -> /usr/local/erc/zbxlog.conf.example && cp zbxlog.conf.example zbxlog.conf

zbxlog -> /usr/local/etc/rc.d/zbxlog && chmod 774 zbxlog

All the .pm modules from Zbxlog folder -> /usr/local/lib/Zbxlog/*.pm

 

Folder usr-local-www-zabbix2-include contains patched .php files along with patches, you can copy already patched files for Zabbix web interface or use patch utility if you prefer.

CScreenHistory.php -> /usr/local/www/zabbix2/include/classes/screens/

defines.inc.php -> /usr/local/www/zabbix2/include/

items.inc.php -> /usr/local/www/zabbix2/include/

Don’t forget to back up original filese!!!

4. Configuring zbxlog.conf

Make sure that the following parameters are configured appropriately:

db_name=

db_user=

db_password=

db_type=mysql

db_host=127.0.0.1

db_port=3306

listen_port=1514 (Rsyslogd uses standard UDP 514 port and forwards all the messages to Zbxlog.pl to UDP 1514)

zabbix_version=2.0

 

db_name, db_user, and db_password should be the same as in /usr/local/etc/zabbix2/zabbix_server.conf

 

5. Replacing Syslogd with Rsyslogd

Disable system’s syslog server first issuing

echo syslogd_enable=”NO” >> /etc/rc.conf

 

Install Rsyslogd VERSION 5cd /usr/ports/sysutils/rsyslog5 && make install clean

Make it as a service echo rsyslogd_enable=”YES” >> /etc/rc.conf

 

Copy rsyslog.conf -> /usr/local/etc/rsyslog.conf

 

6. Rebooting the machine and testing the installation

Issue reboot and check that Zbxlog and Rsyslogd are up and running.

NMS# netstat -an | grep 514

udp4       0      0 *.1514                 *.*

udp4       0      0 *.514                  *.*

udp6       0      0 *.514                  *.*

 

NMS# ps ax|grep syslog

37947   2- S      0:29.85 /usr/local/sbin/rsyslogd -i /var/run/rsyslogd.pid –f

 

7. Creating Items in Zabbix WEB interface

Go to Zabbix web interface and create at least two items.

First one for syslog messages that DO NOT match any configured hosts in Zabbix web interface.

Configuration -> Hosts -> Items -> Create item in Zabbix server

ScreenHunter_04 Dec. 15 13.42

 

The second one has to be configured for every host configured in Zabbix web interface:

ScreenHunter_03 Dec. 15 13.42

As a result you should see the messages in Zabbix like the screenshot below.

ScreenHunter_05 Dec. 15 13.42

 

8. Troubleshooting Zbxlog

If there are no any messages in Zabbix it’s time to make sure that Zbxlog gets the syslog messages.

edit /usr/local/lib/Zbxlog/Controller.pm

set

my $DEBUG = 0;

to

my $DEBUG = 1;

 

Restart the service

/usr/local/etc/rc.d/zbxlog stop

/usr/local/etc/rc.d/zbxlog start

 

Have a look into

/var/log/zbxlog.log

 

You should see one line per syslog message received

Read:remoteip=127.0.0.1 remote_host=127.0.0.1 buf=<118>Sep 26 15:15:49 kernel:

Sep 26 15:15:49 <syslog.err> SW04-MUN01 2258: Sep 26 15:15:49.658 mez: %LINEPROTO-5-UPDOWN:

Line protocol on Interface Vlan666, changed state to down

 

Don’t pay attention on

Prototype mismatch: sub Zbxlog::Controller::AF_INET6: none vs () at lib/Zbxlog/Controller.pm line 23

 

If no line of this type appears in zbxlog.log, Zbxlog is not receiving anything.

netstat, tcpdump, iptraf are your friends:)

 

If you have this kind of lines but still nothing in Zabbix disable DEBUG in Controller.pm

edit /usr/local/lib/Zbxlog/Sender.pm

set

my $DEBUG = 0;

to

my $DEBUG = 1;

 

Restart the service

/usr/local/etc/rc.d/zbxlog stop

/usr/local/etc/rc.d/zbxlog start

Look into

/var/log/zbxlog.log

 

In /var/log/zbxlog.log you should see messages sent to Zabbix, such as:

Zbxlog::Sender::Send item=$VAR1 = [

          ‘Zabbix server’,

          ”,

          ‘syslog[]’,

          ’45’,

          ‘syslog’,

          12,

          1380201349,

          ‘from SW04-MUN01: 2261: Sep 26 15:15:49.658 mez: %LINEPROTO-5-UPDOWN:

Line protocol on Interface Vlan666, changed state to down’

        ];

 

Zbxlog::Sender::Send response=ZBXD.W…….{

        “response”:”success”,

        “info”:”Processed 1 Failed 0 Total 1 Seconds spent 0.000018″}

 

Check that response is success and Failed is 0. Otherwise, it means that Zabbix is rejecting this message. In this case, zabbix_server.log may contain useful error messages.

 

You can send a test messages from local machine using:

nc -w0 -u 127.0.0.1 514 “testing again”

**************************************************************************************************

“RANCID”

RANCID is an utility (a bunch of Perl scripts) that monitors a router’s (or more generally a device’s) configuration, including software and hardware (cards, serial numbers, etc) and uses CVS (Concurrent Version System) to maintain history of changes.

 

RANCID has been installed in every location. Local RANCID pulls the configuration files from local Cisco devices only, i.e. routers, switches, and ASA’s. It can be accessible on every NMS server by http://<IP-address>/config/

London  http://<IP-Address/config/

New York http://<IP-Address>/config/

Munich http://<IP-Address>/config/

 

1. Installation & Configuration

Login as a root and issue cd /usr/ports/net-mgmt/rancid/ && make install clean

DON’T check “Use Subversion instead of CVS”

After the installation issue:

cp /usr/local/etc/rancid/rancid.conf.sample /usr/local/etc/rancid/rancid.conf

Open rancid.conf for editing and put there (search for that option in the file):

LIST_OF_GROUPS=”XXX-devices”

where XXX=MUNICH/LONDON/NEW_YORK – depends on location of installation.

 

XXX-devices – is the name of a group of devices, you will see that name in WEB interface after the whole installation is finished

ScreenHunter_06 Dec. 15 13.58

We need a separate user to run RANCID under that user’s account. Below is the listing of adduser process, use it as an example with the same username/groups/shell/etc:

NMS# adduser

Username: rancid

Full name: RANCID user

Uid (Leave empty for default):

Login group [rancid]: wheel

Login group is wheel. Invite rancid into other groups? []: www

Login class [default]:

Shell (sh csh tcsh bash rbash nologin) [sh]: nologin

Home directory [/home/rancid]:

Home directory permissions (Leave empty for default):

Use password-based authentication? [yes]:

Use an empty password? (yes/no) [no]:

Use a random password? (yes/no) [no]:

Enter password: Password

Enter password again: Password

Lock out the account after creation? [no]:

Username   : rancid

Password   : *****

Full Name  : rancid

Uid        : 1002

Class      :

Groups     : wheel www

Home       : /home/rancid

Home Mode  :

Shell      : /usr/sbin/nologin

Locked     : no

OK? (yes/no): yes

adduser: INFO: Successfully added (rancid) to the user database.

Add another user? (yes/no): no

Goodbye!

RANCID installs under root account in the directory /usr/local/var/rancid/ so the owner has to be changed.

chown -R rancid:wheel /usr/local/var/

 

We have to create a file with usernames, passwords, and device IP addresses which we are going to pull the configs from. That file should be in the home directory of the user rancid.

cp /usr/local/share/rancid/cloginrc.sample /usr/home/rancid/.cloginrc

Comment or delete everything in that file and add the following:

(NEW_YORK  used as an example, TAB is a delimiter)

### NEW_YORK devices

## ASA

add user 10.x.x.x<–>backupasa

add password 10.x.x.x<——>Password<—->Password

add user 10.x.x.x<–>backupasa

add password 10.x.x.x<——>Password<—->Password

 

## routers & switches

add user *<—->backupconfig

add password *<>password

add method *<–>ssh

 

Then change the owner and permissions of that file:

chown rancid:wheel /usr/home/rancid/.cloginrc

chmod 640 /usr/home/rancid/.cloginrc

 

2. TACACS changes

We have to add backupconfig user with ability to issue show, dir, wr t, more, etc commands.

(see full config in /usr/local/etc/tac_plus.conf)

### Fro RANCID configs backup on routers ###

user = backupconfig

 

ASA has issues with some auth commands so I use a user with full access (had no time to troubleshoot that)

### Fro RANCID configs backup on ASA ###

user = backupasa

 

3. Test access to devices

A login ability to every device has to be tested before an addition to the list of monitored devices.

XXX-NMS# su -m rancid -c ‘/usr/local/libexec/rancid/clogin -f /usr/home/rancid/.cloginrc 10.x.x.x’

After issuing that command you should get Priv15 command line console access like XXX1-XXX01#

If not have a look .cloginrc file in /usr/home/rancid/ directory for typos or try to login with local SSH client using ssh –l backupconfig <IP> command

 

4. Creating CVS structure and list of devices

To create initial directory structure issue:

XXX-NMS# su -m rancid -c ‘/usr/local/libexec/rancid/rancid-cvs’

After that you will see a new folder in /usr/local/var/rancid/ called as group name in rancid.conf

 

Open for editing /usr/local/var/rancid/XXX-devices/router.db

Add one string per host (if you were able to login with ‘clogin’, if not add a device here with ‘down’ at the end and solve the problem with login)

10.x.x.x:cisco:up

10.x.x.x:cisco:up

10.x.x.x:cisco:up

10.x.x.x:cisco:up

and so on

 

5. Mailing reports about configuration files changes

Edit /etc/aliases

Add

# For RANCID reports

rancid-XXX-devices:<><——>ITnetworkDL@yourcompany.com

rancid-admin-XXX-devices:<–>ITnetworkDL@yourcompany.com

where XXX=MUNICH/LONDON/NEW_YORK – depends on location of installation and group name in rancid.conf

 

Then issue newaliases in console to rebuild aliases DB.

 

6. Automation

Edit /etc/crontab

# RANCID configs backup utility

15<—->5<—–>*<—–>*<—–>*<—–>rancid<>/usr/local/bin/rancid-run

So that RANCID will run at 5:15am every day, pull all the config files, run diff and send a report if there is a difference in configs in comparison to the old ones.

 

7. Troubleshooting

Log files to see what’s going on with RANCID are here

/usr/local/var/rancid/logs/

 

Devices configuration files are here

/usr/local/var/rancid/XXX-devices/configs/

where XXX=MUNICH/LONDON/NEW_YORK – depends on location of installation and group name in rancid.conf

 

To completely delete a host from list of devices do the following:

– delete the IP from router.db

– delete config files from /usr/local/var/rancid/XXX-devices/configs/ & /usr/local/var/rancid/CVS/XXX-devices/configs folders

– go to /usr/local/var/rancid/XXX-devices/configs/ and issue cvs update command

– check /usr/local/var/rancid/logs/ for any errors

– if you still see the ERROR

cvs diff: cannot find revision control file for configs/10.1.6.253

go -> configs DIR and issue ‘cvs update‘ one more time

 

8. WEB interface for CVS tree

Issue cd /usr/ports/devel/viewvc && make install clean command.

DisableSVN support” option.

After the installation edit /usr/local/viewvc/viewvc.conf

Add cvs_roots = rancid: /usr/local/var/rancid/CVS

 

Edit Apache config file /usr/local/etc/apache22/httpd.conf

Add the alias:

Alias /config “/usr/local/viewvc/bin/cgi/”

  <Directory “/usr/local/viewvc/bin/cgi”>

    AddHandler cgi-script .cgi

    Options NONE +ExecCGI

    DirectoryIndex viewvc.cgi

    Order allow,deny

    Allow from all

  </Directory>

 

Save and issue apachectl graceful

Check availability http://XXX-IP/config