The Big Brother Systems and Network Monitor
Big Brother
A Web-based Systems and Network Monitoring
and Notification System
Installation and Configuration Manual
The Big Brother Systems and Network Monitor
------------------------------------------------------------------------
Installation and Configuration Manual
Version 1.09b - February 21st 1999
------------------------------------------------------------------------
1.0 Quick and Dirty Install for Big Brother
2.0 Configuring the BB hosts file
2.1 The etc/bb-hosts file
2.2 Basic format of etc/bb-hosts
2.3 etc/bb-hosts special directives
3.0 Customizing your BB installation
3.1 The runbb.sh script
3.2 Configuring the etc/bbdef.sh file
3.3 Localizing programs paths
3.4 Creating BB clients
3.5 Configuring the Web Stuff
3.6 History
4.0 Configuring the Notification feature
4.1 A Little History
4.2 Setting up notification
4.3 Creating notification rules
4.4 Acknowledging notification messages
4.5 Manual notification (Web based)
4.6 SMS notification
4.7 Using kermit
4.8 Adding a custom notification
5.0 Miscellaneous Big Brother configuration options
5.1 Customizing the HTMLized status logs
5.2 Specifying filesystem specific values in the disk test
5.3 Enabling security
5.4 Inserting notes about your hosts
5.5 Customizing the footer for bb.html/bb2.html
6.0 Other documentation sources
------------------------------------------------------------------------
This Installation and Configuration Guide is Copyright 1997-1999
by The MacLawran Group Inc. This document may be reproduced, so
long as it is kept in its entirety and in its original format.
------------------------------------------------------------------------
------------------------------------------------------------------------
Section 1: Quick and Dirty Install for Big Brother
From the directory you extracted the BB archive:
1. cd ./doc, and run ./bbconfig
where OS-NAME is freebsd, solaris, linux, hpux9, hpux, sunos,
aix, ultrix, osf, netbsd, freebsd, redhat, bsdi,
sco3, sco, irix, unixware
2. cd ../src
type "make"
then type "make install"
3. cd ..
edit runbb.sh, set BBHOME to the installation
directory.
4. cd etc
edit bb-hosts, put your host entries in there.
This is the core of Big Brother.
Then edit bbdef.sh to set alarm levels and things.
Edit bbwarnsetup.cfg and bbwarnrules.cfg
to set up the notification feature. If you want to use
fully qualified domain name hosts then make sure you set FQDN=TRUE
in etc/bbdef.sh.
5. execute ./bbchkcfg.sh - Checks the bbdef.sh/bbinc.sh/bbsys.sh
source scripts for invalid entries
execute ./bbchkhosts.sh - Checks the bb-hosts file for errors
6. ln -s $BBHOME/www /WWW/bb (where /WWW is the Document Root dir).
Make sure the permissions are correct. Configure your web server
for this directory if need be.
7. For Web-based history, edit web/bb-hist.sh and install
it in your cgi-bin directory. To get text-based history
set the variable WEBHISTORY to "FALSE" is etc/bbdef.sh.
8. cd ..
nohup ./runbb.sh &
examine the nohup.out file for any errors
9. This will have enabled you to have a display/pager server set up.
Point you browser to the /bb path of your web server to view
the results. At this point no clients are running. Define
all of your hosts in the etc/bb-hosts file and when
you have all of your hosts defined in etc/bb-hosts then
use doc/bbclient to create a tarball for BB clients
of the same OS/HW type. If you have different OS/HW platforms then
reinstall BB, as a client, on each one (then use the bbclient to
create a tarball for each identical OS/HW client) and don't forget
to copy your master bb-hosts file to it. Run through the install
procedure to make sure that the clients are also installed
properly (bbchkcfg.sh/bbchkhosts.cfg)
------------------------------------------------------------------------
Section 2: Configuring the Big Brother hosts file
2.1 The etc/bb-hosts file
The etc/bb-hosts file controls where Big Brother looks for things
and the actions that are taken. The format is identical to the
standard /etc/hosts file, except with additional directives
for Big Brother.
The Big Brother hosts file is located at etc/bb-hosts.
2.2 Basic format of etc/bb-hosts
Lines are of the format:
#
IP-ADDR: XXX.XXX.XXX.XXX
HOSTNAME: host.domain.com if FQDN="TRUE"
host if FQDN=""
FQDN is a variable that you set in the etc/bbdef.sh file.
Directives the Big Brother knows about are:
BBDISPLAY This host displays the HTML results
You need a Web server on this host
BBPAGER This host acts as the notification server
BBNET Indicates that this host checks
the ip network services
http:// The host is tested for http connections
using the
ftp Test for ftp service
smtp Test for smtp server
pop3 Test host for pop3 server
telnet Check telnet service
ssh Test ssh server
nntp Test nntp news server
Check /etc/services for proper service name
especially pop3 (sometimes referred as pop-3)
These directives names MUST appear in /etc/services
dns Checks for name resolution server
noping Don't do ping test for this host
dialup If host is down then display clear button
This is a sample bb-hosts file:
#
# BIG BROTHER HOSTS FILE
#
204.101.110.102 solario4 # BBDISPLAY BBPAGER ftp smtp pop3
204.101.110.95 admin # http://admin/
204.101.110.108 ctnet # BBNET
solario4 is the display and notification server.
ctnet is the host that runs ip network services checks
it will check the ftp/smtp/pop3 services on solario4,
then the http service on admin
You can also group hosts within a seperate HTML table. They
are two group tags defined: group and group-compress.
It is best to have the master bb-hosts file identical on all
BB server and client hosts.
Changes to the etc/bb-hosts file are automatic. You do not
have to restart BB to initialize the changes.
2.3 etc/bb-hosts special directives
2.3.1 Grouping on the display
The Web display may be broken into tables to create
a more aesthetic and sensible display. It's also much faster
to load small tables, than one giant table. So consider
grouping your etc/bb-host entries logically and separating
them using the "group" or "group-compress" directives.
group "Intranet Servers
"
This directive defines a block of hosts to be grouped
in the same HTML table. All hosts following the group
directive, until a new group/group-compress is defined, belong
to that group. The text that follows the directive is the
title given to the HTML table. Note that you can embed HTML
code in the title: Italic / H3, but use with caution.
group-compress "Group name"
The group-compress is identical to the "group" directive
except it will only display services (columns) containing data
for that group.
2.3.2 DHCP hosts (no fixed IP address)
0.0.0.0 dhcphost.domain #
The 0.0.0.0 indicates that this host is a DHCP
host and that the BBNET host will not try to run the
connection test on that host.
2.3.3 Modem Banks
dialup modem-bank 204.19.116.20 4
The dialup directive (not to be confused with the other
dialup tag which displays a clear button if the host is down)
is used to specify connectivity for a bank of modems. The
2nd parameter is the name to be displayed on the display page.
The 3rd argument is the starting IP address of the modem bank.
The last argument is the number of modems on that bank.
2.3.4 Summary Lines
summary quebec.mtl 255.255.255.255 http://cafe.domain.com/bb/
This indicates to the BBDISPLAY machine that
summary information about the state of this display is to
be forwarded to the IP address noted on the line. The
summary can be sent to more than one parent machine.
------------------------------------------------------------------------
Section 3: Customizing your BB installation
3.1 The runbb.sh script
The runbb.sh script is the master BB script. It sets up the
BB environment with the values from the configuration files.
You MUST set the BBHOME variable to the directory path where
you have installed BB. Without it, BB will not run properly.
3.2 Configuring the etc/bbdef.sh file
etc/bbdef.sh is where you configure how Big Brother reacts to a
variety of situations that may arise. You can establish levels
for warnings(yellow), panics(red) values.
Please note that any modifications to this file does not come
into effect until the next restart of BB. It is best to
stop/start BB immediately after this file has been modified.
Use FQDN (fully qualified domain naming)
FQDN="" - Host names rendered on the BB status pages will only
include the host name, not including the domain
name.
FQDN="TRUE" - Host names rendered on the BB status pages will
include the host name and the domain name.
Which program to use to check the HTTP connectivity:
LYNX="$BBHOME/bin/bbnet" - This indicates where bbnet (a C program
that checks the IP services) is located.
LYNX="/usr/local/bin/lynx -dump -head" - Use lynx if you need to
check password protected pages. Please use the proper
directory path as per you host installation.
NONETPAGE defines which IP services not to notify on if an error
is encountered. Include the services as they are named
in the etc/bb-hosts file.
BBTMP indicates the location of the BB temporary directory. You
may want to locate outside of the BB structure.
The DFWARN and DFPANIC variables contain the % levels at which
the disk test will fail. By default DFWARN is 90% and
DFPANIC is 95%. These values can be overriden by values
in the etc/bb-dftab file described in section 5.2.
CPUWARN and CPUPANIC are default levels based on the load
average. CPUWARN is set at 150 and CPUPANIC at 300.
These values are the load average multiplied by 100.
You can define the processes for which you want to monitor if
they are up and running. You do this by setting the
PROCS and PAGEPROCS variables. The processes included
in PROCS will lead to a yellow condition if one the
processes is down. Processes in PAGEPROCS will generate
a red condition if a process is down.
MSGS and PAGEMSGS are used to monitor error messages in the
system logs. MSGS contain keywords to look for in
the system logs. If a keyword is found then this
will create a yellow condition and if the keyword
is in the PAGEMSGS variable then it will be upgraded
to a red condition.
BBWEBHTMLLOGS="http://www.maclawran.ca/bb/html/"
This variable defines the path to the HTMLized status logs. This
path and the HTML file is appended to the e-mail
notification message for a quickpath to the error
information.
The PAGELEVELS variable indicates to the 'bb' process at which
level(s) it should send a notification message. The 'bb'
program is used to send status messages and if the status
level of the message is found in the PAGELEVELS variable
then a notification message is also sent to the BBPAGER
host. This is the default value. It can be overridden
by the pagelevels: token in the etc/bbwarnsetup.cfg
file.
3.2 Localizing programs paths
Sometimes the OS specific programs paths found in the
installation might be wrong for a particular host(s) OS.
Change the paths in the etc/bbsys.local file. The variables
contained in this file have all default values in bbsys.sh
The original bbsys.local is copied from the etc/bbsys.OS file
at installation time.
3.4 Creating BB clients
Creating BB clients is very simple:
cd $BBHOME
cd doc
./bbclient
must be found in the etc/bb-hosts file.
This will create a tarball for the client that is the same
OS/HW platform type as the host on which the bblient command
is executed.
If you have different OS/HW platforms then reinstall BB,
as a client, on each OS/HW platform (then use the bbclient to
create a tarball for each identical OS/HW client) and don't forget
to copy your master bb-hosts file to it. Run through the install
procedure to make sure that the clients are also installed
properly(bbchkcfg.sh/bbchkhosts.cfg)
3.5 Configuring the Web Stuff
All Web-based things live in the www directory. In order for
the bb stuff to work correctly, this directory must be linked
into your Web site somewhere. I suggest something like the following:
ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir).
You should then be able to access BB with the URL
http://.../bb/bb.html
where ... is your webserver address.
You also have to configure the script which generates the web-based
history page. This page is accessible from the "History" button when
you click on a colored dot in one of the display pages (bb.html/bb2.html).
You must copy the web/bb-hist.sh script to your /cgi-bin directory
and set the BBHOME and BBWEB variables in the script. Make sure the
script has the proper permissions to execute: system and web server permissions.
If you want only a text based history display then set the WEBHISTORY
variable to "FALSE" in etc/bbdef.sh.
If you intend to enable manual notification ("PAGE/ACK" button)
and the acknowledgement module then copy the web/bb-ack.sh
script to you /cgi-bin directory and set the BBHOME and BBWEB
variables in the script. Make sure the script has the proper
permissions to execute: system and web server permissions.
3.6 History
Each time a status log is received it is checked against its
previous status to see if it has changed. If it has, then
the duration of the previous status is recorded and a new line
is appended to the history file with the timestamp of the
new status.
There a history file for each host.service (status log) received.
The history files are located in the www/hist directory.
Here's a sample:
Sun Jan 24 18:30:45 1999 green 917220645 232068
Wed Jan 27 10:58:33 1999 red 917452713 1560
Wed Jan 27 11:24:33 1999 green 917454273
Notice that last line doesn't have a duration because it is
the current status. It will inserted only when the status
changes.
If you do not wish to have historical data then rename
www/hist to something else or remove it.
------------------------------------------------------------------------
Section 4: Configure the notification feature (BBWARN)
4.1 A little History
In the Beginning :), BB could only e-mail and notify by pager.
The configuration required that all BB hosts (servers and
clients) have the PAGER variable set with the recipients of the
notifications. They were no possibilities of customizing the
notification feature. This led to the:
The Pager Protection Act of 1997
================================
Use of BB has been linked to the untimely death of at least one
pager in Texas. In order to protect pagers from their users, a
notification scheme has been created to allow to specify a delay
between the notifications.
But this wasn't enough. BB netizens got restless and finally a
solution was delivered by Robert-Andre Croteau, and described in
an article in SysAdmin magazine. It was called BBWARN and it was
good. It allowed an admin to specify rules which enable him/her
to get notified based on the source of the error (host) and the
service involved and also based on the day/time. Different rules
could be set up and different recipients could be specified depending
on the characteristics of the problem (host/service/day/time).
Numerous changes have been made and various configuration
options are available to the BB admin(s).
Notification methods
You can currently get notified by email, numeric pager and
SMS communication device. Users have reported custom
notification by alpha-numeric pagers and other PCS devices.
To use the email feature, your BBPAGER host must have an
email program. This is not a problem on UNIX: the mail (mailx)
program is available.
Sending a notification message to a numeric pager requires
kermit and a modem. The path of kermit should be setup in
the KERMIT variable of bbsys.sh/bbsys.local variable.
4.2 Setting up notification
All of the BB notification configuration is specified in the
etc/bbwarnsetup.cfg file. Here are the various options that
can be set. Note that instructions are included in the
configuration file as comments.
All tokens in the etc/bbwarnsetup.cfg file must be defined
on the BBPAGER host. The pagelevels: token must be defined
on all BB hosts (servers/clients) because the 'bb'
executable send the notification and that is done locally
on each client host.
bbwarn: Set to "TRUE" is you want to enable the notification
feature
svcerrlist: This contains a list of service/code pair. The service
name is the column found in the HTML display and
the code is the value displayed on a numeric message.
ignforall: Is a regular expression that is used to temporarely
disable notification a host.service combination(s)
e.g. .*.cpu|.*.msgs|host11.*
Don't notify for any cpu/msgs errors and any errors
concerning host11.
pagehelpcode: Numeric code to use when a user sends a manual
notification. See section 4.5.
ttyline: List of modems devices. You can specify more than one.
prefix: Prefix to use when dialing out
suffix: Suffix to use when dialing out
pagelevels: At which color level to send a notification.
This token is set on every host.
NB. There is the PAGELEVELS variable defined in etc/bbdef.sh
that also needs setting up if you want to notify on other levels
than "red" and "purple". The PAGELEVELS is the default value.
pagetype: Defines how the pager delay is handled. There are
4 choices (RCPT|EVENT|HOST|GROUP).
RCPT: the recipient is not notified until pager delay
expires.
EVENT: the recipient is not notified for a particular
host.service combination until the pager delay
expires.
HOST: the recipient is not notified for a particular
host until the pager delay expires.
GROUP: the recipient is not notified for a particular
host.service within the same etc/bb-hosts group
combination until the pager delay expires.
pagemaster: Recpient(s) to receive an e-mail notification if
a page notification could not be sent.
pageaddhtmlpath: Set to TRUE if you want the HTML path of
the status log to be appended to an em-mail
notification. If this is set to TRUE, make sure
BBWEBHTMLLOGS variable is set correctly in etc/bbdef.sh.
cfgdelim: entry delimiter in the etc/bbwarnrules.cfg
4.3 Creating notification rules
You define the notification rules in the etc/bbwarnrules.cfg
file
Rules are written in the following format:
hosts;exhosts;services;exservices;day;time;recipients
hosts: match on these hosts (* is a wildcard for all hosts)
exhosts: exclude these hosts
services: match on these services (* is wildcard for all hosts)
exservices: exclude these services
day: 0-6 (sunday-saturday)
time: 0000-2359
recipients: email address, numeric pager, sms number
The value set by "cfgdelim" (in bbwarnsetup.cfg) is your
filed delimiter. In this case the value is ';'
Even though egrep regular expressions are allowed do not use
the .* construct, just use *. It will be replaced with
.* in the regexp. It's just that * is more readable than .*
Also the default "pagedelay" value (see bbwarnsetup.cfg)
which indicates how long before the next notification occurs
can be overridden on the rule line for a specific recipeint
by appended the time value to the recipient:
recipient:XX where XX is the value in minutes.
For examples please refer to the etc/bbwarnrules.cfg and
the etc/bbwarnsetup.cfg file for complete information.
4.4 Acknowledging a notification
When an admin is notified, the admin is always sent an
acknowlegement tag with the message. This tag number
is seven digits: First five digits are a unique number
and the last two digits are the recipient's ID for that
notification (recipient's ID can change, it is valid
only for the current notification) valid
In an e-mail notification, the ack number is in the
subject and the body of the message: in the subject
it is found at the beginning !BB - XXXXXXX! and in the
body of the message, it is at the beginning of the
body delimited by [XXXXXXX].
In a numeric page, the acknowledgement is always after
the BB numeric code. It is the last 7 digits of the
numeric message. NOTE, if your pager doesn't support
more than 21 digits, then you're out of luck, you'll
be missing some digits. You could always try to
shorten the ack number but that is left as an exercise
for the reader ;-). If you get the first 5 digits
then you can always use recipient 99 to acknowledge
for all recipients but that may not fly at your site
especially if you have multiple recipients.
4.5 Manual notification (Web based "PAGE/ACK" button)
You can enable the manual notification feature and
let the user reach you by pager/e-mail if they can't get
a hold of you. If this is enabled, then you'll
receive a numeric message starting with 911 (pagecode
in etc/bbwarnsetup.cfg) followed by the phone
number to call.
You can setup who gets called when by adding a rule(s)
in the etc/bbwarnrules.cfg. The pseudo host to use
is notify-admin*. There's an example in the config
file.
4.6 SMS notification
This a user custom notification scheme for SMS devices.
What follows is the README file for his custom hack.
Last updated: 1997-01-11 by jaclu@ibk.se
This section describes the usage of a kermit script that
sends SMS messages.
It is used as a mail -> SMS gateway, and also for the "Big Brother"
network monitor(obviously).
The kermit script for SMS can be found in etc/sms.scr
My operator uses NOKIA server software, hopefully you can use
this also on Motorola servers, without to much hazzle.
Terms
SMS Short Message Service
SMSC Short Message Service Center
CIMD Computer Interface to Message Distribution
Syntax
kermit sms.scr device file number [number [number [...]]]
device is modem device like /dev/cua0
The file should contain the message, only first line of file
will be sent, and it MUST be terminated with LF, otherwise the
kermit script won't be able to extract the message.
SMS messages can only be 160 chars, and since nobody would want
a lengthly message on a 8 character display, I haven't bothered
to handle message splitting. If message is too long, it is truncated.
If one or more SMS phonenumbers can be specified, the message will be
sent to each one of them.
Configuration
I have done this the easy way, that is I start CIMD with
faked checksums, if anybody manages to generate correct checksums,
please mail me.
Log in to SMSC
You have to set two variables in sms.scr:
\%b phonenumber to you operators SMSC
\%c login to access CIMD
For Europolitan (Swedish operator) you can use:
\%b 46708222902
\%c cimd3 # (use faked checksums)
This starts the CIMD server, and from now we must ACK
all responses.
Don't forget to include your modem prefix to \%b!!
I don't use the "prefix" token from BB, since I also use
this script as a mail->SMS gateway. If you are going to
use this, you must anyhow configure this script, so I hope
you don't mind.
Start a CIMD session
The first step in a CIMD session is to identify yourself
You have to set two variables in sms.scr:
\%d CIMD account
\%e CIMD password
Unfourtunately, I cant give you theese values ;)
Now you have configured your script and can start to use it!
Send to all recipients
BB calls the script one time for each recipient, that's
a bit of a waste, but it would just be to much work to
get around. If you call it manually or from sendmail,
you can use multiple recipients.
More info
The (english) spec of NOKIAS CIMD implementation can
be found on:
http://www.europolitan.se/europolitan/fick/tjanst/cimdspec.htm
If you happen to know swedish, you can read more on Europolitans
implementation on:
http://www.europolitan.se/europolitan/fick/tjanst/faqcimd.htm
Feedback
Please let me know if you use this with a non NOKIA server,
or if you fix real checksums!
Jacob Lundqvist (SMS:+46-708-555 456)
Thanks Jacob !!!
4.7 Using kermit
By default, BB expects kermit 5 to be installed on your
BBPAGER host. On certain installation it is V6 and up that is
installed. If that's you case then you must modify the
bin/bb-page1.sh script:
change numeric.scr to numeric-k6.scr in the script.
But, try with numeric.scr at first and if it doesn't
then replace with numeric-k6.scr.
4.8 Adding a custom notification procedure
------------------------------------------------------------------------
Section 5: Miscellaneous Big Brother configuration options
5.1 Customizing the HTMLized status logs
If you don't like the blinking gifs, you can change them to
non-blinking by copying over the files found in www/gifs.
The non-blinking gifs start with the nb- prefix.
You can change Sean's mean face in the bb.html/bb2.html and
HTMLized status logs by replacing the bb.gif file.
You can change the header/footer for the HTMLized status pages.
The files are located in the www/html directory. Note that
your customized version can contain special tags which can be
replaced by values:
&BBDATE: current date/time
&BBBACKGROUND: with the current background color
(red/green/yellow/clear/purple)
&BBRELDATE: Release date of the current version
&BBREL: Current version number
&BBCOLOR: Color of this status file
&BBHOST: Name of host of current status file
&BBSVC: Service name of current status file.
You can add tags by modifying the bbd.c file. Every
time a status log is received by bbd, an HTML version of the
status log is created by prefixing the status with the header
file and suffixed with the footer file.
You can put tags also in the status logs:
&red &green &yellow &clear &purple
These tags are replaced with a corresponding image
source tag for the HTMLized version. Just like the bb-local.sh
script does for the disk and procs tests. So, if you create
your own custom check, enter these tags if you want to see a
colored dot in the HTMLized status.
NOTE: If you do not wish to have HTMLized status files, then
rename the www/html directory and the main BB display
pages will use the plain text status logs which are
located in the www/logs directory.
5.2 Specifying filesystem specific values in the disk test
You can define warning and panic levels for specific
filesystems. They are defined in the etc/bb-dftab file. The
format is as follows:
/filesystem:warning level:panic level
example:
/:85:95
/oradata:98:99
/ uses 85% for warning level and 95% for panic
/oradata uses 98% for warning and 99% for panic
All other filesystems use the default values define
int the etc/bbdef.sh file.
Use the etc/bb-dftab.INFO file as a starting point.
5.3 Enabling security
When BB is installed, it accepts connections from
any hosts. You can specify from which hosts only to accept
status logs using the etc/security file.
You can define individual hosts or networks in the
etc/security file. Using this format:
192.168.1.0
192.168.2.125
10.0.0.0
Accept from network 192.168.1.0, for 192.168.2.125 host
and the 10.0.0.0 network.
5.4 Inserting notes about your hosts
You can setup HTML links on the host name in the
bb.html/bb2.html pages to point to an information
page for that particular host.
If a file exists in the www/notes directory that
matches the system name as displayed on the bb pages,
then they are linked into both the bb.html page and the
bb2.html summary page.
These files can end in .htm, .html, or nothing :)
5.5 Customizing the footer for bb.html/bb2.html
bb.html/bb2.html are the HTML status pages generated
every 5 minutes. You can modify the footer of it
by modifying the www/notes/footer file.
NOTE: The link to maclawran.ca MUST be kept in the
footer page as per your agreement of the
license agreement.
------------------------------------------------------------------------
Section 6: Other documentation sources
$BBHOME/README
$BBHOME/LICENSE
$BBHOME/README.INSTALL
$BBHOME/README.CHANGES
$BBHOME/doc/README
$BBHOME/src/README