Rothirsch Technologies Hintergrund

Monitoring

Information

Use it on
Hardware Banana Pi m64
OperatingSystem Armbian 5.59 Debian Stretch
Kernel 4.14.70-sunxi

Goal

Install and use icinga2 on an ARM based device

Icinga2 is an open source monitoring tool that allows you to overwatch different services on servers in your local and/or remote network.

You can even get notifications onto your smartphone.

Also testing on

Information
Docdate 2018.10.10
Hardware Banana Pi m2+ h3
OperatingSystem Armbian 5.59 Debian Stretch
Kernel 4.14.70-sunxi

Install dependencies

Install maridadb server for icinga2

apt install mariadb-server mariadb-client
mysql_secure_installation

Root password is empty

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
TE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MariaDB
      SERVERS IN PRODUCTION USE! PLEASE READ EACH STEP CAREFULLY!

In order to log into MariaDB to secure it, we'll need the current
password for the root user. If you've just installed MariaDB, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: YES)
Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

Enter current password for root (enter for none):
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MariaDB
root user without the proper authorisation.

Set root password? [Y/n] Y
New password:
Re-enter new password:
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MariaDB installation has an anonymous user, allowing anyone
to log into MariaDB without having to have a user account created for
them. This is intended only for testing, and to make the installation
go a bit smoother. You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] Y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'. This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] Y
 ... Success!

By default, MariaDB comes with a database named 'test' that anyone can
access. This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] Y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] Y
 ... Success!

Cleaning up...

All done! If you've completed all of the above steps, your MariaDB
installation should now be secure.

Thanks for using MariaDB!

OUTPUT

Create databases

mysql -u root -p
#
create database icingadb;
grant all privileges on icingadb.* to 'icinga_user'@'localhost' identified by 'icinga_pass';
flush privileges;
exit

mysql -u root -p
#
create database icinga_users;
grant all privileges on icinga_users.* to 'icinga_user'@'localhost' identified by 'icinga_pass';
flush privileges;
exit

Install Webserver

icingaweb2 is the web administration interface of icinga2. So we need a web-server on this monitoring instance. We use apache2.

apt install apache2

Install PHP 7.0

apt install libapache2-mod-php7.0 php7.0-xml php7.0-opcache php7.0-xml php7.0-mbstring php7.0-json php7.0-curl php7.0-ldap php7.0-cli php7.0-gd php7.0-intl php7.0-readline

You have to define the right timezone for your php instance. Change in /etc/php/7.0/apache2/php.ini

From

;date.timezone =

to

date.timezone = Europe/Vienna #your timezone

Install icingaweb2 dependencies

icingaweb2 needs this package

apt install php-htmlpurifier

Install icinga2

How To Install Icinga 2 Monitoring Tool on Debian / Armbian

Prepare the system

pbuilder setup

apt update && apt -y upgrade
apt install pbuilder debootstrap devscripts

Add icinga repositories and install the packages

sudo -s
wget -O - https://packages.icinga.com/icinga.key | sudo apt-key add -

Get icinga source code from repository

echo -e '# Icinga \
deb-src http://packages.icinga.com/debian icinga-stretch main' > /etc/apt/sources.list.d/icinga.list
apt update

It would do no harm if you reboot your device after these steps

Expand swap

I had some issues with a "to small" swap space. So I expanded it to 1.5GB.

internal compiler error killed (program cc1plus)

OUTPUT

stable armbian images

dd if=/dev/zero of=/var/swap.img bs=1024k count=1500
chmod 600 /var/swap.img
mkswap /var/swap.img
swapon /var/swap.img

beta armbian images

Armbian uses zram and you can configure it within the /etc/default/armbian-zram-config

# configuration values for the armbian-zram-config service
#
# enable the armbian-zram-config service?
ENABLED=true

# percentage of zram usage compared to physically available DRAM.
# Huge overcommitment (300) is possible and sometimes desirable. See
# https://forum.armbian.com/topic/5565-zram-vs-swap/?do=findComment&comment=61082
ZRAM_PERCENTAGE=200

# create how many zram devices max for swap
# ZRAM_MAX_DEVICES=4

# Which algorithm for zram based swapping. Seems lzo is best choice on ARM:
# https://forum.armbian.com/topic/8161-swap-on-sbc/?do=findComment&comment=61668
# SWAP_ALGORITHM=lzo

# Which algorithm to choose for zram based ramlog partition
# RAMLOG_ALGORITHM=zstd

# Which algorithm to choose for zram based /tmp
# TMP_ALGORITHM=zstd

Configuration

Build icinga2

mkdir -p /opt/icinga2
cd /opt/icinga2
apt source icinga2

Reading package lists... Done
NOTICE: 'icinga2' packaging is maintained in the 'Git' version control system at:
https://anonscm.debian.org/git/pkg-nagios/pkg-icinga2.git
Please use:
git clone https://anonscm.debian.org/git/pkg-nagios/pkg-icinga2.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 2,226 kB of source archives.
Get:1 http://packages.icinga.com/debian icinga-stretch/main icinga2 2.9.1-1.stretch (diff) [31.0 kB]
Get:2 http://packages.icinga.com/debian icinga-stretch/main icinga2 2.9.1-1.stretch (dsc) [1,873 B]
Get:3 http://packages.icinga.com/debian icinga-stretch/main icinga2 2.9.1-1.stretch (tar) [2,193 kB]
Fetched 2,226 kB in 0s (2,550 kB/s)
dpkg-source: info: extracting icinga2 in icinga2-2.9.1
dpkg-source: info: unpacking icinga2_2.9.1.orig.tar.gz
dpkg-source: info: unpacking icinga2_2.9.1-1.stretch.debian.tar.xz
dpkg-source: info: applying 21_config_changes
W: Download is performed unsandboxed as root as file 'icinga2_2.9.1-1.stretch.debian.tar.xz' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

OUTPUT

pbuilder create --distribution stretch --debootstrapopts --variant=buildd
# pbuilder create --distribution stretch --mirror http://ftp.tu-graz.ac.at/mirror/debian/ --debootstrapopts --variant=buildd

W: /root/.pbuilderrc does not exist
I: using fakeroot in build.
I: pbuilder: network access will be disabled during build
I: Current time: Wed Aug 1 09:01:29 CEST 2018
I: pbuilder-time-stamp: 1533106889
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/base.tgz]
I: copying local configuration
I: mounting /proc filesystem
I: mounting /sys filesystem
I: creating /{dev,run}/shm
I: mounting /dev/pts filesystem
...
...
...
I: unmounting dev/ptmx filesystem
I: unmounting dev/pts filesystem
I: unmounting dev/shm filesystem
I: unmounting proc filesystem
I: unmounting sys filesystem
I: creating base tarball [/var/cache/pbuilder/base.tgz]
I: cleaning the build env
I: removing directory /var/cache/pbuilder/build/1658 and its subdirectories

OUTPUT

Now we can build the packages for icinga2

pbuilder build --debbuildopts "-j4" icinga2_*.*.*-*.stretch.dsc

and install the packages

cd /var/cache/pbuilder/result/
dpkg -i libicinga2_*.*.*-*.stretch_arm64.deb
dpkg -i icinga2-bin_*.*.*-*.stretch_arm64.deb icinga2-common_*.*.*-*.stretch_all.deb
dpkg -i icinga2_*.*.*-*.stretch_arm64.deb
dpkg -i icinga2-ido-mysql_*.*.*-*.stretch_arm64.deb

Installation steps

Don't let the install configure the database but enable ido-mysql feature.

mysql root password

Click! for installation steps

Note

The installation is done. For later configuration we need to note the root password.

Install icingaweb2

How To Install Icinga 2 Monitoring Tool on Debian / Armbian

Build icingaweb2

mkdir -p /opt/icingaweb2
cd /opt/icingaweb2
apt source icingaweb2

Reading package lists... Done
NOTICE: 'icingaweb2' packaging is maintained in the 'Git' version control system at:
https://anonscm.debian.org/git/pkg-nagios/pkg-icingaweb2.git
Please use:
git clone https://anonscm.debian.org/git/pkg-nagios/pkg-icingaweb2.git
to retrieve the latest (possibly unreleased) updates to the package.
Need to get 8,179 kB of source archives.
Get:1 http://packages.icinga.com/debian icinga-stretch/main icingaweb2 2.6.1-1.stretch (diff) [11.8 kB]
Get:2 http://packages.icinga.com/debian icinga-stretch/main icingaweb2 2.6.1-1.stretch (dsc) [1,535 B]
Get:3 http://packages.icinga.com/debian icinga-stretch/main icingaweb2 2.6.1-1.stretch (tar) [8,165 kB]
Fetched 8,179 kB in 3s (2,541 kB/s)
dpkg-source: info: extracting icingaweb2 in icingaweb2-2.6.1
dpkg-source: info: unpacking icingaweb2_2.6.1.orig.tar.gz
dpkg-source: info: unpacking icingaweb2_2.6.1-1.stretch.debian.tar.xz
W: Download is performed unsandboxed as root as file 'icingaweb2_2.6.1-1.stretch.debian.tar.xz' couldn't be accessed by user '_apt'. - pkgAcquire::Run (13: Permission denied)

OUTPUT

pbuilder build --debbuildopts "-j4" icingaweb2_*.*.*-*.stretch.dsc

W: /root/.pbuilderrc does not exist
I: using fakeroot in build.
I: pbuilder: network access will be disabled during build
I: Current time: Thu Oct 11 12:08:51 UTC 2018
I: pbuilder-time-stamp: 1539259731
I: Building the build Environment
I: extracting base tarball [/var/cache/pbuilder/base.tgz]
I: copying local configuration

OUTPUT

and install the packages

cd /var/cache/pbuilder/result/
dpkg -i php-icinga_*.*.*-*.stretch_all.deb
dpkg -i icingaweb2-common_*.*.*-*.stretch_all.deb
dpkg -i icingaweb2_*.*.*-*.stretch_all.deb icingacli_*.*.*-*.stretch_all.deb

Change permissions and reload the webserver

usermod -a -G icingaweb2 www-data
chown -R www-data:icingaweb2 /etc/icingaweb2/
service apache2 reload

Icingaweb2 basic configuration

Create the icnageweb2 configuration

icingacli setup config directory --group icingaweb2

Create the setup tokens and note them for later configuration

mysql -u root icingadb -p < /usr/share/icinga2-ido-mysql/schema/mysql.sql
icingacli setup token create
The newly generated setup token is: 8fabc89ff4b537dd

You are now able to configure the service over the web interface (icingaweb2). Change following string so it uses your server IP address and use it in you browser.

http://172.17.0.1/icingaweb2/setup

Interface configuration step by step

Installation beendet

Master

Um mehrer Server von einem Master überwachen zu lassen, werden diese unterschiedlich konfiguriert

Checkliste:

  • Port 5665, muss von und zum Master von der Firewall freigegeben sein.
  • Bewerkstelligt wird die Überwachung mit dem icinga2 client.

Master Konfiguration

icinga2 node wizard

Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]: n

Starting the Master setup routine...

Please specify the common name (CN) [your.master.local]:
Reconfiguring Icinga...
Checking for existing certificates for common name 'your.master.local'...
Certificates not yet generated. Running 'api setup' now.
Generating master configuration for Icinga 2.
Enabling feature api. Make sure to restart Icinga 2 for these changes to take effect.

Master zone name master:

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]:
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Do you want to disable the inclusion of the conf.d directory [Y/n]:
Disabling the inclusion of the conf.d directory...
Checking if the api-users.conf file exists...

Done.

Now restart your Icinga 2 daemon to finish the installation!

OUTPUT

service icinga2 restart

Warning: icinga2.service changed on disk. Run 'systemctl daemon-reload' to reload units

OUTPUT

systemctl daemon-reload
service icinga2 status

● icinga2.service - Icinga host/service/network monitoring system
   Loaded: loaded (/lib/systemd/system/icinga2.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/icinga2.service.d
           └─limits.conf
   Active: active (running) since Fri 2018-10-12 09:00:05 UTC; 1min 3s ago
 Main PID: 1716 (icinga2)
   CGroup: /system.slice/icinga2.service
           ├─1716 /usr/lib/aarch64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e
           └─1755 /usr/lib/aarch64-linux-gnu/icinga2/sbin/icinga2 --no-stack-rlimit daemon -e

Oct 12 09:00:36 your icinga2[1716]: Context:
Oct 12 09:00:36 your icinga2[1716]: (0) Reconnecting to MySQL IDO database 'ido-mysql'
Oct 12 09:00:36 your icinga2[1716]: [2018-10-12 09:00:16 +0000] critical/IdoMysqlConnection: Exception during d
Oct 12 09:00:36 your icinga2[1716]: [2018-10-12 09:00:26 +0000] critical/IdoMysqlConnection: Connection to data
Oct 12 09:00:36 your icinga2[1716]: Context:
Oct 12 09:00:36 your icinga2[1716]: (0) Reconnecting to MySQL IDO database 'ido-mysql'
Oct 12 09:00:36 your icinga2[1716]: [2018-10-12 09:00:26 +0000] critical/IdoMysqlConnection: Exception during d
Oct 12 09:00:36 your icinga2[1716]: [2018-10-12 09:00:36 +0000] critical/IdoMysqlConnection: Connection to data
Oct 12 09:00:36 your icinga2[1716]: Context:
Oct 12 09:00:36 your icinga2[1716]: (0) Reconnecting to MySQL IDO database 'ido-mysql'

OUTPUT

Satellite am Master bekannt machen

Ändere /etc/icinga2/zones.conf

/*
* Generated by Icinga 2 node setup commands
* on 2016-06-27 22:04:57 +0200
*/

object Endpoint NodeName {
}

object Zone "master" {
  endpoints = [ NodeName ]
}

object Endpoint "your.satellite.local" {
  host = "your.satellite.local"
}
  object Zone "your.satellite.local" {
  endpoints = [ "your.satellite.local" ]
  parent = "master"
}

Configuration

Füge Satellite zu /etc/icinga2/conf.d/hosts.conf hinzu

object Host "your.satellite.local" {
  import "generic-host"

  address = "your.satellite.local"
  //address6 = "::1"

  vars.os = "Linux"

  vars.remote_client = "your.satellite.local"

  vars.ssh_port = 22
  //vars.users_wgreater = 10
  //vars.users_cgreater = 20

  /* Define disks and attributes for service apply rules in `services.conf`. */
  vars.disks["disk"] = {
      /* No parameters. */
  }
  vars.disks["disk /"] = {
      disk_partitions = "/"
  }

  /* Define notification mail attributes for notification apply rules in `notifications.conf`. */
  vars.notification["mail"] = {
      /* The UserGroup `icingaadmins` is defined in `users.conf`. */
      groups = [ "icingaadmins" ]
  }
}

Configuration

At the end of this tutorial we create a ticket for the satellite

icinga2 pki ticket --cn 'your.satellite.local'
a18672b3ad1fa2ay2a6d84a1234d5ddc2420789

Satellite

icinga Satellite Installation/configuration

Because we will not only monitor ARM64 devices we will show the installation on x86 or x64 architectures here.

Other ARM64 devices

For other ARM64 devices u can use the .deb packages you have build on the master server

Other ARM devices

For other ARM devices u have to build the .deb packages like described on the master server

x86/x64 installation on Debian Stretch

wget -O - https://packages.icinga.com/icinga.key | sudo apt-key add -

Füge zu /etc/apt/sources.list hinzu

echo -e '# Icinga \
deb http://packages.icinga.com/debian icinga-stretch main' > /etc/apt/sources.list.d/icinga.list

Install Icinga2

apt-get update && apt-get install icinga2

That's it ; )

Satellite configuration

icinga2 node wizard

Welcome to the Icinga 2 Setup Wizard!

We will guide you through all required configuration details.

Please specify if this is a satellite/client setup ('n' installs a master setup) [Y/n]:

Starting the Client/Satellite setup routine...

Please specify the common name (CN) [your.satellite.local]:

Please specify the parent endpoint(s) (master or satellite) where this node should connect to:
Master/Satellite Common Name (CN from your master/satellite node): your.master.local

Do you want to establish a connection to the parent node from this node? [Y/n]:
Please specify the master/satellite connection information:
Master/Satellite endpoint host (IP address or FQDN): 172.30.7.203
Master/Satellite endpoint port [5665]:

Add more master/satellite endpoints? [y/N]:
Parent certificate information:

 Subject: CN = your.master.local
 Issuer: CN = Icinga CA
 Valid From: Oct 12 10:37:57 2018 GMT
 Valid Until: Oct 8 10:37:57 2033 GMT
 Fingerprint: AA 0E 94 0B 84 00 0C EE 6A 07 72 6B CD FF DB EF 32 BB CC 67

Is this information correct? [y/N]: y

Please specify the request ticket generated on your Icinga 2 master (optional).
 (Hint: # icinga2 pki ticket --cn 'your.satellite.local'): a18672b3ad1fa2ay2a6d84a1234d5ddc2420789
Please specify the API bind host/port (optional):
Bind Host []:
Bind Port []:

Accept config from parent node? [y/N]: y
Accept commands from parent node? [y/N]: y

Reconfiguring Icinga...

Local zone name [your.satellite.local]:
Parent zone name [master]:

Default global zones: global-templates director-global
Do you want to specify additional global zones? [y/N]:

Do you want to disable the inclusion of the conf.d directory [Y/n]:
Disabling the inclusion of the conf.d directory...

Done.

Now restart your Icinga 2 daemon to finish the installation!

OUTPUT

Now restart your Icinga 2 daemon to finish the installation!

ARM monitoring