Applications with Auto-Monitoring
It is very common for networking devices such as routers and (smart) switches to support monitoring through built-in SNMP support. This habit has not been established for higher-level applications, such as server daemons, although it would be very useful there too.
Why monitoring is useful
In any professional environment, monitoring is a vital part of business. It is a Good Thing |trade| to resolve problems before customers detect it. Not only does it reflect well on Service Level Agreements, but it also helps to keep pressure off the helpdesk.
The standard for monitoring, as established by the IETF, is SNMP. We discussed it in a Good, Bad and Ugly article. We believe that the value of SNMP incorporated into devices could also do wonders for the daemons that we tend to run as servers.
Imagine being able to auto-discover new DNS zones, have their DNSSEC state incorporated and see a report when RRSIGs on the zone start expiring. Or if you like web protocols, imagine hearing about the retraction of a virtual host or backend server before anybody notices it. Or imagine a replicated architecture, where you get to see servers that are down even if your customers cannot see it. These things are an extremely useful part of running a solid online organisation.
The prefix auto means self or by itself. We have forgotten it now, but an automobile was something that moved by itself... without a horse to pull it, that is.
Similarly, auto-monitoring is a term that we coined for systems that present the information with which they can be monitored -- some part of their internal state that signals trouble, either in summary or on a listed object (such as a zone name) or perhaps statistical information (such as traffic flowing through a virtual host).
Not all environments use SNMP -- yet
;-) -- but an application may respond
to the ability to load the Net-SNMP library and registering it as an agent
by actually exporting information over SNMP. This means that adding SNMP
is instantaneous when the Net-SNMP library is installed on a system.
Two Programming Projects
We want you to engage in two concrete projects, and compose in a HOWTO format what needs to be done in general to support SNMP in a piece of software.
The first project is a "push" mode assignment. It provides a variable that may be watched, and optionally sends a trap to a monitoring station if it knows one. The daemon to work on is the SMARTmontools daemon, which runs in the background to check on disks.
The second project is a "pull" mode assignment, and it is somewhat complex because it reports state in one table, composed from two programs; this sort of thing is possible under the AgentX specification. Your target for this is OpenDNSSEC; we want to see a table indexed by zone names, and reporting on the keys (how many there are, when the next rollover takes place and such) from the Enforcer, as well as the next re-signing moment from the Signer (and, if it is easy to add, the first RRSIG to expire might be a nice addition). These daemons each register their objects into the table, and leave it to the SNMPd master agent to combine the results. Use the functionality of the Net-SNMP to register callbacks that inquire for the information, which you can dig up from the common storage structures -- probably the database for the Enforcer, and the queue for the Signer.
As part of your work, construct templates for Zabbix (just one of the many monitoring products that can track SNMP) and use its discovery facilities to demonstrate how newly added zones automatically pop up in the output; also demonstrate how the retraction of a zone leads to an alarm that must be handled, or otherwise silenced by an operator.