condor_triggerd - QMF event manager
-----------------------------------
The Qpid Management Framework (QMF) is a distributed object management framework
that uses Apache Qpid for its transport. It provides code-generation tools to
render native language binding types and objects for C++, Python, and Ruby from
XML schema definitions. That code is then linked to runtime libraries for
client-agent integration mediated by a Qpid broker.

This contribution is a daemon that raises QMF Events based upon user defined
ClassAd queries.  The Triggerd asks the Collector if any ClassAds match the
specific query, and then processes the results and raises events.  A trigger
definition contains 3 elements:
- A Name
- A ClassAd Query
- The text to be raised if the ClassAd Query returns any matches

The QMF Events raised by the condor_triggerd contain two
arguments:
- The time the event was raised
- The event text

The event text can contain attributes from the matching ClassAd by surrounding
the attribute with $().  If there was a trigger with text:

$(Machine) has been Claimed/Idle

Then the $(Machine) would be replaced with the value for the attribute Machine
from the matching ClassAd.

Currently, this code base has external dependencies on:
        * Apache Qpid 0.8-4

For more information about QMF, please refer to:
https://cwiki.apache.org/qpid/qpid-management-framework.html

Building
--------
This triggerd contrib is enabled by setting -DWANT_CONTRIB:BOOL=ON for cmake
at the root of the src tree. This will generate the necessary platform-specific
build files (e.g., make, Visual Studio solution, etc.) for the triggerd
contrib.

Broker Setup
------------
The triggerd daemon must be connected to a Qpid broker. In Condor this broker
is configured by:
 QMF_BROKER_HOST=<hostname/ip>
 QMF_BROKER_PORT=<broker port - internal default is 5672>

Currently, the triggerd does not support communicating with a secured broker.
Run the broker something like this:
 $ sudo qpidd --auth=no

This limitation will change in the near future.

Configuring the Triggerd
------------------------
The Triggerd uses the following configuration parameters:
DATA - The directory to store the configured triggers
TRIGGERD_DEFAULT_EVAL_PERIOD - How often the Triggerd will evaluate the
                               configured triggers.  The Triggerd will start
                               evaluating triggers at this interval unless the
                               interval is changed.
TRIGGERD_QUERY_COLLECTOR - Usually the Triggerd will query all Collectors in a
                           pool.  This parameter defines a specific collector
                           to query, such as a View Collector.

The DATA parameter is required.  Additionally, the following needs to be added
to condor's configuration:
MAX_TRIGGERD_LOG = 1000000
TRIGGERD_DEBUG =
TRIGGERD_LOG = $(LOG)/TriggerLog
TRIGGERD = $(SBIN)/condor_triggerd
DC_DAEMON_LIST =+ TRIGGERD
DAEMON_LIST = $(DAEMON_LIST), TRIGGERD

Configuring the Startds (Optional)
----------------------------------
In addition to the Triggerd itself, this contribution also contains a script
named get_trigger_data which will extend the diagnostic data of linux based
execute nodes (nodes running startds).  This script will query the machine it
is running on and, using condor's cron functionality, insert data into the
machine ad that provides detailed diagnostic information.

To configure the module, add the following to condor's configuration for each
execuate node:
STARTD_CRON_NAME = TRIGGER_DATA
STARTD_CRON_AUTOPUBLISH = If_Changed
TRIGGER_DATA_JOBLIST = GetData
TRIGGER_DATA_GETDATA_PREFIX = Triggerd
TRIGGER_DATA_GETDATA_EXECUTABLE = $(BIN)/get_trigger_data
TRIGGER_DATA_GETDATA_PERIOD = 5m
TRIGGER_DATA_GETDATA_RECONFIG = FALSE

Usage
-----
This contribution contains a tool to manage triggers in the Triggerd called
condor_trigger_config.  This tool can be used to add, delete, and list triggers
configured in the triggerd.  It also contains a set of default diagnostic
triggers that are reliant upon the get_trigger_data script.  These default
triggers are:

Trigger Name:                ClassAd Query:
High CPU Usage               (TriggerdLoadAvg1Min > 5)
Low Free Mem                 (TriggerdMemFree <= 10240)
Low Free Disk Space (/)      (TriggerdFilesystem_slash_Free < 10240)
Busy and Swapping            (State == \"Claimed\" && Activity == \"Busy\" &&
TriggerdSwapInKBSec > 1000 && TriggerdActivityTime > 300)
Busy but Idle                (State == \"Claimed\" && Activity == \"Busy\" &&
CondorLoadAvg < 0.3 && TriggerdActivityTime > 300)
Idle for long time           (State == \"Claimed\" && Activity == \"Idle\" &&
TriggerdActivityTime > 300)
dprintf Logs                 (TriggerdCondorLogDPrintfs != \"\")
Core Files                   (TriggerdCondorCoreFiles != \"\")
Logs with ERROR entries      (TriggerdCondorLogCapitalError != \"\")
Logs with error entries      (TriggerdCondorLogLowerError != \"\")
Logs with DENIED entries     (TriggerdCondorLogCapitalDenied != \"\")
Logs with denied entries     (TriggerdCondorLogLowerDenied != \"\")
Logs with WARNING entries    (TriggerdCondorLogCapitalWarning != \"\")
Logs with warning entries    (TriggerdCondorLogLowerWarning != \"\")
Logs with stack dumps        (TriggerdCondorLogStackDump != \"\")
