Zorka

In addition to standard JVM monitoring, support for JBoss7/Wildfly provides the following functionalities:

(zabbix) access to JBoss-specific JMX metrics - Thread pools, JCA, request processors, HTTP sessions, Hibernate etc. (JBoss 7.1+since 7.0 did not expose anything meaningful over JMX);
(zabbix) direct access to JBoss management interface (DMR) - this provides access to all configuration and runtime informationmaintained by server instance; in the case of JBoss 7.0 this is the only way to access such data, in later versionsit gives access to performance statistics that are maintained by JBoss but normally not accessible through JMX;
(zabbix) standard metrics provided by Zorka instrumentation - HTTP, SQL,EJB, LDAP etc. (in some cases adequate scripts need to be loaded and configured);
(ZICO) tracing for certain types of requests: HTTP, SQL,EJB, LDAP etc.;
(syslog, text files, zabbix) audit log for applications that use authentication and authorization subsytems;
a set of zabbix templates for (most of) above features: both common features and jboss7-specific ones;
for JDK7 add zorka.spy.compute.frames = yes to zorka.properties or -XX:-UseSplitVerifier to JVM startup options;
for JDK8 add zorka.spy.compute.frames = yes to zorka.properties;

JBoss integration has been tested on Sun JDK 6 and Sun JDK 7.

Agent installation

unpack zorka binary distribution in jboss directory;
add ldap.bsh if necessary to enable LDAP support and
for JDK7 add zorka.spy.compute.frames = yes to zorka.properties or -XX:-UseSplitVerifier to JVM startup options;
for JDK8 add zorka.spy.compute.frames = yes to zorka.properties;

JBoss 7 can work in one of two modes, so there are two procedures for agent configuration:

Standalone mode

edit <server-home>/bin/standalone.conf and add the following line:

JAVA_OPTS="$JAVA_OPTS -javaagent:<zorka-home>/zorka.jar=<zorka-home>"

add zorka bootstrap package to JBoss system modules - find and change the setting in standalone.conf file:

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.jitlogic.zorka.core.spy"
fi

Domain mode

In domain mode administrator needs to create separate zorka home directory for every server VM. Each agent instance has to listen on separate ports and use separate log/trace files.

edit <server-home>/bin/standalone.conf and add the following line:
add zorka bootstrap package to JBoss system modules - find and change the setting in domain.conf file:

if [ "x$JBOSS_MODULES_SYSTEM_PKGS" = "x" ]; then
   JBOSS_MODULES_SYSTEM_PKGS="org.jboss.byteman,com.jitlogic.zorka.core.spy"
fi

add java agent option to default JVM configuration in host.xml:

<jvms>
    <jvm name="default">
        <jvm-options>
            <option value="-server"/>
            <option value="-javaagent:<zorka-root>/zorka.jar=<zorka-root>/zorka"/>
        </jvm-options>
    </jvm>
</jvms>

add zorka.home.dir system property to each server JVM:

<servers>
    <server name="server-one" group="main-server-group" auto-start="true">
        <system-properties>
            <property name="zorka.home.dir" value="<zorka-root>/server1"/>
        </system-properties>
    </server>
    <server name="server-two" group="main-server-group" auto-start="true">
        <system-properties>
            <property name="zorka.home.dir" value="<zorka-root>/server2"/>
        </system-properties>
        <socket-bindings port-offset="150"/>
    </server>
    <server name="server-three" group="other-server-group" auto-start="false">
        <socket-bindings port-offset="250"/>
    </server>
</servers>

Configuration Settings

JBoss integration script implement standard HTTP/EJB monitoring, so standard options for HTTP and EJB apply. Also, consider SQL and/or LDAP monitoring if your applications use SQL database(s). There are no other specific settings for JBoss.

HTTP instrumentation for JBoss 7.x can be explicitly disabled by adding http.instrument.catalina = no in zorka.properties. For Wildfly 8.x or newer use http.instrument.undertow property.

Audit logging

Zorka agent supports auditing for JBoss 7. It is disabled by default and uses standard configuration settings for audit as described in Application Auditing section. The following events are supported as for now:

authentication and authorization of web requests;
authentication and authorization of EJB calls;

Auditing is done separately from JBoss built in auditing in order to ensure uniform behavior of zorka auditing across supported platforms and frameworks but it might be eventually incorporated in the future as well.

Tracing JBoss7

By adding tracer = yes to zorka.properties. EJB3 and HTTP requests are supported out of the box (more to come). Tracing of LDAP, SQL will be enabled as soon as proper configuration scripts will be loaded.

Sample traces collected from JBoss 7 (typical ones - HTTP, SQL, etc.).

Expanded method call tree of an EJB call.

Accessing JBoss management data (DMR)

DMR integration is implemented entirely in jboss7.bsh extension script. It provides access to attributes of arbitrary nodes in JBoss management data and simple LLD discovery functionality. Attribute values can be retreived using jboss7.dmr[] function, for example:

jboss7.dmr["/subsystem=datasources/data-source=ExampleDS/statistics=pool", "ActiveCount"]

First argument contains path to object - its syntax is the same as in jboss-cli utlility. Second is attribute name. Above query is equivalent of performing query below in jboss-cli and returning its result part if outcome="success" (or ZBX_NOTSUPPORTED otherwise):

/subsystem=datasources/data-source=ExampleDS/statistics=pool:read-attribute(name="ActiveCount")

If some attribute values contain slash characters (/), commas or other characters can be used instead as path separators. Agent will always use first character of a path as separator. Example below shows how to query for datasource parameters in JBoss 7.0.2 (where data sources contain slash characters):

jboss7.dmr[",subsystem=datasources,data-source=java:jboss/datasources/ExampleDS", "ActiveCount"]

There are two additional functions for performing zabbix discovery. First one lists selected node and looks for child nodes of specific type, second one does the same but looks for child nodes of any type. For example this query specifies path and child type:

jboss7.dmr_discovery["/subsystem=datasources", "data-source"]

will return something like this:

{"data":[{"{#NAME}":"ExampleDS"}]}

Without specifying child type it will look for any child nodes and return both names and types:

jboss7.dmr_discovery["/subsystem=threads"]

Might return something like this (granted that some global thread pools are actually configured):

{"data":[{"{#NAME}":"ajp-executor","{#TYPE}":"blocking-bounded-queue-thread-pool"},{"{#NAME}":"http-executor","{#TYPE}":"blocking-queueless-thread-pool"}]}

For more practical examples, see templates distributed along with zorka agent that use DMR, (eg. thread pools template or datasources template).

How DMR queries work

Internally DMR monitoring is implemented in jboss7.bsh script as follows:

agent intercepts reference to org.jboss.as.controller.ModelController instance - this is singleton create by JBossat boot time;
jboss7.dmr*() functions implemented in jboss7.bsh script translate their arguments into ModelNode structureand call ModelController.execute() to execute queries;

Zabbix Templates for JBoss 7

This script defines jboss7 namespace containing set of helper functions that are used in conjunction with the following zabbix templates:

Template_Zorka_JBoss70_DataSources.xml - performance statistics regarding database connectors: active connections,available connections, connections created, connections destroyed, connections timed out; this is DMR template for JBoss7.0.x;
Template_Zorka_JBoss71_DataSources.xml - performance statistics regarding database connectors: active connections,available connections, connections created, connections destroyed, connections timed out; this is DMR template for JBoss7.1+/8;
Template_Zorka_JBoss7_ThreadPools.xml - collects performance information about globally configured thread pools:number of active threads, current number of threads, maximum number of threads, tasks submitted, tasks completed,tasks rejected; (DMR, JBoss 7.0+);
Template_Zorka_JBoss7_Ejb3Stateless.xml - Stateless EJB beans (JMX, JBoss 7.1+);
Template_Zorka_JBoss7_JCAWorkManager.xml - Thread pools for JCA work manager (JMX, JBoss 7.1+);
Template_Zorka_JBoss7_JpaPersistence.xml - Hibernate persistence metrics (JMX, JBoss 7.1+);
Template_Zorka_JBoss7_RequestProcessors.xml - request processing metrics (JMX, JBoss 7.1+);
Template_Zorka_JBoss7_Servlets.xml - servlet execution metrics (JMX, JBoss 7.1+);
Template_Zorka_JBoss7_Sessions.xml - http session statistics (JMX, JBoss 7.1+);

In addition some generic templates can be used:

Template_Zorka_Audit.xml - if auditing is enabled;
Template_Zorka_EJB.xml - EJB call execution statistics;
Template_Zorka_HTTP.xml - HTTP request handling statistics;
Template_Zorka_LDAP.xml - statistics for LDAP queries (if ldap.bsh is loaded);
Template_Zorka_SQL.xml - statistics for SQL queries (if some JDBC monitoring script is loaded);
Template_Zorka_JMS.xml - statistics for JMS communications;

JBoss7+, Wildfly