Zorka is a flexible java monitoring agent with programmable bytecode instrumentation engine. It is designed to complement existing monitoring systems rather than duplicate their functionality. Zorka agent provides unparalleled flexibility due to its scripting capabilities which makes it unique among comparable products. In addition to core functionalities, set of scripts and zabbix templates distributed with agent provides monitoring capabilities for popular application servers (a.k.a. ‘batteries included’).
End-user features (for application administrators)
Zorka agent has been tested with JDK5, JDK6 and JDK7. Binary package contains agent compiled for JDK6 but it can be easily backported to Java 5 with retrotranslator.
Zorka agent can act in several roles - depending on needs application administrator can enable desired functionalities and configure conventional monitoring systems accordingly.
JMX access - exposing JMX attributes to conventional monitoring systems;
Zorka Stats - agent can aggregate performance statistics on arbitrary application components using its instrumentation engine and expose performance counters similiar to J2EE statistics via JMX;
slow query logs and error logs - eg. SQL, EJB, HTTP et. to local files, via syslog or to zabbix;
transaction tracing with code-level visibility - agent records application execution (eg. HTTP request) and submits method call tree along with potentially interesting information (eg. HTTP params/headers, SQL queries etc.);
audit logs - agent can generate audit logs for arbitrary applications that lack built in auditing functionality; audit logs can be stored in local files, sent to remote syslog server or zabbix server; auditing functionality can be easily extended or adjusted using beanshell scripts;
Monitored application components
Standard java application components are supported in all application servers implementing such features:
HTTP - performance statistics, HTTP request tracing, slow request processing logs, error logs;
SQL - performance statistics, SQL query tracing, slow query logs and error logs (variety of JDBC drivers are now supported, additional can be easily implemented with beanshell scripts);
EJB - performance statistics, EJB call tracing, slow query logs, error logs;
LDAP - performance statistics and tracing for LDAP client requests;
JMS - performance statistics and tracing for JMS listeners, counters and trace attributes for JMS producers and consumers;
Supported monitoring systems and protocols
Zorka agent integrates seamlessly with:
Zabbix Monitoring System - works in both agent mode and trapper mode, implements low-level discovery;
Nagios - simple NRPE integration;
syslog - can send information directly to remote to remote syslog servers;
SNMP - can send SNMP traps directly to remote management servers;
local files - agent can log arbitrary application events it detects in local files (eg. slow SQL queries, EJB errors etc.);
ZICO collector - to collect profiling information (application traces);
Supported application servers
A bunch of popular application servers are supported out of the box. For other applications agent might lack some capabilities of monitoring server-specific information but at least JVM related monitoring should work and possibly some generic things (eg. JMS, embedded Jetty/Catalina) and some common application frameworks can help. The following servers and applications have been tested and instrumented:
Tomcat 6/7 - standard JMX counters provided by Tomcat, HTTP tracing and performance counters;
JBoss 4/5/6 - standard JMX counters provided by JBoss, HTTP, EJB, JMS tracing and performance counters;
JBoss 7, EAP 6 - standard JMX counters provided by JBoss 7, HTTP, EJB, JMS tracing and performance counters, audit;
Jetty 6/7/8/9 - standard JMX counters, HTTP tracing and performance counters;
IBM Websphere 7/8 - standard JMX counters, PMI, HTTP, EJB, JMS tracing and performance counters;
Mule ESB - standard JMX counters,
Oracle Weblogic - standard JMX counters, HTTP, EJB tracing and performance counters;
There are scripts for some applications and application frameworks that can be useful:
Jasig CAS - tracing, auditing and performance counters for standard CAS operations (ticket handling and validation);
Spring Webflow - additional trace information useful for debugging webflow applications;
Note that some common components (eg. SQL, LDAP) are independent of application server and are supported in all above application servers. Administrator must enable appropriate support scripts for those components separately from main application server support script.
Core agent features (advanced, agent scripting)
Features described in this section are useful for application developers, consultants and advanced administrators looking for ways of enhancing or adjusting existing agent functionalities.
Beanshell is well known programming language that has been used to make Zorka programmable. Administrator can implement additional beanshell code which will either configure other agent components (eg. methods instrumentation or implement additional queries that can be accessed by monitoring systems;
JMX objects can be accessed by agent with greater reach and lower overhead than traditional JMX clients. This is possible because agent works in the same JVM as inspected objects, so no serialization is necessary and limits imposed by serialization and RMI communication disappear. In addition, object inspection component built into agent allows for flexible listing traversal through objects of various types. It can use type-specific features (eg. indexes for arrays or keys for hashmaps), recognizes some specific object types (JMX mbeans, composite and tabular types, J2EE stats, zorka stats etc.), uses getters, arbitrary methods or reflection to access attributes/fields of objects unknown to agent. And even if getting through an object is impossible using inspector, administrator can always write a beanshell function that will extract needed value.
Zorka also implements functionality for listing and searching object graph of objects visible from JMX. It has flexible, fluent API useful for querying for objects of specific type.
Zorka Spy is an advanced bytecode instrumentation engine using Agent Supplied Aspects technique. It is configured in a manner similiar to Aspect Oriented Programming. Application administrator can define arbitrary actions at start and end of method execution, fetch and process arbitrary data (method arguments, current time, current thread etc.) and process data in arbitrary manner. There is a set of predefined components for presenting and processing obtained data:
method execution statistics - data from method calls can be aggregated and aggregates can be presented as JMX beans; number of calls, errors, average/peak execution time and congestion (number of threads executing inside instrumented method at any tim) are currently available;
asynchronous signals (traps) - agent can send traps to other monitoring systems using standard protocols; zabbix trapper, syslog messages and SNMP traps are currently supported;
intercepting and presenting application objects - agent can intercept instances of specific objects in application and present some of their attributes via JMX (if application does not do it by itself);
processing intercepted data and filtering events - events generated by intercepted method calls can be transformed and filtered;
performing arbitrary actions with bsh functions;
Zorka Tracer is low-overhead application profiling mechanism suitable to be used in production environments. Tracer generates trace files that can be viewed using zorka viewer. Tracer integrates well with Spy module, so it is very configurable when it comes to deciding where traces should start filtering method call information that end up being saved and attaching additional data fetched and transformed using Spy components. The following functionalities can be used to control tracer:
changing minimum method execution time and maximum number of methods recorded in a single trace: methods that execute faster than defined time will be omitted (albeit method that throw an exception might still be included);
deciding which traces should be saved and which ones should be dropped: administrator can set minimum trace execution time or programmaticaly decide if trace should be saved (eg. HTTP requests with HTTP/500 status code should be recorded regardless of how short will they execute;
- attaching custom attributes to traced methods: any data fetched and processed by Spy can be attached to trace data and later viewed;
Zorka Trappers functionality allows for asynchronous notifications of external systems. Typically trappers are used in conjunction with Spy, so notifications can be sent when certain methods will be invoked (and certain conditions, defined by administrator will be met). The following trappers are currently implemented:
zabbix - uses native zabbix protocol to send data to specific items defined in zabbix templates;
syslog - sends syslog packets to remote server;
SNMP - sends SNMP traps to remote SNMP collector;
local file - writes text messages to a local file;
zorka log - logs messages in standard zorka log;
Data fetched and transformed by Spy functions can be used to construct traps.
Zorka Intranet Collector (ZICO) collects application performance data from agents all across the company. It collects traces in real time and presents them in user friendly HTML5 application. No special client software and no browser plugins are required. Modern browser is all you need.