Zorka

Instrumentation part of Zorka agent is called Zorka Spy.ZOrka features a fluent-style configuration API that uses similiar concepts to AOP (Aspect Oriented Programmin). It basically allows wrapping instrumented methods with

You can define spy configuration properties in fluent style using SpyDefinition object and then submit it to instrumentation engine. In order to be able to configure instrumentations, you need to understand structure of instrumentation engine and how events coming from instrumented methods are processed.

Zorka Spy will insert probes into certain points of instrumented methods. There are three kinds of points: entry points, return points and error points (when exception has been thrown). A probe can fetch some data, eg. current time, method argument, some class (method context) etc. All etched values are packed into a submission record (SpyRecord class) and processed by one of argument processing chains (ON_ENTER, ON_RETURN or ON_ERROR depending of probe type), then results of all probes from a method call are bound together and processed by ON_SUBMIT chain. All these operations are performed in method calling thread context (so these processing stages must be thread safe). After that, record is passed to ON_COLLECT chain which is guaranteed to be single threaded. Finally records are dispatched into collectors (which is also done in a single thread). Collectors can do various things with records: update statistics in some mbean, call some BSH function, log it to file etc. New collector implementations can be added on the fly - either in Java or as BeanShell scripts.

Spy definition example:

__mbsRegister() {
  process(record) {
    synchronized (super) {
        mbs = record.get("THIS").getServer();
        ccl = Thread.currentThread().getContextClassLoader();
        zorka.registerMbs("jboss", mbs, ccl);
    }
    return record;
  }
  return this;
} // __mbsRegister()

spy.add(spy.instance("JBOSS_MBS_REGISTER")
  .onReturn(spy.fetchArg("THIS", 0), (com.jitlogic.zorka.core.spy.SpyProcessor)__mbsRegister())
  .include(spy.byMethod(0, "org.jboss.naming.NamingService", "startService", null)));

This is part of Zorka configuration for JBoss 5.x. It intercepts instantiation of MBeanServerImpl (at the end of its constructor) and calls collect() function of jboss namespace (everything is declared in jboss namespace). SpyDefinition objects are immutable, so when tinkering with sdef multiple times, remember to assign result of last method call to some variable. Method spy.instance() returns empty (unconfigured) object of SpyDefinition type that can be further configured using methods described in next sections.

For more examples see Examples section above.

Choosing processing stages

Argument fetch and argument processing can be done in one of several points (see Diagram 1). Use the following functions to choose stage (or probe point):

sdef = sdef.onEnter(args...);
sdef = sdef.onReturn(args...);
sdef = sdef.onError(args...);
sdef = sdef.onSubmit(args...);

Fetching, processing and collecting data

Data to be fetched by probes can be defined using withArguments() method:

sdef = sdef.onEnter(arg1, arg2, ...);

Fetch probes, processors and collectors can be used as arguments (see spy library functions). Fetch probes produce data that are passed through processors to collectors as dictionary objects (Map<String,Object>). Data are accessed using string keys as in ordinary hash maps. Processors can get named values from map objects or write other values to passed maps. Processors can also filter out records. Collectors are just special kinds of processors that have some side effects. >Collector can update statistics, write to logs, send messages via syslog etc.

Additional notes:

when instrumenting constructors, be sure that this reference (if used) can be fetched only at constructor return -this is because at the beginning of a constructor this points to an uninitialized block of memory that does notrepresent any object, so instrumented class won't load and will crash with class verifier error;

Looking for classes and methods to be instrumented

sdef = sdef.include(matcher1, matcher2, ...);

Using include() method administrator can add matching rules filtering which methods are to be instrumented. Methods matching any of passed matchers will be included. Matchers can be defined using spy.byXXX() functions.

Example: logging catalina requests

This example will instrument org.apache.catalina.core.StandardEngineValve.invoke() method and log its invocations in agent log.

spy.add(spy.instance("CATALINA_LOG_REQUESTS")
  .onEnter(spy.zorkaLog("INFO", "HTTP", "request occured."))
  .include(spy.byMethod("org.apache.catalina.core.StandardEngineValve", "invoke")));

The way it is done is to create empty spy configuration using spy.instance(), add some action on method entry using .onEnter() method and specify which methods that can be instrumented using .include() method. Finally, created spy configuration is passed to spy.add() function and thus registered in agent instrumentation engine. From now on every class matching specification in .include() will be processed by instrumentation engine and all its matching methods will be instrumented.

Slightly extended version of this script will also log URIs of incoming requests:

spy.add(spy.instance("CATALINA_LOG_REQUESTS")
  .onEnter(spy.fetchArg("REQ", 1), spy.zorkaLog("INFO", "HTTP", "${REQ.request.requestURI}"))
  .include(spy.byMethod("org.apache.catalina.core.StandardEngineValve", "invoke")));

Here we use spy.fetchArg() function to fetch first argument (which is request object), and then we use it inside spy.zorkaLog().

Instrumenting Methods

Choosing processing stages

Fetching, processing and collecting data

Looking for classes and methods to be instrumented

Example: logging catalina requests