Zorka

Configuration options for both tracing and monitoring HTTP requests are a convention used by all HTTP related extension scripts (eg. catalina.bsh for Tomcat and all its derivatives/wrappers, eg. JBoss).

HTTP monitoring can provide the following information:

http performance statistics: number of HTTP requests, average and peak processing time for HTTP requests;
http contention statistics: peak number of threads concurrently processing HTTP requests;
logging of slow HTTP requests and HTTP errors;
tracing (profiling) HTTP requests;

HTTP monitoring is implemented in two scripts: general HTTP monitoring parts (http.bsh) and application server specific parts (eg. catalina.bsh). In order to enable HTTP monitoring both scripts have to be present (which is the case after installing cardridge for specific application server as both scripts are typically provided by default with most cardridges).

With Zabbix, use Template_Zorka_HTTP.xml template. Also, there is a bunch of configuration options that can be used to fine-tune HTTP monitoring. See next section for more details.

HTTP support script also has capability of logging slow HTTP requests and HTTP errors. As it is integrated with traps.bsh script, standard properties like traps.slow, traps.error, traps.file etc. can be used to configure slow/error logs. See traps.bsh documentation for more details.

Configuration properties for HTTP monitoring

This section describes options for standard HTTP monitoring elements reported as JMX beans and slow query logs.

Settings for HTTP statistics:

http.stats = yes - monitor HTTP statistis;
http.stats.congestion = yes - monitor congestion on HTTP processing (must have http.stats = yes in order to work correctly);
http.stats.mbean = zorka:type=ZorkaStats,name=HttpStats - object name of mbean that will store HTTP processing statistics;

Settings for HTTP tracing:

http.trace = yes - enable or disable tracing of HTTP requests;
http.trace.time = 1000 - minimum execution time of collected http trace (traces that execute faster will be dropped);
http.trace.exclude - contains comma-separated list of URIs (strings or regular expressions when starting with ~character) that will be excluded from tracing; if this option is used, all URIs will traced with exception of those on list;
http.trace.include - contains comma-separated list of URIs (strings or regular expressions when starting with ~character)

Slow HTTP request log settings:

http.slow = no - enable or disable slow query log;
http.slow.time = 5000 - minimum time HTTP;
http.slow.file = yes - enable or disable logging to a file;
http.slow.file.max = 8M - maximum slow log file size;
http.slow.file.num = 8 - maximum number of archived slow logs;
http.slow.file.path = ${zorka.log.dir}/http-slow.log - path to slow query log file;
http.slow.format = [${TIME}] ${URI}${PARAMS} -> ${STATUS} - query log messages format;

HTTP error log settings:

http.error = no - enable or disable HTTP error log;
http.error.file = yes - enable or disable logging to a file;
http.error.file.max = 8M - maximum error log file size;
http.error.file.num = 8 - maximum number of archived error logs;
http.error.file.path = ${zorka.log.dir}/http-error.log - path
http.error.format = [${TIME}] ${URI}${PARAMS} -> ${STATUS} - query log messages format;
http.error.codes - error status codes; note that there is a default list of HTTP status codes considered as errors,so this property does not need to be set in most cases;

Settings for processing of HTTP parameters:

http.params = no - whenever HTTP parameters should be processed (and used in request logs or by tracer);
http.params.exclude = <none> - comma-separated list of omitted HTTP parameter names (exact names only);
http.param.maxlen = 128 - maximum length of single HTTP parameter - parameter will be truncated if longer;

Settings for processing of HTTP cookies:

http.cookies = yes - whenever HTTP cookies should be processed;
http.cookies.exclude - comma-separated list of omitted HTTP cookie names (exact names or regularexpressions when starting with ~);
http.cookies.include - comma-separated list of HTTP cookies to be collected by tracer (exact names or regularexpressions starting with ~); if this option is used, only cookies from this list will be collected and remainingones will be dropped;
http.cookies.iprefix, http.cookies.oprefix - input and output cookies prefixes in tracer data;
http.cookie.maxlen = 128 - maximum length of a single HTTP cookie - cookie will be truncated if longer;

Settings for HTTP headers processing:

http.headers = yes - whenever HTTP headers should be processed;
http.headers.exclude - comma-separated list of omitted HTTP headers names (exact names or regularexpressions when starting with ~);
http.headers.include - comma-separated list of HTTP headers to be collected by tracer (exact names or regularexpressions starting with ~); if this option is used, only headers from this list will be collected and remainingones will be dropped;
http.headers.iprefix, http.headers.oprefix - input and output headers prefixes in tracer data;
http.headers.maxlen = 128 - maximum length of a single HTTP header - header will be truncated if longer;

The following macros can be used in format strings for HTTP (eg. http.slow.format and http.error.format):

${TIME} - request execution time (as human readable string);
${URI} - request URI (without parameters);
${PARAMS} - request parameters (including POST parameters) - as a single string with params delimited with & character;

Tagging and classifying HTTP requests

Zorka agent can separate incoming HTTP requests and keep separate statistics for various parts of web application. Administrator can add arbitrary number of rules assigning URIs to tags, option enabling tagging and default tag for requests that do not match any rule. The following options are used to enable filtering:

http.tags = yes - enables tagging;
http.tags.0 = MY_DEFAULT_TAG - default tag (for all requests that don't match any of defined rules);
http.tags.<n> = MY_TAG - where <n> is rule number - tag name used to statistics collection;
http.tags.<n>.uri = uri-regex - regular expression that will be used to match HTTP requests;

Numbers should be consecutive and start with 1. Collected statistics will be available using the following queries:

http.discovery["byTag", "MY_VAR"] - zabbix discovery of all tags; tag name will be stored as variable {#MY_VAR};
http.calls["byTag", "MY_TAG"], http.errors["byTag", "MY_TAG"], http.peak["byTag", "MY_TAG"],http.threads["byTag", "MY_TAG"], http.avg5["byTag", "MY_TAG"], http.avg15["byTag", "MY_TAG"] - all usual counters;substitute "MY_TAG" with appropiate tag name;

Zabbix templates

There is one universal zabbix template for HTTP processing which will work on all application servers for which instrumentation of HTTP requests is supported. It monitors number of requests/errors, request handling time averages, peak request handling time and contention (max number of threads handling HTTP requests simultaneously.

Template_Zorka_HTTP.xml - template for HTTP monitoring;

HTTP Monitoring

Configuration properties for HTTP monitoring

Tagging and classifying HTTP requests

Zabbix templates