JASIG CAS Monitoring

Zorka provides two scripts: one for CAS server and one for CAS java client. CAS server script aggregates statistics, implements audit logging of CAS server and enables tracing of basic CAS functions (in addition to standard HTTP tracing).

Solution described below provides the following functionalities:

  • standard JVM / application server monitoring (not described in this section - please read section specific to application server of your choice;

  • basic CAS-related statistics (authentication, ticket validation etc.);

  • audit logging independent from built-in CAS auditing that can store logs in local file or send logs in real time to remote syslog server or zabbix server;

  • tracing - recording detailed call trees of HTTP/CAS requests (similiar to profiling);

CAS Server

Basic agent configuration

In addition to standard configuration for application server CAS is running on, apps/cas.bsh script has to be loaded. For CAS+Tomcat, list of loaded scripts in zorka.properties should look like this:

scripts = jvm.bsh, zabbix.bsh, apache/tomcat.bsh, apps/cas.bsh

This will enable collection of basic statistics and auditing to a local file. Statistics will be accessible via JMX and audit file will be $ZORKA_HOME/log/cas_audit.log.YYYY-MM-DD (where YYYY, MM, DD stands for year, month and day).

Audit file mirrors what is logged by built-in auditing functionality of CAS server - with zorka agent it is possible to have audit logging without modifying (monitored) application. Its content looks like this (log lines are long and thus wrapped):

CAS audit log file

Audit logging can be disabled overall by adding cas.audit = no setting to zorka.properties.

Monitoring basic CAS statistics with Zabbix

The following statistics are collected by zorka agent:

  • numbers of calls/errors and execution times for standard CAS operations (as specified for audit purposes): AUTHENTICATION, TICKET_GRANTING_TICKET,SERVICE_TICKET, PROXY_GRANTING_TICKET, SERVICE_TICKET_VALIDATE;

  • time between issuing and consuming service ticket - TICKET_AGE_MS. This is useful for spotting issues with network operation or load balancers;

All above statistics can be monitored by importing Template_Zorka_CAS.xml template file distributed with zorka agent and attaching it to monitored CAS server. In addition Template_Zorka_HTTP.xml and Template_Zorka_LDAP.xml files can be useful - first one will register standard HTTP stats, second one - LDAP statistics (if LDAP is used in monitored CAS isntance). Collected data can be dispolayed as graphs in zabbix:

CAS audit log file

CAS audit log file

Enabling audit logging

Audit logging to local file is enabled by default. Zorka agent makes remote logging possible - either via syslog or to zabbix server using zabbix trapper protocol. In real world scenarios this is useful as it limits abilities of CAS server administrator to tamper audit logs.

In order to disable audit logging to file, but still be able to send audit logs to other sources, add cas.audit.file = no setting to zorka.properties.

Audit logs with syslog

Audit log can be transmitted to remote syslog server in real time. In order to enable audit logging via syslog, the following properties need to be set in zorka.properties:

cas.audit.syslog = yes
cas.audit.syslog.server = <address-of-syslog-server>

Audit trails will be logged as F_AUDIT - in order to change syslog facility, use cas.audit.syslog.facility property.

Audit logs with zabbix

Audit logs can also be stored in zabbix database. In order to enable submitting audit logs to zabbix, the following properties in zorka.properties file need to be set:

cas.audit.zabbix = yes
cas.audit.zabbix.host = <host-name-in-zabbix>
cas.audit.zabbix.addr = <address-of-zabbix-server>

In zabbix: create host named as in cas.audit.zabbix.host setting, import and attach Template_Zorka_CAS_audit.xml template to it.

Audit logs in Zabbix

Securing audit logs

Plaintext tickets appearing in audit logs can pose a security risk and generally should not be used in production. By adding cas.audit.secure = yes to zorka.properties file administrator can obfuscate this information: agent will calculate CRC-32 sums and log them instead. Configuration scripts can be easily modified to achieve stronger obfuscation than that as zorka agent has MD5 and SHA1 calculation functionalities.

Note that if HTTP are also traced, password parameter and CASTGC cookie has to be excluded:

http.params.exclude = password
http.cookies.exclude = CASTGC
http.headers.exclude = cookie, Set-Cookie

CAS Client

CAS client monitoring script is suitable for official cas java client and has been tested with version 3.2.

scripts = jvm.bsh, zabbix.bsh, apache/tomcat.bsh, apps/cascli.bsh

Above example assumes CAS-ified application runs on Tomcat. Replace apache/tomcat.bsh with config script for your favorite application server. CAS client script can trace (see below) ticket validations and and aggregates some statistics regarding ticket validation. Import and apply Template_Zorka_CAS_client.xml to have client-side of CAS authentication process monitored in Zabbix.

Tracing CAS requests

Both scripts (client and server) will provide tracing of CAS-specific requests if zorka.properties contains tracer = yes property. Note that tracing needs to be configured and tuned properly in order to avoid performance hit on monitored applications. See Tracer Configuration section for more details.

Note that by default tracer collects only requests (HTTP, LDAP, CAS etc.) that took too long or caused errors and filters out short method calls from collected method call trees. This setting is suitable for production monitoring. In order to log all requests (eg. for debugging), add tracer.min.trace.time = 0 property to zorka.properties.

Disabling automatic method filterin in traces is also possible but in most cases it makes traces less readable, so use it with caution: tracer.min.method.time = 0 in zorka.properties.

Traces from CAS specific functionalities will be collected in two cases: when execution time of traced functionality exceeds cas.trace.time (which is by default equal to tracer.min.trace.time) or when error occurs:

CAS Traces

Additional information attached inside trace detail tree might help spot problems:

Method call tree for CAS

Debugging webflow with tracer

In order to add more information about webflow and its state transitions, add the following properties to zorka.properties file:

spring.webflow.trace = yes

spring.webflow.trace.request.scope = yes
spring.webflow.trace.flash.scope   = yes
spring.webflow.trace.view.scope    = yes
spring.webflow.trace.flow.scope    = yes

This will result in adding webflow states, results and flow variables like in this screenshot:

Method call tree for CAS with webflow details

Note that on heavily loaded sites this might generate a lot of data and incur some performance hit (I estimate it at around 50 microseconds per webflow action on Sandy Bridge), so if you plan to record webflow states in production, be sure to perform adequate load (stress) tests and perform adequate tracer tuning if required.

Configuration options reference

  • cas.audit.secure = no - enables obfuscation of CAS tickets;

  • cas.stats = yes - controls maintaining of CAS-specific statistics;

  • cas.stats.mbean = zorka:name=Cas,type=ZorkaStats - mbean name that will maintain statistics for CAS;

  • cas.trace = ${tracer} - whenever tracing CAS specific requests is enabled;

  • cas.trace.time = ${tracer.min.trace.time} - minimum execution time of traced functionalities required to submit their executions to collector;

Server specific options

  • cas.audit = yes - enables/disabled CAS auditing;

  • cas.audit.file = yes - enables logging audit information to local file;

  • cas.audit.syslog = no - enables sending audit information to syslog server;

  • cas.audit.zabbix = no - enables sending audit information to zabbix server;

  • cas.audit.secure = no - replaces all tickets in audit log and trace with CRC32 sums;

  • cas.audit.tag = cas.audit - zabbix item or syslog tag audit records will be marked with;

  • cas.audit.syslog.addr = ${zorka.syslog.server} - syslog server address (127.0.0.1);

  • cas.audit.file.path = ${zorka.log.dir}/cas_audit.log - path to log file;

  • cas.audit.zabbix.addr = ${zabbix.server.addr} - zabbix host (defaults to ${zabbix.host.addr});