ZICO collects and presents trace data from agents. New version brings:

• common list view for traces from all agents - with common filters;
• distributed traces - ability to bind individual traces from various application into one big call tree;
• deep search capabilities with full compressed index;

Note that at this point ZICO2 is developed and tested only on Linux/x64 platforms and JDK11.

Collector can work with two trace data stores:

• :memory - useful for quick development setup; collector stores some data in memory (and discards old data),when collector is restarted all collected data is lost;
• :elastic - stores data in Elastic Server indexes;

Installing collector

Collector requires JDK11, so make sure it is installed before proceeding.

In order to install ZICO collector:

• install OpenJdk11;
• unpack zico2 binary distribution;
• check and fix JAVA_HOME variable in jvm.conf;
• (optional) examine and adjust zico.conf - main collector configuration;
• start collector: ./zico.sh start;
• access collector UI: http://localhost:8640;

Note that collector uses the same port as ZICO 1.x. When running both collectors on the same VM, listen port needs to be changed from 8640 to, for example 8641.

ZICO configuration options

Main configuration file, zico.conf uses EDN format and contains several keys that represent application subsystems. Note that most options have default values, so it is not required to have them all explicitly stated in configuration file.

HTTP service

ZICO uses ring-jetty library, so standard ring settings can be used in :http section.

Default HTTP configuration:

:http {
    :port                8640
    :host                "0.0.0.0"
    :http?               true
    :join?               false,
    :daemon?             false
    :min-threads         2
    :max-threads         32
    :max-queued-requests 256
    :max-form-size       67108864 
}

Interesting settings:

• :host, :port - listen address and port
• :http? - enables or disables plain text HTTP interface (useful for simplified setups);
• :ssl?, :ssl-port - enables TLS and sets listen port for HTTPS;
• :keystore, :keystore-password - path and password to TLS keystore;
• :tls-ciphers, :tls-protocols - limits ciphers and protocols used for communication;
• :min-threads, :max-threads - sets limits for number of threads handling all incoming requests (incl. those from agents);

Logging

Only global log :main is defined at the moment.

Default logging configuration:

:log {
  :path "${zico.home}/log"
  :dump false
  :dump-path "${zico.home}/log/dump.log"
  :mode :time-rolling
  :max-history 7
  :max-size 16777216
  :current-fname "zico.log"
  :history-fname   "zico-%d{yyyy-M-dd-HH-mm}-%i.log.gz"
  :console-pattern "%-12date{YYYY-MM-dd HH:mm:ss.SSS} %-5level - %msg%n"
  :file-pattern    "%-12date{YYYY-MM-dd HH:mm:ss.SSS} %-5level - %msg%n"
  :file-level      :debug
  :console-level   :debug
  :log-levels
    {:ROOT         :debug
     :org.eclipse  :info}
}

Interesting settings:

• :path - log directory;
• :dump, :dump-path - enables dumps of all incoming agent traffic (raw data) and sets path to dump file; useful whensetting
• :mode - log archiving mode (only :time-rolling at the moment);
• :max-history, :max-size - maximum size of a single log file and number of archived files;
• :current-fname, :history-fname - log file name patterns;
• :console-pattern, :file-pattern - log entry patterns (on console and in log file);
• :file-level, :console-level - log levels for file and console logs;
• :log-levels - maps java packages / clojure namespaces to one of: :trace, :debug, :info, :warn, :error;

Authentication settings

There are two authentication modes at the moment: :none and :http-basic.

:auth {
  :type :none
  :users {
    "admin" "zico"
    "user" "SSHA512:iKMGE...2ky2PlToqtQ=="}
  :admin-users #{"admin"}
}

Users database is stored in :users map. OAuth/OIDC more and more secure password hashing for local passwords will be implemented in future iteration. Note that passwords can be hashed using salted SHA-512. In order to generate hash from password, run zico .jar with passwd argument:

java -jar zico.jar passwd

Users with access to administrative APIs should have their logins added to :admin-users.

Trace store configuration

Trace store configuration has following configuration:

 :tstore {
    :type                 :memory
    :url                  "http://127.0.0.1:9200"
    :username             "zico"
    :password             "zico"
    :trust-store          "/path/to/truststore.jks"
    :trust-store-pass     "changeit"
    :keystore             "/path/to/keystore.jks"
    :keystore-pass        "changeit"
    :timeout              60
    :threads              16
    :insecure?            false
    :memstore-size-max    768
    :memstore-size-del    256
    :name                 "ztrace"
    :session-timeout      90
    :index-size           16384
    :number_of_shards     1
    :number_of_replicas   0}

• :type - trace store type, currently :memory and :elastic are possible;
• :url - URL to elastic server;
• :trust-store, :trust-store-pass - trust store path and password for TLS connections between elastic and collector;
• :keystore, :keystore-pass - key store path and password (for client authentication to elastic);
• :insecure? - disables TLS certificate verification when connecting to elastic server;
• :name - name prefix for indexes created in elastic server;
• :username, :password - for authentication in Elastic Server;
• :instance - collector instance ID (should be unique for each collector connected to the same Elastic instance);
• :session-timeout - timeout after which inactive sessions will be dropped;
• :index-size, :index-count - index size, above which rotation process will activate; note that there is some lag between exceedingsize and actual rotation, so resulting indexes might be a bit bigger (approx. 10-15%); collector will keep up to :index-countindexes;
• :memstore-size-max, :memstore-size-del - maximum size of memory store, amount of data to be removed when memory storesize is exceeded (applies to :memory mode); in megabytes;
• :num-shard, :num-replicas - number of shards and replicas for elastic indexes (might be usable when in cluster);
• :threads - number of threads handling elastic connections;

Filter definitions

In :filter-defs it is possible to define attribute filters available under icons in right-upper screen corner:

:filter-defs
 [{:attr "component", :description "Component", :icon "awe/puzzle#light"}
  {:attr "local.service", :description "Service", :icon "awe/cubes#light"}]

Each filter definition has :attr which contains attribute name used in filtering, :description and :icon for defining user interface aspects of defined filter.

Trace types

This map contains settings that help in visualisation of various types of collected trace data. It is possible to select attribute to be displayed in main list, icon and component name that determines given trace type.

:trace-types
 {:db  {:component "db", :render :db.statement, :icon "awe/database#darkgreen"}
  :http {:component "http", :render :http.url, :icon "awe/globe#steelblue"}
  :amqp {:component "amqp", :render :call.method, :icon "awe/paper-plane-empty#darkred"}
  :spring {:component "spring", :render :method, :icon "awe/leaf#darkgreen"}
  :breaker {:component "breaker", :render :call.method, :icon "awe/flash#darkkhaki"}
  :job {:component "job", :render :call.method, :icon "awe/clock#darkred"}
  :jms {:component "jms", :render :call.method, :icon "awe/paper-plane-empty#darkkhaki"}
  :ejb {:component "ejb", :render :call.method, :icon "awe/cube#steelblue"}}

Elastic Search

Securing ZICO with Elastic backend

Note that OSS version of Elastic server does not have any security features, so Elastic with at least BASIC license and X-Pack is required.

Enable X-Pack Security subsystem - edit config/elasticsearch.yml file and add following setting:

xpack.security.enabled: true

Restart elastic server and set up passwords for system accounts:

bin/elasticsearch-setup-passwords interactive

Add role for ZICO collector:

curl -XPOST -u elastic 'localhost:9200/_security/role/zico_admin' -H "Content-Type: application/json" -d '{
  "cluster": [ "monitor" ],
  "indices" : [
    {
      "names" : [ "zico*" ],
      "privileges" : [ "all" ]
    }
  ]
}'

Add user for ZICO collector:

curl -XPOST -u elastic 'localhost:9200/_security/user/zico' -H "Content-Type: application/json" -d '{
  "password" : "zicoPassword",
  "full_name" : "ZICO collector",
  "email" : "zicoadmin@mycompany.com",
  "roles" : [ "zico_admin" ]
}'

Set username and password in zico.edn file and restart collector.

Configuring TLS between Elastic and ZICO

In general, certificates should be generated in company trusted CA but for testing/development it is possible to generate self-signed certificate using openssl tool:

openssl req -x509 -newkey rsa:2048 -keyout localhost.key -out localhost.crt -days 365 -nodes -subj '/CN=localhost'

Create trust store for zico:

keytool -import -alias localhost -file localhost.crt -keystore localhost.jks -storepass changeit

In zico.edn configure :trust-store* and optionally :keystore* settings.

Collector monitoring

ZICO collector exposes its internal metrics using Micrometer library. Monitoring settings are kept in :metrics section:

 :metrics {
   :type :prometheus
   :prefix "zico"
   :host "127.0.0.1:9200"
   :index "zico_metrics"
   :username "zico"
   :password "zico"
   :step 60
   :threads 2
   :conn-timeout 1
   :read-timeout 1
   :batch-size 10000}

• :type - can be :prometheus or :elastic;
• :prefix - name prefix for metrics exported to prometheus;
• :host, :index, :username, :password - address, index name and credentials to elastic server;
• :step, :threads, :conn-timeout, :read-timeout, :batch-size - tuning settings for elastic reporting;

Admin operations

Collector exposes some API operations for administrators.

When index attribute mapping overflow occurs in Elastic, collector will automatically rotate index. In such cases analyzing attribute names and transforming repeatable names into single one can be useful. In order to list all attributes from given index, call /api/admin/attrs/<n> endpoint:

curl -s -u admin:zico -X GET http://localhost:8640/api/admin/attrs/2 | jq -r '.[]'

After determining repeatable names, attribute transform rules can be set in zico.edn in :attr-transforms section, for example:

:attr-transforms [
    {:match "par__\\d+",                 :replace "par__nnn"}
    {:match "cookiein__state-\\d+-\\d+", :replace "cookiein__state"}
    {:match "(par__[a-z]+)_j_idt\\d+",   :replace "$1"}]

This will replace all par_* ending with one or more digits attributes to par_nnn and all cookiein__state-123-456 to cookiein_state. Regex groups can also be, so in third example par_abcd_j_idt123123 will be transformed to par_abcd (replace abcd with any sequence of lower-case letters).

It is also possible to manually rotate elastic index (when problem occurs) with use of /api/admin/rotate endpoint:

curl -s -u admin:zico -X POST http://localhost:8640/api/admin/rotate

Securing agent-collector communications using TLS

Both agent and collector make it possible to encrypt communications using TLS.

Creating CA (optional)

Certificates can be issued by any CA supporting your system or by local CA. This section contains instructions for sample local CA setup:

openssl genrsa -out ca.key 4096
openssl req -new -x509 -key ca.key -subj '/CN=rootca.mycompany.com'  -days 3650 -out ca.crt
echo 1000 > ca.srl
openssl x509 -outform der -in ca.crt -out ca.der
keytool -import -alias ca -keystore ca.jks -file ca.der -storepass changeit -noprompt

Configuring TLS: collector and agent

Web server certificate for ZICO collector is needed.

Creating web server certificate and private key:

openssl genrsa -out zico.mycompany.com.key 2048
openssl req -new -key zico.mycompany.com.key -subj '/CN=zico.mycompany.com' -out zico.mycompany.com.csr
openssl x509 -req -in zico.mycompany.com.csr -CA ca.crt -CAkey ca.key -days 365 -out zico.mycompany.com.crt
openssl pkcs12 -export -in zico.mycompany.com.crt -inkey zico.mycompany.com.key -name zico.mycompany.com -password pass:changeit -out zico.mycompany.com.p12

Configuring collector (zico.edn):

{
  :http {
    :ssl? true
    :http? false
    :keystore "zico.mycompany.com.p12"
    :keystore-type "pkcs12"
    :key-password "changeit"
  }
  ...
}

Configuring agent (zorka.properties):

tracer.net.protocol = cbor
tracer.net.http.url = https://zico.mycompany.com:8643
tracer.net.http.truststore = /opt/myapp/zorka/ca.jks
tracer.net.http.truststore.pass = changeit