inspectIT OpenCensus Edition is a Java agent that provides the automatic collection of metrics and traces using OpenCensus standard. Using Java Instrumentation API, the inspectIT OCE agent is capable to instrument any JVM-based application with a zero-configuration approach. It overcomes the problem of instrumentation libraries like OpenCensus that require you to manually instrument your application and change the source code.

Introduction

inspectIT OpenCensus Edition is a spin-off from the original inspectIT APM project. Unlike the original inspectIT version, this edition provides only a Java agent that is built on top of the OpenCensus implementation, without any other components or dependencies.

Open source APM tools have significantly improved in the past few years and instrumentation libraries like OpenCensus, openTracing, Zipkin, Jaeger, micrometer, etc have gained a lot of popularity. The main problem that inspectIT OCE tries to solve is the time needed to introduce these instrumentation libraries in already active projects. As these libraries require manual instrumentation, this would result in code changes in hundreds of projects for many organizations. This often hinders companies from moving away from commercial APM solutions.

The inspectIT OCE agent uses Java byte-code manipulation to set up the OpenCensus instrumentation library with zero-configuration and requires no source-code changes. Furthermore, inspectIT OCE instruments your application in a way that most important metrics are automatically monitored and traces are collected and propagated as well. This allows DevOps teams to start collecting performance data about their Java applications in seconds.

The decision to base the agent on the OpenCensus implementation was made because of the flexibility that OpenCensus provides with respect to where the collected data can be exported and stored. OpenCensus offers multiple exporters for Java (Prometheus, Zipkin, Jaeger, etc) and inspectIT OCE supports them all. Thus, combining different open source tools you can come to the desired APM solution based on open source software only, as an example openapm.io landscape shows below:

Sunset
Figure 1. Possible landscape with inspectIT OCE and other open source tools (provided by the openapm.io)

It’s important to mention that inspectIT OCE seamlessly integrates with the OpenCensus library if it is used in your application. The data collected by your manual source code instrumentation will be combined with everything that the agent collects as well.

Getting Started

This section describes how to start using the inspectIT OCE agent.

Quick Start

You can find and download all released versions of inspectIT OCE in our GitHub repository. You can get the current version on the following link:

$ wget https://github.com/inspectIT/inspectit-oce/releases/download/0.1.M2/inspectit-oce-agent-0.1.M2.jar

The best way to start using inspectIT OCE is to attach the java agent when starting your Java program. Use the -javaagent command-line option to reference the inspectIT OCE jar:

$ java -javaagent:"/path/to/inspectit-oce-agent-0.1.M2.jar" -jar my-java-program.jar

The Installation section further describes what options are available for installing the agent, as well as how you can attach the agent to an already started JVM. In the Configuration part you can find more details on how to configure the inspectIT OCE agent.

inspectIT OCE Demo

If you would like to check inspectIT OCE in action with a demo application, you can use our docker compose example. We use the distributed version of the Spring PetClinic sample application as the target application in our demo. To demonstrate the flexibility of the OpenCensus bases inspectIT agent, we provide different demo scenarios covering different monitoring and APM components:

Table 1. Demo Scenarios
Docker-compose file Landscape Description

docker-compose-influxdb.yml

demo landscape influxdb

Uses InfluxData Telegraf for metrics gathering, InfluxDB for metrics storage and Grafana for Dashboards.

docker-compose-elastic.yml

demo landscape elastic

Uses InfluxData Telegraf for metrics gathering, Elasticsearch for metrics storage, Kibana and Grafana for visualization and Dashboards.

docker-compose-prometheus.yml

demo landscape prometheus

Uses Prometheus Server for metrics gathering and storage, Grafana for Dashboards.

All of the demo scenarios are fully configured with predefined dashboards, so you can get started in 5 minutes.

Launching the Demo

Pre-requisites: To launch the demo, Docker needs to be installed on your system. If you are using Docker for Windows or running Docker in a virtual machine, ensure that Docker has at least 4GB main memory assigned.

Change into the inspectit-oce-demo/ directory and execute the following command to launch the desired demo scenario (replace [SCENARIO_POSTFIX] with the postfix of the scenario you would like to launch):

$ docker-compose -f docker-compose-[SCENARIO_POSTFIX].yml up

This will start all the Docker containers required for the corresponding demo scenario, including the Petclinic demo application.

You can access the demo application (Petclinic) under http://localhost:8080. Details on accessing monitoring infrastructure components are listed below, depending on the selected demo scenario.

Demo Scenarios

InfluxDB Scenario

In this scenario the following components are preconfigured and used for monitoring:

demo landscape influxdb
  • inspectIT OCE agent: Instruments all the target demo application components.

  • InfluxData Telegraf: Gathers metrics exposed by the agent.

  • InfluxDB: Stores metric data collected by Telegraf as time series.

  • Grafana: Provides predefined example Dashboards visualizing the metrics collected by the inspectIT OCE agent. The query language Flux is used to query the data from InfluxDB.

You can access Grafana through http://localhost:3001 using admin as username and demo as password.

Elastic Scenario

In this scenario the following components are preconfigured and used for monitoring:

demo landscape elastic
  • inspectIT OCE agent: Instruments all the target demo application components.

  • InfluxData Telegraf: Gathers metrics exposed by the agent.

  • Elasticsearch: Stores metric data collected by Telegraf as time series.

  • Kibana: Allows to browse the indices from Elasticsearch.

  • Grafana: Provides predefined example Dashboards visualizing the metrics collected by the inspectIT OCE agent.

You can access Grafana through http://localhost:3001 using admin as username and demo as password.

Kibana can be accessed through http://localhost:5601.

Prometheus Scenario

In this scenario the following components are preconfigured and used for monitoring:

demo landscape prometheus
  • inspectIT OCE agent: Instruments all the target demo application components.

  • Prometheus Server: Gathers metrics exposed by the agent.

  • Grafana: Provides predefined example Dashboards visualizing the metrics collected by the inspectIT OCE agent.

You can access Grafana through http://localhost:3001 using admin as username and demo as password.

Prometheus can be accessed through http://localhost:9090.

Demo Grafana Dashboards

The demo scenarios come with the following predefined Grafana Dashboards:

Table 2. Demo Dashboards
Name / Grafana Marketplace Links Description Screenshot

System Metrics

InfluxDB Version

Elastic Version

Prometheus Version

Shows system metrics, such as system CPU utilization, load average and disk usage.

demo dashboard system

JVM Metrics

InfluxDB Version

Elastic Version

Prometheus Version

Shows JVM metrics related to JVM CPU usage, Memory (Heap and Non-Heap) and Garbage Collection.

demo dashboard jvm

Changing Agent Configurations

In all demo scenarios the inspectIT OCE agents already have their service names and used ports set up. However, if you want to customize any other configuration option you can provide custom configuration files.

The demo starts the following services, of which each is instrumented with an inspectIT OCE Agent:

  • config-server

  • discovery-server

  • customers-service

  • visits-service

  • vets-service

  • api-gateway

For each service you can put your own agent configuration files in the correspondingly named subfolders in inspectit-oce-demo/agentconfig/. For example, if you want to change the configuration of the inspectIT OCE agent attached to the vets-service, you can put a YAML-file into inspectit-oce-demo/agentconfig/vets-service.

Note that it is not required to restart the demo! The agents listen for updates of the corresponding directories and reconfigure themselves when required.

Installation

This section describes the installation details for the inspectIT OCE agent.

Supported Java Runtime Environments

The inspectIT OCE supports Java Runtime Environments in version 1.8.0 and above. You will not be able to use the agent with the lower Java versions. The agent works with different JRE distributions including Oracle, openJDK, Azul, etc.

Adding the Agent to the JVM

The best option for using the inspectIT OCE is to include it to the start of the JVM by using the -javaagent command-line option. This way the agent will be initialized before your application starts.

$ java -javaagent:"/path/to/inspectit-oce-agent-0.1.M2.jar" -jar my-java-program.jar
Some application servers have dedicated scripts that are used to launch the actual JVM that runs the application. In such cases, you must alter the start-up scripts in order to instrument the correct JVM.

Attaching the Agent to an already started JVM

inspectIT OCE also supports attaching the agent to an already running JVM. In such a scenario the collection of metrics and traces will start from the point of the attachment.

The attaching can easily be done with utilities like jattach:

$ ./jattach.sh <pid> load instrument /path/to/inspectit-oce-agent-0.1.M2.jar

In order to find the process ID of a running JVM, you can use the jcmd to list all the running Java processes on your machine:

$ jcmd -l
Using the attach options has some limitations with respect to using the OpenCensus instrumentation library in combination with the inspectIT OCE agent. Please refer to Using OpenCensus Library with inspectIT OCE section to understand these limitations.

Configuration

This section describes the configuration of the inspectIT OCE agent.

Configuration Basics

inspectIT OCE tries to implement the zero-configuration approach but lets you externalize the configuration and influence every bit of its configuration if this is required. Internally it uses the Spring-based PropertySource order to allow overriding of configuration values. Configuration properties are considered in inspectIT OCE in the following order:

When an invalid configuration is given to inspectIT on startup, the agent will use a fallback configuration. In this fallback configuration, the agent is inactive with the exception of listening for configuration updates.

When giving an invalid configuration through a runtime update to the agent, the agent will simply retain the previous configuration.

The overview of all available inspectIT OCE configuration properties is given in the All Configuration Options section.

Java System Properties

You can pass any configuration property as the Java System property to the Java command that you use to start your Java application. For example, to override the service name used to identify your application reporting the performance data, you can change the inspectit.service-name property:

$ java -Dinspectit.service-name="My Custom Service" -javaagent:/path/to/inspectit-oce-agent-0.1.M2.jar -jar my-java-program.jar

OS Environment Variables

Similar to the Java System properties, inspectIT OCE will also consider all the available operating system environment variables. Due to the relaxed bindings, you can use upper case format, which is recommended when using system environment variables.

$ INSPECTIT_SERVICE_NAME="My Custom Service" java -javaagent:/path/to/inspectit-oce-agent-0.1.M2.jar -jar my-java-program.jar

Relaxed Bindings

Note that due to the Spring-powered configuration approach, the inspectIT OCE agent uses Spring support for relaxed bindings. This means that a property can be specified in different formats depending on the property source. As suggested by Spring, the allowed formats are:

Table 3. Relaxed binding options for properties
Property Note

inspectit.service-name

kebab-case, which is recommended for use in .properties and .yml files.

inspectit.serviceName

Standard camelCase syntax.

inspectit.service_name

Underscore notation (snake_case), which is an alternative format for use in .properties and .yml files.

INSPECTIT_SERVICE_NAME

UPPER_CASE format, which is recommended when using system environment variables.

The formats should be used in the following way, based on the type of property source:

Table 4. Relaxed binding rules
Property Source Format

System properties

Camel case, kebab case, or underscore notation

Environment Variables

Upper case format with the underscore as the delimiter.

Property files (.properties)

Camel case, kebab case, or underscore notation

YAML files (.yaml, .yml)

Camel case, kebab case, or underscore notation

Externalized Configuration Methods

In order to support the externalized configuration inspectIT OCE provides several methods. These methods represent a bridge between certain configuration providers and the inspectIT OCE agent. This way, a configuration can be provided from different sources, for example, file systems, the internet, external services, configuration services, vaults, etc.

The configuration method is considered dynamic if it supports run-time updates. For example, if the configuration file changes on disk, the file-based configuration method will report the updated properties to the inspectIT OCE agent. The agent will then update the internal configuration, effectively changing the behavior of the agent during the run-time.

Note that not all inspectIT OCE configuration properties support dynamic updates. In addition, the Java System properties and OS environment variables always have higher priority, thus dynamically changing a property which is also defined in those sources will have no effect.

All externalized configuration methods must be enabled using Java System properties or OS environment variables, as they are disabled by default. You can have more than one method active at the same time, thus pulling the configuration from different sources.

Table 5. Overview on externalized configuration methods
Method Dynamic updates Enabled by default

File-based Configuration

Yes

No

File-based Configuration

The file-based configuration loads all .yaml/.yml and .properties files in alphabetical order from the given directory path in non-recursive mode. The alphabetical order of files defines their priority, thus if the same property exists in my.properties and yours.properties files, then the one from the my.properties file will be used.

The file-based configuration is activated by specifying the path to an existing directory in the inspectit.config.file-based.path property. It automatically watches for changes in the given directory and reports updates to inspectIT OCE agent. If you would like to disable the dynamic updates you can set the inspectit.config.file-based.watch property to false.

Table 6. Properties related to the file-based configuration method
Property Default Description

inspectit.config.file-based.path

-

The path to the directory to scan for configuration files. An existing directory must be specified for the file-based configuration to be enabled.

inspectit.config.file-based.enabled

true

Can be used to disable the file-based configuration while the path is still specified.

inspectit.config.file-based.watch

true

If true the directory will be watched for changes. When changes occur, the configuration is dynamically updated.

Using OpenCensus Library with inspectIT OCE

If you plan to use the OpenCensus library in an application which will be instrumented later on with inspectIT OCE, some special rules do apply. Following these rules will make sure that there are no run-time problems in your application. Furthermore, a correct configuration will make it possible to combine metrics and traces that you manually collect using the OpenCensus instrumentation library with the ones collected by the inspectIT OCE agent.

  1. Make sure you are using the same version of OpenCensus as inspectIT OCE

    The inspectIT OCE agent in version 0.1.M2 internally uses OpenCensus in version 0.18.0. Please adapt any OpenCensus dependency in your application to this version to avoid run-time conflicts.

    <dependency>
        <groupId>io.opencensus</groupId>
        <artifactId>opencensus-api</artifactId>
        <version>0.18.0</version>
    </dependency>
  2. Set the inspectit.publish-opencensus-to-bootstrap property to true

    Setting the above property to true tells inspectIT OCE that you plan to use the OpenCensus library in combination with the agent. The inspectIT OCE agent will make sure that all OpenCensus classes are then loaded by the bootstrap class loader. This ensures that OpenCensus implementation is shared between your manual instrumentation and the agent instrumentation, making the combing of data possible.

  3. Add the agent to the start of a JVM

    You should install the agent at the start of the JVM. You should avoid attaching the agent during the run-time, as at this point it’s already too late for the inspectIT OCE agent to perform needed setup described in the previous point. If you anyhow do attach the agent, metrics and traces collected by inspectIT OCE will not be combined with the manual instrumentation. In such case you can also omit setting the inspectit.publish-opencensus-to-bootstrap property to true.

Logging Configuration

The inspectIT OCE agent uses Logback library for logging. It comes with the default logging configuration that can be manipulated using configuration properties. The default logback configuration appends inspectIT logs to the console and writes two types of log files:

  1. agent.log - includes the complete log

  2. exceptions.log - includes only WARN and ERROR level messages

The file appenders use the rolling policy with a max file size of 20MB and a history of 30 files.

The available properties for manipulating the default logging configuration are the following:

Table 7. Properties for manipulating default logback configuration
Property Default Description

inspectit.logging.trace

false

Sets the inspectIT OCE log level to TRACE.

inspectit.logging.debug

false

Sets the inspectIT OCE log level to DEBUG only if it’s not already set to TRACE.

inspectit.logging.console.enabled

true

Defines if the console output is enabled.

inspectit.logging.console.pattern

-

Sets a custom logback pattern for console output.

inspectit.logging.file.enabled

true

Defines if the file appenders are enabled.

inspectit.logging.file.pattern

-

Sets a custom logback pattern for file output.

inspectit.logging.file.path

-

Sets a path where agent.log and exceptions.log files are created. By default, the path is resolved to $java.io.tmp/inspectit-oce directory, falling back to /tmp/inspectit-oce if $java.io.tmp System property is not defined.

inspectit.logging.file.include-service-name

true

When true the service name defined in the inspectit.service-name is included in the log messages appended to files.

If you are not satisfied with default logback configuration options you can supply your own logback config file in the property inspectit.logging.config-file. This way the properties specified in the table above are not taken into account.

Metrics

This section provides all the information on metrics collection with inspectIT OCE.

Master switch for metrics

The inspectIT OCE configuration has a master switch for metrics, which can be used to completly disable anything that has to do with metrics collection and exporting. Disabling of metrics completly in the agent can be done by setting the inspectit.metrics.enabled property to false. This way any default inspectIT setting or anything else defined for metrics collection will be overruled. If used, the switch makes sure that the inspectIT OCE agent:

  1. disables all metrics recorders

  2. does not set up any metrics exporter

Metrics Recorders

Metrics recorders are responsible for capturing system metrics, such as processor or memory usage. All metrics recorders provided by inspectIT OCE publish the recorded data through the OpenCensus API. Therefore all recorded metrics can be exported via any of the exported via any of the supported metrics exporters. Currently the inspectIT OCE agent is capable of recording the following metrics:

The metrics above and their capturing logic are based on the open-source micrometer project.

In the following sections we provide detailed information on the collected metrics and how they can be configured. In general metrics are grouped together based on the recorder which provides them.

Many recorders poll the system APIs for extracting the metrics. The rate at which this polling occurs can be configured. By default all polling based recorders use the duration specified by inspectit.metrics.frequency. The default value of this property is 15s. Overwriting inspectit.metrics.frequency will cause all recorders to use the given frequency in case they do not have an explicit frequency in their own configuration.

Default metrics settings

By default, all metrics are captured if they are available on the system. If you do not want certain metrics to be recorded, you need to disable them manually. For example, if you want to disable the system.average metric of the processor recorder, you need to use the following configuration:

inspectit:
  metrics:
    processor:
      enabled:
        system.average: false

CPU Metrics

Processor metrics are recorded by the inspectit.metrics.processor recorder. This recorder polls the captured data from the system with a frequency specified by inspectit.metrics.processor.frequency which defaults to inspectit.metrics.frequency. The available metrics are explained in the table below.

Table 8. CPU metrics
Metric Description Unit OpenCensus Metric Name

count

The number of processor cores available to the JVM

cores

system/cpu/count

system.average

The sum of the number of runnable entities queued to the available processors and the number of runnable entities running on the available processors averaged over a minute for the whole system. See the definition of getSystemAverageLoad() for more details.

percentage

system/load/average/1m

system.usage

The recent CPU usage for the whole system

percentage

system/cpu/usage

process.usage

The recent CPU usage for the JVM’s process

percentage

process/cpu/usage

The availability of each processor metric depends on the capabilities of your JVM in combination with your OS. If a metric is not available, the inspectit OCE agent will print a corresponding info in its logs on startup.

Disk Space Metrics

Disk space metrics are recorded by the inspectit.metrics.disk recorder. This recorder polls the captured data from the system with a frequency specified by inspectit.metrics.disk.frequency which defaults to inspectit.metrics.frequency. The available metrics are explained in the table below.

Table 9. Disk metrics
Metric Description Unit OpenCensus Metric Name

free

The free disk space

bytes

disk/free

total

The total size of the disk

bytes

disk/total

Memory Metrics

All memory related metrics are recorded by the inspectit.metrics.memory recorder. This recorder polls the captured data from the system with a frequency specified by inspectit.metrics.memory.frequency which defaults to inspectit.metrics.frequency.

The first set of available metrics are general JVM memory metrics:

Table 10. JVM memory metrics
Metric Description Unit OpenCensus Metric Name

used

The amount of used memory

bytes

jvm/memory/used

committed

The amount of memory that is committed for the Java virtual machine to use

bytes

jvm/memory/committed

max

The maximum amount of memory in bytes that can be used for memory management

bytes

jvm/memory/max

For all these metrics inspectIT adds two tags in addition to the common tags: Firstly area which either is heap or nonheap. Secondly an id tag is added specifying the exact memory region, for example PS Old Gen depending on the used garbage collector.

Most JVMs also provide metrics regarding the usage of buffer pools, which are reflected in the following metrics provided by inspectIT OCE:

Table 11. JVM buffer pool metrics
Metric Description Unit OpenCensus Metric Name

buffer.count

An estimate of the number of buffers for each buffer pool

buffers

jvm/buffer/count

buffer.used

An estimate of the memory that the JVM is currently using for each buffer pool

bytes

jvm/buffer/memory/used

buffer.capacity

An estimate of the total capacity of the buffers in each pool

bytes

jvm/buffer/total/capacity

Again for each metric an id tag is added. This tag hereby contains the name of the buffer pool for which the metrics was captured.

Thread Metrics

Thread metrics provide statistics about the number and the state of all JVM threads. They are recorded by the inspectit.metrics.threads recorder. This recorder polls the captured data from the JVM with a frequency specified by inspectit.metrics.threads.frequency which defaults to inspectit.metrics.frequency. The available thread metrics are explained in the table below.

Table 12. Thread metrics
Metric Description Unit OpenCensus Metric Name

peak

The peak number of live threads since the start of the JVM

threads

jvm/threads/peak

live

The total number of currently live threads including both daemon and non-daemon threads

threads

jvm/threads/live

daemon

The total number of currently live daemon threads

threads

jvm/threads/daemon

states

The total number of currently live threads for each state

threads

jvm/threads/states

The states metric provides the amount of threads grouped by their state. For this purpose, an additional tag state is added whose values correspond to the Java Thread.State enum.

Garbage Collection Metrics

The inspectit.metrics.gc recorder provides metrics about the time spent for garbage collection as well as about the collection effectiveness. This recorder is not polling based. Instead, it listens to garbage collection events published by the JVM and records metrics on occurrence.

The availability of all garbage collection metrics depends on the capabilities of your JVM. If the garbage collection metrics are unavailable, the inspectit OCE agent will print a corresponding info in its logs on startup.

The recorder offers the following timing related metrics:

Table 13. Garbage Collection Timings
Metric Description Unit OpenCensus Metric Name

pause

The total time spent for Garbage Collection Pauses

milliseconds

jvm/gc/pause

concurrent.phase.time

The total time spent in concurrent phases of the Garbage Collector

milliseconds

jvm/gc/concurrent/phase/time

Whether pause or concurrent.phase.time are captured depends on the concurrency of the garbage collector with which the JVM was started. For both metrics an action and a cause tag is added. The action specifies what was was done, e.g. a minor or a major collection. The cause tag provides information on the circumstances which triggered the collection.

The following additional garbage collection metrics are also available:

Table 14. Garbage Collection Statistics
Metric Description Unit OpenCensus Metric Name

live.data.size

The size of the old generation memory pool captured directly after a full GC.

bytes

jvm/gc/live/data/size

max.data.size

The maximum allowed size of the old generation memory pool captured directly after a full GC.

bytes

jvm/gc/max/data/size

memory.allocated

Increase in the size of the young generation memory pool after one GC to before the next

bytes

jvm/gc/memory/allocation

memory.promoted

Increase in the size of the old generation memory pool from before a GC to after the GC

bytes

jvm/gc/memory/allocation

Class Loading Metrics

Class loading metrics are recorded by the inspectit.metrics.classloader recorder. This recorder polls the captured data from the system with a frequency specified by inspectit.metrics.classloader.frequency which defaults to inspectit.metrics.frequency. The available metrics are explained in the table below.

Table 15. Class loader metrics
Metric Description Unit OpenCensus Metric Name

loaded

The total number of currently loaded classes in the JVM

classes

jvm/classes/loaded

unloaded

The total number of unloaded classes since the start of the JVM

classes

jvm/classes/unloaded

Metrics Exporters

Metrics exporters are responsible for passing the recorded metrics to a metric storage. They can implement a push approach where metrics are sent to a collector or a pull approach where metrics are scraped by an external system.

If an exporter supports run-time updates it means that it can be enabled/disabled during the run-time or that any property related to the exporter can be changed. This way you can, for example, change the endpoint where exporter pushes the metrics without a need to restart the application. In order to use run-time updates, you must enable one of the externalized configuration methods that support dynamic updates.

inspectIT OCE current supports the following OpenCensus metrics exporters:

Exporter Supports run-time updates Push / Pull Enabled by default

Prometheus Exporter

Yes

Pull

Yes

Prometheus Exporter

Prometheus exporter exposes the metrics in Prometheus format and is the default metrics exporter set up by inspectIT OCE. When enabled, inspectIT starts a Prometheus HTTP server in parallel with your application. The server is by default started on the port 8888 and metrics can then be accessed by visiting http://localhost:8888/metrics.

Table 16. Properties related to the Prometheus exporter
Property Default Description

inspectit.exporters.metrics.prometheus.enabled

true

If true, the inspectIT OCE agent will try to start the Prometheus metrics exporter and Prometheus HTTP server.

inspectit.exporters.metrics.prometheus.host

0.0.0.0

The hostname or network address to which the Prometheus HTTP server should bind.

inspectit.exporters.metrics.prometheus.port

8888

The port the Prometheus HTTP server should use.

Monitoring Environment

Common Tags

inspectIT OCE provides a map of common tag keys and values that are used when recording all metrics. This feature enables to "label" metrics collected with inspectIT OCE and to share common information about a process or an environment where the process runs. The map can be extended or overwritten by the user when this is required.

Tag provider is responsible for extracting information from a specific source and provides a map of key value pairs that are then combined into the common tag context. Each provider specifies a priority and if the same tag keys are supplied by two providers, then the tag value from the provider with higher priority will be used.

inspectIT OCE current supports the following tag providers:

Provider Tags provided Supports run-time updates Enabled by default Priority

User Defined Tags

-

Yes

No

HIGH

Environment Tags

service-name, host, host-address

Yes

Yes

LOW

Run-time updates currently only support changing of a tag value, but not adding or removing tags.

User Defined Tags

User defined tags can be added to the common tag context by defining the inspectit.tags.extra property.

Setting the user defined tags using properties
inspectit.tags.extra.region=us-west-1
inspectit.tags.extra.stage=preprod
Setting the user defined tags using YAML file
inspectit:
  tags:
    extra:
      region: us-west-1
      stage: preprod

Environment Tags

The environment tag provider extends the common tag context by supplying information about the service name and the network host. The service name is resolved from the inspectit.service-name property, while the host information is extracted using InetAddress.getLocalHost().

On machines that have multiple network interfaces the InetAddress.getLocalHost() method might not provide desired values for host related tags.
Table 17. Properties related to the environment tags provider
Property Default Description

inspectit.tags.providers.environment.enabled

true

Enables or disables the provider.

inspectit.tags.providers.environment.resolve-host-name

true

If false, the tag host containing the resolved host name will not be provided.

inspectit.tags.providers.environment.resolve-host-address

true

If false, the tag host-address containing the resolved host IP address will not be provided.

Self Monitoring

inspectIT OCE is able to monitor itself and report the time spent in components responsible for recording metrics and creating traces. This way a user can have a clear view on what is the overhead introduced by running the inspectIT OCE agent. When self monitoring is enabled, the agent will expose the inspectit/self/duration metric with the total execution time in microseconds. The metric is split by the tag containing the component name.

Exposed self-monitoring metric using the Prometheus exporter
# HELP inspectit_self_duration inspectIT OCE self-monitoring duration [μs]
# TYPE inspectit_self_duration counter
inspectit_self_duration{component_name="ClassLoaderMetricsRecorder",} 33814.0
inspectit_self_duration{component_name="DiskMetricsRecorder",} 38118.0

Self monitoring is disabled by default and can be enabled by setting the inspectit.self-monitoring.enabled property to true.

Components responsible for internal management of inspectIT OCE are at the moment not reporting the time used for internal tasks. Please take the provided numbers only for a basic reference on overhead and don’t assume they are 100% correct.

Appendix

All Configuration Options

inspectit:

  # the name of the service which is being instrumented
  service-name: "InspectIT Agent"

  # defines common tags to be be set on the metrics
  tags:
    # different tag providers that can be configured
    providers:
      # environment provider adds 'service-name', 'host' and 'host-address' tags
      environment:
        # if environment provider is enabled
        enabled: true
        # should the host name be resolved using InetAddress.getLocalHost(), if false 'host' tag is not added by env provider
        resolve-host-name: true
        # should the host address be resolved using InetAddress.getLocalHost(), if false 'host-address' tag is not added by env provider
        resolve-host-address: true

    # specifies user defined tag keys and values as a map
    # these tag values would overwrite any value added by the providers, thus you can easily overwrite tags values by your own
    extra: {}

  # all configurations sources
  config:
    # file based configuration - has the highest priority
    # loads all .yaml/yml and .properties files in alphabetical order from the given path
    # is active when path is not empty or null and enabled is set to true
    file-based:
      # the path to the directory to scan for configuration files
      # scanning happens non-recursive meaning that no subdirectories will be parsed
      path:
      # can be used to disable the file based configuration while the path is still specified
      enabled: true
      # if true the directory will be watched for changes. When changes occur, the configuration is automatically reloaded
      watch: true

  # settings for configuring OpenCensus stats and trace exporters
  exporters:
    # settings for metrics exporters
    metrics:
      # settings for the prometheus exporter (https://github.com/census-instrumentation/opencensus-java/tree/master/exporters/stats/prometheus)
      prometheus:
        # if true, the agent will try to start the Prometheus stats exporter
        enabled: true
        # the hostname or IP-address on which the /metrics endpoint of prometheus will be started.
        host: 0.0.0.0
        # the port on which the /metrics endpoint of prometheus will be started
        port: 8888

  # general settings regarding metrics capturing
  metrics:
    # master switch for metrics capturing. When set to false the following happens:
    #  - all metrics exporters are disabled
    #  - all metrics recorders are disabled
    enabled: true
    # root setting for the polling frequency of all metrics
    # when a metrics has no frequency defined separately, this frequency will be used
    frequency: 15s
    # settings regarding disk related metrics
    disk:
      enabled:
        # if true, the free disk space will be measured and the view "disk/free" is registered
        free: true
        # if true, the total disk space will be measured and the view "disk/total" is registered
        total: true
      # specifies the frequency with which the disk metrics should be polled and recorded
      frequency: ${inspectit.metrics.frequency}
    # settings regarding the measurement of the loaded classes
    classloader:
      enabled:
        # if true, the number of loaded classes will be captured
        loaded: true
        # if true, the number of unloaded classes will be captures
        unloaded: true
      # specifies the frequency with which the classloader metrics should be polled and recorded
      frequency: ${inspectit.metrics.frequency}
    # settings regarding the capturing of processor related metrics
    processor:
      enabled:
        # if true, the number of processors available to the jvm will be captured
        count: true
        # if true, the average load of the last minute of the system load will be captured
        system.average: true
        # if true, the current cpu usage of the system will be captured
        system.usage: true
        # if true, the current cpu usage of this process will be captured
        process.usage: true
      # specifies the frequency with which the processor metrics should be polled and recorded
      frequency: ${inspectit.metrics.frequency}
    # settings regarding the capturing of threads related metrics
    threads:
      enabled:
        # if true, the peak number of live threads since JVM start is captured
        peak: true
        # if true, the current daemon thread count is captured
        daemon: true
        # if true, the current total number of live threads is captured
        live: true
        # if true, for each thread state the number of live threads will be captured
        states: true
      # specifies the frequency with which the thread metrics should be polled and recorded
      frequency: ${inspectit.metrics.frequency}
    # settings regarding the capturing of garbage collection related metrics
    gc:
      enabled:
        # if true, the maximum size of the old generation pool is captured
        max.data.size: true
        # if true, the size of the old generation pool after a full GC is captured
        live.data.size: true
        # if true, the time spent in the concurrent gc phase is captured
        concurrent.phase.time: true
        # if true, the time spent during gc pauses is captured
        pause: true
        # if true, the count of positive increases in the size of the old generation memory pool before GC to after GC is captured
        memory.promoted: true
        # if true, the increase in the size of the young generation memory pool after one GC to before the next is captured
        memory.allocated: true
    # settings regarding the capturing of memory related metrics
    memory:
      enabled:
        # if true, the amount of used memory will be captured
        used: true
        # if true, the amount of committed memory for the JVM to use will be captured
        committed: true
        # if true, the maximum amount of memory that can be used for memory management will be captured
        max: true
        # if true, an estimate of the number of buffers for each buffer pool will be captured
        buffer.count: true
        # if true, an estimate of the used memory for each buffer pool will be captured
        buffer.used: true
        # if true, the total capacity of the buffers for each buffer pool will be captured
        buffer.capacity: true
      # specifies the frequency with which the memory metrics should be polled and recorded
      frequency: ${inspectit.metrics.frequency}

  # logging settings
  logging:
      # path to a custom user-specified logback config file that should be used
      config-file:
      # properties below only work if the default inspectIT OCE logback config file is used
      # sets the inspectIT OCE log level to TRACE
      trace: false
      # sets the level to DEBUG (only if trace is false)
      debug: false
      # settings for the console output
      console:
        # defines if the console output is enabled
        enabled: true
        # defines a custom pattern to output to the console
        pattern:
      # settings for the file-based log output
      # inspectIT OCE will create two log files: agent.log and exceptions.log
      file:
        # defines if the file-based log output is enabled
        enabled: true
        # defines a custom pattern to output to the console
        pattern:
        # defines a custom path where log files should be placed (defaults to /tmp/inspectit-oce)
        path:
        # if the default pattern should include the service name (specified with inspectit.service-name)
        # helpful when you run more than one service on the same host
        include-service-name: true

  # self monitoring, when enabled inspectit reports execution times for metrics and traces collection and processing
  self-monitoring:
    # if self monitoring is enabled
    enabled: false
    # what should be the prefix for the self monitoring measures
    measure-name: inspectit/self/duration

  # defines how many threads inspectIT may start for its internal tasks
  thread-pool-size: 2

  # If true, the OpenCensus API and Implementation will be loaded by the bootstrap classloader.
  # Otherwise they will be loaded by the private inspectIT classloader.
  publish-opencensus-to-bootstrap: false

Default Logback Configuration

<?xml version="1.0" encoding="UTF-8"?>
<configuration>
    <property name="logDir" value="${INSPECTIT_LOG_PATH:-${LOG_TEMP:-${java.io.tmpdir:-/tmp}}/inspectit-oce}" />
    <property name="serviceName" value="${INSPECTIT_LOG_SERVICE_NAME:-[boot]}" />
    <property name="consoleLogPattern" value="${INSPECTIT_LOG_CONSOLE_PATTERN:-%d{ISO8601} %-5p %-6r --- [inspectIT] [%15.15t] %-40.40logger{39} : %m%n%nopex}"/>
    <property name="fileLogPattern" value="${INSPECTIT_LOG_FILE_PATTERN:-%d{ISO8601} %-5p %-6r --- ${serviceName} [%15.15t] %-40.40logger{39} : %m%n%rEx}"/>

    <!-- Console output -->
    <appender name="stdout" class="ch.qos.logback.core.ConsoleAppender">
        <filter class="rocks.inspectit.oce.core.logging.logback.LogbackInitializer$ConsoleFilter" />
        <Target>System.out</Target>
        <encoder>
            <pattern>${consoleLogPattern}</pattern>
        </encoder>
    </appender>

    <!-- Enriched Sysout -->
    <!-- Rollover every day or when file reaches 20MB -->
    <appender name="file" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="rocks.inspectit.oce.core.logging.logback.LogbackInitializer$FileFilter" />
        <file>${logDir}/agent.log</file>
        <encoder>
            <pattern>${fileLogPattern}</pattern>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>logDir/agent.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
            <maxHistory>30</maxHistory>
            <maxFileSize>20MB</maxFileSize>
        </rollingPolicy>
    </appender>

    <!-- Exceptions -->
    <!-- Rollover every day or when file reaches 20MB -->
    <!-- Only warn & error levels-->
    <appender name="exceptions" class="ch.qos.logback.core.rolling.RollingFileAppender">
        <filter class="rocks.inspectit.oce.core.logging.logback.LogbackInitializer$FileFilter" />
        <file>${logDir}/exceptions.log</file>
        <encoder>
            <pattern>${fileLogPattern}</pattern>
        </encoder>
        <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
            <fileNamePattern>${logDir}/exceptions.%d{yyyy-MM-dd}.%i.zip</fileNamePattern>
            <maxHistory>30</maxHistory>
            <maxFileSize>20MB</maxFileSize>
        </rollingPolicy>
        <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
            <level>WARN</level>
        </filter>
    </appender>

    <!-- set log levels - for more verbose logging change inspectit.logging.trace or inspectit.logging.debug to true -->
    <root level="${INSPECTIT_LOG_LEVEL:-INFO}">
        <appender-ref ref="stdout" />
        <appender-ref ref="file" />
        <appender-ref ref="exceptions" />
    </root>

    <!-- deactivate all other loggers, except for errors -->
    <logger name="org" level="error" />
    <logger name="com" level="error" />
    <logger name="rocks.inspectit" level="${INSPECTIT_LOG_LEVEL:-INFO}" />

</configuration>