Installation

Application logs

Kpow uses Logback to record application logs to stdout.

Switch to JSON formatted logs with the environment variableLOG_FORMAT=json

Configure Kpow to write application logs to disk (or any other supported Logback appender) and control the log levels by providing a Logback configuration file.

This example turns on Jetty JAAS debug logging:

<configuration>

<statusListener class="ch.qos.logback.core.status.NopStatusListener" />

<appender name="plain" class="ch.qos.logback.core.ConsoleAppender">
    <encoder>
        <pattern>%d{HH:mm:ss.SSS} %-5level [%thread] %logger{36} – %m%n</pattern>
    </encoder>
</appender>

<appender name="json" class="ch.qos.logback.core.ConsoleAppender">
    <encoder class="ch.qos.logback.core.encoder.LayoutWrappingEncoder">
        <layout class="ch.qos.logback.contrib.json.classic.JsonLayout">
            <timestampFormat>yyyy-MM-dd'T'HH:mm:ss.SSSX</timestampFormat>
            <timestampFormatTimezoneId>Etc/UTC</timestampFormatTimezoneId>
            <appendLineSeparator>true</appendLineSeparator>
            <jsonFormatter class="ch.qos.logback.contrib.jackson.JacksonJsonFormatter">
                <prettyPrint>false</prettyPrint>
            </jsonFormatter>
        </layout>
    </encoder>
</appender>

<logger name="kafka" level="ERROR"/>
<logger name="org.apache.kafka" level="ERROR"/>
<logger name="org.apache.zookeeper" level="ERROR"/>
<logger name="io.confluent.kafka.serializers" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.common.configs" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.serializers" level="ERROR"/>
<logger name="com.amazonaws.services.schemaregistry.deserializers" level="ERROR"/>

<logger name="io.kpow.jaas.spi" level="DEBUG"/>
<logger name="org.eclipse.jetty.jaas" level="DEBUG"/>
<logger name="org.eclipse.jetty.security.authentication" level="DEBUG"/>

<if condition='isDefined("LOG_FORMAT")'>
    <then>
        <root level="INFO">
            <appender-ref ref="${LOG_FORMAT}"/>
        </root>
    </then>
    <else>
        <root level="INFO">
            <appender-ref ref="plain"/>
        </root>
    </else>
</if>
</configuration>

Provide your custom Logback configuration to Kpow as a Java system variable:

java -Dlogback.configurationFile=/kpow/custom-logback.xml -jar /kpow/kpow-latest.jar

The Kpow Docker container can be configured with the JAVA_TOOL_OPTIONS environment variable:

JAVA_TOOL_OPTIONS=-Dlogback.configurationFile=/kpow/custom-logback.xml

In this specific example we turn on debug logging for Jetty JAAS authentication methods (LDAP, etc.).

To put JAAS into debug mode also requires debug="true" in your JAAS config file:

kpow {
         org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
         debug="true"
         file="dev-resources/jaas/hash-realm.properties";
     };

Once configured, application logs are written to file and JAAS debug output is visible:

09:26:32.135 INFO  [oprtr.compute.snapshots.v2-44318b10-b719-46db-96ea-f97495cdce8f-StreamThread-2] operatr.compute-v2 – oprtr.compute.snapshots.v2: streams state changed from REBALANCING to RUNNING
09:26:32.287 INFO  [oprtr.compute.metrics.v2-6126ee6c-0253-4a5d-8d66-cbd3b0a4b0bb-StreamThread-2] operatr.compute-v2 – oprtr.compute.metrics.v2: streams state changed from REBALANCING to RUNNING
09:26:39.349 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [9] configuration captured in 204 ms
09:26:39.605 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [9] configuration captured in 256 ms
09:26:41.409 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – setupPropertyUserStore: Starting new PropertyUserStore. PropertiesFile: dev-resources/jaas/hash-realm.properties refrescallouterval: 0
09:26:41.416 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – Checking PropertyUserStore dev-resources/jaas/hash-realm.properties for admin
09:26:41.417 DEBUG [qtp1357054363-56] o.e.j.j.spi.PropertyFileLoginModule – Found: admin in PropertyUserStore dev-resources/jaas/hash-realm.properties
09:26:41.429 INFO  [qtp1357054363-58] operatr.auth.jetty – login -> user: admin, roles: #{"content-administrators" "kafka-admins" "server-administrators"}
09:27:05.449 INFO  [OperatrScheduler_Worker-3] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 440 ms
09:28:05.250 INFO  [OperatrScheduler_Worker-7] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 243 ms
09:29:05.350 INFO  [OperatrScheduler_Worker-1] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 343 ms
09:30:05.257 INFO  [OperatrScheduler_Worker-5] operatr.observe.kafka – R9AvkXfHTsCiI8g3E_5EVA: [38] kafka telemetry snapshots captured in 250 ms

See the Logback site for full information on configuration options like rolling file appenders.