Serdes

The Ultimate Engineering Toolkit for Apache Kafka®
Search…
Serdes
Configuration of serdes for data inspect
By default, kPow ships with the following serdes available from data inspect:
JSON
JSON Schema
AVRO
Protobuf
String
EDN
Double
Float
Integer
Long
Short
Transit / JSON
Transit / JSON-Verbose
Transit / Msgpack
AVRO, JSON Schema, and Protobuf
Schema Registry serdes in kPow follow the wire format documented here. 
See: Custom Serdes to use kPow with Protobuf messages without a Schema Registry.
kPow integrates with Confluent Schema Registry and allows for AVRO, JSON Schema and Protobuf serdes to be used in data inspect.
See Schema Registry for more on how to configure Confluent Schema Registry with kPow.
Once configured, from within the data inspect UI you will now be able to select the schema and subject strategy when searching for records by key:
Configuring Serdes
kPow offers some configuration on how serdes are presented in the UI.
Default Serdes
Set DEFAULT_KEY_SERDES or DEFAULT_VALUE_SERDES to specify which serdes should be selected from the dropdown by default when using data inspect. 
Available Serdes
To restrict the serdes available to your users set AVAILABLE_KEY_SERDES or AVAILABLE_VALUE_SERDES

Eg: AVAILABLE_VALUE_SERDES=JSON,AVRO to only ever show JSON or AVRO serdes from within kPow's UI
Custom Serdes
kPow works with custom org.apache.kafka.common.serialization.Serdes<String>
Why <String>?
kPow expects your custom serdes to provide String key/value output when deserializing messages and will interpret that String key/value output according to the serdes format you configure.
kPow will provide your custom serdes with String input when your users select a custom serdes to produce data to a topic, it is up to you how you interpret and encode that data for serialization.
Serdes with json or clojure format will have Data Policies applied and can be used with kJQ.
Setup
Provide your custom serdes on the kPow classpath (see How to add a Jar to the Classpath).
Use a plain Java command similar to below to start kpow with a set classpath, note operatr.kpow.
java -Xmx2G -cp /opt/kpow/lib/kpow.jar:/opt/org/custom-serdes.jar operatr.kpow
Configure kPow with the CUSTOM_SERDES environment variable:
1
# CUSTOM_SERDES accepts a comma-separated list of serdes classes
2
​
3
CUSTOM_SERDES=org.corp.XMLSerde,org.corp.MyCustomSerde
Copied!
Configuration
Include a YAML configuration file in your jar file to configure custom serdes further (Optional).
E.g. org.corp.XMLSerde can be configured with org/corp/XMLSerde.yml
The following fields are available to configure your serdes:
name - the display name to use in kPow's Data Inspect UI.
format - json, clojure or string (default)
isKey -  true for key data only, false for value data only, leave unset for both key/value data.
config - a map of config values passed into the serdes configure method 
Config is converted to String/String and passed to your Serdes/configure method. 
Config values starting with $ are resolved as environment variables. (e.g. $BOOTSTRAP below)
E.g. Single Serdes Configuration (json format, available to key and value fields).
1
name: PROTO
2
format: json
3
config:
4
  bootstrap: $BOOTSTRAP
5
  limit: 22
6
  display: another-value
7
  abc: $SOME_ENV
Copied!
E.g. Multi Serdes Configuration (json format, separate configured serdes for key and value fields)
1
serdes:
2
  - name: "PROTO 1"
3
    format: "json"
4
    isKey: true
5
    config:
6
      bootstrap: "some-value"
7
      limit: 22
8
      display: another-value
9
      abc: $SOME_ENV
10
  - name: "PROTO 2"
11
    format: "json"
12
    isKey: false
13
    config:
14
      bootstrap: "some-value"
15
      limit: "100"
16
      display: another-value
17
      abc: $ANOTHER_ENV
Copied!
Serdes default to String format and are available to keys and values where no config provided
On startup you will see details logged of each custom serdes:
1
INFO operatr.kafka.serdes – initializing custom serdes: kpow.serdes.CustomSerdesExample
2
INFO operatr.kafka.serdes – found kpow/serdes/CustomSerdesExample.yml
3
INFO operatr.kafka.serdes – serde configuration: Custom Serde!
4
WARN operatr.kafka.serdes – environment variable $SOME_ENV not set
5
INFO operatr.kafka.serdes – config 'Custom Serde!', isKey?: null, format: json, config: ("bootstrap" "limit" "display" "abc").
Copied!
kJQ Filters
Next - Features
Data Produce
Last modified 5mo ago
Copy link
Contents
AVRO, JSON Schema, and Protobuf
Configuring Serdes
Custom Serdes