Getting Started
Configuration
How to configure Kpow
Your options for configuring and deploying Kpow Community edition:
- Run the Docker container with no arguments and use the Kpow Setup Wizard UI
- Run the Docker container with environment variables configuring your Kafka resources
- Use the Kpow Helm charts to run Kpow Community Edition in Kubernetes
Kpow Setup Wizard UI
If you provide no configuration to Kpow, you will be prompted with a wizard to configure your Kafka resources when you launch Kpow.
If you are using the suggested docker run command (below) the UI can be accessed at http://localhost:3000
Kpow environment variables
You can use environment variables to configure Kpow and avoid the wizard UI.
For example, you can create a kpow.env file:
# Name this Kpow Installation
ENVIRONMENT_NAME=Kpow Community Demo
# Configure Kafka cluster
BOOTSTRAP=bootstrap-url:9092
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="USERNAME" password="PASSWORD";
# Configure Schema Registry
SCHEMA_REGISTRY_URL=https://your-schema-url
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=your-schema-username
SCHEMA_REGISTRY_PASSWORD=your-schema-password
# Configure Connect
CONNECT_NAME=Connect Cluster
CONNECT_REST_URL=https://connect-cluster-url
CONNECT_AUTH=BASIC
CONNECT_BASIC_AUTH_USER=CONNECT_USERNAME
CONNECT_BASIC_AUTH_PASS=CONNECT_PASSWORD
# The license details from your community sign-up email
LICENSE_ID=b2b41e30-a401-4b70-8a36-a830581a85d5
LICENSE_CODE=KPOW_COMMUNITY_ORG
LICENSEE=Factor House
LICENSE_EXPIRY=2024-06-20
LICENSE_SIGNATURE=683013F8BAC1D61560FC0C40C2B340..
And start Kpow like so:
docker run --pull=always -p 3000:3000 --env-file ./kpow.env -m 2G factorhouse/kpow-ce:latest
Confluent Cloud example configuration
Kpow Community fully supports Confluent Cloud, including integration with the metrics API and Confluent Managed Connect and Schema resources.
Example ccloud.env configuration file:
# Name this Kpow Installation
ENVIRONMENT_NAME=Kpow Community Demo
# Configure Kafka cluster
BOOTSTRAP=pkc-abcd1.us-east-2.aws.confluent.cloud:9092
SECURITY_PROTOCOL=SASL_SSL
SASL_MECHANISM=PLAIN
SASL_JAAS_CONFIG=org.apache.kafka.common.security.plain.PlainLoginModule required username="your-username" password="your-password";
SSL_ENDPOINT_IDENTIFICATION_ALGORITHM=https
# Configure Schema Registry
SCHEMA_REGISTRY_URL=https://psrc-abcd2.us-east-2.aws.confluent.cloud
SCHEMA_REGISTRY_AUTH=USER_INFO
SCHEMA_REGISTRY_USER=your-schema-username
SCHEMA_REGISTRY_PASSWORD=your-schema-password
# Configure Connect Cluster
CONNECT_REST_URL=https://api.confluent.cloud/connect/v1/environments/your-env-id/clusters/your-cluster-id
CONNECT_AUTH=BASIC
CONNECT_BASIC_AUTH_USER=your-connect-username
CONNECT_BASIC_AUTH_PASS=your-connect-password
# Configure Confluent API Key+Secret for enhanced metrics
CONFLUENT_API_KEY=your-confluent-api-key
CONFLUENT_API_SECRET=your-confluent-api-secret
# The license details from your community sign-up email
LICENSE_ID=b2b41e30-a401-4b70-8a36-a830581a85d5
LICENSE_CODE=KPOW_COMMUNITY_ORG
LICENSEE=Factor House
LICENSE_EXPIRY=2024-06-20
LICENSE_SIGNATURE=683013F8BAC1D61560FC0C40C2B340..
We can use that ccloud.env config file like so:
docker run --pull=always -p 3000:3000 -m 2G --env-file ./ccloud.env factorhouse/kpow-ce:latest
For more information on the optional, additional api-key for integration with the Confluent Metrics API, see: Confluent Cloud.
Evaluate Kpow with Docker Compose
Our kpow-local GitHub repository contains a reference configuration of Kpow, Kafka, Schema, and Connect.
Simply check out the project, run Docker Compose to start all the resources, and evaluate Kpow Community Edition locally.
Environment Variable Reference
Kafka cluster
Kpow supports all standard Kafka connection properties.
| Environment Variable | Required? | Description |
|---|---|---|
| BOOTSTRAP | true | The Kafka cluster bootstrap URL |
| SECURITY_PROTOCOL | false | Specify the connection security protocol, acceptable values are: PLAINTEXT, SASL_PLAINTEXT, SASL_SSL, or SSL |
| SASL_MECHANISM | false | SASL mechanism used for client connections. This may be any mechanism for which a security provider is available. GSSAPI is the default mechanism. |
| SASL_JAAS_CONFIG | false | JAAS login context parameters for SASL connections in the format used by JAAS configuration files. JAAS configuration file format is described here. The format for the value is: 'loginModuleClass controlFlag (optionName=optionValue)*;'. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.jaas.config=com.example.ScramLoginModule required; |
| SASL_CLIENT_CALLBACK_HANDLER_CLASS | false | The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface. |
| SASL_KERBEROS_SERVICE_NAME | false | The Kerberos principal name that Kafka runs as. This can be defined either in Kafka's JAAS config or in Kafka's config. |
| SASL_LOGIN_CALLBACK_HANDLER_CLASS | false | The fully qualified name of a SASL login callback handler class that implements the AuthenticateCallbackHandler interface. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.callback.handler.class=com.example.CustomScramLoginCallbackHandler |
| SASL_LOGIN_CLASS | false | The fully qualified name of a class that implements the Login interface. For brokers, login config must be prefixed with listener prefix and SASL mechanism name in lower-case. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin |
| SASL_OAUTHBEARER_JWKS_ENDPOINT_URL | false | - |
| SASL_OAUTHBEARER_TOKEN_ENDPOINT_URL | false | - |
| SASL_KERBEROS_KINIT_CMD | false | Kerberos kinit command path. |
| SASL_KERBEROS_MIN_TIME_BEFORE_RELOGIN | false | Login thread sleep time between refresh attempts. |
| SASL_KERBEROS_TICKET_RENEW_JITTER | false | Percentage of random jitter added to the renewal time. |
| SASL_KERBEROS_TICKET_RENEW_WINDOW_FACTOR | false | Login thread will sleep until the specified window factor of time from last refresh to ticket's expiry has been reached, at which time it will try to renew the ticket. |
| SASL_LOGIN_CONNECT_TIMEOUT_MS | false | - |
| SASL_LOGIN_READ_TIMEOUT_MS | false | - |
| SASL_LOGIN_REFRESH_BUFFER_SECONDS | false | The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. If a refresh would otherwise occur closer to expiration than the number of buffer seconds then the refresh will be moved up to maintain as much of the buffer time as possible. Legal values are between 0 and 3600 (1 hour); a default value of 300 (5 minutes) is used if no value is specified. This value and sasl.login.refresh.min.period.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER. |
| SASL_LOGIN_REFRESH_MIN_PERIOD_SECONDS | false | The desired minimum time for the login refresh thread to wait before refreshing a credential, in seconds. Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. This value and sasl.login.refresh.buffer.seconds are both ignored if their sum exceeds the remaining lifetime of a credential. Currently applies only to OAUTHBEARER. |
| SASL_LOGIN_REFRESH_WINDOW_FACTOR | false | Login refresh thread will sleep until the specified window factor relative to the credential's lifetime has been reached, at which time it will try to refresh the credential. Legal values are between 0.5 (50%) and 1.0 (100%) inclusive; a default value of 0.8 (80%) is used if no value is specified. Currently applies only to OAUTHBEARER. |
| SASL_LOGIN_REFRESH_WINDOW_JITTER | false | The maximum amount of random jitter relative to the credential's lifetime that is added to the login refresh thread's sleep time. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. Currently applies only to OAUTHBEARER. |
| SASL_LOGIN_RETRY_BACKOFF_MAX_MS | false | - |
| SASL_LOGIN_RETRY_BACKOFF_MS | false | - |
| SASL_OAUTHBEARER_CLOCK_SKEW_SECONDS | false | - |
| SASL_OAUTHBEARER_EXPECTED_AUDIENCE | false | - |
| SASL_OAUTHBEARER_EXPECTED_ISSUER | false | - |
| SASL_OAUTHBEARER_JWKS_ENDPOINT_REFRESH_MS | false | - |
| SASL_OAUTHBEARER_JWKS_ENDPOINT_RETRY_BACKOFF_MAX_MS | false | - |
| SASL_OAUTHBEARER_JWKS_ENDPOINT_RETRY_BACKOFF_MS | false | - |
| SASL_OAUTHBEARER_SCOPE_CLAIM_NAME | false | - |
| SASL_OAUTHBEARER_SUB_CLAIM_NAME | false | - |
| SSL_KEY_PASSWORD | false | The password of the private key in the key store file. This is optional for client. |
| SSL_KEYSTORE_CERTIFICATE_CHAIN | false | - |
| SSL_KEYSTORE_KEY | false | - |
| SSL_KEYSTORE_LOCATION | false | The location of the key store file. This is optional for client and can be used for two-way authentication for client. |
| SSL_KEYSTORE_PASSWORD | false | The store password for the key store file. This is optional for client and only needed if ssl.keystore.location is configured. |
| SSL_TRUSTSTORE_CERTIFICATES | false | - |
| SSL_TRUSTSTORE_LOCATION | false | The location of the trust store file. |
| SSL_TRUSTSTORE_PASSWORD | false | The password for the trust store file. If a password is not set access to the truststore is still available, but integrity checking is disabled. |
| SSL_ENABLED_PROTOCOLS | false | The list of protocols enabled for SSL connections. |
| SSL_KEYSTORE_TYPE | false | The file format of the key store file. This is optional for client. |
| SSL_PROTOCOL | false | The SSL protocol used to generate the SSLContext. Default setting is TLSv1.2, which is fine for most cases. Allowed values in recent JVMs are TLSv1.2 and TLSv1.3. TLS, TLSv1.1, SSL, SSLv2 and SSLv3 may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. |
| SSL_PROVIDER | false | The name of the security provider used for SSL connections. Default value is the default security provider of the JVM. |
| SSL_TRUSTSTORE_TYPE | false | The file format of the trust store file. |
| SSL_CIPHER_SUITES | false | A list of cipher suites. This is a named combination of authentication, encryption, MAC and key exchange algorithm used to negotiate the security settings for a network connection using TLS or SSL network protocol. By default all the available cipher suites are supported. |
| SSL_ENDPOINT_IDENTIFICATION_ALGORITHM | false | The endpoint identification algorithm to validate server hostname using server certificate. |
| SSL_ENGINE_FACTORY_CLASS | false | - |
| SSL_KEYMANAGER_ALGORITHM | false | The algorithm used by key manager factory for SSL connections. Default value is the key manager factory algorithm configured for the Java Virtual Machine. |
| SSL_SECURE_RANDOM_IMPLEMENTATION | false | The SecureRandom PRNG implementation to use for SSL cryptography operations. |
| SSL_TRUSTMANAGER_ALGORITHM | false | The algorithm used by trust manager factory for SSL connections. Default value is the trust manager factory algorithm configured for the Java Virtual Machine. |
Kafka Connect
Open Source Connect
Kpow supports connections to Kafka Connect
| Environment Variable | Required? | Description |
|---|---|---|
| CONNECT_REST_URL | true | The rest endpoint of the Connect instance |
| CONNECT_AUTH | false | An enum for the authentication method. Set to BASIC if the Connect instance is using basic authentication. |
| CONNECT_AUTH_USER | false | The basic auth username |
| CONNECT_AUTH_PASSWORD | false | The basic auth password |
| CONNECT_PERMISSIVE_SSL | false | Whether to allow insecure https connections (default: false) |
MSK Connect
Kpow supports connections to MSK Connect
Note: you will also need to pass through the appropriate IAM permissions to the Kpow container.
| Environment Variable | Required? | Description |
|---|---|---|
| CONNECT_AWS_REGION | true | The AWS region your MSK Connect instance resides in. |
Schema Registry
Confluent Schema Registry
Kpow supports connections to Confluent Schema Registry
| Environment Variable | Required? | Description |
|---|---|---|
| SCHEMA_REGISTRY_URL | true | The REST endpoint of the Schema Registry. |
| SCHEMA_REGISTRY_AUTH | false | Specifies the value for basic.auth.credentials.source (eg, USER_INFO) |
| SCHEMA_REGISTRY_USER | false | If auth set to USER_INFO, the basic auth username |
| SCHEMA_REGISTRY_PASSWORD | false | If auth set to USER_INFO, the basic auth password |
AWS Glue
Kpow supports connections to AWS Glue Schema Registry
Note: you will also need to pass through the appropriate IAM permissions to the Kpow container.
| Environment Variable | Required? | Description |
|---|---|---|
| SCHEMA_REGISTRY_ARN | true | The ARN endpoint of the Schema Registry. |
| SCHEMA_REGISTRY_REGION | false | The AWS region the Schema Registry instance resides in. |