Minimum ACL permissions

Kafka has the ability to restrict access to objects and operations within a cluster through the use of Kafka Access Control Lists (ACLs). This is different to Kpow's own Role Based Access Control.

You can skip this page if you do not have Kafka ACLs enabled in your cluster/s

When your cluster is secured with Kafka ACLs the Kafka user identified by the cluster connection credentials provided to Kpow will need to have a minimum level of permissions for Kpow to operate.

Kpow required permissions

If you run Kpow with the default PERSISTENCE_MODE, Kpow needs permission to read and write internal topics and read internal groups.

If you choose to enable specific Kpow features (e.g. TOPIC_PRODUCE), Kpow needs permission to action those features.

See: Environment variables for more details about Kpow's persistence modes.

You can reduce or remove Kpow's use of internal topis and groups by configuring PERSISTENCE_MODE=audit|none

Kpow checks for the following internal topics in your primary cluster (the first bootstrap in your configuration) on startup and will attempt to create them if required:

__oprtr_metric_pt1m
__oprtr_snapshot_state
__oprtr_audit_log
oprtr.compute.metrics.v2-oprtr_metric_v2_pt1m-changelog
oprtr.compute.snapshots.v2-oprtr_snaphot_state_v2-changelog

You can manually create these topics if you prefer, see Create Kpow topics.

Once started, Kpow creates two internal streaming compute applications:

oprtr.compute.metrics.v2
oprtr.compute.snapshots.v2

Minimum required permissions

At a minimum, Kpow must have permission to describe clusters, topics, configuration, and groups.

Depending on your configured PERSISTENCE_MODE, Kpow must be able to read and write to and from Kpow internal topics, must be able to read Kpow internal groups.

A basic set of Kafka ACLs that allows Kpow to operate provides ALLOW on the following:

Kafka Resource	Kafka ACL	Detail
Cluster	Describe	`*`
Cluster	DescribeConfigs	`*`
Cluster	Create	`* (if not manually creating kpow topics)`
Topic	Describe	`*`
Topic	DescribeConfigs	`*`
Topic	Read	`* or kpow topics only`
Topic	Write	`* or kpow topics only`
Group	Describe	`*`
Group	Read	`* or kpow groups only`

Kpow does not read from or write to topics other than internal ones as a part of normal operation.

Feature specific ACLS

The following ACLS are optional and only required if you intend to permit the associated Kpow action.

See User authorization for a description of Kpow User actions and Controls.

Kafka Resource	Kafka ACL	Required for user action
Cluster	Alter	`ACL_EDIT`
Cluster	AlterConfigs	`BROKER_EDIT`
Cluster	Create	`TOPIC_CREATE`
Topic	AlterConfigs	`TOPIC_EDIT`
Topic	Create	`TOPIC_CREATE`
Topic	Delete	`TOPIC_DELETE`, `TRUNCATE_TOPIC`
Topic	Read	`TOPIC_INSPECT`
Topic	Write	`TOPIC_PRODUCE`
Group	Read / Delete	`GROUP_EDIT`

Configuring Kafka ACLS

Creating ACLs on a cluster with no existing ACL configuration can cause issues.

Consult your cluster provider documentation first.

For example the Amazon MSK ACL Guide describes extra ACLs required to allow inter-broker replication, and suggests not to set CLUSTER level ACLs.

Create a file containing client configuration for a user who has permissions to create ACLs.

security.protocol=SASL_PLAINTEXT
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="admin" password="admin-secret";

Basic ACL creation script

Use the kafka-acls.sh script provided by Apache Kafka to create the basic set of ACLs described above. This allows Kpow to operate plus the ALTER CLUSTER ACL allowing Kpow to create and delete ACLs.

./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Describe --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation DescribeConfigs  --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Create --cluster '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Write --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Read --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Describe --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation DescribeConfigs --topic '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Describe --group '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Read --group '*'
./kafka-acls.sh --bootstrap-server 127.0.0.1:9092 --command-config client.conf --add --allow-principal User:kpow --operation Alter --cluster '*'

That set of ACLs can then be listed using kafka-acls.sh.

./kafka-acls.sh -bootstrap-server 127.0.0.1:9092 --command-config client.conf --list

Current ACLs for resource `ResourcePattern(resourceType=GROUP, name=*, patternType=LITERAL)`:
 	(principal=User:kpow, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=DESCRIBE, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=*, patternType=LITERAL)`:
 	(principal=User:kpow, host=*, operation=READ, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=WRITE, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)

Current ACLs for resource `ResourcePattern(resourceType=CLUSTER, name=kafka-cluster, patternType=LITERAL)`:
 	(principal=User:kpow, host=*, operation=DESCRIBE_CONFIGS, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=DESCRIBE, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=CREATE, permissionType=ALLOW)
	(principal=User:kpow, host=*, operation=ALTER, permissionType=ALLOW)

Create Kpow topics

Using the same client configuration file as above, the following script correctly creates the required internal Kpow topics. You can run this script and create these topics on the primary cluster in which you expect Kpow to keep data.

./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic __oprtr_audit_log \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=-1
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic __oprtr_metric_pt1m \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=7200000 \
         --config segment.ms=5400000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic __oprtr_snapshot_state \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config segment.bytes=104857600 \
         --config retention.ms=7200000 \
         --config message.timestamp.type=LogAppendTime \
         --config segment.ms=5400000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic oprtr.compute.metrics.v2-oprtr_metric_v2_pt1m-changelog \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config cleanup.policy=compact,delete \
         --config segment.bytes=52428800 \
         --config retention.ms=5400000 \
         --config message.timestamp.type=CreateTime \
         --config segment.ms=1800000
./kafka-topics.sh --create \
         --bootstrap-server 127.0.0.1:9092 \
         --command-config client.conf \
         --topic oprtr.compute.snapshots.v2-oprtr_snaphot_state_v2-changelog \
         --replication-factor 3 \
         --partitions 12 \
         --config compression.type=gzip \
         --config cleanup.policy=compact,delete \
         --config segment.bytes=52428800 \
         --config retention.ms=604800000 \
         --config message.timestamp.type=CreateTime \
         --config segment.ms=604800000