Kafka Offset

You can use the Kafka Offset job entry to reset the offset value of all partitions in a topic based on a given timestamp or epoch value.

General

• Entry name: Specify the unique name of the Kafka Offset entry on the canvas. You can customize the name or leave it as the default.

Options

The Kafka Offset entry includes three tabs: Setup, Options, and Offset settings.

Setup tab

Setup tab for Kafka Offset job

Use this tab to define connections, topics, and the consumer group.

Option	Description
Bootstrap servers	Specify the bootstrap servers to use for the Kafka connection.
Topics	Enter the name of each Kafka topic you want to target. You must include all relevant topics.
Consumer group	Enter the consumer group name. When part of a consumer group, each consumer is assigned a subset of partitions from the subscribed topics. Each Kafka consumer instance runs a single consumer thread.

Options tab

Options tab for Kafka Offset job

Use this tab to send parameters (properties) to the broker when connecting. Parameter values can be encrypted.

A few common options are included for convenience, but you can enter any Kafka property.

For more information, see the Apache Kafka documentation: https://kafka.apache.org/documentation/.

Offset settings tab

To reset the offset value, set the start date in Offset timestamp and select the format in Timestamp format.

Offset settings tab for Kafka Offset job

Option	Description
Offset timestamp	The start timestamp to use as the offset. The timestamp can be in the past or future. When you run the job entry, each partition offset is reset to the timestamp provided.
Timestamp format	The timestamp format for the offset. If you provide an epoch value, you do not need to specify a timestamp format.

Examples

The kafka_job_abort.kjb sample job in design-tools/data-integration/plugins/kafka-job-plugins-zip-9.5.1.0-110/kafka-offset-job/samples/transformations demonstrates this job entry by using the Abort step.

Kafka Offset example kafka_job_abort

The kafka_job.kjb sample job in design-tools/data-integration/plugins/pentaho-streaming-kafka-plugin-zip-9.5.1.0-110/pentaho-streaming-kafka-plugin/samples/transformations demonstrates a job with both successes and failures. Comments in the transformations explain how the steps are used.

Kafka offset example kafka_job.kjb