Kinesis Producer

The Kinesis Producer step pushes data from a PDI transformation to an existing stream in a specific region within the Amazon Kinesis Data Streams (KDS) service. You can use this step to publish a stream of data to KDS for storage in S3, Redshift, or EMR HDFS. For more information about the Amazon Kinesis Data Streams protocol, see https://aws.amazon.com/kinesis/.

For the Kinesis Producer step, configure the connection to Amazon and specify the target Amazon region and KDS stream. Only one region can be selected.

Step name

• Step name: Specifies the unique name of the Kinesis Producer step on the canvas. Default: Kinesis Producer.

Options

The Kinesis Producer step includes the following tabs.

• Setup

• Options

Setup tab

In this tab, define the connection to the Amazon region, the target stream in the Amazon Kinesis Data Streams (KDS) service, the partition key field, and the message field.

Kinesis Producer setup tab

Option	Description
Region	Specify the Amazon geographical region where the stream is located. You can select only one region.
Stream name	Specify the Amazon KDS stream name by using one of the following methods: Specify stream name Specify the Stream name to which the data will be published. Get data from field Specify the Field name from another step that is generating data in the transformation stream. Using the drop-down list, select the name of the field to use.
Partition key field	Select the PDI field that contains the partition key. The partition key is used to group records into shards within the stream.
Message field	Select the PDI field that contains data to write to the stream.

Options tab

In this tab, define the Value for the following write and connection timeout settings.

Kinesis Producer options tab

Name	Value
Write Timeout Seconds	Specify a timeout value (in seconds) for the Kinesis Producer step to wait for the next time to write to the KDS stream. Default: 30.
Connection Timeout Seconds	Specify a timeout value (in seconds) for the Kinesis Producer step to wait while trying to connect to the server. Default: 2.
Connection Acquisition Timeout Seconds	Specify a timeout value (in seconds) for the Kinesis Producer step to wait while trying to connect after the initial connection is made. Default: 10.

Metadata injection support

This step supports metadata injection. You can use it with ETL metadata injection to pass metadata to your transformation at runtime.

See also

• Kinesis consumer