Unique Rows

The Unique Rows step removes duplicate rows from the input stream and outputs only unique rows.

The input stream must be sorted before the Unique Rows step. Otherwise, only consecutive duplicate rows are correctly analyzed and filtered.

If you cannot sort the input stream, use Unique Rows (HashSet).

Step name: Specify the unique name of the step on the canvas. You can customize the name or leave it as the default.

Option

Description

Add counter to output?

Select to add a counter field to the output stream.

Counter field

Name of the counter field.

Redirect duplicate row

Select to send duplicate rows to the step error stream. If cleared, duplicates are discarded.

Error description

Error message used when redirecting duplicate rows. Available only when Redirect duplicate row is selected.

Fields to compare table

Fields used to determine uniqueness.

Select Get to insert all incoming fields.

If you do not specify any fields, the step compares the entire row.

Ignore case

Set to Y to ignore case when comparing strings.

For example, Kettle, KETTLE, and kettle are treated as equivalent when Ignore case is Y.