PDI transformation steps

Steps extend and expand the functionality of Pentaho Data Integration (PDI) transformations. You can use the following steps in PDI.

Steps: A - F

Name	Category	Description
Abort	Flow	Abort a transformation.
Add a Checksum	Transform	Add a checksum column for each input row.
Add constants	Transform	Add one or more constants to the input rows.
Add sequence	Transform	Get the next value from a sequence.
Add value fields changing sequence	Transform	Add sequence depending of fields value change. Each time value of at least one field change, PDI will reset sequence.
Add XML	Transform	Encode several fields into an XML fragment.
AMQP Consumer	Streaming	Pull streaming data from an AMQP broker or clients through an AMQP transformation.
AMQP Producer	Streaming	Publish messages in near-real-time to an AMQP broker.
Analytic query	Statistics	Execute analytic queries over a sorted dataset (LEAD/LAG/FIRST/LAST).
Annotate stream	Flow	Refine your data for the Streamlined Data Refinery by creating measures, link dimensions, or attributes on stream fields.
Append streams	Flow	Append two streams in an ordered way.
ARFF output	Data Mining	Write data in ARFF format to a file.
Automatic Documentation Output	Output	Generate documentation automatically based on input in the form of a list of transformations and jobs.
Avro Input	Big Data	Decode binary or JSON Avro data and extracts fields from the structure it defines, either from flat files or incoming fields.
Avro input (deprecated)	Deprecated	Replaced by Avro Input.
Avro Output	Big Data	Serialize data into Avro binary or JSON format from the PDI data stream, then writes it to file.
Block this step until steps finish	Flow	Block this step until selected steps finish.
Blocking step	Flow	Block flow until all incoming rows have been processed. Subsequent steps only receive the last input row to this step.
Calculator	Transform	Create new fields by performing simple calculations.
Call DB Procedure	Lookup	Get back information by calling a database procedure.
Call Endpoint	Pentaho Server	Call API endpoints from the Pentaho Server within a PDI transformation.
Cassandra Input	Big Data	Read from a Cassandra column family.
Cassandra Output	Big Data	Write to a Cassandra column family.
Catalog Input	Catalog	Reads the payload of a resource registered with the Pentaho Data Catalog and outputs the payload in the form of rows to be used by a transformation.
Catalog Output	Catalog	Creates a new Pentaho Data Catalog resource or updates an existing resource and saves the metadata associated with this resource.
Change file encoding	Utility	Change file encoding and create a new file.
Check if a column exists	Lookup	Check if a column exists in a table on a specified connection.
Check if file is locked	Lookup	Check if a file is locked by another process.
Check if webservice is available	Lookup	Check if a webservice is available.
Clone row	Utility	Clone a row as many times as needed.
Closure Generator	Transform	Generate a closure table using parent-child relationships.
Combination lookup/update	Data Warehouse	Update a junk dimension in a data warehouse. Alternatively, look up information in this dimension. The primary key of a junk dimension are all the fields.
Concat Fields	Transform	Concatenate multiple fields into one target field. The fields can be separated by a separator and the enclosure logic is completely compatible with the Text File Output step.
Copybook Input	Input	Reads binary data files that are mapped by a fixed-length COBOL copybook definition file.
Copy rows to result	Job	Write rows to the executing job. The information will then be passed to the next entry in this job.
CouchDB Input	Big Data	Retrieve all documents from a given view in a given design document from a given database.
Credit card validator	Validation	Determines if a credit card number is valid (uses LUHN10 (MOD-10) algorithm), and which credit card vendor handles that number (VISA, MasterCard, Diners Club, EnRoute, American Express (AMEX),...).
CSV File Input	Input	Read from a simple CSV file input.
Data Grid	Input	Enter rows of static data in a grid, usually for testing, reference or demo purpose.
Data Validator	Validation	Validates passing data based on a set of rules.
Database join	Lookup	Execute a database query using stream values as parameters.
Database lookup	Lookup	Look up values in a database using field values.
De-serialize from file	Input	Read rows of data from a data cube.
Delay row	Utility	Output each input row after a delay.
Delete	Output	Permanently removes a row from a database.
Detect empty stream	Flow	Output one empty row if input stream is empty (I.e. when input stream does not contain any row).
Dimension lookup/update	Data Warehouse	Update a slowly changing dimension in a data warehouse. Alternatively, look up information in this dimension.
Dummy (do nothing)	Flow	Does not do anything. It is useful, however, when testing things or in certain situations where you want to split streams.
Dynamic SQL row	Lookup	Execute dynamic SQL statement build in a previous field.
Edi to XML	Utility	Convert an Edifact message to XML to simplify data extraction.
ElasticSearch Bulk Insert (deprecated)	Deprecated	Replaced by Elasticsearch REST Bulk Insert.
Elasticsearch REST Bulk Insert	Bulk loading	Perform bulk inserts into Elasticsearch.
Email messages input	Input	Read POP3/IMAP server and retrieve messages.
ESRI Shapefile Reader	Input	Read shape file data from an ESRI shape file and linked DBF file.
ETL metadata injection	Flow	Inject metadata into an existing transformation prior to execution. This allows for the creation of dynamic and highly flexible data integration solutions.
Example step (deprecated)	Deprecated	Is an example of a plugin test step.
Execute a process	Utility	Execute a process and return the result.
Execute Row SQL Script	Scripting	Execute an SQL statement or file for every input row.
Execute SQL Script	Scripting	Execute an SQL script, optionally parameterized using input rows.
File exists (Step)	Lookup	Check if a file exists.
Filter Rows	Flow	Filter rows using simple equations.
Fixed file input	Input	Read from a fixed file input.
Formula	Scripting	Calculate a formula using Pentaho's libformula.
Fuzzy match	Lookup	Find the approximate matches to a string using matching algorithms. Read a field from a main stream and output approximative value from lookup stream.

Steps: G - L

Name	Category	Description
Generate random credit card numbers	Input	Generate random valid (luhn check) credit card numbers.
Generate random value	Input	Generate random value.
Generate Rows	Input	Generate a number of empty or equal rows.
Get data from XML	Input	Get data from XML file by using XPath. This step also allows you to parse XML defined in a previous field.
Get File Names	Input	Get file names from the operating system and send them to the next step.
Get files from result	Job	Read filenames used or generated in a previous entry in a job.
Get Files Rows Count	Input	Get files rows count.
Get ID from slave server	Transform	Retrieve unique IDs in blocks from a slave server. The referenced sequence needs to be configured on the slave server in the XML configuration file.
Get records from stream	Streaming	Return records that were previously generated by another transformation in a job.
Get repository names	Input	List detailed information about transformations and/or jobs in a repository.
Get rows from result	Job	Read rows from a previous entry in a job.
Get Session Variables	Pentaho Server	Retrieve the value of a session variable.
Get SubFolder names	Input	Read a parent folder and return all subfolders.
Get System Info	Input	Get information from the system like system date, arguments, etc.
Get table names	Input	Get table names from database connection and send them to the next step.
Get Variables	Job	Determine the values of certain (environment or Kettle) variables and put them in field values.
Google Analytics	Input	Fetch data from google analytics account.
Greenplum Bulk Loader (deprecated)	Deprecated	Bulk load Greenplum data. Replacement step is Greenplum Load.
Greenplum Load	Bulk loading	Bulk load Greenplum data.
Group By	Statistics	Build aggregates in a group by fashion. This works only on a sorted input. If the input is not sorted, only double consecutive rows are handled correctly.
GZIP CSV Input	Input	Read in parallel from a GZIP CSV file.
Hadoop File Input	Big Data	Read data from a variety of different text-file types stored on a Hadoop cluster.
Hadoop File Output	Big Data	Write data to a variety of different text-file types stored on a Hadoop cluster.
HBase Input	Big Data	Read from an HBase column family.
HBase Output	Big Data	Write to an HBase column family.
HBase row decoder	Big Data	Decodes an incoming key and HBase result object to a mapping.
HL7 Input	Input	Read data from HL7 data streams.
HTTP client	Lookup	Call a web service over HTTP by supplying a base URL by allowing parameters to be set dynamically.
HTTP Post	Lookup	Call a web service request over HTTP by supplying a base URL by allowing parameters to be set dynamically.
IBM WebSphere MQ Consumer (deprecated)	Deprecated	Receive messages from any IBM WebSphere MQ Server.
IBM WebSphere MQ Producer (deprecated)	Deprecated	Send messages to any IBM WebSphere MQ Server.
Identify last row in a stream	Flow	Mark the last row.
If field value is null	Utility	Set a field value to a constant if it is null.
Infobright Loader	Bulk loading	Load data to an Infobright database table.
Ingres VectorWise Bulk Loader	Bulk loading	Interface with the Ingres VectorWise Bulk Loader "COPY TABLE" command.
Injector	Inline	Inject rows into the transformation through the java API.
Insert / Update	Output	Update or insert rows in a database based upon keys.
Java Filter	Flow	Filter rows using java code.
JMS Consumer	Streaming	Receive messages from a JMS server.
JMS consumer (deprecated)	Deprecated	Replaced by JMS Consumer.
JMS Producer	Streaming	Send messages to a JMS server.
JMS producer (deprecated)	Deprecated	Replaced by JMS Producer.
Job Executor	Flow	Run a PDI job, and passes parameters and rows.
Join Rows (cartesian product)	Joins	Output the cartesian product of the input streams. The number of rows is the multiplication of the number of rows in the input streams.
JSON Input	Input	Extract relevant portions out of JSON structures (file or incoming field) and output rows.
JSON output	Output	Create JSON block and output it in a field to a file.
Kafka consumer	Streaming	Run a sub-transformation that executes according to message batch size or duration, letting you process a continuous stream of records in near-real-time.
Kafka Producer	Streaming	Publish messages in near-real-time across worker nodes where multiple, subscribed members have access.
Kinesis Consumer	Streaming	Extract data from a specific stream located within the Amazon Kinesis Data Streams service.
Kinesis Producer	Streaming	Push data to an existing region and stream located within the Amazon Kinesis Data Streams service.
Knowledge Flow	Data Mining	Executes a Knowledge Flow data mining process.
LDAP Input	Input	Read data from LDAP host.
LDAP Output	Output	Perform Insert, upsert, update, add or delete operations on records based on their DN (Distinguished Name).
LDIF Input	Input	Read data from LDIF files.
Load file content in memory	Input	Load file content in memory.
LucidDB streaming loader (deprecated)	Deprecated	Load data into LucidDB by using Remote Rows UDX.

Steps: M - R

Name	Category	Description
Mail	Utility	Send e-mail.
Mail Validator	Validation	Check if an email address is valid.
Mapping	Mapping	Run a mapping (sub-transformation), use MappingInput and MappingOutput to specify the fields interface.
Mapping Input Specification	Mapping	Specify the input interface of a mapping.
Mapping Output Specification	Mapping	Specify the output interface of a mapping.
MapReduce Input	Big Data	Enter Key Value pairs from Hadoop MapReduce.
MapReduce Output	Big Data	Exit Key Value pairs, then push into Hadoop MapReduce.
MaxMind GeoIP Lookup	Lookup	Lookup an IPv4 address in a MaxMind database and add fields such as geography, ISP, or organization.
Memory Group By	Statistics	Build aggregates in a group by fashion. This step doesn't require sorted input.
Merge Join	Joins	Join two streams on a given key and outputs a joined set. The input streams must be sorted on the join key.
Merge Rows (diff)	Joins	Merge two streams of rows, sorted on a certain key. The two streams are compared and the equals, changed, deleted and new rows are flagged.
Metadata structure of stream	Utility	Read the metadata of the incoming stream.
Microsoft Access Input	Input	Read data from a Microsoft Access file
Microsoft Access Output	Output	Store records into an MS-Access database table.
Microsoft Excel Input	Input	Read data from Excel and OpenOffice Workbooks (XLS, XLSX, ODS).
Microsoft Excel Output	Output	Store records into an Excel (XLS) document with formatting information.
Microsoft Excel Writer	Output	Write or appends data to an Excel file.
Modified Java Script Value	Scripting	Run JavaScript programs (and much more).
Mondrian Input	Input	Execute and retrieve data using an MDX query against a Pentaho Analyses OLAP server (Mondrian).
MonetDB Agile Mart	Agile
MonetDB Bulk Loader	Bulk loading	Load data into MonetDB by using their bulk load command in streaming mode.
MongoDB Input	Big Data	Read all entries from a MongoDB collection in the specified database.
MongoDB Output	Big Data	Write to a MongoDB collection.
MQTT Consumer	Streaming	Pull streaming data from an MQTT broker or clients through an MQTT transformation.
MQTT Producer	Streaming	Publish messages in near-real-time to an MQTT broker.
Multiway Merge Join	Joins	Join multiple streams. This step supports INNER and FULL OUTER joins.
MySQL Bulk Loader	Bulk loading	Load data over a named pipe (not available on MS Windows).
Null if...	Utility	Set a field value to null if it is equal to a constant value.
Number range	Transform	Create ranges based on numeric field.
OLAP Input	Input	Execute and retrieve data using an MDX query against any XML/A OLAP datasource using olap4j.
OpenERP object delete (deprecated)	Deprecated	Delete data from the OpenERP server using the XMLRPC interface with the 'unlink' function.
OpenERP object input (deprecated)	Deprecated	Retrieve data from the OpenERP server using the XMLRPC interface with the 'read' function.
OpenERP object output (deprecated)	Deprecated	Update data on the OpenERP server using the XMLRPC interface and the 'import' function
Oracle Bulk Loader	Bulk loading	Use Oracle Bulk Loader to load data.
ORC Input	Big Data	Read fields data from ORC files into a PDI data stream.
ORC Output	Big Data	Serialize data from the PDI data stream into an ORC file format and writes it to a file.
Output steps metrics	Statistics	Return metrics for one or several steps.
Palo cell input (deprecated)	Deprecated	Retrieve all cell data from a Palo cube.
Palo cell output (deprecated)	Deprecated	Update cell data in a Palo cube.
Palo dim input (deprecated)	Deprecated	Return elements from a dimension in a Palo database.
Palo dim output (deprecated)	Deprecated	Create/update dimension elements and element consolidations in a Palo database.
Parquet Input	Big Data	Decode Parquet data formats and extracts fields from the structure it defines.
Parquet Output	Big Data	Map fields within data files and choose where you want to process those files.
Pentaho Reporting Output	Output	Execute an existing report file (.prpt).
PostgreSQL Bulk Loader	Bulk loading	Bulk load PostgreSQL data.
Prioritize streams	Flow	Prioritize streams in an order way.
Process files	Utility	Process one file per row (copy or move or delete). This step only accept filename in input.
Properties Output	Output	Write data to properties file.
Property Input	Input	Read data (key, value) from properties files.
Python Executor	Scripting	Map upstream data from a PDI input step or execute a Python script to generate data. When you send all rows, Python stores the dataset in a variable that kicks off your Python script.
Query HCP	Input	Uses the Metadata Query Engine (MQE) to query your Hitachi Content Platform (HCP) repository for objects, their URLs, and system metadata properties.
R script executor	Statistics	Execute an R script within a PDI transformation.
Read metadata from HCP	Input	Identifies an HCP object by its URL path then specifies a target annotation name to read.
Read metadata from Copybook	Input	Reads a binary fixed-length copybook definition file and outputs the file and column descriptor information as fields to PDI rows.
Read metadata	Catalog	Reads metadata from the Pentaho Data Catalog (PDC).
Regex Evaluation	Scripting	Evaluate regular expressions. This step uses a regular expression to evaluate a field. It can also extract new fields out of an existing field with capturing groups.
Replace in String	Transform	Replace all occurrences a word in a string with another word.
Reservoir Sampling	Statistics	Transform Samples a fixed number of rows from the incoming stream.
REST Client	Lookup	Consume RESTful services. REpresentational State Transfer (REST) is a key design idiom that embraces a stateless client-server architecture in which the web services are viewed as resources and can be identified by their URLs
Row Denormaliser	Transform	Denormalise rows by looking up key-value pairs and by assigning them to new fields in the output rows. This method aggregates and needs the input rows to be sorted on the grouping fields.
Row Flattener	Transform	Flatten consecutive rows based on the order in which they appear in the input stream.
Row Normaliser	Transform	Normalise de-normalised information.
RSS Input	Input	Read RSS feeds.
RSS Output	Output	Read RSS stream.
Rule Executor	Scripting	Execute a rule against each row (using Drools).
Rule Accumulator	Scripting	Execute a rule against a set of rows (using Drools).
Run SSH commands	Utility	Run SSH commands and returns result.

Steps: S - Z

Name	Category	Description
S3 CSV Input	Input	Read from an S3 CSV file.
S3 File Output	Output	Export data to a text file on an Amazon Simple Storage Service (S3).
Salesforce Delete	Output	Delete records in a Salesforce module.
Salesforce Input	Input	Read information from Salesforce.
Salesforce Insert	Output	Insert records in a Salesforce module.
Salesforce Update	Output	Update records in a Salesforce module.
Salesforce Upsert	Output	Insert or update records in a Salesforce module.
Sample rows	Statistics	Filter rows based on the line number.
SAP input (deprecated)