MQTT Consumer
The PDI client can pull streaming data from an MQTT broker or clients through an MQTT transformation. The parent MQTT Consumer step runs a child transformation that executes according to the message batch size or duration, allowing you to process a continuous stream of records in near real-time. The child transformation must start with the Get records from stream step.
Additionally, from the MQTT Consumer step, you can select a step in the child transformation to stream records back to the parent transformation. This capability allows records processed by an MQTT Consumer step in a parent transformation to be passed downstream to any other steps included within the same parent transformation.
Step name
Step name: Specifies the unique name of the step on the canvas. Default:
MQTT Consumer.
Transformation
Use Transformation to specify the child transformation to run.
You can specify the child transformation by doing any of the following:
Enter the transformation path.
Select Browse to select an existing child transformation.
Select New to create and save a new child transformation. For details, see Create and save a new child transformation.
The selected child transformation must start with the Get records from stream step.
If you select a transformation that has the same root path as the current transformation, the variable ${Internal.Entry.Current.Directory} is inserted automatically in place of the common root path.
Example:
Current transformation:
/home/admin/transformation.ktrSelected transformation:
/home/admin/path/sub.ktrConverted path:
${Internal.Entry.Current.Directory}/path/sub.ktr
If you are working with a repository, specify the name of the transformation. If you are not working with a repository, specify the XML file name of the transformation.
Transformations previously specified by reference are converted automatically to use the transformation name in the Pentaho Repository.
Create and save a new child transformation
If you do not already have a child transformation, you can create one while setting up the MQTT Consumer step.
When you select New, PDI generates the required Get records from stream step in a new canvas tab. The step’s fields and types match the Fields tab configuration in the parent MQTT Consumer step.
In the MQTT Consumer step, select New.
The Save As dialog box appears.
Navigate to the location where you want to save your new child transformation, then enter the file name.
Select Save.
A notification box informs you that the child transformation has been created and opened in a new tab. If you do not want to see this notification again, select Don't show me this again.
Select the new transformation tab to view and edit the child transformation.
It automatically contains the Get records from stream step. Optionally, you can continue to build this transformation and save it.
When finished, return to the MQTT Consumer step.
Options
The MQTT Consumer step includes the following tabs.
Setup
Security
Batch
Fields
Result fields
Options
Setup tab
In this tab, define the connections used for receiving messages, topics to which you want to subscribe, and the consumer group for the topics.
Connection
Specify the address of the MQTT server to which this step will connect for sending or retrieving messages.
Client ID
Specify a unique ID for the MQTT client. The MQTT server uses this client ID to recognize each distinct client and that client's current state.
Topics
Specify the MQTT topic or topics to subscribe to.
Quality of Service (QoS)
Quality of Service (QoS) is a level of guarantee for message delivery. Select one of the following options:
At most once (0) (default)
At least once (1)
Exactly once (2)
Security tab
Use this tab to define authentication credentials for the MQTT server.
Username
Specify the user name required to access the MQTT server.
Password
Specify the password associated with the user name.
Use secure protocol
Select this option to define SSL properties for the connection.
SSL Properties
ssl.contextProvider
Specify the underlying JSSE provider.
ssl.enabledCipherSuites
Specify which ciphers are enabled. Values are dependent on the provider.
ssl.keyManager
Specify the algorithm that will be used to create a KeyManagerFactory object instead of using the default algorithm available in the platform.
ssl.keyStore
Specify the name of the file that contains the KeyStore object that you want the KeyManager to use.
ssl.keyStorePassword
Specify the password for the KeyStore object that you want the KeyManager to use.
ssl.keyStoreProvider
Specify the identifying name or string for the key store provider.
ssl.keyStoreType
Specify the identifying name or string for the type of key store.
ssl.protocol
Specify the type of SSL protocol to use.
ssl.trustManager
Specify the algorithm that will be used to create a TrustManagerFactory object, instead of using the default algorithm available in the platform.
ssl.trustStore
Specify the name of the file that contains the KeyStore object that you want the TrustManager to use.
ssl.trustStorePassword
Specify the password for the TrustStore object that you want the TrustManager to use.
ssl.trustStoreProvider
Specify the identifier or string for the trust store provider.
ssl.trustStoreType
Specify the type of KeyStore object that you want the TrustManager to use.
Batch tab
Use this tab to specify how many messages to consume before processing. You can specify message count and or a specific amount of time.
The number of messages consumed before processing is defined by either Duration (ms) or Number of records. Messages are consumed when either the specified duration or number of records occurs. If you set either option to 0, PDI ignores that parameter.
Duration (ms)
Specify a time in milliseconds. This value is the amount of time the step will spend collecting records prior to the execution of the transformation. If this option is set to 0, then Number of records triggers consumption. Either Duration or Number of records must be greater than 0 to run the transformation.
Number of records
Specify a number. After every X records, the specified transformation is run and those records are passed to the transformation. If this option is set to 0, then Duration triggers consumption. Either Duration or Number of records must be greater than 0 to run the transformation.
Maximum concurrent batches
Specify the maximum number of batches used to collect records at the same time. Default: 1. Use this option only when your consumer step cannot keep pace with the speed at which the data is streaming and the environment has adequate CPU and memory. An error occurs if the environment cannot handle the maximum number of concurrent batches specified.
Message prefetch limit
Specify a limit for how many incoming messages this step will queue for processing, as they are received from the broker. Setting this value forces the broker to manage the backpressure of messages exceeding the specified limit. Default: 100000.
Fields tab
Use this tab to define the fields in the record format.
Input name
The input name is received from the MQTT streams. The following are received by default:
message
The individual message contained in a record.
topic
The category to which records are published.
Output name
The output name can be mapped to subscriber and member requirements.
Type
This value is always String. This field applies to the message and topic input names.
Result fields tab
Use this tab to select the step from the child transformation that will stream records back to the parent transformation.
Return fields from
Select the name of the step (from the child transformation) that will stream fields back to the parent transformation. The data values in these returned fields are available to any subsequent downstream steps in the parent transformation.
Options tab
This tab includes the following MQTT-specific parameters.
Keep Alive Interval
Specify a maximum number of interval seconds that is permitted to elapse between the point at which the PDI client finishes transmitting one control packet and the point it starts sending the next.
Max Inflight
Specify the maximum number of messages to have in process at any given time.
Connection Timeout
Specify the time (in seconds) to disconnect if a message is not received.
Clean Session
Specify whether the broker stores or purges messages for a session:
True
The broker does not store any information for the client. All information from a previous persistent session is purged.
False
The broker stores all subscriptions for the client. When QoS is set to 1 or 2, missed messages are stored. For details, see the QoS setting in the Setup tab.
Storage Level
Indicates whether messages are stored in memory or on disk:
Blank (default): memory
Disk: enter a valid path
Server URIs
Specify the MQTT server universal resource identifier (URI).
MQTT Version
Specify the MQTT protocol version that this step connects to.
Automatic Reconnect
Enable the client to attempt an automatic reconnect to the server if it becomes disconnected:
True: reconnect to the serverFalse: do not reconnect
Metadata injection support
This step supports metadata injection. You can use it with ETL metadata injection to pass metadata to your transformation at runtime.
See also
Last updated
Was this helpful?