Advanced topics

The following topics help to extend your knowledge of the Adaptive Execution Layer beyond basic setup and use:

• Spark Tuning

  You can customize PDI transformation and step parameters to improve the performance of running your PDI transformations on Spark. These parameters affect memory, cores, and instances used by the Spark engine. These Spark parameters include both application parameters and Spark tuning parameters.

  • See Configuring application tuning parameters for Spark to learn how to define additional Spark properties within the application.properties file or as run modification parameters within a transformation.

  • See About Spark tuning in PDI to learn more about how Spark tuning parameters work in PDI.

  • For the list of Spark tuning parameters available in PDI transformation steps, see Spark Tuning.

• Configuring AEL with Spark in a secure cluster

  If your AEL daemon server and your cluster machines are in a secure environment like a data center, you may only want to configure a secure connection between the PDI client and the AEL daemon server.