Opening Spark application tuning

You can specify values for fine tuning the following types of Spark application parameters within either the data-integration/adaptive-execution/config/application.properties file or the Parameters tab of the PDI Transformation properties dialog box:

• Executor and driver resource sizing

• YARN utilization impacts

• Default partitioning

• Memory splits

You can also control these types of Spark application parameters through PDI environments variables.

As an example of application tuning, the number of default executors may be too low for your PDI transformation to efficiently utilize YARN capacity. You would improve capacity by increasing the number of executors based on data storage memory and cluster resources.

See Configuring application tuning parameters for Spark for more information on setting Spark application parameters in PDI. See the Spark Application Properties documentation for a full list of Spark application parameters.