Before you begin

You must meet the following requirements for using the AEL daemon and operating the Spark engine for transformations:

• Pentaho 9.3 or later installation. See Pentaho installation.

• One of the following Hadoop distributions and versions:

  • Cloudera 6.1 or later

  • Hortonworks 3.0 and 3.1

  • Amazon EMR 5.21 and 5.24

  • Microsoft Azure HDInsight 4.0

• Spark client 2.3 or later.

• Pentaho Spark application 9.3.

• If you are configuring AEL for use with Cloudera, Hortonworks, or Amazon EMR, review Vendor-Supplied Clients.

CAUTION:

Spark does not support having Hadoop 2.x JAR files and Hadoop 3.x JAR files on the same class path as your spark application. Do not mix and match Hadoop versions.

Note: The dependency on Zookeeper was removed in Pentaho 8.0. If you installed AEL for Pentaho 7.1, you must delete the adaptive-execution folder and follow the Pentaho 8.0 or later Installation instructions to use AEL with Pentaho 8.0 or later.