Ctrlk
Try Pentaho! - Start your 30 day evaluation today
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Pentaho configuration

You can use basic configuration tasks to set up the Pentaho Server, data connections, the Pentaho design tools, and Hadoop cluster connections so you can get started creating ETL solutions and data analytics. These configuration tasks can be performed through the PUC (Pentaho User Console), the PDI (Pentaho Data Integration) client, or edits to shell scripts and property files. A Pentaho administrator user name and password are required to perform configuration tasks through the user console.

These tasks are for IT and Pentaho administrators, as described in the following definitions:

  • IT administrator

    Installs, configures, and upgrades the Pentaho Server. An IT administrator knows where the data is stored, how to connect to it, details about the computing environment, and how to use the command line on Microsoft Windows or Linux.

  • Pentaho administrator

    Responsible for creation and management of users and roles, along with managing workstations so the ETL specialists and business analysts can create, publish, and share content.

Note: This guide assumes you have installed the Pentaho software.

Tasks to be performed by an IT administrator

As an IT administrator, you configure the Pentaho Server and define security. If your team uses big data, you also configure access to Hadoop clusters.

Configure the Pentaho Server

Basic server tasks include starting and stopping the Pentaho Server, tuning memory, and setting up connections.

The following IT administrator tasks prepare the system for more specific Pentaho administrator configuration tasks, like defining connections and managing users and roles:

Stop and start the Pentaho Server and repository

To perform maintenance work on the Pentaho Server or components that use the Pentaho Server, such as the Pentaho User Console and Pentaho Data Integration, stop the repository and server, perform the work, and then restart the repository and server.

Start the Pentaho Repository before you start the Pentaho Server.

You must be an IT administrator and know where the data that you are managing is stored, how to access that stored data, details about the computing environment, and how to use the command line for Windows or Linux.

  1. Run the following control scripts to stop and start the Pentaho Server:

    Start on Windows

    ...\\pentaho\\server\\pentaho-server\\start-pentaho.bat

    Stop on Windows

    ...\\pentaho\\server\\pentaho-server\\stop-pentaho.bat

    Start on Linux

    .../pentaho/server/pentaho-server/start-pentaho.sh

    Stop on Linux

    .../pentaho/server/pentaho-server/stop-pentaho.sh

If you installed the Pentaho Repository as PostgreSQL, MySQL, MS SQL Server, or Oracle, consult that database documentation for start and stop instructions.

Script arguments and services (custom start/stop scripts)

Use these standard arguments:

  • start

  • stop

  • restart

  • status

  • help

Common service names:

  • pentahoserver

  • postgresql

Create scripts for automatic stop and start of the Pentaho server and repository on Linux

If you used the manual installation to install Pentaho on Linux, create scripts for automatic start on boot and stop at shutdown for the Pentaho Server and Pentaho Repository.

You must have root permissions.

  1. Navigate to /etc/init.d/ and create a file named pentaho.

  2. Open the pentaho file and enter the following content:

  3. Optional: Update the script for your environment.

    • If you are not using Red Hat Enterprise Linux, modify the details of the script to work with the OS, shells, and init systems that you are using. The script was tested only on Red Hat Enterprise Linux.

    • If you use an account other than the pentaho local user account to start services, replace pentaho with your account name.

    • If you use a MySQL or Oracle repository instead of PostgreSQL, replace postgresql with mysql or oracle.

    • If the solution repository runs on the same machine as the server, change postgresql to the name of the init script for your database.

    • If the solution repository runs on a remote computer, remove postgresql entirely and adjust the paths to the Pentaho Server scripts.

  4. Save and close the pentaho file.

  5. Navigate to /home/pentaho/pentaho/server/pentaho-server and open start-pentaho.sh.

  6. In start-pentaho.sh, change the last if statement to match the following example:

  7. Save and close start-pentaho.sh.

  8. Make the init script executable:

  9. Add the init script to the standard run levels.

    If your distribution supports update-rc.d, run:

Many modern Linux distributions use systemd instead of SysV init scripts. If update-rc.d is not available, consult your distribution documentation for the equivalent.

Increase Pentaho Server memory limit for custom installations on Windows or Linux

If you used a custom method to install PDI on Windows or Linux, consider increasing the PDI memory limit so that the Pentaho Server and the PDI client (also known as Spoon) can perform memory-intensive tasks, like sorting, grouping large datasets, or running complex transformations and jobs.

Increase the memory limit for both the Pentaho Server and the PDI client. If you do not increase the memory limit, PDI uses the default memory settings in the PDI startup scripts.

For instructions on increasing memory limits by editing the Tomcat startup script, see Configure and start the Pentaho Server after manual installation.

For instructions on increasing the memory limit in the PDI client, see Modify the PDI client startup script.

Specify data connections for BA design tools

The BA design tools need a way to connect to where you store data, and to the Pentaho Server where you publish and share what you create in the design tools.

We support connecting to data stored in these ways:

  • Pentaho data sources, such as relational Metadata data models or multidimensional Mondrian data models.

  • JDBC databases.

JDBC database connections

Your database access protocol affects how you create, test, and publish reports and models. Choose one of these protocols:

  • Native (JDBC): Standard JDBC connection details stored in each report or model.

  • JNDI: A named connection. Properties are stored outside the report or model.

  • ODBC: Not supported. Available only for compatibility.

  • OCI: Oracle-only. Use when you must use Oracle Call Interface.

Choose an access protocol

Native (JDBC)

Use Native (JDBC) when you need a quick setup or you are evaluating.

Keep this limitation in mind:

  • If connection details change, you must update every report and model that embeds them.

Behavior by tool:

  • Report Designer and Metadata Editor

    • Connection details are embedded in each report or Metadata model.

    • Report Designer lists the saved connection name after you define it.

    • Metadata Editor requires you to re-enter the connection name when you use it.

    • Before publishing, ensure the connection name matches a Native (JDBC) connection already defined on the Pentaho Server.

  • Schema Workbench and Aggregation Designer

    • Connection details are not stored. Define the connection each time.

    • Before publishing, you need the Pentaho Server URL and the name of a Native (JDBC) connection already defined on that server.

Recommendation:

  • Use for trials and evaluation.

  • For production, switch to JNDI before publishing.

JNDI

Use JNDI when you need centralized maintenance and a distributed team.

Key characteristics:

  • Reports and models store only the JNDI name.

  • You update connection details in one place.

Behavior by tool:

  • Report Designer and Metadata Editor

    • Connection properties live in a file on each workstation.

    • Before publishing, ensure the JNDI name matches a JNDI connection already defined on the Pentaho Server.

  • Schema Workbench and Aggregation Designer

    • JNDI is not available for connecting from the tool.

    • Before publishing, you still need the Pentaho Server URL and a JNDI connection name already defined on the Pentaho Server.

Recommendation:

  • Use for production deployments.

ODBC (not supported)

ODBC is not supported. It is available only if you have a hard dependency.

If you must use ODBC, contact Pentaho Support.

OCI (Oracle only)

If you connect to an Oracle database, you can use OCI. Use the steps in Define JDBC or OCI connections for BA design tools.

Define connections for BA design tools

Use these sections to set up connections on workstations running BA design tools.

Define JDBC or OCI connections for BA design tools

  1. Choose an access protocol in Choose an access protocol.

  2. Install the matching driver.

  3. Set the access protocol later in each design tool when you define the data source.

Add drivers

The driver lets the design tools connect to the Pentaho Server. This connection helps you validate the model.

Your database administrator or IT team can provide the driver. If needed, download it from your database vendor.

Copy the driver to these directories on every machine that runs the design tools. Stop the design tools before you copy the file.

  • Aggregation Designer: /pentaho/design-tools/agg-designer/drivers/

  • Metadata Editor: /pentaho/design-tools/metadata-editor/libext/JDBC/

  • Report Designer: /pentaho/design-tools/report-designer/lib/jdbc/

  • Schema Workbench: /pentaho/design-tools/schema-workbench/drivers/

Keep one driver version per database in each directory. Remove older versions to avoid conflicts.

If you are unsure, contact Pentaho Support.

Microsoft SQL Server driver notes

If you use Microsoft SQL Server, you might need the jTDS driver. jTDS is not vendor supported. Confirm the correct driver with Pentaho Support.

On Windows, many JDBC drivers support Type 2 integrated authentication using the integratedSecurity connection string property. To use it, copy sqljdbc_auth.dll to every machine and directory where you copied the JDBC driver.

You can find sqljdbc_auth.dll at:

<installation directory>\\sqljdbc_<version>\\<language>\\auth\\

  • Use the x64 DLL with a 64-bit JVM on an x64 processor.

  • Use the IA64 DLL with a 64-bit JVM on an Itanium processor.

Define JNDI connections for Report Designer and Metadata Editor

  1. Choose an access protocol in Choose an access protocol.

  2. Complete these tasks on each workstation.

Add drivers

The driver lets design tools connect to the Pentaho Server and validate models.

Your database administrator or IT team can provide the driver. You can also download it from your database vendor.

Check Components Reference to confirm your driver is supported.

Copy the driver to these directories on each machine where you installed design tools. Stop the tools first.

  • Aggregation Designer: /pentaho/design-tools/agg-designer/drivers/

  • Metadata Editor: /pentaho/design-tools/metadata-editor/libext/JDBC/

  • Report Designer: /pentaho/design-tools/report-designer/lib/jdbc/

  • Schema Workbench: /pentaho/design-tools/schema-workbench/drivers/

Keep only one version of the driver in each directory. Remove older versions to avoid conflicts.

If you are unsure, contact Pentaho Support.

Configure Report Designer

  1. Stop Report Designer. Stop the Pentaho Server too, if it runs locally.

    For server steps, see Stop and start the Pentaho Server and repository.

  2. Go to the .pentaho directory in the user profile.

    Examples:

    • Windows: C:\\Users\\username\\.pentaho\\

    • Linux or Solaris: /home/username/.pentaho/

  3. Go to ~/.pentaho/simple-jndi/. Create it if needed.

  4. Edit default.properties. Create it if needed.

    Example:

    In this example, SampleData is the JNDI connection name.

  5. Save the file.

  6. Restart Report Designer. Restart the Pentaho Server too, if it runs locally.

Repeat this process on each workstation where you run Report Designer.

Configure Metadata Editor

  1. Stop Metadata Editor. Stop the Pentaho Server too, if it runs locally.

    For server steps, see Stop and start the Pentaho Server and repository.

  2. Go to the metadata-editor/simple_jndi directory under your Metadata Editor install.

  3. Edit JDBC.properties. Create it if needed.

    Example:

    In this example, SampleData is the JNDI connection name.

  4. Save the file.

  5. Restart Metadata Editor. Restart the Pentaho Server too, if it runs locally.

Repeat this process on each workstation where you run Metadata Editor.

Specify data connections for the Pentaho Server

For business analytics, you can use the Pentaho User Console to define the connection to where you store data.

We support accessing data stored in the following ways:

  • Comma separated values (CSV) files, or any file that can be converted to CSV, such as spreadsheets, XML, or other semi-structured or tabular data files. Use CSV files if you are evaluating Pentaho or you want to get started quickly.

  • JDBC drivers to help with database connections.

In Pentaho Data Integration (PDI), you can make connections in each job and transformation through an input step. Although users can create connections themselves, it is best to set up shared connections for your users so that they can select the connection they need from a list.

JDBC database connections

Use JDBC database connections when the Pentaho Server connects to an RDBMS.

You choose one of these access types:

  • Native (JDBC): Define the connection in Pentaho User Console (PUC).

  • JNDI: Define a data source in your application server (Tomcat).

  • OCI: Oracle-only option for using the Oracle Call Interface.

  • ODBC: Available but not supported. Contact Pentaho Support if you must use it.

Choose between native (JDBC) and JNDI

Use native (JDBC) when you want the simplest setup.

You manage the full connection definition in PUC.

If connection details change, update each affected connection in PUC.

Use JNDI when you want central management in the application server.

This is common in production.

It can also help keep credentials out of Pentaho admin workflows.

If connection details change, update the application server configuration.

Add JDBC drivers

The Pentaho Server needs the correct JDBC driver for your database.

Get the driver from your database vendor or your IT team.

The Components Reference lists supported drivers.

Copy the driver to these directories:

  • Pentaho Server: /pentaho/server/pentaho-server/tomcat/lib/

  • PDI client: data-integration/lib

Restart the Pentaho Server and any PDI clients after you add drivers.

Driver conflict guidance

Keep only one driver per database type in each directory.

Remove older versions to avoid conflicts.

This matters when the driver matches your repository database type.

If you are unsure, contact Pentaho Support.

Microsoft SQL Server driver

For Microsoft SQL Server (MSSQL), you might need the non-vendor driver called jTDS.

Contact Pentaho Support to confirm the correct driver.

Most Microsoft JDBC drivers support Type 2 integrated authentication on Windows.

Use the integratedSecurity property in the connection string.

To use integrated authentication, copy sqljdbc_auth.dll to every location where you copied the JDBC driver.

You can find sqljdbc_auth.dll here:

<installation directory>\\sqljdbc_<version>\\<language>\\auth\\

Choose the correct DLL:

  • 64-bit JVM on an x64 processor: use the x64 folder.

  • 64-bit JVM on an Itanium processor: use the IA64 folder.

Set up JNDI connections for the Pentaho Server

If you installed the Pentaho Server using the manual installation method, you set up JNDI data sources in Tomcat.

Use this section to add JNDI data sources and update existing ones.

Before you begin

  • You need IT administrator access.

  • You must know how to connect to your data sources.

  • You must know your OS and database setup.

Defining JNDI connections for PDI clients

If you publish to the Pentaho Server from a PDI client, you can configure the PDI client with the same JNDI settings as the Pentaho Server.

This lets you develop and test transformations without keeping your application server running.

Edit the jdbc.properties file to mirror your application server data sources:

  • Location: /pentaho/design-tools/data-integration/simple-jndi/jdbc.properties

Tomcat JNDI connections

Use JNDI connections when you manage data sources in Tomcat.

Install the correct JDBC driver before you create or use a JNDI data source.

Next step: define the connection

After the driver is installed, create the connection definition in Pentaho.

See Define data connections.

See also

The native database connections in the Pentaho Suite are based on Java Database Connectivity (JDBC).

For Pentaho Server-specific connection setup, see:

Define security for the Pentaho Server

You also need to establish a security plan for your Pentaho system. Pentaho supports two different security options: Pentaho Security, or advanced security providers, such as LDAP, Single Sign-On, or Microsoft Active Directory.

This table can help you choose the option that is best for you.

Explore Considerations
Choose Options

Pentaho security in PUC and Pentaho security in the PDI client

Advanced security providers

Summary

Pentaho Security is the easiest way to configure security quickly. Both the Pentaho User Console and the PDI client enable you to define and manage users and roles. The Pentaho Server controls which users and roles can access web resources through the User Console or resources in the Pentaho Repository.

Pentaho Security works well if you do not have a security provider or if you have a user community with less than 100 users.

If you are already using a security provider, such as LDAP, Single Sign-On, or Microsoft Active Directory, you can use the users and roles you have already defined with Pentaho. Your security provider controls which users and roles can access Pentaho web resources through the User Console or resources in the Pentaho Repository.

Advanced security scales well for production and enterprise user communities.

Expertise

Knowledge of your user community and which users should have which roles in the Pentaho system. Knowledge about security in general is NOT required.

Knowledge of your user community and which users should have which roles in the Pentaho system. Knowledge about your particular security provider and its options is required.

Recommendation

Recommended for the Pentaho trial download or evaluation only. Caution: Do not use for production.

Recommended for production.

Depending on the security option you choose, refer to the following details in the Administer Pentaho Data Integration and Analytics document:

  • If you want to use Pentaho Security, see Manage Users and Roles in PUC and Manage Users and Roles in the PDI client.

  • If you want to use an advanced security provider, see Implement Advanced Security.

  • For information on using Kerberos for security, see How to Enable Kerberos Authentication.

Use password encryption with Pentaho

Strengthen security by using encrypted passwords for Pentaho applications.

For IT administrators, who have permissions to modify files on the server and the permission to stop and start the server, perform these tasks when you want to enhance your company's security by encrypting the passwords that are currently stored as plain text in configuration files, for example, if you want to meet specific server security levels for regulatory compliance.

As a best practice, stop the server before modifying configuration files, then start the server when finished. After you have configured a Pentaho product to use encrypted passwords, all logins with the Pentaho product will use the encrypted passwords. Connect to any databases that were edited to ensure all changes are correct.

Use this topic to:

Encrypted passwords are supported for:

You can also use encrypted passwords with JDBC security. See the Administer Pentaho Data Integration and Analytics document for instructions on switching to JDBC security.

Encrypting a password

Perform the following steps on the machine with the Pentaho Server to create an encrypted password.

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. At the command line, navigate to the server/pentaho-server directory.

  3. Run encr.bat (Windows) or encr.sh (Linux).

    Example:

    The console prints the encrypted password.

    Note: You must have a JRE or JDK installed.

  4. Restart the server.

Use encrypted passwords with Pentaho products

How you apply an encrypted password varies by product.

Pentaho Data Integration (PDI)

Perform the following steps to use an encrypted password with Pentaho Data Integration (PDI).

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. Navigate to the design-tools/data-integration/simple-jndi directory.

  3. Open the jdbc.properties file in a text editor.

  4. Replace all instances of the password value with the encrypted password.

  5. Save and close the file.

  6. Restart the server and verify the change.

Pentaho User Console (PUC)

Perform the following steps to use an encrypted password with the Pentaho User Console (PUC).

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. Navigate to the server/pentaho-server/tomcat/webapps/pentaho/META-INF directory.

  3. Open the context.xml file in a text editor.

  4. Replace the password value in every Resource element with the encrypted password.

  5. Save and close the file.

  6. Restart the server and verify the change.

PUC email

After you configure PUC to use an encrypted password, you can use that password with PUC email.

  1. Log in to PUC as an administrator.

  2. Open the Administration Perspective.

  3. Select the Mail server section.

  4. Enter your encrypted password value in the password field.

    Note: If you use Gmail, enable Allow less secure apps to access your account.

  5. Select Test Email Configuration.

  6. Verify that PUC sends an email to the address you specified.

Pentaho Aggregation Designer

To use encrypted passwords with Pentaho Aggregation Designer, you must first centralize your passwords in a jndi.properties file.

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. Create a jndi.properties file with the default properties:

  3. Save jndi.properties in the design-tools/aggregation-designer/lib directory.

  4. In the user’s home directory, navigate to the .pentaho/simple-jndi directory.

    • Open default.properties in a text editor.

    • If default.properties does not exist, create it.

    • If you created default.properties under design-tools/aggregation-designer/simple-jndi, update org.osjava.sj.root in design-tools/aggregation-designer/lib/jndi.properties to point to it. Example:

  5. Replace the password value in every property in default.properties with the encrypted password.

    Note: If you use a remote repository, replace localhost with the repository IP address.

  6. Save and close the file.

  7. Restart the server and verify the change.

Pentaho Metadata Editor (PME)

The Pentaho Metadata Editor (PME) stores passwords in the JNDI connection default.properties file. For setup details, see Define JNDI connections for Report Designer and Metadata Editor.

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. In the user’s home directory, navigate to the .pentaho/simple-jndi directory.

  3. Open default.properties in a text editor.

    Note: If default.properties does not exist, create it.

  4. Replace the password value in every property with the encrypted password.

    Note: If you use a remote repository, replace localhost with the repository IP address.

  5. Save and close the file.

  6. Restart the server and verify the change.

Pentaho Report Designer (PRD)

The Pentaho Report Designer (PRD) stores passwords in the JNDI connection default.properties file. For setup details, see Define JNDI connections for Report Designer and Metadata Editor.

  1. Stop the server.

    For instructions, see Stop and start the Pentaho Server and repository.

  2. Navigate to the design-tools/report-designer/configuration-template/simple-jndi directory.

  3. Open default.properties in a text editor.

  4. Replace the password value in every property with the encrypted password.

    Note: If you use a remote repository, replace localhost with the repository IP address.

  5. Save the file.

  6. Copy default.properties to the .pentaho/simple-jndi directory in the user’s home directory. Replace the existing file.

    Note: If the .pentaho/simple-jndi directory does not exist, create it.

  7. Restart the server and verify the change.

After you update a product

After you configure a product to use encrypted passwords, all logins with that product use encrypted passwords.

Connect to any databases you updated to verify the changes.

Set up Pentaho to connect to a Hadoop cluster

Use this topic to configure Pentaho to connect to Hadoop clusters.

Supported distributions include Amazon EMR, Azure HDInsight (HDI), Cloudera Data Platform (CDP), and Google Dataproc.

Pentaho also supports related services such as HDFS, HBase, Hive, Oozie, Sqoop, YARN/MapReduce, ZooKeeper, and Spark.

You can connect to clusters and services from these Pentaho components:

  • PDI client (Spoon), along with Kitchen and Pan command line tools

  • Pentaho Server

  • Analyzer (PAZ)

  • Pentaho Interactive Reports (PIR)

  • Pentaho Report Designer (PRD)

  • Pentaho Metadata Editor (PME)

Pentaho connects to Hadoop clusters through a compatibility layer called a driver (Big Data shim).

To confirm which drivers are supported for your version, see the Components Reference.

Drivers are shipped as vendor-specific builds of the optional pentaho-big-data-ee-plugin.

Download drivers from the Hitachi Vantara Lumada and Pentaho Support Portal.

Note: Pentaho ships with a generic Apache Hadoop driver. For specific vendor drivers, visit the Hitachi Vantara Lumada and Pentaho Support Portal to download the drivers.

Install a new driver

You need a driver for each cluster vendor and version you connect to from:

  • PDI client (Spoon), plus Kitchen and Pan

  • Pentaho Server

  • Analyzer

  • Interactive Reports

  • Pentaho Report Designer (PRD)

  • Pentaho Metadata Editor (PME)

Pentaho ships with a generic Apache Hadoop driver. Download vendor-specific drivers from the Support Portal.

1

Download the driver plugin

  1. Sign in to the Support Portal.

  2. Go to Downloads.

  3. In the 11.0 list, open the full downloads list.

  4. Open Pentaho 11.0 GA Release.

  5. Download the driver plugin from Big Data Shims.

Common driver plugin files:

  • Apache Vanilla: pentaho-big-data-ee-plugin-apachevanilla-11.0.0.0-<build-number>.zip

  • Cloudera Data Platform: pentaho-big-data-ee-plugin-cdpdc71-11.0.0.0-<build-number>.zip

  • Google Dataproc: pentaho-big-data-ee-plugin-dataproc1421-11.0.0.0-<build-number>.zip

  • Amazon EMR: pentaho-big-data-ee-plugin-emr770-11.0.0.0-<build-number>.zip

  • Azure HDInsight: pentaho-big-data-ee-plugin-hdi40-11.0.0.0-<build-number>.zip

2

Install the driver on the PDI client

  1. Stop PDI.

  2. Extract the downloaded .zip into:

    • <pdi-install-dir>/data-integration/plugins

  3. If you are replacing an existing driver plugin, remove the old pentaho-big-data-ee-plugin folder first.

3

Install the driver on the Pentaho Server

  1. Stop the Pentaho Server.

  2. Extract the downloaded .zip into:

    • <pentaho-server>/pentaho-solutions/system/kettle/plugins

  3. If you are replacing an existing driver plugin, remove the old pentaho-big-data-ee-plugin folder first.

4

Restart and verify

  1. Restart the PDI client and the Pentaho Server.

  2. Create or update your cluster connection and verify it connects.

Update drivers

When drivers for new Hadoop versions are released, download the new driver plugin and repeat the install steps.

Additional configurations for specific distributions

Use these settings when you configure Pentaho to connect to specific Hadoop distributions:

Amazon EMR

The following settings are available while you configure Pentaho to connect to a working Amazon EMR cluster.

EMR clusters (version 7.x and later) built with JDK 17 exclude commons-lang-2.6.jar from standard Hadoop library directories (such as $HADOOP_HOME/lib).

To use the EMR driver with EMR 7.x:

  1. Download commons-lang-2.6.jar from a trusted source (for example, Maven Repository: commons-lang » commons-lang » 2.6).

  2. Copy the JAR to $HADOOP_HOME/lib or $HADOOP_MAPRED_HOME/lib on every EMR node.

Before you begin

Before you set up Pentaho to connect to an Amazon EMR cluster, do these tasks:

  1. Check the Components Reference to confirm your Pentaho version supports your EMR version.

  2. Prepare your Amazon EMR cluster:

    1. Configure an Amazon EC2 cluster.

    2. Install required services and service client tools.

    3. Test the cluster.

  3. Install PDI on an Amazon EC2 instance in the same Amazon VPC as the EMR cluster.

  4. Get connection details from your Hadoop administrator.

  5. Add the YARN user on the cluster to the group defined by dfs.permissions.superusergroup in hdfs-site.xml.

As a best practice, install PDI on the Amazon EC2 instance.

Otherwise, you may not be able to read or write cluster files.

For a workaround, see Unable to read or write files to HDFS on the Amazon EMR cluster.

You also need to share connection details with users after setup.

For the full list, see Hadoop connection and access information list.

Edit configuration files for users

Your cluster administrator must download cluster configuration files.

Update the files with Pentaho-specific and user-specific values.

Use these files to create or update a named connection.

Where named connection files live

Named connection files are stored here:

  • Named connection XML: <username>/.pentaho/metastore/pentaho/NamedCluster

  • Named connection config folder: <username>/.pentaho/metastore/pentaho/NamedCluster/Configs/<connection-name>

  • Extra settings file: <username>/.pentaho/metastore/pentaho/NamedCluster/Configs/<connection-name>/config.properties

Save edited files in a safe location.

Files to provide to users

Provide these files to each user:

  • core-site.xml

  • mapred-site.xml

  • hdfs-site.xml

  • yarn-site.xml

Verify or edit core-site.xml file

If you plan to run MapReduce jobs on Amazon EMR, confirm you have read, write, and execute access to the S3 buffer directories specified in core-site.xml.

Edit core-site.xml to add AWS access keys and (optional) LZO compression settings.

1

Open the file

Open core-site.xml from the folder where you saved the other *-site.xml files.

2

Add AWS credentials

Add your AWS Access Key ID and secret access key:

3

Optional: Add S3N credentials

If you use S3N, add these properties:

4

Add filesystem implementation settings

Add these properties:

5

Configure LZO compression

If you are not using LZO compression, remove any references to com.hadoop.compression.lzo.LzoCodec from core-site.xml.

If you are using LZO compression:

  1. Download the LZO JAR.

  2. Add it to pentaho-big-data-plugin/hadoop-configurations/emr3x/lib.

Download: http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/

6

Save and apply the change

Save the file.

Update the named connection.

Upload the updated core-site.xml.

Edit mapred-site.xml file

If you use MapReduce, edit mapred-site.xml.

You also enable cross-platform MapReduce job submission.

1

Open the file

Open mapred-site.xml from the folder where you saved the other *-site.xml files.

2

Add the property

Add this property:

This property is only required for MapReduce jobs on Windows.

3

Save and apply the change

Save the file.

Update the named connection.

Upload the updated mapred-site.xml.

Connect to a Hadoop cluster with the PDI client

After you set up the Pentaho Server to connect to a cluster, configure and test the connection.

See the Pentaho Data Integration documentation for instructions.

Connect other Pentaho components to the Amazon EMR cluster

Use this procedure to create and test a connection to your Amazon EMR cluster from these Pentaho components:

  • Pentaho Server (DI and BA)

  • Pentaho Metadata Editor (PME)

  • Pentaho Report Designer (PRD)

Install a driver for the Pentaho Server

Install a driver for the Pentaho Server.

For instructions, see Install a new driver.

Create and test connections

Create and test a connection for each component:

  • Pentaho Server for DI: Create a transformation in the PDI client and run it remotely.

  • Pentaho Server for BA: Create a connection to the cluster in the Data Source Wizard.

  • PME: Create a connection to the cluster in PME.

  • PRD: Create a connection to the cluster in PRD.

Share connection details with users

After you connect to the cluster and services, share the connection details with users.

Users can access the cluster only from machines configured to connect to it.

To connect, users need:

  • Hadoop distribution and version

  • HDFS, JobTracker, ZooKeeper, and Hive2/Impala hostnames (or IP addresses) and port numbers

  • Oozie URL (if used)

Users also need permissions for required HDFS directories.

For a detailed list of required information, see Hadoop connection and access information list.

Azure HDInsight

Use these settings when you configure Pentaho to connect to Azure HDInsight (HDI).

Before you begin

Before you set up Pentaho to connect to HDI, do the following:

  1. Check Components Reference. Confirm your Pentaho version supports your HDI version.

  2. Prepare your HDI instance:

    1. Configure your Azure HDInsight instance.

    2. Install required services and client tools.

    3. Test the platform.

    4. If HDI uses Kerberos, complete the Kerberos steps in this page.

  3. Get connection details from your platform admin. You will share some of this information with users later. See Hadoop connection and access information list.

  4. Add the YARN user to the group defined by dfs.permissions.superusergroup in hdfs-site.xml.

  5. Set up the Hadoop driver for your HDI version. See Install a new driver.

Kerberos-secured HDInsight instances

If you connect to HDI secured with Kerberos, complete these steps first:

  1. Configure Kerberos security on the platform. Configure the Kerberos realm, KDC, and admin server.

  2. Configure these nodes to accept remote connection requests:

    • NameNode

    • DataNode

    • Secondary NameNode

    • JobTracker

    • TaskTracker

  3. If you deployed HDI using an enterprise program, set up Kerberos for those nodes.

  4. Add user credentials to the Kerberos database for each Pentaho user.

  5. Verify an OS user exists on each HDI node for each Kerberos user. Create users as needed.

User account UIDs should be greater than min.user.id. The default is usually 1000.

  1. Set up Kerberos on your Pentaho machines. See the Administer Pentaho Data Integration and Analytics guide.

Edit configuration files for users

Your Azure admin downloads the site configuration files for the services you use. They update the files with Pentaho-specific and user-specific settings. Users upload the updated files when they create a named connection.

Named connection files are stored in these locations:

  • <username>/.pentaho/metastore/pentaho/NamedCluster

  • <username>/.pentaho/metastore/pentaho/NamedCluster/Configs/<connection-name>/config.properties

Save the updated files in a known location for reuse.

Files to provide

  • core-site.xml (secured HDInsight only)

  • hbase-site.xml

  • hive-site.xml

  • mapred-site.xml

  • yarn-site.xml

If you update these files after creating a named connection, edit the named connection and re-upload the updated files.

Edit Core site XML file

If you use a secured instance of Azure HDInsight, update core-site.xml.

  1. Open core-site.xml.

  2. Add or update properties for your storage type.

    WASB storage

    Add these properties:

    • fs.AbstractFileSystem.wasb.impl: org.apache.hadoop.fs.azure.Wasb

    • pentaho.runtime.fs.default.name: wasb://<container-name>@<storage-account-name>.blob.core.windows.net

    Example:

    ADLS (ABFS) storage

    Add this property:

    • pentaho.runtime.fs.default.name: abfs://<container-name>@<storage-account-name>.dfs.core.windows.net

    Example:

  3. Save the file.

Edit HBase site XML file

If you use HBase, update hbase-site.xml to set the temporary directory.

  1. Open hbase-site.xml.

  2. Add or update this property:

    • hbase.tmp.dir: /tmp/hadoop/hbase

  3. Save the file.

Edit Hive site XML file

If you use Hive, update hive-site.xml to set the Hive metastore location.

  1. Open hive-site.xml.

  2. Add or update these properties:

    • hive.metastore.uris: Hive metastore URI, if different from your HDInsight instance.

    • fs.azure.account.keyprovider.<storage-account>.blob.core.windows.net: Azure storage key provider principal, if required.

    Example:

  3. Save the file.

Edit Mapred site XML file

If you use MapReduce, update mapred-site.xml for job history logging and cross-platform execution.

  1. Open mapred-site.xml.

  2. Ensure these properties exist:

    • mapreduce.jobhistory.address: where MapReduce job history logs are stored

    • mapreduce.job.hdfs-servers: HDFS servers used by YARN to run MapReduce jobs

    Example:

  3. Optional: If YARN containers run on JDK 11 nodes, add this property:

    • mapreduce.jvm.add-opens-as-default: false

Do not add mapreduce.jvm.add-opens-as-default for containers running on JDK 17 nodes.

Example:

  1. Save the file.

Edit YARN site XML file

If you use YARN, verify your yarn-site.xml settings.

  1. Open yarn-site.xml.

  2. Add or update these properties:

    • yarn.resourcemanager.hostname: ResourceManager host name

    • yarn.resourcemanager.address: ResourceManager address and port

    • yarn.resourcemanager.admin.address: ResourceManager admin address and port

    Example:

  3. Save the file.

After you change these files, edit the named connection and upload the updated files.

Oozie configuration

If you use Oozie, configure both the cluster and the Pentaho server.

By default, the Oozie user runs Oozie jobs. If you start an Oozie job from PDI, set up a PDI proxy user.

Set up Oozie on a cluster

Add your PDI user to oozie-site.xml.

  1. Open oozie-site.xml on the cluster.

  2. Add these properties. Replace <pdi-username> with the PDI user name.

  3. Save the file.

Set up Oozie on the server

Set the proxy user for the named cluster on the Pentaho server.

  1. Open config.properties:

    /<username>/.pentaho/metastore/pentaho/NamedCluster/Configs/<connection-name>/config.properties

    This path is created when you create a named connection.

  2. Set pentaho.oozie.proxy.user to the proxy user name.

  3. Save the file.

Windows configuration for a secured cluster

If you run Pentaho Server on Windows and your cluster uses Kerberos, point Tomcat to your krb5.conf or krb5.ini.

  1. Go to server/pentaho-server.

  2. Open start-pentaho.bat.

  3. Set CATALINA_OPTS to include the Kerberos config path:

  4. Save the file.

Connect to HDI with the PDI client

After you set up the Pentaho Server to connect to HDI, configure and test the connection from PDI.

See the Pentaho Data Integration documentation for how to connect the PDI client to a cluster.

Connect other Pentaho components to HDI

Create and test an Azure HDInsight (HDI) connection in:

  • Pentaho Server

  • Pentaho Metadata Editor (PME)

  • Pentaho Report Designer (PRD)

Prerequisites

Install a driver for the Pentaho Server. See Install a new driver.

Create and test connections

Create and test the connection in each product:

  • Pentaho Server (DI): Create a transformation in the PDI client. Run it remotely.

  • Pentaho Server (BA): Create a connection to HDI in the Data Source Wizard.

  • PME: Create a connection to HDI.

  • PRD: Create a connection to HDI.

After you connect, share connection details with users.

Users typically need:

  • HDI distribution and version

  • HDFS, ResourceManager (JobTracker), ZooKeeper, and HiveServer2 hostnames, IP addresses, and ports