# What's New in 11.0
Release 11 of Pentaho Data Integration & Analytics includes several major updates that improve the user experience in both Data Integration and Business Analytics. It also introduces new performance- and security-related features, as well as simplified deployment and upgrades.
Release 11 is a Long Term Supported (LTS) release. For details about the support period and the service pack and update release schedule, see the[ Pentaho Support Lifecycle page](https://support.pentaho.com/hc/en-us/articles/205789159-Pentaho-Product-Lifecycle-Overview).
The **key features and enhancements introduced** in this release are:
1. [**Pipeline Designer**](#pipeline-designer)
2. [**Project Lifecycle Management**](#project-lifecycle-management)
3. [**New Pentaho User Console**](#new-pentaho-user-console-preview) **(Preview)**
4. [**New Data Modeling Workflow (replacing Schema Workbench and Data Source Wizard)**](#new-data-modeling-workflow)
5. [**Out-of-the-box support for OIDC / OAuth 2.0**](#out-of-the-box-support-for-oidc-oauth-2.0)
6. [**Granular Permissions in Pentaho / BA Server**](#granular-permissions-for-pentaho-ba-server)
7. [**Docker Simplification**](#docker-simplification)
8. [**Java 21 Support & Tomcat 10**](#java-21-support-tomcat-10)
9. [**Plugin Manager**](#plugin-manager)
10. [**Karaf/OSGi removal and Big Data plugins**](#karaf-osgi-removal-and-big-data-plugins)
11. [**OTEL-based observability**](#otel-based-observability)
The following sections describe each of these features and enhancements.
## Pipeline Designer
Release 11 introduces a browser-based user experience (UX) that you can use to author ETL pipelines. This experience enables you to build PDI transformations and jobs in a web browser without the need to deploy Spoon.
Conceptually, it is similar to the Spoon interface that users know and rely on. However, the UI is built on a modern UI framework. This interface also maintains compatibility with existing transformations and jobs that you developed by using Spoon.
For more information, see [Pipeline Designer](/pba/pipeline-designer).
## Project Lifecycle Management
In PDI versions prior to Release 11, there was no defined structure for organizing transformations, jobs, and associated configuration. This lack of structure made it challenging for ETL developers and DevOps teams to keep pipeline development work organized and made collaboration and migration across environments difficult.
In addition, there were differences in how configuration references were resolved, resulting in behavior that might appear to be inconsistent.
Project Lifecycle Management introduces several features to address these gaps. For more information, see [Configuring ETL with Projects](https://docs.pentaho.com/pdia-data-integration/pdia-11.0-data-integration/organizing-etl-with-projects#manageability).
## New Pentaho User Console (Preview)
With Release 11, Pentaho introduces a completely revamped user experience (UX) for Pentaho User Console (PUC). This UX aligns with the broader Pentaho platform UX and serves as the entry point for most PDI and PBA users. The new UX presents a more modern and convenient interface and addresses several pain points associated with the existing PUC in 10.2 and earlier.
The existing PUC will continue to be available until all existing functionality has been moved to the new UX. For more information, see [Introducing Modern PUC](https://docs.pentaho.com/pba/11.0-pba/pentaho-user-console/modern-design).
## New Data Modeling Workflow
Release 11 introduces a new tool for building and managing Mondrian data models. Customers have typically used Schema Workbench or Data Source Wizard to build, edit, and deploy Mondrian models. Pentaho now includes a web-based tool, Semantic Model Editor (SME), that enables you to build and manage Mondrian models in an easy-to-use, modern UX.
Both novice and advanced users can use SME to work with Mondrian data models. SME provides a significantly better experience for users of PBA and especially Analyzer. In addition, SME supports all existing Mondrian models and provides advanced capabilities that were not available in either Schema Workbench or Data Source Wizard.
For more information, see [Semantic Model Editor](https://docs.pentaho.com/pba/11.0-pba/semantic-model-editor).
## Out-of-the-box support for OIDC / OAuth 2.0
Release 11 supports OAuth 2.0/OIDC authentication for Pentaho Server, enabling single sign-on (SSO) integration with identity providers such as Google, Okta, and Azure. It supports any OIDC-compliant identity provider (IdP). This support greatly simplifies SSO configuration in Pentaho.
For more information, see [OIDC/OAuth](/pdia-admin/secure-the-pentaho-system/user-security/advanced-security-providers/oidc-oauth-2.0).
## Granular Permissions for Pentaho / BA Server
Release 11 introduces more granular and flexible access control across the platform to address several long-standing challenges, such as:
1. Permissions were not sufficiently fine-grained. For example, the "Read Content" permission allows a user to see any content from any plugin (unless file or folder permissions prevent it).
2. Permissions were not structured in a way that could prevent or provide access to individual plugins.
3. Permissions were not sufficiently fine-grained for data sources or other assets.
4. Execute permissions were very broad.
Release 11 addresses these issues in Pentaho Server. Together with OAuth 2.0/OIDC support, Pentaho now has a more robust authentication and authorization model.
For more information, see [Granular Permissions for Pentaho](https://docs.pentaho.com/pba/11.0-pba/semantic-model-editor/sharing-a-semantic-model/permissions-for-semantic-models).
## Docker Simplification
Release 11 significantly simplifies Docker-based Pentaho deployments. First, Pentaho introduces optimized, prebuilt images for on-premises deployments (plain Docker and Kubernetes \[K8s]), EKS, AKS, and GKE. These images have standardized installation paths and variables. In addition, enhanced entrypoint scripts support runtime configuration overrides (configuration files and licenses), allowing flexible customization by injecting files at startup.
For more information, see [Docker Deployment](https://docs.pentaho.com/install/pentaho-installation-overview-cp/docker-container-deployment-of-pentaho-server).
## Java 21 Support & Tomcat 10
Java 21 is supported in Release 11. This support allows customers to operate Pentaho with an Oracle JVM without associated licensing costs. Customers can continue to use OpenJDK or other supported JVMs (see the list of supported JVMs in the documentation).
Pentaho Server in Release 11 comes with Tomcat 10, addressing several vulnerabilities and other defects associated with Tomcat 9.
For more information about supported JVM and Tomcat versions, see [Components Reference](https://docs.pentaho.com/install/pdia-11.0-installation/components-reference).
## Plugin Manager
Release 11 introduces a Plugin Manager that enables administrators to manage both PDI and PBA plugins. Going forward, Pentaho will release new functionality as plugins where possible. Having a plugin manager to identify, deploy, and manage plugins conveniently is therefore key to managing Pentaho deployments.
For more information, see [Plugin Manager](https://docs.pentaho.com/pba/11.0-pba/pentaho-user-console/modern-design/plugin-manager).
## Karaf / OSGi Removal and Big Data Plugins
With Release 11, Karaf and OSGi are completely removed from PDI. Big data components are now available as plugins. There is no separate PDI deployment with big data add-ons. Big data components are treated the same as any other PDI plugin and can be deployed in the same manner.
This change significantly reduces the size of both the PDI client and Pentaho Server deployments by more than 1 GB.
For more information, see [Plugin Manager](https://docs.pentaho.com/pba/11.0-pba/pentaho-user-console/modern-design/plugin-manager).
## OTEL-based Observability
[OTEL](https://opentelemetry.io/docs/languages/java/) is an open standard for applications to communicate telemetry data, such as traces, metrics, and logs. There is a broad ecosystem of tools that can consume OTEL data, including Datadog, Splunk, Elastic, Amazon CloudWatch, and Azure Monitor.
With the OTEL plugin, you can monitor every Pentaho ETL process with the following:
* Logs that are consolidated in a single place and represented hierarchically
* Traces to view task timing, execution hierarchy, and variables during execution
* Metrics to track data flow trends at specified points of interest
For more information, see [Plugin Manager](https://docs.pentaho.com/pba/11.0-pba/pentaho-user-console/modern-design/plugin-manager).
## Other Enhancements
Release 11 also introduces several other enhancements and defect fixes. These are documented in the release notes.
# Release Notes 11.0
Welcome to the release notes for Pentaho Data Integration & Analytics 11.0. This is a Long Term Supported (LTS) release. LTS releases are stable releases with a longer support lifecycle. See the Support Lifecycle page to review the support terms for an LTS release.
The last LTS release, 10.2, was released in August 2024. This document covers the incremental enhancements from 10.2 to 11.0.
## What's New?
The **key features and enhancements introduced** in this release are:
1. **Pipeline Designer**
2. **Project Lifecycle Management**
3. **New Pentaho User Console (Preview)**
4. **New Data Modeling Workflow (replacing Schema Workbench and Data Source Wizard)**
5. **OOB Support for OIDC / OAuth 2.0**
6. **Granular Permissions in Pentaho / BA Server**
7. **Java 21 Support & Tomcat 10**
8. **Plugin Manager**
9. **Karaf / OSGi removal and Big Data plugins**
10. **OTEL-based observability**
11. **Docker Simplification**
For an overview of the new features, see [What's New in 11.0](/whats-new).
## Bug Fixes
The following issues have been addressed in 11.0.
### Pentaho Data Integration Issues
The following issues in Pentaho Data Integration have been addressed.
Issue ID
Description
PDI-20592
Spoon throws an OutOfMemoryError when running jobs in parallel
PDI-20300
Bug - Carte Object ID is not captured in the channel logs table when execution is done through the executeJob API
PDI-20282
Failure to abort the main job even after the subjobs fail with an error.
PDI-20251
PDI data services - Missing Log4j-related libraries
PDI-20188
NullPointerException when saving a job/transformation to a file-based repository in Pentaho 10.2 and 10.1
PDI-20187
AWS v2 metadata check causes failure to start
PDI-20199
Spoon runs out of memory while running a job
PDI-19731
Separator is not working as expected in Text File Input and Text File Output.
PDI-20540
Avro Input step does not handle expected null fields
PDI-20404
Splash screen text capitalization
PDI-20194
Variable replacement in PVFS implementation is not working as expected
PDI-20319
Unable to read nested fields from a JSON file and throws "cannot find any data" with path error. (.js vs .json)
PDI-20160
Cannot extract files from an SFTP server when multiple copies of a job that uses the SFTP step are called
PDI-20105
MapReduce execution does not wait for KAR file installation before launching the mapper KTR
PDI-20220
Pan cannot run KTRs with PVFS URLs when the associated VFS connection config contains a variable
PDI-20159
REST API step - When processing an API URL, the space value is not rendered correctly
PDI-20589
Mail step in transformation fails when no auth is used
PDI-20546
[Transformation] Bulk load into Snowflake UI issue
PDI-20419
Bulk load into Amazon Redshift throws error: S3ServiceException: The AWS Access Key ID you provided does not exist in our records
PDI-20406
Too many transitive dependencies in the Excel and core plugins cause errors
PDI-20325
[Excel Input] Throws higher override value with IOUtils.SetByteArrayMaxOverride() exception
PDI-20295
10.2.x: Microsoft Excel Writer step is unable to write incoming rows to an MS Excel file.
PDI-20283
Excel Writer step fails to write XLS with java.lang.NoClassDefFoundError: com/zaxxer/sparsebits/SparseBitSet when "Shift existing cells down" is enabled
PDI-20189
Step (or job) names are unchanged to Japanese characters in PDI.
PDI-20601
/opt/pentaho/data-integration/logs is not writable as a mounted volume
PDI-20568
Cannot copy files using the job's Copy Files step from S3 bucket to S3 bucket
PDI-20563
Job Status API fails when multiple instances of the same job exist
PDI-20569
Error when attempting to copy a folder using S3 VFS (MinIO)
PDI-20548
When backing up with "import-export.sh", the value of the "add file name to result filenames" tag in the backed up job file changes from "Y" to "N".
PDI-20514
Using variables in S3 connection parameters: only the "region" variable is not retained after reopening the connection dialog
PDI-20495
Variable is not retained in "Timeout" field of "Get a file with FTP" step
PDI-20494
Variable is not retained when set in "Wait" column of "Check DB connections" entry
PDI-20485
"Put a file with FTP" PDI step "timeout" field does not store a variable from environment variables.
PDI-20484
"FTP Delete" PDI job entry step "timeout" field does not store a variable from environment variables.
PDI-20482
"Block this step until steps finish" PDI step Copy nr field does not store a variable from environment variables.
PDI-20455
NullPointerException encountered when trying to copy files using a VFS connection, combining TGZ notation and PVFS
PDI-20435
PDI fails to start when located in a folder with spaces in the path
PDI-20410
Performance degradation when using more than one Table Input on a transformation
PDI-20403
NullPointerException in Python Executor step during JSON parsing
PDI-20395
Error with Pentaho Reporting Output in PDI if a Groovy scripted datasource is used in the report: Caused by: org.apache.bsf.BSFException: unable to load language: groovy
PDI-20387
Azure ADLS2 VFS Block Blob fails when writing large files (> block size)
PDI-20377
As an ETL developer, I would expect Spoon to select data types that do not cause data corruption.
PDI-20372
Performance drag when comparing Pentaho PDI 9.3, 10.2.0.2 to Pentaho PDI 6.1
PDI-20371
As an ETL developer, I would expect the 'Discover metadata from a Text File' step to produce field sizes on integers that match the fields.
PDI-20370
Poor performance copying files from S3 to S3 using PVFS and MultiPartUpload
PDI-20369
As an ETL developer, I would expect the 'Discover metadata from a Text File' step to produce field sizes on strings.
PDI-20364
PDI (Kitchen) sporadically runtime error: 'Central Log Store is not initialized'
PDI-20342
Reporting Output step fails to create DataCacheFactory resulting in performance regression
PDI-20334
Transformation with Metadata Injection step and "Optional target file" field set causes deadlock, leading to server hang
PDI-20322
Disable gather performance metrics when scheduling jobs/KTRs from Spoon's schedule perspective
PDI-20311
Using "Run SSH commands" step to connect to destination with "OpenSSH_8.9p1 Ubuntu-3ubuntu0.10, OpenSSL 3.0.2 15 Mar 2022": fails
PDI-20301
UnsupportedEncodingException occurred while using Load file content in memory
PDI-20292
Record count is not accurately determined in the Text File Input step when multiple input files are used.
PDI-20262
Uploaded PDI job/transformation in PDI and saved in repository has a file size of 0 on the Properties screen
PDI-20219
Charts do not function in DET in 9.3.0.8
PDI-20206
Spoon PDI client: user unable to connect to repository if password has special characters and server is configured to use LDAP
PDI-20195
Inconsistent variable names recently created
PDI-20190
Issues with encoding when reading files with charset different from UTF-8
PDI-20183
Clustered Mapping (sub-transformation) step run on a Carte cluster causes the job to hang in a Running state if the next step is not clustered
PDI-20171
Modified JavaScript Value step in PDI 9.3.0.8 fails to terminate when setting variables
PDI-20154
Spoon crashes with NullPointerException when connected with Power User role
PDI-20135
HTTP Post step does not process configured file paths correctly
PDI-20106
Memory leak when accessing jobs and transformations concurrently on repository
PDI-20035
When the underlying MDI template steps are removed, errors persist.
PDI-20030
Move Files job entry does not move large files from S3 bucket to S3 bucket, and is missing the partSize logic
PDI-19926
Put a file with FTP: an error doesn't occur when local directory permissions are changed.
PDI-19803
Loss of functionality present in the PDI 'SFTP Put' step due to outdated jsch-0.1.54.jar
PDI-19550
The Carte jobStatus API was changed to always require a name parameter, even when a valid ID is provided
PDI-17941
Data Validator - Read allowed values from another step - Loses values
PDI-17634
Carte Server configured for SSL cannot be stopped from the command line
PDI-17311
Currency sign replaced by currency symbol (¤) specified in Regex Evaluation step errors out.
PDI-17310
Currency sign replaced by currency symbol (¤) specified in Generate Rows step errors out.
PDI-20172
Carte and Pentaho Server notify multiple TransFinished events when transformation fails or aborts
PDI-20161
Memory leak issues encountered on Pentaho 9.3.0.8 SP.
### Pentaho Business Analytics
The following issues have been addressed in the BA platform and Analyzer, Interactive Reporting, Report Designer, and Aggregation Designer components.
Issue ID
Description
BISERVER-15447
Bug - Schedule is displayed with server's timezone instead of browser's timezone, and it will not be triggered on time if the schedule is set up using browser's local timezone
BISERVER-15401
Post-upgrade task URL is still pointing to old documentation link.
BISERVER-15378
PUC - Help > hitachivantara.com... should be Help > pentaho.com...
BISERVER-15303
Run Once scheduler jobs do not terminate after running once.
BISERVER-15305
Cannot schedule blockout jobs in PUC; they get created as normal scheduled jobs
PPP-5768
Start time for monthly jobs is incorrect
BISERVER-15291
Wrong table used on migrate_old_quartz_data scripts
BISERVER-15279
Upgrade from 10.2.0.2 to 10.2.0.3 breaks due to large pentaho-mapreduce-libraries.zip file.
BISERVER-15277
Start/end date reflected in PUC does not consider the time zone when the scheduler is created from PDI
BISERVER-15180
Bug - Duplicate parameters are added to parameter dialog when clicking Tab to leave parameter's textbox
BISERVER-15216
Run Once - Next button disabled by default on Scheduler Page Two in PUC
BISERVER-15176
User-defined internal variables are crossing between schedules
BISERVER-15174
StringIndexOutOfBoundsException displaying schedules when using migration script
PPP-5469
The upgrade utility is ignoring the BACKUP_ROOT_PATH variable
PPP-5466
Upgrade utility does not detect /tmp correctly
PPP-5380
License screen for marketplace images shows no licenses.
BISERVER-15124
Enabling the RepositoryCleanerSystemListener on 10.2 causes server to not start.
BISERVER-15051
Bug - Scheduler showing internal variable usage from other jobs and cannot be cleared.
BISERVER-14374
Schedule is displayed with server's timezone instead of browser's timezone, and it will not be triggered on time if the schedule is set up using browser's local timezone
PRD-6184
HTML streaming format fails to generate report
PMD-1128
In macOS Ventura, unable to use PME
BISERVER-15168
Restored schedules from previous versions are missing on latest scheduler-plugin
BISERVER-15167
Schedules are lost during backup-restore procedure even when restore is successful on same version of Pentaho Server
ANALYZER-4150
Format MDX expression cannot be applied to measures on Analyzer UI
BISERVER-15148
Fresh installation of 10.2.0.0 and 10.2.0.1 will give scheduling errors
ANALYZER-4137
Reintroduce ability to delete filter and column when clicking OK on the alert that appears when deleting column from report that also has Top 10 numeric filter based on it
BISERVER-15406
PUC Administration - Operation permission revert fails – server ignores latest change and sends empty array
BISERVER-15407
CanUpload, CanDownload, and UploadDownloadAndScheduleKjbFunctionalTest automation tests failing with permissions errors
ANALYZER-4177
Column resizing in Analyzer reports inconsistent due to trash can hover zone interfering
PPP-5748
Intermittent redirect to CDF JS script instead of login or home page
PPP-5722
Logging in after session timeout redirects to RequireJS config file
PIR-1547
Control Type Selector for PIR Prompt Definition not displayed in Sapphire and Crystal themes
ANALYZER-4155
Subtotals not being displayed in Analyzer reports migrated from v9.3.0.0 to v10.1 and up.
BISERVER-15190
FileService throws NPE on no access, not found, or a depth of 0
Random redirection to .js or .json files instead of Pentaho Home
BAD-1992
BIG DATA VFS [GCS] - Parquet/ORC IO throws an error when running or executing preview data functionality
BAD-1972
[Bigdata-EMR] EMR cluster is shut down on Amazon Hive Executor step failure when using existing cluster option.
BAD-1971
Amazon Job Executor and Hive Job Executor step do not work due to a change in Amazon API requirement
PPP-5904
Licensing exception because en_ZA locale is not found when trying to use a LLS server
PPP-5706
Spoon 10.2.0.3 crashes when using an expired offline license
PPP-5673
Local License Server is showing used counts of 1040 PDI cores on one client and 48 PDI cores on another client even though both servers only have 8 CPU cores
Licensing - Regression - PUC/PDI - Allows incorrect URL/Activation ID to corrupt an existing validated license
PPP-5598
Existence of unnecessary license libraries is causing an issue with some Mondrian features
BISERVER-15218
IMPORT/EXPORT UTILITY - the listing for emails and groups does not display in logs
PPP-5557
Re-evaluate the host ID implementation to ensure it remains consistent across machine restarts
PIR-1542
Unable to export PIR report data (HTML, CSV, PDF, Excel, Text) in PUC
PPP-5463
The Local License Server requires /tmp to allow executions
BISERVER-15169
There is no useful information that can aid debugging import-export utility
PPP-5378
.elmLicInfo.plt file gets corrupted by switching License Server providers and causes downtime
PPP-5283
Expired Offline Trial License File (.bin) does not show any information about expiration dates
PPP-5264
Licensing - Offline license files show trial expiry even though the trial duration has not been exceeded
PIR-1513
Database query is not cancelled when clicking Cancel during the run/build of an Interactive Report
BAD-1942
[Bigdata|CDP71SecureJ11] YARN job fails when executing Start a PDI Cluster on YARN "locally" leaving PDI Client Archive field blank.
BISERVER-15275
Jobs and transformations are hidden by default when using PUC upload facility
BISERVER-15420
Errors occur due to missing Java opens on Pentaho Executable installer
BISERVER-15271
NoClassDefFoundError: com/zaxxer/sparsebits/SparseBitSet error when executing a transformation with Microsoft Excel Writer step using .xls format and "shift existing cells down" setting on Pentaho Server.
BISERVER-15247
Files written to Local VFS are locked at the OS level
BISERVER-15191
[Bigdata] PMR job fails with ApacheVanilla shim.
BISERVER-15184
[Bigdata] Parquet Input step fails to read files from Azure Cloud Storage (ADLSGen2)
BAD-1961
[Bigdata] Pentaho MapReduce job to read/write from Cloud Storage (S3) fails with emr700 shim
PRD-6169
Cannot preview or export large MS Excel files with PRD and the latest POI
ANALYZER-4144
Large table reports cannot be exported to MS Excel based on changes to POI
BAD-1955
[Bigdata] ORC input/output steps fail when connected to EMR cluster
BISERVER-15128
Excel reports generated using newer Apache POI library leave larger files on Tomcat's temp directory
PIR-1529
PIR Save/Open fails when a calculated field exists
PIR-1528
FATAL ERROR when using calculated fields (with > or < and no space between field and operator) in Pentaho v10.1 Interactive Reports
BISERVER-15414
Pentaho Server does not start if the server folder path has spaces in the name
PIR-1568
PIR 10.2.0.6 – Firefox: Reports with pipe symbol “|” in the saved file name are not editable
BISERVER-15379
Older and newer AWS JARs exist together after applying SP 10.2.0.3.
BISERVER-15374
Inconsistent JAR deployment between incremental and direct SP 10.2.0.5 installation.
PRD-6224
Report Designer: Numeric format not applied to Excel and PDF exports.
PMD-1134
Unable to start Metadata Editor OSGi version; startup gets stuck.
BISERVER-15349
Simple trigger schedules display “undefined” after upgrade to 10.2.0.2 and above
BISERVER-15341
Cannot use "-" dash character on username when security is Jackrabbit
BISERVER-15339
10.2.0.4 introduced regression in Karaf that breaks SAML plugin
BISERVER-15302
Tooltips do not work for 'Browse files' when browser page is reloaded in 10.2
BISERVER-15276
Schedules created via Spoon are missing after restoring the Pentaho Repository using import-export utility.
ANALYZER-4174
PDF export of line (and similar) graph shows inaccurate time horizon on X axis
PRD-6195
Change the error message that is displayed after applying the fix on PRD-6182
BISERVER-15257
Pentaho import issue with special characters in filenames
PPP-5648
Import-export does not import all schedules correctly
PIR-1550
Fix Interactive Reporting plugin unit test (handleMissingFields)
PRD-6186
PRD query editor resets back to the default value of disable_distinct after making changes
ANALYZER-4166
HTTP Status 400 - Bad Request error with Filter link when using German, French and Japanese language in Pentaho 10.2
BAD-1967
[Bigdata] Parquet and ORC Input Output steps fail to read/write from S3 when running jobs using kitchen.sh
PRD-6182
PRD query editor does not correctly parse complex conditions and throws a NullPointerException
BISERVER-15219
Default HyperSQL (HSQLDB) in-memory DB start script with error
BISERVER-15211
PIR in Edit mode gets frozen after pressing Esc after selecting "Select ..." in the General tab of Template Selector dialog
BISERVER-15243
Unable to log in with new non-admin user with a password that includes all special characters on the keyboard Ç-[~!@#$%^&*(){}|.,]-=_+|;'"?<>~`:
PPP-5546
Ops Mart: Generate_DIM_DATE transformation hardcoded to generate dates for about 20 years starting on 20050101
PRD-6179
PRD - Unable to connect to MongoDB in Pentaho Report Designer v10.x due to missing pentaho-mongo-utils.jar
PRD-6177
With Java 17, the property settings in "Edit Chart" within Report Designer are no longer editable.
BISERVER-15179
Revision required for Details section of New Schedule dialog on Scheduler plugin
BISERVER-15178
appendDateFormat attribute is treated as parameter or variable under Scheduler perspective
BISERVER-15173
StringIndexOutOfBoundsException displaying schedules with many variables on them
PRD-6173
RowLimit while exporting Excel for PRD report does not work in PUC
BISERVER-15161
Schedule a Report - Trigger Execute Now - Last run column is not updated
PIR-1538
Manage Data Sources - Delete a field from a datasource present in a saved PIR report (column and group) - Only one message appears
BISERVER-15156
Multiple report executions result when using Execute Now on daily recurrence
BISERVER-15155
File size in KB is calculated through division by 1000 instead of 1024
ANALYZER-4149
Percentage measure displayed incorrectly on the Trend Line in Pentaho v10.2
BISERVER-15147
Shell Job entry step invoking Perl script fails
BISERVER-15145
Prevent the creation of usernames with leading/trailing whitespaces
BISERVER-15144
Invalid database connection is saved and used despite "Cancel"
BISERVER-15139
"Files" text box displays an ellipsis (...) when the file name is very long
BISERVER-15135
Username with prefix or suffix blank spaces on login page triggers UsernameNotFoundException, but session is created
ANALYZER-4145
Multiselect feature for Analyzer report filter no longer requires holding CTRL key
ANALYZER-4143
Analyzer reports are re-executed when saving and editing, creating performance issues
BISERVER-15125
Sparkl (App Builder) plugin not available for Pentaho 9.3 or 10.2
BISERVER-15101
Schedule a Report - Execute Now after report is run for the first time and in the next minute - Reports are generated twice
PPP-5127
Turn autoDeploy from true to false in Tomcat's server.xml due to security concerns
BISERVER-15098
PUC: folders don't update the Last Modified date
PAD-179
ODBC access type does not work and should be removed from connection dialogs
PSW-282
ODBC access type does not work and should be removed from connection dialogs
PMD-1119
ODBC access type does not work and should be removed from connection dialogs
PRD-6162
ODBC access type does not work and should be removed from connection dialogs
PDB-2096
View Mode wrong tooltip text
PIR-1524
Load PIR in blank without error message, InteractiveAdhocReportUtils.ERROR_0002 - Unable to load report query
BISERVER-14919
PRPT any report using formula calculation fails when running Pentaho 9.4 in a docker container or server using UTC timezone
PRD-6138
Bar codes not displayed in Excel or HTML (single-page) report output for newly created reports
BISERVER-13909
Uploaded PDI Job/Transformation has file size of 0 on the Properties screen
MONDRIAN-2752
UserDefinedFunction scripting capability broken due to removal of Nashorn scripting engine from JDK
MONDRIAN-2751
MemberFormatter scripting capability broken due to removal of Nashorn scripting engine from JDK
MONDRIAN-2750
PropertyFormatter scripting capability broken due to removal of Nashorn scripting engine from JDK
MONDRIAN-2749
CellFormatter scripting capability broken due to removal of Nashorn scripting engine from JDK
MONDRIAN-2646
Mondrian does not cache empty children requests - generates numerous queries
PRD-6156
Strange behavior of parameter text box with parameterized query
PDB-2103
Prompts are not in sequence
BISERVER-14394
PUC - XML parsing error: no root element found - Firefox
BISERVER-14274
Schedule: error in console for more than 25 schedules
## Security Issues Addressed
The following vulnerabilities have been addressed in 11.0. Several vulnerabilities are already addressed in 10.2 service packs. These fixes are also included in 11.0 but not listed here. Review the relevant service pack notes for details.
CVE ID
Details
CVE-2020-17521
https://nvd.nist.gov/vuln/detail/CVE-2020-17521
CVE-2025-55752
https://nvd.nist.gov/vuln/detail/CVE-2025-55752
CVE-2025-30065
https://nvd.nist.gov/vuln/detail/CVE-2025-30065
CVE-2025-41248
https://nvd.nist.gov/vuln/detail/CVE-2025-41248
CVE-2025-41249
https://nvd.nist.gov/vuln/detail/CVE-2025-41249
CVE-2025-22228
https://nvd.nist.gov/vuln/detail/CVE-2025-22228
CVE-2025-27820
https://nvd.nist.gov/vuln/detail/CVE-2025-27820
CVE-2015-6420
https://nvd.nist.gov/vuln/detail/CVE-2015-6420
CVE-2015-1832
https://nvd.nist.gov/vuln/detail/CVE-2015-1832
CVE-2009-4611
https://nvd.nist.gov/vuln/detail/CVE-2009-4611
CVE-2025-46762
https://nvd.nist.gov/vuln/detail/CVE-2025-46762
CVE-2021-33813
https://nvd.nist.gov/vuln/detail/CVE-2021-33813
CVE-2025-52099
https://nvd.nist.gov/vuln/detail/CVE-2025-52099
CVE-2025-9230
https://nvd.nist.gov/vuln/detail/CVE-2025-9230
CVE-2024-6162
https://nvd.nist.gov/vuln/detail/CVE-2024-6162
CVE-2025-35036
https://nvd.nist.gov/vuln/detail/CVE-2025-35036
CVE-2023-5072
https://nvd.nist.gov/vuln/detail/CVE-2023-5072
CVE-2024-1635
https://nvd.nist.gov/vuln/detail/CVE-2024-1635
CVE-2024-7885
https://nvd.nist.gov/vuln/detail/CVE-2024-7885
CVE-2023-4639
https://nvd.nist.gov/vuln/detail/CVE-2023-4639
CVE-2023-5685
https://nvd.nist.gov/vuln/detail/CVE-2023-5685
CVE-2023-1973
https://nvd.nist.gov/vuln/detail/CVE-2023-1973
CVE-2024-47554
https://nvd.nist.gov/vuln/detail/CVE-2024-47554
CVE-2024-5971
https://nvd.nist.gov/vuln/detail/CVE-2024-5971
CVE-2024-57699
https://nvd.nist.gov/vuln/detail/CVE-2024-57699
CVE-2025-6297
https://nvd.nist.gov/vuln/detail/CVE-2025-6297
CVE-2023-31484
https://nvd.nist.gov/vuln/detail/CVE-2023-31484
CVE-2023-45853
https://nvd.nist.gov/vuln/detail/CVE-2023-45853
CVE-2025-6020
https://nvd.nist.gov/vuln/detail/CVE-2025-6020
CVE-2024-4741
https://nvd.nist.gov/vuln/detail/CVE-2024-4741
CVE-2024-56406
https://nvd.nist.gov/vuln/detail/CVE-2024-56406
# What's new in Pentaho 10.2
Version 10.2 of Pentaho Data Integration and Analytics Enterprise Edition delivers enhancements across Pentaho Data Integration (PDI) and Pentaho Business Analytics (PBA) that includes both usability improvements and performance across several components.
Along with these features and enhancements, Pentaho 10.2 is a Long Term Support (LTS) release. LTS refers to a widely adopted release that provides stability and flexibility for customers to upgrade when best suited. This type of release is for customers who have rigid environments and cannot frequently upgrade.
The key features in this release are:
## Java 17 support
Use Pentaho now built on Oracle Java 17 and Open JDK Java 17. The following Java 17 support benefits are key to Pentaho:
* Oracle Java 17 is available under a new license that permits free production use.
* Java 17 has several performance improvements over Java 11.
See [Java virtual machine](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia000/components-reference/java-virtual-machine) for details.
## License manager enhancements
Enhance your license manager interactions with the following capabilities:
* Re-introduced support for offline license files. If you are in an air-gapped environment, you can deploy licenses without the need for any license server. Additional audit costs may apply. Please contact your Pentaho representative for your license management options. You can still use Pentaho maintained cloud License Servers or install Local License Servers with no additional audit costs.
* Separate entitlements for development or production usage. You can track and control usage across these two types of scenarios within your environment.
* Separate PDI client (ETL authoring) entitlements. You can track and control the use of PDI within your environment independently from other Pentaho components. These entitlements are instance-based as opposed to other Pentaho components, which are core-based.
* The borrow-time (how often licenses are validated) has now been increased to reduce the number of license validation calls made by Pentaho components to further minimize resource utilization due to license validation.
* Enhancements to enable OEMs to self-manage entitlements to their customers.
See [Acquire and install enterprise licenses](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia002/manage-the-pentaho-system/manage-pentaho-licenses/acquire-and-install-enterprise-licenses) for instructions on using the license manager.
## New charts available for Analyzer reports
In your Analyzer reports, visualize progress or status of a specific metric or goal with the new Gauge E-chart or show a comparison of multiple categories across several variables with the new Radar E-chart. See [Gauge chart](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-analyzer/creating-analyzer-reports/visualizations-for-analyzer/gauge-chart) and [Radar chart](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-analyzer/creating-analyzer-reports/visualizations-for-analyzer/radar-chart) for details.
## CBC support for AES Encryption
Choose between the existing (default) Electronic Code Book (ECB) or the new Cipher Block Chaining (CBC) implementation of the Pentaho Advanced Encryption Standard (AES) security protocol. See [AES security](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia002/secure-the-pentaho-system/user-security/advanced-security-providers/aes-security) for details.
## Scheduling Improvements
Improve your scheduling experience with the following enhancements:
* **Parameter information**
A new **Parameter** column added to the schedule management listings in Pentaho User Console (PUC) and Pentaho Data Integration (PDI). See [PUC Schedules](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/schedules) and [Scheduler perspective in the PDI client](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia003/scheduler-perspective-in-the-pdi-client) for details.
* **Schedule Power User role**
A new role designed for a user who mostly executes schedules. See [Manage Users and Roles in PUC](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/administration/manage-users-and-roles-in-puc) for details.
* **VFS-based scheduling updates**
Virtual File System (VFS) connections added to PUC **Browse File** perspective, VFS access based on roles, and specification of a VFS root folder path. See [Set up a VFS location for schedule outputs](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/schedules/set-up-a-vfs-location-for-schedule-outputs) for details.
* **E-mail address updates**
External LDAP and JDBC support added for syncing and improved search functionality for e-mail addresses. See [Importing and updating email addresses used for scheduling from data sources](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia002/manage-the-pentaho-system/manage-the-pentaho-server/importing-and-updating-email-addresses-used-for-scheduling-from-data-sources) and [Create an email group](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/schedules/schedule-reports/set-up-emails-for-scheduled-reports/create-an-email-group) for details.
* **Report date**
New reporting date added to scheduled email attachment filename. See [Schedule a report](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/schedules/schedule-reports/schedule-a-report) for details
## Support for UNC Paths (SMB)
Connect to your Server Message Block (SMB) files through VFS in both PUC and PDI. See [Set up a VFS location for schedule outputs](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia004/pentaho-user-console/about-pentaho-user-console-perspectives/schedules/set-up-a-vfs-location-for-schedule-outputs) for PUC details and [Connecting to Virtual File Systems](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia003/data-integration-perspective-in-the-pdi-client/connecting-to-virtual-file-systems) for PDI details.
## AWS Athena available as a data source
Access Amazon Web Service (AWS) Athena as a data source in both PUC and PDI. See [AWS Athena](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia001/jdbc-drivers-reference/aws-athena) for details and [Define data connections](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia001/pentaho-configuration/tasks-to-be-performed-by-a-pentaho-administrator/define-data-connections) for instructions on adding a database connection.
## Support for Apache Vanilla Hadoop
Directly connect to plain Hadoop clusters (commonly known as Apache Vanilla Hadoop clusters) through the new Apache Vanilla Hadoop driver. See [Using the Apache Vanilla Hadoop driver](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia003/advanced-pentaho-data-integration-topics/connecting-to-a-hadoop-cluster-with-the-pdi-client/using-the-apache-vanilla-hadoop-driver) for details.
## Data formatting in PDI logging output
Specify the date used in PDI logging with the **Pattern** parameter in the `log4j2.xml` file. See [Set up the log](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia002/optimize-the-pentaho-system/performance-monitoring/pdi-logging/set-up-the-log-file) file for details.
## PDI transformation steps enhancement
The following PDI transformation steps have been modified for this release:
* **Microsoft Excel Writer**
This step has been improved to include functionality from the Microsoft Excel Output step, which is now deprecated in favor of the Microsoft Excel Writer step. See [Microsoft Excel Writer](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia003/pdi-transformation-steps/microsoft-excel-writer) for details on this enhanced step.
* **Cassandra Input and Output moved to the Marketplace**
These steps were moved to the Marketplace to relieve a vulnerability to the PDI core. Contact [Pentaho Support](https://support.pentaho.com/) for details.
## Improved Excel export performance from Pentaho Report Designer
Improve output performance with support of Excel XLSX export. See [Microsoft Excel report takes a long time to generate](https://docs.hitachivantara.com/r/en-us/pentaho-data-integration-and-analytics/10.2.x/mk-95pdia008/report-designer-and-reporting-engine-issues/microsoft-excel-report-takes-a-long-time-to-generate) for details.
# Pentaho EE Marketplace Plugins
The plug-ins listed on this page are available from the [Support Portal](https://support.pentaho.com/home) home page.
## Pentaho 10.0.2
You can use the Salesforce bulk operation step plug-in for bulk operations on Salesforce objects. This step can significantly increase performance of Salesforce operations.
With the Bulk load into Databricks job entry, you can load large amounts of data from files in your cloud accounts into Databricks tables.
## Pentaho 10.0.1
You can use the new Google Analytics v4 plug-in to generate reports and populate your data warehouse.
A new Hitachi Data Connector for SAP and Business Warehouse plug-in is available. This new plug-in is the ideal Pentaho Data Integration plug-in to use SAP data in your data integration work flows. It enables the querying and extraction of complex and nested SAP structures. The plug-in supports both full and incremental (delta) load scenarios and bi-directional (read from and write to SAP) transfers. In addition, it integrates with SAP security allowing you to manage access within your existing setup. For more information, ask your customer support representative.
## Pentaho 10.0
The Elasticsearch REST Bulk Insert step now supports Elasticsearch version 8 and has been made available as a Pentaho EE Marketplace plug-in.
## Pentaho 9.5
For the Pentaho EE 9.5 GA release, two new plugins were released as part of the Pentaho EE Marketplace Plugin release with new features to improve your data management operations. The plugins are available from the [Support Portal](https://support.pentaho.com/home) home page. Sign into the portal using the Pentaho support username and password provided in your Pentaho Welcome Packet. The plugins are:
* A hierarchical data plugin that adds five new steps for working with hierarchical data.
* A Kafka streaming plugin that enhances the Kafka consumer and producer steps and adds a **Kafka Offset** job with the ability to reset the offset.
## Hierarchical data type steps
Pentaho has added an hierarchical data type, and has five new steps for processing structured, complex, and nested data types. This new data type is supported in steps in previous releases of Pentaho that can handle hierarchical data. These five new steps consist of the following:
* **Hierarchical JSON Input**
This step accepts a JSON file or JSONL from a previous step or a file location and converts it into a hierarchical object.
* **Hierarchical JSON Output**
This step accepts hierarchical data from a previous step and converts it into a JSON formatted string.
* **Extract to Rows**
This step parses hierarchical data from input steps.
* **Modify values from a single row**
This step modifies the hierarchical data using incoming columns or create hierarchical data output to another step.
* **Modify values from grouped rows**
This step builds complex hierarchical data or group data based on a field.
The last three steps are used within the transformation flow for interacting with the hierarchical data type structure in-place, without needing to flatten the data to a row structure.
## Kafka Improvements
A new job entry called Kafka Offset has been added to enable you to change the offset of a topic partition. This Job entry has fields to connect to a Kafka broker or cluster in the **Setup** and **Options** tabs.
The following improvements have been made to the Kafka Consumer and Kafka Producer steps:
* Encryption is supported for connection parameters.
* SSL and Kerberos (SASL) connectivity have been certified.
* You can now use variables from Kettle properties, PDI environment variables. and parameter variables in the Kafka properties settings on the **Options** tab.
* The Kafka client library has been upgraded to 3.4.0.
* Logging has been improved to make debugging easier.
* Improved the Kafka consumer step to consume messages until the time stamp set using the **Offset Settings** tab in the Kafka Offset job.
* An offset rebalancer has been added to correctly commit offsets if a rebalance occurs when a new consumer is added or an existing consumer removed from the consumer group.
# What's new in Pentaho 9.3
The Pentaho 9.3 Enterprise Edition delivers a variety of features and enhancements, including the Java upgrade to version 11, Docker container deployment, and component upgrades. Pentaho 9.3 also continues to enhance the Pentaho business analytics experience.
Along with these features and enhancements, Pentaho 9.3 is a Long Term Support (LTS) release. LTS refers to a widely adopted release that provides stability and flexibility for customers to upgrade when best suited. Such a release is for customers who have rigid environments and cannot frequently upgrade.
## Java 11
With Oracle's support of Java 8 ending, Pentaho 9.3 is built on the latest release of Java, version 11. See the **Components reference** in the **Try Pentaho Data Integration and Analytics** document for more information.
## Docker container deployment
You can now create and deploy Docker containers of Pentaho products with a command line tool. For example, you can use your Pentaho Server, based on your licensed Pentaho software installation, to create a standardized Docker container with data integration, business analytics, and Carte server components. See the **Install Pentaho Data Integration and Analytics** document for more information.
## MongoDB improvements
You can now use MongoDB Atlas string format to connect Pentaho to MongoDB. With this type of string, you no longer need to specify all your cluster members in the connection. Also, you can now continue processing aggregation pipeline data when it exceeds the standard 100MB RAM allocation. See the MongoDB Input anf MongoDB Output steps in the **Pentaho Data Integration** document for more information.
## Component upgrades
Pentaho 9.3 contains many upgraded components, including Snowflake, ElasticSearch, IBM MQ, Postgres, Oracle, SQL Server, Windows Server, MacOS 11 Big Sur, and Ubuntu. See the **Components reference** in the **Try Pentaho Data Integration and Analytics** document for more information.
## Security improvements
Log4j has been upgraded to version 2.17.1 to improve logging and address Log4j security issues. See [Pentaho Lumada and Pentaho Support Portal](https://support.pentaho.com/hc/en-us) for more information.
## Reduced Pentaho installation download size and improved startup time
The download size of the Pentaho installation is reduced by almost half, leading to a smaller footprint and quicker download time. Pentaho's start time is also significantly improved now that only your specific drivers are loaded at startup instead of the entire set. See the **Install Pentaho Data Integration and Analytics** document for instructions on downloading and adding your specific drivers.
## Analyzer enhancements
Pentaho 9.3 includes the following Analyzer improvements:
* **Query performance inprovement**
Improved Analyzer's caching of queries to increase performance.
* **Improved handling of daylight savings when scheduling reports**
The new **Ignore daylight saving adjustment** option permits your report to run at least once every 24 hours, regardless of daylight saving. See **Schedule a report** in the **Pentaho Business Analytics** document for more information.
* **Enable or disable report options at the server level**
The new **Show Empty Rows where the Measure cells is blank** property improves performance by avoiding cross-joins. See **Control empty rows in reports** in the **Pentaho Business Analytics** document for more information.
* **Select parameter values when scheduling reports**
You can now add parameters values when scheduling an Analyzer report from Pentaho User Console and set default values at the time a report is scheduled to run. See **Add query parameters to Analyzer reports** in the **Pentaho Business Analytics** document for more information.
* **Drill up or down a dimension**
You have new options for drilling up or down a dimension. See **Working with Analyzer fields** in the **Pentaho Business Analytics** document for more information.
* **Export an Analyzer report as a JSON file through a URL**
You can now export an Analyzer report as a JSON file from a Pentaho repository through a URL, which can be useful when you want to export reports from a different scheduler. See **Export an Analyzer report through a URL** in the **Pentaho Business Analytics** document for more information.
# Try Pentaho Data Integration and Analytics
Learn how to install an evaluation version of Pentaho Data Integration and Analytics, then get started with basic concepts, walk-throughs, and workflows.
Pentaho Data Integration and Analytics is a platform for access and analytics. It supports flat files, relational databases, Hadoop, NoSQL, and cloud sources. Use it to integrate, transform, visualize, and analyze data. Use the APIs to extend reports, queries, and transformations.
## Install the 30-day trial of Pentaho Data Integration and Analytics
The Pentaho Installation Wizard is a streamlined way to install, learn about, and evaluate the Pentaho Suite. With this 30-day trial, you can install and test Pentaho Business Analytics (BA) and Data Integration (DI).
To get started fast after installation, follow the [Pentaho Data Integration (PDI) tutorial](/pdia-try-pdia/archive-flattened-pages/pdi-transformation-tutorial).
If you have issues with the 30-day trial, contact the [Pentaho Trial Experts](mailto:trialexperts.pentaho@hitachivantara.com).
{% hint style="warning" %}
**Important**: Moving to development or production is not supported for installations completed with the Pentaho Installation Wizard.
{% endhint %}
With the Pentaho Installation Wizard you can choose one of two install types:
* **Default**: Select **Keep it simple. Give me everything.**
* **Custom**: Select **Let me decide for myself.**
### Process overview
Installation instructions are the same for Windows, Linux, and macOS.
Complete these tasks in order:
1. [Download the trial software](#download-the-trial-software)
2. [Start the Pentaho Installation Wizard](#start-the-pentaho-installation-wizard)
3. Choose a [default installation](#default-installation) or [custom installation](#custom-installation).
4. [Verify installation](#verify-installation)
5. [Getting started tutorial](#getting-started-tutorial)
| Explore Considerations | |
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You Supply | A computer that meets Pentaho's [operating system and hardware requirements](/pdia-try-pdia/components-reference). |
| We Supply |
Installation Package
Oracle JRE
Repository Platform (PostgreSQL)
Repository Platform JDBC Driver (PostgreSQL)
Web Application Server (Tomcat)
|
| Technologies Used |
Tomcat web application server
PostgreSQL database to house the Pentaho Repository
|
| Expertise |
Basic computer knowledge.
|
| Approximate Installation Time | 30 minutes |
### Download the trial software
To download the software, complete these steps.
1. Make sure you are logged in to the computer where you want to install the software.
You should use an account that can install software.
* On Windows, this is typically an account with administrator privileges.
* On Linux, this is also an account with administrator privileges.
2. Navigate to the [Pentaho trial download](https://www.hitachivantara.com/en-us/products/data-management-analytics/pentaho/download-pentaho.html) website.
3. Click **Start Your 30-Day Trial**.
The **Submission Agreement** form appears.
4. Fill in the requested information and click **Submit**.
The **Getting Started with**
\
**Pentaho Data Integration & Analytics** page opens.
5. In the **On-Prem** section, click **Download** for your operating system.
6. When prompted, choose a directory for the installer and wait for the download to finish.
7. If you are using Linux, make sure you can execute the file.
Open a terminal, go to the directory where you downloaded the file, then run:
```
chmod a+x ./pentaho-business-analytics-10.3.0-x64.bin
```
### Start the Pentaho Installation Wizard
To start the installation wizard, complete the following steps.
{% hint style="info" %}
**Note:** Launch the installation wizard from a locally mounted hard drive only. Network-mounted drives (for example NFS) are not supported.
{% endhint %}
Do one of the following.
* **Windows**: Open File Explorer, go to the installer, then double-click `pentaho-business-analytics-10.3.0-x64.exe`.
* **macOS**: Unpack `pentaho-business-analytics-10.3.0-x64.app.tar.gz`, then double-click `pentaho-business-analytics-10.3.0-x64.app`.
* **Linux (graphics)**: Open a terminal, go to the installer, then run:
```
./pentaho-business-analytics-10.3.0-x64.bin
```
* **Linux (no graphics)**: Open a terminal, go to the installer, then run one of these:
* GTK text mode:
```
./pentaho-business-analytics-10.3.0-x64.bin --mode text
```
* OpenMotif X mode:
```
./pentaho-business-analytics-10.3.0-x64.bin --mode xwindow
```
### Default installation
If you want to install every component and some sample data, complete the following steps. If you want a custom install, see [Custom installation](#custom-installation).
{% hint style="warning" %}
**Important:** You cannot install into a directory that already exists. On Linux, do not install under `/opt/`. It can cause permissions problems.
{% endhint %}
1. After you start the wizard, the splash screen appears, then the **Pentaho Business Analytics installation is ready!** window. Click **Next**.
2. Read the license agreement. Select **Accept**, then click **Next**.
3. In **Installation folder**, accept the default directory or enter a different path, then click **Next**.
4. You are prompted to add a PostgreSQL password. Enter and confirm the password for the `postgres` user.
Important: Do not use these characters in the password: ' " & < > \
5. Click **Next**.
6. In the **What do you want to install?** window, select **Keep it simple. Give me everything**, then click **Next.**
7. If the **Sample Database** window appears, enter a port number, then click **Next**.
This prompt appears only if the default ports are not available.
8. In the **Pentaho License Information** window, leave the box empty and click **Next**. A 30-day license is installed for the trial.
9. In **We're Set. Let's install!** window, click **Next**. Installation begins and takes about 30 minutes to complete.
10. When installation is complete, select **Pentaho Data Integration** to launch Spoon.
11. Click **Finish**.
After PDI starts, you will see the **Welcome!** window.
To get started quickly after installation, follow the [Pentaho Data Integration (PDI) tutorial](/pdia-try-pdia/archive-flattened-pages/pdi-transformation-tutorial).
If you have trouble with the installation wizard, see the [Administer Pentaho Data Integration and Analytics](https://docs.pentaho.com/pdia-admin/) document.
### Custom installation
If you want to choose which components to install and whether to install sample data, complete the following steps. If you want the default install, see [Default installation](#default-installation).
{% hint style="warning" %}
**Note:** You cannot install into a directory that already exists. On Linux, do not install under `/opt/`. It can cause permissions problems.
{% endhint %}
1. After you start the wizard, the splash screen appears, then the **Pentaho Business Analytics installation is ready!** window. Click **Next**.
2. Read the license agreement. Select **Accept**, then click **Next**.
3. In **Installation folder**, accept the default directory or enter a different path, then click **Next**.
4. You are prompted to add a PostgreSQL password. Enter and confirm the password for the `postgres` user.
Important: Do not use these characters in the password: ' " & < > \
5. Click **Next**.
6. If the **Sample Database** window appears, enter a port number, then click **Next**.
This prompt appears only if the default ports are not available.
7. In the **What do you want to install?** window, select **Let me decide for myself**, then click **Next**.
8. Select the components you want to install, then click **Next**.
9. Select whether to include sample content, then click **Next**.
10. If the **Sample Database** window appears, enter a port number, then click **Next**.
11. In the **Pentaho License Information** window, leave the box empty and click **Next**. A 30-day license is installed for the trial.
12. In **We're Set. Let's install!** window, click **Next**. Installation begins and takes about 30 minutes to complete.
13. When installation is complete, select the components you want to launch and click **Finish**.
If you have trouble with the installer, see the [Administer Pentaho Data Integration and Analytics](https://docs.pentaho.com/pdia-admin/) document.
### Verify installation
To verify the installation, review the directory structure and the installation summary file.
1. Open a file explorer or command line tool.
2. Navigate to the install directory and compare it to the structure below.
```
pentaho/
pentaho/design-tools/
pentaho/design-tools/aggregation-designer/
pentaho/design-tools/data-integration/
pentaho/design-tools/metadata-editor/
pentaho/design-tools/report-designer/
pentaho/design-tools/schema-workbench/
pentaho/documentation/
pentaho/java/
pentaho/jdbc-distribution/
pentaho/license-installer/
pentaho/licenses/pentaho/monetdb/
pentaho/postgresql/
pentaho/scripts/
pentaho/server/
```
3. Open `installation-summary.txt`. Make sure it lists the design tools and plugins you installed.
4. Review the file locations below.
| File | Description |
| -------------------------- | ------------------------------------------------------------------------------------------------ |
| `ctlscript.sh` | Starts, stops, restarts, and shows the status of Pentaho services. Available on Linux and macOS. |
| `installation-summary.txt` | Contains the information from the summary screen at the end of the installation process. |
| `uninstall` | A script that removes Pentaho Business Analytics. |
| Tool/Plugin | Location |
| ------------------------ | --------------------------------------------------------------------------------------- |
| Pentaho Server | `pentaho/server/pentaho-server/` |
| Report Designer | `pentaho/design-tools/report-designer/` |
| Schema Workbench | `pentaho/design-tools/schema-workbench/` |
| Data Integration (Spoon) | `pentaho/design-tools/data-integration/` |
| Metadata Editor | `pentaho/design-tools/metadata-editor/` |
| Aggregation Designer | `pentaho/design-tools/aggregation-designer/` |
| Dashboard Designer | `pentaho/server/pentaho-server/pentaho-solutions/system/dashboards/` |
| Analyzer | `pentaho/server/pentaho-server/pentaho-solutions/system/analyzer/` |
| Interactive Reports | `pentaho/server/pentaho-server/pentaho-solutions/system/pentaho-interactive-reporting/` |
| License Installer | `pentaho/license-installer/` |
| Log | Location |
| --------------------------------------------------- | -------------------------------------------- |
| Pentaho Server Logs for BA configuration | `pentaho/server/pentaho-server/logs/` |
| Tomcat Logs for Pentaho Server for BA configuration | `pentaho/server/pentaho-server/tomcat/logs/` |
For macOS, copy the JDBC `.jar` into the location listed for Report Designer.
| JDBC Driver | Location |
| ----------------------------------- | ---------------------------------------------------- |
| Pentaho Server for BA configuration | `pentaho/server/pentaho-server/tomcat/lib/` |
| Report Designer | `pentaho/design-tools/report-designer/lib/jdbc/` |
| Schema Workbench | `pentaho/design-tools/schema-workbench/drivers/` |
| Aggregation Designer | `pentaho/design-tools/aggregation-designer/drivers/` |
| Metadata Editor | `pentaho/design-tools/metadata-editor/libext/JDBC/` |
| PDI client (Spoon) | `pentaho/design-tools/data-integration/lib/` |
| Port Number | Description |
| ----------- | --------------------------------------------- |
| 5432 | PostgreSQL Server |
| 8080 | Pentaho Server Tomcat Web Server Startup Port |
| 8012 | Pentaho Server Shutdown Port |
| 9001 | HSQL Server Port |
| 9092 | Embedded H2 Database |
Note: Your port numbers might differ. If you installed using the installation wizard, check installation-summary.txt for the actual ports.
### Getting started tutorial
To get started quickly after installation, follow the [Pentaho Data Integration (PDI) tutorial](/pdia-try-pdia/archive-flattened-pages/pdi-transformation-tutorial).
## Start and stop the PDI client on Windows
If you installed PDI on Windows using the installation wizard, start Spoon from: **Start** > **All Programs** > **Pentaho Enterprise Edition** > **Design Tools** > **Data Integration**.
## Start and stop the Pentaho Server for configuration on Windows
When you ran the installation wizard on Windows, the Pentaho Server deployed in an included Apache Tomcat application server. Manage Pentaho and Tomcat from: **Start** > **All Programs** > **Pentaho Enterprise Edition** > **Server Management**.
Use one of these menu items:
* **Start Pentaho Server**
* **Stop Pentaho Server**
The installer also registered services for:
* **Pentaho Server**
* **Data Integration**
* **Pentaho Repository**
You can start and stop these services from **Control Panel** > **Administrative Tools** > **Services**.
## Start and stop the Pentaho Server for configuration on Linux
When you ran the installation wizard on Linux, the Pentaho Server deployed in an included Apache Tomcat application server. Use `/pentaho/ctlscript.sh` to start and stop services.
Common arguments:
* `start`
* `stop`
* `restart`
* `status`
* `help`
Common services:
* `pentahoserver`
* `postgresql`
Examples:
```
./ctlscript.sh start pentahoserver
./ctlscript.sh status pentahoserver
./ctlscript.sh status postgresql
./ctlscript.sh help
```
## Adjust Java VM memory limits for an installation on Windows
Windows users might see out-of-memory errors. Increase the Java VM memory allocation to fix them.
{% hint style="info" %}
**Note:** These steps apply when you installed with the installation wizard.
{% endhint %}
1. Stop the Pentaho Server. See [Start and stop the Pentaho Server for configuration on Windows](#start-and-stop-the-pentaho-server-for-configuration-on-windows).
2. Double-click `pentahoserverw.exe` in `server\pentaho-server\tomcat\bin` to open **Pentaho Server Properties**.
You might need to select **Run as Administrator**.
3. Select the **Java** tab.
4. Set:
* **Initial memory pool**: `6144` MB
* **Maximum memory pool**: `8192` MB
5. Start the Pentaho Server. See [Start and stop the Pentaho Server for configuration on Windows](#start-and-stop-the-pentaho-server-for-configuration-on-windows).
If the JVM refuses to start, you might need to add RAM, stop memory-intensive services, or lower the maximum memory limit.
## Adjust Java VM memory limits for a graphical installation on Linux
These steps apply when you installed using the installation wizard, your computer can display graphics, and you are running Linux.
1. Stop the server. See [Start and stop the Pentaho Server for configuration on Linux](#start-and-stop-the-pentaho-server-for-configuration-on-linux).
2. Go to `pentaho-server/tomcat/scripts`.
3. Edit `ctl.sh`.
4. Locate the line under `start tomcat`, which looks like this:
```
export JAVA_OPTS="... -Xms128m -Xmx768m -XX:MaxPermSize=256m ..."
```
5. Set the memory to a minimum of `4096` MB and a maximum of `6144` MB, based on available system memory.
6. Start the server. See [Start and stop the Pentaho Server for configuration on Linux](#start-and-stop-the-pentaho-server-for-configuration-on-linux).
If the JVM refuses to start, add RAM, stop memory-intensive services, or lower the maximum memory limit.
## Increase Pentaho Server memory limit for installations on Windows
If you installed the Pentaho Server on Windows using the installation wizard, you can increase the server memory limits by editing the Tomcat Java settings.
{% hint style="info" %}
**Notes:**
* For Linux, see [Increase Pentaho Server memory limit for installations on Linux](#increase-pentaho-server-memory-limit-for-installations-on-linux).
* For a custom install, see **Configure and start the Pentaho Server after manual installation** in the **Install Pentaho Data Integration and Analytics** document.
{% endhint %}
1. Stop the Pentaho Server if it is running. See [Start and stop the Pentaho Server for configuration on Windows](#start-and-stop-the-pentaho-server-for-configuration-on-windows).
2. Type `services.msc` into the Windows Search box.
3. Find the Pentaho Server entry and note the **service name**.
It should be `pentahoserver`.
4. In `C:\pentaho\server\pentaho-server\tomcat\bin\`, rename `tomcat8w.exe` to match the service name.
Example: `pentahoserverw.exe`.
5. Double-click the renamed file to open the **Properties** window.
You might need to select **Run as Administrator**.
6. Select the **Java** tab.
7. Set:
* Minimum: `4096` MB
* Maximum: `6144` MB
8. Start the Pentaho Server. See [Start and stop the Pentaho Server for configuration on Windows](#start-and-stop-the-pentaho-server-for-configuration-on-windows).
If the JVM refuses to start with increased limits, add RAM, stop memory-intensive services, or reduce the maximum memory limit.
See also: [Increase the PDI client memory limit](#increase-the-pdi-client-memory-limit).
## Increase Pentaho Server memory limit for installations on Linux
If you installed PDI on Linux using the installation wizard, you can increase memory limits by editing a variable in a Pentaho-supplied script.
{% hint style="info" %}
**Note:**
* For Windows, see [Increase Pentaho Server memory limit for installations on Windows](#increase-pentaho-server-memory-limit-for-installations-on-windows).
* For a custom install, see **Configure and start the Pentaho Server after manual installation** in the **Install Pentaho Data Integration and Analytics** document.
{% endhint %}
1. Stop the server. See [Start and stop the Pentaho Server for configuration on Linux](#start-and-stop-the-pentaho-server-for-configuration-on-linux).
2. Go to `pentaho-server/tomcat/scripts`.
3. Edit `ctl.sh`.
4. Locate the line under `start tomcat`, which looks like this:
```java
export JAVA_OPTS="-Dpentaho.installed.licenses.file=/opt/pentaho/.installedLicenses.xml -Xms128m -Xmx768m -XX:MaxPermSize=256m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
```
5. Set the memory to a minimum of `4096` MB and a maximum of `6144` MB, based on available system memory.
6. Start the server. See [Start and stop the Pentaho Server for configuration on Linux](#start-and-stop-the-pentaho-server-for-configuration-on-linux).
See also: [Increase the PDI client memory limit](#increase-the-pdi-client-memory-limit).
## Increase the PDI client memory limit
As a best practice, increase PDI's memory limit so Spoon can perform memory-intensive tasks. You must increase the memory limit for both the Pentaho Server and the PDI client.
{% hint style="info" %}
**Note:** Instead of modifying the PDI client startup script, you can set the environment variable `PENTAHO_DI_JAVA_OPTIONS` to `-Xmx2g -XX:MaxPermSize=256m` on your client.
{% endhint %}
## Uninstalling the Pentaho Suite after evaluation
To uninstall the Pentaho Suite after evaluation:
1. Go to the `pentaho` directory and run the uninstall file.
2. Follow the Uninstall Wizard.
3. When prompted, choose whether to delete the data files.
4. Restart your computer.
Before you install a production version of Pentaho, uninstall the evaluation version first.
## Tutorials
Review these tutorials to start using PDI, reporting tools, and dashboards.
If you are new to PDI, start with [Getting Started with PDI](/pdia-try-pdia/getting-started-with-pdi).
Within this tutorial, you can also view:
* [PDI Transformation Tutorial](/pdia-try-pdia/archive-flattened-pages/pdi-transformation-tutorial)
* [PDI Job Tutorial](/pdia-try-pdia/archive-flattened-pages/pdi-job-tutorial)
* [Getting started with PDI and Hadoop](/pdia-try-pdia/archive-flattened-pages/getting-started-with-pdi-and-hadoop)
The [Getting Started with Analyzer, Interactive Reports, and Dashboard Designer](/pdia-try-pdia/getting-started-with-analyzer-interactive-reports-and-dashboard-designer) tutorial covers product features, best practices, and troubleshooting.
Within that tutorial, you can also view:
* [About Pentaho business analytics tools](/pdia-try-pdia/archive-flattened-pages/about-pentaho-business-analytics-tools)
* [Get started with Pentaho Reporting tools](/pdia-try-pdia/archive-flattened-pages/get-started-with-pentaho-reporting-tools)
* [Quick tour of the Pentaho User Console](/pdia-try-pdia/archive-flattened-pages/quick-tour-of-the-pentaho-user-console-puc)
* [Get started with Interactive Reports](/pdia-try-pdia/archive-flattened-pages/get-started-with-interactive-reports)
* [Get started with Analyzer Reports](/pdia-try-pdia/archive-flattened-pages/get-started-with-analyzer-reports)
* [Get started with Dashboard Designer](/pdia-try-pdia/archive-flattened-pages/get-started-with-dashboard-designer)
* [Next steps](/pdia-try-pdia/archive-flattened-pages/next-steps-analyzer-reports-dashboard)
The [Getting started with Report Designer](/pdia-try-pdia/getting-started-with-report-designer) tutorial includes step-by-step report creation instructions.
Within that tutorial, you can also view:
* [About Pentaho Report Designer](/pdia-try-pdia/archive-flattened-pages/report-designer-merged-pages/about-pentaho-report-designer)
* [Create a report with Report Designer](/pdia-try-pdia/archive-flattened-pages/report-designer-merged-pages/create-a-report-with-report-designer)
## Remove sample data from the Pentaho Server
By default, you have access to a sample data source and example content. When you are ready to move to development or production, remove the sample content.
To remove sample data and content:
1. Stop the Pentaho Server.
2. Delete `samples.zip` from `/pentaho-server/pentaho-solutions/system/default-content`.
If you performed a manual WAR build and deployment, the path is `/pentaho-server/pentaho-solutions/system`.
3. Edit `/pentaho/WEB-INF/web.xml` inside the deployed `pentaho.war`.
For archive installs, the path is usually `/pentaho-server/tomcat/webapps/pentaho/WEB-INF/web.xml`.
4. Remove the `hsqldb-databases` section:
```xml
hsqldb-databasessampledata@../../data/hsqldb/sampledata
```
5. Remove the `hsqldb-starter` section:
```xml
org.pentaho.platform.web.http.context.HsqldbStartupListener
```
6. Remove the `SystemStatusFilter`.
Note: This filter shows status messages useful only for development and testing.
```xml
SystemStatusFiltercom.pentaho.ui.servlet.SystemStatusFilterinitFailurePageInitFailureThis page is displayed if the PentahoSystem fails to properly initialize.
```
7. Remove the filter mapping:
```xml
SystemStatusFilter/*
```
8. Save and close `web.xml`.
9. Delete the `/pentaho-server/data/` directory.
10. Restart the Pentaho Server, then sign in to the Pentaho User Console as an administrator.
On **Browse Files**, delete the sample folders under **Public**.
## Data Integration and Analytics components and tools
Pentaho Data Integration and Analytics includes web-based components and design tools. What you use depends on your workflow and environment.
### Data Integration and Analytics web-based components
Use the Pentaho web-based components to share business intelligence solutions by analyzing data, creating reports, and building dashboards.
These components include:
* **Pentaho User Console (PUC)**
A design environment for accessing Analyzer, Interactive Reports, and Dashboard Designer. PUC also offers administration features for configuring your Pentaho Server.
* **Analyzer**
Visualize data to make informed decisions. Create charts and visualizations, filter data, and configure drill-down links.
* **Interactive Reports**
Create simple and on-demand operational reports without relying on IT.
* **Dashboard Designer**
Create dashboards from templates, themes, and content. Combine Interactive Reports, Analyzer, and more.
* **CTools**
A community-driven framework for creating dashboards with web technologies.
### Data Integration and Analytics design tools
Use Pentaho design tools to model, transform, and store data.
These tools include:
* **Pentaho Data Integration (PDI)**
An ETL engine for capturing data, cleansing it, and storing it in a usable format.
* **Report Designer**
Create pixel-perfect reports from virtually any data source.
* **Aggregation Designer**
Create aggregate tables for OLAP cubes to improve performance.
* **Metadata Editor**
Build metadata domains and models. Map physical database structures into a business model.
* **Schema Workbench**
Create and edit Mondrian models.
### Evaluate Pentaho Data Integration and Analytics
Before you set up and use Pentaho Data Integration and Analytics in production, evaluate it:
1. [Install the 30-day trial of Pentaho Data Integration and Analytics](#install-the-30-day-trial-of-pentaho-data-integration-and-analytics)
2. [Learn about Analyzer, Interactive Reports, and Dashboard Designer](/pdia-try-pdia/getting-started-with-analyzer-interactive-reports-and-dashboard-designer)
3. [Learn about Report Designer](/pdia-try-pdia/getting-started-with-report-designer)
4. [Learn about Pentaho Data Integration and the PDI client](/pdia-try-pdia/getting-started-with-pdi)
5. [Learn about Pentaho, big data, and Hadoop](/pdia-try-pdia/pentaho-big-data-and-hadoop)
6. [Understand common workflows](/pdia-try-pdia/about-pentaho-workflows)
### Data Integration and Analytics supported technologies
Reference material for supported components and JDBC drivers:
* [Components Reference](/pdia-try-pdia/components-reference)
* [JDBC drivers reference](/pdia-try-pdia/jdbc-drivers-reference)
# Getting Started with Analyzer, Interactive Reports, and Dashboard Designer
This guide covers core Pentaho Business Analytics workflows in one place.
Jump to:
* [About Pentaho business analytics tools](#about-pentaho-business-analytics-tools)
* [Quick tour of the Pentaho User Console](#quick-tour-of-the-pentaho-user-console)
* [Get started with Analyzer Reports](#get-started-with-analyzer-reports)
* [Get started with Interactive Reports](#get-started-with-interactive-reports)
* [Get started with Pentaho Reporting tools](#get-started-with-pentaho-reporting-tools)
* [Get started with Dashboard Designer](#get-started-with-dashboard-designer)
* [Next steps](#next-steps)
### About Pentaho business analytics tools
The topics found in this section give you an overview of the reports and dashboards you create with the User Console, to help you become familiar with the look and feel of the console.
The Pentaho User Console is a web-based design environment where you can analyze data, create interactive reports, dashboard reports, and build integrated dashboards to share business intelligence solutions with others in your organization and on the internet. In addition to its design features, the User Console offers a wide variety of system administration features for configuring the Pentaho Server, maintaining the Pentaho licenses, setting up security, managing report scheduling, and tailoring system performance to meet your requirements.
#### Prerequisites
Before you work with the User Console, install the Pentaho software and configure the Pentaho Server.
See [Install the 30-day trial of Pentaho Data Integration and Analytics](/pdia-try-pdia/archive-flattened-pages/pentaho-evaluation).
#### Expertise
You do not need special skills to use the design environment.
To use system administration features, you should understand your data sources, system configuration, and security providers.
#### Tools
In the User Console, you can access the Pentaho Repository on the server and these tools:
* Analyzer
* Interactive Reports
* Dashboard Designer
* Data Source Wizard
* Data Source Model Editor
#### Sign-in credentials
Some tasks require that you [sign in to the User Console](#log-in-to-the-pentaho-user-console) with an evaluator username and password.
### Quick tour of the Pentaho User Console
If you use file management tools or any web browser, you should feel right at home with the Pentaho User Console (PUC). To familiarize yourself with the different features and options of the User Console, take a quick tour.
{% hint style="info" %}
The features and options you see depend on your role and permissions.
See the **Pentaho Business Analytics** documentation for full details.
{% endhint %}
Jump to a section:
* [Log in](#log-in-to-the-pentaho-user-console)
* [Home](#home)
* [Opened](#opened)
* [Use Pentaho tools](#use-pentaho-tools)
* [Browse Files](#browse-files)
* [Schedules](#schedules)
* [Administration](#administration)
#### Log in to the Pentaho User Console
Follow these steps to log in to the User Console.
1. Launch a web browser.
2. Enter the URL for the Pentaho Server.
Your IT administrator can provide the URL.
3. On the Welcome page, enter your username and password.
4. Select **Log in**.
You can also use **Log in as an evaluator** if enabled.
#### Home
After you log in, you land on the Home perspective. Use it to start most tasks.
| Item | Name | Function |
| ---- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | User menu | Shows the name of the user currently logged in to the User Console. To log out or change your password, click the arrow next to your username. |
| 2 | **Home** |
Indicates the Home perspective, which you can use to explore learning resources, create reports, dashboards and data sources, open recent files, view Help documentation, and access other User Console perspectives. Click Home and use the drop-down menu to navigate to the different perspectives:
Home - Shows the Home perspective. See the Pentaho Business Analytics documentation for more information.
Browse Files - Helps you access, view, and manage the files and folders you need.
Schedules - Shows your active scheduled reports, any block out times, and allows you to create, edit, and maintain report schedules.
Administration - Allows you to perform user setup, mail server configuration, revise Pentaho Server authentication settings, and view the available Pentaho licenses.
|
| 3 | **Getting Started** |
Shows resources to help you get familiar with Pentaho. Click the tabs in this section for videos, and report and dashboard examples.- The Welcome tab contains an introductory video about Pentaho products. Click the play icon to view the video.
The Samples tab contains sample reports and dashboards that you can use to get familiar with the features and functionality of the User Console. Click Explore to view the samples.
The Tutorials tab contains tutorial videos that provide a visual tour of the User Console, reports, and dashboards. Click Watch the Video to view the tutorial.
|
| 4 | **Browse Files** | Opens the Browse Files perspective, where you can locate your files and folders, manage files, and schedule reports. Any file that you open appears in a new tab on the Opened perspective. |
| 5 | **Create New** | Allows you to create new reports, dashboards, and data sources, if your user role has permissions. |
| 6 | **Manage Data Sources** | Allows you to manage existing, and add new, data sources. |
| 7 | **Documentation** | Opens the [Pentaho documentation](https://docs.hitachivantara.com/) in a new window or tab. |
| 8 | **Recents** | Shows a list of your most recently opened files. Click the star next to the file name to add it to Favorites. |
| 9 | **Favorites** | Shows a list of your favorite files for quick access. To add a file for future access, use **Recents**, or select **Add to Favorites** in Browse Files. |
#### Opened
The Opened perspective contains your open files. It appears after you open a file from **Recents**, **Favorites**, or the **Browse Files** perspective.
Select **Home** > **Opened**.
The icons and options you see depend on the file type you select. See the **Pentaho Business Analytics** documentation for more information.
**Use Pentaho tools**
Your download includes sample reports and dashboards. It also includes the Steel Wheels sample database.
* [Interactive Reports](#get-started-with-interactive-reports) helps you create operational, tabular reports.
* [Analyzer Reports](#get-started-with-analyzer-reports) helps you explore data visually with filtering and drill-down.
* [Dashboard Designer](#get-started-with-dashboard-designer) helps you combine multiple visuals into one dashboard.
#### Browse Files
The Browse Files perspective helps you organize, find, and manage files. Your files can be local, stored in the repository, or accessed through a virtual file system (VFS) connection.
Select **Home** > **Browse Files**.
You can use this view for file management and actions like sharing and scheduling. See the **Pentaho Business Analytics** documentation for more information.
#### Schedules
The Schedules perspective shows your active scheduled reports.
Select **Home** > **Schedules**.
You can view recurrence patterns, last run time, next run time, and status. You can also edit schedules and create blockout times. See the **Pentaho Business Analytics** documentation for more information.
#### Administration
The Administration perspective is for users with the **Administer Security** permission.
Select **Home** > **Administration**.
If you do not have admin privileges, you do not see **Administration**.
Options include:
* **Users & Roles**
* **Authentication**
* **Mail Server**
* **Licenses**
* **VFS Connections**
* **Settings**
* **Email Groups**
See the **Pentaho Business Analytics** documentation for more information.
### Get started with Analyzer Reports
Analyzer Reports is an analytical visualization tool. It helps you filter and drill into data from Pentaho analysis data sources.
Use Analyzer when you need quick, interactive analysis. You can sort, filter, pivot, and add chart visualizations.
#### View an Analyzer report sample
This section highlights popular Analyzer capabilities. It uses the sample report **European Sales** in the **Getting Started** widget.
1. In the **Getting Started** widget on the Home page, click the **Samples** tab.
2. In the scrolling panel, scroll down and click **European Sales**, then click **Explore**.
A new browser window opens. Click the **Samples** tab to see the report.
#### Tour the Analyzer panels
You can open an editable version of **European Sales** in Analyzer from the **Browse Files** page.
1. From the User Console Home page, click **Browse Files**.
2. In the **Browsing** pane, expand `Public`, then expand **Steel Wheels**.
3. In the center pane, double-click **European Sales**.
The **Opened** page appears with the Analyzer report.
4. On the toolbar, click **Add More Fields** and **Rearrange Fields**.
The **Available Fields** and **Layout** panels expand.
**Panel and toolbar basics**
* **Opened page**
* Quick-access buttons for **Analysis Report**, **Interactive Report**, and **Dashboard**.
* Tabs across the page for opened reports and files.
* **Available Fields** and **Layout**
* Drag levels and measures into a report.
* The canvas updates as you build the layout.
* Remove a field by dragging it off the **Layout** panel.
* **Report canvas**
* Dynamic view of your report as you build it.
* Shown fields depend on the selected chart type.
* **Analyzer toolbar and filters**
* Undo/redo, show or hide panels, and change settings.
* Use the **Filters** panel to view, edit, and delete filters.
#### Create your first Analyzer report
These steps use the **Steel Wheels** sample data.
1. From the User Console Home page, click **Create New**, then select **Analysis Report**.
2. In **Select Data Source**, select **SteelWheels:SteelWheelsSales**, then click **OK**.
A blank Analyzer report appears.
3. Build a basic pivot table:
* Drag **Territory** to **Rows**.
* Drag **Years** to **Columns**.
* Drag **Sales** to **Measures**.
4. Add subtotals:
1. Drag **Line** above **Territory** in the **Layout** column.
2. Right-click the **Line** header, then select **Show Subtotals**.
5. Add conditional formatting:
* Right-click the first **Sales** column.
* Select **Conditional Formatting** > **Data Bar - Green**.
6. Add a user-defined measure:
1. Right-click the same **Sales** column.
2. Select **User Defined Measure** > **% of Rank, Running Sum**.
3. Select **% of Sales**, then click **Next**.
7. Refine the measure:
* Select **Each Line Column/Row Subtotal (Subtotal is 100%)**, then click **Done**.
8. Add a filter:
1. Click **Show Filters** to expand the filters canvas.
2. Drag **Territory** from **Available Fields** to the filter canvas.
The **Filter on Territory** dialog box appears.
9. In **Filter on Territory**, select **APAC**, then click the right arrow to move it to the selected list.
10. Enable **Parameter Name**.
11. In **Parameter Name**, type `region`, then click **OK**.
The report updates and shows APAC sales data. 12. Resize columns as needed for readability.
 13. Save the report:
1. Click **Save As**.
2. Save the report as `Territory - Sales` in your user folder.
You have created a simple Analyzer report from scratch. For deeper Analyzer workflows, see [About Pentaho business analytics tools](#about-pentaho-business-analytics-tools).
### Get started with Interactive Reports
Interactive Reports is a web-based design interface which is used to create both simple and on-demand operational reports without depending on IT or report developers. Use Interactive Reports if you want to create a quick report that answers an immediate business question, looks professional, and provides significant control over formatting elements such as fonts, column width or sorting, background colors, and more.
Jump to:
* [View an Interactive report sample](#view-an-interactive-report-sample)
* [Tour the Interactive panels](#tour-the-interactive-panels)
* [Create your first Interactive report](#create-your-first-interactive-report)
#### View an Interactive report sample
This section highlights some popular Interactive Reports capabilities that are available, using the sample report called Vendor Sales Report, located in the **Getting Started** widget.
1. In the **Getting Started** widget on the Home page, click the **Samples** tab.
2. Click **Vendor Sales** from the scrolling panel on the right.
3. Click **Explore** in the **Samples** pane.
A new window opens showing the Vendor Sales sample report.
#### Tour the Interactive panels
By going to the Browse Files page in the User Console, you can also view an editable version of the Vendor Sales report.
1. Switch to the Browse Files page in the User Console.
2. In the **Folders** pane, click to expand the **Public** folder, then click to highlight the `Steel Wheels` folder.
3. In the **Files** pane, click **Vendor Sales**, then click **Edit** in **File Actions**.
The Opened page appears with the interactive report and toolbars active.
Key areas on the page:
1. **Opened page**\
Provides quick access buttons across the top to create and save a new **Analysis Report**, **Interactive Report**, and **Dashboard**. Opened reports and files show as a series of tabs across the page.
2. **Data**, **Formatting**, and **General** panels\
Use the **Data** panel to drag information into a column or a row on the report. Your report updates as you drag items onto the report canvas. Use **Find** to search for a specific field. Delete a field by dragging it from the layout area to the trash can that appears in the lower right corner of the report canvas.
Use the **Formatting** panel to change font size and type.
Use the **General** panel to set preferences, select a paper size for printing, and select templates for your report.
3. **Report canvas**\
Shows a dynamic view of your report as you build it. The look of your report changes as you use the **Data**, **Formatting**, and **General** panels.
4. **Interactive toolbar and filters**\
Use the toolbar to undo and redo actions, hide lists of fields, add or hide filters, disable auto-refresh, adjust settings, change the report view, and limit the number of rows queried. Use the **Filters** panel to view, edit, and delete filters for the active report.
#### Create your first Interactive report
The instructions below guide you through the creation of your first Interactive report using the Steel Wheels sample data.
1. From the Home page, click **Create New**, then choose **Interactive Report**.
2. Choose the **Inventory** data source from the **Select Data Source** dialog box. Click **OK**.
3. Click **Get Started** on the dialog box that appears.
A blank Interactive report canvas appears.
4. Click and drag the **Product Code** element onto the report canvas until a highlighted vertical line appears. Drop it onto the report canvas.
5. Continue dragging and dropping these fields onto the canvas: **Product Name**, **Product Vendor**, **Quantity in Stock**, **MSRP**, and **Buy Price**.
The data from the chosen fields appears on the report canvas and populates with the information from the server.
**Note:** You can change the order of the columns by clicking the column headings and dragging them left or right. If you want to delete a column, drag the column title to the trash can.
6. Rename your report by double-clicking **Untitled** in the report canvas and typing a name in the field that appears. `In Stock Report` is used in this example.
7. After you have arranged your columns, apply a filter to the data. Click the **Filter** icon in the toolbar. After the **Filter** pane expands, drag the **Product Code** field onto the filter workspace.
8. In the **Filter on** dialog box, click **Select from a list**.
9. Choose items from the filter list using one of these methods. Click the arrows to move your selected filters on or off the filter list.
* To choose more than one item, hold down Ctrl and click the items. Then click the top arrow to move them to the right panel.
* To choose a range, hold down Shift. Then click the first and last item.
* To choose a single item, click it. Then click the top arrow to move it to the right panel.
10. Click **OK**, then click **Save As** on the toolbar.
1. In the **Save As** dialog box, save your report using the title you used in Step 6. `In Stock Report` is used in this example.
2. Choose your user folder as the location. Remember the folder and report title. You use the report in a later tutorial. Click **Save**.
11. If you want to export the report, click the **Export** icon on the toolbar and choose a format from the dropdown list.
The report exports in the selected format. You can print a paper copy from the export.
You have successfully created a simple Interactive report from scratch. See **Pentaho Business Analytics** for details on how to work with more complex interactive reports.
### Get started with Pentaho Reporting tools
After you define the data sources for your Pentaho Server, you are ready to begin working with the Pentaho User Console to create your first reports. Each section below uses sample data sources that are included with the installation.
Use these sections in order:
* [Quick tour of the Pentaho User Console](#quick-tour-of-the-pentaho-user-console)
* [Get started with Interactive Reports](#get-started-with-interactive-reports)
* [Get started with Analyzer Reports](#get-started-with-analyzer-reports)
* [Get started with Dashboard Designer](#get-started-with-dashboard-designer)
* [Next steps](#next-steps)
### Get started with Dashboard Designer
Dashboard Designer lets you build dashboards with minimal training. A dashboard combines several reports in one view. Use it to monitor multiple reports at once, keep quick links to pages you use often, and view charts while you work.
In this topic:
* [View a dashboard sample](#view-a-dashboard-sample)
* [Tour the Dashboard panels](#tour-the-dashboard-panels)
* [Create your first dashboard](#create-your-first-dashboard)
#### View a dashboard sample
This section highlights popular Dashboard Designer capabilities, using the sample dashboard **Sales Performance (Dashboard)** in the **Getting Started** widget.
1. In the **Getting Started** widget on the Home page, click the **Samples** tab.
2. Scroll down to **Sales Performance (Dashboard)**.
3. Click **Explore** to open a new browser window, then click the **Samples** tab.
4. Scroll right in the horizontal list at the bottom.
5. Click **Sales Performance (Dashboard)**.
#### Tour the Dashboard panels
You can open the editable version of **Sales Performance (Dashboard)** in Dashboard Designer from **Browse Files**.
1. In the **Folders** pane, expand `Public`, then select `Steel Wheels`.
2. In the center pane, double-click **Sales Performance (Dashboard)**.
3. After the dashboard opens, in **File Actions**, click **Edit**.
| Item | Name | Function |
| ---- | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| 1 | **Opened** page | Provides quick access buttons across the top to create and save a new **Analysis Report**, **Interactive Report**, and **Dashboard**. Opened reports and files show as a series of tabs across the page. |
| 2 | Prompts panel | The prompts panel gives you a way to add filters to the individual parts of your dashboard. |
| 3 | **Browse** and **Files** panel | Locate your files using the **Browse** and **Files** panels, and add them to dashboards. |
| 4 | Dashboard canvas | Shows a dynamic view of your dashboard as you work to build it. The look of your dashboard refreshes as you add content from the **Browse** and **Files** panels, and work with the prompts or **Objects** panels. |
| 5 | **Objects** panel | Refine the look of your dashboard with the **Objects** panel by choosing a dashboard template or changing the titles for each object in the dashboard. |
#### Create your first dashboard
1. From the User Console Home page, click **Create New**, then select **Dashboard**.
2. In the **Edit** pane, click the **Templates** tab, then select **2 over 1**.
3. In the **Edit** pane, click the **Properties** tab, then enter `My Dashboard` in **Page Title**.
This is the title for your dashboard page.
4. Click the **Themes** tab, then select a theme.
The new theme applies immediately.
5. In the **Browse** pane, open the folder you used earlier.
6. From the **Files** pane, drag `Territory - Sales` onto the top-left dashboard panel.
7. In the **Edit** pane, enter `Territory - Sales` in **Title**, then click **Apply**.
The panel populates with the **Territory - Sales** report.
8. Locate your Interactive report in the **Browse** pane.
9. Drag `In Stock Report` onto the top-right dashboard panel.
10. In the **Edit** pane, enter `In Stock Report` in **Title**, then click **Apply**.
The panel populates with the **In Stock Report**.
11. Drag any report from `Public/Steel Wheels` into the bottom dashboard panel.
12. Enter a title for the bottom panel, then click **Apply**.
13. In the toolbar, click **Save As**.
14. Save the dashboard as `My Dashboard`, then click **Save**.
15. Close the dashboard (click **X** on its tab).
16. Go to **Browse Files**, then double-click `My Dashboard` in the **Files** pane.
You created a simple dashboard. See **Pentaho Business Analytics** for details on complex dashboards.
### Next steps
After you have finished working through the walk-through tutorials, you are ready to learn more about Pentaho reporting with the following documents:
* **Pentaho Business Analytics**
* **Pentaho Report Designer**
# Getting started with Report Designer
Use this topic to create and refine a sample report in Report Designer.
You use Pentaho’s sample database in these steps. Sample data is installed by default with Report Designer.
### In this topic
* [About Pentaho Report Designer](#about-pentaho-report-designer)
* [Create a report with Report Designer](#create-a-report-with-report-designer)
* [Design your report](#design-your-report)
* [Refine your report](#refine-your-report)
* [Row banding, data formatting, and alignment](#row-banding-data-formatting-and-alignment)
* [Add a chart to your report](#add-a-chart-to-your-report)
* [Add parameters to your report](#add-parameters-to-your-report)
* [Publish your report](#publish-your-report)
### About Pentaho Report Designer
Pentaho Report Designer is a report creation tool. You can use it standalone or as part of the Pentaho Business Analytics suite.
It helps you build detailed reports from prepared data. You can connect to most data sources.
### Create a report with Report Designer
Perform the steps below to create a report using Report Designer.
1. Start Report Designer. Go to **Start** > **Programs** > **Pentaho Enterprise Edition** > **Design Tools** > **Report Designer**.
The Report Designer home page appears.
To change the zoom level, drag the percentage in the upper-left corner. Double-click it to reset to 100%.
2. Click **New Report** in the Welcome dialog box.
The design workspace appears.
3. In the right pane, click the **Data** tab.
4. Right-click **Data Sets** and select **JDBC**.
You can also click the yellow database icon.
5. Under **Connections**, select **SampleData (Hypersonic)**.
6. Next to **Available Queries**, click the plus sign to add queries.
Query 1 appears under **Available Queries**.
7. Click the edit icon ().
The SQL Query Designer window opens.
8. Select **PUBLIC** in the **schema filter** menu. Double-click **ORDERFACT** so the table appears in the workspace.
9. In the SQL Query Designer workspace, right-click **ORDERFACT** and select **deselect all**.
10. Select the **ORDERNUMBER**, **QUANTITYORDERED**, **PRICEEACH**, and **ORDERDATE** fields.
11. Double-click **PRODUCTS** so the table also appears in the workspace.
Notice the line joining the **ORDERFACT** and **PRODUCTS** tables.
12. Clear all **PRODUCTS** fields. Then select **PRODUCTNAME** and **PRODUCTLINE**.
13. Click the **Syntax** tab to view the SQL statement.
Notice that **PRODUCTCODE** joins the tables.
14. Click **OK** to return to the JDBC Data Source dialog box.
The SQL statement appears under **Query**.
15. Click **OK** in the JDBC Data Source dialog box.
The fields now appear under **Query 1**.
Next: [Design your report](#design-your-report).
### Design your report
This exercise walks you through designing your report layout.
1. Under **View**, select **Element Alignment Hints** and **Snap to Elements**.
These options help align elements.
2. Under **Query 1**, drag **ORDERNUMBER** into the **Details** band.
3. Add **ORDERDATE**, **PRODUCTNAME**, **QUANTITYORDERED**, and **PRICEEACH** to the **Details** band.
Do not overlap fields.
4. Resize **PRODUCTNAME** larger. Resize **QUANTITYORDERED** smaller.
5. Click **Preview** ().
6. Click **Edit** () to return to the workspace.
Next: [Refine your report](#refine-your-report).
### Refine your report
You created a report in the previous exercise. Now add labels, headers, and row banding.
1. Drag a label () to the **Page Header** band.
The **Structure** tab updates.
2. Click inside the label and type `Order Report`.
3. Select the label text. Set the font size to 18. Apply bold.
4. With the label selected, set a font color.
The page header appears on every page.
5. Create column headers. Click **Details Header** under the **Structure** tab.
The **Style** and **Attributes** tabs appear.
6. Under **common** in **Attributes**, set **hide-on-canvas** to **False**.
The **Details Header** band appears.
7. Click the **Select Objects** icon ().
8. Select all column objects in the **Details** pane.
9. Press Ctrl+C to copy. Press Ctrl+V to paste into the **Details Header** pane.
10. Select **Format** > **Morph** > **label**.
The objects change to labels.
11. Type the header labels: `Order No.`, `Order Date`, `Product Name`, `Quan.`, and `Price Each`.
12. Click **Preview** ().
13. Select **Format** > **Row Banding**.
14. In the Row Banding dialog box, select **Yellow** for **Visible Color**. Click **OK**.
15. Click **Preview**.
16. Select **File** > **Save**. Save to `.../report-designer/samples`. Use `Orders` as the file name.
Next: [Row banding, data formatting, and alignment](#row-banding-data-formatting-and-alignment).
### Row banding, data formatting, and alignment
#### Row banding
Create a row band element to control which fields show banding. You can name the row band element anything.
In this example, the row band element is named **row-band-element**.
After you create the element, select the fields to band. In **Attributes**, type `row-band-element` in the **name** field.
#### Data formatting
Report Designer uses default formats for dates and numbers. You can change formats under **Attributes**.
Select a field. Then select a value for **format**.
In this example, **Order Date** uses `MM-dd-yy`.
Preview the report to confirm the results.
{% hint style="info" %}
You can also type a custom format string. Use the JavaScript date and number format syntax.
{% endhint %}
#### Alignment
To align multiple objects, select them first. Then choose an alignment option under **Format**.
To multi-select, press Shift and click each object. You can also use **Select Objects** () and drag to select.
Next: [Add a chart to your report](#add-a-chart-to-your-report).
### Add a chart to your report
In this exercise, you add a chart to your report.
1. Select **File** > **Open**. Open the report you saved earlier.
2. In the **Palette**, drag a Chart icon () into the **Report Footer** band.
3. Resize and center the chart.
4. Double-click the sample chart.
5. Select the **pie chart** icon.
Chart properties are listed on the left. Data properties are listed on the right.
6. Under **Title**, set **chart-title** to `Product Pie Chart`.
7. Under **Common** in **Primary DataSource**, set **value-column**. Click the ellipsis to open the Select Field dialog box.
8. Select **QUANTITYORDERED** and click **OK**.
9. Under **Series**, click the ellipsis next to **series-by-field**.
The Edit Array dialog box opens.
10. Click the **Add** icon ().
11. Select **PRODUCTLINE** and click **OK**.
12. Click **OK** to close the Edit Chart dialog box.
13. Click **Preview** ().
14. Scroll to the last page.
15. Save your report.
{% hint style="info" %}
To use a bar or line chart, change the chart type. Add **series-by-value** entries for `SALES` and `COST`.
{% endhint %}
Next: [Add parameters to your report](#add-parameters-to-your-report).
### Add parameters to your report
Now make your report interactive by setting parameters. Users get prompted for values when they run the report.
1. In Report Designer, open your Orders report.
2. Select **Data** > **Add Parameter**.
You can also select **Master Report Parameter** () under the **Data** tab.
The Add Parameter dialog box appears.
3. In **Name**, enter `enter_prodline`.
4. In **Label**, enter `Select Line`.
5. For **Display Type**, select **Drop Down**.
6. Under **DataSources**, select **JDBC (SampleData (Hypersonic)**. Click the **Edit** icon ().
The JDBC Data Source dialog box appears.
7. Under **Connections**, select **SampleData (Memory)**.
8. Next to **Available Queries**, click **Add** ().
Query 2 is added.
9. In **Query Name**, enter `prodlineList`.
10. In **Query**, enter the following SQL:
```sql
SELECT DISTINCT
"PRODUCTS"."PRODUCTLINE"
FROM
"PRODUCTS"
```
You can also build the query in SQL Query Designer.
11. Click **OK** to close the Data Source dialog box.
12. In the Add Parameter dialog box, under **DataSources**, select **prodlineList**.
13. For **Value Type**, select **String**.
14. Optional: Set a default value, such as `Motorcycles`.
15. Click **OK** to close the Add Parameter dialog box.
16. Map the parameter back to Query 1. Under **Data**, double-click **Query 1**.
17. Click the **Edit** icon () to open SQL Query Designer. Right-click **PRODUCTLINE** and select **add where condition**.
18. In the condition editor, enter `${enter_prodline}`. Click **OK**.
19. Click **OK** to close SQL Query Designer.
20. Click **OK** to close the Data Source dialog box.
21. Click **Preview**.
22. Save and close the report.
Next: [Publish your report](#publish-your-report).
### Publish your report
Now publish the report to a Pentaho server.
1. In Report Designer, open the report you created.
2. Select **File** > **Publish**.
You can also select **Publishes the report on a Pentaho server** ().
3. In the Login dialog box, confirm the server URL is `http://localhost:8080/pentaho/`.
4. Click **OK**.
The Publish to Server dialog box appears.
5. Enter a report title and description.
6. Under **Location**, save to `...public/Steel Wheels`.
7. Set **Output Type** to **html** and click **OK**.
A success message appears.
8. Click **Yes** to open the User Console and view the report.
To view it later, go to `http://localhost:8080/pentaho/`. Then browse to the `Reporting Examples` directory.
9. Log in as `Admin`.
The default password is `password`.
10. Select a product line parameter value. Keep the default **Output Type**.
Your report is now available to users.
# Getting Started with PDI
If you are new to Pentaho Data Integration, start here.
Use these tutorials to build your first transformations and jobs in Spoon.
### In this topic
* [Pentaho Data Integration (PDI) tutorial](#pentaho-data-integration-pdi-tutorial)
* [PDI job tutorial](#pdi-job-tutorial)
* [Getting started with PDI and Hadoop](#getting-started-with-pdi-and-hadoop)
### Pentaho Data Integration (PDI) tutorial
The following tutorial is intended for users who are new to the Pentaho suite or who are evaluating Pentaho as a data integration and business analysis solution. The tutorial consists of six basic steps, demonstrating how to build a data integration transformation and a job using the features and tools provided by Pentaho Data Integration (PDI).
The Data Integration perspective of PDI allows you to create two basic file types: transformations and jobs. Transformations describe the data flows for ETL such as reading from a source, transforming data and loading it into a target location. Jobs coordinate ETL activities such as defining the flow and dependencies for what order transformations should be run, or prepare for execution by checking conditions such as, "Is my source file available?" or "Does a table exist in my database?"
The aim of this tutorial is to walk you through the basic concepts and processes involved in building a transformation with PDI in a typical business scenario. In this scenario, you are loading a flat file (CSV) of sales data into a database to generate mailing lists. Several of the customer records are missing postal codes that must be resolved before loading into the database. In the preview feature of PDI, you will use a combination of steps to cleanse, format, standardize, and categorize the sample data. The six basic steps are:
1. Step 1: Extract and load data
2. Step 2: Filter for missing codes
3. Step 3: Resolve missing data
4. Step 4: Clean the data
5. Step 5: Run the transformation
6. Step 6: Orchestrate with jobs
#### Prerequisites
To complete this tutorial, you need the following items:
* An installed version of the [Pentaho 30-day trial](https://www.hitachivantara.com/en-us/products/data-management-analytics/pentaho-platform/pentaho-data-integration/pentaho-trial-download.html).
#### Step 1: Extract and load data
In Step 1, you will retrieve data from a CSV flat file and use the Text File Input step to connect to a repository, view the file schema, and retrieve the data contents.
**Create a new transformation**
Follow these steps to create a new transformation.
If you want to insert a variable into a field that accepts variables, you can put your cursor in the fields and press **CTRL+Spacebar** to see a list of variables to insert. Fields that accept variables have a blue diamond.
1. Select **File** > **New** > **Transformation** in the upper-left corner of the PDI window.
2. Under the **Design** tab, expand the **Input** node, then select and drag a Text File Input step onto the canvas.
3. Double-click the Text File input step. In the Text file input window, you can set the properties of the step.
4. In the **Step Name** field, type `Read Sales Data`.
The Text file input step is now renamed to Read Sales Data.
5. Click **Browse** to locate the `sales_data.csv` source file in the `...\design-tools\data-integration\samples\transformations\files` folder. The **Browse** button appears in the upper-right side of the window near the **File or Directory** field.
6. Change **File type** to `*.csv`. Select `sales_data.csv`, then click **OK**.
The path to the source file appears in the **File or directory** field.
7. Click **Add**.
The path to the file appears under **Selected Files**.
**View the content in the sample file**
Follow these steps to look at the contents of the sample file.
1. Click the **Content** tab, then set the **Format** field to **Unix**.
2. Click the **File** tab again and click the **Show file content** in the lower section of the window.
3. The Number of lines (0-all lines) window appears. Click **OK** to accept the default.
4. The Content of first file window shows the file. Examine the file to see how that input file is delimited, what enclosure character is used, and whether or not a header row is present.
In the sample, the input file is comma delimited, using the enclosure character of a quotation mark ("). It contains a single header row containing field names.
5. Click the **Close** button to close the window.
**Edit and save the transformation**
Follow these steps to provide information about the data's content.
1. Click the **Content** tab. Use the fields under the **Content** tab to define how your data is formatted.
2. Verify that the **Separator** is set to comma (,) and that the **Enclosure** is set to quotation mark ("). Select **Header** and enter `1` in the **Number of header lines** field.
3. Click the **Fields** tab and click **Get Fields** to retrieve the input fields from your source file. When the Number of lines to sample window appears, enter `0` in the field, then click **OK**.
4. If the Scan Result window displays, click **Close** to close the window.
5. To verify that the data is read correctly, click the **Content** tab, then click **Preview Rows**.
6. In the Enter the number of rows you would like to preview window, click **OK** to accept the default.
The Examine preview data window appears.
7. Review the data. Do you notice any missing, incomplete, or variations of the data?
* `STATE & POSTALCODE` both contain ``
* `COUNTRY` contains both `USA` and `United States`.
8. Click **OK** to save the information that you entered in the step.
9. Enter a name for the transformation and provide additional properties using the Transformation Properties window. There are multiple ways to open the Transformation Properties window.
* Right-click any empty space on the canvas and select **Properties**.
* Double-click any empty space on the canvas to select **Properties**.
* Enter the CTRL-T keyboard combination.
10. In the **Transformation Name** field, enter `Getting Started Transformation`.
Below the name, the filename is empty.
11. Click **OK** to close the Transformation Properties window.
12. To save the transformation, select **File** > **Save**.
When saving your transformation for the first time, you are prompted for a file location and name of your choice. The file extension `.ktr` is the usual file extension for transformations.
**Load data into a relational database**
Now you are ready to take all the records that are exiting the Filter Rows step (added in Step 2) where the **POSTALCODE** was not null (the **true** condition) and load them into a database table. You will use the Table Output step and a hop from the Text File Input step to direct the data stream into a database table. This section of the tutorial uses a pre-existing database established during the Pentaho installation, which is started along with the server.
**Create the Table Output step**
Follow these instructions to create the Table Output step.
1. Under the **Design** tab, expand the contents of the **Output** node.
2. Click and drag a Table Output step into your transformation.
3. Create a hop between the Read Sales Data and Table Output steps. To create the hop:
1. Press the SHIFT key.
2. Click the Read Sales Data (Text File Input) step and drag the mouse to draw a line to the Table Output step.
3. Release the SHIFT key.
4. Click the Table Output step.
4. Double-click the Table Output step to open its **Edit properties** dialog box.
5. Rename your Table Output step to Write to Database.
**Create a connection to the database**
Follow these steps to create a connection to the database.
1. Click **New** next to the **Connection** field. You must create a connection to the database.
The Database Connection window appears.
2. Provide the settings for connecting to the database.
| Field | Setting |
| ------------------- | ------------------------------------------------------------------------------------ |
| **Connection Name** | Sample Data |
| **Connection Type** | Hypersonic |
| **Host Name** | localhost |
| **Database Name** | sampledata |
| **Port Number** | 9001 |
| **User Name** | pentaho\_admin |
| **Password** | password (If `password` does not work, please check with your system administrator.) |
3. Click **Test** to verify your entries are correct. A success message appears. Click **OK**.
**Note:** If you get an error when testing your connection, ensure that you have provided the correct settings information as described in the table and that the sample database is running. Depending on your platform, see [Start and stop the Pentaho Server for configuration on Windows](https://github.com/pentaho/documentation/blob/main/PDIA/11.0/Get%20Started/Pentaho%20evaluation/Start%20and%20stop%20the%20Pentaho%20Server%20for%20configuration%20on%20Windows=GUID-2DF3CCF0-39D7-4BC4-8129-AE3C6A3FBD60=1=en=.md) or [Start and stop the Pentaho Server for configuration on Linux](/pdia-try-pdia/archive-flattened-pages-do-not-use/install-trial-legacy-child-pages/start-and-stop-the-pentaho-server-for-configuration-on-linux).
4. Click **OK** to exit the Database Connections window.
**Define the Data Definition Language (DDL)**
DDLs are the SQL commands that define the different structures in a database such as `CREATE TABLE`. Fortunately, Pentaho can help you create the necessary DDL.
1. Enter `SALES_DATA` in the **Target Table** text field.
2. This table does not exist in the target database, so Pentaho can generate the DDL to create the table and execute it. In this scenario, the DDL is based on the stream of data coming from the previous step, which is the Read Sales Data step.
3. In the Table Output window, select the **Truncate Table** property.
4. Click the **SQL** button in the bottom of the Table output dialog box to generate the DDL for creating your target table.
5. The Simple SQL editor window appears with the SQL statements needed to create the table.
6. Click **Execute** to execute the SQL statement.
The Results of the SQL statements window appears.
7. Examine the results, then click **OK** to close the Results of the SQL statements window.
8. Click **Close** in the Simple SQL editor window
9. Click **OK** to close the Table output window.
10. Save your transformation.
#### Step 2: Filter for missing codes
After completing Step 1: Extract and load data, you are ready to add a transformation component to your data pipeline. The source file contains several records that are missing postal codes. This section of the tutorial filters out those records that have missing postal codes, where the POSTALCODE is not null (the true condition), and ensures that only complete records are loaded into the database table.
**Preview the rows read by the input step**
Follow these steps to preview the rows read by the input step.
1. Right-click the Read Sales Data step and select **Preview**.
2. Specify the number of rows to preview. Optionally, you can configure breakpoints which pause execution based on a defined condition, such as a field having a specific value or exceeding a threshold.
3. Click the **Quick Launch** button. Preview the data and notice that several of the input rows are missing values for the **POSTALCODE** field.
4. Click **Stop** on the preview window to end the preview.
**Separate the records with missing postal codes**
Follow these instructions to use the Filter Rows transformation step to separate out those records missing postal codes. These records are resolved later in the tutorial.
1. Add a Filter Rows step to your transformation. Under the **Design** tab, select **Flow** > **Filter Rows**.
2. Insert your Filter Rows step between your Read Sales Data step and your Write to Database step.
1. Right-click and delete the hop between the Read Sales Data step and Write to Database steps.
2. Create a hop between the Read Sales Data step and the Filter Rows step. Create a hop by clicking the step, and then hold the SHIFT key down and click-and-drag to draw a line to the next step.
3. Create a hop between the Filter Rows step and Write to Database step.
4. In the dialog box that appears, select **Result is TRUE.**
3. Double-click the Filter Rows step. The Filter Rows window appears.
4. In the **Step Name** field, enter `Filter Missing Zips`.
5. Click in **The condition** field to open the Fields window. The available conditions appear.
6. In the **Fields** window select **POSTALCODE** and click **OK**.
7. Click the comparison operator field, which is set to **=** by default. The Functions window appears.
8. Select **IS NOT NULL** from the list of functions, and then click **OK** to close the Functions window.
9. Click **OK** to exit the Filter Rows window.
**Note:** You will return to this step later to configure the **Send true data to step** and **Send false data to step** settings after adding their target steps to your transformation.
10. Save your transformation.
#### Step 3: Resolve missing data
After completing Step 2: Filter for missing codes, you are ready to resolve the missing postal codes. In this section, you will learn how to use a second text file containing a list of cities, states, and postal codes, to look up the postal codes for those records in which the fields are missing, which is the false branch of your Filter rows step.
First, you will use a Text file input step to read from the source file. Then, you will use a Stream lookup step to bring the resolved postal codes into the stream. Lastly, you will use the Select values step to rename fields on the stream, remove unnecessary fields, and more.
**Retrieve data from your lookup file**
Follow these steps to retrieve data from your lookup file.
1. Add a new Text File Input step to your transformation.
This step retrieves the records from your lookup file. Do not add a hop yet.
2. Open the Text File Input step window, then enter `Read Postal Codes` in the **Step name** property.
3. Click **Browse** to navigate to the `Zipssortedbycitystate.csv` source file located in the directory `...\design-tools\data-integration\samples\transformations\files`.
4. Change **File type** to `*.csv`, select `Zipsortedbycitrystate.csv`, and click **OK**.
The path to the source file appears in the **File or directory** field.
5. Click **Add**.
The path to the file appears under **Selected files**.
**View the contents of the sample file**
Follow these steps to view the contents of the sample file.
1. Click the **Content** tab, then set the **Format** field to **Unix**.
2. Click the **File** tab again and click the **Show file content** near the bottom of the window.
3. The Number of lines(0=all lines) window appears. Click the **OK**button to accept the default.
4. The **Content of first file** window shows the file. Examine the file to see how that input file is delimited, what enclosure character is used, and whether a header row is present. In the example, the input file is comma (,) delimited and the enclosure character is the quotation mark ("). A single header row contains field names.
5. Click **Close** to close the window.
**Edit and save the transformation**
Follow these steps to edit and save your transformation.
1. In the **Content** tab, change the **Separator** character to a comma (,) and confirm that the **Enclosure** setting is a quotation mark ("). Verify that the **Header** option is selected.
2. Under the **Fields** tab, click **Get Fields** to retrieve the data from your CSV file.
3. The Number of lines to sample window appears. Enter `0` in the field, then click **OK.**
4. If the Scan Result window displays, click **Close** to close it.
5. Click **Preview rows** to verify that your entries are correct.
1. When prompted to enter the preview size, click **OK**.
2. Review the information in the window, then click **Close**.
6. Click **OK** to exit the Text File input window.
7. Save the transformation.
**Resolve missing zip code information**
Follow these steps to resolve the missing postal code information.
1. Add a Stream Lookup step to your transformation by clicking the **Design** tab, expanding the **Lookup** folder, then selecting **Stream Lookup**.
2. Draw a hop from the Filter Missing Zips to the Stream lookup step. In the dialog box that appears, select **Result is FALSE**.
3. Create a hop from the Read Postal Codes step to the Stream lookup step.
4. Double-click the Stream lookup step to open the Stream Value Lookup window.
5. Rename Stream Lookup to Lookup Missing Zips.
6. From the Lookup step drop-down box, select **Read Postal Codes** as the lookup step. Perform the following:
1. In the **key(s) to look up the value(s)** table, define the **CITY** and **STATE** fields .
2. In **row #1**, open the drop-down menu in the **Field** column and select **CITY**.
3. Click in the **LookupField** column and select **CITY**.
4. In **row #2**, open the drop-down menu in the **Field** column and select **STATE**.
5. Click in the **LookupField** column and select **STATE**.
7. Click **Get Lookup Fields** to pull the three fields from the Read Postal Code step.
8. **POSTALCODE** is the only field you want to retrieve. To delete the **CITY** and **STATE** lines, right-click in the line and select **Delete Selected Lines**.
9. In the **New Name** field, change the name **POSTALCODE** to **ZIP\_RESOLVED** and verify that **Type** is set to **String**.
10. Select **Use sorted list (i.s.o. hashtable)**.
11. Click **OK** to close the Stream Value Lookup edit properties dialog box.
12. Save your transformation.
**Preview your transformation**
Follow these steps to preview your transformation.
1. To preview the data, select and right-click the Lookup Missing Zips step. From the menu that appears, select **Preview**.
2. In the Transformation debug dialog window, click **Quick Launch** to preview the data flowing through this step.
3. In the Examine preview data window that appears, note that the new field, **ZIP\_RESOLVED**, has been added to the stream containing your resolved postal codes.
4. Click **Close** to close the window.
5. If the Select the preview step window appears, click **Close**.
The execution results near the bottom of the PDI window show updated metrics in the **Step Metrics** tab.
**Apply formatting to your transformation**
Follow these steps to clean up the field layout on your lookup stream so that it matches the format and layout of the other stream going to the Write to Database step.
1. Add a Select Values step to your transformation by expanding the **Transform** folder and clicking Select Values.
2. Create a hop from the Lookup Missing Zips to the Select Values step.
3. Double-click the Select Values step to open its properties dialog box.
4. Rename the Select Values step to Prepare Field Layout.
5. Click **Get fields to select** to retrieve all fields and begin modifying the stream layout.
6. In the **Fields** list, find the **#** column and click the number for the **ZIP\_RESOLVED** field.
Use CTRL+UP (Windows/Linux) or COMMAND+UP (macOS) to move **ZIP\_RESOLVED** just below the **POSTALCODE** field, which is the one that still contains null values.
7. Select the old **POSTALCODE** field in the list (line 20), right-click in the line, and select **Delete Selected Lines**
8. The original **POSTALCODE** field was formatted as a 9-character string. You must modify your new field to match the form. Click the **Meta-Data** tab.
9. In the first row of the **Fields to alter table the meta-data for** section, click in the **Fieldname** column and select **ZIP\_RESOLVED**. Perform the following steps:
1. Enter `POSTALCODE` in the **Rename to** column.
2. Select **String** in the **Type** column and enter `9` in the **Length** column.
3. Click **OK** to exit the **edit properties** dialog box.
10. Draw a hop from the Prepare Field Layout (Select values) step to the Write to Database (Table output) step.
11. When prompted, select the **Main output of the step** option.
12. Save your transformation.
#### Step 4: Clean the data
After completing Step 3: Resolve missing data, you can further cleanse and categorize the data into buckets before loading it into a relational database. In this section, you will cleanse the `COUNTRY` field data by mapping `United States` to `USA` using the Value mapper step. Cleaning the data ensures there is only one version of `USA`.
In addition, you will learn how to use buckets for categorizing the `SALES` data into small, medium, and large categories using the Number range step. You will learn how to insert these cleaning and categorizing functions into your transformation just prior to the Write to Database step on the canvas.
**Add a Value mapper step to the transformation**
Follow these steps to add the Value mapper step to the transformation.
1. Delete both hops connected to the Write to Database step. For each hop, right-click and select **Delete**.
2. Create some extra space on the canvas. Drag the Write to Database step toward the right side of your canvas.
3. Add the Value mapper step to your transformation by expanding the **Transform** folder and choosing **Value mapper**.
4. Create a hop between the Filter Missing Zips and Value mapper steps. In the dialog box that appears, select **Result is TRUE**.
5. Create a hop between the Prepare Field Layout and Value mapper steps. When prompted, select the **Main output of the step** option.
**Set the properties in the Value Mapper step**
Follow these steps to set the properties in the Value mapper step.
1. Double-click the Value mapper step to open its properties dialog box.
2. Click in the **Fieldname to use** field and select **COUNTRY**.
3. In the **Field Values** table, define the `United States` and `USA` field values.
1. In row #1, click the field in the **Source value** column and enter `United States`
2. Then, click the field in the **Target value** column and enter `USA`
4. Click **OK**.
5. Save your transformation.
**Apply ranges**
Follow these steps to apply ranges to your transformation.
1. Add a Number range step to your transformation by expanding the **Transform** folder and selecting **Number range**.
2. Create a hop between the **Value mapper** and **Number range** steps.
3. Create a hop between the Number range and Write to Database (which was built using Table output) steps. When prompted, select the **Main output of the step** option.
4. Double-click the Number range step to open its **properties** dialog box.
5. Click in the **Input field** and select **SALES** from the list.
6. In the **Output field** enter `DEALSIZE`.
7. In the **Ranges (min <=x< max)** table, define the **Lower Bound** and **Upper Bound** field ranges along with the bucket **Value**.
1. In row #1, click the field in the **Upper Bound** column and enter `3000.0`. Then, click the field in the **Value** column and enter `Small`.
2. In row #2, click the field in the **Lower Bound** column and enter `3000.0`. Then, click the field in the **Upper Bound** column and enter `7000.0`. Click the field in the **Value** column and enter `Medium`.
3. In row #3, click the field in the **Lower Bound** column and enter `7000.0`. Then, click the field in the **Value** column and enter `Large`.
8. Click **OK**.
**Execute the SQL statement**
Your database table does not yet contain the field `DEALSIZE`. Perform these steps to execute the SQL statement.
1. Double-click the Write to Database step to open its properties dialog box.
2. Click the **SQL** button at the bottom of the window to generate the new DDL for editing your original target table. Note that the Write to Database step was built using Table output.
1. The **Simple SQL editor** window appears with the SQL statements needed to `alter` the table.
2. Click **Execute** to execute the SQL statement.
3. The Results of the SQL statements window appears. Examine the results, then click **OK** to close the window.
4. Click **Close** in the Simple SQL editor window to close it.
5. Click **OK** to close the Write to Database window. Note that the Write to Database step was built using Table output
3. Save your transformation.
#### Step 5: Run the transformation
Pentaho Data Integration provides a number of deployment options. The **Running a Transformation** section in the **Pentaho Data Integration** document explains these and other options available for execution. In this section of the tutorial, you create a transformation using the **Local** run option.
1. In the PDI client window, select **Action** > **Run**.
The Run Options window appears.
2. Keep the default **Pentaho local** option for this exercise.
It uses the native Pentaho engine and runs the transformation on your local machine. See the **Pentaho Data Integration** document if you are interested in setting up configurations that use another engine.
3. Click **Run**.
The transformation executes.
After the transformation runs, the **Execution Results** panel opens below the canvas.
**Viewing the execution results**
Use the tabs in the **Execution Results** section of the window to view how the transformation executed, pinpoint errors, and monitor performance.
* **Step Metrics**
Provides statistics for each step in your transformation including how many records were read, written, or caused an error, as well as processing speed (rows per second) and more. This tab also indicates whether an error occurred in a transformation step.
This tutorial introduces no intentional transformation errors, so the transformation should run correctly. If a mistake does occur, you can view the steps that caused the transformation to fail highlighted in red. In the example below, the Lookup Missing Zips step caused an error.
* **Logging**
Shows the logging details for the most recent execution of the transformation. It also allows you to drill deeper to determine where errors occur. Error lines are highlighted in red. In the example below, the **Lookup Missing Zips** step caused an error because it attempted to look up values on a field called **POSTALCODE2** which did not exist in the lookup stream.
* **Execution History**
Provides access to the step metrics and log information from previous executions of the transformation. This feature works only if you have configured your transformation to log to a database through the **Logging** tab of the Transformation Settings dialog box.
* **Performance Graph**
Analyzes the performance of steps based on a variety of metrics including how many records were read, written, or caused an error, as well as processing speed (rows per second) and more. Like the execution history, this feature requires you to configure your transformation to log to a database through the **Logging** tab found in the Transformation Settings dialog box.
* **Metrics tab**
Shows a Gantt chart after the transformation or job runs. This information includes how long it takes to connect to a database, the time spent executing a SQL query, or the load time of a transformation.
* **Preview Data**
Shows a preview of the data.
#### Step 6: Orchestrate with jobs
Jobs are used to coordinate ETL activities such as:
* Defining the flow and dependencies that control the linear order for the transformations to run.
* Preparing for execution by checking conditions such as, "Is my source file available?" or "Does a table exist?"
* Performing bulk load database operations.
* Assisting file management, such as posting or retrieving files using FTP, copying files, and deleting files.
* Sending success or failure notifications through email.
For this part of the tutorial, imagine that an external system is responsible for placing your `sales_data.csv` input in its source location every Saturday night at 9 p.m. You want to create a job that will verify that the file has arrived and then run the transformation to load the records into the database. In a subsequent exercise, you will schedule the job to run every Sunday morning at 9 a.m.
The following steps assume that you have built a Getting Started transformation as described in Step 1: Extract and load data of the tutorial.
1. Go to **File** > **New** > **Job**.
2. Expand the **General** folder and drag a Start job entry onto the canvas.
The **Start** job entry defines where the execution will begin.
**Note:** Jobs run in a sequential order of steps and transformations can run in a parallel order of steps.
3. Expand the **Conditions** folder and add a File Exists job entry.
4. Draw a hop from the Start job entry to the File Exists job entry.
5. Double-click the File Exists job entry to open its properties dialog box. Click **Browse** and set the filter near the bottom of the window to **All Files**. Select the `sales_data.csv` from the following directory: `...\design-tools\data-integration\samples\transformations\files`.
6. Click **OK** to exit the Open File window.
7. Click **OK** to exit the Check if a file exists window.
8. Expand the **General** folder and add a Transformation job entry.
9. Draw a hop between the File Exists and the Transformation job entries.
10. Double-click the Transformation job entry to open its properties dialog box.
11. Click **Browse** to open the **Select repository object** window. Browse to and select the **Getting Started** transformation.
12. Click **OK** to close the Transformation window.
13. Save your job as **Sample Job**.
14. Click **Run** icon in the toolbar. When the Run Options window appears, select **Local** environment type and click **Run**. The **Execution Results** panel should open showing you the job metrics and log information for the job execution.
### PDI job tutorial
This is a shorter, standalone version of the job exercise.
Jobs are used to coordinate ETL activities such as:
* Defining the flow and dependencies for what order transformations should be run.
* Preparing for execution by checking conditions such as, "Is my source file available?" or "Does a table exist?"
* Performing bulk load database operations.
* File management such as posting or retrieving files using FTP, copying files and deleting files.
* Sending success or failure notifications through email.
For this exercise, imagine that an external system is responsible for placing your `sales_data.csv` input in its source location every Saturday night at 9 p.m. You want to create a job that will check to see that the file has arrived and run your transformation to load the records into the database. In a subsequent exercise, you will schedule the job to be run every Sunday morning at 9 a.m.
To complete this exercise, you must have completed the exercises in the [Pentaho Data Integration (PDI) tutorial](#pentaho-data-integration-pdi-tutorial).
1. Go to **File** > **New** > **Job**.
2. Expand the **General** folder and drag a Start job entry onto the graphical workspace.
The **Start** job entry defines where the execution will begin.
3. Expand the **Conditions** folder and add a File Exists job entry.
4. Draw a hop from the Start job entry to the File Exists job entry.
5. Double-click the File Exists job entry to open its Edit Properties dialog box. Click **Browse** and set the filter near the bottom of the window to **All Files**. Select the `sales_data.csv` from the following location: `...\design-tools\data-integration\samples\transformations\files`.
6. Click **OK** to exit from the Open File window.
7. Click **OK** to exit from the Check if a file exists window.
8. In Spoon, expand the **General** folder and add a Transformation job entry.
9. Draw a hop between the File Exists and the Transformation job entries.
10. Double-click the Transformation job entry to open its edit Properties dialog box.
11. Click **Browse** to open the **Select repository object** window. Browse to and select the transformation you created in the [Pentaho Data Integration (PDI) tutorial](#pentaho-data-integration-pdi-tutorial).
12. Expand the repository tree to find your sample transformation. Select it and click **OK**.
13. Save your job as Sample Job.
14. Click **Run** icon in the toolbar. When the Run Options window appears, choose **Local** environment type and click **Run**. The **Execution Results** panel should open showing you the job metrics and log information for the job execution.
### Getting started with PDI and Hadoop
Pentaho provides a complete big data analytics solution that supports the entire big data analytics process. From big data aggregation, preparation, and integration, to interactive visualization, analysis, and prediction, Pentaho allows you to harvest the meaningful patterns buried in big data stores. Analyzing your big data sets gives you the ability to identify new revenue sources, develop loyal and profitable customer relationships, and run your organization more efficiently and cost effectively.
### Next steps
The tutorials above are designed to quickly demonstrate basic PDI features.
For more detailed information about PDI features and functions, see the following topics in the **Pentaho Data Integration** document:
* **Learn about the PDI Client**
* **Use Pentaho Repositories in PDI**
* **Schedule Perspective in the PDI Client**
# Pentaho, big data, and Hadoop
The term big data applies to very large, complex, or dynamic datasets that need to be stored and managed over a long time. To derive benefits from big data, you need the ability to access, process, and analyze data as it is being created. However, the size and structure of big data makes it very inefficient to maintain and process it using traditional relational databases.
Big data solutions re-engineer the components of traditional databases, such as data storage, retrieval, query, processing, and massively scale them.
### In this topic
* [Big data overview](#big-data-overview)
* [About Hadoop](#about-hadoop)
* [Big data resources](#big-data-resources)
### Big data overview
Pentaho increases speed-of-thought analysis against even the largest of big data stores by focusing on the features that deliver performance.
* Instant access: Pentaho provides visual tools to make it easy to define the sets of data that are important to you for interactive analysis. These data sets and associated analytics can be easily shared with others, and as new business questions arise, new views of data can be defined for interactive analysis.
* High performance platform: Pentaho is built on a modern, lightweight, high performance platform. This platform fully leverages 64-bit, multi-core processors and large memory spaces to efficiently leverage the power of contemporary hardware.
* Extreme-scale, in-memory caching: Pentaho is unique in leveraging external data grid technologies, such as Infinispan and Memcached to load vast amounts of data into memory so that it is instantly available for speed-of-thought analysis.
* Federated data integration: Data can be extracted from multiple sources, including big data and traditional data stores, integrated together and then flowed directly into reports, without needing an enterprise data warehouse or data mart.
### About Hadoop
The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. It is designed to scale up from single servers to thousands of machines, each offering local computation and storage. Rather than rely on hardware to deliver high-availability, the library itself is designed to detect and handle failures at the application layer, so delivering a highly-available service on top of a cluster of computers, each of which may be prone to failures.
A Hadoop platform consists of a Hadoop kernel, a [MapReduce](http://wiki.apache.org/hadoop/MapReduce) model, a distributed file system, and often a number of related projects—such as [Apache Hive](http://hive.apache.org/), [Apache HBase](http://hbase.apache.org/), and others.
A Hadoop Distributed File System, commonly referred to as HDFS, is a Java-based, distributed, scalable, and portable file system for the Hadoop framework.
### Big data resources
The following resources may help in understanding big data architecture and components:
* [Pentaho Big Data Analytics Center](http://www.pentahobigdata.com/resources)
* [Apache Hadoop project](http://hadoop.apache.org/) -- A project that contains libraries that allows for the distributed processing of large data sets across clusters of computers using simple programming models. There are several modules, including the [Hadoop Distributed File System (HDFS)](http://wiki.apache.org/hadoop/HDFS), which is a distributed file system that provides high-throughput access to application data and [Hadoop MapReduce](http://wiki.apache.org/hadoop/MapReduce), which is a key algorithm to distribute work around a cluster.
* [Avro](http://avro.apache.org/)—A data serialization system
* [HBase](http://hbase.apache.org/)—A scalable, distributed database that supports structured data storage for large tables
* [Hive](http://hive.apache.org/)—A data warehouse infrastructure that provides data summarization and on-demand querying
* [ZooKeeper](http://zookeeper.apache.org/)—A high-performance coordination service for distributed applications
* [MongoDB](http://www.mongodb.org/)— A NoSQL open source document-oriented database system developed and supported by 10gen
* [Splunk](http://www.splunk.com/) - A data collection, visualization and indexing engine for operational intelligence that is developed by Splunk, Inc.
* [CouchDB](http://couchdb.apache.org/)—A NoSQL open source document-oriented database system developed and supported by Apache
* [Sqoop](http://sqoop.apache.org/)—Software for transferring data between relational databases and Hadoop
* [Oozie](http://oozie.apache.org/)—A workflow scheduler system to manage Hadoop jobs
# About Pentaho workflows
The route that you take with Pentaho depends on your expertise, business needs, and data. It also depends on what you want to analyze and report. During an evaluation, you might use both tracks.
All of the products are integrated to work smoothly together, regardless of which track you ultimately choose. We provide specific details within the workflow discussions, however, here are the high-level use cases for each track.
* Business Analytics (BA) Track: Great for analysis and reporting. Meant primarily for business users and does not require special skills to successfully use the components involved. This track enables anyone to build Pentaho solutions without using programming or having deep understanding of data structures.
* Data Integration (DI) Track: Meant for data design professionals and requires a working knowledge of data structures and modeling, as well as extract, transform, and load (ETL) processes. With this track, you can directly manipulate data from multiple sources, making it scalable and efficient for enterprise-wide analysis and reporting.
Each track has three workflows: one for Evaluation, one for Development, and one for Production.
* Evaluate and Learn: If you used the trial download on the Pentaho website and want to get a hands-on feel for the components that are best for your implementation, follow the Evaluation Workflow.
* Develop Pentaho Solutions: After you have figured out which components are best for you and how to use them, the Develop Workflow is the process you use to build, change, and test Pentaho solutions until they meet your production requirements.
* Go Live for Production: When your solution is working just right, the Go Live Workflow shows how to move your solution from development to production.
### In this topic
* [Prepare for the evaluation](#prepare-for-the-evaluation)
* [Pentaho Data Integration workflows](#pentaho-data-integration-workflows)
* [Pentaho Business Analytics workflow](#pentaho-business-analytics-workflow)
### Prepare for the evaluation
This table guides you through the differences between the Business Analytics and Data Integration tracks. It also helps you decide which track to follow for evaluation. You may choose to follow one track, then the other, while you are exploring the software.
| Explore Considerations | Choose Options | |
| --------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Business Analytics Evaluation](/pdia-try-pdia/archive-flattened-pages/evaluate-and-learn-pentaho-business-analytics) | [Data Integration Evaluation](/pdia-try-pdia/archive-flattened-pages/evaluate-and-learn-pentaho-data-integration-pdi) | |
| Expertise |
No special skills required
Knowledge of business requirements and what reports and analysis should show
|
Knowledge of business requirements
Understanding of data structures and modeling
Knowledge of extract, transform, and load (ETL) processes
|
| Data Set Description |
Single source of data
Data from multiple sources that have been transformed and joined into a single data mart or warehouse
Small data sets
|
Multiple sources of data
Data you want to transform and join in one or more data marts or warehouses
Large to enormously vast data sets
|
| Reporting Options | Offers a wide variety of visualization and reporting options. | Offers more limited but focused reporting options that help you visualize and analyze data. BA tools can be used to generate reports based on DI-processed data. |
| Data Storage Types |
Relational databases
CSV data sources
SQL queries
|
Relational databases
NoSQL or Hadoop databases
Big data of any types
Data from a web service
|
| Recommendation | Best used by business analysts, managers, report designers, individual business units within an organization or enterprise | Best used by data scientists, data modelers, data integration and ETL developers, individual business units within an organization or enterprise, and enterprise-wide implementations |
Now that you have an idea of which track you want to follow for evaluation, choose an evaluation method. This decision table explains the different options for evaluation so you can pick the option that works best for you.
| Explore Considerations | Choose Options | | |
| ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| [Hosted Demo](http://www.pentaho.com/hosted-demo) | [Custom Prototype](http://www.pentaho.com/helpmeout) | [Trial Download](http://www.pentaho.com/download) | |
| Track | Business Analytics | Business Analytics or Data Integration | Business Analytics or Data Integration |
| Summary | A cloud-based, hands-on, interactive exploration of Business Analytics reports, analysis, visualizations, and dashboards. Here you can see how easy and fun it can be to use Pentaho. | Work with Pentaho analysts and data integration specialists to plan and build a complimentary custom prototype that illustrates what Pentaho can do with your data. A representative will guide you through the entire process. | Using our trial software, tutorials, and documentation, install and configure your own work environment. Then, build a prototype to get a complete Pentaho experience from installation and administration, through creating your first data models and build reports, analysis, dashboards, and data integration ETL transformations. |
| Data Source | Pentaho sample data in CSV format | Your sample data, including a range of typical data characteristics in CSV format. | Your sample data, including a range of typical data characteristics in the format that you commonly use |
| Hardware/Software Requirements | Web browser | Varies, depending on your requirements. | One computer that meets the server requirements stated in the [Components Reference](/pdia-try-pdia/components-reference). |
| Recommendation |
Any evaluator who wants an overview of Business Analytics features.
Recommended for business analysts and report designers.
Any evaluator who wants to independently work with Business Analytics, Data Integration tools, and big data. - Recommended for evaluators who want to explore Business Analytics and Data Integration features using their own data.
|
### Pentaho Data Integration workflows
Pentaho Data Integration is a robust extract, transform, and load (ETL) tool that you can use to integrate, manipulate, and visualize your data. You can use PDI to import, transform, and export data from multiple data sources, including flat files, relational databases, Hadoop, NoSQL databases, analytic databases, social media streams, and operational stores. You can also use PDI to clean and enrich the data, move data between databases, and to visualize your data.
#### In this topic
* [Evaluate and learn PDI](#evaluate-and-learn-pdi)
* [Develop your PDI solution](#develop-your-pdi-solution)
* [Go Live for production - DI](#go-live-for-production---di)
* [Commonly used PDI steps and entries](#commonly-used-pdi-steps-and-entries)
#### Evaluate and learn PDI
As you explore Pentaho Data Integration (PDI), you will be introduced to the major components, watch videos, work through hands-on examples, and read about the different features.
Review the documentation and contact Pentaho [sales support](https://www.pentaho.com/services) if you have questions.
**PDI basics**
This section familiarizes you with PDI and introduces you to basic terminology and concepts. Then, you learn how to start and configure Spoon and take a spin through the interface.
* Get a basic understanding of what PDI does.
* View a video that explains how PDI fits into the [Business Analytics Platform](http://www.youtube.com/watch?v=hCMtrLCsBuE).
* Read about Pentaho Data Integration architecture in the **Pentaho Data Integration** document.
**Get acquainted with the PDI client**
Spoon is the PDI design tool. In this section you will set up Spoon, take a tour of the Spoon interface, and learn about the different Spoon perspectives.
* Check out the [hardware and software requirements](/pdia-try-pdia/components-reference) for PDI.
* [Download trial version](http://www.pentaho.com/download) of the Pentaho Suite and install the software. (The platform includes PDI.)
* Learn how to install PDI only. See [Custom installation](/pdia-try-pdia/archive-flattened-pages-do-not-use/install-trial-legacy-child-pages/custom-installation) for details.
* Configure the Pentaho Server. Depending on your platform, see [Increase Pentaho Server memory limit for installations on Linux](https://github.com/pentaho/documentation/blob/main/PDIA/11.0/Get%20Started/Pentaho%20evaluation/Increase%20Pentaho%20Server%20memory%20limit%20for%20installations%20on%20Linux=GUID-EBDD031B-404B-4E8C-B48E-75AC258CDF52=3=en=.md) or [Increase Pentaho Server memory limit for installations on Windows](/pdia-try-pdia/archive-flattened-pages-do-not-use/install-trial-legacy-child-pages/increase-pentaho-server-memory-limit-for-installations-on-windows) for details.
* Start the Pentaho Server. Depending on your platform, see [Start and stop the Pentaho Server for configuration on Linux](https://github.com/pentaho/documentation/blob/main/PDIA/11.0/Get%20Started/Pentaho%20evaluation/Start%20and%20stop%20the%20Pentaho%20Server%20for%20configuration%20on%20Linux=GUID-2AF5899D-DD81-4F5E-B884-BA998F6DBC2A=1=en=.md) or [Start and stop the Pentaho Server for configuration on Windows](/pdia-try-pdia/archive-flattened-pages-do-not-use/install-trial-legacy-child-pages/start-and-stop-the-pentaho-server-for-configuration-on-windows) for details.
* Access the PDI client. See the **Pentaho Data Integration** document for details.
* Tour the PDI client perspectives. See the **Pentaho Data Integration** document for details.
* Read about terminology and basic concepts in the **Pentaho Data Integration** document.
**Build transformations and jobs**
Now that your environment is set up and you are familiar with the PDI client, you are ready to build transformations and jobs. Trying the following task may be helpful.
* Create a connection to the Pentaho Repository.
* Work through the exercise on [Creating a Transformation](/pdia-try-pdia/archive-flattened-pages/pdi-transformation-tutorial) that involves a flat file. Click through the links at the bottom of the page to complete the exercise.
* Create a job to execute the transformation.
* Schedule a job to execute the transformation at a later time.
* Review [commonly used steps and job entries](#commonly-used-pdi-steps-and-entries).
**Explore Big Data and Streamlined Data Refinery**
In this section, you will learn how to use transformation steps to connect to a variety of big data sources, including Hadoop, NoSQL, and analytical databases such as MongoDB. You can then try working through the detailed, step-by-step tutorials, and peruse the out-of-the-box steps that Spoon provides. Learn how to work with Streamlined Data Refinery. Then, you will have an opportunity to move beyond the basics and learn how to edit transformations and metadata models.
* Watch one of our [Big Data Videos](http://www.youtube.com/watch?v=vOMOFPMnXgk).
* Learn how to work with Streamlined Data Refinery. See **Pentaho Data Integration** for details.
* Learn how to auto model using the Build Model. See **Pentaho Data Integration** for details. job entry and how this feature intersects with Analyzer.
* Find out what big data steps are available out-of-the-box. See [Commonly used PDI steps and entries](#commonly-used-pdi-steps-and-entries) for details.
* Find out which Hadoop distributions are available and how to configure them. See [Pentaho, big data, and Hadoop](/pdia-try-pdia/pentaho-big-data-and-hadoop) for details.
**Note:** You should already have a cluster set up to perform this task.
* Edit transformations and metadata models. See **Pentaho Data Integration** for details.
* Watch a video about how to use PDI to [blend Big Data](http://www.youtube.com/watch?v=JyypUdwySrQ).
**About Kitchen, Pan, and Carte**
Kitchen, Pan, and Carte are command line tools for executing transformations and jobs modeled in the PDI client.
* Use Pan and Kitchen command line tools to work with transformations and jobs
* Use Carte clusters to:
* Run transformations and jobs on a Carte cluster.
* Schedule jobs to run on a remote Carte server.
* Start or stop Carte from the command line interface or a URL.
* Run transformations and jobs from the repository on the Carte server
See the **Pentaho Data Integration** document for details on Kitchen, Pan, and Carte.
**Learn more**
Now that you have completed an initial evaluation of PDI, dig a little deeper. Find out how to:
* Use newer steps and entries, like Spark Submit. See the **Pentaho Data Integration** document for details.
* Read about how to turn a transformation into a data service. See the **Pentaho Data Integration** document for details.
* Use the ETL Metadata Injection step. See the **Pentaho Data Integration** document for details.
* Check out our **What's New** document.
* Create other Data Integration solutions. See the **Pentaho Data Integration** document for details.
* Administer PDI. See the administration documentation for details.
* Integrate with different security protocols, like Pentaho security, LDAP, MSAD, and Kerberos. See the administration documentation for details.
* Check out our developer center section in the administration documentation.
#### Develop your PDI solution
This workflow helps you to set up and configure the DI development and test environments, then build, test, and tune your Pentaho DI solution prototype. This process is similar to the trial download evaluation experience, except that you will be completely configuring the Pentaho Server for data integration and working with your own ETL developers.
If you need extra help, Pentaho [professional services](https://www.pentaho.com/services) is available. The end result is to learn DI implementation best practices and deploy your DI solution to a production server. Most development and testing for DI occurs in Spoon.
Before you begin developing your DI solution, we recommend that you attend Pentaho [training classes](http://www.pentaho.com/service/training) to learn how to install and configure the Pentaho Server, as well as how to develop data models.
This section is grouped into parts that will guide you during the development of your DI solution. These parts are iterative and you might bounce between them during development. For example, as you tune a job, you might find that although you have built a solution that produces the right results, it takes a long time to run. You might need to rebuild and test a transformation to improve efficiency, and then retest it.
**Design DI solution**
Design helps you think critically about the problem you want to solve and possible solutions. Consider these questions as you gather your requirements and design the solution.
* **Output**
What does the overall solution look like? What questions are posing and how do you want the answers formatted?
* **Data Sources**
What type(s) of data sources are you querying? Where are they located? How much data do you need to process? Are you using big data? Are you using relational or non-relational data sources? Will you have a target data source? If so, where are they located?
* **Content/Processing**
What data quality issues do you have? How is the input data mapped to the output data? Where do you want to process the content, in PDI or in the data source? What hardware will you include in your development environment? Will you need one or more quality assurance test environments or production environments?
Also, consider templates or standards, naming conventions, and other requirements of your end users if you have them. Consider how you will back up your data as well.
**Set up a development environment**
Setting up the environment includes installing and configuring PDI on development computers, configuring clustering if needed, and connecting to data sources. If you have one or more quality assurance environments, you will need to set those up also.
| Task | Do This | Objective |
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| Verify System Requirements |
Acquire one or more servers that meet the requirements.
Obtain the correct drivers for your system.
|
| Obtain Software and Install PDI |
See the Install Pentaho Data Integration and Analytics document for following instructions:- Installing PDI
Starting the Pentaho Server
Starting the PDI client (also known as Spoon)
|
Get the software from your Sales Support representative.
Install the software.
Start the Pentaho Server and Spoon.
|
| Install licenses for the Pentaho Server | See the **Administer Pentaho Data Integration and Analytics** document for instructions on installing licenses. |
Add all acquired Pentaho licenses.
|
| Connect to the Pentaho Repository | See the **Pentaho Data Integration** for instructions on connecting to the Pentaho Repository. |
Connect to the Pentaho Repository.
|
| Apply Advanced Security (if needed) | See the **Administer Pentaho Data Integration and Analytics** document for details on Advanced Security. |
Determine whether you need to apply Advanced Security.
|
**Build and test solution**
During this step, you develop transformations, jobs, and models, then test what you have developed. You will tune the transformations, jobs, and models for optimal performance.
Development occurs in the PDI client design tool. The PDI client's streamlined design tightly couples the build and test activities so that you can easily perform them iteratively. The PDI client has perspectives to help you perform ETL and visualize data. The PDI client also provides a scheduling perspective that can be used to automate testing. Testing encompasses verifying the quality of transformations and jobs, reviewing visualizations, and debugging issues. One common method of testing is to include steps in a transformation or job that calculates hash totals, checksums, record counts, and so forth to determine whether data is being properly processed. You can also visualize your data in analyzer and report designer and review the results as you develop. This can not only help you find errors and issues with processing but can help you get a jump on user acceptance testing if you show these reports to your customers or business analysts to get early feedback.
One basic question is how you can determine the number of transformations and jobs needed, as well as the order in which they should be executed. A good rule of thumb is to create one transformation for each combination of source system and target tables. You can often identify combinations in your mapping documents. Once you have identified the number of transformations that you need, you can use the same process to determine that number of jobs that you need. When considering the order of execution for transformations and jobs, consider how referential integrity is enforced. Run target table transformations that have no dependencies first, then run transformations that depend on those tables, and so forth.
| Task | Do This | Objective |
| ---------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Understand the Basics |
Read the overview of the PDI client process in the Pentaho Data Integration document.
|
Review information about the process and perspectives.
Review available transformations and determine how you can use them for your solution.
Review job step references to identify which steps can be used in your solution.
|
| Create and Run Transformations |
Create and run a transformation. See the Pentaho Data Integration document for details.
|
Identify the transformations needed for your job and implement them.
Save transformation.
Run transformations locally.
|
| Create and Run a Job |
Create and run a job. See the Pentaho Data Integration document for details.
|
Create a job.
Arrange transformations in a job so that they execute logically.
Run a job.
|
**Tune solution**
Fine tune transformations and jobs to optimize performance. This involves using various tools such as the DI Operation and Audit Mart to determine where bottlenecks or other performance issues occur, and addressing them.
| Task | Do This | Objective |
| ------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Review the Performance Tuning Checklist and Make Changes to Transformations and Jobs |
Review tuning tips. See the Administer Pentaho Data Integration and Analytics document for tuning tips.
|
Get familiar with things that you can do to optimize performance.
Apply tuning tips as needed.
|
| Consider other performance tuning options |
Read about transactional databases. See the Pentaho Data Integration document for details on transactional databases.
Read about using logs. See the Administer Pentaho Data Integration and Analytics document for details on logging.
|
Learn how to apply transactional databases.
Learn how to use logs to tune transformations and jobs.
|
**Next steps**
These resources will be helpful to you as you prepare to Go Live for Production:
* Prepare to [Go Live for Production - DI](#go-live-for-production---di).
* [Support Portal](https://support.pentaho.com/hc/en-us): check with Support for service packs.
#### Go Live for production - DI
Go Live is the process by which you migrate a prototype to production. This process is divided into four parts:
* Setting up the production environment
* Deploying the solution
* Tuning the solution
* Scheduling the runs
**Set up production environment**
Setting up the environment includes installing the software on production computers, configuring clustering, and connecting to data sources. To set up the environment, install and configure the Pentaho Server, Spoon, and any plugins required. Then set up data sources and clusters.
| Task | Do This | Objective |
| ---------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Verify system requirements |
Acquire one or more servers that meet the requirements.
Obtain the correct drivers for your system.
|
| Obtain software and install the Pentaho Server |
Download the Pentaho software.
Start the Pentaho Server. See Install Pentaho Data Integration and Analytics for details.
Start the PDI client. See Pentaho Data Integration for details.
Install the licenses (if necessary). See Administer Pentaho Data Integration and Analytics for details.
|
Get the software from your Sales Support representative.
Install the software.
|
| Change the Server Fully Qualified URL |
Change the ports and URLs. See Administer Pentaho Data Integration and Analytics for details.
|
Change the server's URL so that you do not have a conflict.
|
| Connect to the Pentaho Repository |
Create a connection to the Pentaho Repository. See Pentaho Data Integration for details.
|
Connect to the Pentaho Repository.
|
| Set up clusters |
Optional: Set up clusters. See Pentaho Data Integration for details.
|
Become familiar with clustering.
Set up clusters, if they are needed in your environment.
|
| Copy configuration files | Copy `shared.xml`, `repositories.xml`, `kettle.properties`, and JAR files from the development environment to the production environment. |
System is set up and ready for production.
|
| Logging and monitoring your server |
Review logging and monitoring operations. See Pentaho Data Integration for details.
Enable logging. See Administer Pentaho Data Integration and Analytics for details.
Monitor PDI and SNMP traps. See Administer Pentaho Data Integration and Analytics for details.
|
Learn about the different ways to log and monitor Pentaho Server operations:
Log through Spoon and Carte
Use SNMP traps with PDI
|
**Deploy solution**
Export solutions from the Pentaho Repository that is in the development or test environments, to the Pentaho Repository that is in the production environment.
| Task | Do This | Objective |
| ------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| Export and Import Pentaho Repository |
See Export and Import Pentaho Repository Content in the Administer Pentaho Data Integration and Analytics document.
|
Export Pentaho Repository content from test environment
Import Pentaho Repository content to production environment
|
**Tune solution**
Fine tune transformations and jobs to optimize performance. This involves using various tools such as the DI Operations and Audit Marts to determine where bottlenecks or other performance issues occur, and attempting to address them.
| Task | Do This | Objective |
| ------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| Review the Performance Tuning Checklist and Make Changes to Transformations and Jobs |
Consult the tuning tips. See the Administer Pentaho Data Integration and Analytics document for tuning tips.
|
Get familiar with things that you can do to optimize performance.
Apply tuning tips as needed.
|
| Consider other performance tuning options |
Learn about transactional databases. See the Pentaho Data Integration document for details on transactional databases.
Learn about using logs. See the Administer Pentaho Data Integration and Analytics document for details on logging.
|
Learn how to apply transactional databases.
Learn how to use logs to tune transformations and jobs.
|
**Schedule runs**
Use the PDI client, Pan, or Kitchen to schedule executions of transformations and jobs.
| Task | Do This | Objective |
| ---------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------- |
| Schedule Transformations and Jobs From Spoon |
Schedule transformations and jobs. See the Pentaho Data Integration document for details.
|
Schedule transformations and jobs
|
| Command Line Scripting Through Pan and Kitchen |
Learn about Pan's options. See the Pentaho Data Integration document for details.
Learn about Kitchen's options. See the Pentaho Data Integration document for details.
|
Use Pan and Kitchen to schedule transformations and jobs.
|
**Next steps**
These resources will be helpful to you after your production server is live.
* Fine-tune Pentaho systems: Provides guidance on how to maintain and fine-tune your Pentaho Server. See the **Administer Pentaho Data Integration and Analytics** document for details.
* Pentaho [Training and Education](https://www.hitachivantara.com/en-us/services/training-certification.html)
* [Support Portal](https://support.pentaho.com/hc/en-us): Check with support for service packs.
#### Commonly used PDI steps and entries
Although there are over 330 transformation steps and job entries, some steps and entries are used more often than others. If you are creating a transformation and job, but do not know where to begin, this list might be helpful to you.
**Top ten transformation steps**
PDI transformation steps are documented in Pentaho Data Integration.
* Text File Input
* Table Input
* Microsoft Excel Input
* Text File Output
* Table Output
* Microsoft Excel writer
* Select Values
* Filter Rows
* Group By
* Stream Lookup
**Other commonly used transformation steps**
PDI transformation steps are documented in Documentation
* INPUT: Generate Rows, Data Grid, Get Data from XML, CSV File Input, Fixed File Input
* OUTPUT: XML Output
* TRANSFORM: Split Fields, Calculator, Add Constants, Add Sequence, Replacing Strings, Split Fields, Sort Rows, String Operations, Strings Cut
* SCRIPTING: User Defined Java Class, Modified Java Script Value, User Defined Java Expression
* FLOW: Abort, Append Streams, Block this step until steps finish, Blocking Step, Detect Empty Stream, Dummy, ETL Metadata Injection, Filter Rows, Identify Last Row in a Stream, Java Filter, Job Executor, Prioritize Streams, Single Threader, Switch/Case, Transformation Executor
* LOOKUP
* JOINS: Join Rows, Merge Join
* JOB: Get Variables, Set Variables
**Commonly used job entries**
PDI job entries are documented in documentation.
* GENERAL: Start, Job, Transformation, Success
* UTILITY: Abort
* MAIL: Mail
* FILE MANAGEMENT: Add filenames to result, Compare folders, Convert file between Windows and Unix, Copy Files, Create a folder, Create file, Delete file, Delete filenames from result, Delete files, Delete folders, File Compare, HTTP, Move Files, Process result filenames, Unzip file, Wait for file, Write to file, Zip file
* UTILITIES: Write to log
### Pentaho Business Analytics workflow
Pentaho Business Analytics is a combined business analytics and data integration platform that allows business users, data scientists, and IT administrators to easily access, explore, and visualize their data. Pentaho empowers business users to make information-driven decisions that positively impact their organization’s performance, data scientists to use a full-spectrum of tools to create robust data models, and IT to rapidly deliver a secure, scalable, flexible, and easy to manage business analytics platform for the broadest set of users.
#### Workflow stages
Use these sections to move from evaluation to production:
* [Evaluate and learn Pentaho Business Analytics](#evaluate-and-learn-pentaho-business-analytics)
* [Develop your BA environment](#develop-your-ba-environment)
* [Go live for production - BA](#go-live-for-production---ba)
#### Evaluate and learn Pentaho Business Analytics
As you explore Pentaho Business Analytics, you will be introduced to the major components, watch videos, work through hands-on examples, and learn about the different features.
Go at your own pace. Feel free to dig into the documentation or to contact Pentaho [sales support](https://www.pentaho.com/services) if you have questions.
Use the sections below to get familiar with Business Analytics:
* [Tour the User Console and create your first reports](#tour-the-user-console-and-create-your-first-reports)
* [Explore and learn data source basics](#explore-and-learn-data-source-basics)
* [Learn about Report Designer](#learn-about-report-designer)
* [Discover more about Pentaho Business Analytics](#discover-more-about-pentaho-business-analytics)
* [Next steps](#next-steps-evaluation)
**Tour the User Console and create your first reports**
The User Console is a web-based design environment where you can analyze data, create interactive reports, dashboard reports, and build integrated dashboards to share business intelligence solutions with others in your organization and on the internet. In addition to its design features, the User Console offers a wide variety of system administration features for configuring the Pentaho Server, managing Pentaho licenses, setting up security, managing report scheduling, and tailoring system performance to meet your requirements.
If you have installed the trial download on your laptop or desktop machine, you are ready to get started exploring. If you have the software installed on a server, and want to use your machine to point to it, see [Develop your BA environment](#develop-your-ba-environment) for details.
| Lesson | Do This | Notes |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Tour the User Console | [Quick tour of the Pentaho User Console](/pdia-try-pdia/archive-flattened-pages/quick-tour-of-the-pentaho-user-console-puc) |
Understand the features of the User Console
View the sample reports on the Samples tab of the Getting Started section
|
| Create Your First Reports and Dashboards | [Getting Started with Analyzer, Interactive Reports, and Dashboard Designer](/pdia-try-pdia/getting-started-with-analyzer-interactive-reports-and-dashboard-designer) |
Created and saved an Interactive Report
Created and saved an Analysis Report
Created and saved a custom Dashboard
|
| Schedule Your Report |
Learn about scheduling reports. See the Pentaho Business Analytics document for details.
|
Scheduled a report to run and email automatically.
Received your report through email after the schedule runs.
|
**Explore and learn data source basics**
If you have already worked with the Steel Wheels sample data and want to learn how to create your own data sources and data models with Pentaho, use the Data Source Wizard. The Data Source Wizard helps you define a data source that contains the data you want to use and guides you through the creation of your evaluation data model for use in creating reports.
After you define a data source, you can make it available to other evaluators so they can create reports and analysis by simply picking the data source from the data source list. Any number of reports can be created using a single data source.
| Lesson | Do This | Notes |
| ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Create Your First Data Source |
Create a Data Source
Tour the Data Source Wizard
See Pentaho Business Analytics for instructions.
|
Understand how the Pentaho Server and Data Source Wizard work together to create usable data sources and data models.
Explore the Data Source Wizard interface.
Learn the basics of creating a data source using the Data Source Wizard.
|
| Choose Data Source Types |
Choose a data source type
See Pentaho Business Analytics for instructions.
|
Learn about the different data source types supported by the Data Source Wizard.
We recommend using a CSV data source for evaluation.
|
| Create Your First CSV Data Source |
Create a CSV data source
See Pentaho Business Analytics for instructions.
|
Import a CSV data file using the Data Source Wizard.
Create the CSV data source.
We recommend creating a report using this new CSV data source, then refining the data model with the Data Source Model Editor as needed.
|
| Refine Your Data Source Model |
Edit multidimensional data source models.
See Pentaho Business Analytics for instructions.
| (Optional) Edit your evaluation data source model using the Data Source Model Editor. |
| Inline Model Editing |
Read Working with Analyzer measures in the Pentaho Business Analytics document.
|
Learn how to edit your data models while working in Analyzer.
|
| Learn about Streamlined Data Refinery |
Learn how to work with Streamlined Data Refinery
See Pentaho Data Integration for instructions.
|
Learn how Streamlined Data Refinery works.
|
**Learn about Report Designer**
Pentaho Report Designer is a report creation tool that you can use by itself, or as part of the Pentaho Suite. It allows professionals to create print-quality reports based on data from virtually any type of data source.
These resources in the **Pentaho Report Designer** document will help you get familiar with the Report Designer interface, and guide you through the creation and publishing of a print-quality report.
| Lesson | Section in document | Notes |
| ------------------------------------- | ------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| Explore the Report Designer Interface |
Explore Report Designer
|
Tour the Report Designer interface before you begin building reports.
|
| Report Designer Workflow Overview |
Learn about Report Designer workflow
|
Look over the workflow concepts for Report Designer.
|
| Create Your First Report |
Create your first print-quality report
|
Create a report.
Add a chart and parameters to your report.
View and then publish your report.
|
| Refine the Look of Your Report |
Design print-quality reports
|
Explore more advanced features of Report Designer, beginning with report elements.
|
| Add a PDI Data Source |
Add a PDI data source
|
Add a PDI data source and use it to create a report in Report Designer.
|
**Discover more about Pentaho Business Analytics**
* The Pentaho Analyzer, Interactive Reports, and Dashboard Designer plugins provide in-depth details about creating eye-catching business intelligence deliverables for your user community. See the **Pentaho Business Analytics** document for details.
* If you are a system administrator, check out the **Install Pentaho Data Integration and Analytics** document. Both have details on configuring and administering your Pentaho Server using the User Console, as well as a section on the variety of things you can do to maintain your server manually.
**Next steps (evaluation)**
* Contact [Pentaho](https://www.pentaho.com/helpmeout) to learn more about how Business Analytics can be tailored to meet your business needs.
* Continue with [Develop your BA environment](#develop-your-ba-environment).
#### Develop your BA environment
This workflow outlines how to set up a Pentaho Server for BA development. It also covers how to build, refine, and test BA content.
This workflow is similar to the Trial Download Evaluation experience. The difference is you configure the server fully. You also work with your own report designers and data scientists. You can also engage Pentaho [professional services](https://www.hitachivantara.com/en-us/services/big-data-analytics-services.html).
Before you start, consider Pentaho [training classes](https://www.hitachivantara.com/en-us/services/training-certification.html). Training helps you install and configure the server. Training also helps you build data models and BA applications.
**Set up your Pentaho Server**
Use this checklist to verify requirements. Then install and configure the Pentaho Server and BA design tools.
**Verify system requirements**
* Review required components in [Components Reference](/pdia-try-pdia/components-reference).
* Review required drivers in [JDBC drivers reference](/pdia-try-pdia/jdbc-drivers-reference).
* Acquire one or more servers that meet requirements.
* Obtain the correct drivers for your system.
**Obtain software and install the Pentaho Server**
* Download the Pentaho software from your Sales Support representative.
* Install the software using [Install the 30-day trial of Pentaho Data Integration and Analytics](/pdia-try-pdia/archive-flattened-pages/pentaho-evaluation).
* Sign in using [Quick tour of the Pentaho User Console](/pdia-try-pdia/archive-flattened-pages/quick-tour-of-the-pentaho-user-console-puc).
* Tour **Administration**.
* Change the default administrator password.
**Change the Pentaho Server fully qualified URL**
* Follow **Administer Pentaho Data Integration and Analytics** instructions to change the server URL.
* If multiple machines point to one server, confirm all clients use the new URL.
**Configure the Pentaho Server**
* Manage licenses. See **Administer Pentaho Data Integration and Analytics**.
* Configure server data connections. See **Install Pentaho Data Integration and Analytics**.
* Configure email for scheduled reports. See **Pentaho Business Analytics**.
* Review schedule management. See **Pentaho Business Analytics**.
**Configure BA design tools**
Do this only on a development system. Do not configure design tools on your production server.
* Configure design tools and utilities. See **Install Pentaho Data Integration and Analytics**.
* Configure each tool’s data connections. See **Install Pentaho Data Integration and Analytics**.
**Import data sources and data models**
Create data sources and models that support agile BA development.
**Choose data source types**
* Choose a data source type. See **Pentaho Business Analytics**.
* Review relational versus multidimensional models.
**Create data sources and models**
* Tour the Data Source Wizard. See **Pentaho Business Analytics**.
* Learn how the server and wizard produce usable sources and models.
**Create database table data sources**
* Create a database table source. See **Pentaho Business Analytics**.
* Create initial data sources and preliminary models.
**Learn about Mondrian schemas**
* Create and modify Mondrian schemas. See **Pentaho Schema Workbench**.
* Add a Mondrian data source.
* Adapt the schema for Analyzer.
* Refine the schema in Schema Workbench.
**Create reports and further refine data models**
Work with data scientists and business analysts at this stage. This improves the quality of models and reports.
As you prepare to move to production, use data sources from:
* Pentaho Schema Workbench
* Pentaho Metadata Editor
**Create Analyzer reports, Interactive reports, and dashboards**
* Follow **Pentaho Business Analytics** instructions.
* Create Interactive and Analyzer reports.
* Create a dashboard.
* Verify results match what you need.
* If needed, refine models with your data team.
**Create a report with Report Designer (optional)**
* Follow **Pentaho Report Designer** instructions.
**Refine your data source model**
* Edit multidimensional models. See **Pentaho Business Analytics**.
* Refine Mondrian schemas. See **Pentaho Schema Workbench**.
* Refine relational models. See **Pentaho Metadata Editor**.
* Recreate reports to validate changes.
* Repeat until results meet requirements.
**Test environment quality**
If you do quality assurance testing, upload content to the Pentaho Repository. Then download it to the QA server. See **Administer Pentaho Data Integration and Analytics** for details.
Some organizations also run user acceptance testing after QA.
**Next steps (development)**
* Investigate security. See **Administer Pentaho Data Integration and Analytics**.
* Plan scheduling for production. See **Pentaho Business Analytics**.
* Decide what content to promote to production. See **Administer Pentaho Data Integration and Analytics**.
* Check the [Support Portal](https://support.pentaho.com/hc/en-us) for service packs.
* Prepare to [Go live for production - BA](#go-live-for-production---ba).
#### Go live for production - BA
This section explains how to move Pentaho content and server settings between servers.
This process usually uses two or three servers with identical configurations:
* BA content development
* Testing and QA (optional)
* Production
We recommend working with Pentaho [professional services](https://www.pentaho.com/services) during production deployment.
**Prepare for going live**
This section has two parts:
* A checklist for setting up a Pentaho Server
* Prerequisites to complete before you go live
If your production server is already set up, start with the prerequisites.
**Pentaho Server setup checklist**
| Task | Do this | Notes |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Verify system requirements |
- Acquire one or more servers that meet requirements. - Obtain the correct drivers.
|
| Obtain software and install the Pentaho Server |
- Install Pentaho Suite. See Install Pentaho Data Integration and Analytics. - Download and install the latest service pack. See Administer Pentaho Data Integration and Analytics. - Access the User Console. See Pentaho Business Analytics.
|
- Install the software. - Install the latest service pack. - Access the User Console, review Administration, and change the default administrator password.
If needed, change the fully qualified URL for the Pentaho Server.
|
| Change the server fully qualified URL | Change the Pentaho Server fully qualified URL if needed. See **Administer Pentaho Data Integration and Analytics**. | If many machines point to one server, change the URL and verify connectivity. |
| Configure the server |
- Manage licenses. See Administer Pentaho Data Integration and Analytics. - Specify data connections. See Install Pentaho Data Integration and Analytics. - Set up email for scheduled reports. See Pentaho Business Analytics.
|
- Set up data connections. - Configure email through Administration.
|
**Prerequisites before you go live**
| Task | Do this | Notes |
| --------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| Compare configuration files |
- Compare server configuration files. - Verify and increase memory settings. See Administer Pentaho Data Integration and Analytics.
|
- Identify configuration differences. - Commit a unified properties file to version control. - Increase memory settings as needed.
|
| Verify data sources |
- Specify data connections. See Install Pentaho Data Integration and Analytics. - Define JNDI connections. See Install Pentaho Data Integration and Analytics.
|
- Confirm data sources can be promoted. - Establish JNDI sources as replacements if needed.
|
| Define security |
- Define Pentaho Server security. See Administer Pentaho Data Integration and Analytics. - Manage users and roles. See Pentaho Business Analytics. - Implement advanced security. See Administer Pentaho Data Integration and Analytics.
|
- Implement security. - Define users, roles, and permissions.
|
| Upload content | Upload and download from the Pentaho Repository. See **Administer Pentaho Data Integration and Analytics**. | - Upload files and folders. |
**Compare configuration files**
The most important server configuration settings are stored in the `/server/pentaho-server/pentaho-solutions/system/` directory.
Some core settings are also inside the Pentaho WAR archive deployed to your application server. These settings should not change after initial setup.
{% hint style="warning" %}
Do not change the names of content files, data sources, solution directories, or other file names during promotion.
Set names during solution development. Keep names consistent through promotion.
Renaming can cause issues that you will not detect immediately. This can break QA and production content.
{% endhint %}
To ensure you selected all server configuration files, compare these directories in full:
* `/pentaho-solutions/system/`
* `/WEB-INF/` inside your deployed `pentaho.war`
* `/META-INF/` inside your deployed `pentaho.war`
{% hint style="info" %}
Plugin directories for Analyzer, Dashboard Designer, Interactive Reports, and Community Dashboard Framework include binaries.
Binary differences usually indicate version differences. Focus on XML and properties files.
If you customized plugins, promote those changes too.
{% endhint %}
**Move content to production server**
This checklist summarizes best practices to promote Pentaho Server settings, data sources, and content.
Before you promote from development to production, complete the preparation and prerequisite tasks earlier in this page.
| Task | Do this | Notes |
| ----------------------------------- | ----------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| Download content | - Upload and download from the Pentaho Repository. |
- Move all desired content to production. - See Administer Pentaho Data Integration and Analytics for details.
|
| Set up schedules and blockout times |
- Manage schedules. - Prevent scheduling by setting blockout times.
|
- Set up production schedules. - Set up blockout times for maintenance. - See Pentaho Business Analytics for details.
|
**Next steps (production)**
These resources are helpful after your production server is live:
* See **Administer Pentaho Data Integration and Analytics** for guidance on maintenance and tuning.
* Pentaho [Training and Education](https://www.hitachivantara.com/en-us/services/training-certification.html)
* [Support Portal](https://support.pentaho.com/hc/en-us) for service packs
# Components reference
Pentaho aims to accommodate diverse computing environments. This list provides details about the environment components and versions we support. Where applicable, versions are listed as certified or supported:
* **Certified**
The version has been tested and validated for compatibility with Pentaho.
* **Supported**
Support is available for listed non-certified versions.
If you have questions about your particular computing environment, contact [Pentaho Support](https://support.pentaho.com/).
### Server operating system
The Pentaho Server is hardware-independent and runs on server-class computers.
Your server-class computer must comply with the specifications for minimum hardware and required operating systems:
| Hardware | Certified Operating System - 64 bit | Supported Operating System - 64 bit |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
Processor
Intel EM64T or AMD64 Dual-Core or later
RAM
8 GB with 4 GB dedicated to Pentaho servers
Disk Space
20 GB free after installation
|
Microsoft Windows 2025 Server
Red Hat Enterprise 9\*
Ubuntu Server 22.04 LTS
|
Microsoft Windows 2022 Server
Red Hat Enterprise 8\*
|
\*Pentaho Data Integration and Analytics is supported on any Linux distribution binary-compatible with RHEL 9 and Ubuntu Server 22, including in virtualized and cloud environments. If you have any questions, contact [Pentaho Support](https://support.pentaho.com/).
{% hint style="info" %}
**Note:** Mac servers are not supported as an operating system.
{% endhint %}
### Container deployment
Supported technology for deploying Pentaho in containers.
Technology
Certified
Docker
27.5.1
{% hint style="info" %}
**Note:** Kubernetes environments that use this Docker version are also supported.
{% endhint %}
You can also deploy pre-configured Docker images of specific Pentaho products in your AWS environments. See [Docker container deployment of Pentaho Server](/install/pentaho-installation-overview-cp/docker-container-deployment-of-pentaho-server) and [Docker container deployment of Carte, Pan, and Kitchen](/install/pentaho-installation-overview-cp/docker-container-deployment-of-carte-pan-and-kitchen) for details.
### Workstation operating system
These Pentaho design tools are hardware-independent and run on client-class computers that comply with these specifications for minimum hardware and required operation systems.
* Pentaho Aggregation Designer
* Pentaho Data Integration
* Pentaho Metadata Editor
* Pentaho Report Designer
* Pentaho Schema Workbench
| Hardware - 64 bit | Certified Operating System - 64 bit | Supported Operating System - 64 bit |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | ----------------------------------- |
|
Processors
Apple Macintosh Dual-Core
Apple Mac M1, M2, and M3 chipset
Intel EM64T or AMD64 Dual-Core or later
RAM
2 GB RAM for most of the design tools, PDI requires 2 GB dedicated
Disk Space
2 GB free after installation
Minimum Screen Size
1280 x 960 pixels
|
Ubuntu Desktop 22.04
Microsoft Windows 11
macOS 15 (Sequioa)
|
macOS 14 (Sonoma)
|
{% hint style="info" %}
**Note:** Ubuntu Linux requires \`libwebkitgtk-1.0\`. See **Install Pentaho Data Integration and Analytics** for more information.
{% endhint %}
### Embedded software
When embedding Pentaho software into other applications, the computing environment should comply with these specifications for minimum hardware and required operation systems.
* Embedded Pentaho Reporting
* Embedded Pentaho Analysis
* Embedded Pentaho Data Integration
{% hint style="info" %}
**Note:** Pentaho Data Integration and Analytics is officially certified to run on the Red Hat Enterprise and Ubuntu Linux distributions. It is compatible with any binary-compatible Linux distribution that meets the necessary software and hardware requirements, including in virtualized and cloud environments. If you have any questions, contact [Pentaho Support](https://support.pentaho.com/). The following specifications comply with minimum hardware and required operating systems for embedding Pentaho reporting, analysis, and data integration:
{% endhint %}
| Hardware—64 bit | Certified Operating System—64 bit |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
Processors
Intel EM64T or AMD64 Dual-Core
RAM
8 GB with 4 GB dedicated to Pentaho servers
Disk Space
20 GB free after installation
|
Microsoft Windows 2022 Server
Red Hat Enterprise 9
Ubuntu Server 22.04 LTS
|
### Application servers
The server to which you deploy Pentaho software must run the following application server:
* Tomcat 10.1.48 (Certified)
### Solution database repositories
Pentaho software stores processing artifacts in these database repositories:
Repository
Certified
Supported
PostgreSQL
16
15
MySQL
8.4
Oracle
23c & 23ai
MS SQL Server
2022
2019 (including patched versions)
Maria DB
11.4
\* The default installed solution database.
### Apache Hadoop vendors
Pentaho software has certified or supported data sources from these Hadoop Vendors.
| Vendor | Driver Version |
| ------------------------------------------ | -------------- |
| Amazon EMR | 7.7.0 |
| Cloudera Data Platform (CDP) Private Cloud | 7.1.x, 7.3.1 |
### Data Sources: Pentaho Tools
This table summarizes which data sources are compatible with the main Pentaho tools.
Pentaho Software
Data Source
Pentaho Reporting
JDBC 3/41
ODBC
OLAP4J
XML
Pentaho Analysis
Pentaho Data Integration
Pentaho Metadata
Scriptable
Snowflake
Pentaho Server, Action Sequences
Relational (JDBC)
Hibernate
Javascript
Metadata (MQL)
Mondrian (MDX)
XML (XQuery)
Security User/Role List Provider
Snowflake
Data Integration Steps (PDI)
Other Action Sequences
Web Services
XMLA
Pentaho Data Integration
JDBC 3/41
OLAP4J
Salesforce
Snowflake
XML
CSV
Microsoft Excel
1 Use a JDBC 3.x or 4.x compliant driver that is compatible with SQL-92 standards when communicating with relational data sources. For a list of drivers to use with relational JDBC databases, see the [JDBC drivers reference](/pdia-try-pdia/jdbc-drivers-reference).
### Big Data Sources: General
Pentaho software supports the following Big Data sources. Check this list if you are evaluating Pentaho or checking for general compatibility with a specific vendor.
| Data Source | Certified | Supported |
| -------------------------------------------------------- | --------- | --------- |
| Amazon EMR (via Hive) | 7.7.0 | |
| Cassandra (Datastax) | 6.8 | |
| Cloudera Data Platform (CDP) on premises (private cloud) | 7.1.9 | |
| Google BigQuery (Simba) | 1.6.2 | 1.2.25 |
| MongoDB | 7.0 | |
| Vertica\* | 24 | |
\* Deprecated beginning in version 11.0.
### Big Data Sources: Details
This table shows the Big Data sources that are compatible with specific Pentaho tools.
1 EMR clusters (version 7.x and later) built with JDK 17 exclude the `commons-lang-2.6.jar` library from their standard Hadoop library directories (`$HADOOP_HOME/lib`). To use the EMR driver for EMR 7.x, obtain the `commons-lang-2.6.jar` file from a trusted source, such as the official Maven repository ([Maven Repository: commons-lang » commons-lang » 2.6](https://mvnrepository.com/artifact/commons-lang/commons-lang/2.6)). Then manually copy the downloaded JAR file to the `$HADOOP_HOME/lib` or `$HADOOP_MAPRED_HOME/lib` directory on each node within the EMR cluster to ensure that all worker nodes have access to the library.
2 Hive3 as a data source for CDP also supports Hive LLAP, and Hive3 on Tez.
3 The Simba driver required for Google BigQuery is the JDBC 4.2-compatible version, which you can download from .
4 Deprecated beginning in version 11.0.
{% hint style="info" %}
**Note:** A generic Apache Hadoop driver is included in the Pentaho distribution for version 11.0. Other supported drivers can be downloaded from the [Support Portal](https://support.pentaho.com/hc/en-us).
{% endhint %}
### SQL Dialect-Specific
Pentaho software generates dialect-specific SQL when communicating with these data sources. Certified indicates the SQL dialect has been tested for compatibility with Pentaho.
| Pentaho Software | Data Source |
| ------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Pentaho Analyzer |
Certified
Amazon Redshift
Azure SQL
Impala
MySQL
Microsoft SQL Server
Oracle
PostgreSQL
Snowflake
Supported
Access
Firebird
Hsqldb
IBM DB2
IBM MQ 9.2
Informix
Ingres1
Interbase1
Neoview1
SqlStream
Sybase1
Vectorwise1
Vertica1
Other SQL-89 compliant2
|
| Pentaho Metadata |
Certified
Azure SQL
Hive 2
Impala
MySQL
PostgreSQL
Supported
Amazon Redshift
ASSQL
Firebird
H2
Hypersonic
IBM DB2
IBM MQ 9.2
Ingres1
Interbase1
MS Access
MS SQL Server (JTDS Driver)
MS SQL Server (Microsoft Driver)
Snowflake
Sybase1
Vertica1
Other SQL-92 compliant2
|
| Pentaho Data Integration |
Certified
Amazon Redshift
Azure SQL
Hive1
Hive 2
Impala
MS SQL Server (JTDS Driver)
MS SQL Server (Microsoft Driver)
MySQL
Oracle
PostgreSQL
Snowflake
Vertica1
Supported
AS/400
InfiniDB1
Exasol 4
Firebird SQL
H2
Hypersonic
IBM DB2
IBM MQ 9.2
Informix
Ingres1
Ingres VectorWise1
MaxDB (SAP DB)
Neoview1
Oracle RDB
SQLite
UniVerse database
Other SQL-92 compliant2
|
1 Deprecated beginning in version 11.0.
2 If your data source is not in this list and is compatible with SQL-92, Pentaho software uses a generic SQL dialect.
### Security
Pentaho software integrates with these third-party security authentication systems:
* CAS 7 (Certified)
* Integrated Windows Authentication with Internet Information Services 10 (Certified)
* Spring 6.2.12 (Certified)
### Java virtual machine
Pentaho software requirements for Java Runtime Environment (JRE).
| Pentaho Software | Certified | Supported |
| -------------------- | ---------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| All Pentaho software |
Oracle Java 21
Oracle OpenJDK 21
|
Oracle Java 17
Oracle OpenJDK 17
Adoptium) Temurin JDK 17
(Adoptium) Temurin JDK 21
Azul Zulu JDK 17
Azul Zulu JDK 21
|
{% hint style="info" %}
**Note:** The PDI client requires at least Java 11.x to run on Windows 11.
{% endhint %}
### Web browsers
Pentaho supports major versions of web browsers that are publicly available six weeks before the finalization of a Pentaho release.
| Certified Browsers | Supported Browsers |
| ------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
Apple Safari 26.1 (On macOS only)
Google Chrome 142
Microsoft Edge 142
Mozilla Firefox 144
|
Apple Safari 26.1 and later (On macOS only)
Google Chrome 142 and later
Microsoft Edge 142 and later
Mozilla Firefox 144 and later
|
### Support Statement for Analyzer on Impala
These are the minimum requirements for Analyzer to work with Impala:
* Pentaho 7.1 or later
* Impala 1.3.x or later
* Recommend using Parquet compressed file format for tables in Impala
* Make sure that the JDBC driver is dropped into the Pentaho Server and Schema Workbench directories. See the **Install Pentaho Data Integration and Analytics** document for details.
* Turn off connection pooling in Pentaho Server.
* In Mondrian schemas, divide dimension tables with high cardinality into several levels
**Note:** As with any data source, the performance of Pentaho Analyzer on Impala will be dependent upon the data shape, Impala’s configuration, and the types of queries. See the best practice, "Pentaho Analyzer with Impala as a Data Source" located at: [https://support.pentaho.com/hc/en-us/articles/208652846](https://support.pentaho.com/hc/en-us/articles/360002913871-Big-Data-and-Pentaho) or download the [PDF](https://support.pentaho.com/hc/en-us/article_attachments/360012191711/Pentaho_Analyzer_and_Impala_Data_Source_Settings_and_Recommendations.pdf).
There are some compiled Mondrian automated test suite results for Analyzer on Impala with OEM Simba, as well as the community Apache Hive driver:
* [Analyzer on Impala with OEM Simba](http://wiki.pentaho.com/display/analysis/PA_CR_PA-3.7.0.0-752_impalad-1.3.0_simba-2.5.2)
* [Analyzer on Impala with community Apache Hive driver](http://wiki.pentaho.com/display/analysis/PA_CR_PA-3.7.0.0-752_impalad-1.3.0)
### Google BigQuery
You can use Google BigQuery as a data source with the Pentaho User Console or with the PDI client.
Before you begin, you must have a Google account and must create service account credentials in the form of a key file in JSON format to connect to Google BigQuery. To create service account credentials, see the [Google Cloud Storage Authentication documentation](https://cloud.google.com/storage/docs/authentication).
Additionally, you must set permissions for your BigQuery and Google Cloud accounts. To configure your service account authentication, see the [Google Service Account documentation](https://www.simba.com/products/BigQuery/doc/v1/JDBC_InstallGuide/content/jdbc/bq/authenticating/serviceaccount.htm).
Perform the following steps to create a JDBC connection to a Google BigQuery data source from the User Console or PDI client.
1. Stop the Pentaho Server.
2. Download the ZIP file containing the Simba version 1.5.4.1008 JDBC 4.2 driver for Google BigQuery from .
3. Navigate to the `server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib` directory for the User Console or the `design-tools/data-integration/lib` directory for the PDI client and delete any files associated with previous versions of Google BigQuery.
Visually verify each file to ensure the older version is deleted.
4. Extract the following files to the `server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib` folder for the User Console or the `design-tools/data-integration/lib` directory for the PDI client.
* `animal-sniffer-annotations-1.14.jar`
* `api-common-1.7.0.jar`
* `avro-1.9.0.jar`
* `checker-compat-qual-2.5.2.jar`
* `error_prone_annotations-2.1.3.jar`
* `gax-1.42.0.jar`
* `gax-grpc-1.42.0.jar`
* `google-api-client-1.28.0.jar`
* `google-api-services-bigquery-v2-rev426-1.25.0.jar`
* `google-auth-library-credentials-0.15.0.jar`
* `google-auth-library-oauth2-http-0.13.0.jar`
* `GoogleBigQueryJDBC42.jar`
* `google-cloud-bigquerystorage-0.85.0-alpha.jar`
* `google-cloud-core-1.67.0.jar`
* `google-cloud-core-grpc-1.67.0.jar`
* `google-http-client-1.29.0.jar`
* `google-http-client-apache-2.0.0.jar`
* `google-http-client-jackson2-1.28.0.jar`
* `google-oauth-client-1.28.0.jar`
* `grpc-alts-1.18.0.jar`
* `grpc-auth-1.18.0.jar`
* `grpc-context-1.18.0.jar`
* `grpc-core-1.18.0.jar`
* `grpc-google-cloud-bigquerystorage-v1beta1-0.50.0.jar`
* `grpc-grpclb-1.18.0.jar`
* `grpc-netty-shaded-1.18.0.jar`
* `grpc-protobuf-1.18.0.jar`
* `grpc-protobuf-lite-1.18.0.jar`
* `grpc-stub-1.18.0.jar`
* `gson-2.7.jar`
* `j2objc-annotations-1.1.jar`
* `javax.annotation-api-1.3.2.jar`
* `jsr305-3.0.2.jar`
* `opencensus-api-0.18.0.jar`
* `opencensus-contrib-grpc-metrics-0.18.0.jar`
* `opencensus-contrib-http-util-0.18.0.jar`
* `protobuf-java-3.7.0.jar`
* `protobuf-java-util-3.7.0.jar`
* `proto-google-cloud-bigquerystorage-v1beta1-0.50.0.jar`
* `proto-google-common-protos-1.15.0.jar`
* `proto-google-iam-v1-0.12.0.jar`
* `threetenbp-1.3.3.jar`**Note:** The Google BigQuery connection name does not display in the User Console Database Connection dialog box until you copy these files.
5. Restart the Pentaho Server.
6. Log on to the User Console or the PDI client, then open the Database Connection dialog box.
See the **Install Pentaho Data Integration and Analytics** document for more information on the Database Connection dialog box.
7. In the Database Connection dialog box, select **General**, then select **Google BigQuery** as the **Database Type**.
8. In the **Settings** area, enter the information for your Google BigQuery account.
* The **Host Name** is the URL to Google's BigQuery web services API. For example,
* The **Project ID** in the PDI client and the **Database name** in the User Console are identical.
* The **Port Number** is `443`.
9. Click **Options**, then add the following parameters and values.
| Parameter | Value |
| ------------------------- | ---------------------------------------------------------------------------------------------------------------------- |
| **OAuthType** | `0` (Zero) |
| **OAuthServiceAcctEmail** | Specify your service account email address. |
| **OAuthPvtKeyPath** | Specify the path to your private key credential file. |
| **Timeout** | Specify the amount of time, in seconds, before the server closes the connection. The recommended value is 120 seconds. |
10. Click **Test** to verify that you can connect to your data.
# JDBC drivers reference
This reference can change.\
Report issues using the [Support Portal](https://support.pentaho.com).
## JDBC driver download links
| Database | Vendor | URL |
| ----------------------------------------- | ----------------------------------- | -------------------------------------------------------------------------------- |
| Amazon Redshift | Amazon | |
| AWS Aurora | Amazon | Aurora is MySQL- and PostgreSQL-compatible. Use the MySQL or PostgreSQL driver. |
| AWS Athena | Amazon Web Services | |
| Caché1 | InterSystems | |
| CUBRID | CUBRID | |
| Daffodil DB | Daffodil Software | |
| DB2 AS/400 | IBM | |
| DB2 Universal Database | IBM | |
| Firebird | Firebird Foundation | |
| FrontBase | FrontBase | |
| Google BigQuery | Google BigQuery | |
| Greenplum | EMC2 | |
| H2 Database | H2 | [http://www.h2database.com](http://www.h2database.com/) |
| Hive1 | Apache | |
| Hive2 | Apache | |
| HSQLDB | HyperSQL | |
| Impala | Cloudera | |
| Informix | IBM | |
| Ingres1 | Actian | |
| InterBase1 | Embarcadero | [http://edn.embarcadero.com](http://edn.embarcadero.com/) |
| jTDS Free MS SQL Sybase1 | jTDS | |
| MariaDB | MariaDB | |
| MaxDB | SAP | [http://maxdb.sap.com](http://maxdb.sap.com/) |
| Mckoi SQL Database | Mckoi SQL Database | |
| Mimer | Mimer Information Technology | [http://www.mimer.com](http://www.mimer.com/) |
| MySQL | Oracle | |
| Neoview1 | HP | Contact your local HP representative for information on this product. |
| Netezza | IBM | [http://www.netezza.com](http://www.netezza.com/) |
| OpenBase SQL | OpenBase International | |
| Oracle | Oracle | |
| Pervasive | Pervasive | |
| PostgreSQL | PostgreSQL Global Development Group | |
| SAP ASE (formerly Sybase ASE)1 | SAP | |
| SAP DB | SAP MaxDB | |
| SAP HANA | SAP | |
| SAP SQL Anywhere | SAP | |
| SmallSQL | SmallSQL | |
| Snowflake | Snowflake | |
| SQLite | Xerial | |
| SQL Server | Microsoft | |
| Teradata | Teradata | |
| Vertica1 | HP | [http://www.vertica.com](http://www.vertica.com/) |
1 Deprecated beginning in version 11.0.
## Driver details by database
### Amazon Redshift
| Vendor Name | Details |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| Amazon Web Services |
| |
### Hive
{% hint style="info" %}
**Note:** Deprecated beginning in version 11.0.
{% endhint %}
| Vendor Name | Details |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| Apache |
Company URL:
|
|
Driver URL:
or
| |
|
JDBC URL Syntax by Type:
Server
jdbc:hive://\\[:\]/default
|
Default Port:
10000
|
|
JDBC Class:
org.apache.hadoop.hive.jdbc. HiveDriver
|
JDBC JAR File Name:
pentaho-hadoop-shims-common-fragment-Vx-x.x.x.jar
|
|
Shipped with Pentaho products:
Pentaho Server
Pentaho Data Integration
Pentaho Metadata Editor
Pentaho Report Designer
| |
|
Comments:
The pentaho-hadoop-shims-common-fragment-Vx-x.x.x.jar library includes a proxy driver. The actual Hive JDBC implementation for the specific distribution and version of Hadoop is located in the Pentaho driver for that distro. See the Install Pentaho Data Integration and Analytics document for more information.
Hive does not support the full SQL capabilities. It uses a subset and is more accurately referred to as HiveQL.
The pentaho-hadoop-shims-common-fragment-Vx-x.x.x.jar library includes a proxy driver. The actual Hive JDBC implementation for the specific distribution and version of Hadoop is located in the Pentaho driver for that distro. See the Install Pentaho Data Integration and Analytics document for more information.
Hive2 does not support the full SQL capabilities. It uses a subset and is more accurately referred to as HiveQL.
The pentaho-hadoop-shims-common-fragment-Vx-x.x.x.jar library isncludesa proxy driver. The actual Hive JDBC implementation for the specific distribution and version of Hadoop is located in the Pentaho driver for that distro. See the Install Pentaho Data Integration and Analytics document for more information.
Hive does not support the full SQL capabilities. It uses a subset and is more accurately referred to as HiveQL.
| |
### PostgreSQL
| Vendor Name | Details |
| -------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| PostgreSQL Global Development Group |
Company URL:
|
|
Driver URL:
| |
|
JDBC URL Syntax by Type:
Server
jdbc:postgresql://\\[:\]/\
|
Default Port:
5342
|
|
JDBC Class:
org.postgresql.Driver
|
JDBC JAR File Name:
postgresql-9.x-xxx.jdbc4.2.jar
|
|
Shipped with Pentaho products:
Pentaho Data Integration
Pentaho Report Designer
| |
### SAP ASE (formerly Sybase ASE)
{% hint style="info" %}
**Note:** Deprecated beginning in version 11.0.
{% endhint %}
| Vendor Name | Details |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| SAP |
Company URL:
|
|
Driver URL:
The jConnect JDBC driver can only be installed from the SAP Adaptive Server Enterprise Installer. See the SAP website for more information.
| |
|
JDBC URL Syntax by Type:
Server
jdbc:sybase:Tds:\\[:\]/\
|
Default Port:
5000
|
|
JDBC Class:
com.sybase.jdbc4.jdbc.SybDriver
|
JDBC JAR File Name:
N/A
|
|
Comments:
The open source jTDS driver works with SAP ASE (formerly Sybase) as well. Note that although you can use jTDS open source JDBC driver, we recommend that you use the SAP-supplied JDBC driver instead. Connections might not work reliably if you use the jTDS JDBC driver.
| |
### SAP DB
| Vendor Name | Details |
| ------------------------------------------------------------------------------------------------------------------------------------ | -------------------------------------------------------------- |
| Recommended Native Driver | |
| SAP DB |
Company URL:
N/A
|
|
Driver URL:
| |
|
JDBC URL Syntax by Type:
Server
jdbc:sapdb://\/\
|
Default Port:
N/A
|
|
JDBC Class:
com.sap.dbtech.jdbc.DriverSapDB
|
JDBC JAR File Name:
sapdbc-x.x.x.jar
|
|
Shipped with Pentaho products:
Pentaho Data Integration
| |
|
Comments:
FREE Enterprise Open Source Database.
| |
### SAP HANA
| Vendor Name | Details |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| SAP HANA |
Company URL:
|
|
Driver URL:
For SAP customers, the driver is part of your client tools. Contact your SAP representative for more information.
| |
|
JDBC URL Syntax by Type:
Server
jdbc:sap\://\:\/?autocommit=false
|
Default Port:
30015
|
|
JDBC Class:
com.sap.db.jdbc.Driver
|
JDBC JAR File Name:
ngdbc.jar
|
|
Comments:
Note that the default port number is '30015' where '00' is the instance of the machine you are connecting to. For example, you can connect to the same machine using '30015', '30115', or '31015'.
| |
### SAP SQL Anywhere
| Vendor Name | Details |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| SAP |
Company URL:
|
|
Driver URL:
The jConnect JDBC driver can only be installed from the SAP Adaptive Server Enterprise Installer. Visit the SAP website for more information.
| |
|
JDBC URL Syntax by Type:
Server
jdbc:sybase:Tds:\\[:\]/\
|
Default Port:
2638
|
|
JDBC Class:
com.sybase.jdbc4.jdbc.SybDriver
|
JDBC JAR File Name:
N/A
|
|
Comments:
This open source jTDS driver works with SAP ASE (formerly Sybase) as well. Note that although you can use jTDS open source JDBC driver, we recommend that you use the SAP-supplied JDBC driver instead. Connections might not work reliably if you use the jTDS JDBC driver.
| |
### SQL Server
| Vendor Name | Details |
| --------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| Microsoft |
Company URL:
|
|
Driver URL:
| |
|
JDBC URL Syntax by Type:
Server
jdbc:sqlserver://\\[:\];DatabaseName= \
|
Default Port:
1433
|
|
JDBC Class:
com.microsoft.sqlserver.jdbc. SQLServerDriver
|
JDBC JAR File Name:
sqljdbc4.jar
|
|
Comments:
The open source jtds driver also works with MSSQL.
| |
### Teradata
| Vendor Name | Details |
| ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------- |
| Please see the [Teradata JDBC Driver Reference](http://developer.teradata.com/connectivity/reference/jdbc-driver) for information about required drivers. | |
| Teradata |
Company URL:
|
|
Driver URL:
| |
|
JDBC URL Syntax by Type:
Server
jdbc:teradata://\\[/\\[;\]…]
|
Default Port:
N/A
|
|
JDBC Class:
com.teradata.jdbc.TeraDriver
|
JDBC JAR File Name:
terajdbc4.jar
tdgssconfig.jar
|
### Vertica
{% hint style="info" %}
**Note:** Deprecated beginning in version 11.0.
{% endhint %}
| Vendor Name | Details |
| ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------- |
| Recommended Native Driver | |
| HP |
Company URL:
|
|
Driver URL:
| |
|
JDBC URL Syntax by Type:
Server
jdbc:vertica://\\[:\]/\
|
Default Port:
5433
|
|
JDBC Class:
com.vertica.Driver
|
JDBC JAR File Name:
N/A
|
## Install drivers with the JDBC distribution tool
To connect to a database, including the Pentaho Repository database, download and install the JDBC driver:
* In the appropriate Pentaho component directories.
* In the web application server that hosts Pentaho Server.
{% hint style="info" %}
Due to licensing restrictions, Pentaho cannot redistribute some third-party database drivers.\
You must download and install those drivers yourself.
{% endhint %}
1. Download the JDBC driver JAR from your database vendor or a third-party driver developer.
2. Copy the JDBC driver JAR to `pentaho/jdbc-distribution`.
3. Open a command prompt or shell, then run one of these commands:
* Windows: `distribute-files.bat `
* Linux: `./distribute-files.sh `
4. If you ran this utility during installation, continue with your install steps.
5. If you ran this utility to connect to a new repository:
* Restart Pentaho Server and the design tools.
* Try the connection again.
* If the connection still fails, verify driver locations below.
| Product | JDBC driver location |
| ---------------------------------- | --------------------------------------------------- |
| Pentaho Server | `pentaho/server/pentaho-server/tomcat/lib` |
| Pentaho Data Integration (Spoon) | `pentaho/design-tools/data-integration/lib` |
| Pentaho Report Designer (PRD) | `pentaho/design-tools/report-designer/lib/jdbc` |
| Pentaho Aggregation Designer (PAD) | `pentaho/design-tools/aggregation-designer/drivers` |
| Pentaho Schema Workbench (PSW) | `pentaho/design-tools/schema-workbench/drivers` |
| Pentaho Metadata Editor (PME) | `pentaho/design-tools/metadata-editor/libext/JDBC` |
# Schedules
All your active scheduled reports appear on the Schedules perspective.
Click **Home** > **Schedules**.
The Schedules perspective lists your reports that are scheduled to run, the recurrence pattern of the schedule, when it was last run, when it is set to run again, and the current status. You can edit and maintain your schedules using the toolbar. You can also click **Create Blockout Time** to stop reports from running. See the **Pentaho Business Analytics** document for more information.
# Pentaho Data Integration 11.0
Pentaho Data Integration (PDI) provides the Extract, Transform, and Load (ETL) capabilities that facilitate the process of capturing, cleansing, and storing data using a uniform and consistent format that is accessible and relevant to end users and IoT technologies.
If you or your administrator has not already installed PDI on your system, see the [Install Pentaho Data Integration and Analytics](https://docs.pentaho.com/install/) document for details.
Get started with Pentaho Data Integration (PDI) by learning core ETL concepts, data types, client setup, and project organization.
* [Basic concepts of ETL in PDI](/pdia-data-integration/basic-concepts-of-pdi)
PDI uses a workflow metaphor as building blocks for transforming your data and other tasks. Workflows are built using steps or entries as you create transformations and jobs. Each step or entry is joined by a hop which passes the flow of data from one item to the next.
* [Understanding PDI data types and field metadata](/pdia-data-integration/understanding-pdi-data-types-and-field-metadata)
As a best practice for producing consistent, predictable outcomes when working with your data in PDI, you must consider how the Pentaho engine processes different data types and field metadata in transformations and jobs.
* [Starting the PDI client](/pdia-data-integration/start-the-pdi-client)
After you have installed Pentaho Data Integration (PDI), you can use the PDI client (also known as Spoon) desktop application to start building transformations of your data.
* [Use the PDI client perspectives](/pdia-data-integration/use-the-pdi-client-perspectives)
Pentaho Data Integration (PDI) empowers you with tools that include ETL and scheduling in one unified environment — the PDI client interface. This integrated environment enables you to work in close cooperation with business users to build business intelligence solutions more quickly and efficiently.
# Basic concepts of ETL in PDI
PDI uses a workflow metaphor as building blocks for transforming your data and other tasks. Workflows are built using steps or entries as you create transformations and jobs. Each step or entry is joined by a hop which passes the flow of data from one item to the next.
## Transformations
A transformation is a network of logical tasks called steps. Transformations are essentially data flows. In the example below, the database developer has created a transformation that reads a flat file, filters it, sorts it, and loads it to a relational database table. Suppose the database developer detects an error condition and instead of sending the data to a Dummy step (which does nothing), the data is logged back to a table. The transformation is, in essence, a directed graph of a logical set of data transformation configurations. Transformation file names have a `.ktr` extension.
The two main components associated with transformations are steps and hops:
* Steps are the building blocks of a transformation, for example a text file input or a table output. There are many steps available in Pentaho Data Integration and they are grouped according to function; for example, input, output, scripting, and so on. Each step in a transformation is designed to perform a specific task, such as reading data from a flat file, filtering rows, and logging to a database as shown in the example above. You can add a step by dragging it from the **Design** tab onto the canvas, or by double-clicking the step. Steps can be configured to perform the tasks you require. See [PDI transformation steps](/pdia-data-integration/pdi-transformation-steps-reference-overview) for details about the features and ETL functions of the various transformation steps available in PDI.
* Hops are data pathways that connect steps together and allow schema metadata to pass from one step to another. In the image above, it seems like there is a sequential execution occurring; however, that is not true. Hops determine the flow of data through the steps not necessarily the sequence in which they run. When you run a transformation, each step starts up in its own thread and pushes and passes data.
**Note:** All steps in a transformation are started and run in parallel so the initialization sequence is not predictable. That is why you cannot, for example, set a variable in a first step and attempt to use that variable in a subsequent step.
You can connect steps together, edit steps, and open the step **contextual** menu by clicking to edit a step. For more information about connecting steps with hops, see [Hops](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Data%20Integration%20Perspective/Data%20Integration%20perspective%20in%20the%20PDI%20client/broken-reference/README.md).
A step can have many connections. Some steps join other steps together, while some serve as an input or output for another step. The data stream flows through steps to the various steps in a transformation. Hops are represented in Spoon as arrows. Hops allow data to be passed from step to step, and also determine the direction and flow of data through the steps. If a step sends outputs to more than one step, the data can either be copied to each step or distributed among them.
## Jobs
Jobs are workflow-like models for coordinating resources, execution, and dependencies of ETL activities.
Jobs aggregate individual pieces of functionality to implement an entire process. Examples of common tasks performed in a job include getting FTP files, checking conditions such as existence of a necessary target database table, running a transformation that populates that table, and e-mailing an error log if a transformation fails. The final job outcome might be a nightly warehouse update, for example.
Job entries are the individual configured pieces as shown in the example above; they are the primary building blocks of a job. In data transformations these individual pieces are called steps. Job entries can provide you with a wide range of functionality ranging from executing transformations to getting files from a Web server. A single job entry can be placed multiple times on the canvas; for example, you can take a single job entry such as a transformation run and place it on the canvas multiple times using different configurations. Job settings are the options that control the behavior of a job and the method of logging a job’s actions. Job file names have a `.kjb` extension. See [PDI job entries](/pdia-data-integration/pdi-job-entries-reference-overview) for details about the features and ETL functions of the various job entries available in PDI.
Job hops specify the execution order and the condition on which the next job entry will be executed. You can specify the **Evaluation** mode by right clicking on the job hop. A job hop is just a flow of control. Hops link to job entries and, based on the results of the previous job entry, determine what happens next.
**Note:** Hops behave differently when used in a job than when used in a transformation.
Job hop conditions are specified in the following table:
| Condition | Description |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Unconditional** | Specifies that the next job entry will be executed regardless of the result of the originating job entry |
| **Follow when result is true** | Specifies that the next job entry will be executed only when the result of the originating job entry is true; this means a successful execution such as, file found, table found, without error, and so on |
| **Follow when result is false** | Specifies that the next job entry will only be executed when the result of the originating job entry was false, meaning unsuccessful execution, file not found, table not found, error(s) occurred, and so on |
## Hops
A hop connects one transformation step or job entry with another. The direction of the data flow is indicated by an arrow. To create the hop, click the source step, then press the SHIFT key and draw a line to the target step. Alternatively, you can draw hops by hovering over a step until the **hover** menu appears. Drag the **hop painter** icon from the source step to your target step.
Additional methods for creating hops include:
* Click on the source step, hold down the middle mouse button, and drag the hop to the target step.
* Use CTRL and left-click to select two steps the right-click on the step and choose **New Hop**.
To split a hop, insert a new step into the hop between two steps by dragging the step over a hop. Confirm that you want to split the hop. This feature works with steps that have not yet been connected to another step only.
Loops are not allowed in transformations because Spoon depends heavily on the previous steps to determine the field values that are passed from one step to another. Allowing loops in transformations may result in endless loops and other problems. Loops are allowed in jobs because Spoon executes job entries sequentially; however, make sure you do not create endless loops.
Mixing rows that have a different layout is not allowed in a transformation; for example, if you have two table input steps that use a varying number of fields. Mixing row layouts causes steps to fail because fields cannot be found where expected or the data type changes unexpectedly. The trap detector displays warnings at design time if a step is receiving mixed layouts.
You can specify if data can either be **copied**, **distributed**, or **load balanced** between multiple hops leaving a step. Select the step, right-click and choose **Data Movement**.
A hop can be enabled or disabled (for testing purposes for example). Right-click on the hop to display the **Options** menu.
# Understanding PDI data types and field metadata
This section is for users who want to maximize the efficiency of their transformation and job results.
As a best practice for producing consistent, predictable outcomes when working with your data in PDI, you must consider how the Pentaho engine processes different data types and field metadata in transformations and jobs. For example, steps like [Avro Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-input) and [Text File Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/text-file-input-cp), require additional considerations to best meet your working requirements for specific data types, mathematical operations, number conversions, and formatting.
**Note:** As a rule, data is never modified by metadata inside of PDI. Data is only modified when PDI writes to files or similar objects, but not to databases. Refer to the sections below that apply to your use case.
## Data type mappings
PDI data types map internally to Java data types, so the Java behavior of these data types applies to the associated fields, parameters, and variables used in your transformations and jobs. The following table describes these mappings.
| PDI data type | Java data type | Description |
| ---------------- | -------------- | ----------------------------------------------------------------------------- |
| BigNumber | BigDecimal | An arbitrary unlimited precision number. |
| Binary | Byte\[] | An array of bytes that contain any type of binary data. |
| Boolean | Boolean | A boolean value `true` or `false.` |
| Date | Date | A date-time value with millisecond precision. |
| Integer | Long | A signed long 64-bit integer. |
| Internet Address | InetAddress | An Internet Protocol (IP) address. |
| Number | Double | A double precision floating point value. |
| String | String | A variable unlimited length text encoded in UTF-8 (Unicode). |
| Timestamp | Timestamp | Allows the specification of fractional seconds to a precision of nanoseconds. |
### Using the correct data type for math operations
Using the correct data type for math operations helps ensure expected results from your transformations and jobs. The Number, BigNumber, and Integer types offer specific solutions for different computing needs. The following table highlights the behaviors and possible uses for each data type. For information about the proper method to round or truncate numbers, see [Applying calculations and rounding](#applying-calculations-and-rounding).
| PDI data type | Description |
| ------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Number |
Use for general math with real numbers:
Decimal precision is not guaranteed.
Normally precise within 15 to 16 decimal points.
15.4 may be represented as 15.400000000000000000001111111
15.498723528092515678989712397 may be 15.498723528092515701252…
|
| BigNumber |
Use to get exact results from math of decimal numbers:
Guarantees precision to about 2 billion decimal places.
Requires more memory than Integer or Number.
15.498723528092515678989712397 will always be 15.498723528092515678989712397
|
| Integer |
Use for math without a fraction or a decimal component:
Handles minimum and maximum values ranging from -9,223,372,036,854,775,808 to 9,223,372,036,854,775,807
|
## Using the fields table properties
You define properties for the fields to read or write using the fields table. The properties in the fields table determine the field-level processing options for your row data, including the metadata attributes. Some commonly used steps that include a fields table are [Split Fields](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Transformation%20steps/PDI%20transformation%20steps%20reference%20\(overview\)/Split%20Fields=GUID-36055D0C-1602-4F21-BEA7-BEEDF325CAAD=2=en=.md), [Select Values](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Transformation%20steps/PDI%20transformation%20steps%20reference%20\(overview\)/Select%20Values=GUID-14FD2047-C05A-4E28-A0E9-8C8696823EC4=2=en=.md), [Text File Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/text-file-output-cp), and [Concat Fields](http://wiki.pentaho.com/display/EAI/Concat+Fields).
When using the fields table the following definitions and processing rules apply.
**Note:** Depending on the transformation step or job entry, some fields tables may feature only a portion of the columns listed below.
* **Name**
The name of the field.
* **Type**
The type of the field. For example, String, Date, or Number. See [Data type mappings](#data-type-mappings) for more information.
* **Format**
Defines the format mask to use when converting the value to, or reading the value from, a string. The **Format** drop-down menu offers suggestions, but you can enter your own mask. **Format** is only used when converting a non-string data type to a string data type. **Format** overrides **Length** and **Precision**. See [Applying formatting](#applying-formatting) for formatting details.
* **Length**
Defines the length to use when converting the value to, or reading the value from, a string. The numbers before the decimal point, or a value that is longer than the maximum length, will not be truncated. **Length**, also called Precision in some databases, is a metadata component. PDI converts to the required metadata type when the data is resulted to a string, not during the transformation (or job) or if resulted to non-string data types. See [Output type examples](#output-type-examples) for a listing of string and non-string types. **Length** is not used when **Format** is specified.
* **Precision**
Defines the number of digits after the decimal point to use when converting the value to, or reading the value from, a string. The numbers before the decimal point will not be truncated. **Precision**, also called Scale in some databases, is a metadata component. PDI converts to the required metadata type when the data is resulted to a string, not during the transformation (or job) or if resulted to non-string data types. See [Output type examples](#output-type-examples) for a listing of string and non-string types. **Precision** is not used when **Format** is specified.
* **Currency**
Used in conjunction with **Format** to interpret numbers such as `$10,000.00` or `E5.000,00`. If the format mask contains the Unicode currency symbol ¤ (`\u00A4`), then it replaces the symbol by the value in the currency column. In Pentaho, you must use the copy and paste method to apply this symbol. See [Common Formats](/pdia-data-integration/pdi-transformation-steps-reference-overview/common-formats) for information on valid number formats.
* **Decimal**
Represents the character that replaces the period (.) in the format mask. Only applies when converting the value to, or reading the value from, a string.
* **Group**
Represents the character that replaces the comma (,) in the format mask. Only applies when converting the value to, or reading the value from, a string.
* **Null if**
Converts the value to null if the input value matches.
**Note:** This value is case-sensitive.
* **Default**
Defaults to this value if the value is null.
* **Trim type**
Defines the type of trimming to perform on the input or the output string. Trimming removes the white space on either side of a string. Options are both, left, right, or none.
* **Repeat**
Determines how null rows are handled. If the value in this row is null, then the value from the last row where the column was not null is used.
### Applying formatting
Format masks define how data returned for a field is converted to, or from, a string. For example, a field might return the value "`7000`", but you want to display it as "`$7,000.00`". To do this, you apply a format mask to the field. The original data is not truncated when using a format mask.
As shown in the table below, when **Format** is used with **Decimal**, the period (.) in the format mask is replaced with the indicated character. Alternatively, when **Format** is used with **Group**, the comma (,) in the format mask is replaced with the indicated character. See [Common Formats](/pdia-data-integration/pdi-transformation-steps-reference-overview/common-formats) for information on valid number formats.
| Input / Output value | **Format** | **Decimal** | **Group** |
| -------------------- | ---------- | ----------- | --------- |
| 10.0 | #.# | . | , |
| 1,546.99 | #,###.## | . | , |
| 1g546d99 | #,###.## | d | g |
| €1.546,99 | €#,###.00 | , | . |
| $1,546.99 | $#,###.00 | . | , |
The following table shows that when \*\*Format\*\*, \*\*Decimal\*\*, \*\*Group\*\*, \*\*Length\*\*, and \*\*Precision\*\* are used together. \*\*Format\*\* always overrides \*\*Length\*\* and \*\*Precision\*\*.
| Input | **Format** | **Decimal** | **Group** | **Length** | **Precision** | String output | Number output |
| -------- | ---------- | ----------- | --------- | ---------- | ------------- | ------------- | ------------- |
| 10.0 | #.# | . | , | 5 | 2 | 10.0 | 10.0 |
| 10.0 | | . | , | 5 | 2 | 010.00 | 10.0 |
| 10.01 | | . | , | 2 | 1 | 10.0 | 10.01 |
| 1,546.99 | #,###.## | . | , | 10 | 3 | 1,546.99 | 1546.99 |
| 1,546.99 | 0#,###.000 | . | , | | | 01,546.990 | 1546.99 |
### Applying calculations and rounding
Number and date calculations performed in PDI do not apply the **Format**, **Length**, and **Precision** properties. For example, using the table below, A + B + B = `30.1` If you preview B, it will appear as `10.0`, so you would think `10.02 + 10.0 + 10.0` `= 30.02`. However, because B was never converted to a string for the calculation, `10.02 + 10.04 + 10.04` `= 30.1.`
| Field | Input | **Format** | **Decimal** | **Group** | **Length** | **Precision** |
| ----- | ------- | ---------- | ----------- | --------- | ---------- | ------------- |
| A | `10.02` | #.0 | . | , | 5 | 1 |
| B | `10.04` | | . | , | 5 | 1 |
If you want to truncate a string, use the [Strings cut](/pdia-data-integration/pdi-transformation-steps-reference-overview/strings-cut) step.
If you want to round or truncate a number, use the following [Calculator](/pdia-data-integration/pdi-transformation-steps-reference-overview/calculator) step features:
* Round function
* Floor and Ceil functions
Alternatively, you can convert the date or number to a string in the [Select Values](/pdia-data-integration/pdi-transformation-steps-reference-overview/select-values) step, which applies the formatting specified in the metadata.
## Output type examples
The following table provides examples of the string and non-string output types in PDI. Note that **Format**, **Length**, **Precision**, **Decimal**, and **Group** apply only when reading from, or outputting to, a string.
| String output type example | Non-string output type example |
| ---------------------------------------------------------------------- | ------------------------------------------------- |
| Preview | Table Output when the target field is a number. |
| Text File Output | Avro Output when the target field is a number. |
| JSON Output | Parquet Output when the target field is a number. |
| XML Output | ORC Output when the target field is a number. |
| Table Output when the target field is a varchar. | Any binary output type |
| Anything displayed by PDI including logs, error messages, and prompts. | |
# Starting the PDI client
After you have installed Pentaho Data Integration (PDI), you can use the PDI client (also known as Spoon) desktop application to start building transformations of your data.
Perform the following steps to launch the PDI client from the `Pentaho` directory.
1. Start the Pentaho Server.
2. Navigate to the folder where you have installed PDI. For example `..\pentaho\design-tools\data-integration`.
3. Launch the PDI client in the best way for your operating system.
* For Windows: Double-click `Spoon.bat`
* For Linux: Double-click `spoon.sh`
* For Mac: Go to `../pdi-ee/data-integration` and double-click the **Data Integration** icon.
4. If you do not have a license installed, complete the following substeps to install a license:
1. When the License Manager dialog box appears, click the plus sign (**+**) to open the Add License window.
2. Select **License Server**, enter either the cloud license server URL or the local license server URL, and then click **OK**.
The URL for requesting a license from the local license server looks like the following example:
**Note:** The **Activation Code** option is only for activating a trial license.
3. Review the license summary to see which components you are now entitled to use, and then click **Close**.
The PDI client application opens on your desktop.
# Use the PDI client perspectives
Pentaho Data Integration (PDI) empowers you with tools that include ETL and scheduling in one unified environment — the PDI client interface. This integrated environment enables you to work in close cooperation with business users to build business intelligence solutions more quickly and efficiently.
When you are working in the PDI client, you can change perspectives to easily switch back and forth from:
* Designing ETL jobs and transformations, and
* Scheduling jobs and transformations.
As users provide you with feedback about how the data is presented to them, you can also quickly make iterative changes to your data directly using our data inspection tools in the PDI client.
From within the PDI client, you can change perspectives using the **Perspective** icon in the toolbar.
Perspectives in PDI help you focus how you work with different tasks.
## Data Integration perspective
You can use the **Data Integration** perspective to develop ETL programs known as transformations and jobs.
Consider the following example and table as a guide to the major sections of the **Data Integration** perspective.
In the table below, match the numbered items in the example illustration above to reference the toolbars, buttons, and areas on the PDI client.
| Item | Feature | Description |
| ---- | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| 1 | Toolbar |
Use this toolbar to access commonly performed actions: - New file button ( ) to create a new job, transformation, database connection, or slave server.
Open file button ( ) to open a transformation or job from a file.
Explore Repository button ( ) to explore the repositories.
Save button ( ) to save the current transformation or job to a file or repository.
Save As button ( ) to save the transformation or job under a different file name or type.
Perspectives button ( ) to switch between the different perspectives:
Data Integration Perspective: Create ETL transformations and jobs.
Schedule Perspective: Manage scheduled ETL activities on the Pentaho Server.
|
| 2 | **Connect** button | Use this button to access the menu to create and connect to [repositories](/pdia-data-integration/redirects/use-a-pentaho-repository-in-pdi) for central storage of your ETL jobs and transformations. |
| 3 | Sub-toolbar |
Use this toolbar to perform transformation or job actions:- Run button ( ) to run a transformation or job: - Run: Runs the current transformation or job from an XML file or a repository. - Run Options: Sets the Run Options and then runs the current transformation or job from an XML file or a repository.
Pause button ( ) to pause a running transformation or job.
Stop button ( ) to stop a running transformation or job:
Stop: Stops the transformation or job immediately.
Stop input processing: Stops the input steps to the transformation or job, while allowing any records already retrieved or initiated to be processed and then stopped.
Preview button ( ) to run the transformation in preview mode to examine the rows produced by the selected steps.
Debug button ( ) to run the transformation in debug mode to troubleshoot execution errors.
Replay button ( ) to replay the processing of a transformation.
Verify button ( ) to verify the transformation.
Analyze button ( ) to run an impact analysis on the database.
SQL button ( ) to generate the SQL that is needed to run the loaded transformation.
Explore DB button ( ) to launch the Database Explorer to perform actions such as preview data, run SQL queries, and generate DDL.
Results button ( ) to show the Execution Results pane.
Lock button ( ) to lock the transformation.
|
| 4 | Explore pane |
Use this pane to access the Design and View tabs:
The Design tab provides a list of steps or entries that are used to build transformations or jobs.
The View tab provides information about available database connections and the steps and hops used for the transformation or job.
|
| 5 | Canvas | Use this canvas for designing and building transformations and jobs for the ETL activities you want to perform. |
## Scheduler perspective
You can plan when to run transformations and jobs and set timed intervals to automatically send the output to your preferred destinations. See [Schedule a transformation or job](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/manage-pdi-transformations-and-job-schedules#schedule-a-transformation-or-job) for more details.
The table below outlines the features in the Scheduler perspective.
| Item | Feature | Description |
| ---- | ----------------- | ------------------------------------------------------------------------------------------------- |
| 1 | Perspectives icon | Allows you to switch between perspectives in PDI. |
| 2 | Scheduler toolbar | Provides single-click access to common actions such as edit, refresh, enable, disable, or delete. |
| 3 | Schedules table | Contains a list of schedules. |
# Pipeline Designer
Pipeline Designer is a web-based interface that you can use to design, execute, and manage data integration workflows directly in your browser. A wide range of database connections, advanced transformation steps, and robust execution monitoring is supported in Pipeline Designer.
By default, Pipeline Designer is installed when you install the Pentaho Server. To update the Pipeline Designer plugin, use the Plugin Manager. For details, see [Update plugins](/pdia-data-integration/redirects/update-plugins-in-pdi#use-plugin-manager-to-update-plugins).
**Note:** The Project option appears on the main page of the Pipeline Designer but is not currently supported in Pipeline Designer. You can use Projects in the Pentaho Data Integration client. For details, see [Organizing data integration with projects](/pdia-data-integration/organizing-etl-with-projects).
* [Views in Pipeline Designer](/pdia-data-integration/pipeline-designer/views-in-pipeline-designer)
In Pipeline Designer, you can work with transformations and jobs in the Design View and review the raw XML in the XML View.
* [Basic concepts of ETL in Pipeline Designer](/pdia-data-integration/basic-concepts-of-pdi)
Pipeline Designer uses a workflow metaphor as building blocks for transforming your data and other tasks. Workflows are built using steps as you create transformations and jobs. Each step is joined by a hop which passes the flow of data from one item to the next.
* [Manage transformations and jobs](/pdia-data-integration/transforming-data-with-pdi)
You can manage transformations and jobs on the Pipeline Designer's main page. Available actions include marking transformations and jobs as favorites, downloading them as files, creating duplicates, moving them to different folders or to the trash, renaming them, and viewing their details.
* [Working with transformations](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp)
Create, configure, and then run transformations to perform ETL tasks as part of specified workflow. After running a transformation, analyze its results to explore the data or to identify possible improvements or problems.
* [Working with jobs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs)
Create, configure, and then run jobs to orchestrate ETL activities. After running a transformation, analyze its results to identify possible improvements or problems.
* Stop transformations and jobs
Use [Stop your transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/stop-your-transformation) or [Stop your job](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs/stop-your-job).
* [Transformation steps in Pipeline Designer](/pdia-data-integration/pdi-transformation-steps-reference-overview)
List of steps available for extending and expanding the functionality of transformations.
* [Job steps in Pipeline Designer](/pdia-data-integration/pdi-job-entries-reference-overview)
List of available steps for extending and expanding the functionality of jobs.
# Views in Pipeline Designer
In Pipeline Designer, you can work with transformations and jobs in the Design View and review the raw XML in the XML View.
## Design View
The Design View is the primary view to use for creating and running transformations and jobs in the Pipeline Designer.
### Canvas Action toolbar
While working with a transformation or job, use the following options in the Canvas Action toolbar:
| Option | Description |
| ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Add note | Adds a note to the canvas. For details, see [Add notes to transformations and jobs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/add-notes-to-transformations-and-jobs). |
| Reset |
Clears all steps in the transformation or job.
Note: Resetting a transformation or job cannot be undone.
|
| Run | Runs the transformation or job. |
| Run > Run Options | Opens the Run Options window for the transformation or job. For details, see [Run your transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/run-your-transformation) or [Run your job](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs/run-your-job). |
| Pause | Pauses a running transformation or job. |
| Stop | Stops processing all data in the transformation or job immediately. |
| Stop > Stop Processing | Makes the transformation or job finish any records that were initiated or retrieved and then stops processing data. |
| Logs | Opens the Execution Results pane for a transformation or job. For details, see [Analyze your transformation results](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/run-your-transformation/analyze-your-transformation-results) or [Logging and performance monitoring](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring). |
| Kettle status | Opens the Kettle status page ([http://server:port/pentaho/kettle/status](https://docs.pentaho.com/pdia-data-integration/pipeline-designer/http:/server:port/pentaho/kettle/status)) in a new tab of your browser, showing the execution status and details of your transformation or job. |
| Export | Exports a transformation as a `.ktr` file or a job as a `.kjb` file. For details, see [Save a transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/save-a-transformation) or [Save a job](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs/save-a-job). |
| Settings | Opens the Transformation Properties or Job Properties window, where you can configure properties to describe or control the transformation or job. For details, see [Configure transformation properties](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/configure-transformation-properties) or [Configure job properties](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs/configure-job-properties). |
### Canvas toolbar
Move around the canvas while working with a semantic model by using the following options in the canvas toolbar:
| Option | Description |
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Zoom In | Enlarges the size of the semantic model on the canvas so that you can see more details of individual elements in the model. |
| Zoom Out | Decreases the size of the semantic model on the canvas so that you can see more elements in the model at the same time, with less detail for individual elements. |
| Fit View | Resizes the semantic model and centers it on the canvas so that you can see as many elements in the model at the same time as possible. Whether the entire model can fit in the view depends on the size of the model. |
| Interactive | Locks and unlocks the canvas to control whether elements can be moved on the canvas. Turn the Interactive toggle on when you want to lock the canvas so that you can move around it without moving elements on the canvas. |
| More actions > Auto format | Uniformly arranges all steps in the transformation or job so that the steps all fit on the canvas, but do not overlap. |
| More actions > Horizontal format | Uniformly arranges steps in each data stream horizontally (left to right) without overlap. Separate streams are distributed vertically to maintain a clear and organized canvas layout. |
| More actions > Vertical format | Uniformly arranges steps in each data stream vertically (top to bottom) without overlap. Separate streams are distributed horizontally to maintain a clear and organized canvas layout. |
| \ XML View | Switches to the XML View, which shows the raw XML for the transformation or job. |
## XML View
In the XML View, you can review the raw XML for a transformation or job.
You can click **Design View** to continue working on the transformation in the Design View.
# Basic Concepts of ETL in Pipeline Designer
## Basic concepts of ETL
The Pentaho Data Integration (PDI) platform, which includes the PDI client and Pipeline Designer, uses a workflow metaphor as building blocks for transforming your data and other tasks. Workflows are built using steps as you create transformations and jobs. Each step is joined by a hop which passes the flow of data from one item to the next.
### Transformations
A transformation is a network of logical tasks called steps. Transformations are essentially data flows. The transformation is, in essence, a directed graph of a logical set of data transformation configurations. Transformation file names have a `.ktr` extension.
The two main components associated with transformations are steps and hops:
* Steps are the building blocks of a transformation, for example a text file input or a table output. There are many steps available in the Pipeline Designer and they are grouped according to their function; for example, input, output, transform, and so on. Each step in a transformation is designed to perform a specific task, such as reading data from a flat file, filtering rows, or logging to a database. You can add a step by dragging it from the **Design** pane onto the canvas. Steps can be configured to perform the tasks you require. See [Transformation steps in Pipeline Designer](https://docs.pentaho.com/pba/11.0-pba/pipeline-designer/working-with-transformations/transformation-steps-in-pipeline-designer) for details about the features and ETL functions of the various transformation steps available in Pipeline Designer.
* Hops are data pathways that connect steps together and allow schema metadata to pass from one step to another. Hops determine the flow of data through the steps not necessarily the sequence in which they run. When you run a transformation, each step starts up in its own thread and pushes and passes data.
**Note:** All steps in a transformation are started and run in parallel, so the initialization sequence is not predictable. That is why you cannot, for example, set a variable in a first step and attempt to use that variable in a subsequent step.
You can connect steps together with hops. For details, see [Hops](https://docs.pentaho.com/pba/11.0-pba/pipeline-designer/basic-concepts-of-etl#hops). A step can have many connections. Some steps join other steps together, while some serve as an input or output for another step. The data stream flows through steps to the various steps in a transformation. Hops are represented in Pipeline Designer as arrows. Hops allow data to be passed from step to step and also determine the direction and flow of data through the steps. If a step sends outputs to more than one step, the data can either be copied to each step or distributed among them.
### Jobs
Jobs are workflow-like models for coordinating resources, execution, and dependencies of ETL activities.
Jobs aggregate individual pieces of functionality to implement an entire process. Examples of common tasks performed in a job include getting FTP files, checking conditions such as existence of a necessary target database table, running a transformation that populates that table, and e-mailing an error log if a transformation fails. The final job outcome might be a nightly warehouse update, for example.
Job entries are the individual configured pieces; they are the primary building blocks of a job. In data transformations these individual pieces are called steps. Job entries can provide you with a wide range of functionality ranging from executing transformations to getting files from a Web server. A single job entry can be placed multiple times on the canvas; for example, you can take a single job entry such as a transformation run and place it on the canvas multiple times using different configurations. Job settings are the options that control the behavior of a job and the method of logging a job’s actions. Job file names have a `.kjb` extension. See [Job steps in Pipeline Designer](https://docs.pentaho.com/pba/11.0-pba/pipeline-designer/working-with-jobs/job-steps-in-pipeline-designer) for details about the features and ETL functions of the various job entries available in Pipeline Designer.
Job hops control the execution order and the condition on which the next job entry will be executed. A job hop is just a flow of control. Hops link to job entries and, based on the results of the previous job entry, determine what happens next.
**Note:** Hops behave differently when used in a job than when used in a transformation.
Job hop conditions are described in the following table:
ConditionDescription
**Unconditional**
The next job entry is executed regardless of the result of the originating job entry
**Follow when result is true**
The next job entry is executed only when the result of the originating job entry is true; this means a successful execution such as, file found, table found, without error, and so on
**Follow when result is false**
The next job entry is executed only when the result of the originating job entry was false, meaning unsuccessful execution, file not found, table not found, error(s) occurred, and so on
### Hops
A hop connects one transformation step or job entry with another. The direction of the data flow is indicated by an arrow. To create the hop, click the handle of one step and drag the connection to the handle of another step.
Loops are not allowed in transformations because Pipeline Designer depends heavily on the previous steps to determine the field values that are passed from one step to another. Allowing loops in transformations may result in endless loops and other problems. Loops are allowed in jobs because Pipeline Designer executes job entries sequentially; however, make sure you do not create endless loops.
Mixing rows that have a different layout is not allowed in a transformation; for example, if you have two table input steps that use a varying number of fields. Mixing row layouts causes steps to fail because fields cannot be found where expected or the data type changes unexpectedly. The trap detector displays warnings at design time if a step is receiving mixed layouts.
In transformations, you can specify if data can either be **copied**, **distributed**, or **load balanced** between multiple hops leaving a step. Select the step, right-click and choose **Data Movement**.
A hop can be enabled or disabled (for testing purposes for example). Click the hop to enable or disable it.
# Working with transformations
Create, configure, and run transformations to perform ETL tasks in Pipeline Designer. After you run a transformation, review logs and step metrics to validate results and troubleshoot issues.
{% hint style="info" %}
To schedule transformations to run one time or on a recurring schedule, use the Pentaho Data Integration client (Spoon). See [Manage PDI transformations and job schedules](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/manage-pdi-transformations-and-job-schedules).
{% endhint %}
### Create a transformation
Create a transformation to arrange a network of ETL tasks (steps) into a data workflow.
1. Log in to the Pentaho User Console.
2. Open **Pipeline Designer**:
* If you are using the **Modern Design**, select **Pipeline Designer** from the left menu.
* If you are using the **Classic Design**, select **Switch to the Modern Design**, then select **Pipeline Designer**.
3. In the **Transformation** card, select **Create Transformation**.
4. Add steps:
1. In the **Design** pane, search for or browse to the steps you need.
2. Drag steps onto the canvas.
5. Configure steps by hovering over a step and selecting an action:
* **Delete**: Remove the step from the canvas.
* **Edit**: Configure step properties.
* To learn more about a step while configuring it, select **Help** in the step dialog.
* **Duplicate**: Add a copy of the step to the canvas.
* **More Actions** > **Change Number of Copies**: Run multiple copies of a step in parallel.
* **More Actions** > **Data Movement**: Control how rows flow to downstream step copies:
* **Round-robin** (default): Distribute rows evenly across downstream step copies.
* **Load balance**: Send rows to the downstream copy with the lightest load.
* **Copy data to next steps**: Send each row to all downstream copies.
6. Add hops (connections) between steps:
* Hover over a step handle until the plus sign (+) appears.
* Drag from the plus sign to another step handle.
7. Optional: Add a note to the canvas.
* Use the canvas toolbar, or see [Use notes on canvas](/pdia-data-integration/pipeline-designer/use-notes-on-canvas).
8. Save the transformation:
1. Select **Save**.
2. In **Select File or Directory**, pick the target folder.
3. Optional: Create a folder by selecting **New Folder**.
4. Select **Save**, then confirm in **Save Change**.
### Edit transformation properties
Transformation properties describe the transformation and control how it runs.
#### Dependencies tab
Use **Dependencies** to document and validate upstream tables your transformation depends on. This is useful when you need to detect source table changes by using a “last changed” field.
Select **Get dependencies** to detect dependencies automatically.
| Setting | Description |
| ---------- | -------------------------------------------------------- |
| Connection | Select a database connection used by the transformation. |
| Table | Select a table from the selected connection. |
| Field | Select a field within the selected table. |
#### Miscellaneous tab
Use **Miscellaneous** to tune buffers and logging feedback, and to configure transactional behavior.
| Setting | Description |
| ---------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| Number of rows in rowset | Size of the buffers between connected steps. Change only if you are memory constrained. |
| Show a feedback row in transformation steps? | Adds periodic feedback entries to the log during execution. Default is enabled. |
| The feedback size | Number of rows between feedback log entries. Increase this for large runs to reduce log volume. |
| Make the transformation database transactional | Uses a single transactional connection per database connection and rolls back on failure. Commit happens after the last step finishes. |
| Shared objects file | Location of the XML file for shared objects (connections, cluster schemas, and more). |
| Manage thread priorities? | Enables thread-priority tuning based on rowset buffer levels. Disable if the overhead outweighs benefits. |
#### Monitoring tab
Use **Monitoring** to enable step performance monitoring and control how often metrics are sampled.
| Setting | Description |
| ------------------------------------------ | ------------------------------------------------------------------------------------------------------ |
| Enable step performance monitoring? | Collects per-step throughput and I/O metrics for viewing in the execution results (performance graph). |
| Step performance measurement interval (ms) | Snapshot interval in milliseconds (example: `10`). |
| Maximum number of snapshots in memory | Maximum number of metric snapshots retained during runtime. |
### Run a transformation
Run a transformation to execute steps in the order defined by the hops on the canvas.
1. Log in to the Pentaho User Console.
2. Open **Pipeline Designer**:
* If you are using the **Modern Design**, select **Pipeline Designer** from the left menu.
* If you are using the **Classic Design**, select **Switch to the Modern Design**, then select **Pipeline Designer**.
3. In the bottom table, select **Recently opened** or **Favorites**.
4. Open the transformation:
1. Browse to the transformation and select **Open**.
2. Or select **Open files**, then browse to the transformation and select **Open**.
5. In the **Canvas Action** toolbar, select **Run**, then choose one:
* **Run**: Start with defaults.
* **Run options**: Configure options, then select **Run**.
The transformation runs and the **Preview** panel opens with **Logging** selected.
{% hint style="info" %}
To stop a running transformation, see [Stop transformations and jobs](/pdia-data-integration/pipeline-designer/stop-transformations-and-jobs).
{% endhint %}
#### Run options
| Option | What it does |
| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Select configuration | Select the run configuration. |
| Clear log before running | Clears existing logs before execution. |
| Enable safe mode | Validates that each row matches the first row’s layout. |
| Gather performance metrics | Collects runtime metrics for step performance. See [Use performance graphs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring/monitor-performance/use-performance-graphs). |
| Log level | Controls the amount of logging. Higher levels can expose sensitive data. |
| Parameters | Applies parameter values for this run. See [Parameters](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/parameters). |
| Variables | Applies temporary values for variables for this run. See [Variables](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/variables). |
| Arguments | Supplies runtime arguments (max 10). See [Arguments](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/arguments). |
### Analyze transformation results
Analyze results to identify errors, preview data, and assess performance using logs and step metrics. After you run a transformation, **Execution Results** opens at the bottom of the canvas.
You can do the following:
* Expand results by selecting **Expand Preview**.
* Delete logs by selecting **Delete all logs**.
* Close the panel, then re-open it later from the **Canvas Action** toolbar by selecting **Logs**.
#### Logging
The **Logging** tab shows logging details for the most recent execution. Error lines are highlighted in red.
#### Preview data
The **Preview Data** tab shows a preview of data for each step. Select a step to view its output.
#### Step metrics
The **Step Metrics** tab shows per-step statistics, including rows read and written, errors, and processing speed (rows per second). Steps that caused a failure are highlighted in red.
### Transformation steps
Steps extend and expand transformation functionality. Add steps from the **Design** pane, then connect them with hops.
For the full reference list, see [PDI transformation steps](/pdia-data-integration/pdi-transformation-steps-reference-overview).
# Working with jobs
### Working with jobs
Create, configure, and then run jobs to orchestrate ETL activities. After running a transformation, analyze its results to identify possible improvements or problems.
* [Create a job](/pba/pipeline-designer/working-with-jobs/create-a-job)Create a job to coordinate resources, execution, and dependencies of an ETL activity.
* [Edit job properties](/pba/pipeline-designer/working-with-jobs/edit-job-properties)Job properties control how a job behaves and how it logs what it is doing.
* [Run a job](/pba/pipeline-designer/working-with-jobs/run-a-job)After you create a job and configure its properties, you can run the job. You can control how the job runs without modifying the job itself by configuring run options before running the job.
* [Analyze your job results](/pba/pipeline-designer/working-with-jobs/analyze-job-results)After running a job, see how it performed, if errors occurred, and explore the data by viewing the logs.
* [Job steps in Pipeline Designer](/pba/pipeline-designer/working-with-jobs/job-steps-in-pipeline-designer)Steps extend and expand the functionality of jobs. You can use the following steps in Pipeline Designer.
# Create a job
Create a job to coordinate resources, execution, and dependencies of an ETL activity.
To create a job in **Pipeline Designer**, complete the following steps:
1. Log into the Pentaho User Console.
2. Open **Pipeline Designer**:
* If you are using the **Modern Design**, in the menu on the left side of the page, click **Pipeline Designer**.
* If you are using the **Classic Design**, click **Switch to the Modern Design,** and then in the menu on the left side of the page, click **Pipeline Designer**.
**Pipeline Designer** opens with the **Quick Access** section expanded.
3. In the **Job** card, click **Create Job**. A new, blank job is opened with the **Design** pane selected.
4. Add steps to the job:
1. In the **Design** pane, search for or browse to each step you want to use in the job. You may need to expand sections in the **Design** pane to find steps.
2. Drag the steps you want to use onto the canvas.
5. Work with steps on the canvas by hovering over a step to open the step menu and selecting one of the following options:
Menu option
Description
Delete
Deletes the step from the canvas.
Edit
Opens the Step Name window where you can configure the properties of the step. Step properties may appear in multiple sections, tabs, or both.
Note: To learn more about the step you're configuring, in the lower-left corner of the Step Name window, click Help.
Duplicate
Adds a copy of the step to the canvas.
6. To add hops between steps, hover over a step’s handle until a plus sign (+) appears, then drag the connection to the handle of another step.
7. (Optional) To add a note on the canvas, in the canvas toolbar, click the **Add Note** icon, and then in **Notes** dialog box, enter your note and click **Save**.
**Note:** You can format the note in the **Notes** dialog box, by clicking **Style**, and then entering the font, color, and shadow options you want to use for the note.
8. Save the job:
1. Click **Save**. The **Select File or Directory** dialog box opens.
2. Search for or browse to the folder in the repository where you want to save the job.
3. (Optional) To create a new folder in the repository, click the **New Folder** icon, and then in the **New folder** dialog box, enter a **New folder name** and click **Save**.
4. (Optional) To delete a folder from the repository, select the folder and click the **Delete** icon.
5. In the **Select File or Directory** dialog box, click **Save**. The **Save Change** dialog box opens.
6. Click **Yes** to confirm that you want to save the job.
# Edit job properties
\
Edit job properties
-------------------
Job properties control how a job behaves and how it logs what it is doing.To configure job properties, complete the following steps:Log into the Pentaho User Console.Open Pipeline Designer:If you are using the Modern Design, in the menu on the left side of the page, click Pipeline Designer.If you are using the Classic Design, click Switch to the Modern Design, and then in the menu on the left side of the page, click Pipeline Designer.Pipeline Designer opens with the Quick Access section expanded.In the table at the bottom of the screen, select either the Recently opened or Favorites tab.Open the job:Search for or browse to the job, and then click Open.Click Open files, and then in the Select File or Directory dialog box, search for or browse to the job and click Open.In the Canvas Action toolbar, click the Settings icon. The Job Properties window opens.Configure the properties in each tab. To learn more about the properties in each tab, see theJob propertiesin this topic.To generate the SQL code necessary for creating the logging table, take following actions:Click SQL. The Simple SQL editor opens with DDL (Data Definition Language) generated from the properties of the job.(Optional) Edit the SQL statements. For details, seeUse the SQL Editor.(Optional) To remove stored query results, metadata, or temporary data that the editor has cached from previous SQL executions, click Clear cache.Click Execute. The SQL statements run.Click Save. The job properties are saved.Job propertiesThe following sections provide a detailed description of the available settings in the Job Properties window:Job tabParameters tabSettings tabLog tabTransactions tabJob tabGeneral properties for jobs are found on the Job tab.This table describes all of the general job properties found on the Job tab:Job NameThe name of the job.Note: This information is required if you want to save to a repository.Job filenameThe file name of the job if it is not stored in the repository.DescriptionA user-defined short description of the job which is shown in the repository explorer.Extended descriptionA user-defined longer description of the job.StatusThe status of the job. The values are draft and production.VersionA description of the version.DirectoryThe directory in the repository where the job is kept.Created byThe original creator of the job.Created atThe date and time when the job was created.Last modified byThe name of the last user who modified the job.Last modified atThe date and time when the job was last modified.Parameters tabYou can use the Parameters tab to define parameters for your jobs.This table describes all of the general job properties found on the Parameters tab:
| | |
| ----------------- | ------------------------------------------------ |
| **Parameter** | A user-defined parameter. |
| **Default value** | The default value of the user-defined parameter. |
| **Description** | A description of the parameter. |
#### Settings tab
The following options are available on the **Settings** tab:
| | |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Pass batch ID?** | Select to pass the identification number of the batch to the transformation. |
| **Shared objects file** | PDI uses a single shared objects file for each user. The default filename is `shared.xml` and is located in the `.kettle` directory in the user’s home directory. You can define a different shared objects file, location, and name. |
#### Logging tab
Use the **Logging** tab to specify logging settings.This table describes all of the general job properties found on the **Log** tab:
| | |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Log connection** | Specify the database connection you are using for logging. You can configure a new connection by clicking **New**. |
| **Log Schema** | Specify the schema name, if supported by your database. |
| **Log table** | Specify the name of the log table. f you are also using transformation logging, you must use a different table name for job logging. |
| **Logging interval (seconds)** | Specify the interval in which logs are written to the table. This property only applies to Transformation and Performance logging types. |
| **Log line timeout (days)** | Specify the number of days to keep log entries in the table before they are deleted. This property only applies to Transformation and Performance logging types. |
| **Log size limit in lines** | Enter the limit for the number of lines that are stored in the `LOG_FIELD`. PDI stores logging for the transformation in a long text field (CLOB). This property only applies to Transformation and Performance logging types. |
| **SQL button** | Generates the SQL needed to create the logging table and allows you to execute this SQL statement. |
# Run a job
After you create a job and configure its properties, you can run the job. You can control how the job runs without modifying the job itself by configuring run options before running the job.Run a job by completing steps in the following sections:
1. Log into the Pentaho User Console.
2. Open **Pipeline Designer**:
* If you are using the **Modern Design**, in the menu on the left side of the page, click **Pipeline Designer**.
* If you are using the **Classic Design**, click **Switch to the Modern Design,** and then in the menu on the left side of the page, click **Pipeline Designer**.
**Pipeline Designer** opens with the **Quick Access** section expanded.
3. In the table at the bottom of the screen, select either the **Recently opened** or **Favorites** tab.
4. Open the job:
1. Search for or browse to the job, and then click **Open**.
2. Click **Open files**, and then in the **Select File or Directory** dialog box, search for or browse to the job and click **Open**.
5. In the **Canvas Action** toolbar, click the **Run** icon and select one of the following options:
1. To run the job, select **Run**.
2. To run the job with run options, select **Run Options,** and then in the **Run Options** window, configure options and click **Run**. For details about run options, see [Job run options](/pba/pipeline-designer/working-with-jobs/run-a-job#job-run-options).
The job runs and the **Preview** panel opens with the **Logging** tab selected.
**Note:** For information about stopping the job while it is running, see [Stop transformations and jobs](/pba/pipeline-designer/managing-transformations-and-jobs/stop-transformations-and-jobs).
### Job run options
| | |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Select configuration** | All jobs are run using the **Pentaho server** configuration. |
| **Clear log before running** | Indicates whether to clear all your logs before you run your job. If your log is large, you might need to clear it before the next execution to conserve space. |
| **Enable safe mode** | Checks every row passed through your job and ensures all layouts are identical. If a row does not have the same layout as the first row, an error is generated and reported. |
| **Gather performance metrics** | Monitors the performance of your job execution. You can view performance metrics in the Pentaho Data Integration client. For details see [Use performance graphs](https://docs.pentaho.com/pdia-data-integration/data-integration-perspective-in-the-pdi-client/logging-and-performance-monitoring/monitor-performance/use-performance-graphs). |
| **Log level** |
Specifies how much logging is performed and the amount of information captured:
Nothing: No logging occurs.
Error: Only errors are logged.
Minimal: Only use minimal logging.
Basic: This is the default level.
Detailed: Give detailed logging output.
Debug: For debugging purposes, very detailed output.
Row Level (very detailed): Logging at a row level, which generates a lot of log data.
Debug and Row Level logging levels contain information you may consider too sensitive to be shown. Consider the sensitivity of your data when selecting these logging levels. See the Administer Pentaho Data Integration and Analytics guide for instructions on how best to use these logging methods.
|
| **Expand Remote Job** | Bundles all required files for a job, including its sub-components, so they can be sent to a remote server for execution. The remote server runs the complete job without needing to retrieve additional files from the original environment. |
| **Start job at** | Specifies the step where the job begins execution. By default, execution begins at the **Start** step. |
| **Parameters** | Applies parameter values pertaining to your job during runtime. A parameter is a local variable. For more details, see [Parameters](/pba/pipeline-designer/managing-transformations-and-jobs/run-modifiers#parameters). |
| **Variables** | Applies temporary values for user-defined and environment variables pertaining to your job during runtime. For more details, see [Variables](/pba/pipeline-designer/managing-transformations-and-jobs/run-modifiers#variables). |
| **Arguments** | Applies a named, user-supplied, single-value input given as a command line argument when running the job manually or with a script. Arguments are handled according to a job's design. If the job is not designed to handle arguments, nothing happens.Typically, argument values are numbers, words (strings), or system or script variables. Each job can have a maximum of 10 arguments.For more details, see [Arguments](/pba/pipeline-designer/managing-transformations-and-jobs/run-modifiers#arguments). |
# Analyze job results
You can see how your job performed and if errors occurred by viewing logs and job metrics. After you [Run a job](broken://pages/Bm8c4iLK23obaoLjpSgv), the Logs panel opens with tabs that help you to see how the transformation ran and pinpoint errors.
### Logging
The **Logging** tab displays the logging details for the most recent execution of the job. You can also drill in deeper to determine where errors occur. Error lines are highlighted in red.
### Job Metrics
The **Job Metrics** tab shows statistics for each step in your job including how many records were read, written, caused an error, processing speed (rows per second) and more. This tab also indicates whether an error occurred in your job step. If a mistake occurs, steps that caused the job to fail are highlighted in red.
# Job steps in Pipeline Designer
Steps extend and expand the functionality of jobs. You can use the following steps in Pipeline Designer.
| [Abort job](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558484/Abort+Job) | Utility | Abort the job. |
| ------------------------------------------------------------------------------------------------------------------------- | --------------- | ------------------------------------------------------------------------------------------- |
| [Checks if files exist](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/388311875/Checks+if+files+exists) | Conditions | Checks if files exist. |
| [Create a folder](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/372703482/Create+a+folder) | File management | Create a folder. |
| [Create file](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558469/Create+a+file) | File management | Create an empty file. |
| [Delete file](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558476/Delete+a+file) | File management | Delete a file. |
| [Delete Files](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/372703421/Delete+Files) | File management | Delete files. |
| [Delete folders](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/372703488/Delete+folders) | File management | Delete specified folders. If a the folder contains files, PDI will delete them all. |
| [File compare](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558478/File+compare) | File management | Compare two files. |
| [HTTP](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558468/HTTP) | File management | Get or upload a file using HTTP (Hypertext Transfer Protocol). |
| [Job](/pdia-data-integration/pdi-job-entries-reference-overview/job-job-entry) | General | Execute a job. |
| [Set variables](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/388311910/Set+variables+job+entry) | General | Set one or several variables. |
| [Shell](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558460/Shell) | Scripting | Execute a shell script. |
| [Start](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/370967988/Start) | General | Defines the starting point for job execution. Every job must have one (and only one) Start. |
| [Success](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/372703425/Success) | General | Clear any error state encountered in a job and forces it to a success state. |
| [Transformation](/pdia-data-integration/pdi-job-entries-reference-overview/transformation-job-entry-cp) | General | Run a transformation. |
| [Wait for](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558481/Wait+for) | Conditions | Wait for a delay. |
| [Wait For File](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/371558477/Wait+for+a+file) | File management | Wait for a file. |
| [Write To Log](https://pentaho-community.atlassian.net/wiki/spaces/EAI/pages/372703439/Write+to+log) | Utility | Write message to log. |
# Managing transformations and jobs
Managing transformations and jobs includes managing the files for them inside Pipeline Designer, managing their connections, and performing tasks that are common to both transformations and jobs.
* [Manage transformations and jobs](broken://pages/e6Ag0lB3iurK8xvWClAk)
Manage transformations and jobs on the Pipeline Designer's main page. Available actions include marking transformations and jobs as favorites, downloading them as files, creating duplicates, moving them to different folders or to the trash, renaming them, and viewing their details.
* [Manage connections for transformations and jobs](broken://pages/DCYxygIcvElMqkWqcQPm)
While creating or editing a transformation or job, you can edit database connections that are used by your transformations and jobs.
* [Edit a transformation or job](broken://pages/H2AkMmSZ5npWhak5xXPD)
Edit an existing transformation or job to rename it, add new steps, edit steps, configure hops, edit properties, reset it, or save it in a different location.
* [Use the SQL Editor](broken://pages/ejyOtYRZ7US6wvYulh8k)
The SQL Editor is good tool to use when you must execute standard SQL commands for tasks such as creating tables, dropping indexes, and modifying fields.
* [Use notes on the canvas](broken://pages/rIVe7wNWn1zibMSnzpVa)
While creating or editing a transformation or job, you can add notes to the canvas to help you and others understand the structure, design decisions, business rules, dependencies, and other aspects of your transformations and jobs.
* [Run modifiers](broken://pages/FQMJzYsQstZLoOf29V6V)
Run modifiers include parameters, variables, and arguments, which you can use to modify how you run transformations and jobs.
* [Export a transformation or job](broken://pages/ujyA942DAcweJFOn4cPn)
Export transformations and jobs to save backup copies, migrate them to other systems, share them with other users, or use them in other tools. You can export a transformation as a `.ktr` file or a job as a `.kjb` files.
* [Stop transformations and jobs](broken://pages/VPH09pmw6ch7eLyAm3EO)
There are two different methods you can use to stop a transformation or job running in the Pipeline Designer. The method you use depends on the processing requirements of your ETL task.
# Manage transformations and jobs
Manage transformations and jobs on the Pipeline Designer's main page. Available actions include marking transformations and jobs as favorites, downloading them as files, creating duplicates, moving them to different folders or to the trash, renaming them, and viewing their details.To manage a transformation or job, complete the following steps:
1. Log into the Pentaho User Console.
2. Open Pipeline Designer:
* If you are using the **Modern Design**, in the menu on the left side of the page, click **Pipeline Designer**.
* If you are using the **Classic Design**, click **Switch to the Modern Design,** and then in the menu on the left side of the page, click **Pipeline Designer**.
**Pipeline Designer** opens with the **Quick Access** section expanded.
3. In the table at the bottom of the screen, select either the **Recently opened** or **Favorites** tab.
4. Search for or browse to the transformation or job you want to manage and click the **More Actions** icon.
5. In the **More Actions** menu, select one of the following options:
| | |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Not favorite (click to add)** | Adds the transformation or job to the **Favorites** tab and changes the heart icon in the **Favorite** column from empty to red. |
| **Favorite (click to remove)** | Removes the transformation or job from the **Favorites** tab and changes the heart icon in the **Favorite** column from red to empty. |
| **Download** | Downloads the transformation or job as a file to your default download folder.Transformations are downloaded as KTR files. Jobs are downloaded as KJB files. |
| **Move** | Opens the **Move to** window. In the **Move to** window, you can navigate to a different folder or create a new folder and click **Move here** to move the transformation or job to that folder. |
| **Duplicate** | Opens the **Duplicate to** window. In the **Duplicate to** window, you can navigate to a different folder or create a new folder and click **Paste here** to create a duplicate of the transformation or job in that folder. |
| **Send to trash** | Opens the **Send to trash?** dialog box. You can click **Yes** to move the transformation or job into the trash. |
| **Info** | Opens either the **Transformation Name** or **Job Name** window, where you can view information about the transformation or job. You can also copy the file path for the transformation or job by clicking **Copy source**. |
| **Rename** | **Important**: After you rename a transformation or job, any dashboards, schedules, or favorites linked to it become inactive and stop functioning.Makes the transformation or job name editable in the table. After you enter a new name, you must click **OK** to rename the transformation or job. |
# Manage connections for transformations and jobs
When you create or edit a transformation or job in Pipeline Designer, you can connect to multiple databases. Pipeline Designer supports many database vendors, including MySQL and Oracle.
Pipeline Designer includes JDBC drivers for PostgreSQL, the default database.
Pentaho recommends avoiding ODBC connections. The ODBC-to-JDBC bridge is not exact. It also adds overhead. Use ODBC only when a JDBC driver is unavailable. For details, see [Why you should avoid ODBC](http://wiki.pentaho.com/pages/viewpage.action?pageId=14850644).
When you define a database connection, Pipeline Designer stores connection details in the Pentaho Repository. These details include username, password, and port. Other users can reuse the connection through the repository.
If you do not use the Pentaho Repository, Pipeline Designer stores connection details in the transformation or job XML.
Collect your connection details before you start. You typically need database type, host, port, username, and password. You can also set connection properties as variables.
Use ANSI SQL when possible. It improves portability across databases.
You must have a transformation or job open to manage connections. For help, see [Create a transformation](broken://pages/Bj0Q1QSvXmarGMgvgLUq), [Create a job](broken://pages/eI9zUxWpJJGBsGcLiYMc), or [Edit a transformation or job](broken://pages/H2AkMmSZ5npWhak5xXPD).
### In this topic
* [Define a new database connection](#define-a-new-database-connection)
* [Clear cached database metadata](#clear-cached-database-metadata)
* [Edit a database connection](#edit-a-database-connection)
* [Delete a database connection](#delete-a-database-connection)
* [Explore configured database connections](#explore-configured-database-connections)
* [Show dependencies](#show-dependencies)
### Define a new database connection
1. In **Connection Name**, enter a descriptive name.
The name can include spaces. Do not use special characters like `#`, `$`, or `%`.
2. In **Connection Type**, select the database type.
3. In **Access Type**, select **Native (JDBC)**.
4. In **Settings**, enter the connection values:
* **Host Name**: Server host name or IP address.
* **Database Name**: Database name. For ODBC, enter the DSN.
* **Port Number**: TCP/IP port, if not the default.
* **Username**: Optional username.
* **Password**: Optional password.
5. Select **Test Connection**.
6. Select **OK** to close the test dialog.
7. Select **Save** to save the connection.
#### OCI protocol information
Use these steps to create an Oracle OCI connection:
1. In **Connection Name**, enter a descriptive name.
2. In **Connection Type**, select **Oracle**.
3. In **Access**, select **OCI**.
4. In **Settings**, enter values as described in the [Oracle OCI documentation](http://docs.oracle.com/cd/B28359_01/java.111/b31224/instclnt.htm):
* **SID**: Oracle system ID.
* **Tablespace for Data**: Tablespace for data.
* **Tablespace for Indices**: Tablespace for indexes.
* **User Name**: Database username.
* **Password**: Database password.
5. Select **Test Connection**.
6. Select **OK** to close the test dialog.
7. Select **OK** to close the Database Connection dialog.
If you want to use **Advanced**, **Options**, or **Pooling**, see the [Oracle OCI documentation](http://docs.oracle.com/cd/B28359_01/java.111/b31224/instclnt.htm).
#### Connect to Snowflake using strong authentication
Snowflake key pair authentication uses a private key file instead of a password.
1. After you enter the connection values on the **General** tab, select **Options**.
2. Add these parameters:
* `authenticator`: `snowflake_jwt`
* `private_key_file`: Path to your private key file. Example: `/rsa_key.p8`
* `private_key_file_pwd`: Password for the private key file. Example: `PentahoSnowFlake123`
See the Snowflake JDBC docs: .
3. Select **Test Connection**.
4. Select **OK** to close the test dialog.
5. Select **OK** to close the Database Connection dialog.
#### Connect to an Azure SQL database
You can use an Azure SQL database as a data source in Pipeline Designer.
This connection is required for the **Bulk load into Azure SQL DB** job entry.
Pipeline Designer supports:
* Always Encrypted
* Dynamic masking
* Multiple authentication methods
Azure SQL differs from on-premises SQL Server. See Microsoft documentation: .
**Before you begin**
* Ensure you have an Azure subscription and an Azure SQL database instance.
* Install the Azure SQL database drivers.
* Get these values from your administrator:
* Host name
* Database name
* Port number
* Authentication method
* Username
* Password
* If you use **Always Encryption Enabled**, also get:
* Client ID
* Client Secret Key
**Authentication method**
Pipeline Designer supports four Azure SQL authentication methods:
* **SQL Authentication**
* **Azure Active Directory**
* **Azure Active Directory with password**
* **Azure Active Directory with integrated authentication**
**Connect to an Azure SQL database**
1. In **Connection Name**, enter a descriptive name.
2. In **Connection Type**, select **Azure SQL DB**.
3. In **Access**, select **Native (JDBC)**.
4. Enter your Azure SQL connection values:
* **Host Name**: Azure SQL server instance name.
* **Database Name**: Azure SQL database name.
* **Port Number**: TCP port. Azure SQL uses `1433`.
* **Authentication method**: Default is SQL Authentication.
* **Username**
* **Password**
* **Always Encryption Enabled** (optional)
* **Client id** (if Always Encryption Enabled)
* **Client Secret Key** (if Always Encryption Enabled)
5. Select **Test Connection**.
**Use the Always Encryption Enabled option**
Before you enable **Always Encryption Enabled**, complete these steps using Azure tools:
1. Generate a column master key in Azure Key Vault.
2. Encrypt the column using the column master key.
3. Register the app in Azure Active Directory. Collect **Client id** and **Client Secret Key**.
4. Grant the app permissions to access Azure Key Vault.
5. Enable **Always Encryption Enabled**. Enter **Client id** and **Client Secret Key**.
### Clear cached database metadata
When you work with complex transformations or jobs, Pipeline Designer can cache outdated metadata. Use **Clear Complete DB Cache** to refresh it.
Cached metadata can include:
* Table structures
* Column types
* Indexes
* Primary and foreign keys
* Other schema metadata
{% hint style="info" %}
Clearing cached database metadata does not delete database data. It also does not change transformation or job files.
{% endhint %}
1. Open a transformation or job.
2. On the left, select **View** to open the **View** pane.
3. Under **Database Connections**, select **More Actions**, then select **Clear Complete DB Cache**.
### Edit a database connection
Open the connection editor, then review these tabs and options.
#### Advanced options
| Option | Description |
| ------------------------------------------------- | ------------------------------------------------------------ |
| Supports the Boolean data type | Uses the database native Boolean data type. |
| Supports the timestamp data type | Uses the database native timestamp data type. |
| Quote all in database | Uses quoted identifiers. This can make names case-sensitive. |
| Force all to lower-case | Converts identifiers to lowercase. |
| Force all to upper-case | Converts identifiers to uppercase. |
| Preserve case of reserved words | Uses the database reserved word list. |
| The Preferred Schema name where no schema is used | Sets a default schema name, such as `MYSCHEMA`. |
| SQL Code Editor | SQL statements to run immediately after connecting. |
#### Options tab
Use the **Options** tab to add or delete driver parameters.
* To add a parameter, select **Add Row**.
* To delete a parameter row, select the **Delete** icon.
#### Pooling tab
Use **Pooling** to configure a connection pool. Pooling can reduce connection overhead. It can also help when database licensing limits concurrent connections.
* To add a pool parameter, select **Add Row**, then enter **Parameter** and **Value**.
* To delete a pool parameter, select the **Delete** icon.
* To control pagination, select a new **Items per page** value.
Common pooling parameters include `validationQuery`.
* For Oracle and PostgreSQL, use `Select 1 from dual`.
* For Microsoft SQL Server and MySQL, use `Select 1`.
Typical pooling options include:
* **Enable Connection Pooling**
* **Pool Size (Initial, Maximum)**
* **Parameters**
* **Description**
#### Clustering tab
Use **Clustering** to cluster a database connection and create connections to data partitions.
To create a partition connection, enter:
* Partition ID
* Host Name
* Port
* Database Name
* User Name
* Password
If you use the Data Source Wizard (DSW) in a clustered Pentaho Server setup, disable DSW data source caching. This helps ensure new data sources appear on all nodes.
To disable DSW caching, set `enableDomainIdCache=false` in `server/pentaho-server/pentaho-solutions/system/system.properties`.
### Delete a database connection
Delete a database connection that you no longer need.
{% hint style="danger" %}
Deleting a connection can affect reports, charts, dashboards, and other content that uses the connection.
{% endhint %}
1. Open a transformation or job.
2. On the left, select **View** to open the **View** pane.
3. Expand **Database Connections**.
4. Find the connection, select **More Actions**, then select **Delete**.
5. In **Confirm deletion**, select **Yes**.
### Explore configured database connections
Use the Database Explorer to inspect configured connections. You can browse tables, views, synonyms, catalogs, and schemas.
1. Open a transformation or job.
2. On the left, select **View** to open the **View** pane.
3. Expand **Database Connections**.
4. Find the connection, select **More Actions**, then select **Explore**.
5. (Optional) Select **Refresh** to reload the list.
6. Browse to the object you want, then select **Actions**:
* **Preview first 100**: Shows the first 100 rows.
* **Preview x Rows**: Prompts for the number of rows.
* **Row Count**: Shows the total row count.
* **Show Layout**: Shows columns and data types.
* **DDL**: Generates DDL for the selected object.
* **View SQL**: Opens the Simple SQL Editor for the selected table.
* **Truncate Table**: Generates a `TRUNCATE TABLE` statement. The statement is commented out by default.
* **Data Profile**: Shows basic profiling information.
7. Select **OK** to close the Database Explorer.
### Show dependencies
Show all platform dependencies for a connection, including transformations and jobs.
1. Open a transformation or job.
2. On the left, select **View** to open the **View** pane.
3. Expand **Database Connections**.
4. Find the connection, select **More Actions**, then select **Show dependencies**.
### Related tasks
* [Use the SQL Editor](broken://pages/ejyOtYRZ7US6wvYulh8k)
# Edit a transformation or job
Edit an existing transformation or job to rename it, add new steps, edit steps, configure hops, edit properties, reset it, or save it in a different location.
To edit a transformation or job, complete the following steps:
1. Log into the Pentaho User Console.
2. Open **Pipeline Designer**:
* If you are using the **Modern Design**, in the menu on the left side of the page, click **Pipeline Designer**.
* If you are using the **Classic Design**, click **Switch to the Modern Design,** and then in the menu on the left side of the page, click **Pipeline Designer**.
**Pipeline Designer** opens with the **Quick Access** section expanded.
3. In the table at the bottom of the screen, select either the **Recently opened** or **Favorites** tab.
4. Open a transformation or job:
1. Search for or browse to the transformation or job, and then click **Open**.
2. Click **Open files**, and then in the **Select File or Directory** dialog box, search for or browse to the transformation or job and click **Open**.
5. Edit the transformation or job by taking one or more of the following actions:
* **Rename a transformation or job**
1. In the transformation or job tab, click the name to make it editable.
2. Type a new name and press **Enter**.
**Note:** You can also change the name of a transformation or job in the properties window. For details see [Configure transformation properties](broken://pages/6e17OnYYsjLTPYnuFNSA) or [Configure job properties](broken://pages/qQx1eENOpzh3YwPbl0Eu).
* **Add a new step**
1. In the **Design** pane, search for or browse to a step you want to use in the transformation. You may need to expand sections in the **Design** pane to find steps.
2. Drag the step you want to add onto the canvas.
* **Edit steps**
Hover over a step to open the step menu, and then select one of the following options for either a transformation or job:
Step option
Description
Delete
Deletes the step from the canvas.
Edit
Opens the Step Name window where you can configure the properties of the step. Step properties may appear in multiple sections, tabs, or both.
Note: To learn more about the step you're configuring, in the lower-left corner of the Step Name window, click Help.
Duplicate
Adds a copy of the step to the canvas.
For transformations only, you can select one of the following additional options:
Step option
Description
More Actions > Change Number of Copies
Opens the Number of copies dialog box, where you can enter a number or a variable to specify how many copies of the step are processed in parallel when the transformation or job is run. To find a variable, in the Number of copies (1 or higher) box, click the Select variable to insert icon.
More Actions > Data Movement
Opens a list of data movement options for you to select from to specify how data rows are distributed to the next steps of the transformation or job. Round-Robin is the default setting.
Round-Robin: Distributes rows evenly across all parallel step copies using round-robin logic. This setting optimizes load balancing when the transformation includes multiple instances of the next step.
Load Balance: Routes rows to the step copy with the lightest processing load. This setting can improve performance when processing times vary across parallel step instances.
Copy Data to Next Steps: Sends each row to all parallel step copies. Use this setting when every downstream branch must process the complete dataset independently.
* **Configure hops**
* To add hops between steps, hover over a step’s handle until a plus sign (+) appears, then drag the connection to the handle of another step.
* Disable a hop by selecting it and clicking the **Disable** icon.
* Delete the hop by selecting it and clicking the **Delete** icon.
* **Edit transformation or job properties**
* In the **Canvas Action** toolbar, click the **Settings** icon. The **Transformation Properties** or **Job Properties** window opens.
* Change properties in one or more tabs. For details, see [Transformation properties](#transformation-properties) or [Job properties](#job-properties).
* **Reset the transformation or job**
**Note:** Resetting a transformation or job clears all steps and connections from the transformation or job and cannot be undone.
1. In the **Canvas Action** toolbar, click the **Reset** icon. The **Confirm Reset** window opens.
2. Click **Reset**. All steps and connections are removed from the transformation or job.
* **Save transformation or job in a different location**
1. Click **Save as**. The **Select File or Directory** dialog box opens.
2. Search for or browse to the folder in the repository where you want to save the transformation or job.
3. (Optional) To create a new folder in the repository, click the **New Folder** icon, and then in the **New folder** dialog box, enter a **New folder name** and click **Save**.
4. (Optional) To delete a folder from the repository, select the folder and click the **Delete** icon.
5. In the **Select File or Directory** dialog box, click **Save**. The **Save Change** dialog box opens.
6. Click **Yes** to confirm that you want to save the transformation or job.
# Use the SQL Editor
The SQL Editor is good tool to use when you must execute standard SQL commands for tasks such as creating tables, dropping indexes, and modifying fields. The SQL Editor is used to preview and execute DDL (Data Definition Language) generated by the Pipeline Designer such as create and alter table, create index, and create sequence SQL commands. For example, if you add a Table Output step to a transformation and click the **SQL** button at the bottom of the **Table Input** dialog box, Pipeline Designer automatically generates the necessary DDL for the output step to function properly and presents it to the end user through the SQL Editor.
Below are some points to consider:
* Multiple SQL Statements must be separated by semi-colons.
* Before SQL Statements are sent to the database to be executed, Pipeline Designer removes returns, line-feeds, and separating semi-colons.
* Pentaho Data Integration clears the database cache for the database connection on which you launch DDL statements.
The SQL Editor does not recognize the dialects of all supported databases. That means that creating stored procedures, triggers, and other database-specific objects may pose problems. Consider using the tools that came with the database in these instances.
# Use notes on canvas
While creating or editing a transformation or job, you can add notes to the canvas to help you and others understand the structure, design decisions, business rules, dependencies, and other aspects of your transformations and jobs.
For instructions on creating and editing transformations and jobs, see [Create a job](broken://pages/eI9zUxWpJJGBsGcLiYMc), [Create a transformation](broken://pages/Bj0Q1QSvXmarGMgvgLUq), and [Edit a transformation or job](broken://pages/H2AkMmSZ5npWhak5xXPD).
With a transformation or job open, take one or more of the following actions.
* [Add a note](#add-a-note)
* [Edit a note](#edit-a-note)
* [Reposition a note](#reposition-a-note)
* [Delete a note](#delete-a-note)
## Add a note
1. In the canvas toolbar, click the **Add Note** icon. The **Notes** dialog box opens.
2. In the **Note** box, enter the content for the note.
3. To format the note, click **Style**, and then enter the font, color, and shadow options you want to use for the note.
4. Click **Save**. The note appears on the canvas.
## Edit a note
1. Hover over the note on the canvas. The Delete and Edit icons are displayed above the note.
2. Click the Edit icon. The Notes dialog box opens.
3. In the **Note** box, edit the content of the note.
4. To edit the format of the note, click **Style**, and then edit the font, color, and shadow options for the note.
5. Click **Save**. The edited note is saved.
## Reposition a note
To reposition a note, click and drag the note on the canvas to the location where you want it to appear.
## Delete a note
1. Hover over the note on the canvas. The Delete and Edit icons are displayed above the note.
2. Click the **Delete** icon. The note is deleted.
# Run modifiers
* `${VARIABLE}`
* `%%VARIABLE%%`
Both formats can be used and even mixed. In fact, you can create variable recursion by alternating between the Unix and Windows syntax. For example, if you wanted to resolve a variable that depends on another variable, then you could use this example: `${%%inner_var%%}`.
**Note:** If there is a name collision with a parameter or argument, variables will defer.
You can also use ASCII or hexadecimal character codes in place of variables, using the same format: `$[hex value]`. This makes it possible to escape the variable syntax in instances where you need to put variable-like text into a variable. For instance if you wanted to use `${foobar}` in your data stream, then you can escape it like this: `$[24]{foobar}`. Pipeline Designer will replace `$[24]` with a `$` without resolving it as a variable.
### Environment variables
This is the traditional variable type in Pipeline Designer. You define an environment variable through the Set Environment Variables dialog box in the **Edit** menu, or by hand by passing it as an option to the Java Virtual Machine (JVM) with the -D flag.
Environment variables are an easy way to specify the location of temporary files in a platform-independent way; for example, the\*\*${java.io.tmpdir}\*\* variable points to the `/tmp/` directory on Unix/Linux/OS X and to the `C:\Documents and Settings\
Item
Description
1
Use the Projects list to switch between recently opened projects. You can also select <No Project> to close the currently open project.
2
Use Project Files in the View tab to manage every file, folder, and subfolder for a project. You can create, rename, delete, or move files and folders, directly from this list.
3
Use Configurations in the View tab to view and manage all of the configurations for the project, such as database connections, VFS connections, and run configurations. Notice that configurations appear shaded when another configuration with the same name and higher priority is active.
### Benefits of projects
When you use projects to manage your ETL workflows, you gain the following benefits and capabilities:
#### Manageability
* **Improved organization:** Consolidate files and configurations for the ETL workflow into a single logical container (the project folder) and use subfolders to organize your workflow.
* **Scalability:** Use multiple projects to segment and modularize integration work, making large ETL workflows easier to manage.
* **Visibility:** In the **View** tab, see **Project Files** and **Configurations** for better visibility and improved management of your project files, connections, schemas, and run configurations.
* **Quick switching:** Switch between all your open projects while you continue to work in the same location or between up to 10 recently opened projects when you change locations in the PDI client.
* **Portability:** Move a project folder (including all files and configurations) without losing the context for the project. You can transfer the project between your local system, a Pentaho Repository, or a VFS.
#### Flexible collaboration
* **Ease of sharing:** Share a project folder with other users and teams that need to review, contribute, or deploy the project. You can share a project folder as a ZIP file, upload it to a Pentaho Repository, save it in a shared VFS, or check it into a version control system.
* **Version control:** Commit the project folder to version control so you can track changes and preserve shared history for the project.
#### Configuration management and security
* **Isolated configurations:** Reduce conflicts by isolating ETL workflows in separate projects with their own unique configurations, connection parameters, and transformation strategies.
* **Enhanced security:** Isolate configurations at the project level to help protect sensitive information (for example, database credentials).
* **Configuration prioritization:** Define configurations at the appropriate level. When a configuration exists in multiple places, the PDI client uses the highest-priority version.
* **Backward compatibility:** Continue working with existing ETL workflow configurations until you are ready to migrate to projects. Projects are an optional feature. You can even execute transformations and jobs with a project context even if the target file lives outside the project.
### Configuration levels
Configurations are automatically applied in priority order to ensure the correct settings are used and to prevent unexpected conflicts. When a configuration exists in multiple locations, the PDI client applies the highest-priority version, reducing duplication and maintaining consistency for shared settings.
When configurations share the same name, the configuration at the highest level is applied. Configurations include connections, schemas, run configurations, slave servers, variables, and parameters.
Configurations are applied in the following priority order, from highest to lowest:
1. **Default:** A default configuration that cannot be edited or removed. **Example:** The **Pentaho local** run configuration.
2. **Project:** A configuration defined in the project that applies only to that project. Project-level configurations are only shown if you have an open project.
3. **System:** A configuration defined for projects in your local system. System-level configurations are not accessible to projects in a repository.
4. **Repository:** A configuration defined for projects in the connected repository. Repository-level configurations are not accessible to projects in your local system.
5. **(Deprecated) File:** File-level configurations have the lowest priority and can no longer be created. File-level configurations can only be edited or deleted. If you move a file with a file-level configuration into the Pentaho Repository, and the repository contains a configuration with the same name, a warning appears to indicate that the file configuration will overwrite the repository configuration.
## Creating projects
You can create a project folder in any of the following locations:
* **Local file system:** A folder on your computer.
* **Pentaho Repository:** A folder in a connected Pentaho Repository.
* **VFS:** A folder in a supported VFS\* (for example, cloud storage) that the PDI client can access.
Note: Using folders in a VFS might affect performance.
### Create a project
Create a new project when you want to work with all the files and settings for an ETL workflow in a self-contained project folder that you can port to other locations and share with other users.
To create a new project, complete the following steps:
1. Open the PDI client.
2. Click **Project** > **New Project**. The **New project** dialog box opens.
3. Enter a **Name** for the project.
4. To navigate to the **Project Path**, click **Browse**. The **Select a Folder** dialog box opens.
5. Select a folder for your project by taking one of the following actions:
1. For an existing ETL workflow that you do not need to move, navigate to the root directory that contains every item in your workflow and select it.
Note: If you have an existing ETL workflow that you want to move to a different location before selecting it as a project, see Move a project.
2. To create a new project for a new ETL workflow, navigate to the folder where you want to create the project, click the **Add folder** icon, and in the **Folder Name** dialog box enter a **New Folder Name** and click **OK**.
6. Click **Open**.
Notes:
A project folder can contain only one project.
It is a recommended best practice to use the same name for the project and project folder.
7. In the **New project** dialog box, enter a **Description** for the project.
8. Click **OK**. The project is created and opened in the PDI client.
### Create folders in a project
Create folders within your project to organize transformations, jobs, configurations, and related content, making it easier to manage the project as it becomes more complex.
Create a folder in your project by completing the following steps:
1. Open the PDI client.
2. Click **Project** > **Open Project**. The **Select a Folder** window opens.
3. Search or browse to the project folder and then click **Open**. The project opens.
4. Click the **View** tab.
5. Expand **Project Files** and then navigate to the existing folder in which you want to create your new folder.
6. Right-click the existing folder and select **New Folder**.
Note: You can create a new folder in the project’s root folder by right-clicking Project Files.
7. In the **Folder Name** dialog box, enter a folder name.
8. Click **OK**. The folder is created.
{% hint style="info" %}
**Note:** If your project is saved in a repository, you can also use the Repository Explorer to create a folder inside the project folder. For instructions, see [Use the Repository Explorer](/pdia-data-integration/redirects/use-the-repository-explorer).
{% endhint %}
### Share a project
Use one of the following options to share your project with other users:
* Create the project in the Pentaho Repository or a shared VFS. See [Create projects](#create-projects).
* Move an existing project into a Pentaho Repository or shared VFS. See [Move a project](#move-a-project).
* Check the project folder into your organization’s version control system.
* Outside of the PDI client, save the project folder as a ZIP file and share it directly with another user. Projects shared as ZIP files can be opened in the recipient's chosen location. After the recipient saves the project to their chosen location, they can open it and then refresh the project by clicking **Project** > **Refresh Project**.
## Editing projects
You can edit a project's name or description, rename or delete folders in the project folder, or move the entire project folder.
### Edit project details
Edit a project details by completing the following steps:
1. Open the PDI client.
2. Click **Project** > **Open Project**. The **Select a Folder** window opens.
3. Search or browse to the project folder and then click **Open**. The project opens.
Note: You can have only one project open at a time.
4. Click **Project** > **Project Details**. The **Project Details** window opens.
5. Update one or more of the following options:
* **Name**
* **Description**
Note: You cannot edit the Project Path. To change the project path, move the project folder and then open the project from the new location.
6. Click **OK**. The details for the project are updated.
### Edit a folder in a project
You can edit a folder in a project to change its name or delete it.
Edit a folder in a project by completing the following steps:
1. Open the PDI client.
2. Click **Project** > **Open Project**. The **Select a Folder** window opens.
3. Search or browse to the project folder and then click **Open**. The project opens.
Note: You can have only one project open at a time.
4. Edit the folder by taking one of the following actions:
* **Rename a folder**
1. Click the **View** tab.
2. Expand **Project Files** and then navigate to the folder you want to rename.
3. Right-click the folder and select **Rename**.
4. In the **Rename** dialog box, enter a new name and click **OK**.
* **Delete a folder**
1. Click the **View** tab.
2. Expand **Project Files** and then navigate to the folder you want to delete.
3. Right-click the folder and select **Delete**.
4. Click **Yes** to confirm the deletion.
### Move a project
{% hint style="warning" %}
**Important:**
* If you move a project, you must move the entire project and its contents.
* If you move individual files out of a project, such as transformation (.ktr) or job (.kjb) files, any references to other project items, such as database connections, will break.
* Before moving a project, close it in the PDI client.
{% endhint %}
Projects can be moved between any of the following locations:
* **Local file system:** A folder on your computer.
* **Pentaho Repository:** A folder in a connected Pentaho Repository.
* **VFS:** A folder in a supported VFS (for example, cloud storage) that the PDI client can access.
Note: Using folders in a VFS might affect performance.
* To move a project folder in your local system, use your operating system’s file explorer or command line.
* To move a project folder into or out of the Pentaho Repository, see one of the following topics: [Use the Repository Explorer](/pdia-data-integration/redirects/use-the-repository-explorer) or [Upload and download from the Pentaho Repository](/pdia-admin/manage-the-pentaho-system/manage-the-pentaho-repository/upload-and-download-from-the-pentaho-repository).
* To move a project into or out of a VFS, use your VFS management tool or file system interface.
After you have moved a project to its new location, open the project and [refresh it](#refresh-project-data).
## Working with projects
You can switch between projects, refresh project data when it is modified outside of the PDI client, close a project, or migrate your existing ETL workflows into a project.
### Switch between projects
You can quickly switch between open or recently opened projects in the PDI client.
To switch projects, above the **View** and **Design** tabs, click the **Projects** list and select the project that you want to switch to.
#### How the Projects list works
* If you work in the same location, on your local system or in a connected repository, you can switch between all open projects.
* If you restart the PDI client or connect to or disconnect from the Pentaho Repository, the 10 most recently opened projects remain in the **Projects** list for each location. The PDI client keeps separate lists of the 10 most recently opened projects for the local system and each user in the Pentaho Repository.
* If a project is open, selecting another project closes the current project and opens the new one. Selecting **\** closes the current project.
* If projects have the same name in the **Projects** list, each name shows the full path to help you tell them apart.
* If a project is moved, deleted, or on an inaccessible VFS, it does not appear in the **Projects** list.
* If a project is not in the **Projects** list, you must re-open it. See [Open a project](#open-a-project).
### Refresh project data
Refresh an open project when files are added, removed, or modified outside of the PDI client.
To refresh a project, click **Project** > **Refresh Project**.
To refresh configurations that were edited outside of the PDI client, click the **Refresh** icon.
### Close a project
With a project open, click **Project** > **Close Project**.
A warning is displayed if any open files in the project have unsaved changes.
As long as a project remains in the **Projects** list, its files are automatically restored the next time you open it.
## Project variables
Project variables are a way to store small pieces of information dynamically in a narrower scope than Kettle or environment variables. Project variables are loaded either when a project is opened or when a transformation or job in the project is opened in the PDI client or run in a scheduled task. Use project variables to define variables that apply only to the specific project.
{% hint style="warning" %}
**Important:** When variables are defined in multiple places, the last one applied overrides the others.
{% endhint %}
Variables are applied in the following order:
1. Environment variables (JVM `-D` arguments from the JVM process)
2. Variables defined in `kettle.properties`
3. Variables defined in `project.properties`
4. Variables and parameters defined directly in the PDI client or in a scheduled job.
5. Variables set at runtime by steps such as `Set Variables` and `Set Session Variables`.
{% hint style="info" %}
**Note:** Internal variables are also available for transformations and jobs within a project. For details, see [Internal Variables](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/variables/internal-variables).
{% endhint %}
### Set project variables on your local system
To edit project variables for projects saved on your local system, complete the following steps:
1. Open the `project.properties` file in a text editor. By default, the `project.properties` file is stored in the project folder, inside the `.config` folder.
2. Edit the file and save it.
3. (Optional) If the project you set the variables for is open in the PDI client, you must close the project and reopen it for the new variable values to be applied.
### Set project variables in the Pentaho Repository
To edit project variables for projects saved in a Pentaho Repository:
1. Download the `project.properties` file from the project folder in the repository.
2. Open the `project.properties` file in a text editor.
3. Edit the file and save it.
4. Upload the edited `project.properties` file to the repository folder you downloaded it from.
{% hint style="info" %}
**Note:** The `project.properties` file uses the same format as the `kettle.properties` file. For details about `kettle.properties`, see [Kettle Variables](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/variables/kettle-variables).
{% endhint %}
### Project run modifier
You can use `-project` argument when executing a transformation or job to specify the path for a project. The path can be to a folder in the Pentaho Repository, in a VFS, or on your local system.
# Extracting data into PDI
Connect Pentaho Data Integration (PDI) to databases, file systems, clusters, and other data sources, and configure advanced options for integration.
* [Defining PDI database connections](/pdia-data-integration/extracting-data-into-pdi/defining-pdi-database-connections)
You can use Pentaho Data Integration (PDI) to access data from various databases. You\
must connect to the database before accessing its records. You define database connections\
in PDI through the Database Connection dialog box.
* [Edit database connections in PDI](/pdia-data-integration/extracting-data-into-pdi/edit-database-connections-in-pdi)
Once a connection has been established, you can open the Database Connection dialog box to refine and change aspects of the connection.
* [Specify advanced configuration of PDI database connections](/pdia-data-integration/extracting-data-into-pdi/specify-advanced-configuration-of-pdi-database-connections)
Use the **Advanced** option in the Database Connection dialog box to configure properties associated with how SQL is generated. With these properties, you can set a standard across all your SQL tools, ETL tools, and design tools.
* [Quoting PDI database connections](/pdia-data-integration/extracting-data-into-pdi/quoting-pdi-database-connections)
Pentaho uses a database-specific quoting system. With this system, you can use any name or character that complies with the supported databases' naming conventions.
* [Set specific options for PDI database connections](/pdia-data-integration/extracting-data-into-pdi/set-specific-options-for-pdi-database-connections)
Use the **Advanced** option in the Database Connection dialog box to configure properties associated with how SQL is generated. With these properties, you can set a standard across all your SQL tools, ETL tools, and design tools.
* [Define PDI database connection pooling](/pdia-data-integration/extracting-data-into-pdi/define-pdi-database-connection-pooling)
You can use the **Pooling** option in the Database Connection dialog box to set up a connection pool and define options like the initial pool size, maximum pool size, and connection pool parameters. By default, a connection remains open for each individual report or set of reports in PUC and for each individual step in a transformation in PDI.
* [Connect to clusters (PDI only)](/pdia-data-integration/extracting-data-into-pdi/connect-to-clusters-pdi-only)
Use the **Clustering** options in the Database Connection dialog box to cluster the database connection and create connections to data partitions in PDI.
* [Modify connections from PDI](/pdia-data-integration/extracting-data-into-pdi/modify-connections-from-pdi)
Access other database-related connection tasks in PDI by right-clicking on the connection name in the **View** tab of the **Explorer** pane.
* [PDI and Hitachi Content Platform (HCP)](/pdia-data-integration/extracting-data-into-pdi/pdi-and-hitachi-content-platform-hcp)
Pentaho Content Platform (HCP) is the distributed, fixed-content, data storage system from Pentaho. HCP provides a scalable, easy-to-use repository that can accommodate all types of data, from simple text files to medical images to multigigabyte database images.
* [Hierarchical data](/pdia-data-integration/extracting-data-into-pdi/hierarchical-data)
Pentaho supports a hierarchical data type (HDT) by means of the Pentaho EE Marketplace hierarchical data type plugin that adds the data type and creates five steps. These steps are designed to simplify string manipulation, with the ability to convert between HDT fields and formatted strings.
* [PDI and Snowflake](/pdia-data-integration/extracting-data-into-pdi/pdi-and-snowflake-cp)
Snowflake is an analytic data warehouse running completely on a cloud infrastructure. Snowflake supports loading popular data formats like JSON, Avro, Parquet, ORC, and XML. Using Pentaho Data Integration (PDI), you can load your data into Snowflake and define jobs in PDI to efficiently orchestrate warehouse operations, paying only for the storage and computing resources actually used when you use them.
* [Copybook steps in PDI](/pdia-data-integration/extracting-data-into-pdi/copybook-steps-in-pdi-cp)
Pentaho Data Integration supports simplified integration with fixed-length records in binary mainframe data files, so more users can ingest, integrate, and blend mainframe data as part of their data integration pipelines. This capability is critical if your business relies on massive amounts of customer and transactional datasets generated in mainframes that you want to search and query to create reports.
* [Work with the Streamlined Data Refinery](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery)
The Streamlined Data Refinery (SDR) is a simplified and specific ETL refinery composed of a series of Pentaho Data Integration (PDI) jobs that take raw data, augment and blend it through the request form, and then publish it for report designers to use in Analyzer.
* [Connecting to a Hadoop cluster with the PDI client](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article)
To connect to a Hadoop cluster, you must access a driver, create a named connection, then configure and test the connection.
* [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser)
You can connect to most Virtual File Systems (VFS) through VFS connections in PDI. A VFS connection is a stored set of VFS properties that you can use to connect to a specific file system.
* [Streaming analytics](/pdia-data-integration/extracting-data-into-pdi/streaming-analytics)
With streaming analytics, you can constantly perform statistical analysis while moving within a data stream.
* [Web services steps](/pdia-data-integration/extracting-data-into-pdi/web-services-steps)
PDI jobs and transformations can interact with a variety of Web services through specialized steps. How you use these steps, and which ones you use, is largely determined by your definition of Web services.
# Defining PDI database connections
You can use Pentaho Data Integration (PDI) to access data from various databases. You\
must connect to the database before accessing its records. You define database connections\
in PDI through the Database Connection dialog box.
Before you can create a connection, the appropriate driver must be installed for your\
particular data connection. Your IT administrator should be able to install the appropriate\
driver for you.
## Open the Database Connection dialog box from PDI
Perform the following steps to open a new database connection in PDI:
1. Start the PDI client (Spoon) and create a new transformation or job.
2. In the **View** tab of the **Explorer** pane, double-click on the **Database connections** folder.
The Database Connection dialog box appears, as shown below:
3. Enter your data connection information and test.
See [Enter database connection information](broken://pages/slgCGouGbxLQL2mFpWJz) for further details.
In PDI, you can define connections to multiple databases provided by multiple database vendors such as MySQL and Oracle. PDI ships with the most suitable JDBC drivers for PostgreSQL, our default database.
Pentaho recommends avoiding ODBC connections. The ODBC to JDBC bridge driver does not always provide an exact match and adds another level of complexity, which affects performance. The only time you may have to use ODBC is if no JDBC driver is available. For details, see the [Pentaho Community article on why you should avoid ODBC](http://wiki.pentaho.com/pages/viewpage.action?pageId=14850644).
When you define a database connection in PDI, the connection information (such as the user name, password, and port number) is stored in the Pentaho Repository and is available to other users when they connect to the repository. If you are not using the Pentaho Repository, the database connection information is stored in the XML file associated with your transformation or job. See the **Pentaho Data Integration** document for details on the Pentaho Repository.
You must have information about your database (such as your database type, port number, user name and password) before you define a JDBC connection. In PDI, you can also set connection properties as variables. Through such variables, your transformations and jobs can access data from multiple database types.
Make sure to use clean ANSI SQL that works on all the database types used.
## Enter database connection information
The type of database connection information entered depends on your access protocol. The following sections show examples for Native (JDBC) and OCI protocols:
### Native (JDBC) protocol information
Perform the following steps to create a Native (JDBC) connection in the Database Connection dialog box:
1. In the **Connection Name** field, enter a name that uniquely describes this connection.
The name can have spaces, but it cannot have special characters (such as #, $, and %).
2. In the **Database Type** list, select the database you want to use (for example, MySQL or Oracle).
3. In the **Access** list, select **Native (JDBC)**. The access protocol which appears depends on the database type you select.
4. In the **Settings** section, enter the following information:
| Field | Description |
| ------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| Host Name | The name of the server that hosts the database to which you are connecting. Alternatively, you can specify the host by IP address. |
| Database Name | The name of the database to which you are connecting. If you are using a ODBC connection, enter the Data Source Name (DSN) in this field. |
| Port Number | The TCP/IP port number (if it is different from the default) |
| User Name | Optional user name used to connect to the database |
| Password | Optional password used to connect to the database |
5. Click **Test**.
A success message appears if the connection is established.
6. Click **OK** to close the connection test dialog box.
7. To save the connection, click **OK** to close the Database Connection dialog box.
* In PUC, your connection name appears in the list of available data sources in the Manage Data Sources dialog box.
* In PDI, your connection name appears under the **Database connections** folder in the **View** tab.
### OCI protocol information (PDI only)
Perform the following steps to create an OCI connection in the PDI Database Connection dialog box:
1. In the **Connection Name** field, enter a name that uniquely describes this connection.
The name can have spaces, but it cannot have special characters (such as #, $, and %).
2. In the **Database Type** list, select **Oracle**.
3. In the **Access** list, select **OCI**. The access protocol which appears depends on the database type you select.
4. In the **Settings** section, enter the following information as directed by the [Oracle OCI documentation](http://docs.oracle.com/cd/B28359_01/java.111/b31224/instclnt.htm).
| Field | Description |
| ---------------------- | ------------------------------------------------------------------------ |
| SID | The Oracle system ID that uniquely identifies the database on the system |
| Tablespace for Data | The name of the tablespace where the data is stored |
| Tablespace for Indices | The name of the tablespace where the indices is stored |
| User Name | The user name used to connect to the database |
| Password | The password used to connect to the database |
5. Click **Test**.
A success message appears if the connection is established.
6. Click **OK** to close the connection test dialog box.
7. To save the connection, click **OK** to close the Database Connection dialog box.
If you want to use **Advanced**, **Options**, or **Pooling** for your OCI connection, refer to the [Oracle OCI documentation](http://docs.oracle.com/cd/B28359_01/java.111/b31224/instclnt.htm) to understand how to specify these settings.
### Connect to Snowflake using strong authentication
If you are defining a data connection to Pentaho Data Integration and Analytics from a Snowflake data warehouse in the cloud, you can improve connection security by applying strong authentication.
You can apply strong authentication to your defined Pentaho data connection from Snowflake through a key pair. Perform the following steps to configure key pair strong authentication for your Snowflake data connection:
1. After [entering the information for your Snowflake data connection](broken://pages/slgCGouGbxLQL2mFpWJz) in the **General** tab of the Database Connection dialog box, select the **Options** tab.
2. Set the key pair parameters as indicated in the following table:
| Parameter | Value |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| `authenticator` | `snowflake_jwt` |
| `private_key_file` | Specify the name of the private key file you use in your environment. For example, `/rsa_key.p8` |
| `private_key_file_pwd` | Specify the password for accessing the private key file you use in your environment. For example, `PentahoSnowFlake123` |
See for details on the private key file and its password.
3. Click **Test** to verify your connection. A success message appears if the connection is established.
4. Click **OK** to close the connection test dialog box.
5. To save the connection, click **OK** to close the Database Connection dialog box.
You have applied key pair authentication to your defined data connection between Pentaho and Snowflake.
### Connect to an Azure SQL database
You can use an Azure SQL database as a data source with the PDI client. This connection is required if you want to use the PDI Bulk load into Azure SQL DB job entry to load data into your Azure SQL database from Azure Data Lake Storage. Pentaho supports the [Always Encrypted](https://docs.microsoft.com/en-us/sql/relational-databases/security/encryption/always-encrypted-database-engine?view=sql-server-ver15) option, [dynamic masking](https://docs.microsoft.com/en-us/azure/azure-sql/database/dynamic-data-masking-overview), and multiple authentication methods for connecting to an Azure SQL database.
Because one physical server may host databases for multiple customers, keep in mind that SQL for Azure is different from MSSQL. For more information regarding the differences between Azure SQL and MSSQL, see
#### Before you begin
You must have an Azure account with an active subscription and an instance of an Azure SQL database. You also need to install the Azure SQL database drivers. For help installing your drivers, see your Microsoft documentation for details.
Additionally, you need to obtain the following information from your system administrator:
* Host name
* Database name
* Port number
* Authentication method
* Username
* Password
If you use the **Always Encryption Enabled** option, you also need to obtain the **Client id** and **Client Secret Key**.
#### Authentication method
Pentaho supports four authentication methods for connecting to the Azure SQL DB instance:
* **SQL Authentication**
Connect using the Azure SQL Server username and password.
* **Azure Active Directory**
Connect using Multi Factor Authentication (MFA). The MFA password must be entered on the displayed webpage.
* **Azure Active Directory with password**
Connect using an Azure AD username and password.
* **Azure Active Directory with integrated authentication**
Connect using the federated on-premises Active Directory Federation Services (ADFS) with Azure Active Directory in the cloud.
#### Connect to an Azure database
Perform the following steps to connect to your database:
1. Start the PDI client and create a new transformation or job.
See the **Pentaho Data Integration** document for instructions on creating a PDI transformation.
**Note:** You can also use the Pentaho User Console to make this connection. See the **Pentaho Business Analytics** document for instructions.
2. In the **View** tab of the **Explorer** pane, double-click on the **Database connections** folder. The **Database Connection** dialog box appears, as shown below:
3. Enter your database connection information.
| Field | Description |
| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Host Name** | The name of the Azure SQL server instance. |
| **Database Name** | The name of the Azure SQL database to which you are connecting. |
| **Port Number** | The TCP/IP port number. The Azure SQL Database service is only available through TCP port 1433. You must set your firewall to allow outgoing TCP communication on port 1433. |
| **Authentication method** | The authentication method used to connect to the Azure SQL DB instance. The default is SQL Authentication. |
| **Username** | The username used to connect to the database. |
| **Password** | The password used to connect to the database. |
| **Always Encryption Enabled** | Select to use encryption. See [Use the Always Encryption Enabled option](broken://pages/9ZjngCc7m5E8hdg0KJ75) for instructions on using this option. |
| **Client id** | The unique client identifier, used to identify and set up a durable connection path to the server. |
| **Client Secret Key** | The unique name of the key value in the Azure Key Vault. |
4. Click **Test**to verify your connection.
#### Use the Always Encryption Enabled option
Before you can use the **Always Encryption Enabled** option, you must perform the following steps. Consult the [Microsoft Azure SQL documentation](https://docs.microsoft.com/en-us/azure/azure-sql/) for assistance with your Azure SQL tools.
1. Generate a column master key in the Azure Key Vault.
2. Encrypt the column using the column master key.
3. Register the app under Azure Active Directory and obtain both the **Client id** and **Client Secret Key**.
4. Grant permissions to the **Client id** for accessing the Azure Key Vault.
5. Select **Always Encryption Enabled** and provide the **Client id** and **Client Secret Key**.
The Azure Always Encrypted feature is now active.
\
\
# Edit database connections in PDI
Once a connection has been established, you can open the Database Connection dialog box to refine and change aspects of the connection.
Perform the following steps to edit an existing database connection in PDI:
1. With a transformation or job open in the PDI client canvas, expand the **Database connections** folder in the **View** tab of the **Explorer** pane
2. Right-click on a connection name and select **Edit**. The Database Connection dialog box appears with General highlighted in the left navigation pane.
3. Select items in the left navigation pane to further refine your connection or to enable/disable connection options.
# Specify advanced configuration of PDI database connections
Use the **Advanced** option in the Database Connection dialog box to configure properties associated with how SQL is generated. With these properties, you can set a standard across all your SQL tools, ETL tools, and design tools.
1. Open the Database Connection dialog box in [PUC](https://docs.pentaho.com/pba/) or [PDI](/pdia-data-integration).
2. Click **Advanced** on the left pane.
The available options depend on whether you are using PUC or PDI as shown below:
3. Check the appropriate boxes and enter the SQL statements as described in the following table:
| Identifier | Description |
| ---------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Supports the Boolean data type\* | Instructs PDI to use native Boolean data types supported by the database. |
| Supports the timestamp data type\* | Instructs PDI to use the timestamp data type supported by the database. |
| Quote all in database | Enables case-sensitive table names. For example, MySQL is case-sensitive on Linux, but not case-sensitive on Microsoft Windows. If you quote the identifiers, the databases uses a case-sensitive table name. |
| Force all to lower-case | Enables the system to change the case of all database to lower-case. |
| Force all to upper-case | Enables the system to change the case of all identifiers to upper-case. |
| Preserve case of reserved words\* | Instructs PDI to use a list of reserved words supported by the database. |
| Preferred schema name\* | For PDI, enter the preferred schema name (for example, `MYSCHEMA`). |
| SQL statements | Enter the SQL statement used to initialize this connection. |
**Note:** Which preferences appear depends on if you are accessing the dialog box from PUC or PDI. The additional fields available in PDI are indicated with an asterisk (\*).
4. Click **Test**. A success message appears if the connection is established. Click **OK** to close the connection test dialog box.
5. To save the connection, click **OK** to close the Database Connection dialog box.
# Quoting PDI database connections
Pentaho uses a database-specific quoting system. With this system, you can use any name or character that complies with the supported databases' naming conventions.
Both PUC and PDI contain a list of reserved words for most of the supported databases. Pentaho maintains a strict separation between the schema (the user or owner of a table) and the table name itself to correctly quote table or field names that contain one or more periods in them. Placing periods in table and field names is common practice in some ERP systems (for example, fields such as `V.A.T.`)
To avoid quoting-related errors, a rule stops the Pentaho software from performing quoting activity when there is a start or end quotation mark in the table or schema name. This allows you to specify the quoting mechanism yourself.
# Set specific options for PDI database connections
Use the **Advanced** option in the Database Connection dialog box to configure properties associated with how SQL is generated. With these properties, you can set a standard across all your SQL tools, ETL tools, and design tools.
1. Open the Database Connection dialog box in [PUC](https://docs.pentaho.com/pba/) or [PDI](/pdia-data-integration).
2. Click **Options** in the left pane.
The **Parameters** table appears as shown as shown below:
3. In the next available row of the **Parameters** table, enter a valid parameter name and its corresponding value. For JDBC database-specific configuration help, click **Help**.
A new browser window opens and displays additional information about configuring the JDBC connection for the database type that is currently selected in the **General** pane.
4. Click **Test**.
A success message appears if the connection is established.
5. Click **OK** to close the connection test dialog box.
6. To save the connection, click **OK** to close the Database Connection dialog box.
# Define PDI database connection pooling
You can use the **Pooling** option in the Database Connection dialog box to set up a connection pool and define options like the initial pool size, maximum pool size, and connection pool parameters. By default, a connection remains open for each individual report or set of reports in PUC and for each individual step in a transformation in PDI. For example, you might start by specifying a pool of ten or fifteen connections, and as you run reports in PUC or transformations in PDI, the unused connections drop off. Pooling helps control database access, especially if you have dashboards that contain many reports and require a large number of connections. Pooling can also be implemented when your database licensing restricts the number of active concurrent connections.
Perform the following steps to specify pooling options:
1. Open the Database Connection dialog box in [PUC](https://docs.pentaho.com/pba/) or [PDI](/pdia-data-integration).
2. Click **Pooling** in the left pane.
Options appear for your JDBC driver as shown in the example below:
The following table shows an example of **Pooling** options that might be available in a typical JDBC driver. Check your driver documentation for driver-specific pooling details.
| Option | Description |
| ----------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Enable Connection Pooling** | Enables connection pooling. |
| **Pool Size** |
Initial
Set the initial size of the connection pool.
Maximum
Set the maximum number of connections in the connection pool.
|
| **Parameters** |
You can define additional custom pool parameters. Click on any parameter to view a short description of that parameter. Click Restore Defaults when to restore the default values for selected parameters.
The most commonly-used parameter is validationQuery. The parameter differs slightly depending on your RDBMS connection. The basic set of Pentaho databases use the following values for validationQuery:
For Oracle and PostgreSQL, use Select 1 from dual.
For MS SQL Server and MySQL, use Select 1.
|
| **Description** | Enter a description for your parameters. |
3\. Click **Test**.
```
A success message appears if the connection is established.
```
4\. Click **OK** to close the connection test dialog box.
5. To save the connection, click **OK** to close the Database Connection dialog box.
# Connect to clusters (PDI only)
Use the **Clustering** options in the Database Connection dialog box to cluster the database connection and create connections to data partitions in PDI. To create a new connection to a data partition, enter a **Partition ID**, the **Host Name**, the **Port**, the **Database Name**, **User Name**, and **Password** for the connection.
If you have the Pentaho Server configured in a cluster of servers, and use the Data Source Wizard(DSW) in PUC to add a new data source, the new data source will only be seen on the cluster node where the user has a session. For the new data source to be seen by all the cluster nodes, you must disable DSW data source caching. This may cause the loading of the data source list to be slower since the list is not cached.
To disable the cache, navigate to the `server/pentaho-server/pentaho-solutions/system` folder and set the **enableDomainIdCache** value in the `system.properties` file to false.
# Modify connections from PDI
Access other database-related connection tasks in PDI by right-clicking on the connection name in the **View** tab of the **Explorer** pane, as shown below:
Other database-related tasks
The following table describes these tasks:
| Task | Description |
| --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Duplicate** | Duplicate the database connection. The duplicate will not be created unless you specify a different **Connection Name** in the Database Connection dialog box when it appears. |
| **Copy to clipboard** | Copy the XML defining the step to the clipboard. |
| **SQL Editor** | Execute SQL commands against an existing connection within the SQL Editor. See the **Pentaho Data Integration** document for details on the SQL Editor. |
| **Clear DB Cache** | Clear out the database cache used by PDI to speed up connections. This command is commonly used when databases tables have been changed, created, or deleted (when the information in the cache no longer represents the layout of the database). |
| **Share** | Share the connection information among transformations and jobs. |
| **Explore** | Use the Database Explorer to explore the schemas and tables of your connected database. See the **Pentaho Data Integration** document for details on the Database Explorer. |
| **Show dependences** | Show all of the transformations and jobs that use this database connection. |
## Delete connections from PDI
Perform the following steps to delete a connection in PDI:
1. Expand the **Database connections** folder in the **View** tab of the **Explorer** pane.
2. Right-click on a connection name and select **Delete**.
The data source no longer appears under the Database connections folder.
# PDI and Hitachi Content Platform (HCP)
Hitachi Content Platform (HCP) is the distributed, fixed-content, data storage system from Hitachi Vantara. HCP provides a scalable, easy-to-use repository that can accommodate all types of data, from simple text files to medical images to multigigabyte database images. Read the [About Hitachi Content Platform](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/System_administration/Introduction_to_Hitachi_Content_Platform/01_About_Hitachi_Content_Platform) article to learn more about HCP and how it works.
For working with Pentaho Data Integration (PDI), it's useful to remember a few key terms. HCP stores objects in a repository. Each object permanently associates data HCP receives (for example, a document, an image, or a movie) with information about that data called metadata. In PDI, you can query the metadata to locate and access HCP objects. The HCP object (such as the image file below) consists of a read-only file, a unique URL, system metadata properties, and custom metadata annotations.
When using the PDI steps for HCP, you should be familiar with how the HCP repository is partitioned into namespaces. A namespace is a logical grouping of objects such that the objects in one namespace are not visible in any other namespace. Before you can query HCP from PDI, you will need to verify settings and options for your namespaces in the [System Management Console](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/System_administration/HCP_administration/01_System_Management_Console) and the [Tenant Management Console](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/Tenants_and_Namespaces/General_administrative_information/03_Tenant_Management_Console).
**Note:**
PDI currently supports HCP version 8.0.0.9.
## Using PDI for your HCP data
You can use PDI transformation steps to improve your HCP data quality before storing the data in other formats, such as JSON , XML, or Parquet. By using PDI to cleanse your HCP data, you can:
* Connect to a wide variety of data sources, import data from those sources, and then store that data in HCP as metadata of your objects.
* Blend data from sources that are in different formats, and then store the aggregated data as object metadata in HCP.
* Extract metadata from HCP, transform the data, then send the data to cloud storage.
PDI has three transformation steps you can use to work with your metadata in HCP.
* [**Query HCP**](/pdia-data-integration/pdi-transformation-steps-reference-overview/query-hcp)
Locate data objects by searching system and custom metadata annotations. HCP returns the unique URL of objects matching your search terms. For example, a radiology practice can search for an X-ray (object) of a specific patient or all X-rays performed by a specified physician.
* [**Read metadata from HCP**](/pdia-data-integration/pdi-transformation-steps-reference-overview/read-metadata-from-hcp)
Identify and select an HCP object by its URL path and then select a specific target annotation name to read. The step returns the requested custom metadata from the annotation back to your PDI transformation for downstream processing.
* [**Write metadata to HCP**](/pdia-data-integration/pdi-transformation-steps-reference-overview/write-metadata-to-hcp)
Identify and select an HCP object by its URL and then write custom metadata annotations to the object associated with the object URL, enriching and validating the data stored in your HCP repository. For example, a radiology practice could add patient medication data and associated medical conditions to an X-ray (object) or remove invalid diagnosis codes.
These steps use a Virtual File System (VFS) connection to access the HCP repository. See [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser) for details.
# Hierarchical data
Pentaho supports a hierarchical data type (HDT) by means of the Pentaho EE Marketplace hierarchical data type plugin that adds the data type and creates five steps. These steps are designed to simplify string manipulation, with the ability to convert between HDT fields and formatted strings.
These steps improve performance on the steps that that treat hierarchical representations in string fields and enable you to address or modify nested array indices and keys. This data type can hold nested or complex data that is built by using objects and arrays and can also hold single elements. It can be used with any PDI step that uses hierarchical data.
The steps are:
* [Modify values from a single row](/pdia-data-integration/pdi-transformation-steps-reference-overview/modify-values-from-a-single-row)
* [Modify values from grouped rows](/pdia-data-integration/pdi-transformation-steps-reference-overview/modify-values-from-grouped-rows)
* [Extract to Rows](/pdia-data-integration/pdi-transformation-steps-reference-overview/extract-to-rows)
* [Hierarchical JSON Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/hierarchical-json-input)
* [Hierarchical JSON Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/hierarchical-json-output)
Of the five steps, the first three steps have been created to manipulate generic hierarchical data and the last two steps work with data in the JSON format. The first two steps are used to create or modify hierarchical data, and the third step is used to deserialize the data. A common use for these steps is described in the following table:
| Step | Usage |
| ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| Modify values from a single row | You can take one row of data from an incoming step and then use that to either create a new hierarchical data type or modify a hierarchical data type that's on that incoming data stream. |
| Modify values from grouped rows | You can take multiple rows of data, group them based on a common field or set of fields, and then create a new hierarchical data type or modify and existing data type. |
| Extract to Rows | You can take data out of the hierarchical data type and place it onto the PDI data stream and use the data with other steps. |
The last two steps are specifically for working with JSON and JSON Lines ([JSONL](https://jsonlines.org/))) files. Differences between the Hierarchical JSON input step and the JSON Input step are:
* The JSON input step uses the [JSONPath](https://github.com/json-path/JsonPath) path specification, the Hierarchical JSON input step uses an [HDT path specification](broken://pages/ji5ygWJ7r8yORvFw3Rjj).
* The JSON input step reads the entire JSON file into memory, the Hierarchical JSON input step processes the JSON as an input stream and loads only the specified content . You can load filtered subsets of large JSON files, or separate lines from a JSONL directly into a HDT field for extraction or modification by other steps.
* The JSON input step loads content onto multiple fields of a stream, the Hierarchical JSON input step has filters you can apply to fetch the subset of a JSON file.
| Step | Usage |
| ------------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Hierarchical JSON input | You can read an entire JSON file into memory and send it to a single HDT field on the steps output field. You can use the **Split rows across path** option on the **Output** tab to extract each JSON parent to a row, allowing you to process the data in blocks and reduce the amount of memory needed for processing the data. |
| Hierarchical JSON Output | You can convert hierarchical data to a JSON file and write it to an output field or pass the output to a PDI servlet. |
You can convert data between JSON and HDT types with the **Type** and **Format** field options in steps that support HDT. In the following example, the Select values step is set to convert hierarchical data to JSON:
## Hierarchical data path specifications
The Hierarchical data path (HDT) path specification for extractions is different than the specification for modifications. Each specification is described below.
### HDT path specification for extractions
The HDT path specifications for extractions are:
* You must always start with the dollar sign (`$`) for the root.
* Simple alphanumeric key values are separated by periods (`.`).
* Numeric indices are designated like arrays with square brackets `[ n ]`.
* Complex string identifiers are designated with single quotation marks in square brackets `[‘Complex string identifier’]`.
* If your identifier has a special characters like a single quote mark, you must escape that character with a backslash (`\`). For example `['\‘Complex string identifier\’']`.
* You can use the asterisk (\*) wildcard for string or numeric keys.
For example: `$['l am a perfectly'].normal[0].example`
### HDT path specification for modifications
The HDT path specifications for modifications are:
* There is an additional array push operator of square brackets `[ ]` when added to the end of an array that will create an index.
* There are four special reference quote marks that you can use:
* `$[?string_field?]` - Takes the string value of a field from an incoming PDI row and uses that value as the map key fragment.
* `$[#numeric_field#]` - Takes the numeric value of an incoming PDI field and uses the value for array or list access with a resolved numeric index.
* `$[$string_backreference$]` – Matches the string key fragment from a previous paths segment.
* `$[@numeric_backreference@]` - Matches the numeric array or list index from a previous paths segment.
**Note:** The backreference numbering is based on the index in the path. In the code example: `$.first[ 4 ][‘hello’]`, the `$1$ = ‘first`’, `@2@ = 4, $3$ = ‘hello’`. The backreference is a 1-based index, and the surrounding `$` or `@` specifies if a string or numeric value was used.
# PDI and Snowflake
Snowflake is an analytic data warehouse running completely on a cloud infrastructure. Snowflake supports loading popular data formats like JSON, Avro, Parquet, ORC, and XML. Using Pentaho Data Integration (PDI), you can load your data into Snowflake and define jobs in PDI to efficiently orchestrate warehouse operations, paying only for the storage and computing resources actually used when you use them. See the [Snowflake Documentation](https://docs.snowflake.net/manuals/index.html) to learn more about Snowflake and how it works.
Using the Snowflake job entries in PDI, data engineers can set up virtual warehouses, bulk load data, and stop the warehouse when the process is complete. You can scale Snowflake virtual warehouses up and down, or suspend them when not in use to reduce costs.
## Snowflake job entries in PDI
PDI has six job entries you can use to load data and manage warehouses in Snowflake.
In PDI, you can bulk load files into your Snowflake data warehouse:
* [**Bulk load into Snowflake**](/pdia-data-integration/pdi-job-entries-reference-overview/bulk-load-into-snowflake)
Using this job entry, you can load a vast amount of data into a warehouse on Snowflake in a single session, provided you have sized your warehouse correctly. You can load data from Snowflake stages using the Snowflake VFS connection or directly from AWS S3 using the S3 VFS connection. For example, you may want to upload six months of ORC data from an S3 bucket. Using this job entry, you can define the source and type of data to load, specify the target data warehouse, and provide any needed parameters.
In PDI, you can create, modify, and even delete a Snowflake virtual warehouse to help you automate your virtual warehouse scaling activities. These orchestration entries include:
* [**Create Snowflake warehouse**](/pdia-data-integration/pdi-job-entries-reference-overview/create-warehouse-pdi-job-entry)
You can use this job entry to create a Snowflake virtual warehouse. You can set size, scaling, automated suspension, and other properties for your warehouse.
* [**Modify Snowflake warehouse**](/pdia-data-integration/pdi-job-entries-reference-overview/modify-warehouse-pdi-job-entry)
Once you create a warehouse, you can edit its settings using this job entry. Modifying a warehouse is useful if your users typically perform simple queries and only require a small warehouse. However, to meet your ETL service-level agreements (SLA), you may need a larger warehouse during the ETL process. Using this job entry, you can modify the warehouse at the beginning of the ETL process to scale it up, and then modify it to scale it back down when the ETL process is complete.
* [**Delete Snowflake warehouse**](/pdia-data-integration/pdi-job-entries-reference-overview/delete-snowflake-warehouse)
Use this job entry to delete virtual warehouses. Deleting unwanted virtual warehouses helps you clean up the Snowflake management console.
In PDI, you can dynamically start and stop Snowflake virtual warehouses to help you better control your Snowflake costs. For example, if your employees only work 8 hours a day, then you don’t need to keep your warehouse up for 24 hours a day. Using the Start and Stop job entries, you can turn on the warehouse from 8 AM to 5 PM for day-to-day business activities and again from 11 PM to 2 AM while your ETL processes are running.
See the [Snowflake Documentation](https://docs.snowflake.net/manuals/index.html) to learn more about how credits are billed for running virtual warehouses in Snowflake.
* [**Start Snowflake warehouse**](/pdia-data-integration/pdi-job-entries-reference-overview/start-snowflake-warehouse)
Use this job entry to start/resume a virtual warehouse on Snowflake. Warehouses consume credits while running.
* [**Stop Snowflake warehouse**](/pdia-data-integration/pdi-job-entries-reference-overview/stop-snowflake-warehouse)
You can set this job entry to stop/suspend a virtual warehouse on Snowflake. Suspending a warehouse stops the warehouse from consuming credits once all the servers shut down.
# Copybook steps in PDI
Pentaho Data Integration supports simplified integration with fixed-length records in binary mainframe data files, so more users can ingest, integrate, and blend mainframe data as part of their data integration pipelines. This capability is critical if your business relies on massive amounts of customer and transactional datasets generated in mainframes that you want to search and query to create reports.
Mainframe file records are typically defined by a COBOL copybook. A COBOL copybook is a selection of code that defines the data layout of items from a data source, including records, segments, fields, and keys. Copybooks allow developers to reuse data structures in multiple instances.
Copybook data is usually extracted from the mainframes in a block of records and then stored in binary files, along with a definition file, that can be read by PDI. Based on the definition file, the Copybook input step and the Read metadata from Copybook step read the binary content in the data files and convert it to PDI rows which makes the data easy to integrate into your transformations.
These steps navigate you through challenging conversion issues, such as packed decimal numbers and multibyte data type storage, which are typical of COBOL copybooks. The steps can also handle REDEFINES clauses, which change some of the fields in a record based on other values in the record.
## Copybook transformation steps in PDI
PDI has two transformation steps you can use to read mainframe records from a file and transform them into PDI rows. These steps are intended for data scientists with mainframe file and copybook experience.
* [**Copybook input**](/pdia-data-integration/pdi-transformation-steps-reference-overview/copybook-input-pdi-step)
This step reads the mainframe files that were originally created using the copybook definition file and outputs to the PDI stream for use in transformations.
* [**Read metadata from Copybook**](/pdia-data-integration/pdi-transformation-steps-reference-overview/read-metadata-from-copybook)
This step reads the metadata of a copybook definition file to use with [ETL Metadata Injection](/pdia-data-integration/pdi-transformation-steps-reference-overview/etl-metadata-injection) in PDI.
## Metadata discovery
You can use metadata discovery automate the tedious process of manually identifying and determining metadata from Cobol Copybook and JDBC databases.
* [**Read metadata from Copybook**](/pdia-data-integration/pdi-transformation-steps-reference-overview/read-metadata-from-copybook)
The Read metadata from Copybook step reads a binary fixed-length copybook definition file and outputs the file and column descriptor information as fields to Pentaho Data Integration rows.
* [**Query metadata from a database**](/pdia-data-integration/pdi-transformation-steps-reference-overview/query-metadata-from-a-database-article)
The Query metadata from a database step discovers metadata from six different JDBC metadata types to use with [ETL Metadata Injection](/pdia-data-integration/pdi-transformation-steps-reference-overview/etl-metadata-injection) in PDI.
# Work with the Streamlined Data Refinery
The Streamlined Data Refinery (SDR) is a simplified and specific ETL refinery composed of a series of Pentaho Data Integration (PDI) jobs that take raw data, augment and blend it through the request form, and then publish it for report designers to use in Analyzer.
To introduce you to the SDR structure, we have created the Movie Ratings-SDR [sample](https://pentaho.app.box.com/SDRSample60). This sample was developed by Pentaho and is based on CTools.
# How does SDR work?
The components that make up the data refinery are PDI, used for parameter entry, working in conjunction with an app for refining the data. This app calls to the Pentaho Server for the main job: refining data through Spoon using the new [Build Model](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Advanced%20Pentaho%20Data%20Integration%20topics/Advanced%20topics%20\(Pentaho%20Data%20Integration%20overview\)/Work%20with%20the%20Streamlined%20Data%20Refinery/Use%20the%20Streamlined%20Data%20Refinery/Building%20blocks%20for%20the%20SDR/Use%20the%20Build%20Model%20job%20entry%20for%20SDR=GUID-7FAB840B-9B79-440E-AE59-31C3F5B5EACA=4=en=.md) job entry, then publishing the data source back to the Pentaho Server through the [Publish Model](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) job entry. Once it is published, the refined data is available for use in creating Analyzer reports. This process is shown in the graphic.
## App Builder, CDE, and CTools
[App Builder](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/app-builder-and-community-dashboard-editor) is an application builder for people who may not have Java knowledge, but who may have plenty of interesting ideas for new plugins. All that is required to use App Builder is knowledge of CTools and PDI.
Community Dashboard Editor (CDE), when integrated with the Pentaho User Console (PUC), simplifies the process of creating, refining, and previewing Pentaho dashboards. You can use CDE to design dashboards, either from scratch or using a template.
See **Pentaho CTools** for details.
## Get started with App Builder
App Builder is an application builder for people who may not have Java knowledge, but who may have plenty of interesting ideas for new plugins. All that is required to use App Builder is knowledge of CTools and Pentaho Data Integration. Follow these steps to get started building impressive applications with App Builder.
1. Log in to the Pentaho User Console (PUC).
Access App Builder by copying this URL in your web browser: `*\{your PUC location\}*/pentaho/plugin/sparkl/api/main`
The App Builder splash page appears.
2. Click the **Plus Sign** to begin creating a new app.
3. Name your app, then fill in the plugin information on the **About** tab.
4. On the **Element** tab, click to add an element, then add two dashboards by clicking on the **Add** bar and filling out the fields.
5. Click **Edit** to alter the dashboards and add a simple row.
6. Click on **Preview** to see your dashboard.
7. Once you are happy with the way the dashboards look, you can share it with your users.
Learn more about Pentaho products and CTools with this list of [SDR resources](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery).
# Install SDR
Before you can install Streamlined Data Refinery (SDR), you must first install the Pentaho Data Services plugin. For instructions, see the **Install Pentaho Data Services** section in [Expose transformation output with Pentaho Data Services](/pdia-data-integration/loading-data-from-pdi#expose-transformation-output-with-pentaho-data-services).
To install the SDR plugin in Pentaho Data Integration, use the Plugin Manager. For instructions, see [Install plugins in PDI client](/pdia-data-integration/use-plugin-manager#install-plugins).
{% hint style="info" %}
**Note:** You can also install the SDR plugin in the Pentaho Server. For instructions, see [Install plugins in PUC](/pba/pentaho-user-console/modern-design/plugin-manager#install-plugins-in-puc).
{% endhint %}
# Install and configure the SDR sample
Install and configure the Streamlined Data Refinery (SDR) sample as an example or testing purposes to help you understand the SDR structure.
Before you begin, you must install Pentaho Data Integration and Analytics. For instructions, see **Install Pentaho Data Integration and Analytics 11.0**.
For information about installing and configuring the SDR sample for your environment, see the following topics:
* [Install the SDR sample](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/installing-and-configuring-the-sdr-sample)
Install the SDR sample to use as an example or for testing purposes.
* [Configure KTR files for your environment](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/configure-ktr-files-for-your-environment)
If you are not using Postgres and a default installation, configure KTRs in Spoon to use the SDR form.
* [Configure KTR files for your environment](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/configure-ktr-files-for-your-environment)
Delete items from the **All Requests Processed** list to clean your **Latest Requests** list.
* [Install the Vertica JDBC driver](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/install-the-vertica-jdbc-driver)
If you are using Vertica, install the Vertica JDBC driver.
* [Use Hadoop with the SDR](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/use-hadoop-with-the-sdr)
Set up the Streamlined Data Refinery to use Hadoop
* [App endpoints for SDR forms](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/app-endpoints-for-sdr-forms)
Learn about API endpoints that you can use to run the app.
* [App Builder and Community Dashboard Editor](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/app-builder-and-community-dashboard-editor)
Learn more about getting started with App Builder and Community Dashboard Editor to work most effectively with the Streamlined Data Refinery.
# Install the SDR sample
Install the SDR sample to use as an example or for testing purposes. The SDR sample was developed by Pentaho, based on CTools.
If you are evaluating Pentaho software, we recommend using the graphical installation method for a quick set-up. If you already have Pentaho software installed, you can skip right to the section for downloading and installing the sample.
These steps will get the sample installed and running.
1. Download the [sample SDR.zip](https://pentaho.app.box.com/folder/41400845668) and save it to your desktop.
2. Extract the contents of the `SDR.zip` file to this directory: `pentaho/server/pentaho-server/pentaho-solutions/system`.
It should create an SDR folder in the directory.
3. If you are using the default evaluation method, start the Pentaho Server and log into PUC.
4. Go to **Tools** at the top of PUC and verify that **Movie Ratings-SDR Sample** appears in the drop-down menu.
If you are evaluating Pentaho and used the default installation instructions, you are ready to get started experimenting with the [sample form](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Advanced%20Pentaho%20Data%20Integration%20topics/Advanced%20topics%20\(Pentaho%20Data%20Integration%20overview\)/Work%20with%20the%20Streamlined%20Data%20Refinery/Use%20the%20Streamlined%20Data%20Refinery/How%20to%20use%20the%20SDR%20sample%20form=GUID-FBAAA9DF-7CA6-4CEB-B265-CCCE2332FFC2=5=en=.md). If you are doing anything else, follow the appropriate instructions in the [Administration](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery) section before continuing with the sample form.
# Configure KTR files for your environment
If you are not using Postgres and a default installation, you will need to configure a few of the sample KTR files in Spoon to use the SDR form.
1. Click **File** > **Open** and navigate to find the `SDR_data.ktr` in this directory: `pentaho/server/pentaho-server/pentaho-solutions/system/SDR/endpoints/kettle`.
1. Right-click on the Set Local Variables step to edit, and enter the URL for your Pentaho Server.
2. Click **OK**.
2. In the same transformation, right-click on the Call\_ML\_SDR job and select **Open Referenced Object** > **Job** to open the `_ML_SDR.job`.
1. Right-click to edit the Create Table step to point to your staging database.
2. Click **OK**.
3. Right-click to edit the Publish Model step and, if requested, enter your user id and password for the Pentaho Server.
1. Click **Test Connection**.
2. Click **OK**.
4. Open the `_ML_SDR_REFINERY.ktr`.
1. Right-click to edit the Out to Staging DB step.
2. Select your staging database from the drop-down menu.
5. Save all of these files and exit out of the PDI client.
6. Restart the Pentaho Server.
# Clean up the All Requests Processed list
You can delete items from the **All Requests Processed** list in order to clean your **Latest Requests** list.
**CAUTION:**
Deleting entries from the **Latest Requests** list also deletes the tables that contain your data and removes the metadata or Analyzer data source from the list of data sources that appear in PUC.
Follow these quick steps to remove items from the **Latest Requests** list.
1. Click on **All Requests Processed** to open the **Latest Requests** list.
2. Click on **Edit**.
3. Select the check box next to the data sources that you want to delete.
4. Click on **Delete** to remove the selected data sources from the list.
Your **All Requests Processed** list is now cleaned and the corresponding tables are deleted.
# Install the Vertica JDBC driver
If you are using Vertica, you will need to follow these steps to install the Vertica JDBC driver.
1. Exit the PDI client, if you have it running.
2. Stop the Pentaho Server.
3. Copy the Vertica JDBC driver to these three directories:
* `INSTALL_DIR/server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib`
* `INSTALL_DIR/server/pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib`
* `INSTALL_DIR/design-tools/data-integration/lib`
Next, we are going to [download and install](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/install-and-configure-the-streamlined-data-refinery/installing-and-configuring-the-sdr-sample) the SDR sample.
# Use Hadoop with the SDR
There are a [few prerequisites](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article) that you need to make sure are satisfied before you can begin using SDR with Hadoop.
1. Open the `ML_SDR_REFINERY.ktr` and locate the Hadoop File Input step in the upper-left.
2. Click to activate the hop between the Hadoop File Input and Parse weblog steps.
3. Click to deactivate all of the hops between the Log File List and Read Weblog Files steps.
4. Right-click to edit the Out to Staging DB step and select your staging database from the drop-down menu.
5. Click **OK** to close the window and save the transformation.
# App endpoints for SDR forms
These are a few API endpoints that you can also use to run the app. To perform a command, alter this example to match your parameters:
```
http://*\{host\}*/pentaho/plugin/*\{pluginID\}*/api/*\{command\}*
```
Here are a couple of examples using some parameters:
* **Genre Selector API**
```
http://localhost:8080/pentaho/plugin/SDR/api/genre
```
* **Example Response**
```
{"metadata":[{"colIndex":0,"colType":"String","colName":"GenreID"},{"colIndex":1,"colType":"String","colName":"Genre"}],"queryInfo":{"totalRows":18},"resultset":[["Action","Action"],["Adventure","Adventure"],["Animation","Animation"],["Childrens","Childrens"],["Comedy","Comedy"],["Crime","Crime"],["Documentary","Documentary"],["Drama","Drama"],["Fantasy","Fantasy"],["Film-Noir","Film-Noir"],["Horror","Horror"],["Musical","Musical"],["Mystery","Mystery"],["Romance","Romance"],["Sci-Fi","Sci-Fi"],["Thriller","Thriller"],["War","War"],["Western","Western"]]}
```
* **Gender Selector API**
```
http://localhost:8080/pentaho/plugin/SDR/api/gender
```
* **Example Response**
```
{"metadata":[{"colIndex":0,"colType":"String","colName":"GenderID"},{"colIndex":1,"colType":"String","colName":"Gender"}],"queryInfo":{"totalRows":2},"resultset":[["M","Male"],["F","Female"]]}
```
Here is a list of app endpoint parameters:
| Endpoint | Description |
| ---------------------- | -------------------------------------------------------------------------------------------------------------------- |
| **genre** | Populates the options for the **genre** selector. |
| **gender** | Populates the options for the **gender** selector. |
| **occupation** | Populates the options for the **occupation** selector. |
| **income** | Populates the options for the **income** selector. |
| **firstdate** | Returns the limit dates for the data to be processed. |
| **data\_source\_name** | Returns the names of all data sources available on the server. |
| **latest\_requests** | Returns the latest 10 requests made in a table, instead of inside of a popup. |
| **sdr\_data** | Processes the request and returns the status of the data. |
| **refresh** | Refreshes the kettle and dashboard elements to reflect any saved changes. Clears the cache for all kettle endpoints. |
# App Builder and Community Dashboard Editor
Learn more about getting started with App Builder and Community Dashboard Editor to work most effectively with the Streamlined Data Refinery.
## Get started with App Builder
App Builder is an application builder for people who may not have Java knowledge, but who may have plenty of interesting ideas for new plugins. All that is required to use App Builder is knowledge of CTools and Pentaho Data Integration. Follow these steps to get started building impressive applications with App Builder.
1. Log in to the Pentaho User Console (PUC).
Access App Builder by copying this URL in your web browser: `*\{your PUC location\}*/pentaho/plugin/sparkl/api/main`
The App Builder splash page appears.
2. Click the **Plus Sign** to begin creating a new app.
3. Name your app, then fill in the plugin information on the **About** tab.
4. On the **Element** tab, click to add an element, then add two dashboards by clicking on the **Add** bar and filling out the fields.
5. Click **Edit** to alter the dashboards and add a simple row.
6. Click on **Preview** to see your dashboard.
7. Once you are happy with the way the dashboards look, you can share it with your users.
Learn more about Pentaho products and CTools with this list of [SDR resources](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery).
## Community Dashboard Editor and CTools
Community Dashboard Editor (CDE), when integrated with PUC, simplifies the process of creating, refining, and previewing Pentaho dashboards. You can design dashboards either from scratch or using a template. The layout panel allows you to style your dashboard and add elements such as text or images.
The [Community Dashboard Editor](https://community.hitachivantara.com/s/article/cdf-tutorial) tutorials have more in-depth details on working with CDE.
# Use the Streamlined Data Refinery
Learn how to use the SDR Sample form and the PDI steps and job entries used to create SDR models.
The best method to learn the SDR is to use the SDR Sample Form to create Analyzer reports. [Learn more](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/how-to-use-the-sdr-sample-form).
To learn more about the building blocks for the SDR, view the following articles:
* [Use the Build Model job entry for SDR](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/use-the-build-model-job-entry-for-sdr)
* [Using the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step)
* [Creating measures on stream fields](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-measures-on-stream-fields)
* [Creating attributes](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-attributes)
* [Creating link dimensions](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-link-dimensions)
* [Using the Shared Dimension step for SDR](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-shared-dimension-step-for-sdr)
* [Using the Publish Model job entry for SDR](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr)
# How to use the SDR sample form
The Streamlined Data Refinery (SDR) is made up of a few different tools which work in conjunction to make the data refining process easier for the user. Data is refined in PDI with the [Build Model](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Advanced%20Pentaho%20Data%20Integration%20topics/Advanced%20topics%20\(Pentaho%20Data%20Integration%20overview\)/Work%20with%20the%20Streamlined%20Data%20Refinery/Use%20the%20Streamlined%20Data%20Refinery/Building%20blocks%20for%20the%20SDR/Use%20the%20Build%20Model%20job%20entry%20for%20SDR=GUID-7FAB840B-9B79-440E-AE59-31C3F5B5EACA=4=en=.md) job entry and the [Annotate Stream](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Advanced%20Pentaho%20Data%20Integration%20topics/Advanced%20topics%20\(Pentaho%20Data%20Integration%20overview\)/Work%20with%20the%20Streamlined%20Data%20Refinery/Use%20the%20Streamlined%20Data%20Refinery/Building%20blocks%20for%20the%20SDR/Using%20the%20Annotate%20Stream%20step=GUID-283DA7C6-8AD8-4FCA-999C-F7AA69A96BE4=3=en=.md) step, then pushed to your business analytics tools through the [Publish Model](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) job entry in the Pentaho Server. From there, report designers can use the SDR form to cherry-pick their data for Pentaho Analyzer reports. These instructions walk you through how to use the sample SDR form created by Pentaho for example purposes.
## Edit the Movie Ratings - SDR Sample form
The Movie Ratings-SDR Sample form is easy to use. All you will need to do is select the check boxes next to the filters which you want to apply and the SDR will refine the data for your Analyzer report.
**Note:** We recommend that you make sure to do at least some selecting of data for optimum results.
1. Log on to PUC using a logon with administrator permissions.
2. Open the Movie Ratings-SDR Sample form.
The **All Requests Processed** field contains a list of up to ten instances of data set activity. You can click an item in the list to work with that data set or create a new one.
3. Click to select the **Genre** filters which you want to use from the **Movie Review Filter** panel.
4. Click to select profile items in the **Member Profile** panel to narrow down your data to specific points.
5. Enter the **Start Date** for the data set. Optionally, use the date picker to choose the start date.
6. Enter the **End Date** for the data set. Optionally, use the date picker to choose the end date.
7. Indicate whether you would like to **Grant Access** to the model to **Everyone** or keep it for only your use.
8. Enter a data source name in the field.
9. Click the **Let's do this** button to run the data set and launch a new Analyzer report.
10. Depending on whether you keep the check box selected, Pentaho Analyzer launches with your data set parameters in place. Optionally, you can click **Go to Analyzer** when you are ready to begin creating reports.
# Building blocks for the SDR
Learn how to use the steps and job entries to use for SDR models: the Build Model and Publish Model job entries, and the Annotate Stream and Shared Dimension steps.
* Use the [Build Model job entry](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/use-the-build-model-job-entry-for-sdr) to create data sources to publish on the Pentaho Server.
* Further refine your data for the SDR with annotations using the [Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step).
* Create dimensions to share with other users by using the [Shared Dimension step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-shared-dimension-step-for-sdr).
* The [Publish Model job entry](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) allows you to publish the data source created with the Build Model job entry.
# Use the Build Model job entry for SDR
The Build Model job entry can be used to create Data Source Wizard data sources, which includes both a Metadata and a Mondrian model. It works by searching upstream for an output step or data service upon which to base your model, or you can select an existing Data Source Wizard or Analyzer model from the Pentaho Server. Note that if you are using a data service as the source in your Build Model job entry, you must be connected to a Pentaho Repository to successfully publish your model.
After you have run this job entry, a data model is created and can be published to the server for the creation of Analyzer reports in the Pentaho User Console. Business users will also be able to refine the data model with the Data Source Model Editor. If you selected an Analyzer model to build from, the Build Model job entry will only create an Analyzer model.
A Build Model job entry builds the model and sets variables into the job that can be modified or picked up by the [Publish Model](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) job entry. An example workflow is shown below.
After the Build Model job runs successfully, you can publish the data source to the Pentaho Server.
## Create a Build Model job entry
This task assumes you are in the job canvas of the PDI client.
Use the Build Model job entry to create a model for publishing directly to the Pentaho Server.
1. In the **Design** tab, click the **Modeling** folder, and then double-click the Build Model job entry. Alternatively, you can drag the job entry icon on to the job canvas.
2. Double-click the **Build Model** icon to open the Build Model dialog box.
3. Enter a name for the entry in the **Entry name** field.
**Note:** The following characters are not valid in Data Source Wizard source names:
```
% ? : [] * \t \r \n
```
4. In the **Source** field, select an output step or a Pentaho Data Service from the drop-down menu, or press CTRL Space to select a variable.
The selected output step or data service provides the source data in the model. If no data is written to the table by the selected source, then the Build Model job entry will not build a model.
**Note:** When you select a Pentaho Data Service as the source, previously defined link dimensions will not be linked to your model. Link dimension annotations are dependent on shared dimensions, and shared dimensions cannot use a data service for their source. See [Pentaho Data Services](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/pentaho-data-services) for more information about data services.
5. Enter a name for your data model in the**Model Name** field. Optionally, click in the field and press CTRL Space to select a variable.
The model name entered here will appear in the list of data sources in the Pentaho Server.
6. In the **Modeling Method** section, you can select the modeling method for the entry.
* **Auto Model** if you want to build a new model. When you choose to auto model, your annotations will be applied to the model when it is opened in Analyzer.
* **Use Existing** if you want to base your model on one which already exists in the Pentaho Server. Click the **Select** button below to enter the name of an existing model from the server in the required field. See [Select existing model options](broken://pages/v8SPOimxw9N7Ks6O2Pu1). When you use an existing model, your annotations will not be applied to the model when it is opened in Analyzer.**Note:** When you select **Auto Model** and you have modified a measure in Analyzer, the inline model editing changes you made in Analyzer will take priority over any annotations added to the measure in PDI. In addition, annotations you add in PDI will not be visible for that measure when viewed in Analyzer.
7. Click **OK** to save your changes and close the dialog box, or click **Cancel** to discard your changes and close the dialog box.
## Select existing model options
If you selected to use an existing model to build upon, you will need to be connected to the Pentaho Server. Once you are connected, you can select an existing model with the same physical schema to base your new model on.
**Note:** If you select **Use Existing**, your annotations are not applied to the model when it is opened in Analyzer. If you want to apply your annotations to the model in Analyzer, select **Auto Model**.
1. Fill in or edit the fields in the **Pentaho Server Connection** section.
| Field | Description |
| ------------- | ------------------------------------------------------------------------------------------------------ |
| **URL** | The base URL string used to connect to the server. |
| **User Name** | The user name required to access the server. |
| **Password** | The password associated with the provided user name which is passed during the authentication process. |
2. Click the **Connect** button to connect to the server using the information provided in the above fields.
If the connection was successful, the **Existing Models** drop-down menu will be available for selection with a list of models to choose from.
3. Select a model from the **Existing Models** drop-down menu.
If no models are listed, verify that you are connected to the Pentaho Server.
4. (Optional) Select the **Create model if not found** check box to generate a model if the system cannot find an existing model with the entered name.
5. When finished, click **OK** to save your changes and close the dialog box, or **Cancel** to close the dialog box without saving your changes.
This is an example of the Select Existing Model dialog box.
## Variables for Build Model job entry
These variables are generated when the model is created. You can only see these specific variables for the Build Model job entry if you press CTRL Space after the first run.
You can use these variables to know more about the model you created. For example, if you have scheduled a job to run at regular intervals, you can include these variables in the Mail job entry to alert you to any issues during the run.
| Variable | Variable Type | Definition |
| ------------------------------------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **JobEntryBuildModel.DatabaseConnection.\** | String | Returns the name of the database connection used by the transformation on the source. This connection information is specified in the Build Model job entry. |
| **JobEntryBuildModel.XMI.\** | String | Returns all the contents of the XML file published on the Pentaho Server (It may or may not contain the OLAP components.) |
| **JobEntryBuildModel.XMI.DSW.\** | Boolean (True or False) | Indicates if the XMI should be delivered as a Data Source Wizard data source. Typically, when this is 'true', we only generate the DSW data source and no other contents. |
# Using the Annotate Stream step
The Annotate Stream step helps you refine your data for the Streamlined Data Refinery by creating measures, link dimensions, or attributes on stream fields you specify.
If you want, you can create multiple annotations on the same field. For example, you might want to create an average measure and a sum measure on the same field. You can also annotate multiple streams to modify the same data model. Additionally, you can create and add calculated measures to the data model.
The Annotate Stream step modifies the default model produced from the [Build Model](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/use-the-build-model-job-entry-for-sdr) job entry.
After you are done annotating your data model, you are ready to [publish](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) it.
### Metadata injection support
All fields of this step support metadata injection. You can use it with [ETL metadata injection](/pdia-data-integration/pdi-transformation-steps-reference-overview/etl-metadata-injection) to pass metadata to your transformation at runtime.
When using metadata injection with the Annotate Stream step, you can reuse shared annotation groups that already exist, but you cannot create a new shared annotation group. If you inject a shared annotation group by providing a value for **SHARED\_ANNOTATION\_GROUP**, then the step assumes you are reusing an existing shared annotation group. As a result, any annotations defined in the ETL metadata injection step are ignored.
# Use the Annotate Stream step
You can create annotations in different ways. The annotation type which you create determines which properties are shown in the dialog box to complete that annotation. This task assumes you are in the transformation canvas of the PDI client.
1. In the **Design** tab, click the **Flow** folder, and then double-click the Annotate Stream step.
Alternatively, you can drag the step icon on to the transformation canvas.
2. Double-click the **Annotate Stream** icon to open the Annotate Stream dialog box.
3. Enter a name for the step in the **Step name** field.
4. Select if you want to save the step locally or if you want to share it.
* **Local**: The annotations will be saved locally into the transformation.
* **Shared**: Allows you to select, create, or rename a shared group of annotations for use by PDI users.
5. (Optional) Enter a description for the annotations in the **Description** field.
6. Select available fields for annotation.
1. Click **Select Fields** to open the Select Fields to Annotate dialog box.
2. Double-click the fields in the **Available Fields** list to add them to the Selected Fields list. Optionally, you can use the arrows to move one or more fields to the **Selected Fields** list.
3. When finished, click **OK** to close the dialog box.
The selected fields now display in the **Annotations** table featuring the following columns:
| Column | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------- |
| **Field** | Lists the names of the fields selected for annotation. |
| **Model Action** | Specifies which model action is being taken: **Create Measure**, **Create Attribute**, or **Link Dimension**. |
| **Summary** | Displays a summary of that specific annotation. |
7. Now you can annotate the selected fields.
1. To annotate a field, double-click it in the **Annotations** section. Optionally, you can select the field in the **Annotations** table and then click the **Edit** (pencil) icon in the upper-right corner.
The Annotate dialog box appears for the selected field.
2. In the **Actions** field, click the drop-down arrow to select one of the following actions.
For help on selecting values, select the topic link beside each action.
* Create Measure. See [Creating measures on stream fields](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-measures-on-stream-fields).
* Create Attribute. See [Creating attributes](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-attributes).
* Link Dimension. See [Creating link dimensions](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/creating-link-dimensions).
3. You can use the **Previous** and **Next** buttons to navigate through the fields. When finished, click **OK** to continue or **Cancel** to close the dialog box without saving your annotations.
**Note:** You can remove a field from the **Annotations** section by selecting it in the table and then clicking the **Delete** (X icon) in the upper-right corner.
8. (Optional) You can create a calculated measure to add to the model.
1. Click the **Add Calculated Measure** button to open the Annotate dialog box.
2. Fill in the following fields:
| Field | Description |
| --------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Measure Name** | Enter the name of the calculated measure you are creating. |
| **Format** |
Specify how you want your calculated measure to appear in a report, such as currency, general number, or percentage. Use the drop-down arrow to select a format from a system-defined list, or type in the field to enter a custom format. For example, to display the measure as a percentage, you might select 0.00 %.
See the Pentaho Business Analytics document for more information on selecting the appropriate format string.
|
| **Formula** | Enter the formula of your calculated measure. This is an MDX statement. For more information, see the **Pentaho Business Analytics** document. |
| **When calculating subtotals use this formula** check box | (Optional) Select this check box if you want this calculated measure to be used in calculations of subtotals in your reports. |
| **Hide this calculated measure in the model** | (Optional) Select this check box if you want to hide this calculated measure in the data model. When selected, the calculated measure will be a part of the model, but will not be visible to users when the data source is opened in Analyzer. This check box is useful for calculated measures needed to build a proper data model, but not needed for analytic purposes. |
3\. Use the \*\*Previous\*\* and \*\*Next\*\* buttons to navigate through the fields. When finished, click \*\*OK\*\* to add the attribute to the annotations list or \*\*Cancel\*\* to close the dialog box without saving your annotations. The calculated measure displays in the \*\*Annotation\*\* table with the \*\*Model Action\*\* option \*\*Create Calculated Measure\*\* and a summary detailing the measure name and formula.
```
**Note:** The calculated measure is not validated until it is selected in Analyzer. Calculated measures will display as base measures in the **Available Fields** list in Analyzer.
```
9\. Click **Apply** to save your changes.
```
You can continue to create or edit annotations.
```
10\. When finished, click **OK** to save your changes and close the dialog box, or **Cancel** to discard your changes and close the dialog box.
# Creating measures on stream fields
A measure is a property that is used to make specific calculations, such as SUM, AVERAGE, COUNT, COUNT\_DISTINCT, MINIMUM, or MAXIMUM. You can create, as part of the Annotate Stream and Shared Dimension steps, a measure or series of measures to further refine your data model.
## Create a measure on a stream field
These steps guide you through creating a measure for a field in the stream using the Annotate Stream step.
1. If you have not done so already, complete steps 1-7 in the [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. Select **Create Measure** from the **Actions** menu.
The **Name - Value** table auto-populates with a list of annotation properties for the measure.
3. Enter or edit the values for each annotation component.
| Component | Description |
| -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Measure Name** | Enter a name for your annotation. |
| **Aggregation Type** |
Specify a format string or choose from a list of aggregation types:- SUM
AVERAGE
COUNT
COUNT\_DISTINCT
MINIMUM
MAXIMUM Note that COUNT and COUNT-DISTINCT are only available for non-numeric fields.
|
| **Format String** | Choose how this field should be formatted, such as currency, general number, or percentage. |
| **Description** | Enter a brief description for the annotation. This description will be available on the Pentaho Server. |
| **Hidden** | Set to `True` to prevent this annotation from displaying as an available measure in Analyzer. When set to `True`, this annotation is included in the model, but it is not visible to users when the model is published and opened in Analyzer. Set to `False` to allow this annotation to display as an available measure in Analyzer. When set to `False`, this annotation is visible to users when the model is published and opened in Analyzer. The **Hidden** property is useful for annotations needed to build a proper data model, but not needed for analytic purposes. For example, if you are in a multi-tenant environment, you may want to hide the **Tenant ID** annotation while still including it in the data model. |
4\. Use the **Previous** and **Next** buttons to navigate through the fields.
5. When finished, click **OK** to save your changes and close the dialog box, or **Cancel** to close the dialog box without saving your annotations.
# Creating attributes
An attribute is used to create a single level hierarchy, using either a field or existing hierarchy level.
## Create an attribute on a field
These steps guide you through creating an attribute using the Annotate Stream step.
1. If you have not done so already, complete steps 1-7 in [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. Select **Create Attribute** from the **Actions** menu.
The **Name - Value** table auto-populates with a list of annotation properties for the attribute.
3. Enter or edit the value for each annotation component.
| Property | Description |
| ---------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Attribute Name** | Enter a name for your attribute. This is a required field. |
| **Time Level Type** |
For time level attributes, specify a time level type from the drop-down menu. Options may include:- TimeYears
TimeHalfYears
TimeQuarters
TimeMonths
TimeWeeks
TimeDays
TimeHours
TimeMinutes
TimeSeconds This is a required field for Time Level attributes.
|
| **Time Source Format** |
For time level attributes, specify a time format for the attribute to appear in. Options are dependent on the selected time level type. For example, if you select TimeYears in the Time Level Type field, then you can only select yyyy or yy in the Time Source Format field. This is a required field for Time Level attributes.
For geo attributes, specify the geo type from the drop-down menu. Options may include:- Location
Country
City
State
County
Postal\_Code
Continent
Territory This is a required field for Geo Type attributes.
|
| **Latitude** |
If you select a Geo Type of Location, select an existing value from the drop-down menu.
If you do not select values for Latitude and Longitude, then the system will attempt to assign approximate values for these two fields.
|
| **Longitude** |
If you select a Geo Type of Location, select an existing value from the drop-down menu.
If you do not select values for Latitude and Longitude, then the system will attempt to assign approximate values for these two fields.
|
| **Ordinal Field** | Select a field to use for sorting members in ascending order. For example, months will display in chronological order, such as January, February, etc. |
| **Format** |
Choose how this attribute should be formatted, such as currency, general number, percentage, or date. Use the drop-down arrow to select a format from a system-defined list, or type in the field to enter a custom format. For example, to display the attribute as a simple date, you might select MM/dd/yy. Note that the Format field only displays when the value for the attribute is a number or a date.
See Pentaho Business Analytics for more information on selecting the appropriate format.
|
| **Description** | Enter a description for the attributes. This description is visible when a user mouses over an attribute in Analyzer. |
| **Parent Attribute** | If you want to create multi-level hierarchies, select a parent attribute for the attribute you are creating. This relationship will display in the **Summary** column for the annotation on the Annotate Stream dialog box. For example, if you are working with Month, then you can select **Year** as the parent attribute. Double-clicking **Year** in the model displays **Months**. |
| **Dimension** | Select an existing dimension or create a new dimension for the attribute. This is a required field. |
| **Hierarchy** | Select an existing hierarchy or create a new hierarchy for the attribute. This field may be left blank. |
| **Is Unique** |
Set to True when member names do not repeat under different parents. Setting this field to True will optimize SQL generation in Mondrian.
Set to False when member names may repeat under different parents.
|
| **Hidden** |
Set to True to prevent this attribute from displaying as an available level in Analyzer. When set to True, this attribute is included in the model, but is not visible to users when the model is published and opened in Analyzer.
Set to False to allow this attribute to display as an available level in Analyzer. When set to False, this annotation is visible to users when the model is published and opened in Analyzer.
The Hidden property is useful for attributes needed to build a proper data model, but not needed for analytic purposes. For example, if you are in a multi-tenant environment, you may want to hide the Tenant ID attribute while still including it in the data model.
|
\- For all annotations, enter the \*\*Attribute Name\*\* and select the \*\*Dimension\*\* type. Optionally, you can enter an ordinal field, write a description, choose a hierarchy, and designate if the attribute \*\*Is Unique\*\*.\
\- For Time Level attributes, select a \*\*Time Level Type\*\*, a \*\*Time Source Format\*\*, and optionally set a parent attribute if you want to create a relationship between two time attributes.\
\- For Geo attributes, select a \*\*Geo Type\*\* and optionally set a parent attribute if you want to create a relationship between two geo attributes.\
4\. Use the \*\*Previous\*\* and \*\*Next\*\* buttons to navigate through the fields. When finished, click \*\*OK\*\* to add the attribute to the annotations list or \*\*Cancel\*\* to close the dialog box without saving your annotations.
# Creating link dimensions
Dimensions organize attributes and measures so that users can answer business questions quickly. The most common dimension used is a time dimension, but dimensions are also used to classify people (such as customers), geographical places, and product lines.
Shared Dimensions are dimensions which can be shared for later use. Shared dimensions are particularly useful if you are connected to a Pentaho Repository or your workstation is shared by different users. They allow you to define and save your annotations, and then share them with other users. The Link Dimension annotation allows you to reuse a previously created [shared dimension](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-shared-dimension-step-for-sdr). For example, if you want to create a generic dimension for reuse, such as a standard time dimension, you can create a Shared Dimension and then link directly to that dimension's key. Linking to the dimension key will import all hierarchies from the Shared Dimension into your data model, resulting in a star schema which joins the shared dimension table to the fact table.
When creating a shared dimension, you will need to select a field to use as a Dimension Key which allows you to identify and use the Shared Dimension in your transformation. For example, in the Shared Dimension step, you can select the field **Year** and annotate it as a **Dimension Key**. You may also select the field **Year** again along with the fields **Month** and **Week** to annotate as attributes. Then, in theAnnotate Stream step, select the field **Year** to annotate as a **Link Dimension**. Here, select the Shared Dimension previously created using the **Year** field. All annotations defined in the Shared Dimension will be populated in run time. When the model is published, the annotations **Year**, **Month**, and **Week** will be in the model.
**Note:** If you are using link dimensions as part of your Streamlined Data Refinery (SDR) process, be aware that link dimension annotations are dependent on shared dimensions, and shared dimensions cannot use a data service for their source. Therefore, if you select a Pentaho Data Service as the source of your model in the [Build Model job entry](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Advanced%20Pentaho%20Data%20Integration%20topics/Advanced%20topics%20\(Pentaho%20Data%20Integration%20overview\)/Work%20with%20the%20Streamlined%20Data%20Refinery/Use%20the%20Streamlined%20Data%20Refinery/Building%20blocks%20for%20the%20SDR/Use%20the%20Build%20Model%20job%20entry%20for%20SDR=GUID-7FAB840B-9B79-440E-AE59-31C3F5B5EACA=4=en=.md), the link dimensions which you have defined here will not be linked to your model. See [Pentaho Data Services](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/pentaho-data-services) for more information about data services.
## Create a link dimension
This step assumes you have created a shared dimension which you want to use.
These steps guide you through creating a link dimension using the Annotate Stream step.
1. If you haven't done so already, complete steps 1-7 in [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. Select **Link Dimension** from the **Actions** drop-down menu. The **Name - Value** table auto-populates with a list of annotation properties for the dimension.
| Component | Description |
| ---------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Dimension Name | Enter a name for the dimension. Note that you can overwrite the dimension name set in the [Shared Dimension](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-shared-dimension-step-for-sdr) step. |
| Shared Dimension | Enter or select a previously created **Shared Dimension** to link to from the drop-down menu. |
3. Use the **Previous** and **Next** buttons to navigate through the fields. When finished, click **OK** to save your annotations and close the dialog box, or **Cancel** to close the dialog box without saving your annotations.
Remember that before your link dimension can be used by others, you must create a dimension key for the shared dimension to link to. When creating annotations, you can use a single field more than once, so you will be able to use the same field for both the key and to create annotations.
## Create a dimension key
Before your link dimension can be used by others, you must create a dimension key for the shared dimension to link to.
**Note:** You can create multiple annotations on the same field. For example, you might want to create an attribute and a dimension key on the same field, such as Year, in your time dimension.
1. If you haven't done so already, complete steps 1-7 in [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. From the list in the **Annotations** table, choose a field to use as the dimension key by double-clicking it.
For example, you may want to use the field `Calendar Year` as the dimension key for the shared dimension `Year`. Optionally, you can select the field in the **Annotations** table and then click the **Edit** (Pencil icon) in the upper-right corner.
The Annotate dialog box appears for the selected field.
3. Select **Create Dimension Key** from the **Actions** drop-down menu for the field.
The **Name - Value** table auto-populates with the name and value of the dimension key. You can edit the value to use as the dimension key.
4. Click **OK** to save the annotation and close the dialog box.
In the [Shared Dimension](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-shared-dimension-step-for-sdr) dialog box, the selected field will now appear with the summary that it is the key for the shared dimension. For example, the field `Calendar Year` may display `Calendar Year is key for dimension Year` in the **Summary** column.
# Creating annotation groups
Annotation groups are useful when data sources, such as a weblog table, are reused in many transformations. Whenever this table is used, you can link to the shared annotation group to get model information on each table field. If the table were to ever change, then the annotations would only need to be updated in one place.
You can create multiple annotations based on the same annotation group by copying the group, and then saving it with a different name. You can do this as many times as you need to make a series of related annotation groups, such as annotations for time dimensions.
You must create a group to save your annotations. This group can be saved as just a local group or as a shared group. When the group is saved [locally](broken://pages/Totxzd2FSdqzglSEQF41), it is saved to the transformation on your machine. When you select **Shared**, it will also be stored in the metastore and available to other users.
## Create an annotation group for sharing with other users
This task assumes you are in the transformation canvas of the PDI client.
If you want to share your annotation group with other users, select **Shared** and ensure the annotation group has a unique name. Once you click **Apply**, the group will be available to other users for creating PDI jobs by selecting from the **Shared** menu in the Annotate Stream step.
1. Complete steps 1-3 in the [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. Select **Shared**, then click on the **Add Annotation Group** (Plus Sign) next to the drop-down field.
3. Enter a name for your annotation group in the **Shared** field, and then click **Select Fields** to begin creating annotations to populate the group. See steps 6-8 in [Use the Annotate Stream Step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step) for more information on selecting and annotating fields.
4. When you are done, click **Apply** to save your annotations.
## Create an annotation group locally
This task assumes you are in the transformation canvas of the PDI client.
You can create an annotation group and save it locally.
1. Complete steps 1-3 in the [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step).
2. Select **Local**. Your annotation group will only be saved into your transformation. Any user running this transformation can see and use the annotations group.
3. Click **Select Fields** to begin creating annotations to populate the group. See steps 6-8 in [Use the Annotate Stream step](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step/use-the-annotate-stream-step) for more information on selecting and annotating fields.
4. When you are done, click **Apply** to save your annotation group locally.
If you later decide that you want to share the annotation group, you can reopen it and select the **Shared** radio button, then click **Apply**. The group will then be shared to the metastore and be available to other users.
# Using the Shared Dimension step for SDR
The Shared Dimension step works in much the same way as the [Annotate Stream](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-annotate-stream-step) step, refining your data through the creation of dimensions that can be shared for later use. You must designate one annotation in the Shared Dimension step as the dimension key.
You can create multiple annotations on the same field, such as an attribute and a dimension key. For example, in the Shared Dimension step, you can select the field **Year** and annotate it as a dimension key. You can also select the field **Year** again to annotate it as an attribute. You can also create multiple annotations to modify the same data model.
The transformation that builds a shared dimension should run before the [Build Model job entry](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/use-the-build-model-job-entry-for-sdr), or in a separate job that runs before Build Model. Here is an example transformation for a shared dimension.
After you are done with your data model, you are ready to [publish](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/using-the-publish-model-job-entry-for-sdr) it.
## Metadata injection support
All fields of this step support metadata injection. You can use it with [ETL metadata injection](/pdia-data-integration/pdi-transformation-steps-reference-overview/etl-metadata-injection) to pass metadata to your transformation at runtime.
When using metadata injection with the Shared Dimension step, the system loads the currently defined annotations in the Shared Dimension step, whether or not the shared dimension name is a match. The new set of annotations is saved to the transformation and in the metastore. If a shared dimension with the same name already exists in the metastore, it is overwritten with the new shared dimension.
## Create a shared dimension
This section describes how to create a shared dimension.
**Note:** Be sure to create one annotation to serve as the dimension key.
1. In the **Design** tab, click the **Flow** folder, and then double-click the **Shared Dimension** step icon. Alternatively, you can drag the step icon on to the transformation canvas.
2. Double-click the **Shared Dimension** icon to open the Shared Dimension dialog box.
3. Enter a name for the step in the **Step name** field.
4. If you are basing your new dimension on an existing shared dimension, select it from the **Shared Dimension Name** drop-down list of available shared dimensions, or enter a new name for it.
5. Select a source to associate your dimension from the **Data Provider Step** drop-down list.
The following steps are valid sources:
* Combination Lookup
* Database Lookup
* Dimension Lookup
* Insert/Update
* Table Output
* Vertica Bulk Loader
6. Enter a description of the shared dimension in the **Description** field.
7. Select available fields for annotation.
1. Click the **Select Fields** button to open the Select Fields to Annotate dialog box.
2. Double-click the fields in the **Available Fields** list to add them to the**Selected Fields** list.
For example, you might select **Year** to annotate as your dimension key and **Year**, **Month**, and **Week** to annotate as attributes Optionally, you can use the arrows to move one or more fields to the **Selected Fields** list.
3. When finished, click **OK** to close the dialog box. The selected fields now display in the **Annotations** table featuring the following columns:
| Column | Description |
| ---------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Field** | Lists the names of the fields selected for annotation. |
| **Model Action** |
Create a dimension key: the selected field will be designated as the dimension key for the shared dimension.
|
| **Summary** | Displays a summary of that specific annotation. |
8\. Now you can create \[annotations]\(../Using%20the%20Annotate%20Stream%20step.md) for the selected fields.
```
1. To annotate a field, double-click it in the **Annotations** section. Optionally, you can select the select the field in the **Annotations** table and then select the **Edit** \(Pencil icon\) in the upper-right corner.
The Annotate dialog box appears for the selected field.
2. In the **Actions** field, click the drop-down arrow to select one of the following actions:
- **Create Attribute**. See [Creating attributes](../Using%20the%20Annotate%20Stream%20step/Use%20the%20Annotate%20Stream%20step/Creating%20attributes.md).
- **Create Dimension Key**. See [Create a dimension key](../Using%20the%20Annotate%20Stream%20step/Use%20the%20Annotate%20Stream%20step/Creating%20Link%20Dimensions%20for%20SDR/Create%20a%20dimension%20key.md).
3. You can use the **Previous** and **Next** buttons to navigate through the fields. When finished, click **OK** to continue or **Cancel** to close the dialog box without saving your annotations.
**Note:** You can remove a field from the **Annotations** section by selecting it in the table and then clicking the **Delete** \(X icon\) in the upper-right corner.
```
9\. Click **Apply** to save changes you made to an annotation. Click **OK** to continue or **Cancel** to close the dialog box.
This is an example of the Shared Dimensions dialog box.
## Create a dimension key in Shared Dimension step
To use a shared dimension, you will need to designate one annotation in the Shared Dimension step as the dimension key. Since you can create multiple annotations on the same field, you can use a selected field as an annotation and as the dimension key. For example, you might want to create an attribute and a dimension key on the field Year in your time dimension.
1. If you have not done so already, complete steps 1-8 of [Create a shared dimension](broken://pages/sVi0cwNqtjOGwoINDzzV).
2. Select **Create Dimension Key** from the **Actions** menu.
The **Name - Value** table auto-populates with the **Dimension** attribute.
3. Enter or select a dimension in the **Value** field for **Dimension**. The selected field will be used as the key for this dimension.
4. Click **OK** to finish or click **Cancel** to exit.
In the Shared Dimension dialog box, the selected field will now appear with the summary that it is the key for the shared dimension. For example, the field `'Calendar Year'` may display `'Calendar Year is key for dimension Year'` in the **Summary** column.
You can use a single field more than once, so you will be able to use the field you selected here for both the key and to create annotations.
# Using the Publish Model job entry for SDR
The Publish Model job entry allows you to publish the data model created with the [Build Model](/pdia-data-integration/extracting-data-into-pdi/work-with-the-streamlined-data-refinery/use-the-streamlined-data-refinery/building-blocks-for-the-sdr/use-the-build-model-job-entry-for-sdr) job entry so it is available for use on the Pentaho Server. Before publishing the model, you will have the option of sharing the data model with everyone, or a specific user or role.
When you are building your job in Pentaho Data Integration (PDI), you will need to place the Build Model job entry before the Publish Model job entry, as shown here in this example, to publish the model properly.
After it is published to the Pentaho Server, the Data Source Wizard data source has the same name as the model in the Build Model job entry. Note that once the model is published, you won't be able to use the Data Source Wizard to edit the connection information since the connection information for this model is stored in the Build Model job entry in PDI.
## Use the Publish Model job entry
This task assumes you are in the job canvas in Pentaho Data Integration. You must have permissions to **Publish Content** and **Manage Data Sources** in the Pentaho Server in order to use this job entry. If you are using a data service as the source in your Build Model job entry, you must be connected to a Pentaho Repository to successfully publish your model.
Use this job entry to publish the data source created with the Build Model job entry.
1. In the **Design** tab, click the **Modeling** folder, and then double-click the Publish Model job entry. Alternatively, you can drag the job entry icon to the job canvas.
2. Double-click the **Publish Model** icon to open the Publish Model dialog box.
3. Enter a name for the entry in the **Entry name** field.
4. (Optional) Select the **Replace Existing Published Model** check box to overwrite an existing Data Source Wizard data source and database connection.
**Note:** To successfully complete an SDR job, it is recommended that you select this check box. If this check box is cleared and you attempt to publish a model with the same name, the Publish Data Source job entry will fail.
5. Fill in or edit the fields in the **Pentaho Server Connection** section and then test your connection:
| Option | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **URL** | The base URL string used to connect to the server. |
| **User Name** | The user name required to access the server. |
| **Password** | The password associated with the provided user name which is passed during the authentication process. |
| **Test Connection** | Click to test the connection to the Pentaho Server using the information provided in the above fields. When you click this button, the system will also check that the associated user is granted the **Publish Content** and **Manage Data Sources** permissions. |
6. Fill in or edit the fields in the **Share** section:
| Field | Description |
| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **Grant Access To** |
Grant access to the data model to everyone, specific users, or specific roles.
The permission granted is read-only. From the drop-down menu, you can select Everyone, User, or Role. If you select User or Role, theUser/Role Name field is available.
You can use variables to populate this field. The variable can contain one of three values: everyone, user, or role. These values must be lower-case to work properly.
|
| **User/Role Name** | Enter the name of the user or role who will have access to the model. |
7\. When finished, click \*\*OK\*\* to save your changes and close the dialog box, or click \*\*Cancel\*\* to discard your changes and close the dialog box.
This is an example of the Publish Model dialog box.
# Connecting to a Hadoop cluster with the PDI client
To connect to a Hadoop cluster, you must access a driver, create a named connection, then configure and test the connection. A named connection is information, including the IP address and port number, used to connect to the Hadoop cluster which is then stored by the name you assign to the connection for later use. You can create named connections to any supported vendor cluster and vendor version.
After you have a named connection set up, you can edit or duplicate that connection. For example, if you want to use a configuration with different security credentials, you can duplicate a connection, then edit the security settings on the copy. Named connections are useful when you move the jobs and transformations from a development server to a production server because you only need to update the connection information for the cluster name in the Hadoop Clusters dialog box. The jobs and transformations use the new connection information from the named connection.
# Audience and prerequisites
The audience for this article is ETL developers, data engineers, and data analysts.
Before you begin, verify that the Hadoop administrator has set up your user account on the cluster and granted permissions to access the applicable HDFS directories. You need access to the home directory and any other directories required for the tasks.
Pentaho ships with a default Apache Hadoop driver already installed. Supported versions of other drivers, including Amazon EMR, Apache Vanilla, Cloudera (CDP), and Google Dataproc, must be downloaded from the [Support Portal](https://support.pentaho.com/hc/en-us). You must have a driver for each vendor of Hadoop for connecting to each cluster, which should be preinstalled and available for selection before adding a cluster connection.
* To install a driver for the PDI client, see [Install a driver for the PDI client](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/install-a-driver-for-the-pdi-client).
When drivers for new Hadoop versions are released, you can download them from the [Support Portal](https://support.pentaho.com/hc/en-us) and then add them to Pentaho to connect to the new Hadoop distributions. Install these drivers using the procedure specified in the **Install Pentaho Data Integration and Analytics** document.
Verify that the Hadoop administrator has configured the Pentaho Server to connect to the Hadoop cluster on the computer. Ask the Hadoop administrator to provide you with a copy of the `site.xml` files from the cluster and the following information:
* Distribution and version of the cluster.
* IP addresses and port numbers for HDFS, JobTracker, and Zookeeper (if used).
* Kerberos and cluster credentials if you are connecting to a secured cluster.
* Oozie URL (if used).
# Using the pre-installed Apache Hadoop driver
You can access and use the installed Apache Hadoop driver for HDFS copy file operations as well as for executing input and output transformations and jobs. The driver works with both secure and unsecured clusters. Because the driver pre-installed, you do not have to install a KAR file.
The supported big data steps in Pentaho include:
* [Avro Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-input)
* [Avro Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-output)
* [ORC Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-input)
* [ORC Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-output)
* [Parquet Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-input)
* [Parquet Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-output)
Both operating system file browsers and the Pentaho virtual file system browsers are supported, as well as basic HDFS and VFS operations. For more information, see [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser).
**Note:** Only Hadoop clusters that conform with standard Hadoop connection rules work with the Apache Hadoop Driver. For example, while EMR clusters may work, MapR does not work with this driver because the connection rules for MapR are not standard. The Apache Hadoop Driver is not intended to support higher level Hadoop operations such as Hive, HBase, Sqoop, and Oozie. If you require these operations, install the KAR file for the applicable vendor.
# Using the Apache Vanilla Hadoop driver
Although the Apache Vanilla Hadoop driver is not shipped as installed with PDI as with the Apache Hadoop driver, once you install this driver, you can directly connect to plain Hadoop clusters, commonly known as Apache Vanilla Hadoop clusters. As a best practice when working with plain Hadoop clusters, use the Apache Vanilla Hadoop driver for expanded functionality beyond the pre-installed Apache Hadoop driver for both secure and unsecured environments. For plain Hadoop clusters, the Apache Vanilla Hadoop driver supports a range of services including HDFS file copy, big data file formats (such as ORC, Avro, and Parquet), Hive operations, MapReduce jobs, and Sqoop.
# Install a driver for the PDI client
Before you can add a named connection to a Hadoop cluster in Pentaho Data Integration (PDI), you must first install the vendor-specific Big Data plugin for your Hadoop cluster.
{% hint style="info" %}
If you are connected to the Pentaho Repository and you plan to run Hadoop-related transformations on the Pentaho Server, install and configure the same driver on the Pentaho Server.
See the Manually install a driver for the Pentaho Server subsection of the [Set up Pentaho to connect to a Hadoop cluster](/install/pentaho-configuration#set-up-pentaho-to-connect-to-a-hadoop-cluster) section.
{% endhint %}
{% hint style="info" %}
**Note:** This task assumes that you downloaded the vendor-specific Big Data plugin from the [Support Portal](https://support.pentaho.com). The files are named `pentaho-big-data-ee-plugin--.zip`.
{% endhint %}
The vendor-specific plugin files contain vendor abbreviations in the `.zip` file name:
* Amazon EMR (`emr`)
* Apache vanilla Hadoop (`apachevanilla`)
* Azure HDInsight (`hdi`)
* Cloudera Data Platform (`cdp`)
* Google Dataproc (`dataproc`)
Perform the following steps to install the Big Data plugin in the PDI client:
1. Go to the `data-integration/plugins` folder in your PDI installation.
2. Extract the `pentaho-big-data-ee-plugin--.zip` file into the `plugins` folder.
3. Restart PDI.
In the **New cluster** and **Import cluster** dialog boxes, the **Current configured driver** field shows the driver provided by the plugin you installed.
### Next steps
* To create a named connection, see [Adding a cluster connection](broken://spaces/iFWuQjAZNxh1EoQbRnsT/pages/MxKCIWFUTMyitKDhHy3j).
* For end-to-end connection setup context, see [Connecting to a Hadoop cluster with the PDI client](broken://spaces/iFWuQjAZNxh1EoQbRnsT/pages/cSGwDQkv1NAqOEXGHqhs).
# Configure CDP Public Cloud cluster with the PDI client
If you are working with a CDP Public Cloud cluster in the PDI client, you must perform some additional configuration steps.
CDP Public Cloud’s datahub cluster offerings have gateway nodes for installing any third-party applications that need access to those clusters. Before you begin, install gateway nodes for running Pentaho. Use CentOS to deploy these gateway nodes.
For additional information about how to connect to the Cloudera Data Platform (CDP), see the advanced settings section in the **Install Pentaho Data Integration and Analytics** document.
Perform the following steps to configure the CDP Public Cloud cluster to work with the PDI client:
1. Use `sudo su` to log in as a `root` user and add users to `sudoers` list, as shown in the following example code:
```
usermod -a -G wheel username
```
**Note:** To install packages, we need `sudo` access. To attain that access, log (SSH) into the cluster gateway node as the `cloudbreak` user.
2. Install `tigervnc-server` to access the gateway node with the user interface, as shown in the following example:
```
sudo yum install tigervnc-server
vncpasswd
vncserver
```
The `vncpasswd` command is used to set a password for the VNC login and the `vncserver` command is used to start the VNC viewer.
3. Install `libwebkitgtk-1.0.0` in the gateway node. As this module is deprecated and not present in CentOS repository, you must install `webkitgtk` using a third-party RPM called `nux-dextop`, as shown in the following example commands:
```
sudo rpm --import http://li.nux.ro/download/nux/RPM-GPG-KEY-nux.ro
sudo yum -y install epel-release && sudo rpm -Uvh http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-05.el7.nux.noarch.rpm
sudo yum install webkitgtk
```
4. Verify Kerberos is pre-installed and set up on the cluster:
1. Download the keytab for the Cloudera Management Console.
2. Upload the keytab to the gateway node.
3. Run either one of the following commands:
* ```
```
kinit
````
- ```
kinit -kt username
````
# Adding a cluster connection
You can add named connections manually or by importing them. If you are using high availability (HA) clusters, you must manually add the connection information in the New cluster dialog box to create your connection.
If you are connected to the Pentaho Repository when you add a new cluster connection, you and other users can reuse the connection. Cluster connections are registered by all transformations and jobs in the repository and are loaded during execution unless the connections cannot be found.
If you are not connected to the Pentaho Repository when you create the connection, then only you can reuse the connection, as it is registered exclusively by the local file system for transformations and jobs.
**Note:** Security is set up on a per-user basis. Security information is not stored in the repository.
* To add a new cluster connection using import, see [Import a cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/add-a-cluster-connection-by-import-add-hadoop-cluster-connection)
* To add a new cluster connection manually, see [Manually add a cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/add-a-cluster-connection-manually-add-hadoop-cluster-connection)
# Import a cluster connection
You can add a cluster by importing the `site.xml` files from an existing cluster. Perform the following steps to import a cluster connection.
1. In the PDI client, create a new transformation or job or open an existing transformation or job.
2. Click the **View** tab and then right-click the **Hadoop Clusters** folder.
3. Click **Import cluster**.
The **Hadoop Clusters** dialog box appears.
4. Enter a user-defined name for the cluster connection in the **Cluster name** field.
Valid cluster names may include uppercase and lowercase letters, and numbers. In addition, the only special character allowed is the dash (`-`). To ensure a valid cluster name, do not use any other symbols, punctuation characters, or blank spaces.
After you create the connection, you can locate this named connection in the **View** tab on the PDI client.
**Note:** If the **Cluster name** is already in use, you will be notified that proceeding will overwrite an existing cluster, which cannot be undone. In this case:
* Click **Cancel** then enter a unique name to create the cluster.
* Click **Yes, Overwrite** to overwrite the existing cluster.
5. Use the **Driver** and **Version** options to select the distribution of Hadoop on the cluster and its version number. The [Support Portal](https://support.pentaho.com/hc/en-us) provides supported drivers that you can download and install.
6. Click **Browse to add file(s)** and browse to the directory containing the `site.xml` files that were provided to you by the cluster administrator.
The required files include:
* `hive-site.xml`
* `mapred-site.xml`
* `yarn-site.xml`
* `core-site.xml`
* `hbase-site.xml`
* `hdfs-site.xml`
* `oozie-site.xml` (if you are using Oozie in the configuration)
7. Click **Open**.
The **Site XML files** section displays the files you selected.
8. If you are connecting to a secure cluster, then enter the credentials in the **Username** and **Password** fields in the **HDFS** section.
9. Click **Next** and then specify the security option for the cluster.
* If the Hadoop cluster is non-secure, select **None** and then click **Next** to [test the connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/test-the-cluster-connection-add-hadoop-cluster-connection).
* If your Hadoop cluster is secure, you need to add security to the cluster connection. See [Add security to cluster connections](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/secure-cluster-connections-add-hadoop-cluster-connection) for instructions.
# Manually add a cluster connection
You can manually create a cluster connection by supplying the `site.xml` files, which are typically provided by the cluster administrator.
**Note:** If you are using high availability (HA) clusters, you must manually add the connection information to create the cluster connection.
Perform the following steps to manually add a cluster connection.
1. In the PDI client, create a new job or transformation or open an existing one.
2. Click the **View** tab and then right-click the **Hadoop Clusters** folder.
3. Click **New cluster**.
The Hadoop Cluster dialog box appears.
4. Enter the connection information from the cluster administrator in the Hadoop Cluster dialog box.
**Note:** As a best practice, use Kettle variables for each connection parameter value to reduce risks associated with running jobs and transformations in environments that are disconnected from the repository.
| Option | Description |
| --------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Cluster Name** |
Enter the name you want to assign to the cluster connection. Note: Valid cluster names may include uppercase and lowercase letters and numbers. In addition, the only special character allowed is a dash (-). To ensure a valid cluster name, do not use any other symbols, punctuation characters, or blank spaces.
After you create the connection, you can locate this named connection in the View tab on the PDI client.
|
| Current Configured **Driver** and **Version** | Read only information about distribution of Hadoop on the cluster and its version number. |
| **Site XML files** |
Enter the location of the site.xml files provided by the cluster administrator. Click \*\*Browse to add file(s)\*\*and browse to the directory containing the site.xml files. Pentaho creates the applicable directory on the machine where the PDI client is located and copies the site.xml files to that directory.
Alternatively, if you leave this option blank, Pentaho creates the directory for the distribution and version of Hadoop you selected in the Driver and Version options. You must then copy the site.xml files to that directory.
|
| **HDFS** |
Provide the following information for the HDFS node:- Enter the Hostname for the HDFS node in the Hadoop cluster.
Enter the Port for the HDFS node in the Hadoop cluster.
Note that if the cluster is enabled for high availability (HA), then a port number is not needed, and you should clear the port number.
Enter the Username and Password for the HDFS node, which are typically provided by the cluster administrator.
|
| **JobTracker** |
If you have a separate JobTracker node, provide the following information:- Enter the Hostname for the JobTracker node in the Hadoop cluster.
Enter the Port for the JobTracker node in the Hadoop cluster.
|
| **ZooKeeper** |
If you have a Zookeeper node and want to connect a Zookeeper service, provide the following information:- Enter the Hostname for the Zookeeper node in the Hadoop cluster.
Enter the Port for the Zookeeper node in the Hadoop cluster.
|
| **Oozie** | Enter the Oozie client address in the **Hostname** field. Supply this URL only if you want to connect to the Oozie service. |
| **Kafka** | Enter the host:port pair(s) for the initial connection to the Kafka cluster in the **Bootstrap servers** field. Use a comma-separated list for multiple servers, for example, `host1:port1,host2:port2`. Although there is no need to include all servers used for Kafka, you might want to include more than one in case a server is down. |
5\. Click \*\*Next\*\* and specify the security option for the cluster.
```
- If the Hadoop cluster is non-secure, select **None** and click **Next** to [test the connection](Test%20the%20cluster%20connection%20(Add%20Hadoop%20cluster%20connection).md).
- If the Hadoop cluster is secure, you need to add security to the cluster connection. See [Add security to cluster connections](Secure%20cluster%20connections%20(Add%20Hadoop%20cluster%20connection).md) for instructions.
```
# Add security to cluster connections
If you have a secure Hadoop cluster, the security options depend on the driver. All drivers have the **Kerberos** option. If you are using a Hortonworks driver, you can also select **Knox** as the security type. If you are connected to a Pentaho Repository, you can specify additional Kerberos options for secure impersonation. See the **Administer Pentaho Data Integration and Analytics** document for further information on secure impersonation.
If you are not sure what security type is set up for the Hadoop cluster, contact the cluster administrator.
**Note:** For Kerberos, you need the authentication user name and either a password or a keytab file. For Knox, you need the Gateway URL, user name, and password.
# Specify Kerberos security
Perform the following steps to specify the credentials for the Kerberos security.
**Note:** You can define different principal users for each of the named connections only if all the clusters for these connections are in the same Kerberos realm. See [MIT Kerberos Documentation](https://web.mit.edu/kerberos/krb5-devel/doc/admin/realm_config.html) for more information about Kerberos realms.
1. Select **Kerberos** as the security type.
2. Click **Next**.
3. Choose one of the following security methods and specify the Kerberos credentials you obtained from the cluster administrator:
* **Password**: Specify the **Authentication username** and **Password** options. Additionally, if you are connected to the Pentaho Repository and are using secure impersonation, specify the **Impersonation username** and **Password**. See **Try Pentaho Data Integration and Analytics** if the environment requires advanced settings, the server is on Windows, or when you are using a Cloudera Impala database for secure impersonation.
* **Keytab**: Specify the **Authentication username** and **Authentication Keytab** options. Click **Browse** to navigate to your keytab file. Additionally, if you are connected to the Pentaho Repository and are using secure impersonation, specify the **Impersonation username** and **Impersonation Keytab**. See **Try Pentaho Data Integration and Analytics** if the environment requires advanced settings, the server is on Windows, or when you are using a Cloudera Impala database for secure impersonation.
4. Click **Next** to test the connection. See [Test a cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/test-the-cluster-connection-add-hadoop-cluster-connection) for more information.
The **Test results** dialog box appears.
For each tested connection, the dialog box displays one of the following icons to indicate the results:
*  A green checkmark indicates the connection to the cluster service was successful.
*  A yellow caution symbol indicates the cluster service information was not supplied, so the test for that component was skipped.
*  A red circle-backslash indicates the connection failed. Check the connection information and then test the connection again. If you suspect a different issue, see the troubleshooting section in the **Install Pentaho Data Integration and Analytics** document or consult the cluster administrator.**Note:** You can click the drop-down arrow in the **Hadoop file system** test for more details.
5. Click **Finish**.
If no errors occur during the connection, PDI is successfully connected.
If you have errors, see the troubleshooting section in the **Administer Pentaho Data Integration and Analytics** document to resolve the issues or consult the cluster administrator, then run the test again. When no error messages are returned, the connection is properly configured.
# Test a cluster connection
Perform the following steps to test a cluster connection:
1. In the PDI client, select the **View** tab and navigate to the **Hadoop clusters** folder.
2. If needed, expand the **Hadoop clusters** folder then right-click the cluster you want to test and select **Test cluster**.
The **Test results** dialog box appears.
For each tested connection, the dialog box displays one of the following icons to indicate the results:
*  A green checkmark indicates the connection to the cluster service was successful.
*  A yellow caution symbol indicates the cluster service information was not supplied, so the test for that component was skipped.
*  A red circle-backslash indicates the connection failed. Check the connection information and then test the connection again. If you suspect a different issue, see the troubleshooting section in the **Install Pentaho Data Integration and Analytics** document or consult the cluster administrator.**Note:** You can click the drop-down arrow in the **Hadoop file system** test for more details.
3. Click **Finish**.
If no errors occur during the connection, PDI is successfully connected.
If you have errors, see troubleshooting section of the **Administer Pentaho Data Integration and Analytics** document to resolve the issues then test your connection again. When no error messages are returned, the connection is properly configured.
# Managing Hadoop cluster connections
After cluster connections are added, you can edit, copy, and test them as needed. Once a connection is no longer required, you can delete that named connection.
* [Edit Hadoop cluster connections](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/managing-hadoop-cluster-connections-connect-to-a-hadoop-cluster-with-the-pdi-client/edit-hadoop-cluster-connections)
* [Duplicate a Hadoop cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/managing-hadoop-cluster-connections-connect-to-a-hadoop-cluster-with-the-pdi-client/duplicate-a-hadoop-cluster-connection)
* [Test a cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/test-the-cluster-connection-add-hadoop-cluster-connection)
* [Delete a Hadoop cluster connection](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/managing-hadoop-cluster-connections-connect-to-a-hadoop-cluster-with-the-pdi-client/delete-a-hadoop-cluster-connection)
# Edit Hadoop cluster connections
How updates occur depend on whether you are connected to the repository.
* **If you are connected to a repository**
Hadoop cluster connection changes are registered by all transformations and jobs in the repository. The Hadoop cluster connection information is loaded during execution unless it cannot be found.
* **If you are not connected to a repository**
Hadoop cluster connection changes are registered by the local (file system) transformations and jobs. Note that changes to the Hadoop cluster connection are not updated in any transformations or jobs for the purpose of fallback unless they are saved again.
Perform the following steps to edit a Hadoop cluster connection:
1. Click the **Hadoop Clusters** folder in the **View** tab.
2. Right-click the existing connection, then select **Edit**. Optionally, you can double-click the existing connection.
The Edit cluster dialog box appears.
3. Make the changes, then click **Next**.
If the cluster is enabled for high availability (HA), then a port number is not needed, and you should clear the port number.
4. For the security type, select **None** and click **Next** or see [Add security to cluster connections](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/secure-cluster-connections-add-hadoop-cluster-connection) to add or edit security.
The Test results dialog box displays.
5. Click **Close** to save the changes.
# Duplicate a Hadoop cluster connection
You can duplicate a cluster connection. This task is useful if you want to test a change to a named connection without affecting the existing setup or if you want to add different security permissions.
To duplicate a cluster connection, perform the following steps:
1. Click the **Hadoop clusters** folder in the **View** tab.
2. Right-click an existing connection and select **Duplicate cluster**.
The Hadoop clusters (Edit cluster) dialog box appears.
3. Enter a different name in the **Cluster Name** field.
The system automatically adds `copy-of-` to the beginning of the cluster name.
4. Click **Browse to add files(s)**. Use the file browser to select the `site.xml` files you want to import.
**Note:** Duplicating a cluster connection copies the existing `site.xml` files to a new metastore directory. If you select `site.xml` files in this step, these files replace the copied `site.xml`.
5. Click **Next**.
6. For the security type, select **None** and click **Next** or see [Add security to cluster connections](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article/adding-a-cluster-connection-connect-to-a-hadoop-cluster-with-the-pdi-client/secure-cluster-connections-add-hadoop-cluster-connection) to add or edit security.
7. Click **Edit cluster** to open the **Edit cluster** dialog box
8. Make the applicable changes to the cluster configuration values, then click **Next**.
The **Congratulations** dialog box appears.
9. Click **Close**.
# Delete a Hadoop cluster connection
**CAUTION:**
If you delete a named connection, the deleted connection cannot be restored. You must recreate the connection.
To delete a Hadoop cluster connection in a transformation or job, perform the following steps:
1. Click the **Hadoop clusters** folder in the **View** tab.
2. Right-click the Hadoop cluster connection you want to delete and select **Delete cluster**.
A message appears asking whether you to confirm the operation.
3. Click **Yes, Delete**.
The cluster connection is deleted, including all security credentials.
# Connect other Pentaho components to a cluster
See the **Install Pentaho Data Integration and Analytics** document for information on advanced settings for connecting other Pentaho components.
# Connecting to Virtual File Systems
You can connect to most Virtual File Systems (VFS) through VFS connections in PDI. A VFS connection stores VFS properties for a specific file system. You can reuse the connection whenever you [access files or folders](#access-files-with-a-vfs-connection). For example, you can use an HCP connection in HCP steps without re-entering credentials.
With a VFS connection, you can set your VFS properties with a single instance that can be used multiple times. The VFS connection supports the following file systems:
* **Amazon S3/Minio/HCP**
* Simple Storage Service (S3) accesses the resources on Amazon Web Services. See [Working with AWS Credentials](https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/credentials.html) for Amazon S3 setup instructions.
**Note:** If a connectivity issue occurs with AWS / S3, perform either of the following actions:
* Set the Environment Variables for `AWS_REGION` or `AWS_DEFAULT_REGION` to the applicable Default Region.
* Set the correct Default Region in the shared configuration file (`~/.aws/config`) or the credentials file (`~/.aws/credentials`). For example:
See and for more information.
* Minio accesses data objects on an Amazon compatible storage server. See the [Minio Quickstart Guide](https://docs.min.io/docs/) for Minio setup instructions.
* HCP uses the S3 protocol to access HCP. See [Access to HCP REST](#access-to-hcp-rest) for setup details.
* **Azure Data Lake Gen 1**
Accesses data objects on Microsoft Azure Gen 1 storage services. You must create an Azure account and configure Azure Data Lake Storage Gen 1. See [Access to Microsoft Azure](#access-to-microsoft-azure).
**Note:** Support for Azure Data Lake Gen 1 is discontinued and limited to users with existing Gen 1 accounts. As a best practice, use Azure Data Lake Storage Gen 2. See [Azure](https://azure.microsoft.com/en-us/updates/action-required-switch-to-azure-data-lake-storage-gen2-by-29-february-2024/) for details.
* **Azure Data Lake Gen 2/Blob**
Accesses data objects on Microsoft Azure Gen 2 or Blob storage services. You must create an Azure account and configure Azure Data Lake Storage Gen 2 and Blob Storage. See [Access to Microsoft Azure](#access-to-microsoft-azure).
* **Google Cloud Storage**
Accesses data in the Google Cloud Storage file system. See [Google Cloud Storage](https://cloud.google.com/storage/docs) for more information on this protocol.
* **HCP REST**
Accesses data in the Hitachi Content Platform. You must configure HCP and PDI before accessing the platform. See [Access to HCP REST](#access-to-hcp-rest) for more information.
* **Local**
Accesses data in your local physical file system.
* **SMB/UNC Provider**
Accesses data in a Windows platform that uses the Server Message Block (SMB) protocol and Universal Naming Convention (UNC) string to specify the resource location path.
* **Snowflake Staging**
Accesses a staging area used by Snowflake to load files. See [Snowflake staging area](https://docs.snowflake.net/) for more information on this protocol.
After you create a VFS connection, you can use it with PDI steps and entries that support the use of VFS connections. If you are connected to a repository, the VFS connection is saved in the repository. If you are not connected to a repository, the connection is saved locally on the machine where it was created.
If a VFS connection is not available for your file system, you may be able to access it with the [VFS browser](#vfs-browser).
### Before you begin
You may need to set up access for specific providers before you start.
#### Access to Google Cloud
To access Google Cloud from PDI, you must have a Google account and a service account key file in JSON format. You must also set permissions for your Google Cloud accounts. To create service account credentials, see .
Perform the following steps to set up Google Cloud Storage access:
1. Download the service account credentials file from the Google API Console.
2. Create a system environment variable named **GOOGLE\_APPLICATION\_CREDENTIALS**.
3. Set the variable value to the full path of the JSON key file.
You can now access Google Cloud Storage from PDI.
#### Access to HCP REST
Hitachi Content Platform (HCP) is a distributed storage system that you can access through a VFS connection in the PDI client.
Within HCP, access control lists (ACLs) grant privileges for file operations. [Namespaces](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/System_administration/Introduction_to_Hitachi_Content_Platform/01_About_Hitachi_Content_Platform/) are used for logical groupings, permissions, and object metadata. For more information, see the [Introduction to Hitachi Content Platform](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/Tenants_and_Namespaces/Introduction_to_Hitachi_Content_Platform).
Perform the following steps to set up access to HCP:
{% hint style="info" %}
This process assumes you have tenant permissions and existing namespaces. See [Tenant Management Console](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/Tenants_and_Namespaces/General_administrative_information/03_Tenant_Management_Console).
{% endhint %}
{% hint style="info" %}
To create a successful VFS connection to HCP, configure object versioning in your HCP [Namespaces](https://knowledge.hitachivantara.com/Documents/Storage/Content_Platform/8.1.2/Tenants_and_Namespaces/Managing_namespaces).
{% endhint %}
1. Sign in to the HCP Tenant Management Console.
2. Click **Namespaces**, then select the namespace **Name** you want to configure.
3. On the **Protocols** tab, click **HTTP(S)**.
4. Verify these settings:
* **Enable HTTPS**
* **Enable REST API** with **Authenticated access only**
5. On the **Settings** tab, select **ACLs**.
6. Select **Enable ACLs**.
7. When prompted, click **Enable ACLs** to confirm.
HCP is now set up for access from the PDI client.
#### Access to Microsoft Azure
To access Azure services from PDI, create and configure the following:
* Azure Data Lake Gen 1, or
* Azure Data Lake Storage Gen2 and Blob Storage services
Enable the hierarchical namespace to maximize file system performance.
* Access requires an Azure account with an active subscription. See [Create an account for free](https://azure.microsoft.com/en-us/free/?ref=microsoft.com\&utm_source=microsoft.com\&utm_medium=docs\&utm_campaign=visualstudio).
* Access to Azure Storage requires an Azure Storage account. See [Create a storage account](https://docs.microsoft.com/en-us/azure/storage/common/storage-account-create?tabs=azure-portal).
### Create a VFS connection
Perform the following steps to create a VFS connection in PDI:
1. Start the PDI client (Spoon).
2. In the **View** tab of the Explorer pane, right-click **VFS Connections**, then click **New**.
The New VFS connection dialog box opens.
3. In **Connection Name**, enter a unique name. Optionally, add a **Description**.
The name can include spaces. Do not use special characters. Avoid `#`, `$`, `/`, `\`, `%`.
4. In **Connection Type**, select a type:
* **Amazon S3/Minio/HCP**
* **Azure Data Lake Gen 1**
* **Azure Data Lake Gen 2 / Blob**
* **Google Cloud Storage**
* **HCP REST**
* **Local**
* **SMB/UNC Provider**
* **Snowflake Staging**
5. In the connection details panel, set the options for your connection type.
You can add a predefined variable to fields that have the “insert variable” icon. Place your cursor in the field, then press Ctrl+Space. Variables must be predefined in kettle.properties. Runtime variables are not supported.
Access Key/Secret Key, then enter the Access Key and Secret Key, and optionally enter the Session Token.
Credentials File, then enter the Profile Name and the File Location.
Select the Default S3 Connection checkbox to make Amazon the default S3 connection.
|
| **Minio/HCP** |
Click S3 Connection Type and select Minio/HCP from the list to use a Minio/HCP S3 connection.
Minio accesses data objects on an Amazon compatible storage server. See the Minio Quickstart Guide for Minio setup instructions.
Enter the Access Key.
Enter the Secret Key.
Enter the Endpoint.
Enter the Signature Version.
Select the PathStyle Access checkbox to use path-style requests. Otherwise, Amazon S3 bucket-style access is used.
Select the Default S3 Connection checkbox to make Minio/HCP the default S3 connection.
|
| **Azure Data Lake Gen 1** |
Accesses data objects on Microsoft Azure Gen 1 storage services. You must create an Azure account and configure Azure Data Lake Storage Gen 1. See Access to Microsoft Azure.
The Authentication Type is Service-to-service authentication only.
Enter the Account Fully Qualified Domain Name.
Enter the Application (client) ID.
Enter the Client Secret.
Enter the OAuth 2.0 token endpoint.
|
| **Azure Data Lake Gen 2 / Blob** |
Accesses data objects on Microsoft Azure Gen 2 and Blob storage services. You must create an Azure account and configure Azure Data Lake Storage Gen 2 and Blob Storage. See Access to Microsoft Azure.
Select the Authentication Type: - Account Shared Key - Azure Active Directory - Shared Access Signature
Enter the Service Account Name.
Enter the Block Size (Min 1 MB to Max 100 MB). The default is 50.
Enter the Buffer Count (Min 2). The default is 5.
Enter the Max Block Upload Size (Min 1 MB to 900 MB). The default is 100.
Select the Access Tier. The default value is Hot.
When Authentication Type is:
Account Shared Key, then enter the Service Account Shared Key.
Azure Active Directory, then enter the Application (client) ID, Client Secret, and Directory (tenant) ID.
Shared Access Signature, then enter the Shared Access Signature.
|
| **Google Cloud Storage** |
Accesses data objects on the Google Cloud Storage file system. See Google Cloud Storage for more information on this protocol.
Enter the Service Account Key Location.
|
| **HCP REST** |
Accesses data objects on the Hitachi Content Platform. You must configure HCP and PDI before accessing the platform. You must also configure object versioning in HCP namespaces. See Access to HCP REST.
Enter the Host and Port.
Enter the Tenant, Namespace, Username, and Password.
Click More options, then enter the Proxy Host and Proxy Port.
Select whether to use Accept self-signed certificate. Default: No.
Select whether the Proxy is secure. Default: No.
|
| **Local** |
Accesses a file system on your local machine.
Enter the Root Folder Path or click Browse to set a folder connection. Optionally, use an empty path to allow access to the root directory and its folders.
|
| **SMB/UNC Provider** |
Accesses Server Message Block data using a Universal Naming Convention string to specify the file location.
Enter the Domain. The domain name of the target machine hosting the resource. If the machine has no domain name, use the machine name.
Enter the Port Number. Default: 445.
Enter the Server, User Name, and Password.
|
| **Snowflake Staging** |
Accesses a staging area used by Snowflake to load files. See Snowflake staging area for more information.
Enter the Host Name.
Enter the Port Number. Default: 443.
Enter the Database.
Enter the Namespace, User Name, and Password.
|
6. For all connection types except **Local**, enter the **Root Folder Path** for your VFS connection. Enter the full path to connect to a specific folder. Optionally, use an empty path to allow access to all folders in the root.
```
The default is the root and its folders in your local physical file system.
```
7. Optional: Click **Test** to verify the connection.
8. Click **OK**.
You can now use the connection in steps and entries that support VFS connections, such as Snowflake entries or HCP steps. For related information, see:
* [PDI and Snowflake](/pdia-data-integration/extracting-data-into-pdi/pdi-and-snowflake-cp)
* [PDI and Hitachi Content Platform (HCP)](/pdia-data-integration/extracting-data-into-pdi/pdi-and-hitachi-content-platform-hcp)
For general access details, see [Access files with the VFS browser](#access-files-with-the-vfs-browser).
### Edit a VFS connection
Perform the following steps to edit an existing VFS connection:
1. Right-click **VFS Connections** and select **Edit**.
2. In the Edit VFS Connection dialog box, select the pencil icon next to the section you want to edit.
### Delete a VFS connection
Perform the following steps to delete a VFS connection:
1. Right-click **VFS Connections**.
2. Select **Delete**, then **Yes, Delete**.
The deleted connection no longer appears under **VFS Connections** in the **View** tab.
### Access files with a VFS connection
After you create a VFS connection, you can use the VFS Open and Save dialog boxes to access files in the PDI client.
1. In the PDI client, select **File** > **Open URL** to open a file, or **File** > **Save as** to save a file.
The VFS Open or Save As dialog box opens.
2. In the left pane, select **VFS connection**, then navigate to your folders and files.
3. Optional: Click the navigation path to show and copy the Pentaho file path. See [Pentaho address for a VFS connection](#pentaho-address-for-a-vfs-connection).
4. Select the file and click **Open** or **Save**.
{% hint style="info" %}
If you are not connected to a repository, you can rename a folder or file. Click the item again to edit its name.
{% endhint %}
### Pentaho address for a VFS connection
The Pentaho address is the Pentaho virtual file system (`pvfs`) location within your VFS connection. When you browse in the file access dialog box, the address bar shows the path for your VFS location.
When you click in the address bar, the Pentaho address appears.
You can copy and paste a Pentaho address into file path fields in steps and entries that support VFS connections.
{% hint style="info" %}
Use the Pentaho virtual file system for Amazon S3. Existing transformations and jobs that use Amazon S3 are supported when **Amazon S3** is set as the **Default S3 Connection**.
{% endhint %}
### Create a VFS metastore
A PDI metastore is a location for storing resources shared by multiple transformations. It enables hyperscaler deployments to access the metastore in the cloud. It also lets the PDI client and Pentaho Server reference the same VFS metastore.
The VFS connection information is stored in an XML file. The metastore can be located in one of these places:
* On the machine where you run PDI, in your user directory or in a repository
* On Pentaho Server, as a remote metastore in the server repository
* In a cloud location that is accessible through a VFS connection
Multiple users can access the metastore when it is stored in a remote location. The remote metastore has priority over a local metastore. For example, if you configure a local `metastore-config` file and then connect to a Pentaho Server repository, transformations still use the remote metastore.
#### Enable a VFS metastore
Before you can use a remote metastore, enable a VFS connection in the PDI client. You do this by creating a metastore configuration file, then editing it.
Perform the following steps to enable a VFS metastore:
1. Open the PDI client and create a VFS connection to the storage location you want to use as your metastore. See [Create a VFS connection](#create-a-vfs-connection).
2. Close the PDI client.
3. Go to `Users\\.pentaho\metastore\pentaho\Amazon S3 Connection\` and copy the VFS connection file you created into `Users\\.kettle`.
4. Rename the file to `metastore-config`.
5. Open `metastore-config` in a text editor. Add the `scheme` and `rootPath` elements and their values. See [Metastore configuration](#metastore-configuration).
6. Save the file.
7. Restart the PDI client.
The remote VFS metastore is now enabled. Previous local connections still exist in your local metastore directory. They no longer display in the PDI client. New VFS connections are stored in the location specified in `metastore-config`.
#### Metastore configuration
The elements listed in this section are required for all remote environments. When you create a VFS connection in the PDI client, you do not need to manually edit anything in the `` section.
**Common elements**
These elements are required for all VFS connections:
Element
Value
Description
scheme
<string>
The type of connection. The values are:
s3 - Amazon, MinIO, and HCP
gs - Google cloud storage
abfss - Azure Data Lake Storage Gen2
rootPath
<bucket-name>[/<path>]
The bucket name and optional folder path where you want to create the VFS metastore. The rootPath element must point to the location where you will store the metastore file on the cloud location.
This is analogous to the .pentaho folder in a local metastore.
Examples:
miniobucket/dir1
gcpbucket/dir1
children
A container for type-specific configurations. For example:
**S3 elements**
The elements listed below apply to S3 environments. Some elements are conditional.
| Element | Value | Description |
| --------------------- | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `accessKey` | `` | The S3 user’s access key. |
| `secretKey` | `` | The S3 user’s secret key. |
| `endPoint` | `` |
The URL to access the S3 location. Examples:
http\://\:port
|
| `region` | `` | The user-designated region. For example, `us-east-1`. |
| `connectionType` | 0 or 1 |
The connection type value. The values are:
0 - connect to AWS
1 - connect to MinIO or HCP
|
| `credentialFile` | | An encrypted string that is not user editable |
| `profileName` | `` | The AWS user profile connection when the Type is 0 (AWS) and the `authType` is 1 (credentials file) |
| `defaultS3Config` | true or false | Controls whether the default S3 configuration is used. Set to `true` to use the default S3 configuration |
| `credentialsFilePath` | `` | The path to the AWS credentials file when the connectionType is 0 (AWS) and the authType is 1 (credentials file) |
| `pathStyleAccess` | true or false | Controls which access style is used. Specify `true` for path-style access. Specify `false` for bucket-style access |
| `signatureVersion` | `AWSS3V4SignerType` | The signature version used when communicating with the AWS S3 metastore location. |
| `name` | `vfsMetastore` | The connection name. |
| `description` | `` | A description of the connection. |
| `sessionToken` | `` | Optional. A temporary credential used if the AWS bucket requires a session token for access |
| `authType` | 0 or 1 |
The authentication type when connectionType is 0 (AWS):
0 – Access key/Secret key
1 – Credentials file
|
**GCP elements**
The elements listed below apply to GCP environments:
| Element | Value | Description |
| ------------------- | ---------- | -------------------------------------------------------------------------- |
| `serviceAccountKey` | `` | A key that is generated based on the contents of the service account JSON. |
| `keyPath` | `` | The path to the file containing the GCP service account JSON. |
| `name` | `` | The name of the connection. |
| `description` | `` | A description of the connection. |
**Azure Data Lake Storage Gen2 elements**
The elements listed below apply to Azure Data Lake Storage Gen2 environments. See [Azure Blob Storage](https://learn.microsoft.com/en-us/azure/storage/blobs/) for more information.
| Element | Value | Description |
| --------------------- | -------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
| `sharedKey` | `` | The shared key for accessing the service. |
| `accountName` | `` | The name of the account. |
| `accessTier` | `` | The access tier value. Default: `Hot`. |
| `blockSize` | `` | Default: 50. |
| `maxSingleUploadSize` | `` | Default: 100. |
| `bufferCount` | `` | Default: 5. |
| `name` | `` | The connection name. |
| `authType` | `0`, `1`, or `2` |
The authorization type. Values:
0 - Account Shared Key
1 - Azure Active Directory
2 - Shared Access Signature
|
### Steps and entries supporting VFS connections
You may have a transformation or job containing a step or entry that accesses a file on a Virtual File System.
The following steps and entries support VFS connections:
* [Avro Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-input)
* [Avro Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-output)
* [Bulk load from MySQL into file](http://wiki.pentaho.com/display/EAI/BulkLoad+from+Mysql+to+file)
* [Bulk load into MSSQL](http://wiki.pentaho.com/display/EAI/BulkLoad+into+MSSQL)
* [Bulk load into MySQL](http://wiki.pentaho.com/display/EAI/Bulkload+into+MySQL)
* [Copybook Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/copybook-input-pdi-step)
* [CSV File Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/csv-file-input)
* [De-serialize from file](http://wiki.pentaho.com/display/EAI/De-serialize+from+file)
* [Fixed file input](http://wiki.pentaho.com/display/EAI/Fixed+File+Input)
* [Get data from XML](http://wiki.pentaho.com/display/EAI/Get+Data+From+XML)
* [Get File Names](http://wiki.pentaho.com/display/EAI/Get+File+Names)
* [Get Files Rows Count](http://wiki.pentaho.com/display/EAI/Get+Files+Rows+Count)
* [Get SubFolder names](http://wiki.pentaho.com/display/EAI/Get+SubFolder+names)
* [Google Analytics](http://wiki.pentaho.com/display/EAI/Google+Analytics)
* [GZIP CSV Input](http://wiki.pentaho.com/display/EAI/GZIP+CSV+Input)
* [Job (job entry)](/pdia-data-integration/pdi-job-entries-reference-overview/job-job-entry)
* [JSON Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/json-input)
* [JSON output](http://wiki.pentaho.com/display/EAI/JSON+output)
* [ORC Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-input)
* [ORC Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-output)
* [Parquet Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-input)
* [Parquet Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-output)
* [Query HCP](/pdia-data-integration/pdi-transformation-steps-reference-overview/query-hcp)
* [Read metadata from Copybook](/pdia-data-integration/pdi-transformation-steps-reference-overview/read-metadata-from-copybook)
* [Read metadata from HCP](/pdia-data-integration/pdi-transformation-steps-reference-overview/read-metadata-from-hcp)
* [Text File Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/text-file-output-cp)
* [Transformation (job entry)](/pdia-data-integration/pdi-job-entries-reference-overview/transformation-job-entry-cp)
* [Write metadata to HCP](/pdia-data-integration/pdi-transformation-steps-reference-overview/write-metadata-to-hcp)
### VFS browser
Some transformation steps and job entries use a Virtual File System (VFS) browser instead of VFS connections and the Open dialog box. When you use the VFS browser, you specify a VFS URL instead of a VFS connection. Files are accessed using HTTP. The URLs include schema data that identifies the protocol.
Files can be local or remote. Files can also be compressed formats, such as TAR and ZIP. For more information, see the [Apache Commons VFS documentation](http://commons.apache.org/proper/commons-vfs/).
#### Before you begin
If you need to access Google Drive, see [Access to a Google Drive](#access-to-a-google-drive).
#### Access to a Google Drive
Perform the following setup steps to initially access Google Drive.
1. Follow the “Step 1” procedure in [Build your first Drive app (Java)](https://developers.google.com/drive/api/v3/quickstart/java) in the [Google Drive APIs documentation](https://developers.google.com/drive/).
This procedure turns on the Google Drive API and creates a `credentials.json` file.
2. Rename `credentials.json` to `client_secret.json`. Copy it to `data-integration/plugins/pentaho-googledrive-vfs/credentials`.
3. Restart PDI.
The **Google Drive** option does not appear for the VFS browser until you copy `client_secret.json` into the `credentials` directory and restart PDI.
4. Sign in to your Google account.
5. Enter your Google account credentials.
6. In the permission window, click **Allow**.
After initialization, Pentaho stores a token named **StoredCredential** in `data-integration/plugins/pentaho-googledrive-vfs/credentials`. This token lets you access Google Drive resources without signing in again. If you delete the token, you are prompted to sign in after restarting PDI. If you change account permissions, delete the token and repeat the setup.
{% hint style="info" %}
To access Google Drive from a transformation that runs on Pentaho Server, copy **StoredCredential** and `client_secret.json` into `pentaho-server/pentaho-solutions/system/kettle/plugins/pentaho-googledrive-vfs/credentials` on the server.
{% endhint %}
#### Access files with the VFS browser
Perform the following steps to access files with the VFS browser.
1. Select **File** > **Open** in the PDI client.
The Open dialog box appears.
2. In the left pane, select the file system type. Supported file systems include:
* **Local**: Files on your local machine.
* **Hadoop Cluster**: Files on any Hadoop cluster except S3.
* **HDFS**: Files on Hadoop distributed file systems.
* **Google Drive**: Files on Google Drive. See [Access to a Google Drive](#access-to-a-google-drive).
* **VFS Connections**: Files using a stored VFS connection.
3. Optional: In the **Address** bar, enter a VFS URI.
Examples:
* **Local**: `ftp://userID:password@ftp.myhost.com/path_to/file.txt`
* **HDFS**: `hdfs://myusername:mypassword@mynamenode:port/path`
* **SMB/UNC Provider**: `smb://;:@:/`
For SMB, “domain” is the Windows host name. “Domain” and “server” can be the same when using an IP address.
4. Optional: Use **File type** to filter on file types other than transformations and jobs.
5. Optional: Select a file or folder and click the **X** icon to delete it.
6. Optional: Click the **+** icon to create a new folder.
{% hint style="info" %}
VFS dialog boxes are configured through transformation parameters. See [Configure VFS options](#configure-vfs-options).
{% endhint %}
#### Supported steps and entries
The following steps and entries support the VFS browser:
* [Amazon EMR Job Executor](/pdia-data-integration/pdi-job-entries-reference-overview/amazon-emr-job-executor) (introduced in v9.0)
* [Amazon Hive Job Executor](/pdia-data-integration/pdi-job-entries-reference-overview/amazon-hive-job-executor) (introduced in v9.0)
* [AMQP Consumer](/pdia-data-integration/pdi-transformation-steps-reference-overview/amqp-consumer) (introduced in v9.0)
* [Avro Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-input) (introduced in v8.3)
* [Avro Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/avro-output) (introduced in v8.3)
* [ETL metadata injection](/pdia-data-integration/pdi-transformation-steps-reference-overview/etl-metadata-injection)
* [File Exists (Job Entry)](/pdia-data-integration/pdi-job-entries-reference-overview/file-exists-job-entry)
* [Hadoop Copy Files](/pdia-data-integration/pdi-job-entries-reference-overview/hadoop-copy-files)
* [Hadoop File Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/hadoop-file-input-cp-main-page)
* [Hadoop File Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/hadoop-file-output-cp-main-page)
* [JMS Consumer](/pdia-data-integration/pdi-transformation-steps-reference-overview/jms-consumer) (introduced in v9.0)
* [Job Executor](/pdia-data-integration/pdi-transformation-steps-reference-overview/job-executor) (introduced in v9.0)
* [Kafka consumer](/pdia-data-integration/pdi-transformation-steps-reference-overview/kafka-consumer) (introduced in v9.0)
* [Kinesis consumer](/pdia-data-integration/pdi-transformation-steps-reference-overview/kinesis-consumer) (introduced in v9.0)
* [Mapping](/pdia-data-integration/pdi-transformation-steps-reference-overview/mapping) (sub-transformation)
* [MQTT Consumer](/pdia-data-integration/pdi-transformation-steps-reference-overview/mqtt-consumer) (introduced in v9.0)
* [ORC Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-input) (introduced in v8.3)
* [ORC Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/orc-output) (introduced in v8.3)
* [Parquet Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-input) (introduced in v8.3)
* [Parquet Output](/pdia-data-integration/pdi-transformation-steps-reference-overview/parquet-output) (introduced in v8.3)
* [Oozie Job Executor](http://wiki.pentaho.com/display/EAI/Oozie+Job+Executor) (introduced in v9.0)
* [Simple Mapping](/pdia-data-integration/pdi-transformation-steps-reference-overview/simple-mapping-sub-transformation) (introduced in v9.0)
* [Single Threader](/pdia-data-integration/pdi-transformation-steps-reference-overview/single-threader) (introduced in v9.0)
* [Sqoop Export](http://wiki.pentaho.com/display/EAI/Sqoop+Export) (introduced in v9.0)
* [Sqoop Import](http://wiki.pentaho.com/display/EAI/Sqoop+Import) (introduced in v9.0)
* [Transformation Executor](/pdia-data-integration/pdi-transformation-steps-reference-overview/transformation-executor) (introduced in v9.0)
* [Weka Scoring](https://wiki.pentaho.com/display/EAI/Weka+Scoring) (introduced in v9.0)
{% hint style="info" %}
If you have a Pentaho address for an existing VFS connection, you can paste the `pvfs` location into file or folder fields. You do not need to use **Browse**.
{% endhint %}
For more information on configuring options for SFTP, see [Configure SFTP VFS](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Data%20Integration%20Perspective/Data%20Integration%20perspective%20in%20the%20PDI%20client/Connecting%20to%20Virtual%20File%20Systems%20cp/VFS%20browser%20\(Connecting%20to%20Virtual%20File%20Systems\)/broken-reference/README.md).
#### Configure VFS options
The VFS browser can be configured to set variables as parameters at runtime. The sample transformation `VFS Configuration Sample.ktr` is located in `data-integration/samples/transformations`.
For more information on setting variables, see [VFS properties](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers/parameters/vfs-properties).
For an example of configuring an SFTP VFS connection, see [Configure SFTP VFS](https://github.com/pentaho/documentation/tree/main/PDIA/11.0/PDI/Data%20Integration%20Perspective/Data%20Integration%20perspective%20in%20the%20PDI%20client/Connecting%20to%20Virtual%20File%20Systems%20cp/VFS%20browser%20\(Connecting%20to%20Virtual%20File%20Systems\)/broken-reference/README.md).
# Streaming analytics
With streaming analytics, you can constantly perform statistical analysis while moving within a data stream.
You can use streaming analytics to manage, monitor, and record real-time analytics of live streaming data so you can quickly extract the necessary information from big volumes of data to react to changing conditions in real time. Businesses generate continuous data from the following sources:
* Log files generated by customers using mobile or web applications, e-commerce purchases, and in-game player activity.
* Telemetry, such as data from connected devices, sensors, and instrumentation in data centers.
* Data collected from social networks, financial trading systems, and geospatial services.
Once collected, the streaming data values from these sources will be processed sequentially and incrementally on a record-by-record basis or a time-based sliding window. Ingesting a window of values allows for both processing and analysis of the data, such as through correlating, aggregating, filtering, and sampling. The following figure is an example of a time-based sliding window.
Companies use this information to gain insights into their business and customer activity, such as service usage for billing rates, server activity, website clicks, and geo-locations of devices, people, or physical goods. For example, businesses can track changes in public sentiment on their brands and products by continuously monitoring and analyzing social media streams, and then quickly respond as needed.
The Internet of Things (IoT) also creates large stores of streaming data. Smart objects, such as cars, appliances, and electronic devices, produce data points throughout their operations, activities, and behaviors. Businesses can analyze these points in the data streams to reduce operating costs, improve product reliability, or optimize usage models. For example, you can monitor equipment performance based on its data output. Continuous pattern detection finds anomalies referred to as data gaps. These gaps help to pinpoint when to buy material, plan modifications, and staff personnel.
IoT devices and communication protocols, including text data and transmissions from both legacy and modern equipment sensors, for example, create streaming data of various formats. These multiple formats must be normalized, cleansed, and standardized to process individual events in-memory. Data must be continually corrected and assessed in windows before analysis.
Before you can use streaming analytics, you must ingest the data into PDI as it is received. Within PDI, you can also send event messages to trigger a process of Extract, Transform, and Load (ETL) alerts.
# Get started with streaming analytics in PDI
You can think of Pentaho Data Integration (PDI) as a series of pipes through which water flows and is joined and mixed with other flows. PDI products are designed to work as if data flows like running water. No matter how big the source, the water keeps flowing, such that all the data will be processed if the data keeps flowing. The size of the “pipe” in PDI is directly linked to the number of data records and to the amount of memory needed to hold all those records. The key to successfully transforming your data with high performance is to understand which PDI steps may increase and speed up the flow.
You can develop a PDI transformation that is always waiting for new data. All the steps continue running, awaiting new data. In this transformation, the input steps ingest data records in PDI from the stream. Once ingested, you can process the data to refine it. After processing, you can push it back into the stream or retain it for analysis.
# Data ingestion
Data is ingested into PDI by pulling messages from a stream into a transformation through a specified window. A consumer step in a parent transformation pulls the data into PDI, then runs a child sub-transformation, which executes according to the window parameters. The window creates a continuous stream of records in near real-time.
In the consumer step itself, you can define the number of messages to accept for processing, as well as the specific data formats to stream data. You can set up this step to collect events, monitor alerts, and track user consumption of data streams. Additionally, you can select a step in the child transformation to stream records back to the parent transformation, which passes records downstream to any other steps included within the same parent transformation.
The following consumer steps ingest streaming data into PDI from the specified sources:
* [**AMQP Consumer**](/pdia-data-integration/pdi-transformation-steps-reference-overview/amqp-consumer)
Advanced Message Queuing Protocol brokers
* [**JMS Consumer**](/pdia-data-integration/pdi-transformation-steps-reference-overview/jms-consumer)
Apache ActiveMQ Java Messaging Service server or IBM MQ middleware
* [**Kafka Consumer**](/pdia-data-integration/pdi-transformation-steps-reference-overview/kafka-consumer)
Kafka server
* [**Kinesis Consumer**](/pdia-data-integration/pdi-transformation-steps-reference-overview/kinesis-consumer)
Amazon Kinesis Data Streams service
* [**MQTT Consumer**](/pdia-data-integration/pdi-transformation-steps-reference-overview/mqtt-consumer)
Message Queuing Telemetry Transport broker or clients
In PDI, the data stream window is defined by either duration (in milliseconds) or number of rows. The window of data is created when either the duration or number of rows occur. For example, if the duration is set to `1000` milliseconds and the number of rows is 1000, windows of data are created whenever time intervals of 1000 milliseconds are reached or 1000 rows have been received. If you set either duration or number of rows to `0` (zero), PDI will ignore that parameter. For example, if duration is set to `1000` milliseconds and the number of rows is zero, windows are created only every 1000 milliseconds.
You can specify the maximum number of these batches used to collect records at the same time. However, you should only specify a maximum number of these concurrent batches when your consumer step cannot keep pace with the speed at which the data is streaming. Your computing environment must have adequate CPU and memory for this implementation. An error will occur if your environment cannot handle the maximum number of concurrent batches specified.
Depending on your setup, you can run the transformation within PDI or using Spark within the Adaptive Execution Layer (AEL) as the execution engine, which is set in the [Run Options](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/run-your-transformation) dialog box. The Spark engine executes the child transformation by duration only, and not by the number of rows.
**Note:** If you use the Spark engine on streaming data, your transformation will use the native Spark Streaming. PDI will not report the execution results. This information will appear in Spark on your cluster.
Before using a consumer step with big data, you must set up Pentaho to connect to a cluster. See [Connecting to a Hadoop cluster with the PDI client](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article) for instructions.
# Data processing
Once the data stream is ingested through windowing, you can process these windows in your child transformation. Use the child transformation to adjust the data and handle event alerts as needed. After processing, you can either load the windowed data into various outputs or publish it back into the data message stream. You can publish data back into the message stream by using the following producer steps for your specified target:
* [AMQP Producer](/pdia-data-integration/pdi-transformation-steps-reference-overview/amqp-producer): Advanced Message Queuing Protocol brokers
* [JMS Producer](/pdia-data-integration/pdi-transformation-steps-reference-overview/jms-producer): Apache ActiveMQ Java Messaging Service server or the IBM MQ middleware
* [Kafka Producer](/pdia-data-integration/pdi-transformation-steps-reference-overview/kafka-producer): Kafka server
* [Kinesis Producer](/pdia-data-integration/pdi-transformation-steps-reference-overview/kinesis-producer): Support for pushing data to a specific region and stream located within the Amazon Kinesis Data Streams service
* [MQTT Producer](/pdia-data-integration/pdi-transformation-steps-reference-overview/mqtt-producer): Message Queuing Telemetry Transport broker or clients
You can also use the data streaming window to capture data for analysis. Streaming Pentaho data services can be created from output steps in the child transformation. You can use CTools to create dashboards using these services as data sources. See **Pentaho CTools** for more information.
Once started, streaming data transformations run continuously. You can stop these transformations using the following tools:
* The stop option in the PDI client.
* The [Abort](/pdia-data-integration/pdi-transformation-steps-reference-overview/abort) step in either the parent or child transformation.
* Restarting the Pentaho or Spark execution engine.
**CAUTION:** Stopping or aborting a continuous transformation may cause data loss. Changing the flow of a data stream affects the data that is ingested and processed. Please plan accordingly. If you are working with a Kafka server, you have the option to control when an offset is committed to your window. Use this option to retain the data if the message flow is interrupted.
# Web services steps
[PDI jobs and transformations](/pdia-data-integration/basic-concepts-of-pdi) can interact with a variety of Web services through specialized steps. How you use these steps, and which ones you use, is largely determined by your definition of Web services. The most commonly used Web services steps are:
* [Web Service Lookup](http://wiki.pentaho.com/display/EAI/Web+services+lookup)
* [Modified Java Script Value](/pdia-data-integration/pdi-transformation-steps-reference-overview/modified-java-script-value)
* [RSS Input](http://wiki.pentaho.com/display/EAI/RSS+Input)
* [HTTP Post](http://wiki.pentaho.com/display/EAI/HTTP+Post)
The Web Service Lookup step is useful for selecting and setting input and output parameters via WSDL, but only if you do not need to modify the SOAP request. You can see this step in action in the `Web Services - NOAA Latitude and Longitude.ktr` sample transformation included with PDI in the `/data-integration/samples/transformations/` directory.
There are times when the SOAP message generated by the Web Service Lookup step is insufficient. Many Web services require the security credentials be placed in the SOAP request headers. There may also be a need to parse the response XML to get more information than the response values provide (such as namespaces). In cases like these, you can use the Modified Java Script Value step to create whatever SOAP envelope you need. You would then hop to an HTTP Post step to accept the SOAP request through the input stream and post it to the Web service, then hop to another Modified Java Script Value step to parse the response. The `General - Annotated SOAP Web Service call.ktr` sample in the `/data-integration/samples/transformations/` directory shows this theory in practice.
# Transforming data with PDI
Transform data and manage jobs in Pentaho Data Integration (PDI).
Use this page to build, run, monitor, and optimize transformations and jobs.
### In this article
* [Work with transformations](#work-with-transformations)
* [Work with jobs](#work-with-jobs)
* [PDI run modifiers](#pdi-run-modifiers)
* [Partitioning data](#partitioning-data)
* [Logging and performance monitoring](#logging-and-performance-monitoring)
* [Add notes to transformations and jobs](#add-notes-to-transformations-and-jobs)
* [Manage PDI transformations and job schedules](#manage-pdi-transformations-and-job-schedules)
### Work with transformations
In the [PDI client (Spoon)](/pdia-data-integration/basic-concepts-of-pdi), you can develop transformations.
Transformations are data workflows representing your ETL activities.
Transformation steps define the individual ETL activities.
Transformations are stored in `.ktr` files.
#### Create a transformation
Follow these steps:
1. Do one of the following:
* Select **File** > **New** > **Transformation**.
* Select **New file** on the toolbar, then select **Transformation**.
* Press `Ctrl+N`.
2. Select the **Design** tab.
3. Expand folders or search in **Steps**.
4. Drag a step onto the canvas.
5. Double-click the step to open its properties.
6. Add more steps as needed:
* Drag a step, then press `Shift` and draw a hop to connect steps.
* Double-click a step to add it with a hop from the previous step.
Save the transformation before you run it.
#### Open a transformation
How you open a transformation depends on where it lives.
You can open a local file, a repository object, or a file on a Virtual File System (VFS).
{% hint style="info" %}
If you get a missing plugin error, see [Troubleshooting transformation steps and job entries](/pdia-data-integration/data-integration-issues/troubleshooting-transformation-steps-and-job-entries).
{% endhint %}
**Open a local transformation**
1. In the PDI client, do one of the following:
* Select **File** > **Open**.
* Select **Open file** on the toolbar.
* Select **OPEN Files** on the Welcome screen.
* Press `Ctrl+O`.
2. Select the `.ktr` file, then select **Open**.
**Open a transformation from the Pentaho Repository**
1. Verify you are [connected to a repository](/pdia-data-integration/redirects/use-a-pentaho-repository-in-pdi).
2. Open the repository browser:
* Select **File** > **Open**.
* Select **Open file** on the toolbar.
* Select **OPEN Files** on the Welcome screen.
* Press `Ctrl+O`.
3. Use **Recents**, search, or browse folders to find your transformation.
4. Select the transformation, then select **Open**.
**Open a transformation on a Virtual File System**
Select **File** > **Open**.
For details, see [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser).
#### Rename a folder or file (local only)
You can rename folders and files from the **Open** window.
You can rename only when you are not connected to the Pentaho Repository.
1. In the **Open** window, select a folder or file.
2. Right-click the folder or file.
3. Select **Rename**.
#### Save a transformation
How you save a transformation depends on where it lives.
**Save a local transformation**
1. In the PDI client, do one of the following:
* Select **File** > **Save** or **File** > **Save as**.
* Select **Save current file** on the toolbar.
* Press `Ctrl+S`.
2. Enter a name and choose a location.
3. Select **Save**.
**Save a transformation to the Pentaho Repository**
1. Verify you are [connected to a repository](/pdia-data-integration/redirects/use-a-pentaho-repository-in-pdi).
2. In the PDI client, do one of the following:
* Select **File** > **Save** or **File** > **Save as**.
* Select **Save current file** on the toolbar.
* Press `Ctrl+S`.
3. Browse to the repository folder.
4. Enter a name, then select **Save**.
**Save a transformation on a Virtual File System**
Select **File** > **Open** to save a transformation on a Virtual File System (VFS).
For details, see [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser).
#### Run a transformation
Run a transformation to test how it performs.
The **Run Options** window lets you set run configurations, logging, options, and temporary parameter values.
Open **Run Options** in one of these ways:
* Select **Run** on the toolbar.
* Select **Action** > **Run**.
* Press `F9`.
The **Run Options** window appears.
In **Run Options**, you choose a run configuration.
To set up run configurations, see [Run configurations (transformations)](#run-configurations-transformations).
After you run a transformation, use the [Execution Results](#analyze-transformation-results) section to review output.
**Run configurations (transformations)**
Some ETL activities are lightweight.
Others need dedicated servers or cluster execution.
You can create or edit run configurations in **View** > **Run configurations**.
**Pentaho local** is the default run configuration.
You cannot edit it.
**Create or edit a run configuration**
Right-click **Run configurations**, then select **New**.
Or, right-click a configuration and select **Edit**.
The dialog contains:
| Field | Description |
| --------------- | ----------------------------------------- |
| **Name** | Name of the run configuration. |
| **Description** | Optional details about the configuration. |
**Select an engine**
You can select the **Pentaho engine** to run transformations in the default environment.
You can also use **Spark Submit** to run big data transformations on a Hadoop cluster.
See [Spark Submit](/pdia-data-integration/pdi-job-entries-reference-overview/spark-submit).
The Pentaho engine does not execute sub-transformations or sub-jobs when you select **Pentaho server** or **Slave server**.
If you need sub-transformations to run on the same host as the parent job, use **Local**.
**Run options**
Errors, warnings, and other information are stored in logs.
You set log verbosity and other behavior in the **Options** section.
For background, see [Logging and performance monitoring](#logging-and-performance-monitoring).
| Option | Description |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **Clear log before running** | Clears logs before each run. |
| **Log level** | Controls how much information is logged. |
| **Enable safe mode** | Checks every row to ensure layout consistency. |
| **Gather performance metrics** | Captures performance metrics during the run. See [Use performance graphs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring/monitor-performance/use-performance-graphs). |
**Parameters and variables (run-time overrides)**
You can temporarily override parameters and variables for each run.
These overrides apply only to the current run.
* Use [parameters](#parameters) to set run-time parameter values.
* Use [variables](#variables) to set run-time variable values.
* Use [arguments](#arguments) for command-line style arguments.
#### Analyze transformation results
After you run a transformation, the **Execution Results** panel appears.
It helps you inspect step behavior, errors, and performance.
**Step metrics**
The **Step Metrics** tab shows per-step statistics.
It includes records read, written, errors, and row throughput.
Steps that failed are highlighted in red.
**Logging**
The **Logging** tab shows log details for the most recent run.
Error lines are highlighted in red.
**Execution history**
The **Execution History** tab shows metrics and logs from previous runs.
It requires database logging configured in transformation properties.
See [Set up transformation logging](#set-up-transformation-logging).
**Performance graph**
The **Performance Graph** tab shows step performance over time.
It requires database logging enabled.
**Metrics**
The **Metrics** tab shows a Gantt chart for run timings.
**Preview data**
Use **Preview Data** to inspect step output rows.
Select a step to view its data.
#### Stop your transformation
You can stop transformations in two ways.
* Use **Stop** to stop immediately.
* Use **Stop input processing** to stop input safely after in-flight records complete.
Transformation properties reference
Transformation properties describe the transformation and configure its behavior.
To open properties, press `Ctrl+T` or right-click the canvas and select **Properties**.
The settings are grouped into these tabs:
* Transformation
* Parameters
* Logging
* Dates
* Dependencies
* Miscellaneous
* Monitoring
After you adjust settings, select **SQL** to generate SQL for logging tables.
For SQL execution details, see [Use the SQL Editor](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring/use-the-sql-editor).
#### Transformation tab
Use the **Transformation** tab to specify general properties.
| Property | Description |
| ----------------------- | ----------------------------------------------- |
| Transformation name | Required to save settings to a repository. |
| Transformation filename | `.ktr` file name. |
| Description | Short description shown in Repository Explorer. |
| Extended description | Long description. |
| Status | Draft or production. |
| Version | Version description. |
| Directory | Repository directory. |
| Created by | Original creator. |
| Created at | Create time. |
| Last modified by | Last editor. |
| Last modified at | Last update time. |
#### Parameters tab
Use the **Parameters** tab to add parameters.
| Property | Description |
| ------------- | --------------------------------------------------------- |
| Parameter | Local variable shared across steps in the transformation. |
| Default Value | Used if a parameter value is not set elsewhere. |
| Description | Description of the parameter. |
#### Logging tab
Use the **Logging** tab to configure logging.
For a guided setup, see [Set up transformation logging](#set-up-transformation-logging).
#### Dates tab
Use the **Dates** tab to configure date ranges and limits.
#### Dependencies tab
Use the **Dependencies** tab to list transformation dependencies.
#### Miscellaneous tab
Use the **Miscellaneous** tab to configure buffer sizes and admin settings.
This tab includes the **Make the transformation database transactional** option.
For rollback patterns, see [Transactional databases and job rollback](#transactional-databases-and-job-rollback).
#### Monitoring tab
Use the **Monitoring** tab to enable step performance monitoring.
Transformation canvas context menu (Transformation menu)
Use the **Transformation** menu to access settings, options, and properties.
Right-click any step in the canvas.
Each item is described in this table.
| Menu Item | Description |
| ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **New Hop** | Creates a new hop. |
| **Edit** | Shows the step configuration window. |
| **Description** | Adds a description to the step. |
| **Open Referenced Object** | Maps a sub-transformation. See [Mapping](/pdia-data-integration/pdi-transformation-steps-reference-overview/mapping). |
| **Data Movement** | Round robin, load balance, or copy rows across hops. |
| **Change Number of Copies to Start** | Starts step copies in parallel. |
| **Copy** | Copies selected items to the clipboard. |
| **Duplicate** | Duplicates the selection on the canvas. |
| **Delete** | Deletes selected items. |
| **Hide** | Hides the step. You must edit the XML to show it again. |
| **Detach** | Detaches the step or entry. |
| **Input Fields** | Shows incoming field metadata. |
| **Output Fields** | Shows outgoing field metadata. |
| **Sniff Test During Execution** | Shows row data while executing. See [Sniff Test tool](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring/monitor-performance/sniff-test-tool). |
| **Check Selected Step(s)** | Checks steps for configuration problems. |
| **Error Handling** | Configures step error handling. |
| **Preview** | Launches the debug dialog. |
| **Align/Distribute** | Aligns or distributes steps on the canvas. |
| **Data Services** | Creates or manages data services. See [Pentaho Data Services](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/pentaho-data-services). |
| **Mapping** | Creates a Select/Rename Values step for field mappings. |
| **Partitions** | Configures partitioning. See [Partitioning data](#partitioning-data). |
| **Clusters** | Configures Carte clusters. See [Use Carte Clusters](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/use-carte-clusters). |
### Work with jobs
In the [PDI client (Spoon)](/pdia-data-integration/basic-concepts-of-pdi), you can develop jobs to orchestrate ETL activities.
Job entries define the work a job performs.
Jobs are stored in `.kjb` files.
#### Create a job
Follow these steps:
1. Do one of the following:
* Select **File** > **New** > **Job**.
* Select **New file** on the toolbar, then select **Job**.
* Press `Ctrl+Alt+N`.
2. Select the **Design** tab.
3. Expand folders or search in **Entries**.
4. Drag an entry onto the canvas.
5. Double-click the entry to open its properties.
6. Add more entries as needed:
* Drag an entry, then press `Shift` and draw a hop.
* Double-click an entry to add it with a hop from the previous entry.
Save the job when you are done.
#### Open a job
How you open a job depends on where it lives.
You can open a local file, a repository object, or a file on a Virtual File System (VFS).
{% hint style="info" %}
If you get a missing plugin error, see [Troubleshooting transformation steps and job entries](/pdia-data-integration/data-integration-issues/troubleshooting-transformation-steps-and-job-entries).
{% endhint %}
**Open a local job**
1. In the PDI client, do one of the following:
* Select **File** > **Open**.
* Select **Open file** on the toolbar.
* Select **OPEN Files** on the Welcome screen.
* Press `Ctrl+O`.
2. Select the `.kjb` file, then select **Open**.
**Open a job from the Pentaho Repository**
1. Verify you are [connected to a repository](/pdia-data-integration/redirects/use-a-pentaho-repository-in-pdi).
2. Open the repository browser:
* Select **File** > **Open**.
* Select **Open file** on the toolbar.
* Press `Ctrl+O`.
3. Use **Recents**, search, or browse folders to find your job.
4. Select the job, then select **Open**.
**Open a job on a Virtual File System**
Select **File** > **Open**.
For details, see [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser).
#### Rename a folder or file (local only)
Use [Rename a folder or file (local only)](#rename-a-folder-or-file-local-only).
#### Save a job
How you save a job depends on where it lives.
**Save a local job**
1. In the PDI client, do one of the following:
* Select **File** > **Save** or **File** > **Save as**.
* Select **Save current file** on the toolbar.
* Press `Ctrl+S`.
2. Enter a name and choose a location.
3. Select **Save**.
**Save a job to the Pentaho Repository**
1. Verify you are [connected to a repository](/pdia-data-integration/redirects/use-a-pentaho-repository-in-pdi).
2. In the PDI client, do one of the following:
* Select **File** > **Save** or **File** > **Save as**.
* Select **Save current file** on the toolbar.
* Press `Ctrl+S`.
3. Browse to the repository folder.
4. Enter a name, then select **Save**.
**Save a job on a Virtual File System**
Select **File** > **Open** to save a job on a Virtual File System (VFS).
For details, see [Connecting to Virtual File Systems](/pdia-data-integration/extracting-data-into-pdi/virtual-file-system-browser).
#### Run a job
Run a job to test how it performs.
The **Run Options** window lets you set run configurations, logging, options, and temporary values.
Open **Run Options** in one of these ways:
* Select **Run** on the toolbar.
* Select **Action** > **Run**.
* Press `F9`.
In **Run Options**, you choose a run configuration.
To set up run configurations, see [Run configurations (jobs)](#run-configurations-jobs).
**Run configurations (jobs)**
You can create or edit configurations in **View** > **Run configurations**.
**Pentaho local** is the default run configuration.
You cannot edit it.
**Pentaho engine**
The Pentaho engine does not execute sub-transformations or sub-jobs when you select **Pentaho server** or **Slave server**.
If you need sub-transformations to run on the same host as the parent job, use **Local**.
**Run options**
| Option | Description |
| ------------------------------ | ---------------------------------------------- |
| **Clear log before running** | Clears logs before each run. |
| **Log level** | Controls how much information is logged. |
| **Enable safe mode** | Checks every row to ensure layout consistency. |
| **Start job at** | Starts the run at an alternative entry. |
| **Gather performance metrics** | Captures performance metrics during the run. |
**Parameters and variables (run-time overrides)**
You can temporarily override parameters and variables for each run.
These overrides apply only to the current run.
* Use [parameters](#parameters) to set run-time parameter values.
* Use [variables](#variables) to set run-time variable values.
* Use [arguments](#arguments) for command-line style arguments.
#### Stop your job
You can stop jobs in two ways.
* Use **Stop** to stop immediately.
* Use **Stop input processing** to stop input safely after in-flight records complete.
Job properties reference
Job properties control job behavior and logging.
To open properties, press `Ctrl+T` or right-click the canvas and select **Properties**.
The properties are grouped into these tabs:
* Job
* Parameters
* Settings
* Log
* Transactions
Job canvas context menu (Job menu)
Right-click any entry in the job canvas to view the **Job** menu.
| Menu Item | Description |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **New Hop** | Creates a new hop. |
| **Edit** | Shows the entry configuration window. |
| **Description** | Adds a description to the entry. |
| **Open Referenced Object** | Opens referenced transformations. |
| **Copy** | Copies selected items to the clipboard. |
| **Duplicate** | Duplicates the selection on the canvas. |
| **Delete** | Deletes selected items. |
| **Hide** | Hides the entry. You must edit the XML to show it again. |
| **Detach** | Detaches the entry from the job. |
| **Align/Distribute** | Aligns or distributes entries on the canvas. |
| **Restartable Checkpoint** | Adds a checkpoint to restart failed jobs. See [Use checkpoints to restart jobs](#use-checkpoints-to-restart-jobs). |
| **Run Next Entries in Parallel** | Runs next entries in parallel. |
### PDI run modifiers
This section describes the types of run modifiers, their uses, and configuration.
You can use arguments, parameters, or variables to modify how you run transformations and jobs.
#### Arguments
A PDI argument is a named, user-supplied, single-value input.
Arguments are passed as command-line values after the Pan or Kitchen options.
Each transformation or job supports up to 10 arguments.
Example:
```sh
sh pan.sh -file:/example_transformations/example.ktr argOne argTwo argThree
```
In Spoon, you can test arguments in the **Run Options** window.
Use the **Arguments** button to enter values.
#### Parameters
Parameters are local variables that apply only to the transformation where you define them.
You can assign a default value.
If a parameter name collides with a variable, the parameter takes precedence.
Define parameters in transformation settings:
* Right-click the transformation canvas and select **Transformation settings**.
* Or press `Ctrl+T`.
* Select the **Parameters** tab.
**VFS properties**
You can specify VFS properties as parameters.
**Specifying VFS properties as parameters**
VFS properties can be specified as parameters.
The format of the reference to a VFS property is **vfs.scheme.property.host**.
The following list describes the subparts of the format:
* The **vfs** subpart is required to identify this as a virtual file system configuration property.
* The **scheme** subpart represents the VFS driver's scheme (or VFS type), such as HTTP, SFTP, or ZIP.
* The **property** subpart is the name of a VFS driver's ConfigBuilder's setter (the specific VFS element that you want to set).
* The **host** optionally defines a specific IP address or hostname that this setting applies to.
You must consult each scheme's API reference to determine which properties you can create variables for.
Apache provides VFS scheme documentation at .
The **org.apache.commons.vfs.provider** package lists each of the configurable VFS providers (FTP, HTTP, SFTP, and others).
Each provider has a **FileSystemConfigBuilder** class that in turn has **set\*(FileSystemOptions, Object)** methods.
If a method's second parameter is a **String** or a number (Integer, Long, and others), then you can create a PDI variable to set the value for VFS dialog boxes.
The table below explains VFS properties for the SFTP scheme.
Each property must be declared as a PDI variable and preceded by the `vfs.sftp` prefix as defined above.
| SFTP VFS Property | Purpose |
| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| **compression** | Specifies whether ZLIB compression is used for the destination files. Possible values are `zlib` and `none`. |
| **identity** | The private key file (fully qualified local or remote path and filename) to use for host authentication. |
| **authkeypassphrase** | The passphrase for the private key specified by the **identity** property. |
| **StrictHostKeyChecking** | If this is set to `no`, the certificate of any remote host will be accepted. If set to `yes`, the remote host must exist in the known hosts file (`~/.ssh/known_hosts`). |
**Note:** All of these properties are optional.
The following examples show how to specify parameters as VFS properties:
**Configure SFTP VFS**
To configure the connection settings for SFTP dialog boxes in PDI, you must create either variables or parameters for each relevant value.
Possible values are determined by the VFS driver you are using.
You can use parameters to substitute VFS connection details, then use them in the VFS dialog box.
For example, assuming the parameters have been set:
`sftp://${username}@${host}/${path}`
This technique enables you to hide sensitive connection details, such as usernames and passwords.
You can see examples of these techniques in the VFS Configuration Sample transformation in the `/data-integration/samples/transformations/` directory.
#### Variables
A variable is user-supplied information used dynamically in different scopes.
Variables can be local to a step or available to the whole JVM.
You can define variables in these ways:
* Set Variable step or Set Session Variables step.
* `kettle.properties`.
* **Edit** > **Set Environment Variables**.
You can reference variables in fields using:
* `${VARIABLE}`
* `%%VARIABLE%%`
If a name collides with a parameter or argument, variables defer.
**Environment variables**
Environment variables are traditional variables in PDI.
You define them in **Edit** > **Set Environment Variables**.
Or you pass JVM system properties using the `-D` flag.
Environment variables are not safe for dynamic concurrent runs.
They are visible to all software running in the same JVM.
**Kettle variables**
Kettle variables are scoped to Kettle and can be limited to a job or transformation.
You can set Kettle variables in these ways:
* [Set Kettle variables in the PDI client](#set-kettle-variables-in-the-pdi-client)
* [Set Kettle variables manually](#set-kettle-variables-manually)
* [Set Kettle or Java environment variables in the Pentaho MapReduce job entry](#set-kettle-or-java-environment-variables-in-the-pentaho-mapreduce-job-entry)
* [Set the LAZY\_REPOSITORY variable in the PDI client](#set-the-lazy_repository-variable-in-the-pdi-client)
**Set Kettle variables in the PDI client**
1. Select **Edit** > **Edit the kettle.properties file**.
2. Update existing values.
3. To add a variable:
1. Right-click a line number and select **Insert before this row** or **Insert after this row**.
2. Enter the variable name and value.
3. To reorder, right-click the line and select **Move Up** or **Move Down**.
4. Select **OK**.
**Set Kettle variables manually**
1. Open `kettle.properties` in a text editor.
2. Edit values.
3. Save the file.
**Set Kettle or Java environment variables in the Pentaho MapReduce job entry**
MapReduce jobs run distributed across nodes.
Set node-specific variables in the MapReduce job entry.
1. Double-click the **Pentaho MapReduce** job entry, then select **User Defined**.
2. In **Name**, set a variable name:
* Kettle variable: `KETTLE_SAMPLE_VAR`
* Java system property: prefix with `java.system.` (example `java.system.SAMPLE_PATH_VAR`)
3. In **Value**, enter the variable value.
4. Select **OK**.
**Set the LAZY\_REPOSITORY variable in the PDI client**
This variable restores repository directory-loading behavior from before Pentaho 6.1.
1. Select **Edit** > **Edit the kettle.properties file**.
2. Set `KETTLE_LAZY_REPOSITORY=true`.
3. Select **OK**, then restart the PDI client.
#### Internal variables
These variables are always defined:
| Variable Name | Sample Value |
| --------------------------------- | --------------------- |
| **Internal.Kettle.Build.Date** | `2010/05/22 18:01:39` |
| **Internal.Kettle.Build.Version** | `2045` |
| **Internal.Kettle.Version** | `4.3` |
These variables are defined in a transformation:
| Variable Name | Sample Value |
| ------------------------------------------------ | --------------------------------------------------- |
| **Internal.Transformation.Filename.Directory** | `D:\\Kettle\\samples` |
| **Internal.Transformation.Filename.Name** | `Denormaliser - 2 series of key-value pairs.ktr` |
| **Internal.Transformation.Name** | `Denormaliser - 2 series of key-value pairs sample` |
| **Internal.Transformation.Repository.Directory** | `/` |
These variables are defined in a job:
| Variable Name | Sample Value |
| ------------------------------------- | ------------------------ |
| **Internal.Job.Filename.Directory** | `file:///home/matt/jobs` |
| **Internal.Job.Filename.Name** | `Nested jobs.kjb` |
| **Internal.Job.Name** | `Nested job test case` |
| **Internal.Job.Repository.Directory** | `/` |
These variables are defined in transformations and jobs within a project:
| Variable Name | Sample Value |
| ---------------------------------------- | -------------------------------------------------- |
| **Internal.Project.Data.Directory** | `pvfs://Repository/home/admin/projects/myproject/` |
| **Internal.Project.Execution.Directory** | `/home/admin/projects/myproject` |
| **Internal.Project.Name** | `My project` |
| **Internal.Project.Description** | `Description of my project` |
{% hint style="info" %}
Project directory variables are used in different situations.
* **Internal.Project.Data.Directory** is used by steps that are not repository-aware, like [Text File Input](/pdia-data-integration/pdi-transformation-steps-reference-overview/text-file-input-cp).
* **Internal.Project.Execution.Directory** is used by repository-aware steps, like [Transformation Executor](/pdia-data-integration/pdi-transformation-steps-reference-overview/transformation-executor).
{% endhint %}
These variables are defined in a clustered transformation on a slave server:
| Variable Name | Sample Value |
| ---------------------------------------- | --------------------- |
| **Internal.Slave.Transformation.Number** | `0..` |
| **Internal.Cluster.Size** | `` |
### Partitioning data
Partitioning distributes rows into subsets according to a rule.
Use partitioning to scale up (more CPU cores) and scale out (multiple servers).
#### Get started
By default, each step in a transformation runs in parallel in a separate thread.
#### Partitioning during data processing
You can scale up using **Change Number of Copies to Start**.
This creates multiple copies of a step at runtime.
Without partition rules, parallel aggregation can produce incorrect results.
#### Understand repartitioning logic
When a step needs to repartition data, it creates buffers from each source copy to each target copy.
Partitioning applies a rule-based distribution so like rows go to the same copy.
#### Partitioning data over tables
The Table Output step supports partitioning rows to different tables.
It can accept the table name from a **Partitioning field**.
You can also partition per month or per day.
#### Use partitioning
Partitioning methods can be based on any criteria.
You can also use a partitioning plugin.
1. Set up a partition schema.
2. Apply the schema to a step.
3. Select a partitioning method for row distribution.
#### Use data swimlanes
When a partitioned step passes data to another partitioned step with the same schema, data stays in swimlanes.
No repartitioning is needed.
#### Rules for partitioning
These rules drive distribution and buffer allocation:
* A partitioned step runs one copy per partition.
* Repartitioning creates buffers from each source copy to each target copy.
* Non-partitioned to partitioned causes repartitioning.
* Same schema between partitioned steps avoids repartitioning.
* Different schemas between partitioned steps causes repartitioning.
#### Partitioning clustered transformations
Partitioning can scale out on a cluster of slave servers.
Keep repartitioning to a minimum to avoid network overhead.
Try to keep data in swimlanes for as long as possible.
#### Learn more
* [Set up a Carte cluster](/pdia-data-integration/archived-merged-pages/carte-clusters-archive/set-up-a-carte-cluster)
### Logging and performance monitoring
You can use logging and performance monitoring to troubleshoot, tune, and plan capacity.
You can also run an impact analysis from **Action** > **Impact**.
#### Set up transformation logging
Follow these steps to create a log table for a transformation:
1. Ask your system administrator to create a database or table space called `PdiLog`.
2. Open transformation properties (`Ctrl+T`).
3. Select the **Logging** tab.
4. Configure connection, schema, and table name.
5. Select fields to log.
6. Select **SQL**, then execute the generated SQL.
{% hint style="info" %}
For effective deletion of expired logs, keep **LOGDATE** and **TRANSNAME** enabled.
{% endhint %}
{% hint style="warning" %}
Monitoring **LOG\_FIELD** can negatively affect Pentaho Server performance.
{% endhint %}
#### Set up job logging
Follow these steps to create a log table for a job:
1. Ask your system administrator to create a database or table space called `PdiLog`.
2. Open job properties (`Ctrl+T`).
3. Select the **Log** tab.
4. Configure connection, schema, and table name.
5. Select fields to log.
6. Select **SQL**, then execute the generated SQL.
{% hint style="info" %}
For effective deletion of expired logs, keep **LOGDATE** and **JOBNAME** enabled.
{% endhint %}
#### Logging levels
| Log Level | Description |
| ------------- | ----------------------------------------------- |
| **Nothing** | No logging. |
| **Error** | Only errors. |
| **Minimal** | Minimal logging. |
| **Basic** | Default. |
| **Detailed** | Detailed logging output. |
| **Debug** | Very detailed output for debugging. |
| **Row Level** | Row-level logging. Generates a lot of log data. |
#### Monitor performance
Use these tools:
* [Sniff Test tool](#sniff-test-tool)
* [Monitoring tab](#monitoring-tab)
* [Use performance graphs](#use-performance-graphs)
**Sniff Test tool**
The Sniff Test displays row data as it travels from one step to another.
It is designed as a supplement to logs.
{% hint style="warning" %}
Sniff Test slows transformation run speed.
{% endhint %}
1. Right-click a step while the transformation runs.
2. Select **Sniff Test During Execution**.
3. Select an option:
* **Sniff test input rows**
* **Sniff test output rows**
* **Sniff test error handling**
**Monitoring tab**
Enable step performance monitoring in transformation properties:
1. Open transformation properties (`Ctrl+T`).
2. Select **Enable step performance monitoring?**.
Step performance monitoring can increase memory consumption in long-running transformations.
**Use performance graphs**
If you configured performance monitoring with database logging, you can view performance graphs.
PDI performance tuning tips
The following tips can help diagnose performance issues.
| Step | Tip | Description |
| ------------- | ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| JS | Turn off compatibility mode |
Rewriting JavaScript to use a format that is not compatible with previous versions is, in most instances, easy to do and makes scripts easier to work with and to read. By default, old JavaScript programs run in compatibility mode. That means that the step will process like it did in a previous version. You may see a small performance drop because of the overload associated with forcing compatibility. If you want to make use of the new architecture, disable compatibility mode and change the code as shown below:
intField.getInteger() > intField
numberField.getNumber() > numberField
dateField.getDate() > dateField
bigNumberField.getBigNumber() > bigNumberField
and so on...
Instead of Java methods, use the built-in library. Notice that the resulting program code is more intuitive. For example:
checking for null is now: field.isNull() > field==null
Converting string to date: field.Clone().str2dat() > str2date(field)
and so on...
If you convert your code as shown above, you may get significant performance benefits.
Note: It is no longer possible to modify data in-place using the value methods. This was a design decision to ensure that no data with the wrong type would end up in the output rows of the step. Instead of modifying fields in-place, create new fields using the table at the bottom of the Modified JavaScript transformation.
|
| JS | Combine steps | One large JavaScript step runs faster than three consecutive smaller steps. Combining processes in one larger step helps to reduce overhead. |
| JS | Avoid the JavaScript step or write a custom plug in | Remember that while JavaScript is the fastest scripting language for Java, it is still a scripting language. If you do the same amount of work in a native step or plugin, you avoid the overhead of the JS scripting engine. This has been known to result in significant performance gains. It is also the primary reason why the Calculator step was created — to avoid the use of JavaScript for simple calculations. |
| JS | Create a copy of a field | No JavaScript is required for this, a Select Values step does the trick. You can specify the same field twice. Once without a rename, once (or more) with a rename. Another trick is to use B=NVL(A,A) in a Calculator step where B is forced to be a copy of A. |
| JS | Data conversion | Consider performing conversions between data types (dates, numeric data, and so on) in a Select Values step. You can do this in the **Metadata** tab of the step. |
| JS | Variable creation | If you have variables that can be declared once at the beginning of the transformation, make sure you put them in a separate script and mark that script as a startup script (right click on the script name in the tab). JavaScript object creation is time consuming so if you can avoid creating a new object for every row you are transforming, this will translate to a performance boost for the step. |
| N/A | Launch several copies of a step |
There are two important reasons why launching multiple copies of a step may result in better performance:1. The step uses a lot of CPU resources and you have multiple processor cores in your computer. Example: a JavaScript step. 2. Network latencies and launching multiple copies of a step can reduce average latency. If you have a low network latency of say 5ms and you need to do a round trip to the database, the maximum performance you get is 200 (x5) rows per second, even if the database is running smoothly. You can try to reduce the round trips with caching, but if not, you can try to run multiple copies. Example: a database lookup or table output.
|
| N/A | Manage thread priorities | This feature that is found in the Transformation Settings dialog box under the (**Misc** tab) improves performance by reducing the locking overhead in certain situations. This feature is enabled by default for new transformations that are created in recent versions, but for older transformations this can be different. |
| Select Values | If possible, don't remove fields in Select Values | Don't remove fields in Select Value unless you must. It's a CPU-intensive task as the engine needs to reconstruct the complete row. It is almost always faster to add fields to a row rather than delete fields from a row. |
| Get Variables | Watch your use of Get Variables | May cause bottlenecks if you use it in a high-volume stream (accepting input). To solve the problem, take the Get Variables step out of the transformation (right click, detach) then insert it in with a Join Rows step. Make sure to specify the main step from which to read in the Join Rows step. Set it to the step that originally provided the Get Variables step with data. |
| N/A | Use new text file input | The CSV File Input or Fixed File Input steps provide optimal performance. If you have a fixed width (field/row) input file, you can even read data in parallel. (multiple copies) These new steps have been rewritten using Non-blocking I/O (NIO) features. Typically, the larger the NIO buffer you specify in the step, the better your read performance will be. |
| N/A | When appropriate, use lazy conversion | In instances in which you are reading data from a text file and you write the data back to a text file, use Lazy conversion to speed up the process. The principle behind lazy conversion that it delays data conversion in hopes that it isn't necessary (reading from a file and writing it back comes to mind). Beyond helping with data conversion, lazy conversion also helps to keep the data in "binary" storage form. This, in turn, helps the internal Kettle engine to perform faster data serialization (sort, clustering, and so on). The **Lazy Conversion** option is available in the CSV File Input and Fixed File Input text file reading steps. |
| Join Rows | Use Join Rows | You need to specify the main step from which to read. This prevents the step from performing any unnecessary spooling to disk. If you are joining with a set of data that can fit into memory, make sure that the cache size (in rows of data) is large enough. This prevents (slow) spooling to disk. |
| N/A | Review the big picture: database, commit size, row set size and other factors | Consider how the whole environment influences performance. There can be limiting factors in the transformation itself and limiting factors that result from other applications and PDI. Performance depends on your database, your tables, indexes, the JDBC driver, your hardware, speed of the LAN connection to the database, the row size of data and your transformation itself. Test performance using different commit sizes and changing the number of rows in row sets in your transformation settings. Change buffer sizes in your JDBC drivers or database. |
| N/A | Step Performance Monitoring | Step Performance Monitoring is an important tool that allows you identify the slowest step in your transformation. |
#### Logging best practices
You can improve logging with rotation and other best practices.
For more information, see the **Administer Pentaho Data Integration and Analytics** document.
#### Use checkpoints to restart jobs
Checkpoints let you restart jobs that fail without rerunning from the beginning.
**Add a checkpoint**
1. Open a job.
2. Right-click an entry, then select **Restartable Checkpoint**.
**Delete a checkpoint**
1. Open a job.
2. Right-click an entry, then select **Clear Checkpoint Marker**.
**Set up a checkpoint log**
1. Open a job.
2. Right-click the job canvas, then select **Properties**.
3. Select the **Log** tab, then select **Checkpoints log table**.
4. Configure the log connection and table name.
5. Select **OK**.
#### Use the SQL Editor
Use the SQL Editor to preview and execute DDL generated by the PDI client.
Keep these points in mind:
* Separate statements with semicolons.
* Spoon removes line breaks and semicolons before execution.
* PDI clears database cache for the connection you execute on.
#### Use the Database Explorer
Use Database Explorer to explore configured database connections.
You open it from the Database Connections dialog box.
#### Transactional databases and job rollback
By default, changes are committed as a job or transformation executes.
If you need rollback behavior, make the database transactional.
**Make a transformation database transactional**
1. Open a transformation.
2. Open transformation properties.
3. Select the **Miscellaneous** tab.
4. Select **Make the transformation database transactional**.
5. Select **OK**.
**Make a job database transactional**
1. Open a job.
2. Open job properties.
3. Select the **Transactions** tab.
4. Select **Make the job database transactional**.
5. Select **OK**.
### Add notes to transformations and jobs
Notes help document structure, design decisions, business rules, and dependencies.
#### Create a note
1. Right-click the canvas and select **New Note**.
2. Select **Font Style** to change font and color.
3. Select **Note**, then type your note.
4. Select **OK**.
#### Edit a note
1. Double-click the note.
2. Select **Font Style** to change font and color.
3. Select **Note**, then edit your note.
4. Select **OK**.
#### Reposition a note
1. Drag the note to a new position.
2. Optional: right-click the note:
* Select **Raise Note** to move it above other items.
* Select **Lower Note** to move it below other items.
#### Delete a note
1. Right-click the note.
2. Select **Delete Note**.
### Manage PDI transformations and job schedules
Schedule transformations and jobs to run at specific times or intervals.
#### Schedule a transformation or job
1. Connect to the Pentaho Repository.
2. Open a job or transformation, then select **Action** > **Schedule**.
3. Set start time:
* **Now**, or
* **Date** with date, time, and time zone.
4. Set repeat schedule.
5. Set end time or select **No end**.
6. Optional: enable **Safe mode**.
7. Select **Log Level**.
8. Update argument, parameter, or variable values if needed.
9. Select **OK**.
#### Edit a scheduled run
1. Select the **Scheduler** perspective.
2. Select the schedule, then select **Edit Scheduled Task**.
#### Stop a schedule
1. Select the **Scheduler** perspective.
2. Select the schedule, then select **Stop Scheduled Task**.
#### Enable or disable a schedule
1. Select the **Scheduler** perspective.
2. Select the schedule.
3. Select **Start Scheduler** to enable.
4. Select **Stop Scheduler** to disable.
#### Delete a scheduled run
1. Select the **Scheduler** perspective.
2. Select the schedule, then select **Remove**.
#### Refresh the schedule list
1. Select the **Scheduler** perspective.
2. Select **Refresh**.
### Archived source pages
These pages were merged into this single topic page and moved under [Transforming data with PDI (archive)](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive):
* [Work with transformations](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp)
* [Work with jobs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-jobs)
* [PDI run modifiers](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/pdi-run-modifiers)
* [Partitioning data](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/partitioning-data)
* [Logging and performance monitoring](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/logging-and-performance-monitoring)
* [Add notes to transformations and jobs](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/add-notes-to-transformations-and-jobs)
* [Manage PDI transformations and job schedules](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/manage-pdi-transformations-and-job-schedules)
# Loading data from PDI
Load data from Pentaho Data Integration (PDI) to external tools.
Use these features to run outside the PDI client, run remotely, expose virtual tables, and capture lineage.
### In this article
* [Run transformations and jobs from the command line](#run-transformations-and-jobs-from-the-command-line)
* [Run and monitor remotely with Carte](#run-and-monitor-remotely-with-carte)
* [Expose transformation output with Pentaho Data Services](#expose-transformation-output-with-pentaho-data-services)
* [Capture lineage with the OpenLineage plugin](#capture-lineage-with-the-openlineage-plugin)
### Run transformations and jobs from the command line
You can use command line tools to execute PDI content outside the PDI client.
Use them in scripts and schedulers, like `cron`.
Use **Pan** to run transformations.
Use **Kitchen** to run jobs.
Command-line reference (Pan, Kitchen, ZIP, export, Hadoop)
#### Startup script options
Pan and Kitchen recognize the startup-script options used by the PDI client.
These options are in `Spoon.bat` (Windows) and `Spoon.sh` (Linux).
To use these options with Pan or Kitchen, add them to your startup script.
**Note:** The default directory for the startup script is `design-tools/data-integration`.
| Option | Description |
| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------- |
| **FILTER\_GTK\_WARNINGS** | Suppresses GTK warnings from `spoon.sh` and `kitchen.sh`. Set to `true` to suppress warnings. Leave empty to show warnings. |
| **SKIP\_WEBKITGTK\_CHECK** | Suppresses warnings about missing `libwebkitgtk` when launching the PDI client. Set to `true` to suppress warnings. Leave empty to show warnings. |
| **KETTLE\_HOME** | Identifies the user's home directory for PDI configuration files. Use it to change the location of files normally in `/.kettle`. |
| **KETTLE\_LOG\_SIZE\_LIMIT** | Limits the log size for transformations and jobs that do not set a log size limit property. |
| **KETTLE\_JNDI\_ROOT** | Changes the Simple JNDI path, which contains `jdbc.properties`. |
| **KETTLE\_DIR** | Directory where the PDI client is installed. |
| **KETTLE\_REPOSITORY** | Repository that Kettle connects to at startup. |
| **LIBPATH** | Value passed as the `-Djava.library.path` Java parameter. |
| **PENTAHO\_DI\_JAVA\_OPTIONS** | Additional Java arguments when running Kettle. Use it for settings like memory limits. |
#### Pan (run transformations)
Pan runs transformations from a PDI repository (database or enterprise) or a local file.
The options are the same for the shell script and batch file.
**Note:** Windows uses the forward slash (`/`) and colon (`:`) syntax. If option values contain spaces, quote the full argument. Example: `"-param:MASTER_HOST=192.168.1.3" "-param:MASTER_PORT=8181"`.
```
pan.sh -option=value arg1 arg2
```
```
pan.bat /option:value arg1 arg2
```
Example:
```
sh pan.sh -rep=initech_pdi_repo -user=pgibbons -pass=lumburgh -trans=TPS_reports_2011
```
```
pan.bat /rep:initech_pdi_repo /user:pgibbons /pass:lumburgh /trans:TPS_reports_2011
```
| Switch | Purpose |
| ------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------- |
| rep | Enterprise repository name. |
| user | Repository username. |
| pass | Repository password. |
| trans | Name of the transformation to run. |
| dir | Repository directory that contains the transformation, including the leading slash. |
| file | Local `.ktr` file path. |
| level | Logging level: Basic, Detailed, Debug, Rowlevel, Error, Nothing. |
| logfile | Log file path. |
| listdir | Lists directories in the specified repository. |
| listtrans | Lists transformations in the specified repository directory. |
| listrep | Lists available repositories. |
| exprep | Exports all repository objects to one XML file. |
| norep | Prevents Pan from logging into a repository. Useful when environment variables like `KETTLE_REPOSITORY` are set, but you want to run a local `.ktr`. |
| safemode | Runs in safe mode with extra checking. |
| version | Shows version, revision, and build date. |
| param | Sets a named parameter in `name=value` format. Example: `-param:Foo=bar`. |
| listparam | Lists information about named parameters in the specified transformation. |
| metrics | Gathers metrics during execution. |
| maxloglines | Maximum number of log lines kept internally. `0` keeps all lines (default). |
| maxlogtimeout | Maximum age (minutes) of a log line kept internally. `0` keeps lines indefinitely (default). |
**Pan status codes**
Pan returns one of these status codes:
| Status code | Definition |
| ----------- | -------------------------------------------------------------- |
| 0 | Transformation ran without a problem. |
| 1 | Errors occurred during processing. |
| 2 | Unexpected error during loading or running the transformation. |
| 3 | Unable to prepare and initialize the transformation. |
| 7 | Transformation could not be loaded from XML or the repository. |
| 8 | Error loading steps or plugins. |
| 9 | Command line usage was printed. |
#### Kitchen (run jobs)
Kitchen runs jobs from a PDI repository (database or enterprise) or a local file.
The options are the same for the shell script and batch file.
**Note:** Windows uses the forward slash (`/`) and colon (`:`) syntax. If option values contain spaces, quote the full argument. Example: `"-param:MASTER_HOST=192.168.1.3" "-param:MASTER_PORT=8181"`.
```
kitchen.sh -option=value arg1 arg2
```
```
kitchen.bat /option:value arg1 arg2
```
| Switch | Purpose |
| ------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------- |
| rep | Enterprise or database repository name. |
| user | Repository username. |
| pass | Repository password. |
| job | Name of the job (as it appears in the repository) to run. |
| dir | Repository directory that contains the job, including the leading slash. |
| file | Local `.kjb` file path. |
| level | Logging level: Basic, Detailed, Debug, Rowlevel, Error, Nothing. |
| logfile | Log file path. |
| listdir | Lists subdirectories within the specified repository directory. |
| listjob | Lists jobs in the specified repository directory. |
| listrep | Lists available repositories. |
| export | Exports all linked resources of the specified job. Argument is a ZIP filename. |
| norep | Prevents Kitchen from logging into a repository. Useful when environment variables like `KETTLE_REPOSITORY` are set, but you want to run a local `.kjb`. |
| version | Shows version, revision, and build date. |
| param | Sets a named parameter in `name=value` format. Example: `-param:FOO=bar`. |
| listparam | Lists information about named parameters in the specified job. |
| maxloglines | Maximum number of log lines kept internally. `0` keeps all lines (default). |
| maxlogtimeout | Maximum age (minutes) of a log line kept internally. `0` keeps lines indefinitely (default). |
Example:
```
sh kitchen.sh -rep=initech_pdi_repo -user=pgibbons -pass=lumburghsux -job=TPS_reports_2011
```
```
kitchen.bat /rep:initech_pdi_repo /user:pgibbons /pass:lumburghsux /job:TPS_reports_2011
```
**Kitchen status codes**
Kitchen returns one of these status codes:
| Status code | Definition |
| ----------- | --------------------------------------------------- |
| 0 | Job ran without a problem. |
| 1 | Errors occurred during processing. |
| 2 | Unexpected error during loading or running the job. |
| 7 | Job could not be loaded from XML or the repository. |
| 8 | Error loading steps or plugins. |
| 9 | Command line usage was printed. |
#### Import `.kjb` or `.ktr` files from a ZIP archive
Pan and Kitchen can read PDI content from ZIP files.
Use the `!` switch.
Windows example:
```
Kitchen.bat /file:"zip:file:///C:/Pentaho/PDI Examples/Sandbox/linked_executable_job_and_transform.zip!Hourly_Stats_Job_Unix.kjb"
```
Linux and Solaris example (escape `!`):
```
./kitchen.sh -file:"zip:file:////home/user/pentaho/pdi-ee/my_package/linked_executable_job_and_transform.zip\!Hourly_Stats_Job_Unix.kjb"
```
#### Export repository content from the command line
To export repository objects into XML format using command-line tools, pass named parameters when calling Kitchen or Pan.
Example (Kitchen):
```
call kitchen.bat /file:C:\Pentaho_samples\repository\repository_export.kjb ^
"/param:rep_name=PDI2000" "/param:rep_user=admin" "/param:rep_password=password" ^
"/param:rep_folder=/public/dev" ^
"/param:target_filename=C:\Pentaho_samples\repository\export\dev.xml"
```
| Parameter | Description |
| ----------------- | ------------------- |
| `rep_folder` | Repository folder |
| `rep_name` | Repository name |
| `rep_password` | Repository password |
| `rep_user` | Repository username |
| `target_filename` | Target filename |
**Note:** You can use obfuscated passwords with **Encr**, the command line tool for encrypting strings for storage and use by PDI.
Example batch file that checks for errors:
```bat
@echo off
ECHO This is an example of a batch file calling repository_export.kjb
cd C:\Pentaho\pdi-ee-\data-integration
call kitchen.bat /file:C:\Pentaho_samples\repository\repository_export.kjb "/param:rep_name=PDI2000" ^
"/param:rep_user=admin" "/param:rep_password=password" "/param:rep_folder=/public/dev" ^
"/param:target_filename=C:\Pentaho_samples\repository\export\dev.xml"
if errorlevel 1 goto error
echo Export finished successful.
goto finished
:error
echo ERROR: An error occurred during repository export.
:finished
REM Allow the user to read the message when testing.
pause
```
#### Use Pan and Kitchen with a Hadoop cluster
To use Pan or Kitchen on a Hadoop cluster, configure Pentaho to run transformations and jobs with the PDI client or the Pentaho Server.
You do not need these configurations if the PDI client connects to the Pentaho Repository.
To use Pan and Kitchen from a repository directly on the Pentaho Server, create the named cluster definition in the server repository.
See [Connecting to a Hadoop cluster with the PDI client](/pdia-data-integration/extracting-data-into-pdi/connecting-to-a-hadoop-cluster-with-the-pdi-client-article).
**Note:** If the PDI client and Pentaho Server run on the same platform, cluster configuration files in `/home//.pentaho/metastore` can be overwritten. Use the same cluster connection names on both hosts.
**Configure the PDI client**
1. Create a connection to the Hadoop cluster where you want to run the job or transformation.
2. Create and test the job or transformation in the PDI client.
3. Go to `design-tools/data-integration/plugins/pentaho-big-data-plugin`.
4. Open `plugin.properties` in a text editor.
5. Set `hadoop.configurations.path` to the directory that contains `metastore`.
Example: `hadoop.configurations.path=/home//.pentaho`
The default metastore location is `/home//.pentaho/metastore`.
6. Save and close `plugin.properties`.
**Configure the Pentaho Server**
1. If the server is on a different host, copy the `metastore` directory and its contents from the PDI client to a location the server can access.
The default metastore location for the PDI client is `/home//.pentaho/metastore`.
2. Go to `pentaho-server/pentaho-solutions/system/kettle/plugins/pentaho-big-data-plugin`.
3. Open `plugin.properties` in a text editor.
4. Set `hadoop.configurations.path` to the directory that contains `metastore`.
5. Save and close `plugin.properties`.
### Run and monitor remotely with Carte
Carte is a lightweight web server for running PDI transformations and jobs remotely.
It receives the transformation or job (as XML) plus the run configuration.
It also exposes endpoints to monitor, start, and stop executions.
Carte setup and reference (clusters, security, client config)
* [Carte clusters](#carte-clusters)
* [Set up servers](#set-up-servers)
* [Configure the PDI client](#configure-the-pdi-client)
* [Run transformations in a cluster](#run-transformations-in-a-cluster)
* [Schedule and run remotely](#schedule-and-run-remotely)
* [Stop Carte](#stop-carte)
#### Carte clusters
Use a Carte cluster to distribute transformation processing across multiple Carte servers.
A cluster includes:
* One **master** node that tracks execution.
* Two or more **slave** nodes that do the work.
You can also run a single Carte instance as a standalone remote execution engine.
Define one or more Carte servers in the PDI client (Spoon), then send jobs and transformations to them.
{% hint style="info" %}
You can cluster Pentaho Server for failover. If you use Pentaho Server as the cluster master (dynamic cluster), enable the proxy trusting filter. See [Schedule jobs to run on a remote Carte server](#schedule-jobs-to-run-on-a-remote-carte-server).
{% endhint %}
**Cluster types**
**Static cluster**
Static clusters have a fixed schema.
You define the master and slave nodes at design time.
Static clusters fit smaller, stable environments.
**Dynamic cluster**
Dynamic clusters discover slave nodes at run time.
Slave nodes are registered with the master.
PDI monitors slaves every 30 seconds to see if they are available.
Dynamic clusters fit cloud-like environments where nodes come and go.
#### Set up servers
**Prerequisites**
* Copy required JDBC drivers and PDI plugins from your dev system to each Carte instance.
* If you will run content from a Pentaho Repository, copy `repositories.xml` from your workstation’s `.kettle` directory to the same location on each Carte server.
**Set up a static cluster (start slave servers)**
1. Start each slave server with the host and port you want to expose:
```sh
./carte.sh 127.0.0.1 8081
```
2. Verify each server is reachable from your PDI client.
3. (Optional) Create an init/startup script to start Carte on boot.
{% hint style="info" %}
When Carte runs embedded in Pentaho Server, configuration is controlled by `slave-server-config.xml` under `.../pentaho-solutions/system/kettle/`. Stop Pentaho Server before editing that file.
{% endhint %}
**Set up a dynamic cluster**
Dynamic clusters use two configuration files:
* `carte-master-config.xml` for the master.
* `carte-slave-config.xml` for each slave.
You can rename the files.
Keep the required XML structure and values.
**Configure a Carte master server**
1. Copy required JDBC drivers and plugins to the master host.
2. Create `carte-master-config.xml` using this template:
```xml
Masteryourhostname9001clusterclusterY
```
The master `` must be unique in the cluster.
3. Start Carte using the master config file:
```sh
./carte.sh carte-master-config.xml
```
4. Verify the master is running.
5. (Optional) Create an init/startup script for boot-time startup.
**Configure Carte slave servers**
1. Ensure the master is running.
2. Copy required JDBC drivers and plugins to each slave host.
3. Create `carte-slave-config.xml` using this template:
```xml
Masteryourhostname9000clusterclusterYYSlaveOneyourhostname9001clusterclusterN
```
Each slave `` must be unique in the cluster.
4. (Optional) To use the master’s Kettle properties on a slave, add these tags inside the slave’s ``:
```xml
MasterY
```
5. Start Carte using the slave config file:
```sh
./carte.sh carte-slave-config.xml
```
6. If you use Pentaho Repository content, copy `repositories.xml` to each slave’s `.kettle` directory.
7. Restart the master and slave servers. Restart Pentaho Server if it participates.
{% hint style="info" %}
Carte and PDI track object age for transformations and jobs. Objects are purged only when servers are idle. Purge verification runs every 20 seconds.
{% endhint %}
**Configure schedule and remote execution log cleanup**
These settings live in `slave-server-config.xml`.
Stop Pentaho Server before editing this file.
* `max_log_lines`: Max log lines per execution. Use `0` for no limit.
* `max_log_timeout_minutes`: Remove log lines older than this value. Use `0` for no timeout.
* `object_timeout_minutes`: Remove execution entries older than this value. Use `0` for no timeout.
Example:
```xml
000
```
#### Security and advanced server settings
**Configure Carte servers for SSL**
Carte SSL uses the JKS keystore format.
Keep the keystore in a restricted-access directory.
Carte runs on Jetty.
For Jetty SSL details, see: .
1. Stop Carte.
2. Open `carte-master-config.xml`.
3. Add these values inside the master server ``:
* `keyStore` (required): Path to the keystore file.
* `keyStorePassword` (required): Keystore password.
* `keyPassword` (optional): Private key password. Omit if it matches `keyStorePassword`.
Example:
```xml
D:\KEY_STORE\PentahoOBF:...OBF:...
```
Use the encr tool in the data-integration directory to obfuscate passwords: encr.bat -carte <password> (Windows) or encr.sh -carte <password> (Linux).
4. Add the same `` block to each `carte-slave-config.xml`.
5. Start Carte.
6. Access Carte over HTTPS:
```
https://:/
```
**Configure Carte servers for JAAS**
You can use JAAS for user authentication.
1. Create a JAAS config file (example below) and save it as `carte-ldap.jaas.conf` on the Carte host:
```conf
Kettle {
org.eclipse.jetty.jaas.spi.LdapLoginModule required
debug="true"
contextFactory="com.sun.jndi.ldap.LdapCtxFactory"
hostname="localhost"
port="389"
bindDn="cn=admin,dc=example,dc=com"
bindPassword="admin"
authenticationMethod="simple"
forceBindingLogin="true"
userBaseDn="ou=People,dc=example,dc=com"
userRdnAttribute="uid"
userIdAttribute="uid"
userPasswordAttribute="userPassword"
userObjectClass="inetOrgPerson";
};
Kettle2 {
org.eclipse.jetty.jaas.spi.PropertyFileLoginModule required
debug="true"
file="/installs/common/carte.users";
};
```
Set debug="false" in production environments.
2. Add these Java options to `Spoon.bat` (Windows) or `spoon.sh` (Linux), updating the path:
```
-Djava.security.auth.login.config=/openldap/carte-ldap.jaas.conf -Dloginmodulename=Kettle
```
3. Start Carte. Verify the server does not prompt for BASIC authentication.
**Change Jetty server parameters**
Carte uses an embedded Jetty server.
Only change these settings if you need to tune connection handling.
* `acceptors`: Threads dedicated to accepting connections. Keep it at or below CPU count.
* `acceptQueueSize`: Backlog size before the OS starts rejecting connections.
* `lowResourcesMaxIdleTime`: Close idle connections faster under high load.
Jetty docs:
*
*
**Set Jetty parameters in a Carte config file**
Add this block inside `` in `carte-slave-config.xml`:
```xml
222
```
Adjust values, then save the file.
**Set Jetty parameters in `kettle.properties`**
Set these variables to numeric values:
* `KETTLE_CARTE_JETTY_ACCEPTORS`
* `KETTLE_CARTE_JETTY_ACCEPT_QUEUE_SIZE`
* `KETTLE_CARTE_JETTY_RES_MAX_IDLE_TIME`
#### Configure the PDI client
**Initialize slave servers**
1. Open a transformation.
2. In **Explorer View**, select the **Slave** tab.
3. Select **New**.
4. Enter the slave server connection details:
* Server name
* Hostname or IP address
* Port (leave blank for port 80)
* Web App Name (required only for Pentaho Server)
* User name and password
* **Is the master**
For clustered executions, define one master and the rest as slaves.
5. Select **OK**.
**Create a cluster schema**
In **Explorer View**, right-click **Kettle cluster schemas**, then select **New**.
Configure:
* **Schema name**
* **Port**: Starting port for slave step numbering.
* **Sockets buffer size**
* **Sockets flush interval rows**
* **Sockets data compressed?**
* **Dynamic cluster**: Enable if a master Carte server performs failover.
* **Slave Servers**: Add one master and any number of slaves.
#### Run transformations in a cluster
* Open the **Run Options** window (toolbar **Run** context menu or `F8`).
* Select a run configuration that runs the transformation in clustered mode.
* To run a clustered transformation from a job, open the **Transformation** job entry, then set **Run this transformation in a clustered mode?** on the **Advanced** tab.
* To assign a cluster to a step, right-click the step, select **Clusters**, then pick a cluster schema.
* When running clustered transformations, enable **Show transformations** to see the generated transformations that run on the cluster.
#### Schedule and run remotely
**Schedule jobs to run on a remote Carte server**
These changes are required to schedule a job to run on a remote Carte server.
They are also required if Pentaho Server acts as the load balancer in a dynamic Carte cluster.
1. Stop Pentaho Server and the remote Carte server.
2. Copy `repositories.xml` from your workstation’s `.kettle` directory to the same location on the Carte host.
3. Open `.../tomcat/webapps/pentaho/WEB-INF/web.xml`.
4. In the **Proxy Trusting Filter** section, add the Carte server IP to `TrustedIpAddrs`.
5. Uncomment the proxy trusting filter mappings between the `` and `` markers.
6. Save `web.xml`.
7. Add `-Dpentaho.repository.client.attemptTrust=true` to the Carte startup script:
* **Windows (`Carte.bat`)**: add to the `OPT` line.
* **Linux (`Carte.sh`)**: add to the `OPT` variable before `export OPT`.
8. Start the Carte server and Pentaho Server.
**Run transformations and jobs from a repository on the Carte server**
Copy `repositories.xml` from the user’s `.kettle` directory to the Carte host’s `$HOME/.kettle` directory.
Carte also looks for `repositories.xml` in the directory where you started Carte.
#### Stop Carte
You can stop Carte from the command line or from a URL.
**Stop from the CLI**
Arguments:
```
Carte [-s] [-p ] [-u ]
```
Example:
```
Carte 127.0.0.1 8080 -s -p amidala4ever -u dvader
```
Options:
* `-h, --help`: Help text.
* `-s, --stop`: Stop the running Carte server.
* `-u, --username `: Admin user name.
* `-p, --password `: Admin password.
**Stop from a URL**
```
http://localhost:8080/kettle/stopCarte
```
### Expose transformation output with Pentaho Data Services
Use Pentaho Data Services to expose a transformation step as a virtual table.
Query it over JDBC using SQL.
{% hint style="info" %}
You need a Pentaho Server and repository to publish a data service.
{% endhint %}
Pentaho Data Services guide (install, create, test, optimize)
Prototyping a data model can be time consuming, particularly when it involves setting up databases, creating the data model and setting up a data warehouse, then negotiating accesses so that analysts can visualize the data and provide feedback.
One way to streamline this process is to make the output of a [transformation](/pdia-data-integration/basic-concepts-of-pdi) step a Pentaho Data Service.
The output of the transformation step is exposed by the data service so that the output data can be queried as if it were stored in a physical table, even though the results of the transformation are not stored in a physical database.
Instead, results are published to the Pentaho Server as a virtual table.
{% hint style="info" %}
You must have a Pentaho Server and repository to publish the data service.
{% endhint %}
The virtual table is a JDBC-compliant data source that you and others can connect to or query with SQL, provided they can access the server and the transformation.
The Pentaho Data Service can be connected to or queried by a JDBC-compliant tool such as Pentaho Report Designer, Pentaho Interactive Reports, and CTools as well as other compatible tools like RStudio, DBVisualizer, or SQuirreL.
The Pentaho Data Service can also be used in some instances where building and maintaining a data warehouse is sometimes impractical or inefficient, especially when you need to quickly blend and visualize fast-moving or quickly evolving data sets on the fly.
For example, if you want to compare your product prices with your competitors, you can create a transformation that blends prices from your in-house data sources and competitor prices.
Then, you can convert the output step in the transformation into a Pentaho Data Service that creates a virtual table for querying when you connect to the Pentaho Server.
You or others can connect to and query the virtual table, as you would any other JDBC data source to visualize the results in Analyzer or another tool.
The Pentaho Data Service also has a testing tool.
This tool generates several logs and reports that you can use to refine the data service and determine where to apply specialized optimizations.
You can also define parameters that others can use to pose customized queries.
For example, you can create a data service that publishes a virtual “fact” table of a moderately-sized research dataset to a Pentaho Server.
You can test and add optimizations and parameters, such as gender or test type so that the data service runs more quickly.
Then, you can share connection and parameter information with a group of researchers, who can query the virtual table.
Researchers can use Pentaho Interactive Reporting, a dashboard created with CTools, or an application of their choice, such as RStudio, to analyze and visualize the research dataset.
Pentaho Data Services support a subset of SQL.
For details, see [SQL support reference](#sql-support-reference).
{% hint style="info" %}
Install the Pentaho Data Services plugin before you create data services.
{% endhint %}
#### In this article
* [Install](#install-pentaho-data-services)
* [Create](#create-a-regular-or-streaming-pentaho-data-service)
* [Open or edit](#open-or-edit-a-pentaho-data-service)
* [Delete](#delete-a-pentaho-data-service)
* [Test](#test-a-pentaho-data-service)
* [SQL support reference](#sql-support-reference)
* [Optimize](#optimize-a-pentaho-data-service)
* [Publish](#publish-a-pentaho-data-service)
* [Share](#share-a-pentaho-data-service-with-others)
* [Monitor](#monitor-a-pentaho-data-service)
#### Install Pentaho Data Services
Pentaho Data Services plugin is an optional plugin.
**Pentaho Data Integration**
Install the plugin in Pentaho Data Integration by using the Plugin Manager. For instructions, see [Install plugins in PDI client](/pdia-data-integration/use-plugin-manager#install-plugins).
**Pentaho Server**
Install the plugin on Pentaho Server by using the Plugin manager. For instructions, see [Install plugins in PUC](/pba/pentaho-user-console/modern-design/plugin-manager#install-plugins-in-puc).
After you have installed the plugin, add the driver for the Pentaho Data Services plugin by completing the following steps:
1. In the PDI client, open a new transformation.
2. In the **View** tab, expand the **Configurations** section.
3. Right-click **Data Service** and select **Driver Details**.
4. In the **Driver Details** window, click **Get Pentaho Driver**.
5. In the **Save As** window, indicate where you want to save the `PDI-Data-Service-Driver-Pentaho.jar` file, then click **Save**.
6. Move or copy the `PDI-Data-Service-Driver-Pentaho.jar` into the `pentaho-server/tomcat/webapps/pentaho/WEB-INF/lib` folder.
7. Close the transformation without saving.
8. Restart the Pentaho Server.
#### Create a regular or streaming Pentaho Data Service
You can create either a regular data service or a [streaming analytics](/pdia-data-integration/extracting-data-into-pdi/streaming-analytics) service.
Streaming data services are commonly used when creating streaming dashboards with CTools.
1. [Create or open a transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp).
2. Save the transformation to the Pentaho Server.
3. Right-click the transformation step that outputs the data you want.
4. Select **Data Services** > **New**.
5. Enter a unique name in **Service Name (Virtual Table Name)**.
6. Confirm **Output step** is the step you selected.
7. Optional: Select **Streaming** for **Data Service Type**.
8. Select **OK**.
**Data service badge**
After you create a Pentaho Data Service from a step, a data service badge is added to that step.
The badge indicates whether the step has a regular or streaming data service.
#### Open or edit a Pentaho Data Service
To open a data service for viewing or editing, do one of the following in the PDI client:
* In **Explore** > **View**, select **Data Services**. Right-click a data service, then select **Edit**.
* Right-click the step with the data service badge, then select **Data Services** > **Edit**.
#### Delete a Pentaho Data Service
To delete a data service, do one of the following in the PDI client:
* In **Explore** > **View**, select **Data Services**. Right-click a data service, then select **Delete**.
* Right-click the step with the data service badge, then select **Data Services** > **Delete**.
#### Test a Pentaho Data Service
After creating your data service, test it to ensure that it runs properly and generates the data you need.
Testing can uncover bottlenecks in your transformation and help you decide which [optimization techniques](#optimize-a-pentaho-data-service) to apply.
**Run a basic test**
To run a basic test on a regular data service:
1. Verify **Data Service Type** is set to **Regular**.
2. Open the **Test Data Service** window using one of the following:
* In the Data Service window, select **Test Data Service**.
* In **Explore** > **View** > **Data Services**, right-click a data service and select **Test**.
* Right-click the step with the data service badge, then select **Data Services** > **Test**.
3. Optional: Adjust settings:
* **Log Level**. Controls how much detail appears in logs.
* **Max Rows**. Limits how many rows appear in results.
4. Select **Execute SQL**.
5. Review the output in [Examine test results](#examine-test-results).
6. Optional: If you need a clean run, [clear the cache](#clear-the-cache) and test again.
7. Select **Close**.
8. Optional: Add an [optimization](#optimize-a-pentaho-data-service).
9. [Publish the data service](#publish-a-pentaho-data-service).
**Run a streaming optimization test**
When you test streaming data, the stream is partitioned into windows (batches).
Windows can be time-based or row-based.
To test a streaming data service:
1. Verify **Data Service Type** is set to **Streaming**.
2. Open the **Test Data Service** window.
3. Select a window mode (**Time Based** or **Row Based**).
4. Configure window settings:
* **Window Size**
* **Every**
* **Limit**
5. Optional: Adjust **Log Level** and **Max Rows**.
6. Select **Execute SQL**.
7. Review the output in [Examine test results](#examine-test-results).
8. Select **Stop** to stop execution.
9. Select **Close**.
10. Optional: Add an [optimization](#optimize-a-pentaho-data-service).
11. [Publish the data service](#publish-a-pentaho-data-service).
**Run an optimization test**
If you have added an optimization, run a test that passes the optimization parameter.
Example query:
```sql
SELECT * FROM Mars WHERE rover='Curiosity'
```
* Select **Preview Optimization** in the Test Data Service window.
* Use [Examine test results](#examine-test-results) to interpret results.
**Examine test results**
Test results appear in the tabs in the bottom half of the Test Data Service window.
When you test or run a data service, two transformations run:
* The service transformation. This is the transformation you built.
* The generated transformation. PDI generates this based on executed SQL.
The following table describes the tabs and what to look for:
| Tab | Description | Tips |
| ------------------------------------ | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------ |
| **Query Results** | Events during the test run and query results. | Verify results. For streaming, watch updates per window. |
| **Optimized Queries** | Processing information and results of optimizations. | Verify optimizations applied correctly. Compare with **Service Metrics** and **Generated Transformation Logging**. |
| **Service Transformation Logging** | Logs from the service transformation. | Check for design or runtime issues. Compare with **Generated Transformation Logging**. |
| **Generated Transformation Logging** | Logs from the generated transformation. | Check SQL support issues. See [SQL support reference](#sql-support-reference). |
| **Service Metrics** | GANTT chart timings for the service transformation. | Find bottlenecks. Compare with **SQL Trans Metrics**. |
| **SQL Trans Metrics** | GANTT chart timings for the generated transformation. | Find bottlenecks. Compare with **Service Metrics**. |
#### SQL support reference
The Pentaho Data Service is designed to support a subset of SQL clauses and literals that are useful for data blending, optimizations, and other scenarios.
Limitations and constraints are listed at the end.
**Supported SQL literals**
The Pentaho Data Service supports the following literals:
* Strings use single quotation marks. Escape a single quote using another single quote: `''`.
* Dates use square brackets. Supported formats:
* `[yyyy/MM/dd HH:mm:ss.SSS]`
* `[yyyy/MM/dd HH:mm:ss]`
* `[yyyy/MM/dd]`
* For an `IN` list, date formats can use single quotes and dashes, for example:
* `SELECT * FROM BUILDS WHERE BuildDate IN ('2015-03-18', '2015-03-22')`
* You cannot use bracketed date formats in an `IN` list.
* `Number` and `BigNumber` cannot use grouping symbols. Use `.` for decimals, for example `123.45`.
* Integers contain digits only.
* Boolean values can be `TRUE` or `FALSE`.
**Supported SQL clauses**
The Pentaho Data Service supports the following clauses.
| Clause | What is supported |
| ---------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `SELECT` |
COUNT(field)
COUNT(\*)
COUNT(DISTINCT field)
DISTINCT \
IIF (condition, true-value or field, false-value or field)
CASE WHEN condition THEN true-value ELSE false-value END
SUM
AVG
MIN
MAX
Aliases with AS or with spaces. Example: SUM(sales) AS "Total Sales" or SUM(sales) TotalSales
Constant expressions. See Supported SQL literals.
|
| `FROM` |
Only one Pentaho service name.
Aliases for the service name.
Omit the service name to query an empty row. SELECT 1 and SELECT 1 FROM dual are the same.
|
| `WHERE` |
Nested brackets
AND, OR, NOT followed by brackets (example: NOT (A=5 OR C=3))
Precedence
Literals (String and Integer)
PARAMETER('parameter-name')='value' (always evaluates to TRUE)
Operators: =, <, >, <=, =<, >=, =>, <>
LIKE (wildcards: %, ?)
REGEX
IS NULL, IS NOT NULL
IN
Conditions on IIF or its alias
DATE\_TO\_STR(date-field, \)
|
| `GROUP BY` |
Group on fields. Not on IIF().
|
| `LIMIT` |
LIMIT
LIMIT offset, count
LIMIT count OFFSET offset
|
| `HAVING` |
Apply conditions to aggregates, not aliases.
Use identical strings for expressions.
Use conditions on aggregations not in SELECT.
|
| `ORDER BY` |
Order on any column, even if not in the result.
Order on IIF or CASE-WHEN expressions.
|
**Other development considerations**
Keep these constraints in mind:
* You cannot `JOIN` one data service virtual table to another.
* Data services use the Memory Group By step to group. Watch memory consumption for many groups.
* You cannot specify the same field twice in the same `SELECT` clause.
* Calculations and functions like string concatenation are not supported in queries. Do them in the transformation.
#### Optimize a Pentaho Data Service
As you [test your data service](#test-a-pentaho-data-service), you might see bottlenecks or steps that could run more efficiently.
If you want to improve performance, apply an optimization technique.
**Service cache**
This optimization stores results in a cache.
By default, caching is enabled and results are stored for an hour.
Use this technique when result sets are modest and you expect repeat queries.
**How the service cache optimization works**
If you run the data service while results are cached, PDI can run your query against cached data instead of running the full transformation.
This behavior depends on:
* Whether the cached dataset contains all the records required by the new query
* Other optimizations you apply
When you run tests from the Test Data Service window and change the SQL, PDI does not use the cache.
Tests only return up to **Max Rows**, so using cached results could return incomplete results.
**Adjust the cache duration**
1. In the Data Service window, select the **Service Cache** tab.
2. Select **Enable Caching**.
3. Update **Cache Duration (seconds)**.
4. Select **OK**.
5. [Run an optimization test](#run-an-optimization-test).
6. [Publish the data service](#publish-a-pentaho-data-service).
**Disable the cache**
1. In the Data Service window, select the **Service Cache** tab.
2. Clear **Enable Caching**.
3. Select **OK**.
**Clear the cache**
1. In the Data Service window, select the **Service Cache** tab.
2. Clear **Enable Caching**.
3. Select **OK**.
4. Open the Data Service window again and return to the **Service Cache** tab.
5. Select **Enable Caching**.
**Query pushdown**
Use Query Pushdown to translate the SQL `WHERE` clause into a corresponding `WHERE` clause in:
* Table Input steps
* MongoDB Input steps
The input queries are filtered and handled at the source.
**How query pushdown works**
To apply Query Pushdown, set the optimization values, then add the optimization parameter to the input step query.
The optimization uses a parameter in place of the `WHERE` clause, for example:
`SELECT * FROM 'employee' WHERE ${countryParam}`
**Add the query pushdown parameter to Table Input or MongoDB Input**
1. [Create a transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/create-a-transformation) with a Table Input or MongoDB Input step.
2. [Run your transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/run-your-transformation).
3. Open the input step.
4. Add a parameter where your `WHERE` clause value belongs:
* SQL: `SELECT * FROM media WHERE ${countryParam}`
* MongoDB: `{$match : ${mongoDbParam}}`
5. Press CTRL+SPACE to list parameters.
6. Select **Replace Variables in Script?**.
7. Select **OK**.
8. [Set up query pushdown parameter optimization](#set-up-query-pushdown-parameter-optimization).
**Set up query pushdown parameter optimization**
1. Open the Data Service window and select the **Query Pushdown** tab.
2. Select the plus sign near **Parameters**.
3. Enter the name of the optimization parameter you used in the input query.
4. Select **OK**.
5. Select the step that contains the parameter in **Step Name**.
6. In **Definitions**, map fields:
* **Data Service Field**. Field name as it appears in transformation output.
* **Step Field**. Field name as it appears in the data source.
7. Optional: Select **Get Optimizations** to generate optimizations automatically.
8. Select **OK**.
9. Test and publish:
* [Run an optimization test](#run-an-optimization-test).
* [Publish the data service](#publish-a-pentaho-data-service).
**Disable the query pushdown optimization**
To disable this optimization, select **Disable an Optimization** in the Data Services window.
**Parameter pushdown**
Parameter Pushdown can be applied to any step in the transformation.
It maps a field value to a transformation parameter for simple equality conditions (example: `WHERE region="South"`).
**How parameter pushdown works**
To set up Parameter Pushdown, configure the optimization, then add the parameter to the step.
**Add the parameter pushdown parameter to the step**
1. [Create a transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/create-a-transformation).
2. [Run your transformation](/pdia-data-integration/archived-merged-pages/transforming-data-with-pdi-archive/work-with-transformations-cp/run-your-transformation).
3. Add the parameter to a step, such as JSON Input or REST Client.
4. Select **OK**.
5. [Set up parameter pushdown optimization](#set-up-parameter-pushdown-optimization).
**Set up parameter pushdown optimization**
1. Open the Data Service window and select the **Parameter Pushdown** tab.
2. In **WHERE Clause Column**, enter the data service field name from the existing list.
3. Update **Transformation Parameter** as needed. The name must be unique in the data service.
4. Optional: Update **Value Format**. Default `%s` is usually sufficient.
5. Select **OK**.
6. Test and publish:
* [Run an optimization test](#run-an-optimization-test).
* [Publish the data service](#publish-a-pentaho-data-service).
**Streaming optimization**
This optimization limits the batch size used for processing.
Streaming records are partitioned into windows (batches) for processing.
**How streaming optimization works**
You can optimize processing by setting either:
* **Rows Limit**. Maximum rows in a window.
* **Time Limit**. Maximum elapsed time to create a window.
**Adjust the row or time limits**
1. In the Data Service window, select the **Streaming** tab.
2. Update **Rows Limit** or **Time Limit**.
3. Select **OK**.
4. Test and publish:
* [Run an optimization test](#run-an-optimization-test).
* [Publish the data service](#publish-a-pentaho-data-service).
#### Publish a Pentaho Data Service
To publish your data service, save the transformation containing the data service to the Pentaho Repository.
To validate that your data service was published:
1. Ensure external assets required by the transformation are accessible on the server.
2. Open a browser and log in to the Pentaho Server.
3. List data services on the server, for example:
```
http:///pentaho/kettle/listServices
```
#### Share a Pentaho Data Service with others
Once your data service is created and tested, you can share it so others can connect and query it.
**Share prerequisites**
1. Ensure the user or group has permissions to:
* Access the Pentaho Server and run queries.
* Read and run the transformation that contains the data service.
2. Provide the data service name as saved in the Pentaho Repository.
3. If you use optimizations, provide names and definitions for any parameters.
4. Provide connection instructions:
* [Connect from a Pentaho tool](#connect-to-the-pentaho-data-service-from-a-pentaho-tool)
* [Connect from a non-Pentaho tool](#connect-to-the-pentaho-data-service-from-a-non-pentaho-tool)
**Connect to the Pentaho Data Service from a Pentaho tool**
A Pentaho Data Service is a virtual table containing the output of a step in a PDI transformation.
You can connect to and query:
* A regular data service from Pentaho tools such as Report Designer, Analyzer, and the PDI client.
* A streaming data service from a dashboard created with CTools.
{% hint style="info" %}
You need the data service name and permission to run the transformation and access the Pentaho Server.
{% endhint %}
Connection parameters:
* **Connection Name**. Unique name of the data service.
* **Connection Type**. Pentaho Data Services.
* **Access**. Native (JDBC).
* **Hostname**. Pentaho Server hostname or IP address. Default is `localhost` for local installs.
* **Port Number**. Pentaho Server port. Default is `8080`.
* **Web App Name**. Default is `pentaho`.
* **Username**. A user allowed to run the data service.
* **Password**. Password for that user.
Optional connection parameters:
* `proxyhostname`. Proxy server for HTTP connections.
* `proxyport`. Proxy server port.
* `nonproxyhosts`. Hosts that bypass the proxy. Separate multiple hosts with commas.
* `debugtrans`. File path where PDI saves the generated debug transformation. Example: `/tmp/debug.ktr`.
* `PARAMETER_[optionname]=value`. Sets a transformation parameter. Example: `PARAMETER_model=E6530`.
* `secure`. Set to `TRUE` for HTTPS. Default is HTTP.
**Connect to the Pentaho Data Service from a non-Pentaho tool**
You can connect to and query a data service from tools like RStudio or SQuirreL.
To connect and query, you need the data service name and permission to run the transformation and access the server.
To use a non-Pentaho tool, install the data service JDBC driver, then create a connection in your tool.
**Step 1: Download the Pentaho Data Service JDBC driver**
You can download the driver using the PDI client or manually.
**Download using the PDI client**
1. Open the transformation and identify the step with the data service badge.
2. Open **Driver Details** using one of the following:
* Right-click the step and select **Data Service** > **Driver Details**.
* In **Explore** > **View** > **Data Services**, right-click a data service and select **Driver Details**.
* In the Data Services window, select **Driver Details**.
3. In **Driver Details**, select **Get Third-Party Driver**.
4. Save `Pentaho-Data-Service-Driver.zip`.
5. Select **Close**.
**Download manually**
1. Go to `/design-tools/data-integration/plugins/pdi-dataservices-ee-plugin/Data Service JDBC Driver`.
2. Copy `pdi-dataservice-driver-thirdparty-.jar`.
{% hint style="info" %}
These steps assume you installed the Pentaho Data Services plugin. See [Install Pentaho Data Services](#install-pentaho-data-services).
{% endhint %}
**Step 2: Install the Pentaho Data Service JDBC driver**
1. Extract the driver files and copy them to your application's JDBC driver directory.
2. Start and stop the application.
3. Create a connection using the instructions in [Connect to the Pentaho Data Service from a non-Pentaho tool](#connect-to-the-pentaho-data-service-from-a-non-pentaho-tool).
**Step 3: Create a connection from a non-Pentaho tool**
Most tools let you create a JDBC connection by specifying a driver class and connection string.
Driver class:
`org.pentaho.di.trans.dataservice.jdbc.ThinDriver`
Connection string format:
```
jdbc:pdi:///kettle?option=value&option=value
```
Example:
```
jdbc:pdi://localhost:8080/kettle?webappname=pentaho
```
`webappname` is required when the data service runs on Pentaho Server.
**Query a Pentaho Data Service**
You can query a data service using SQL.
If the transformation uses a parameter, you can assign a value in your SQL query.
Limitations:
* SQL support is limited. See [SQL support reference](#sql-support-reference).
* Data services can be queried only with SQL.
**Example: query with a parameter**
Syntax:
```sql
SELECT * FROM ''
WHERE PARAMETER('') = ''
```
Example:
```sql
SELECT *
FROM 'employeeList'
WHERE PARAMETER('employeeRegion') = 'USA EAST'
```
#### Monitor a Pentaho Data Service
To monitor a data service, enter one of the following URLs in your browser:
* List names of data services:
```
http:///pentaho/kettle/listServices
```
* List data service status:
```
http:///pentaho/kettle/status
```
Replace `` with the host name or IP address and port for the Pentaho Server running the data service.
You must be authenticated and authorized on the Pentaho Server to access these endpoints.
### Capture lineage with the OpenLineage plugin
Use the OpenLineage plugin to emit standardized lineage events from PDI.
Use these events in Pentaho Data Catalog (PDC) for end-to-end data flow visibility.
OpenLineage plugin guide (setup, configuration, supported steps)
The Pentaho Data Integration (PDI) OpenLineage plugin enables PDI to emit rich, standardized OpenLineage events that can be consumed by Pentaho Data Catalog (PDC) to capture how data moves and is transformed in PDI ETL pipelines.
PDC uses information it captures to provide visual end-to-end transparency of data flows, which improves data observability, strengthens compliance and governance, aids in troubleshooting data issues, and enhances data trust and quality for business users.
OpenLineage events are emitted from PDI when supported transformations are executed by discovering input and output datasets and, when possible, generating column-level lineage.
The OpenLineage plugin emits events for:
* **Start**: transformation starts
* **Complete**: transformations ends
* **Abort**: transformation was stopped without errors
* **Fail**: transformation ended with errors
#### Compatibility matrix
OpenLineage plugin functionality is certified to work as intended for the following versions of PDI:
* 10.2.0.1 (SP1)
* 10.2.0.2 (SP2)
* 10.2.0.3 (SP3)
* 10.2.0.4 (SP4)
* 10.2.0.5 (SP5)
* 10.2.0.6 (SP6)
* 11.0
#### Setting up the plugin
Before you begin, verify that you have a valid license for the OpenLineage plugin.
For information about licenses, see [Acquire and install enterprise licenses](/install/pentaho-installation-overview-cp/acquire-and-install-enterprise-licenses).
To set up the OpenLineage plugin, you must complete the following tasks:
* [Download the plugin](#download-the-plugin)
* [Install the plugin](#install-the-plugin)
* [Create a configuration file for the plugin](#create-a-configuration-file-for-the-plugin)
* [Enable the plugin](#enable-the-plugin)
* [Validate the plugin works](#validate-the-plugin-works)
**Download the plugin**
Download the OpenLineage plugin from the Pentaho Support Portal.
1. On the [Support Portal](https://support.pentaho.com/hc/en-us) home page, sign in using the Pentaho support username and password provided in your Pentaho Welcome Packet.
2. In the **Pentaho** card, click **Download**. The **Downloads** page opens.
3. In the **.x** list, click **Pentaho EE Marketplace Plugins Release**.
4. Scroll to the bottom of the page.
5. In the **Marketplace Plugins** section, click **Open Lineage**.
6. Download the `pdi-openlineage-plugin--.zip` file.
**Install the plugin**
Install the OpenLineage plugin in the PDI client and Pentaho Server by running commands appropriate for your operating system.
{% hint style="info" %}
**Note:** The plugin can be installed in the PDI client, Pentaho Server, or both.
{% endhint %}
Installation commands include the following placeholders that must be replaced:
* ``: Replace with full path to the PDI client.
* ``: Replace with full path to the Pentaho Server.
* ``: Replace with one of the following options:
* `none`: Installs the plugin on any version of Pentaho. If the Pentaho version is unsupported, an error is shown.
* `loose`: Default option. Installs the plugin on certified and compatible, newer Pentaho versions.
* `strict`: Installs plugin only on certified Pentaho versions.
To install the OpenLineage plugin, complete the following steps:
1. Stop the PDI client and Pentaho Server.
2. Extract the `pdi-openlineage-plugin--.zip` file to a folder on the computer where the PDI client or PDI Server is installed.
3. In the `pdi-openlineage-plugin--` folder, open a command prompt as an administrator.
4. In the command prompt, run the following installation commands for your operating system, replacing the placeholders for paths and version check options.
* Windows
* PDI client
`install.bat -t --platformVersionCheck `
* PDI Server
`install.bat -t --platformVersionCheck `
* Linux
* PDI client
`./install.sh -t --platformVersionCheck `
* PDI Server
`./install.sh -t --platformVersionCheck `
5. Start the PDI client and Pentaho Server.
**Generate an encrypted password**
If you plan to emit events to PDC, and want to secure your password so that it's not in plain text, you can generate an encrypted password to authenticate to PDC.
The encrypted password is used in the configuration file for the OpenLineage plugin.
1. On the computer where the PDI client or PDI Server is installed, open a command prompt.
2. Run one of the following commands for your operating system:
* **Windows**
* To generate a password using the default Pentaho encryption seed, run the following command:
```bash
cd # or
sh encr.bat
```
* To generate a password using your own custom encryption seed, run the following command:
```bash
export KETTLE_TWO_WAY_PASSWORD_ENCODER_SEED=your-custom-seed"
cd # or
sh encr.bat
```
* **Linux**
* To generate a password using the default Pentaho encryption seed, run the following command:
```bash
cd / # or
sh encr.sh
```
* To generate a password using your own custom encryption seed, run the following command:
```bash
export KETTLE_TWO_WAY_PASSWORD_ENCODER_SEED=your-custom-seed"
cd / # or
sh encr.sh
```
An encrypted password is generated and displayed in the command prompt, like the following example:
```
Encrypted 2be98afc86aa7f297a414ab3dce93bcc9
```
**Create a configuration file for the plugin**
After you install the plugin, create a configuration file that specifies where to send open lineage events.
You can create a simple configuration file for testing or a custom configuration to use in production.
1. In a text editor, create a configuration file with content from one of the following examples, based on your needs:
* To create a simple configuration file that you can use to quickly validate that the plugin is working, include only the following content:
```
version: 0.0.1
consumers:
console:
```
* To create a custom configuration file that includes OpenLineage event consumers in your Pentaho deployment, such as a PDC Server, include the following content:
```
version: 0.0.1
localHostname: # optional
debugMode: false # PDI client (Spoon) only
consumers:
console:
file:
- path: //openlineage.json
http:
- name: PDC
url: https://
endpoint: /lineage/api/events
authenticationParameters:
endpoint: /keycloak/realms/pdc/protocol/openid-connect/token
username:
password:
client_id: pdc-client
scope: openid
```
2. Save the file as `openlineageConfig.yml` in the PDI directory that contains your user-specific configuration files.
**Notes:**
* By default, user-specific configuration files are stored in the `.kettle` directory, which is usually in one of the following locations:
* Windows: `C:\Documents and Settings\example_user\.kettle`
* Linux: `~/.kettle)`
However, if you run PDI in a container, configuration files might resolve to the `/root/.kettle` directory.
* You can add multiple http consumers in the configuration file.
**Enable the plugin**
After you install the OpenLineage plugin and create its configuration file, you must enable the plugin so that it can send open lineage events to the consumers you specified in the configuration file.
**Enable in PDI client**
Enable the plugin in the PDI client by completing the following steps:
1. Log into the PDI client and click **Edit** > **Edit the Kettle.properties file**. The Kettle properties window opens.
2. To make the plugin active, add the following variable and value: `KETTLE_OPEN_LINEAGE_ACTIVE=true`
3. To point PDI to your `openlineageConfig.yml` file, add the following variable with the `` placeholder replaced by the full path to your configuration file directory: `KETTLE_OPEN_LINEAGE_CONFIG_FILE=//openlineageConfig.yml`
4. Click **OK**. The `kettle.properties` file is saved and the OpenLineage plugin is enabled.
**Enable in Pentaho Server**
Enable the client in the Pentaho Server, by completing the following steps:
1. Navigate to the `kettle.properties` file.
**Note:** The `kettle.properties` file is usually in one of the following locations:
* Windows: `C:\Documents and Settings\example_user\.kettle`
* Linux: `~/.kettle)`
If you run PDI in a container, the `kettle.properties` file is in the `/root/.kettle` directory.
2. Open the `kettle.properties` file in a text editor.
3. Enable the plugin with its configuration file by adding the following variables and values:
`KETTLE_OPEN_LINEAGE_ACTIVE=true`
`KETTLE_OPEN_LINEAGE_CONFIG_FILE=//openlineageConfig.yml`
4. Save the `kettle.properties` file.
**Validate the plugin works**
You can validate that the plugin is working by verifying that text related to OpenLineage appears in the appropriate logs and files.
To validate that the plugin is working, complete the following steps:
1. In the PDI client, click **File > Open**, and then navigate to sample transformations in your Pentaho folder. For example, in Windows the sampls are in `\Pentaho\design-tools\data-integration\samples\transformations`.
2. Select the sample transformation, `TextInput and Output using variables.ktr`, and click **Open**.
3. To run the transformation click **Action** > **Run,** and then in the **Run Options** window, click **Run**. The transformation runs and **Execution Results** pane appears at the bottom of the PDI client.
4. Validate that consumers you have enabled are receiving OpenLineage events by taking one of the following actions:
* If the `console` consumer is enabled, in the **Execution Results** pane of the PDI client, click the **Logging** tab and verify that the log contains lines with the text, "`OpenLineage-Plugin`".
* If a `file` consumer is enabled, open the `openlineage.json` file in a text editor and verify that it contains lines with the text, "`OpenLineage-Plugin`". The `openlineage.json` file location is defined in the `openlineageConfig.yml` file.
* If an `HTTP` consumer is enabled, confirm OpenLineage events are arriving for that consumer. For example, if the PDC is a configured consumer, verify the events arrive in PDC.
**Troubleshoot plugin**
If you are unable to validate that the plugin is working, perform the following troubleshooting actions:
* Verify dataset lineage (input text file -> output text file) and column lineage mappings.
* Validate that the `Kettle.properties` file contains the following variable and value: `KETTLE_OPEN_LINEAGE_ACTIVE=true`.
* Verify that the credentials specified in the `openlineageConfig.yml` file are correct.
* Check your network and firewall settings.
#### Supported steps
Note: This list of supported steps is for version 0.5.0 of the plugin.
**Steps that support dataset lineage and column-level lineage**
* Abort
* Append Streams
* Block this step until steps finish
* Blocking Step
* Data Grid
* Delay Row
* Delete
* Dummy
* Filter Rows
* Generate Rows
* Get Variables
* Group By
* Java Filter
* Mail
* Merge Join
* Microsoft Excel Input
Lineage is supported for local files, AWS, Mineo, HCP, and other S3-compatible connections.
* Microsoft Excel Output (deprecated)
Lineage is supported for local files, AWS, Mineo, HCP, and other S3-compatible connections. \[1]
* Microsoft Excel Writer
Lineage is supported for local files, AWS, Mineo, HCP, and other S3-compatible connections. \[1]
* Prioritize streams
* S3 CSV Input
* S3 File Output \[1]
* Send message to syslog
* Set Variables
* Sort Rows
* Switch/Case
* Table input
Lineage is supported for the following connections, using the listed SQL functions and clauses:
* Connection types: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Google BigQuery, Redshift, and Generic Connection \[2]
* SQL functions: aliases, joins, subqueries, functions, aggregations, constants, expressions, cases, window functions, CTEs, and the set operators: unions, intersects, and excepts.
* Clauses: GROUP BY, ORDER BY, WHERE, WITH, and HAVING.
* Table output
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* Text file input
Lineage is supported for local files, AWS, Mineo, HCP, and other S3- compatible connections. Fixed filetype is not supported.
* Text file output
Lineage is supported for local files, AWS, Mineo, HCP, and other S3- compatible file systems. \[1] Fixed filetype is not supported.
* Write to Log
**Steps that support only dataset lineage, not column-level lineage**
* Combination lookup/update
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* CSV File Input
* Database Lookup
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* De-serialize from file
* Dimension lookup/update
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* Fixed file input
* Gzip Csv Input
* Insert/Update
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* JSON Input
* JSON Output \[1]
* LDIF Input
* Load file content in memory
* Property Input
* Properties Output \[1]
* Sql File Output \[1]
* Synchronize after merge
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* Update
Lineage is supported for the following connections: MySQL, PostgreSQL, Denodo, Sybase, Oracle, Vertica, SQL Server, Snowflake, Redshift, and Generic Connection. \[2]
* XBase Input
**Notes:**
\[1] Step, which can create multiple files as its output, can be configured to add filenames to its results file so that the name of each file is recorded in lineage. If the `Add filenames to result` option is disabled for the step, only a single, generic target is recorded in lineage. For example, if the `Add filenames to result` option is enabled for the step, the output is recorded in lineage as `_001.csv`, `_002.csv`, `_003.csv`, and so on. But, if the option is disabled, the output is recorded as only `.csv`.
\[2] Step allows generic connections, but lineage works only with generic connections that are listed as supported.
{% hint style="info" %}
**Note:** The Google Big Query connection is not supported on table output step. An OpenLineage event won't have any dataset outputs from any Google Big Query storage.
{% endhint %}
#### Uninstall plugin
Uninstall the OpenLineage plugin from the PDI client and Pentaho Server by running commands appropriate for your operating system.
Before you begin, you must download the OpenLineage plugin from the Pentaho Support Portal, which contains script files for uninstalling the plugin. For details, see [Download the plugin](#download-the-plugin).
{% hint style="info" %}
**Note:** The plugin can be uninstalled from the PDI client, Pentaho Server, or both.
{% endhint %}
Commands for uninstalling the plugin include the following placeholders that must be replaced:
* ``: Replace with full path to the PDI client.
* ``: Replace with full path to the Pentaho Server.
* ``: Replace with one of the following options:
* `none`: Installs the plugin on any version of Pentaho. If the Pentaho version is unsupported, an error is shown.
* `loose`: Default option. Installs the plugin on certified and compatible, newer Pentaho versions.
* `strict`: Installs plugin only on certified Pentaho versions.
To uninstall the OpenLineage plugin, complete the following steps:
1. Stop the PDI client and Pentaho Server.
2. Extract the `pdi-openlineage-plugin--.zip` file to a folder on the computer where the PDI client or PDI Server is installed.
3. In the `pdi-openlineage-plugin--` folder, open a command prompt as an administrator.
4. In the command prompt, run the following installation commands for your operating system, replacing the placeholders for paths and version check options.
* Windows
* PDI client
`uninstall.bat -t --platformVersionCheck `
* PDI Server
`uninstall.bat -t --platformVersionCheck `
* Linux
* PDI client
`./uninstall.sh -t --platformVersionCheck `
* PDI Server
`./uninstall.sh -t --platformVersionCheck `
5. Start the PDI client and Pentaho Server.
#### Upgrade plugin
{% hint style="info" %}
**Important**: Do not install a new version of the OpenLineage plugin over an existing installation of the plugin.
{% endhint %}
To upgrade the OpenLineage plugin, you must uninstall the plugin and then download and install the new version of the plugin.
For details, see the following sections:
* [Uninstall the plugin](#uninstall-plugin)
* [Download the plugin](#download-the-plugin)
* [Install the plugin](#install-the-plugin)
### Archived source pages
These pages were merged into this single topic page:
* [Use Command Line Tools to Run Transformations and Jobs](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/use-command-line-tools-to-run-transformations-and-jobs)
* [Use Carte Clusters](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/use-carte-clusters)
* [Pentaho Data Services](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/pentaho-data-services)
* [OpenLineage Plugin](/pdia-data-integration/archived-merged-pages/loading-data-from-pdi-archive/openlineage-plugin)
---
[Next Page](/llms-full.txt/1)