Adding a data source

If your role has permission to administer data sources, you can add and edit data sources.

The number of data sources you can add is determined by your license agreement. You receive a message when you have reached 75% of your data source creation quota.

If you have reached the limit of data sources allowed by your license agreement, the Add Data Source button on the Resources card is unavailable, and a message appears when you hover your cursor over the button.

If you encounter an error while connecting to a data source, refer to the documentation of the specific data source provider for more information about the error.

Active Directory as a data source

Data Catalog supports integration with both Windows-based Active Directory (AD) and Azure Active Directory (Azure AD). You can add Active Directory as a data source to import file system security identifiers (SIDs), GUIDs, and security descriptors, and map them to user identities. With this integration, Data Catalog displays the ownership and group information for files and folders from SMB, CIFS, and NFS data sources in Data Canvas.

Perform the following steps to add Active Directory as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Active Directory in the Data Source Type field.

   Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Configuration Method	Select the method used to connect to Active Directory. Options include: LDAP: Establishes a standard, non-encrypted connection. Secure LDAP: Establishes an encrypted connection using SSL/TLS. When you select Secure LDAP, two additional fields appear: Certificate File and Certificate Password.
   Host	The fully qualified domain name (FQDN) or IP address of the Active Directory server.
   Port	The port number used to connect to the Active Directory server. The default port is usually 389 for LDAP or 636 for LDAPS (secure LDAP).
   Domain	The domain name associated with the Active Directory environment.
   User Name	The username that has permission to query the Active Directory. Include the domain if you have not provided the domain name.
   Password	The password associated with the username. This credential is used to authenticate the connection to the AD server.
   Certificate File (Visible only when Configuration Method is Secure LDAP)	Upload the SSL certificate file (in .crt or .pem format) required to establish a secure connection.
   Certificate Password (Visible only when Configuration Method is Secure LDAP)	Specify the password associated with the uploaded certificate file, if applicable.

6. Click Test Connection to test your connection to the specified data source.
7. Click Create Data Source to establish your data source connection.
8. Click Import Users.

   This process imports file system security identifiers (SIDs), GUIDs, and security descriptors, and maps them to user identities, which helps to display ownership and group information for files and folders from SMB and CIFS data sources in Data Canvas. You can also monitor the status of the job on the Workers page

You have successfully created a connection to Active Directory as a data source in Data Catalog.

After completing the Import Users job, you can run the Metadata Ingest process for SMB and CIFS data sources to see the user information in the Properties panel. For more information, see the Processing unstructured data topic in the Explore your data section in the Use Pentaho Data Catalog document. Additionally, you can see the list of users who have access to a particular file or folder. For more information, see the Users with Access topic in the Use Pentaho Data Catalog document.

Amazon Redshift data source

Amazon Redshift is a fully managed, petabyte-scale data warehouse solution offered by Amazon Web Services (AWS). It allows users to run complex queries and perform real-time data analytics on large datasets by utilizing massively parallel processing (MPP) technology. Integrating Amazon Redshift as a data source within Data Catalog, you can access and manage metadata from the Amazon Redshift database. It enables data discovery to search, explore, and understand Amazon Redshift data. Additionally, it enhances data lineage and compliance by providing detailed tracking of data movements.

Perform the following steps to add Amazon Redshift as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the connection information, select Redshift in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Configuration Method	Select Credentials, or URI as a configuration method.
   Driver	If you select the configuration method as Credentials or URI, then you must use the driver. Select an existing driver or upload a new driver to ensure efficient, secure communication between the application and the database that meets the required standards. To upload a new driver, click Manage Drivers, and click Add New, upload the driver, and then click Add Driver.
   Driver Class Name	The fully qualified Java class name of the JDBC driver.
   User Name	The username required to authenticate to the Amazon Redshift database. This field is optional when using Secret Manager Key.
   Password	The password associated with the specified Redshift user account. This field is optional when using Secret Manager Key.
   Secret Manager Key	The name or identifier of the secret stored in Secret Manager that contains the Redshift credentials (username and password). When provided, Data Catalog retrieves the credentials securely from Secret Manager instead of requiring manual entry.
   Role	The IAM role that grants Data Catalog permission to read secret versions from Secret Manager. This role must include permissions such as secretmanager.versions.access.
   Region	The region where the Redshift secret is stored in Secret Manager. Data Catalog uses this region to access the correct Secret Manager endpoint.
   Host (Only for the Credentials method)	The address of the machine where the Amazon Redshift database server is running. It can be an IP address or a domain name.
   Port (Only for the Credentials method)	The port number on which the Amazon Redshift server is listening for incoming connections.
   URI (Only for the URI method)	URIs are used to access and manage various objects and services within the Amazon Redshift environment. For example, URL would look like `jdbc:redshift://:/`.
   Database Name	The name of the database within the Amazon Redshift server that you want to connect with.

6. Click Test Connection to test your connection to the specified data source.
7. Click Ingest Schema. The Select schemas for ingestion dialog opens. You can search for schemas using the search bar at the top, using starts with or regular expressions. Once you have located the relevant schemas, select them, then click Ingest to load the database schema and related metadata.

   Although you can select all schemas, it is a best practice to avoid selecting certain system-related schemas that are unnecessary for your needs.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   Note: To use storage optimization options, you need a Pentaho Data Optimizer license.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.
   Available for Data Mastering	Enables or disables the data source for data mastering purposes.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.

You have successfully created the connection to the Amazon Redshift as a data source.

Apache Iceberg data source

Apache Iceberg is an open table format designed to manage large analytic datasets in modern data lake architectures. It provides a standardized way to define table metadata, track schema and partition changes, and manage transactional updates independently of the underlying storage systems and query engines.

In Data Catalog, you can configure Apache Iceberg as a data source to discover, catalog, and govern Iceberg-managed tables. This integration allows Data Catalog to connect directly to an Iceberg catalog and ingest table metadata based on Iceberg’s native metadata model, rather than relying on file system structures or inferred schemas.

The Apache Iceberg connector in Pentaho Data Catalog supports Iceberg warehouses deployed on AWS, HDFS, and Azure Data Lake. Perform the following steps to add Apache Iceberg as a data source in Data Catalog:

▶️ Watch a walkthrough

You can watch a guided walkthrough that demonstrates how to configure an Apache Iceberg data source in Pentaho Data Catalog.

Apache Iceberg Data Sourceassets.demos.hitachivantara.com

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Apache Iceberg in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   URI	URL to the Apache Iceberg catalog service. For example, URL would look like http://<ip_address_or_hostname_of_iceberg_catalog:8181
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Warehouse	This is where data and metadata are stored. Currently, Data Catalog supports AWS, Azure Data Lake, and HDFS.

   Based on the Warehouse selected, provide the additional connection details:

AWS

Field	Description
Region	The AWS region where the S3 bucket hosting the Iceberg warehouse is located.
Endpoint	The S3-compatible endpoint used to access the Iceberg warehouse. Example: http://<ip_address_or_hostname_of_warehouse_location:9000
Access Key	User credentials to access data in Iceberg Catalog. This key authenticates your access to the S3 bucket.
Secret Access Key	Password credential to access data on Iceberg Catalog. This key authenticates your access to the S3 bucket.
Path Style Access	True when the user is accessing resources via a path-style access. For example, /user/home/resource . If using an alternate access style, then False.

HDFS

Azure Data Lake

1. Click Test Connection to test your connection to the specified data source. A Test Connection confirmation message window opens.
2. Click Ingest Schema. The Select schemas for ingestion dialog opens.

   1. You can search and select schemas using the search bar at the top (starts with or using regular expressions), then click Next.
   2. On the Ingest Schema dialog box, add include or exclude patterns to filter the tables to be ingested, and then click Ingest to load the database schema and related metadata. Note: Although you can select all schemas, it is a best practice to avoid selecting specific system-related schemas that are unnecessary for your needs.
3. (Optional) In the Physical Location field, specify the physical location details of the data source.
4. (Optional) Configure the following storage optimization options for the data source.

   To use storage optimization options, you need a Pentaho Data Optimizer license.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.
   Available for Data Mastering	Enables or disables the data source for data mastering purposes.

5. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
6. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
7. (Optional) Enter a Note for any additional information to share with others who might access this data source.
8. Click Create Data Source to establish your data source connection.

You have successfully created a connection to Apache Iceberg as a data source.

AWS S3 data source

Perform the following steps to add AWS S3 as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select AWS S3 in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Region1	The geographical location where AWS maintains a cluster of data centers. If connecting directly to S3, enter the region where the S3 bucket resides. If using Secrets Manager, enter the region where the Secrets Manager key resides.
   Bucket Name1	Enter the name of the Amazon S3 bucket where your data resides. For S3 access from non-EMR environments, Data Catalog uses the AWS Command Line Interface (CLI) to connect to S3 using your access credentials (Access Key and Secret Access Key). If you are connecting to Hadoop Distributed File System (HDFS) or MapR, provide the logical root directory name defined in the hdfssite.xml configuration file using the dfs.nameservices property. For MapR file systems, identify the root using the maprfs:/// URI scheme.
   Access Key3	User credentials to access data on the bucket. If connecting directly to S3, this key authenticates your access to the S3 bucket. If using Secrets Manager, this key authenticates your access to the Secrets Manager key so you can retrieve the S3 bucket credentials.
   Secret Access Key3	Password credential to access data on the bucket. If connecting directly to S3, this key authenticates your access to the S3 bucket. If using Secrets Manager, this key authenticates your access to the Secrets Manager key so you can retrieve the S3 bucket credentials.
   Secret Manager Key2	Name of the AWS Secrets Manager key to retrieve the credentials securely, instead of specifying them here.
   AWS role	ARN of the AWS IAM Role to assume when accessing the S3 bucket, allowing cross-account or role-based access.
   Endpoint	Location of the bucket. For example, s3.<region containing S3 bucket>.amazonaws.com
   Path	The directory where this data source is included.

   ¹ These fields are mandatory. ² When using Secrets Manager, you must provide:  • the Region where the Secrets Manager key resides  • the Secret Manager Key  • and the Access Key and Secret Access Key (to retrieve the S3 bucket credentials from Secrets Manager). In this case, the access keys authenticate your access to Secrets Manager, not directly to the S3 bucket. ³ When not using Secrets Manager, you must provide the Access Key and Secret Access Key to access the S3 bucket directly.
6. (Optional) Select Skip SSL validation if the data source uses a self-signed SSL certificate and you do not want to import the certificate into Data Catalog. This option bypasses SSL certificate validation for the data source. Although supported in any environment, it is not recommended for production use. To enable SSL validation, import the self-signed certificate into Data Catalog. For more information, see Install user-provided SSL certificates.
7. Click Test Connection to test your connection to the specified data source.
8. Click Scan Files. The Scan Files dialog box appears. Here you can refine metadata ingestion using the following options.

   1. Delete Empty Folders: When selected, it deletes folders without any child entities from the Data Catalog metadata store. This option is intended only for full metadata scans.
   2. Incremental Ingest: Ingests data for filesystems or object stores where files or objects created or modified time stamps match the time period selected. This doesn't remove any existing data from previous ingests and can be used when you only want to ingest data from a particular timeframe. Available time periods are (Last) 12 Hours, 24 Hours, 1 Week, 1 Month, and 3 Months.

      You cannot select Delete Empty Folders and Incremental Ingest at the same time. The Delete Empty Folders option applies only to full metadata scans and is not supported for incremental ingest operations.
   3. Include and Exclude Patterns: This feature enables you to specify which folders or files should be scanned or excluded. By applying these filters, you can shorten scan duration and control the scope of metadata ingestion. For more information, see the feature walkthrough Include or exclude patterns.

   This process loads files and folders into the system. You can monitor the status of the file scan on the Workers page.

   If you are nearing or have exceeded the data scanning limit specified in your license agreement, a message will appear in the upper corner of the screen.
9. (Optional) In the Physical Location field, specify the physical location details of the data source.
10. (Optional) Configure the following storage optimization options for the data source.

    Note: To use storage optimization options, you need a Pentaho Data Optimizer license.

    Field	Description
    Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
    Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.

11. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
12. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
13. (Optional) Enter a Note for any additional information to share with others who might access this data source.
14. Click Create Data Source to establish your data source connection.

You have successfully created a connection to the AWS S3 data source.

Feature walkthrough: Support for Secret Manager

▶️ Launch the Support for Secret Manager walkthrough

This interactive walkthrough demonstrates how to configure Secret Manager in Pentaho Data Catalog. It covers:

• Storing and managing sensitive credentials in Secret Manager
• Configuring PDC to securely fetch these credentials for AWS S3 and other data sources
• Reducing manual handling of secrets during data source setup

Azure Blob Storage data source

Perform the following steps to add Azure Blob Storage as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Azure Blob Storage in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	The Default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Configuration Method	Shared Key (default)
   Account Name	Name of an Azure storage account contains all of your Azure Storage data objects.
   Shared Key	A password-like credential that gives full access to an Azure storage account's data and configuration.
   Container	The top-level object that logically groups blob data that holds an unlimited number of large object data.
   Path	Folder where this data source is included.

6. Click Test Connection to test your connection to the specified data source.
7. Click Scan Files. The Scan Files dialog box appears. Here you can refine metadata ingestion using the following options.

   1. Delete Empty Folders: When selected, it deletes folders without any child entities from the Data Catalog metadata store. This option is intended only for full metadata scans.
   2. Incremental Ingest: Ingests data for filesystems or object stores where files or objects created or modified time stamps match the time period selected. This doesn't remove any existing data from previous ingests and can be used when you only want to ingest data from a particular timeframe. Available time periods are (Last) 12 Hours, 24 Hours, 1 Week, 1 Month, and 3 Months.

      You cannot select Delete Empty Folders and Incremental Ingest at the same time. The Delete Empty Folders option applies only to full metadata scans and is not supported for incremental ingest operations.
   3. Include and Exclude Patterns: This feature enables you to specify which folders or files should be scanned or excluded. By applying these filters, you can shorten scan duration and control the scope of metadata ingestion. For more information, see the feature walkthrough Include or exclude patterns.

   This process loads files and folders into the system. You can monitor the status of the file scan on the Workers page.

   If you are nearing or have exceeded the data scanning limit specified in your license agreement, a message will appear in the upper corner of the screen.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   Note: To use storage optimization options, you need a Pentaho Data Optimizer license.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.
14. Specify the following additional connection information.

    Field	Description
    Affinity	The Default setting specifies which agents should be associated with the data source in a multi-agent deployment.
    Configuration Method	Shared Key (default)
    Account Name	Name of an Azure storage account contains all of your Azure Storage data objects.
    Shared Key	A password-like credential that gives full access to an Azure storage account's data and configuration.
    Container	The top-level object that logically groups blob data that holds an unlimited number of large object data.
    Path	Folder where this data source is included.

15. Click Test Connection to test your connection to the specified data source.

You have successfully created the connection to the Azure Blob Storage data source.

Databricks data source

You can configure Databricks as a data source in Data Catalog to discover, catalog, and govern data stored in Databricks SQL warehouses and the underlying Delta Lake storage layer. This integration enables Pentaho Data Catalog to connect to Databricks using JDBC and ingest metadata for data identification, profiling, and governance. By adding Databricks as a data source, organizations gain centralized visibility into Databricks-managed data and can support governed analytics, machine learning, and AI workflows across structured, semi-structured, unstructured, and streaming data.

Perform the following steps to add Databricks as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

▶️ Watch a walkthrough

You can watch a guided walkthrough that demonstrates how to configure a Databricks data source in Pentaho Data Catalog.

Databricks Catalog Data Sourceassets.demos.hitachivantara.com

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Databricks in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Configuration method	The method is used to configure the connection. The only option is Credentials.
   Driver	Select an existing driver or upload a new driver to ensure that the communication between the application and the database is efficient, secure, and follows the required standards.To upload a new driver, click Manage Drivers, and click Add New, upload the driver, and then click Add Driver.
   Driver Class Name	The Java class name corresponding to the selected Databricks driver.
   Host	The Databricks workspace hostname. This value identifies the Databricks compute resource. Example: https://dbc-xxxx.cloud.databricks.com.
   Port	The port used to connect to the Databricks workspace.
   Path	The HTTP path of the Databricks SQL warehouse. Pentaho Data Catalog appends this value to the host and port to form the JDBC connection URL. Example: /sql/1.0/warehouses/<warehouse-id>.
   Catalog name	A user-defined name to identify the Databricks data source within Pentaho Data Catalog.
   Personal access token	A Databricks personal access token (PAT) used to authenticate the connection. The token must have permission to access the specified SQL warehouse.

6. Click Test Connection to test your connection to the specified data source. A Test Connection confirmation message window opens.
7. Click Ingest Schema. The Select schemas for ingestion dialog opens.

   1. You can search and select schemas using the search bar at the top (starts with or using regular expressions), then click Next.
   2. On the Ingest Schema dialog box, add include or exclude patterns to filter the tables to be ingested, and then click Ingest to load the database schema and related metadata.

      Although you can select all schemas, it is a best practice to avoid selecting specific system-related schemas that are unnecessary for your needs.

8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   Note: To utilize storage optimization options, a Pentaho Data Optimizer license is required.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.
   Available for Data Mastering	Enables or disables the data source for data mastering purposes.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.

You have successfully created a connection to the Databricks data source.

Denodo data source

Perform the following steps to add Denodo as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Denodo in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Configuration method	By default, it is a URI, as the connection is configured using a URL.
   Driver	Select an existing driver or upload a new driver to ensure that the communication between the application and the database is efficient, secure, and follows the required standards.To upload a new driver, click Manage Drivers, and click Add New, upload the driver, and then click Add Driver.
   Driver Class Name	The Java class name corresponding to the selected Denodo driver.
   User Name	The username required to authenticate to the Denodo environment. This field is optional when using Secret Manager Key.
   Password	The password associated with the Denodo user account. This field is optional when using Secret Manager Key.
   Secret Manager Key	The name or identifier of the secret stored in Secret Manager that contains the Denodo connection credentials. When this field is provided, Data Catalog securely retrieves credentials from Secret Manager, eliminating the need to enter the username and password manually.
   Region	The region where the Denodo secret is stored in Secret Manager. Data Catalog uses this region to connect to the correct Secret Manager endpoint.
   Role	The IAM role that grants Data Catalog permission to read secret versions from Secret Manager. The role must include permissions such as secretmanager.versions.access.
   URI	URIs are used to access and manage various objects and services within the Denodo environment. For example, the URI would look like jdbc:vdb://<denodo-host>:<port>/<database-name>?publishCatalogsAsSchemas=true Example: jdbc:vdb://ec2-1-2-3-4.compute-1.amazonaws.com:49999/mydatabase?publishCatalogsAsSchemas=true
   Database Name	The name of the data sources within the Denodo environment that contain the data you want to access.

6. Click Test Connection to test your connection to the specified data source. A Test Connection confirmation message window opens.

   Before finalizing and saving your new data source configuration, you must perform a process called 'Scan files'. If you are nearing or have exceeded the data scanning limit set by your license agreement, a message appears in the upper corner of the screen. Databases do not have a data scan quota.
7. Click Ingest Schema. The Select schemas for ingestion dialog opens. You can search for schemas in the search bar at the top, using starts-with or regular expressions. Once you have located the relevant schemas, select the schemas and then click Ingest to load the database schema and related metadata.

   Although you can select all schemas, it is a best practice to avoid selecting specific system-related schemas that are unnecessary for your needs.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   Note: To utilize storage optimization options, a Pentaho Data Optimizer license is required.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.
   Available for Data Mastering	Enables or disables the data source for data mastering purposes.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.

You have successfully created a connection to the Denodo data source.

DynamoDB data source

Perform the following steps to add Amazon DynamoDB as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Active Directory in the Data Source Type field.

   Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Region	Geographical location where AWS maintains a cluster of data centers.
   Access Key and Secret Access Key	AWS Access Key ID and Secret Access Key that are used for authentication and authorization when interacting with DynamoDB.
   Role	The AWS IAM Role that the Pentaho Data Catalog assumes to access DynamoDB securely. Use this field when authentication is performed through AWS IAM Role-based access instead of (or in addition to) Access Key credentials.

6. Click Test Connection to test your connection to the specified data source.jkljk
7. Click Scan Files. The Scan Files dialog box appears. Here you can refine metadata ingestion using the following options.

   1. Delete Empty Folders: When selected, it deletes folders without any child entities from the Data Catalog metadata store. This option is intended only for full metadata scans.
   2. Incremental Ingest: Ingests data for filesystems or object stores where files or objects created or modified time stamps match the time period selected. This doesn't remove any existing data from previous ingests and can be used when you only want to ingest data from a particular timeframe. Available time periods are (Last) 12 Hours, 24 Hours, 1 Week, 1 Month, and 3 Months.

      You cannot select Delete Empty Folders and Incremental Ingest at the same time. The Delete Empty Folders option applies only to full metadata scans and is not supported for incremental ingest operations.
   3. Include and Exclude Patterns: This feature enables you to specify which folders or files should be scanned or excluded. By applying these filters, you can shorten scan duration and control the scope of metadata ingestion. For more information, see the feature walkthrough Include or exclude patterns.

   This process loads files and folders into the system. You can monitor the status of the file scan on the Workers page.

   If you are nearing or have exceeded the data scanning limit specified in your license agreement, a message will appear in the upper corner of the screen.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   To use storage optimization options, you need a Pentaho Data Optimizer license.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.

You have successfully created the connection to the Amazon DynamoDB data source.

Google Cloud Storage data source

Google Cloud Storage (GCS) is a storage service that enables the storage, retrieval, and management of unstructured data, including files, images, videos, and large datasets. By integrating GCS as a data source within Data Catalog, you can access and manage the metadata of files stored. You can perform data discovery to search, explore, and understand your Google Cloud Storage data. Additionally, it enhances data lineage and compliance by providing detailed tracking of data movements.

Perform the following steps to add Google Cloud Storage as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Google Object Store in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Bucket Name	The name of the Google Cloud Storage bucket in which the data resides.
   Path	The path within the bucket where the files are stored.
   Key Path	The authentication key file, which is used to connect to Google Cloud Storage (a JSON file).

6. Click Test Connection to test your connection to the specified data source.
7. Click Scan Files. The Scan Files dialog box appears. Here you can refine metadata ingestion using the following options.

   1. Delete Empty Folders: When selected, it deletes folders without any child entities from the Data Catalog metadata store. This option is intended only for full metadata scans.
   2. Incremental Ingest: Ingests data for filesystems or object stores where files or objects created or modified time stamps match the time period selected. This doesn't remove any existing data from previous ingests and can be used when you only want to ingest data from a particular timeframe. Available time periods are (Last) 12 Hours, 24 Hours, 1 Week, 1 Month, and 3 Months.

      You cannot select Delete Empty Folders and Incremental Ingest at the same time. The Delete Empty Folders option applies only to full metadata scans and is not supported for incremental ingest operations.
   3. Include and Exclude Patterns: This feature enables you to specify which folders or files should be scanned or excluded. By applying these filters, you can shorten scan duration and control the scope of metadata ingestion. For more information, see the feature walkthrough Include or exclude patterns.

   This process loads files and folders into the system. You can monitor the status of the file scan on the Workers page.

   If you are nearing or have exceeded the data scanning limit specified in your license agreement, a message will appear in the upper corner of the screen.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   To use storage optimization options, you need a Pentaho Data Optimizer license.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.

You have successfully created a connection to Google Cloud Storage as a data source in Data Catalog.

Google BigQuery data source

Google BigQuery is a serverless, highly scalable, and cost-effective multi-cloud data warehouse designed for analytics. Integrating BigQuery as a data source within Data Catalog, you can access and manage metadata from the BigQuery database. It enables data discovery to search, explore, and understand BigQuery data. Additionally, it enhances data lineage and compliance by providing detailed tracking of data movements.

Perform the following steps to add BigQuery as a data source in Data Catalog:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.
   Data Source ID (Optional)	Specify a permanent identifier for your data source. CAUTION: If this field is left blank, Data Catalog generates a permanent identifier, which cannot be modified.
   Description (Optional)	Specify a description of your data source.

4. After you have specified the basic connection information, select Google BigQuery in the Data Source Type field. Data Catalog then prompts you to specify additional connection information based on the file system or database type you are trying to access.
5. Specify the following additional connection information.

   Field	Description
   Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
   Driver	The standard used to establish communication between the application and the database.Select an existing driver or upload a new driver to ensure that the communication between the application and the database is efficient, secure, and follows the required standards. To upload a new driver, click Manage Drivers, and click Add New, upload the driver, and then click Add Driver.
   Host	The Google BigQuery API endpoint. By default, the host is set to https://www.googleapis.com/bigquery/v2, which communicates with BigQuery's REST API for data processing.
   Port	The port number to connect to the BigQuery data source. The default port for HTTPS connections is 443.
   Project	The Google Cloud project ID that contains the BigQuery datasets you want to access.
   Database Name	The name of the dataset within the BigQuery that you want to connect with.
   Secret Manager Key	The name of the secret stored in Secret Manager that contains your Google Cloud service account key (in JSON format). When provided, Data Catalog retrieves the key securely from Secret Manager instead of using a local key file.
   Role	The IAM role that grants access to read secret versions from Secret Manager. This role must include permissions such as secretmanager.versions.access so that Data Catalog can retrieve the service account key.
   Region	The region where the secret is stored in Secret Manager. Data Catalog uses this region to connect to the correct Secret Manager endpoint.
   Key Path	The file path to your Google Cloud service account's key (a JSON file).
   Oauth Type	The authentication method to connect to BigQuery. By default, it is Service-based. It uses a service account and a key file for authentication.
   Client Email	The email associated with your Google Cloud service account for service-based OAuth. The service account email is usually in the format [e-mail].

6. Click Test Connection to test your connection to the specified data source. A Test Connection confirmation message window opens.

   Before you finalize and save your new data source configuration, you need to perform a process called Scan files. If you are nearing or have exceeded the data scanning limit set by your license agreement, a message appears in the upper corner of the screen. Databases do not have a data scan quota.
7. Click Ingest Schema. The Select schemas for ingestion dialog opens. You can search for schemas using the search bar at the top, using starts with or regular expressions. Once you have located the relevant schemas, select the schemas and then click Ingest to load the database schema and related metadata.

   Although you can select all schemas, it is a best practice to avoid selecting certain system-related schemas that are unnecessary for your needs.
8. (Optional) In the Physical Location field, specify the physical location details of the data source.
9. (Optional) Configure the following storage optimization options for the data source.

   To utilize storage optimization options, a Pentaho Data Optimizer license is required.

   Field	Description
   Available for Migration	Enables or disables the data source for storage optimization. When enabled, it includes the data source for data optimizer activities.
   Available for Writing	Enables or disables writing capabilities for the data source and enables migration when turned on.
   Available for Data Mastering	Enables or disables the data source for data mastering purposes.

10. (Optional) In the Cost per Terabyte field, specify the data source pricing details like currency, price per terabyte, and billing frequency.
11. (Optional) In the Total Capacity field, specify the total capacity of the data source in terabytes.
12. (Optional) Enter a Note for any additional information to share with others who might access this data source.
13. Click Create Data Source to establish your data source connection.
14. Specify the following additional connection information.

    Field	Description
    Affinity	This default setting specifies which agents should be associated with the data source in a multi-agent deployment.
    Driver	The standard used to establish communication between the application and the database.Select an existing driver or upload a new driver to ensure that the communication between the application and the database is efficient, secure, and follows the required standards. To upload a new driver, click Manage Drivers, and click Add New, upload the driver, and then click Add Driver.
    Host	The Google BigQuery API endpoint. By default, the host is set to https://www.googleapis.com/bigquery/v2, which communicates with BigQuery's REST API for data processing.
    Port	The port number to connect to the BigQuery data source. The default port for HTTPS connections is 443.
    Project	The Google Cloud project ID that contains the BigQuery datasets you want to access.
    Database Name	The name of the dataset within the BigQuery that you want to connect with.
    Secret Manager Key	The name of the secret stored in Secret Manager that contains your Google Cloud service account key (in JSON format). When provided, Data Catalog retrieves the key securely from Secret Manager instead of using a local key file.
    Role	The IAM role that grants access to read secret versions from Secret Manager. This role must include permissions such as secretmanager.versions.access so that Data Catalog can retrieve the service account key.
    Region	The region where the secret is stored in Secret Manager. Data Catalog uses this region to connect to the correct Secret Manager endpoint.
    Key Path	The file path to your Google Cloud service account's key (a JSON file).
    Oauth Type	The authentication method to connect to BigQuery. By default, it is Service-based. It uses a service account and a key file for authentication.
    Client Email	The email associated with your Google Cloud service account for service-based OAuth. The service account email is usually in the format [e-mail].

15. Click Test Connection to test your connection to the specified data source. A Test Connection confirmation message window opens. Note: Before you finalize and save your new data source configuration, you need to perform a process called Scan files. If you are nearing or have exceeded the data scanning limit set by your license agreement, a message appears in the upper corner of the screen. Databases do not have a data scan quota.
16. Click Ingest Schema. The Select schemas for ingestion dialog opens. You can search for schemas using the search bar at the top, using starts with or regular expressions. Once you have located the relevant schemas, select the schemas, and then click Ingest to load the database schema and related metadata. Note: Although you can select all schemas, it is a best practice to avoid selecting certain system-related schemas that are unnecessary for your needs.
17. (Optional) In the Physical Location field, specify the physical location details of the data source.

You have successfully created a connection to the BigQuery data source.

HCP data source

You can add data to Data Catalog from Hitachi Content Platform (HCP) by adding HCP as data source. Perform the following steps to add HCP as data source:

Refer to the Component Reference section in the Install Pentaho Data Catalog document to confirm that you have met all the necessary requirements listed for the data source you want to connect.

1. Click Management in the left navigation menu.

   The Manage Your Environment page opens.
2. In the Resources card, click Add Data Source.

   The Create Data Source page opens.

   If you are nearing or have exceeded the limit of data sources allowed by your license agreement, a message appears when you try to add a new data source.
3. Specify the following information for the connection to your data source.

   Data Catalog encrypts your data source connection details, such as user name and password, before storing them.

   Field	Description
   Data Source Name	Specify the name of your data source. This name is used in the Data Catalog interface. It should be something your Data Catalog users recognize. Note: Names must start with a letter, and must contain only letters, digits, and underscores. Spaces in names are not supported.