Ctrlk
Try Pentaho! - Start your 30 day evaluation today
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Advanced configuration

After installing Data Catalog, you may need to set up additional components, depending on your environment. Use the following topics as needed to finish setting up your environment.

Configure system environment variables

Although not common, there might be instances where you need to change the default settings for Data Catalog system environment variables. These configuration modifications allow you to override default system behavior to align with the specific needs.

Modifying these settings can have system-wide implications, and incorrect changes might negatively impact the functionality of the other platforms. It is a best practice to collaborate with your Pentaho Data Catalog partner to ensure that any modifications align with the intended objectives.

  1. In a terminal window, navigate to the pdc-docker-deployment folder and open the hidden environment variable configuration file (.env). This file is located in the /opt folder by default.

  2. Verify the system environment variables set in the /opt/pentaho/pdc-docker-deployment/vendor/.env.default file:

    For example, the number of worker instances that Data Catalog uses to run processes is set to 5:

    PDC_WS_DEFAULT_OPS_JOBPOOLMINSIZE=5
PDC_WS_DEFAULT_OPS_JOBPOOLMAXSIZE=5

    Make sure that PDC_WS_DEFAULT_OPS_JOBPOOLMINSIZE and PDC_WS_DEFAULT_OPS_JOBPOOLMAXSIZE have the same value for consistent worker instance management.

  3. To override an environment variable set in the vendor/.env.default file, you can create a new .env file in the opt/pentaho/pdc-docker-deployment/conf/ folder:

    vi opt/pentaho/pdc-docker-deployment/conf/.env

  4. (Optional) The data in the Business Intelligence Database refreshes daily by default, as set in the .env file. To modify the data refresh frequency, update the variable in the .env file to one of the options listed in the following table:

    Value
    Description

    @yearly (or @annually)

    Run once a year, midnight, Jan 1st

    @monthly

    Run once a month, midnight, first of the month

    @weekly

    Run once a week, midnight between Sat/Sun

    @daily (or @midnight)

    Run once a day, midnight

    @hourly

    Run once an hour, the beginning of the hour

    @every <number>m

    Run at a custom interval, where <number> specifies the number of minutes. For example, @every 5m runs the job every 5 minutes.

    Example:

    PDC_CRON_BI_VIEWS_INIT_SCHEDULE=@daily

  5. After adding all required system variables, save the changes and restart the Data Catalog system services.

    ./pdc.sh stop
./pdc.sh up

Configure chatbot in Data Catalog

The chatbot in Pentaho Data Catalog enables users to interact with cataloged metadata using natural language queries. The chatbot supports two response modes:

  • Standard (PDO-based) responses, which retrieve structured metadata from the Business Intelligence Database (BIDB).

  • Conversational responses, which use vector embeddings stored in Qdrant and a configured large language model (LLM) to provide contextual, semantic answers.

Before users can access chatbot capabilities, administrators must configure and validate several backend components. These configurations include enabling the chatbot service, configuring the large language model, populating indexing data stores, and defining role-based access for conversational search.

This section describes how to configure, enable, disable, and manage the chatbot feature in both Docker and Kubernetes (Amazon EKS) deployments.

Initial setup for chatbot

Before users can interact with the chatbot in Data Catalog, administrators must complete several mandatory configuration steps. The chatbot depends on a large language model (LLM) configuration, metadata indexing, vector embedding generation, and role-based access control to function correctly.

Perform the following steps to complete the initial setup for the chatbot in Data Catalog:

Prerequisites

  • Pentaho Data Catalog is installed and running.

  • OpenSearch is deployed and accessible.

  • The chatbot feature is supported in the selected deployment profile.

  • You have administrative access to the deployment environment.

  • You have valid LLM provider details, including API key, model name, and embedding model.

Procedure

  1. Enable the chatbot frontend and backend services in your deployment.

    • For Docker deployments, configure the chatbot services in the Docker profile and restart the application.

    • For Kubernetes deployments, enable chatbot-frontend and chatbot-backend in the active profile YAML file and apply the Helm upgrade.

    For detailed steps, see Enable chatbot in Data Catalog.

  2. Configure the required LLM and embedding settings:

    • API key

    • LLM model

    • Embedding model

    • Base URL (if applicable)

    • Target vector dimensions

    • Vector indexing schedule

    Ensure that the embedding model and target vector dimensions match. Important: You cannot change the embedding model after the initial setup without rebuilding the vector index. For detailed steps, see Configure a large language model (LLM) for the chatbot in Data Catalog.

  3. Configure conversational roles to define which user roles are allowed to receive conversational chatbot responses. For detailed steps, see Configure Data Catalog user roles for conversational search chatbot.

    Users without the configured roles receive standard (PDO-based) chatbot responses.

  4. For the chatbot to return responses, the required data stores must be populated. Verify data population and indexing:

    1. For standard (PDO) queries, the Business Intelligence Database (BIDB) must contain indexed metadata.

    2. For conversational queries, the Qdrant vector database must contain generated embeddings.

    Both BIDB and Qdrant indexing processes run as scheduled jobs. By default, these jobs run once per day at midnight. The schedule is configurable. If you encounter any issues, see Chatbot vector indexing issues.

  5. Once the configuration is completed, sign in to Data Catalog and verify that:

    • The chatbot icon is visible in the user interface.

    • The chatbot accepts and processes queries.

    • Users with configured conversational roles receive contextual responses.

    • Users without conversational roles receive standard responses.

    • Review backend logs if responses are missing or incomplete.

Result

The chatbot is fully configured and operational.

  • Standard (PDO) queries return results from BIDB.

  • Conversational queries return semantic responses from Qdrant embeddings.

  • Indexing runs on schedule and updates incrementally.

The chatbot is now ready for production use.

What next

  • If responses are incomplete or missing, review the Chatbot vector indexing issues.

  • Monitor indexing logs during initial production deployment.

  • Periodically verify indexing schedules and role assignments.

Enable chatbot in Data Catalog

In Pentaho Data Catalog, the chatbot enables users to interact with cataloged data via natural language, retrieve insights, explore business glossary terms, and access related dashboards directly within the Data Catalog interface. This procedure explains how to enable the chatbot feature in Data Catalog. After you enable the chatbot, the chatbot icon appears in the user interface, and users can start conversational data discovery.

Prerequisites

  • Data Catalog is deployed using Docker Compose or Kubernetes.

  • You have administrative access to the deployment environment, such as Docker or Amazon EKS.

  • Ensure that any one large language model (LLM) is configured for chatbot in Data Catalog. For more information, see Configure a large language model (LLM) for the chatbot in Data Catalog.

Docker deployment

Perform the following steps to enable the chatbot in the Docker deployment:

Procedure

  1. Open a terminal session on the server where Data Catalog is deployed using Docker Compose.

  2. Change to the Data Catalog Docker deployment directory.

  3. Stop all running Data Catalog services.

  4. Open the Docker Compose configuration file.

  5. Change the profiles value from the disabled profile name (for example, core1) back to core for the chatbot backend and frontend services.

    The profiles value should match with the profile name defined for COMPOSE_PROFILES parameter in vendor/env.default (for example, core).

  6. Save the file and exit the editor.

  7. Start the Data Catalog services.

  8. Open the Data Catalog UI and confirm that the chatbot icon is visible.

Result

The chatbot frontend and backend services are enabled, and the chatbot icon appears in the Data Catalog interface. Users can now use the chatbot to explore catalog data using natural language.

What next

You can disable the chatbot if no longer required by reversing the configuration changes. For more information, see Disable chatbot in Data Catalog.

Disable chatbot in Data Catalog

In Pentaho Data Catalog, the chatbot enables users to interact with cataloged data via natural language, retrieve insights, explore business glossary terms, and access related dashboards directly within the Data Catalog interface. Sometimes, you might disable the chatbot when the feature is not required. This procedure explains how to disable the chatbot in Data Catalog.

Disabling the chatbot stops the chatbot frontend and backend services and removes the chatbot icon from the user interface.

Prerequisites

  • Data Catalog is deployed using Docker Compose or Kubernetes.

  • You have administrative access to the deployment environment.

  • You can open a terminal session on the deployment server or management system.

Docker deployment

Perform the following steps to disable the chatbot in the Docker deployment:

Procedure

  1. Open a terminal session on the server where Data Catalog is deployed using Docker Compose.

  2. Change to the Data Catalog Docker deployment directory.

  3. Stop all running Data Catalog services.

  4. Open the Docker Compose configuration file.

  5. Change the profiles value from core to an unused value (for example, core1) for the chatbot backend and frontend services.

    The profiles value used to disable the chatbot must not match any active profile defined for COMPOSE_PROFILES parameter in vendor/env.default (for example, core). If an admin later updates the profile's value, a name that is listed in vendor/env.default , the chatbot frontend and backend services start automatically.

  6. Save the file and exit the editor.

  7. Start the Data Catalog services.

  8. Open the Data Catalog application and confirm that the chatbot icon is no longer visible.

Result

The chatbot frontend and backend services are disabled, and the chatbot icon is removed from the Data Catalog interface. Users can no longer access or use the chatbot feature. Additionally, you can verify that no chatbot services are running by checking containers or pods.

What next

You can re-enable the chatbot if required by reversing the configuration changes. For more information, see Enable chatbot in Data Catalog.

Configure a large language model (LLM) for the chatbot in Data Catalog

In Data Catalog, the chatbot requires a configured large language model (LLM) to generate responses, create vector embeddings, and support conversational search for authorized users. This procedure explains how, as an admin, you can configure a LLM for the chatbot feature in Data Catalog. After you configure the LLM, the chatbot backend can process user queries, generate embeddings, and return contextual responses based on catalog metadata and permissions.

Prerequisites

  • The chatbot feature is enabled in Pentaho Data Catalog.

  • You have valid LLM provider details, including API key, model name, and embedding model.

Docker deployment

Perform the following steps to configure a LLM for the chatbot feature in Data Catalog Docker deployment:

Procedure

  1. Open a terminal session on the server where Data Catalog is deployed using Docker Compose.

  2. Change to the Pentaho Data Catalog Docker deployment directory.

  3. Stop all running Data Catalog services.

  4. Open the environment configuration file.

  5. Configure the chatbot LLM and embedding settings.

    Environment variable
    Description

    PDC_CHATBOT_API_KEY

    API key used to authenticate with the configured LLM provider. The chatbot backend does not start if this value is missing or invalid.

    PDC_CHATBOT_LLM

    Name of the LLM used to generate chatbot responses.

    PDC_CHATBOT_LLM_BASE_URL

    Base URL of the LLM provider endpoint. Use this value for self-hosted or non-default LLM providers.

    PDC_CHATBOT_EMBEDDING_MODEL

    Embedding model used to generate vector representations for conversational search. You cannot change this value after the initial setup.

    PDC_CHATBOT_TARGET_VECTOR_DIMENSIONS

    Number of dimensions produced by the embedding model. This value must match the dimensions supported by the embedding model.

    PDC_CHATBOT_TARGET_VECTOR_BATCH_SIZE

    Number of records processed in a single batch during vector indexing. Higher values improve throughput but increase memory usage.

    PDC_CHATBOT_TARGET_VECTOR_MAX_CONCURRENT_BATCHES

    Maximum number of vector indexing batches processed in parallel. Increasing this value increases the load on the system.

    PDC_CHATBOT_CONVERSATIONAL_ROLES

    User roles allowed to receive conversational (vector-based) chatbot responses. You can add additional roles separated by a comma.

    PDC_CHATBOT_SCHEDULE_VECTOR_INDEX_UPDATE

    Schedule that controls how often the chatbot refreshes the vector index (for example, @daily).

    You cannot change the embedding model after the initial setup. If you need to use a different embedding model, you must redeploy the chatbot with a fresh vector index.

  6. Save the file and exit the editor.

  7. Restart Pentaho Data Catalog services.

  8. Sign in to Data Catalog and verify that the chatbot returns responses.

Result

The chatbot is configured with a large language model and embedding settings. Users can use the chatbot to receive contextual responses in the chatbot.

What next

You can configure additional roles for conversational search to control which users receive vector-based responses. For more information, see Configure Data Catalog user roles for conversational search chatbot.

Configure Data Catalog user roles for conversational search chatbot

In Pentaho Data Catalog, conversational search uses vector-based retrieval and large language models to return contextual responses, while other users continue to receive standard reporting (BIDB-based) responses.

In Data Catalog, by configuring roles for conversational search, you can control access to advanced chatbot capabilities based on user responsibilities, such as allowing users with roles, such as Data Stewards and Business Stewards, to receive deeper, semantic responses. This procedure explains how to configure which user roles can use conversational search in the Data Catalog chatbot.

Prerequisites

  • Data Catalog is deployed using Docker Compose or Amazon EKS.

  • The chatbot feature is enabled.

  • You have administrative access to the deployment environment.

Docker deployment

Perform the following steps to configure Data Catalog user roles for the conversational search chatbot in the Docker deployment:

Procedure

  1. Open a terminal session on the server where Data Catalog is deployed using Docker Compose.

  2. Change to the Data Catalog Docker deployment directory.

  3. Stop all running Data Catalog services.

  4. Open the environment configuration file.

  5. Set the conversational search roles separated by a comma.

    By default, conversational search is enabled only for the Data Steward role. To know more about user roles in Data Catalog, see User roles and permissions in Data Catalog.

  6. Save the file and exit the editor.

  7. Start the Data Catalog services.

Result

Conversational search is enabled only for the configured user roles. Users with these roles receive vector-based, context-aware chatbot responses, while other users continue to receive standard chatbot responses.

Enable or disable business rules in Data Catalog

In Data Catalog, business rules are disabled by default. Administrators can enable or disable business rules by setting a deployment flag, then restarting the affected services in Docker or redeploying them in Amazon EKS deployments.

Perform the following steps to enable or disable business rules in the Data Catalog:

Docker deployments

Procedure

  1. Sign in to the server where Data Catalog is installed and open a terminal.

  2. Go to the deployment folder.

  3. Open the environment configuration file.

    Data Catalog supports overriding variables from vendor/.env.default by placing them in conf/.env.

  4. Enable business rules by adding the following line:

    To disable business rules, set the value to false:

  5. Save the file and restart Data Catalog services.

Result

Business rules are enabled (or disabled) after services restart or the updated configuration is rolled out.

Enable verbose logging for profiling

Verbose logging increases the level of detail written to the application logs during profiling operations. As a Data Catalog admin, when you enable verbose logging, Data Catalog records additional diagnostic information about how profiling jobs are executed, including internal processing steps, execution flow, and runtime details. This detailed logging helps administrators and support teams troubleshoot profiling failures, investigate unexpected results, and analyze performance issues.

Verbose logging generates a higher volume of log data and can increase disk usage.

Perform the following steps to enable verbose logging in Data Catalog:

Procedure

  1. Open a terminal on the Data Catalog deployment server.

  2. Go to the Data Catalog deployment directory.

  3. Open the conf/.env file in a text editor.

  4. Add the following environment variable, or update the value if it already exists:

    This setting enables detailed logging for profiling operations performed by the Web Services component. To disable verbose profiling logging, set the parameter to false:

  5. Save and close the file.

  6. Restart the Data Catalog services to apply the change.

Result

Verbose logging is enabled for profiling jobs. Additional diagnostic information is written to the Data Catalog application logs during profiling operations.

Disable the Physical Assets feature from Data Catalog deployment

By default, the Physical Assets feature is included in the Data Catalog deployment to support OT assets metadata through Pentaho Edge. However, if your deployment does not require this feature, you can disable it by removing its reference from the Compose file. This helps reduce the size of the deployment and saves compute resources. This guide depicts how to disable the Physical Assets feature in Data Catalog.

This procedure only disables the container associated with Physical Assets. It does not remove any binaries or metadata files. The service can be re-enabled at any time by restoring the profile reference.

Before you begin, make sure the following conditions are met:

  • Data Catalog is already installed using the Docker deployment method. To know more about installation, see Install Pentaho Data Catalog.

  • You have access to the deployment directory (pdc-docker-deployment) and permission to edit the .env.default or .env file.

Perform the following steps to disable the Physical Assets feature:

  1. Navigate to the following directory:

  2. Open the .env.default file in a text editor:

  3. Locate the COMPOSE_PROFILES line. It may look like this:

  4. Remove physical-assets from the list:

  5. Save the file and return to the deployment root folder:

  6. Restart the deployment using the following command:

You have successfully disabled the Physical Assets service, and it will no longer be started when deploying or restarting Data Catalog. This reduces the number of running containers and optimizes resource usage for environments where OT asset lineage is not required.

Running Data Catalog workloads using node affinity, taints, and tolerations

Data Catalog is a distributed application that runs several containerized components, including the application, database, and worker pods. Worker pods perform data scanning, profiling, and metadata ingestion from enterprise data sources. These operations often involve direct access to sensitive or large-scale datasets.

In large or security-sensitive deployments, administrators often need precise control over where workloads run within a Kubernetes (EKS) cluster. This control helps ensure that critical services and data processing tasks run in secure, compliant, and performance-optimized environments. It becomes especially important when certain components of Data Catalog handle data that is confidential, high-volume, or requires specialized compute resources such as GPUs or high-performance storage.

By using Kubernetes node affinity, taints, and tolerations, you can configure Data Catalog so that:

  • Worker pods run only on specific nodes. For example, nodes in a restricted security group or a separate availability zone.

  • Non-worker workloads, such as the user interface or metadata services, are prevented from running on those nodes.

  • The system continues to meet data segregation and compliance requirements without affecting performance or scalability.

Running PDC workloads using node affinity, taints, and tolerations provides a secure, compliant, and efficient way to manage distributed deployments across different availability zones or network segments.

Perform the following steps to configure node affinity, taints, and tolerations for Data Catalog workloads.

Before you begin

  • Ensure that your Data Catalog deployment is running on Amazon Elastic Kubernetes Service (EKS).

  • Verify that you have kubectl and AWS CLI installed and configured with permissions to update node groups and apply taints.

  • Identify the node group where you want to run PDC worker workloads.

  • Obtain access to the custom-values.yaml file used for your PDC Helm deployment.

  • Confirm that your deployment uses Helmfile for orchestration.

Procedure

  1. Open a terminal on the machine that manages your Kubernetes cluster.

  2. Apply a taint to the node group you want to reserve for worker workloads.

    Replace the placeholders with your cluster and node group names.

    • key=dedicated specifies the taint key.

    • value=ws indicates that the node is dedicated for worker services.

    • effect=NO_SCHEDULE prevents other pods from being scheduled on these nodes unless they have a matching toleration.

    Tip: Use descriptive taint keys and values that reflect the node group’s purpose, such as key=data-processing or value=worker.

  3. Edit the custom-values.yaml file for your Helm deployment.

    For PDC 10.2.8 and later, locate the job-server section and add:

    Replace <your-nodegroup-name> with the name of the dedicated node group.

  4. Save the file and deploy the configuration.

    The -n pentaho flag ensures that the deployment targets the correct namespace.

Result

You have successfully configure node affinity, taints, and tolerations for Data Catalog workloads. After deployment:

  • Data Catalog worker pods are scheduled only on the nodes defined by the node affinity rules.

  • Other pods are prevented from being scheduled on those nodes unless they include a matching toleration.

  • Your Data Catalog deployment now supports workload segregation across different availability zones or network segments in AWS.

This configuration helps validate compliance and segregation requirements for deployments where worker nodes must belong to distinct security groups or availability zones.

Next steps

  • To verify the configuration, run:

    Confirm that worker pods are assigned to nodes in the expected node group.

  • Monitor node usage using Amazon EKS Console or the kubectl describe node command.

Additional information

For more information, see the official AWS documentation: Place Kubernetes pods on Amazon EKS by using node affinity, taints, and tolerations

Install user-provided SSL certificates

To provide a greater level of security to your data, you can use signed Secure Sockets Layer (SSL) certificates from your Certificate Authority (CA) with Data Catalog.

Data Catalog automatically installs self-signed certs in the <install-directory>/conf/https directory as server.key (PEM-encoded private key) and server.crt (PEM-encoded self-signed certificate). You can replace these files with certificates signed by your CA.

Use this procedure to install signed SSL certificates for Data Catalog:

  1. On your Data Catalog server, navigate to the Data Catalog installation directory <install-directory>/conf/https, where <install-directory> is the directory where Data Catalog is installed.

    • server.key is a PEM-formatted file that contains the private key of a specific certificate.

    • server.crt is a PEM-formatted file containing the certificate.

  2. Replace the <install-directory>/conf/https/server.key file with the PEM-encoded private key used to sign the SSL certificate or generate a new private key in PEM-encoded format.

  3. Replace the <install-directory>/conf/https/server.crt file with the PEM-encoded signed certificate associated with the private key in Step 1.

    If a new private key is generated, then you need to download a new PEM-encoded signed SSL certificate from your CA.

  4. Append the <install-directory>/conf/extra-certs/bundle.pem file with the following three certificates in this order:

    1. Top level PEM-encoded signed SSL certificate (basically the content of the <install-directory>/conf/https/server.crt file).

    2. Intermediate PEM-encoded certificate, if any, from your CA.

    3. Root PEM-encoded certificate, if any, from your CA.

  5. Navigate to the Data Catalog <install-directory>.

  6. Use the following command to restart Data Catalog:

    ./pdc.sh restart

The SSL certificates are installed.

Check and remove outdated certificates

If Data Catalog services fail to start or show SSL-related errors, the issue might be caused by expired or outdated certificates in the bundle.pem file. You can identify and remove outdated certificates by checking their validity dates.

Before you begin

  • Ensure you have access to the server where Data Catalog is installed.

  • Verify that you have permission to view and edit files in the conf/extra-certs directory.

Procedure

  1. Go to the directory where the bundle.pem file is stored:

  2. Run the following command to check the validity dates of all certificates in the bundle.pem file:

  3. Review the command output. You see sections similar to the following:

  4. Interpret the fields in the output:

    Field Description notBefore Date from which the certificate becomes valid. notAfter Date after which the certificate expires.

  5. Compare the current date with the notAfter value: If the current date is later than notAfter, the certificate has expired. For example:

  6. Edit the bundle.pem file to remove the expired certificates.

    • Use any text editor such as vi or nano.

    • Remove the complete certificate block starting from -----BEGIN CERTIFICATE----- to -----END CERTIFICATE-----.

  7. Save the file and restart the Data Catalog services.

Result

The bundle.pem file now contains only active certificates, preventing SSL validation issues during Data Catalog startup or connectivity.

Add email domains to the safe list in Data Catalog after deployment

During the initial deployment of Data Catalog, it is typically configured to allow only a predefined set of email domains for user authentication. However, you might grant access to users with email addresses from new domains. Instead of redeploying Data Catalog, which can cause downtime and operational delays, you can dynamically update the list of allowed email domains using the Identity & Access Management (IAM) APIs.

Adding email domains and SMTP details during the initial Data Catalog deployment is always a best practice. For more information, see the Install Pentaho Data Catalog.

Perform the following steps to add email domains to the safe list using IAM APIs after deployment:

Prerequisites

  • You must have administrative access to use the IAM APIs.

  • Identify your Data Catalog DNS (for example, catalog.example.com).

  • Obtain admin credentials to generate a Bearer token.

Procedure

  1. Open the CMD prompt and run the following cURL command to generate an authentication token to interact with the IAM APIs:

    • Replace <your-server-url> with your Data Catalog server URL.

    • Replace <admin-username> and <admin-password> with the actual Keycloak master realm admin user credentials. The response includes the token value.

  2. Before updating, you can view the currently configured email domains using the following GET command.

    The response displays the current domain configuration:

  3. Run the following IAM API cURL request to update email domains:

    • Replace <your-server-url> with your Data Catalog server URL.

    • Replace <ACCESS_TOKEN> with the token obtained in the previous step.

    • Replace <provider-id> with your Data Catalog server’s domain or IP address used during installation.

    • Modify the "emailDomains" list as needed.

    Note: Do not remove hv.com and hitachivantara.com from the email domain list.

  4. Run the GET command again (see Step 2) to verify that the email domains are added.

    The response displays the current domain configuration:

New email domains have been added to the Data Catalog safe list, and users with those domains can now sign in successfully.

Set up an email server to send Data Catalog notifications

To set up Data Catalog to send email notifications to users, you can configure any Simple Mail Transfer Protocol (SMTP) server that meets your needs.

Examples of notifications are when a user is tagged with '@' in a comment or set up in a data pipe template to be notified when a job completes.

The steps to set up an SMTP server that are in the Installing Data Catalog topic in Install Pentaho Data Catalog only set up the forgot password functionality.

To integrate an SMTP server with Data Catalog, use the following steps:

  1. Gather the following information for the SMTP server you want to use:

    • Host name of SMTP server (IP address or domain name)

    • Port number for SMTP server

    • Username on SMTP server in <mail userID>@<domain>.com format

    • Password for username

    • Sender mail ID in <mail userID>@<domain>.com format

    • Whether to use Transport Layer Security (TLS) or Secure Sockets Layer (SSL) security.

    • TLS or SSL port number For example, you can use Gmail’s SMTP server to send emails from your application. Here are the SMTP server configuration settings for Gmail:

    • SMTP Server Address

      smtp.gmail.com

    • Secure Connection

      TLS/SSL based on your mail client/website SMTP plugin

    • SMTP Username

      your Gmail account (xxxx@gmail.com)

    • SMTP Password

      your Gmail password

    • Gmail SMTP port

      465 (SSL) or 587 (TLS)

  2. Log into Data Catalog using root user credentials to configure Data Catalog to use the SMTP server, as in the following example:

    https://*&lt;full domain name for PDC server&gt;*/

  3. Navigate to the configuresystem/smtp directory on the Data Catalog server, as in the following example:

    https://*&lt;full domain name for PDC server&gt;*/configuresystem/smtp

    The Configure Your System page opens.

  4. Specify the SMTP server information as detailed in the following table:

Field
Description

Host

IP address or domain name of SMTP server

Port

Port number for SMTP server

Username

User name in *&lt;mail userID&gt;*@*&lt;domain&gt;*.com format

Password

Password for user name specified above

Sender Mail

Sender mail ID in *&lt;mail userID&gt;*@*&lt;domain&gt;*.com format

Encryption

  • TLS: Default value (leave the Use SSL checkbox blank)

  • SSL: Select the Use SSL checkbox

5. Click Test Connection to test the integration. A success confirmation message is displayed next to the Test Connection button.

6. Click Save Changes.

The SMTP server is configured.

Update SMTP details in Data Catalog after deployment

Adding Simple Mail Transfer Protocol (SMTP) details in Data Catalog enables email notifications and alerts within the application, such as:

  • Alerts about Data Catalog changes, approvals, and errors like data ingestion, metadata extraction, or synchronization failures.

  • Password reset links when users forget their credentials.

  • Notification alerts when tagged in the comments tab.

SMTP details are typically configured during the initial deployment of Data Catalog. However, if you want to update SMTP details post-deployment, you can use the Identity & Access Management (IAM) APIs without redeploying Data Catalog, which might cause downtime and operational delays.

Adding email domains and SMTP details during the initial Data Catalog deployment is always the best practice. For more information, see the Installing Data Catalog topic in Get started with Pentaho Data Catalog document.

Perform the following steps to update SMTP details in Data Catalog using IAM APIs after deployment:

Ensure you have sufficient access to use the IAM APIs.

  1. To generate an authentication token to interact with the IAM APIs, open the CMD prompt, and run the following cURL command:

    • Replace <your-server-url> with your Data Catalog server URL.

    • Replace <admin-username> and <admin-password> credentials with the actual admin credentials. The response includes the token value.

  2. To update SMTP details, run the following IAM API cURL request:

    Parameter
    Description

    <PDC_HOST>

    The host name or IP address of your Data Catalog instance.

    <TENANT_NAME>

    The tenant name, typically "pdc".

    <TOKEN_VALUE>

    A valid authentication token (must be obtained through IAM authentication).

    <SMTP_PASSWORD>

    The password for the SMTP server authentication.

    <REPLY_TO_DISPLAY_NAME>

    The display name for the reply-to email address.

    <SMTP_PORT>

    The port number used by the SMTP server.

    <SMTP_HOST>

    The SMTP server host address).

    <REPLY_TO_EMAIL>

    The reply-to email address.

    <FROM_EMAIL>

    The email address used to send notifications.

    <FROM_DISPLAY_NAME>

    The display name associated with the sender’s email.

    <ENVELOPE_FROM>

    The envelope sender address (optional).

    <SMTP_USERNAME>

    The username for SMTP authentication.

You have successfully updated SMTP details in Data Catalog.

Configure proxy server settings for the Licensing-API service

In Pentaho Data Catalog, the Licensing-API service is responsible for managing and validating software licenses, ensuring that only authorized users and services can access Data Catalog features. When Data Catalog is deployed in an enterprise environment that restricts direct internet access, services like the Licensing-API require a proxy server to reach external licensing servers and authenticate endpoints.

Post deployment of Data Catalog, perform the following steps to configure the proxy server for the Licensing-API service:

When configuring the proxy server for the Licensing-API service, use the domain name instead of the IP address. SSL certificates are typically issued for domain names, ensuring secure communication.

Ensure that you have:

  • Access to the conf/.env and vendor/docker-compose.licensing.yml files.

  • Administrative privileges to modify configuration files and restart services.

  • The required proxy server details (domain, port, username, and password).

  • The SSL certificate file (proxy-cert.pem) if required for secure proxy connections.

Procedure

  1. To configure proxy environment variables, go to Data Catalog root folder and then open the conf/.env file.

  2. In the conf/.env file, update the following proxy variables with respective values:

    Variable
    Description
    Example Value

    LICENSING_SERVER_PROXY_ENABLED

    Enables or disables proxy configuration.

    true or false

    LICENSING_SERVER_PROXY_DOMAIN

    The domain or IP address of the proxy server.

    10.177.176.126

    LICENSING_SERVER_PROXY_PORT

    The port number used for proxy communication.

    443

    LICENSING_SERVER_PROXY_USER

    The username for proxy authentication.

    admin

    LICENSING_SERVER_PROXY_PASSWORD

    The password for proxy authentication.

    password

    Note: It is a best practice to avoid hard coding sensitive credentials like PROXY_USER and PROXY_PASSWORD. Use secret management tools or environment variables to secure them.

  3. To update proxy server configuration in Docker Compose, open the vendor/docker-compose.licensing.yml file and update the licensing-api service configuration as follows:

    Note:

    • The PROXY_ENABLED, PROXY_HOST, PROXY_PORT, PROXY_USER, and PROXY_PASSWORD environment variables are mapped inside the Docker container.

    • The JAVA_EXTRA_CERTS is set to "cert.pem" to configure SSL certificates for proxy authentication.

    • A volume mount is added to ensure that the SSL certificate file proxy-cert.pem is accessible within the container.

  4. (Optional) If the proxy server requires SSL authentication, place the SSL certificate file (proxy-cert.pem) in the specified directory:

    Note: Ensure that the file permissions allow access by the Licensing-API service.

  5. After updating the configuration, restart the Data Catalog services to apply the changes:

You have successfully configured the proxy server settings for Licensing APIs in Data Catalog.

Configure metadata index refresh threshold for profiling

In Data Catalog, the PDC_WS_REFRESH_MDS_INDEX_EVERY_N environment variable controls how frequently the metadata index (MDS index) is refreshed during processing. By default, Data Catalog refreshes the metadata index at predefined intervals while handling profiling and metadata operations. In high-volume environments or during large profiling jobs, you might want to override this threshold to control how often the index refresh occurs.

Changing the metadata index refresh threshold can affect system performance and indexing behavior.

Perform the following steps to override the default refresh interval in Data Catalog:

Procedure

  1. Open a terminal on the Data Catalog deployment server.

  2. Go to the Data Catalog deployment directory.

  3. Open the conf/.env file in a text editor.

  4. Add or update the following environment variable:

    This value specifies the number of processed records or operations after which the metadata index is refreshed.

  5. Save the file and close the editor.

  6. Restart the Data Catalog services to apply the change.

Result

Data Catalog refreshes the metadata index after every 100000 operations, based on the configured threshold. Use this configuration only when necessary and revert to the default setting if it is no longer required.

Configure job server auto-scaling in Amazon EKS

The Job Server in Pentaho Data Catalog can automatically scale its pods in an Amazon Elastic Kubernetes Service (EKS) cluster based on CPU and memory utilization. This auto-scaling applies to all jobs that are executed through the Job Server, including data profiling, data identification, and metadata ingestion tasks. By enabling auto-scaling, Data Catalog maintains consistent performance for job executions while optimizing compute resource usage.

Scaling operations are managed through the Kubernetes Horizontal Pod Autoscaler (HPA) for pod-level scaling and the EKS Cluster Autoscaler for node-level scaling.

Perform the following steps to configure job server auto-scaling in Amazon EKS:

Before you begin

Ensure the following prerequisites are met before enabling Job Server auto-scaling:

  • Data Catalog is deployed in an Amazon EKS cluster.

  • The Cluster Autoscaler is enabled for the EKS node groups.

  • You have administrative privileges to modify and redeploy the Helm configuration.

  • Identify the namespace where Data Catalog is installed.

  • Verify that your workload profiling results (for example, file profiling or JDBC profiling) are available. These results help you define the appropriate CPU and memory utilization thresholds for scaling.

Procedure

  1. Open the Job Server Helm values file.

  2. Enable auto-scaling and define utilization parameters.

    Add or update the following configuration under the deployments section:

    Note:

    • Adjust the CPU and memory utilization percentages according to the workload behavior.

    • For high-throughput profiling or ingestion, a lower threshold (for example, 50–60%) helps scale out faster.

    • The minReplicas and maxReplicas values define the lower and upper bounds for the Job Server pods.

  3. Save the configuration file.

  4. Redeploy Data Catalog using Helmfile.

    This command applies the updated Helm values and triggers a rolling update for the Job Server deployment.

  5. Verify that the Horizontal Pod Autoscaler (HPA) is active.

    The output lists the Job Server HPA with its configured CPU and memory thresholds.

  6. Monitor scaling activity.

    As workload intensity increases, additional Job Server pods are created automatically. When utilization decreases, the pods scale in to conserve resources.

Result

Data Catalog dynamically scales the Job Server pods based on CPU and memory utilization thresholds. The scaling behavior ensures optimal resource consumption while maintaining consistent performance for heavy profiling or ingestion workloads.

Example Reference Configuration

Parameter
Value
Description

minReplicas

1

Minimum number of Job Server pods

maxReplicas

100

Maximum number of Job Server pods

targetCPUUtilizationPercentage

80

Scale-out threshold for CPU usage

targetMemoryUtilizationPercentage

80

Scale-out threshold for memory usage

Next steps

Adjust jobPoolMaxSize in the configuration to control the number of concurrent jobs per pod. It defines the number of simultaneous jobs that each Job Server pod can execute. For example, if jobPoolMaxSize=10 and there are 4 Job Server pods, up to 40 jobs can run in parallel across the cluster.

Configure worker profiling overrides for memory management

In Data Catalog, you can use advanced worker-level environment variables to tune profiling behavior and memory usage. These variables help control how structured and unstructured profiling tasks are processed, especially in environments with large datasets or limited system resources.

Adjust these parameters only when you need to optimize memory consumption, control concurrency, or manage the processing of large unstructured files. It is recommended to apply these overrides in consultation with engineering or support for production environments.

Perform the following steps to configure worker profiling overrides for memory management in Data Catalog:

Procedure

  1. Open a terminal on the Data Catalog deployment server.

  2. Go to the Data Catalog deployment directory.

  3. Open the conf/.env file in a text editor.

  4. Add or update the following environment variables:

    • PDC_WS_MAX_COLUMNS_IN_MEMORY=40 Limits the number of columns processed in memory during profiling. Reducing this value can help control memory usage for wide tables.

    • PDC_WS_PROFILE_HEAVY_THREADS=4 Defines the number of worker threads allocated for heavy profiling tasks. Adjust this value based on available CPU and memory resources.

    • PDC_WS_HEAVY_UNSTRUCTURED_MAX_SIZE=50MB Sets the maximum file size for heavy unstructured processing. Files larger than this threshold are handled differently to prevent excessive memory consumption.

    • PDC_WS_SKIP_FILE_BITCOUNT=false Controls whether file bit-count calculations are skipped during profiling. Setting this to false ensures full processing, while true can reduce processing overhead.

  5. Save the file and close the editor.

  6. Restart the Data Catalog services to apply the changes.

Result

The worker profiling configuration is updated. Profiling operations now use the specified memory and concurrency limits.

Improper tuning of these parameters can lead to performance degradation or resource exhaustion. Monitor system performance and logs after applying changes, and revert to default values if issues occur.

Configure data discovery batch size for worker processing

Data Catalog can split data discovery jobs into multiple secondary workers to process files in parallel. The PDC_WS_DATA_DISCOVERY_ENTITY_BATCH_SIZE parameter controls the number of files each secondary worker processes in a single batch.

By default, each worker processes 20,000 files. In environments with large datasets, increasing this value can reduce the number of worker jobs and improve overall processing efficiency. However, higher values can increase memory and resource consumption. Adjust this parameter based on system capacity and workload.

For Docker deployments

Perform the following steps to increase the number of files processed by each secondary worker in a Docker deployment.

Before you begin

  • Ensure that you have administrator access to the PDC deployment server.

  • Ensure that you have permission to edit the conf/.env file.

  • Review system resource capacity before increasing the batch size.

Procedure

  1. Open a terminal on the PDC deployment server.

  2. Go to the PDC deployment directory.

  3. Open the conf/.env file in a text editor.

  4. Add or update the following environment variable:

    This value specifies the number of files each secondary worker processes.

  5. Save the file and close the editor.

  6. Restart the Docker deployment.

Result

The data discovery batch size is updated. Each secondary worker processes up to 50,000 files per batch.

Configure the generic metadata worker retry duration

In Data Catalog, as a admin, you can configure the retry duration for the generic metadata importer worker. This setting controls the maximum elapsed time that the worker continues retrying a metadata import job before it stops attempting further retries. By adjusting this parameter, you can control how long Data Catalog attempts to recover from transient failures, such as temporary connectivity issues or short-lived infrastructure disruptions. Increasing the retry duration can help improve resilience in unstable environments. Reducing the value can prevent prolonged retry cycles that consume system resources.

Perform the following steps to configure the generic metadata worker retry duration in Data Catalog:

Procedure

  1. Open a terminal on the Data Catalog deployment server.

  2. Go to the Data Catalog deployment directory.

  3. Open the conf/.env file in a text editor.

  4. Add or update the following environment variable:

    This value specifies the maximum number of hours that the generic metadata importer worker continues retrying a failed import job before terminating the retry process.

  5. Save the file and close the editor.

  6. Restart the Data Catalog services to apply the change.

Result

The generic metadata worker execution duration is configured in Data Catalog. When you rerun the metadata import job, it runs for the duration specified in the PDC_WS_DEFAULT_GENERIC_METADATA_IMPORTER_MAX_ELAPSED_HOURS setting.

Configure Smart Type to SQL feature in Data Catalog

In Data Catalog, you can use the Smart Type to SQL feature, which converts natural-language text into executable SQL queries within Data Pipes. This feature uses a large-language-model (LLM) service to interpret user input and automatically generate valid SQL statements for the selected database tables.

Perform the following steps to configure the Smart Type to SQL feature in Data Catalog:

Before you begin

  • Ensure your Data Catalog deployment includes the ml-gateway-service.

  • Confirm that the aiml profile is enabled. The feature is unavailable without it.

  • Obtain valid credentials or API keys for your LLM provider.

Procedure

For Docker Compose deployment

  1. Go to the conf/.env file in your Data Catalog installation directory.

  2. Add the following environment variables:

    ML_LLM_MODEL="" ML_LLM_API_KEY="" ML_LLM_INFERENCE_BASE_URL=""

  3. Save the file and restart the containers for the configuration to take effect.

For Kubernetes deployment

  1. Open the values.yaml file of the ml-gateway-service.

  2. Update the following configuration parameters:

    llmModel: llmApiKey: llmInferenceBaseUrl:

  3. Save the file and redeploy the ml-gateway-service.

Result

You have successfully configured Smart Type to SQL feature. When the aiml profile is active, users can enter plain-language prompts in the SQL Editor to generate valid SQL queries automatically.

Configure database password encoding for special characters

By default, the pg-migration service in Pentaho Data Catalog automatically encodes special characters (such as @, :, /, and ?) in PostgreSQL database passwords to ensure they are safely included in the database connection URL, preventing connection failures caused by unescaped reserved characters. If your environment requires passing passwords without URI encoding, you can disable this behavior by setting the PDC_PG_MIGRATIONS_DB_PASSWORD_REQUIRED_ENCODING environment variable to false.

Disabling URI encoding is not recommended, as it may cause connection failures if passwords contain special characters.

Perform the following steps to disable URI encoding for passwords used by the pg-migration service:

Before you begin

Ensure that you have permission to edit the PDC environment configuration file (.env or .env.default).

Procedure

  1. Open the environment configuration file: