Troubleshooting Pentaho Data Catalog

The Pentaho Data Catalog log files contain information that can help you determine the root cause of error messages you might see. Refer to the following topics for information on how to resolve the issues causing the error messages.

Low disk space message

If you see a Low disk space message from Pentaho Data Catalog while loading images into the Docker repository, you can resolve this issue by linking the Docker root directory to another directory.

Important: The other directory should have at least 100 GB of free space.

Use the following steps to resolve this issue:

1. Enter the following commands to link the /var/lib/docker directory to a directory with at least 100 GB of free space.

   Note: In this example, the directory with at least 100 GB of free space is <dir with min 100 GB free>. You should replace <dir with min 100 GB free> in the command with the full path to your directory with a minimum of 100 GB of free space.

   sudo systemctl stop docker
   sudo mv /var/lib/docker <dir with min 100 GB free>
   sudo ln -s <dir with min 100 GB free> /var/lib/docker
   sudo systemctl start docker

2. Repeat the action that produced the Low disk space message.

The action should succeed without producing a Low disk space message.

Authentication failure after upgrading Remote Worker from 10.2.7 to 10.2.9

When upgrading the Remote Worker from version 10.2.7 to 10.2.9, the Remote Worker container fails to start. The startup log displays an error indicating that authentication failed with the SASL mechanism SCRAM-SHA-512.

This issue occurs because the Kafka user credentials used by the Remote Worker become invalid during the upgrade.

After the upgrade, the Remote Worker container (pdc-ws-remote) fails to start and shows an authentication error similar to the following in the log:

pdc-ws-remote-1  | {"timestamp":"2025-10-23T14:58:57.283873123Z","sequence":447,"loggerClassName":"org.jboss.logging.Logger","loggerName":"io.quarkus.runtime.Application","level":"ERROR","message":"Failed to start application","threadName":"main","threadId":1,"mdc":{},"ndc":"","hostName":"b6d234061919","processName":"/opt/java/openjdk/bin/java","processId":1,"exception":{"refId":1,"exceptionType":"java.lang.RuntimeException","message":"Failed to start quarkus","frames":[{"class":"io.quarkus.runner.ApplicationImpl","method":"doStart"},{"class":"io.quarkus.runtime.Application","method":"start","line":101},{"class":"io.quarkus.runtime.ApplicationLifecycleManager","method":"run","line":121},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":77},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":48},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":137},{"class":"io.quarkus.runner.GeneratedMain","method":"main"}],"causedBy":{"exception":{"refId":2,"exceptionType":"jakarta.enterprise.inject.CreationException","message":"java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512","frames":[{"class":"com.pentaho.job.server.services.JobServiceMain_Bean","method":"create"},{"class":"com.pentaho.job.server.services.JobServiceMain_Bean","method":"create"},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"createInstanceHandle","line":119},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":38},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":35},{"class":"io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances","method":"c9"},{"class":"io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances","method":"computeIfAbsent"},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"get","line":35},{"class":"io.quarkus.arc.impl.ClientProxies","method":"getApplicationScopedDelegate","line":23},{"class":"com.pentaho.job.server.services.JobServiceMain_ClientProxy","method":"arc$delegate"},{"class":"com.pentaho.job.server.services.JobServiceMain_ClientProxy","method":"arc_contextualInstance"},{"class":"com.pentaho.job.server.services.JobServiceMain_Observer_Synthetic_Fe9tBUvfDGKBAGnSqsrAvtpRsn8","method":"notify"},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notifyObservers","line":365},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notify","line":347},{"class":"io.quarkus.arc.impl.EventImpl","method":"fire","line":81},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"fireLifecycleEvent","line":163},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"handleLifecycleEvents","line":114},{"class":"io.quarkus.runner.recorded.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy_0"},{"class":"io.quarkus.runner.recorded.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy"},{"class":"io.quarkus.runner.ApplicationImpl","method":"doStart"},{"class":"io.quarkus.runtime.Application","method":"start","line":101},{"class":"io.quarkus.runtime.ApplicationLifecycleManager","method":"run","line":121},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":77},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":48},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":137},{"class":"io.quarkus.runner.GeneratedMain","method":"main"}],"causedBy":{"exception":{"refId":3,"exceptionType":"java.util.concurrent.ExecutionException","message":"org.apache.kafka.common.errors.SaslAuthenticationException: Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512","frames":[{"class":"java.util.concurrent.CompletableFuture","method":"reportGet"},{"class":"java.util.concurrent.CompletableFuture","method":"get"},{"class":"org.apache.kafka.common.internals.KafkaFutureImpl","method":"get","line":165},{"class":"com.pentaho.job.server.utils.Consumer","method":"createTopicIfNotExists","line":138},{"class":"com.pentaho.job.server.utils.Consumer","method":"startConsumer","line":43},{"class":"com.pentaho.job.server.services.JobServiceMain","method":"onStart","line":80},{"class":"com.pentaho.job.server.services.JobServiceMain_Bean","method":"doCreate"},{"class":"com.pentaho.job.server.services.JobServiceMain_Bean","method":"create"},{"class":"com.pentaho.job.server.services.JobServiceMain_Bean","method":"create"},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"createInstanceHandle","line":119},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":38},{"class":"io.quarkus.arc.impl.AbstractSharedContext$1","method":"get","line":35},{"class":"io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances","method":"c9"},{"class":"io.quarkus.arc.generator.Default_jakarta_enterprise_context_ApplicationScoped_ContextInstances","method":"computeIfAbsent"},{"class":"io.quarkus.arc.impl.AbstractSharedContext","method":"get","line":35},{"class":"io.quarkus.arc.impl.ClientProxies","method":"getApplicationScopedDelegate","line":23},{"class":"com.pentaho.job.server.services.JobServiceMain_ClientProxy","method":"arc$delegate"},{"class":"com.pentaho.job.server.services.JobServiceMain_ClientProxy","method":"arc_contextualInstance"},{"class":"com.pentaho.job.server.services.JobServiceMain_Observer_Synthetic_Fe9tBUvfDGKBAGnSqsrAvtpRsn8","method":"notify"},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notifyObservers","line":365},{"class":"io.quarkus.arc.impl.EventImpl$Notifier","method":"notify","line":347},{"class":"io.quarkus.arc.impl.EventImpl","method":"fire","line":81},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"fireLifecycleEvent","line":163},{"class":"io.quarkus.arc.runtime.ArcRecorder","method":"handleLifecycleEvents","line":114},{"class":"io.quarkus.runner.recorded.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy_0"},{"class":"io.quarkus.runner.recorded.LifecycleEventsBuildStep$startupEvent1144526294","method":"deploy"},{"class":"io.quarkus.runner.ApplicationImpl","method":"doStart"},{"class":"io.quarkus.runtime.Application","method":"start","line":101},{"class":"io.quarkus.runtime.ApplicationLifecycleManager","method":"run","line":121},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":77},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":48},{"class":"io.quarkus.runtime.Quarkus","method":"run","line":137},{"class":"io.quarkus.runner.GeneratedMain","method":"main"}],"causedBy":{"exception":{"refId":4,"exceptionType":"org.apache.kafka.common.errors.SaslAuthenticationException","message":"Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-512","frames":[]}}}}}}}}
pdc-ws-remote-1  | {"timestamp":"2025-10-23T14:58:57.293651565Z","sequence":448,"loggerClassName":"org.jboss.logging.Logger","loggerName":"com.pentaho.exec.utils.Monitor","level":"INFO","message":"monitor: pool exiting","threadName":"Thread-6","threadId":31,"mdc":{},"ndc":"","hostName":"b6d234061919","processName":"/opt/java/openjdk/bin/java","processId":1}

Workaround

You can fix this issue by resetting the Kafka SCRAM-SHA-512 password for the pdcuser on the PDC main server and restarting the Remote Worker.

Use the following steps to resolve this issue:

1. Log in to the Kafka container on the PDC main server:

   sudo ./pdc.sh shell kafka

2. Run the following command to reset the SCRAM-SHA-512 password for the Kafka user pdcuser:

   kafka-configs.sh --bootstrap-server kafka:9092 \
   --alter --add-config 'SCRAM-SHA-512=[password=GvTEOULMgqjOrlfE13]' \
   --entity-type users --entity-name pdcuser

3. Exit the Kafka container.

4. Restart the Remote Worker service:

   sudo ./pdc.sh restart

After resetting the Kafka password and restarting the Remote Worker, authentication succeeds and the Remote Worker starts successfully.

Unable to connect to OpenSearch using HTTPS (Security plugin not initialized)

When accessing OpenSearch over HTTPS, the system may fail to connect because the OpenSearch Security plugin is enabled but not yet initialized. This occurs when the .opendistro_security index does not exist, preventing OpenSearch from recognizing user credentials, roles, TLS settings, and other security configurations.

You may see the following error in the logs:

[2025-06-16T16:53:35,311][ERROR][o.o.s.c.ConfigurationLoaderSecurity7] [af38e3ceb454] Failure no such index [.opendistro_security] retrieving configuration for [ACTIONGROUPS, ALLOWLIST, AUDIT, CONFIG, INTERNALUSERS, NODESDN, ROLES, ROLESMAPPING, TENANTS, WHITELIST] (index=.opendistro_security) 

This issue typically appears during an upgrade (for example, from PDC 10.2.1 to 10.2.6), not in fresh installations of Data Catalog 10.2.5 or later, where the security index is initialized by default.

For fresh installations, the initialization process should be performed only under the supervision of Pentaho Data Catalog Customer Support.

Workaround

Perform the following steps to resolve the issue:

1. Log in to the deployment server where Data Catalog is running.

2. Stop all Pentaho Data Catalog containers:

   --alter --add-config 'SCRAM-SHA-512=[password=GvTEOULMgqjOrlfE13]' \
   --entity-type users --entity-name pdcuser 

./pdc.sh stop

1. Identify the OpenSearch container IDs.

   docker ps -a | grep pdc-opensearch 

2. Remove the OpenSearch containers.

   docker rm <container_id1> <container_id2> <container_id3> 

3. List all Docker volumes related to OpenSearch to confirm their presence:

   docker volume ls | grep pdc_opensearch

   Typical volumes include:

   pdc_opensearch_data 
   pdc_opensearch_snapshots 

⚠️ Caution: Do not delete OpenSearch volumes unless explicitly instructed by Customer Support. Deleting these volumes will permanently remove all OpenSearch data, including indexes, and should only be done under the supervision of support.

1. Delete the pdc_opensearch_data volume:

   docker volume rm pdc_opensearch_data 

2. Delete the pdc_opensearch_snapshots volume:

   docker volume rm pdc_opensearch_snapshots 

3. Restart Data Catalog services:

   ./pdc.sh up 

4. Verify that all the services are up and running now.

   ./pdc.sh ps

After removing the existing OpenSearch volumes and restart the system, the .opendistro_security index is reinitialized. OpenSearch initializes the Security plugin, loads its configuration successfully, and connects over HTTPS without errors.