Upgrade Data Catalog

Upgrading Pentaho Data Catalog ensures that you can take advantage of the latest features, improvements, and security updates while maintaining your existing configurations and data. This section provides step-by-step instructions to upgrade your Data Catalog deployment from earlier versions to newer releases.

Supported upgrades

Pentaho Data Catalog supports upgrades only across the next two versions. You can upgrade directly to the immediate next version or the version after that, but skipping more than two versions in a single upgrade is not supported. If your target version is more than two versions ahead, you must perform the upgrade in multiple stages by first upgrading to an intermediate supported version and then moving to the final target version.

The following matrix shows the supported upgrade paths:

From \ To	11.0	10.2.11	10.2.9	10.2.8	10.2.7	10.2.6	10.2.5	10.2.1
10.2.11	✅ Direct
10.2.9	—	✅ Direct	—	—	—	—	—	—
10.2.8	—	✅ Direct	✅ Direct	—	—	—	—	—
10.2.7		—	✅ Direct	✅ Direct	—	—	—	—
10.2.6		—	—	✅ Direct	✅ Direct	—	—	—
10.2.5			—	❌	✅ Direct	✅ Direct	—	—
10.2.1				❌ via 10.2.5 → 10.2.8	❌ via 10.2.5 → 10.2.7	❌ via 10.2.5	✅ Direct	—
10.2.0				❌	❌	❌	❌	✅ Direct

To avoid problems, always back up configuration files and Docker volumes before upgrading, and review any custom changes in Docker compose files so they can be reapplied after extracting the new package.

Upgrade procedures

The topics in this section guide you through the specific upgrade paths supported for Pentaho Data Catalog:

11.0

Upgrade PDC from 10.2.11 to 11.0 in Docker Deployment: Follow this procedure if your current deployment is on version 10.2.11 and you want to upgrade to 11.0.

10.2.x

• Upgrade PDC 10.2.5 to 10.2.6, 10.2.7, 10.2.8, 10.2.9, or 10.2.11: Follow this procedure if your current deployment is on version 10.2.5 and you want to upgrade to 10.2.6, 10.2.7, 10.2.8, or 10.2.9.
• Upgrade PDC 10.2.1 to 10.2.5: Follow this procedure if your current deployment is on version 10.2.1 and you want to upgrade to 10.2.5.
• Upgrade PDC 10.2.0 to 10.2.1: Follow this procedure if your current deployment is on version 10.2.0 and you want to upgrade to 10.2.1.

If you want to upgrade Data Catalog to a patch version, see Upgrade Data Catalog to a patch version.

Before starting, ensure you review the prerequisites, back up your configuration and data, and prepare the deployment environment as described in the respective upgrade topics.

Upgrade PDC from 10.2.11 to 11.0 in Docker Deployment

Pentaho Data Catalog 11.0 introduces an updated database architecture that contains private schemas, which improve security, scalability, and deployment flexibility. This release supports dedicated PostgreSQL schemas for PDC application data, Business Intelligence Database (BIDB) data, and Keycloak authentication data. These schemas help separate database objects by function and make it easier for administrators to manage access, maintenance, and troubleshooting.

This procedure describes how to upgrade PDC 10.2.11 to PDC 11.0 in a Docker deployment. The upgrade migrates the existing PDC 10.2.11 deployment to the PDC 11.0 architecture. It includes backing up MongoDB data, consolidating PostgreSQL schemas, upgrading the Docker deployment package, restoring MongoDB data to FerretDB, and resetting lineage graph aggregation.

This procedure applies only to Docker deployments. For Kubernetes deployments, use the Kubernetes upgrade procedure when it is available.

Prerequisites

Before you begin, ensure that:

• PDC 10.2.11 is installed and running in a Docker deployment.
• The PDC Docker deployment is installed in /opt/pentaho/pdc-docker-deployment.
• You have administrative access to the deployment server.
• You have permissions to run Docker commands and update files in the PDC deployment directory.
• No metadata ingestion, profiling, data discovery, data pipe, lineage, or other worker jobs are running.
• You have the PDC 11.0 Docker image archive, such as pdc-11.0.0-images.tgz.
• You have the correct PDC 11.0 Docker Compose deployment package for your licensed deployment type, such as full, lineage, PDO, PDM, or classic.
• You have access to the required migration images:

  • repo.pentaho.com/docker/pentaho/cat-mongo2ferret:<version>
  • repo.pentaho.com/docker/pentaho/cat-pgdbmigrate:<version>
• The Docker network used by PDC is available. For a default Docker deployment, the network is pdc_default.
• You have the database credentials required to connect to MongoDB, PostgreSQL, and FerretDB.
• You have reviewed and recorded any deployment-specific customizations, including changes to environment variables, Docker Compose files, certificates, secrets, mounted volumes, and service configuration files.
• You have enough disk space to store MongoDB dumps, PostgreSQL migration dumps, Docker volume backups, and configuration backups.
• If you want to use custom database names for PDC application data, BIDB data, or Keycloak authentication data, decide the database names before you run the PostgreSQL schema consolidation migration. You must use the same database names in the migration configuration files and in the PDC 11.0 environment configuration.

Do not start the PDC 11.0 services until you complete the MongoDB backup and PostgreSQL schema consolidation steps. The upgrade changes the database architecture and requires the migration steps to be completed in the correct order.

The commands in this procedure use the default Docker deployment path, default Docker network, and example file names. If your deployment uses a custom path, network, credentials, or package name, replace the example values with the values from your environment.

Upgrade workflow

The PDC 10.2.11 to PDC 11.0 upgrade is a multi-phase procedure. Complete the upgrade phases in the order.

1

Back up MongoDB data

In this phase, you create a temporary mongotool container, connect it to the PDC Docker network, take a MongoDB dump, and preprocess the dump for later restoration to FerretDB during the upgrade.

Perform the following steps to back up MongoDB data:

1. Load the PDC 11.0 Docker images.

   sudo docker load -i <pdc-11-images-archive>

   Replace <pdc-11-images-archive> with the PDC 11.0 Docker image archive provided for your deployment.

   Example:

   sudo docker load -i pdc-11.0.0-images.tgz

2. Verify that the PDC Docker network exists.

   sudo docker network ls

   Identify the Docker network used by the PDC deployment.

   In a default Docker deployment, the network name is typically pdc_default. If your deployment uses a different Docker network, replace pdc_default with the network name from your environment.
3. Create and start the mongotool container in interactive mode.

   sudo docker run \
    -it \
    --name mongotool \
    --network <pdc-docker-network> \
    --volume <ferretdb-mongo-volume>:/home/data/ \
    <mongotool-image>

   Replace the placeholders as follows:

   • <pdc-docker-network>: Docker network used by the PDC deployment.
   • <ferretdb-mongo-volume>: Docker volume used to store the MongoDB dump and make it available for the restore phase.
   • <mongotool-image>: MongoDB-to-FerretDB migration image provided with the PDC 11.0 upgrade package. Example:

     sudo docker run \
       -it \
       --name mongotool \
       --network pdc_default \
       --volume ferret_mongo_data:/home/data/ \
       repo.pentaho.com/docker/pentaho/cat-mongo2ferret:1-260406122242

     The terminal connects to the mongotool container.
4. From the mongotool container terminal, set the MongoDB connection variables and create the MongoDB dump.

   export SOURCE_HOST="<mongodb-host>"
   export SOURCE_PORT="<mongodb-port>"
   export DUMP_DIR="mongodump_$(date +%Y%m%d_%H%M%S)"
   export SOURCE_USER="<mongodb-user>"
   export SOURCE_PASS="<mongodb-password>"

   Replace the placeholders as follows:

   • <mongodb-host>: Host name or container name for the MongoDB service.
   • <mongodb-port>: MongoDB service port.
   • <mongodb-user>: MongoDB user name with permission to create the dump.
   • <mongodb-password>: Password for the MongoDB user. Example:

     export SOURCE_HOST="mongodb"
     export SOURCE_PORT="27017"
     export DUMP_DIR="mongodump_$(date +%Y%m%d_%H%M%S)"
     export SOURCE_USER="root"
     export SOURCE_PASS="broot"

5. Create the MongoDB dump.

   mongodump \
     --uri="mongodb://${SOURCE_USER}:${SOURCE_PASS}@${SOURCE_HOST}:${SOURCE_PORT}/?authSource=admin" \
     --out="${DUMP_DIR}" \
     --gzip

   The values for SOURCE_USER and SOURCE_PASS are examples from the default Docker deployment. If your deployment uses custom MongoDB credentials, replace these values with the credentials from your environment.
6. Preprocess the MongoDB dump.

   python3 pre_process_dump.py "${DUMP_DIR}"

7. Review the preprocessing output.

   You can ignore the following unsupported collections if they are removed during preprocessing because they do not contain data:

   mds/datasource_capacityusage.metadata.json.gz
   fe/userfqdnaccess.metadata.json.gz

   Important: If preprocessing removes any collections other than the listed unsupported collections, record the collection names and contact Pentaho Support before you continue with the upgrade.

Result

The MongoDB data is backed up and preprocessed in the mongotool container. The generated dump is used later when you restore MongoDB backup data to FerretDB.

2

Migrate PostgreSQL schemas to the consolidated databases

In this phase, you migrate PostgreSQL schemas from the PDC 10.2.11 service databases to the consolidated databases used by PDC 11.0.

The migration includes the following tasks:

• Migrate service schemas to the consolidated pentaho database.
• Migrate the BIDB schema to the consolidated pentaho_bidb database.

The default consolidated database names are pentaho and pentaho_bidb. If your deployment uses custom database names, replace the database name at the end of each target dbUrl with the custom database name. The same names must be configured later in the PDC 11.0 environment configuration before you start the upgraded deployment.

If a database password contains special characters, such as #, URL-encode the password in the dbUrl value. For example, replace # with %23.

Migrate service schemas to pentaho database

Perform the following steps to migrate service schemas to the consolidated pentaho database:

1. Create a working directory for the PostgreSQL schema migration.

   mkdir -p migrate-databases
   cd migrate-databases

2. Create a file named migrate-dbs-config.json and add the following content to the file.

   {
     "target": {
       "name": "pentaho",
       "dbUrl": "postgres://<target-postgres-user>:<target-postgres-password>@<target-postgres-host>:<target-postgres-port>/<pentaho-db>"
     },
     "sources": [
       {
         "name": "rules-postgres-db",
         "dbUrl": "postgres://<rules-postgres-user>:<rules-postgres-password>@<rules-postgres-host>:<rules-postgres-port>/<rules-postgres-database>",
         "schemas": ["rules_engine", "rules_engine_quartz"]
       },
       {
         "name": "user-notifications-postgres-db",
         "dbUrl": "postgres://<user-notifications-postgres-user>:<user-notifications-postgres-password>@<user-notifications-postgres-host>:<user-notifications-postgres-port>/<user-notifications-postgres-database>",
         "schemas": ["user_notifications"]
       },
       {
         "name": "css_auth_proxy-db",
         "dbUrl": "postgres://<css-auth-proxy-postgres-user>:<css-auth-proxy-postgres-password>@<css-auth-proxy-postgres-host>:<css-auth-proxy-postgres-port>/css_auth_proxy",
         "schemas": ["css_auth_proxy", "permission"]
       },
       {
         "name": "global_search-postgres-db",
         "dbUrl": "postgres://<global-search-postgres-user>:<global-search-postgres-password>@<global-search-postgres-host>:<global-search-postgres-port>/global_search",
         "schemas": ["global_search"]
       },
       {
         "name": "pdc_access_request_db",
         "dbUrl": "postgres://<access-request-postgres-user>:<access-request-postgres-password>@<access-request-postgres-host>:<access-request-postgres-port>/pdc_access_request_db",
         "schemas": ["pdc_access_request"]
       },
       {
         "name": "pdc_common",
         "dbUrl": "postgres://<pdc-common-postgres-user>:<pdc-common-postgres-password>@<pdc-common-postgres-host>:<pdc-common-postgres-port>/pdc_common",
         "schemas": ["glossary", "licensing", "workflows"]
       },
       {
         "name": "refdata-db",
         "dbUrl": "postgres://<refdata-postgres-user>:<refdata-postgres-password>@<refdata-postgres-host>:<refdata-postgres-port>/<refdata-postgres-database>",
         "schemas": ["public"],
         "schemaMap": {
           "public": "refdata"
         }
       },
       {
         "name": "mdm",
         "dbUrl": "postgres://<mdm-postgres-user>:<mdm-postgres-password>@<mdm-postgres-host>:<mdm-postgres-port>/<mdm-postgres-database>",
         "schemas": ["public", "jdbci", "sim", "trgm"],
         "schemaMap": {
           "public": "mdm"
         },
         "extensionMap": {
           "mdm": ["dblink"],
           "jdbci": ["jdbc_fdw"],
           "sim": ["pg_similarity"],
           "trgm": ["pg_trgm"]
         }
       },
       {
         "name": "crypto_service",
         "dbUrl": "postgres://<crypto-postgres-user>:<crypto-postgres-password>@<crypto-postgres-host>:<crypto-postgres-port>/<crypto-postgres-database>",
         "schemas": ["crypto"]
       },
       {
         "name": "pdi_logging",
         "dbUrl": "postgres://<pdi-logging-postgres-user>:<pdi-logging-postgres-password>@<pdi-logging-postgres-host>:<pdi-logging-postgres-port>/pdi_logging",
         "schemas": ["public"],
         "schemaMap": {
           "public": "pdi_logging"
         }
       }
     ]
   }

   Replace the placeholders in migrate-dbs-config.json with the values from your deployment.

   • <pentaho-db>: The default value is pentaho. If your deployment uses a custom database name, replace it with that name.
   • <target-postgres-*>: Connection details for the PostgreSQL instance that contains the consolidated `pentaho` database.
   • <rules-postgres-*>: Connection details for the Rules service PostgreSQL database.
   • <user-notifications-postgres-*>: Connection details for the User Notifications service PostgreSQL database.
   • <css-auth-proxy-postgres-*>: Connection details for the CSS Auth Proxy PostgreSQL database.
   • <global-search-postgres-*>: Connection details for the Global Search PostgreSQL database.
   • <access-request-postgres-*>: Connection details for the Access Request PostgreSQL database.
   • <pdc-common-postgres-*>: Connection details for the PDC Common PostgreSQL database
   • <refdata-postgres-*>: Connection details for the Reference Data PostgreSQL database.
   • <mdm-postgres-*>: Connection details for the MDM PostgreSQL database.
   • <crypto-postgres-*>: Connection details for the Crypto service PostgreSQL database.
   • <pdi-logging-postgres-*>: Connection details for the PDI logging PostgreSQL database.

     Example file

     {
       "target": {
         "name": "pentaho",
         "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/pentaho"
       },
       "sources": [
         {
           "name": "rules-postgres-db",
           "dbUrl": "postgres://postgres:secret@rules-postgres:5432/postgres",
           "schemas": ["rules_engine", "rules_engine_quartz"]
         },
         {
           "name": "user-notifications-postgres-db",
           "dbUrl": "postgres://postgres:secret@user-notifications-postgres:5432/postgres",
           "schemas": ["user_notifications"]
         },
         {
           "name": "css_auth_proxy-db",
           "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/css_auth_proxy",
           "schemas": ["css_auth_proxy", "permission"]
         },
         {
           "name": "global_search-postgres-db",
           "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/global_search",
           "schemas": ["global_search"]
         },
         {
           "name": "pdc_access_request_db",
           "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/pdc_access_request_db",
           "schemas": ["pdc_access_request"]
         },
         {
           "name": "pdc_common",
           "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/pdc_common",
           "schemas": ["glossary", "licensing", "workflows"]
         },
         {
           "name": "refdata-db",
           "dbUrl": "postgres://postgres:secret@refdata-postgres:5432/postgres",
           "schemas": ["public"],
           "schemaMap": {
             "public": "refdata"
           }
         },
         {
           "name": "mdm",
           "dbUrl": "postgres://postgres:secret@mdm-postgres:5432/postgres",
           "schemas": ["public", "jdbci", "sim", "trgm"],
           "schemaMap": {
             "public": "mdm"
           },
           "extensionMap": {
             "mdm": ["dblink"],
             "jdbci": ["jdbc_fdw"],
             "sim": ["pg_similarity"],
             "trgm": ["pg_trgm"]
           }
         },
         {
           "name": "crypto_service",
           "dbUrl": "postgres://postgres:secret@crypto-postgres:5432/crypto-service",
           "schemas": ["crypto"]
         },
         {
           "name": "pdi_logging",
           "dbUrl": "postgres://postgres:admin123#@um-postgresql:5432/pdi_logging",
           "schemas": ["public"],
           "schemaMap": {
             "public": "pdi_logging"
           }
         }
       ]
     }

3. Set the file permissions.

   chmod 644 migrate-dbs-config.json

4. Create a dump directory and set the required permissions.

   mkdir -p dumps-dbs
   chmod 777 dumps-dbs

   The migration container must be able to write to the dump directory. If your organization does not allow 777 permissions, set permissions that allow the container user to write to this directory.
5. Run the PostgreSQL database migration container.

    sudo docker run --rm \
      --network <pdc-docker-network> \
      -v "$(pwd)/migrate-dbs-config.json:/usr/src/app/migrate-dbs-config.json" \
      -v "$(pwd)/dumps-dbs:/tmp" \
      -e MIGRATE_DBS_CONFIG_PATH=/usr/src/app/migrate-dbs-config.json \
      <pgdbmigrate-image> \
      migrate-databases

   Replace the placeholders as follows:

   • <pdc-docker-network>: Docker network used by the PDC deployment.
   • <pgdbmigrate-image>: PostgreSQL database migration image provided with the PDC 11.0 upgrade package.

Migrate BIDB schema to pentaho_bidb

Perform the following steps to migrate the BIDB schema to the consolidated pentaho_bidb database:

1. In the same migrate-databases working directory, create a file named migrate-bidb-config.json.
2. Add the following content to the file.

   {
     "target": {
       "name": "pentaho_bidb",
       "dbUrl": "postgres://<target-postgres-user>:<target-postgres-password>@<target-postgres-host>:<target-postgres-port>/<pentaho_bidb>"
     },
     "sources": [
       {
         "name": "bidb",
         "dbUrl": "postgres://<bidb-postgres-user>:<bidb-postgres-password>@<bidb-postgres-host>:<bidb-postgres-port>/bidb",
         "schemas": ["bidb"]
       }
     ]
   }

   Replace the placeholders in migrate-bidb-config.json with the values from your deployment.

   • <pentaho-bidb>: The default name for the consolidated BIDB database is pentaho-bidb. If your deployment uses a custom name, replace it with that name.
   • <target-postgres-user>: User name for the PostgreSQL instance that contains the consolidated `pentaho_bidb` database.
   • <target-postgres-password>: Password for the PostgreSQL user.
   • <target-postgres-host>: Host name or container name for the target PostgreSQL service.
   • <target-postgres-port>: Port for the target PostgreSQL service.
   • <bidb-postgres-user>: User name for the source BIDB PostgreSQL database.
   • <bidb-postgres-password>: Password for the source BIDB PostgreSQL database.
   • <bidb-postgres-host>: Host name or container name for the source BIDB PostgreSQL service.
   • <bidb-postgres-port>: Port for the source BIDB PostgreSQL service.
3. Set the file permissions.

   chmod 644 migrate-bidb-config.json

4. Create a dump directory and set the required permissions.

   mkdir -p dumps-bidb
   chmod 777 dumps-bidb

   The migration container must be able to write to the dump directory. If your organization does not allow 777 permissions, set permissions that allow the container user to write to this directory.
5. Run the PostgreSQL database migration container.

   sudo docker run --rm \
    --network <pdc-docker-network> \
    -v "$(pwd)/migrate-bidb-config.json:/usr/src/app/migrate-bidb-config.json" \
    -v "$(pwd)/dumps-bidb:/tmp" \
    -e MIGRATE_DBS_CONFIG_PATH=/usr/src/app/migrate-bidb-config.json \
    <pgdbmigrate-image> \
    migrate-databases

   Replace the placeholders as follows:

   • <pdc-docker-network>: Docker network used by the PDC deployment.
   • <pgdbmigrate-image>: PostgreSQL database migration image provided with the PDC 11.0 upgrade package.

The required PostgreSQL service schemas are migrated to the consolidated pentaho database, and the BIDB schema is migrated to the consolidated pentaho_bidb database.

3

Upgrade PDC deployment from 10.2.11 to 11.0

In this phase, you disable MongoDB migrations, stop the existing PDC 10.2.11 services, back up the deployment configuration and Docker volumes, replace the Docker deployment package, load the PDC 11.0 images, and start PDC 11.0.

Perform the following steps to upgrade the PDC Docker deployment:

1. Disable mongodb-migrations by adding an override to the Docker Compose configuration file.

   1. Open the conf/docker-compose.yml file.

      sudo vi <pdc-deployment-directory>/conf/docker-compose.yml

   2. Add the following lines and save the file.

      services:
        mongodb-migrations:
          entrypoint: ["sh", "-c", "echo 'MongoDB migrations disabled by override'; exit 0"]
          command: []

      Replace <pdc-deployment-directory> with the path where PDC is installed.

      In a default Docker deployment, the PDC deployment directory is /opt/pentaho/pdc-docker-deployment.

      Important: Disable mongodb-migrations before you start PDC 11.0. The MongoDB data is restored to FerretDB in a later phase of this procedure.
2. Go to the PDC Docker deployment directory.

   cd <pdc-deployment-directory>

   Example:

   cd /opt/pentaho/pdc-docker-deployment

3. Stop the currently running PDC services.

   sudo ./pdc.sh stop

4. Create a backup directory.

   sudo mkdir -p <backup-directory>/docker_volumes

   Replace <backup-directory> with the directory where you want to store the PDC 10.2.11 backup.

   Example:

   sudo mkdir -p /opt/pdc_backup_10_2_11/docker_volumes

5. Back up the conf directory.

   sudo cp -a <pdc-deployment-directory>/conf <backup-directory>

   Example:

   sudo cp -a /opt/pentaho/pdc-docker-deployment/conf /opt/pdc_backup_10_2_11

6. Back up the vendor directory.

   sudo cp -a <pdc-deployment-directory>/vendor <backup-directory>

   Example:

   sudo cp -a /opt/pentaho/pdc-docker-deployment/vendor /opt/pdc_backup_10_2_11

7. Back up the PDC Docker volumes.

   sudo su
   cp -a /var/lib/docker/volumes/<pdc-volume-prefix>* <backup-directory>/docker_volumes/
   exit

   Replace <pdc-volume-prefix> with the prefix used by the PDC Docker volumes.

   Note: In a default Docker deployment, PDC Docker volume names typically start with pdc.
8. Remove the vendor directory from the existing deployment.

   sudo rm -rf <pdc-deployment-directory>/vendor

   Example:

   sudo rm -rf /opt/pentaho/pdc-docker-deployment/vendor

9. Extract the PDC 11.0 Docker Compose deployment package to /opt.

   sudo tar -xvf pdc-<deployment-package>-11.0.0-compose.tgz -C /opt/

   Replace pdc-<deployment-package>-11.0.0-compose.tgz with the PDC 11.0 Docker Compose deployment package for your licensed deployment type.

   Example:

   sudo tar -xvf pdc-<deployment-package>-11.0.0-compose.tgz -C /opt/

   Replace <deployment-package> with the applicable package name, such as full, lineage, pdo, pdm, or classic.
10. If you used custom database names during PostgreSQL schema consolidation, update the database name variables in the PDC 11.0 environment defaults file.

    sudo vi <pdc-deployment-directory>/vendor/.env.default

    Update the following variables with the database names used in the Phase 2 migration configuration files:

    POSTGRES_PDC_PENTAHO_DB="<pentaho-db>"
    POSTGRES_PDC_BI_DB="<pentaho-bidb>"

    Replace the placeholders as follows:

    • <pentaho-db>: Name of the consolidated PDC application database. The default value is pentaho.
    • <pentaho-bidb>: Name of the consolidated BIDB database. The default value is pentaho_bidb.

      The values in POSTGRES_PDC_PENTAHO_DB and POSTGRES_PDC_BI_DB must match the target database names used in the Phase 2 PostgreSQL schema consolidation configuration files. If the values do not match, PDC 11.0 can start with database connection or missing data issues.
11. Start the PDC 11.0 services.

    sudo <pdc-deployment-directory>/pdc.sh up

    Example:

    sudo /opt/pentaho/pdc-docker-deployment/pdc.sh up

The PDC Docker deployment has been upgraded from 10.2.11 to 11.0, and the PDC 11.0 services have been started.

4

Restore MongoDB data to FerretDB

In this phase, you restart the mongotool container that you created in Phase 1, restore the preprocessed MongoDB dump to FerretDB, and re-create the required CollabDB index.

Perform the following steps to restore MongoDB data to FerretDB:

1. Start the stopped mongotool container in interactive mode.

   sudo docker start -ai mongotool

   The terminal connects to the mongotool container.
2. If the DUMP_DIR variable is not set, identify the MongoDB dump directory.

   ls -d mongodump_*

3. Set the dump directory variable.

   export DUMP_DIR="<mongo-dump-directory>"

   Replace <mongo-dump-directory> with the dump directory that was created and preprocessed in Phase 1.
4. Set the FerretDB connection variables.

   export FERRET_HOST="<ferretdb-host>"
   export FERRET_PORT="<ferretdb-port>"
   export FERRET_USER="<ferretdb-user>"
   export FERRET_PASS="<ferretdb-password>"

   Replace the placeholders as follows:

   • <ferretdb-host>: Host name or container name for the FerretDB service.
   • <ferretdb-port>: FerretDB service port.
   • <ferretdb-user>: FerretDB user name with permission to restore data.
   • <ferretdb-password>: Password for the FerretDB user.

     If the FerretDB password contains special characters, URL-encode the password in the connection string. For example, replace `#` with `%23`.
5. Restore the MongoDB dump to FerretDB.

   mongorestore \
    --uri="mongodb://${FERRET_USER}:${FERRET_PASS}@${FERRET_HOST}:${FERRET_PORT}/?authSource=admin&tls=true&tlsInsecure=true" \
    --dir="${DUMP_DIR}" \
    --nsExclude="admin.*" \
    --nsExclude="fe.businessglossaryterms" \
    --gzip \
    --drop

   This command restores the preprocessed MongoDB dump to FerretDB and excludes the following namespaces: Namespace: Reason

   • admin.*: The administrative database is not restored as part of the PDC data migration.
   • fe.businessglossaryterms: This collection is no longer in use and contains unsupported features.
6. Review the restore output.

   • If the restore completes successfully, continue to the next step.
   • If the restore fails for a specific collection, restore that collection individually.

     mongorestore \
       --uri="mongodb://${FERRET_USER}:${FERRET_PASS}@${FERRET_HOST}:${FERRET_PORT}/?authSource=admin&tls=true&tlsInsecure=true" \
       --nsInclude="<database>.<collection>" \
       --dir="${DUMP_DIR}" \
       --gzip \
       --drop

     Replace <database>.<collection> with the database and collection that failed to restore.

     Important: Restore collections individually only for collections that failed during the full restore. Record the names of the failed collections and review the restore output before you continue.
7. Open a FerretDB shell. Use the database shell available in the mongotool container and connect to FerretDB by using the connection details from your environment.

   Example:

   mongosh "mongodb://${FERRET_USER}:${FERRET_PASS}@${FERRET_HOST}:${FERRET_PORT}/?authSource=admin&tls=true&tlsInsecure=true"

   If your environment uses a different shell command or connection method, use the method recommended for your deployment.
8. Re-create the CollabDB index.

   use collabdb;
   
   db.messages.createIndex({
     "attributes.entityName": "text",
     "attributes.entityPath": "text"
   });

   This index is required because CollabDB index creation can fail during the restore process.
9. Exit the FerretDB shell.

   exit

The preprocessed MongoDB backup data is restored to FerretDB, and the required CollabDB index is re-created.

5

Reset lineage graph aggregation

In this phase, you connect to the PDC PostgreSQL container and re-create the graph used for lineage graph aggregation.

Perform the following steps to reset lineage graph aggregation:

1. Go to the PDC Docker deployment directory.

   cd <pdc-deployment-directory>

   Replace <pdc-deployment-directory> with the path where PDC is installed.

   In a default Docker deployment, the PDC deployment directory is /opt/pentaho/pdc-docker-deployment.
2. Connect to the PDC PostgreSQL container.

   sudo ./pdc.sh shell <postgres-service-name>

   Replace <postgres-service-name> with the PostgreSQL service name from your deployment.

   In a default Docker deployment, the PostgreSQL service name is typically um-postgresql.
3. Set the PostgreSQL connection variables.

   export POSTGRES_HOST="<postgres-host>"
   export POSTGRES_PORT="<postgres-port>"
   export POSTGRES_DATABASE="pentaho"
   export POSTGRES_USER="<postgres-user>"
   export POSTGRES_PASSWORD="<postgres-password>"

   Replace the placeholders as follows:

   • <postgres-host>: Host name or container name for the PostgreSQL service.
   • <postgres-port>: PostgreSQL service port.
   • <postgres-user>: PostgreSQL user name with permission to update the `pentaho` database.
   • <postgres-password>: Password for the PostgreSQL user.

     If the PostgreSQL password contains special characters, enclose the value in single quotation marks when setting `POSTGRES_PASSWORD`.
4. Re-create the lineage graph aggregation.

   PGPASSWORD="${POSTGRES_PASSWORD}" \
   psql \
     -h "${POSTGRES_HOST}" \
     -p "${POSTGRES_PORT}" \
     -U "${POSTGRES_USER}" \
     -d "${POSTGRES_DATABASE}" \
     -c "SELECT ag_catalog.drop_graph('lineage_graph', true); SELECT ag_catalog.create_graph('lineage_graph');"

   Important: Run this command only as part of the PDC 10.2.11 to PDC 11.0 upgrade procedure. The command drops and re-creates the lineage_graph used for lineage graph aggregation.
5. Review the command output and confirm that the graph was re-created successfully.
6. Exit the PostgreSQL container shell.

   exit

The lineage_graph AGE graph is re-created in the consolidated pentaho database. Continue to Phase 6 to re-enable MongoDB migrations and restart the PDC services.

6

Restart PDC services

In this phase, you re-enable the mongodb-migrations service and restart the PDC services. Restarting the services runs the MongoDB migrations that were disabled temporarily during the upgrade.

Perform the following steps to restart the PDC services:

1. Go to the PDC Docker deployment directory.

   cd <pdc-deployment-directory>

   Replace <pdc-deployment-directory> with the directory where PDC is installed.

   In a default Docker deployment, the directory is:

   /opt/pentaho/pdc-docker-deployment

2. Open the conf/docker-compose.yml file.

   sudo vi conf/docker-compose.yml

3. Locate the mongodb-migrations override that you added in Phase 3.

   services:
     mongodb-migrations:
       entrypoint: ["sh", "-c", "echo 'MongoDB migrations disabled by override'; exit 0"]
       command: []

4. Comment out or remove the override to re-enable mongodb-migrations.

   If the services section contains only the mongodb-migrations override, you can comment out the complete block:

   # services:
   #   mongodb-migrations:
   #     entrypoint: ["sh", "-c", "echo 'MongoDB migrations disabled by override'; exit 0"]
   #     command: []

   Important: If the services section contains configuration for other services, do not comment out the services line or any unrelated service configuration. Comment out or remove only the mongodb-migrations override.
5. Save the file.
6. Restart the PDC services.

   sudo ./pdc.sh restart

   Restarting the services allows mongodb-migrations to run with its default PDC 11.0 configuration.
7. Review the command output and confirm that the services restart without errors.

The mongodb-migrations service is re-enabled, and the PDC services are restarted. The PDC 10.2.11 to PDC 11.0 migration phases are complete.

---

Upgrade PDC 10.2.5 to 10.2.6, 10.2.7, 10.2.8, 10.2.9, or 10.2.11

Upgrading Pentaho Data Catalog (PDC) to the latest version ensures access to new features, improved performance, and security updates.

Perform the following steps to upgrade from PDC 10.2.5 to PDC 10.2.6, 10.2.7, 10.2.8, 10.2.9, or 10.2.11:

Prerequisites

Before you begin, make sure that:

• PDC 10.2.5 or higher version is installed. For the new installation of Data Catalog, see Installing Data Catalog.
• You have a backup of your configurations and data.
• You have administrative access to the server where Data Catalog is installed.

Data Catalog supports upgrades only across the next two versions. From version 10.2.5, you can upgrade directly to 10.2.6 or 10.2.7. Direct upgrade from 10.2.5 to 10.2.8, 10.2.9, or 10.2.11 is not supported. To move to 10.2.11, you must first upgrade to 10.2.8 and then perform a second upgrade to 10.2.9 or 10.2.11. For details, see the Supported upgrades section.

Procedure

1. Go to the Data Catalog installation folder, where PDC 10.2.5 or higher version is installed:

   cd /opt/pentaho/pdc-docker-deployment/

2. Stop the currently running PDC services.

   ./pdc.sh stop

3. Back up configuration files:

   1. Create a backup folder in /opt/ to store the container data and configuration backups:

      mkdir /opt/pdc_backup_10_2_5

   2. Copy the config folder to the backup folder you have created:

      cp -a /opt/pentaho/pdc-docker-deployment/conf/ /opt/pdc_backup_10_2_5

   3. Copy the vendor folder to the backup folder you have created:

      cp -a /opt/pentaho/pdc-docker-deployment/vendor/ /opt/pdc_backup_10_2_5

4. Back up Docker volumes:

   1. Identify PDC Docker volumes:

      docker volume ls | grep 'pdc*'

   2. Go to the Docker volumes path:

      cd /var/lib/docker/volumes/

   3. Copy PDC Docker volumes to the backup folder:

      cp -a pdc* /opt/pdc_backup_10_2_5/

5. Verify that all configuration files and Docker volumes are backed up correctly in the backup folder.
6. Remove the vendor folder of PDC 10.2.5 build:

   rm –rf /opt/pentaho/pdc-docker-deployment/vendor/

7. Load the new images (choose either 10.2.6, 10.2.7, 10.2.8, 10.2.9, or 10.2.11) into the Docker repository:

   docker load -i pdc-<version>-images.tgz

   Replace <version> with 10.2.6, 10.2.7, 10.2.8, 10.2.9 , or 10.2.11 depending on the upgrade target.
8. Extract the contents of the new package and overwrite the existing files in the deployment folder (/opt/pentaho/pdc-docker-deployment/):

   Important: Before overriding the files, review any custom configurations made in the Docker compose files and ensure those changes are noted and applied to the new files.

   tar -xzvf pdc-<DEPLOYMENT_PACKAGE_TYPE>-<version>-compose.tgz -C /opt/

   The <DEPLOYMENT_PACKAGE_TYPE> placeholder corresponds to the type of PDC service you want to deploy. For example, in the case of PDC 10.2.6:

   • For PDC full services, use: pdc-full-10.2.6-compose.tgz
   • For PDC with Pentaho Data Optimizer services, use: pdc-pdo-10.2.6-compose.tgz
   • For PDC with Pentaho Data Mastering services, use: pdc-pdm-10.2.6-compose.tgz .

   Similarly, for PDC 10.2.7, replace it with 10.2.7, for PDC 10.2.8, replace it with 10.2.8, for PDC 10.2.9, replace it with 10.2.9, or for PDC 10.2.11, replace it with 10.2.11. If you are unsure which deployment package to use, contact Pentaho Support for guidance.
9. Start PDC services with the new version:

   ./pdc.sh up

Result

You have successfully upgraded Pentaho Data Catalog version 10.2.5 to 10.2.6, 10.2.7, 10.2.8, 10.2.9, or 10.2.11.

Upgrade PDC 10.2.1 to 10.2.5

Upgrading Pentaho Data Catalog (PDC) to the latest version ensures access to new features, improved performance, and security updates.

Perform the following steps to upgrade from PDC 10.2.1 to PDC 10.2.5:

Prerequisites

Before you begin, make sure that:

• PDC 10.2.1 is installed. For the new installation of Data Catalog, see Installing Data Catalog.
• You have a backup of your configurations and data.
• You have administrative access to the server where Data Catalog is installed.

Procedure

1. Go to the Data Catalog installation folder, where PDC 10.2.1 is installed:

   cd /opt/pentaho/pdc-docker-deployment/

2. Stop the currently running PDC services.

   ./pdc.sh stop

3. Back up configuration files:

   1. Create a backup folder in /opt/ to store the container data and configuration backups:

      mkdir /opt/pdc_backup_10_2_1

   2. Copy the config folder to the backup folder you have created:

      cp -a /opt/pentaho/pdc-docker-deployment/conf/ /opt/ pdc_backup_10_2_1

   3. Copy the vendor folder to the backup folder you have created:

      cp -a /opt/pentaho/pdc-docker-deployment/vendor/ /opt/ pdc_backup_10_2_1

4. Back up Docker volumes:

   1. Identify PDC Docker volumes:

      docker volume ls | grep pdc*

   2. Go to the Docker volumes path:

      cd /var/lib/docker/volumes/

   3. Copy PDC Docker volumes to the backup folder:

      cp -a pdc* /opt/pdc_backup_10_2_1/

5. Verify that all configuration files and Docker volumes are backed up correctly in the backup folder.
6. Remove the vendor folder of PDC 10.2.1 build:

   rm –rf /opt/pentaho/pdc-docker-deployment/vendor

7. Load the PDC 10.2.5 images into the Docker repository:

   docker load –i pdc-10.2.5-images.tgz

8. Extract the contents of the PDC 10.2.5 package and overwrite the existing files in the deployment folder (/opt/pentaho/pdc-docker-deployment/):

   Important: Before overriding the files, review any custom configurations made in the Docker Compose files and ensure those changes are noted and applied to the new files.

   tar -xzvf pdc-<DEPLOYMENT_PACKAGE_TYPE>-10.2.5-compose.tgz -C /opt/

   The <DEPLOYMENT_PACKAGE_TYPE> placeholder corresponds to the type of PDC service you want to deploy. For example:

   • For PDC full services, use: pdc-full-10.2.5-compose.tgz
   • For PDC with Pentaho Data Optimizer services, use: pdc-pdo-10.2.5-compose.tgz
   • For PDC with Pentaho Data Mastering services, use: pdc-pdm-10.2.5-compose.tgz If you are unsure which deployment package to use, contact Pentaho Support for guidance.
9. Start PDC services with the new version:

   ./pdc.sh up

Result

You have successfully upgraded PDC version 10.2.1 to 10.2.5.

Upgrade PDC 10.2.0 to 10.2.1

Upgrading Pentaho Data Catalog (PDC) to the latest version ensures access to new features, improved performance, and security updates.

Perform the following steps to upgrade from PDC 10.2.0 to PDC 10.2.1:

Prerequisites

Before you begin, make sure you have a backup of your configurations and data, and you have administrative access to the server where Data Catalog is installed.

Procedure

1. Go to the Data Catalog installation directory, where PDC 10.2.0 is installed.

   cd /opt/pentaho/pdc-docker-deployment/

2. Stop the currently running PDC services.

   ./pdc.sh stop

3. Back up configuration files:

   1. Create a backup folder in /opt/ to store the container data and configuration backups:

      mkdir pdc_backup_10_2_0

   2. Copy the config directory to the backup folder you have created:

      cp -a /opt/pentaho/pdc-docker-deployment/conf/ /opt/pdc_backup_10_2_0/

4. Back up Docker volumes:

   1. Identify PDC Docker volumes:

      docker volume ls | grep pdc*

   2. Go to the Docker volumes directory:

      cd /var/lib/docker/volumes/

   3. Copy PDC Docker volumes to the backup folder.

      cp -a pdc* /opt/pdc_backup_10_2_0/

5. Verify that all configuration files and docker volumes are backed up correctly in the backup folder.
6. Extract the contents of the PDC 10.2.1 package and overwrite the existing files in the deployment directory (/opt/pentaho/pdc-docker-deployment/):

   Important: Before overriding the files, review any custom configurations made in the Docker-compose files and ensure those changes are noted and applied to the new files.

   tar -xzvf pdc-10.2.0.tar.gz -C /opt/

7. Open the .env file located in the /opt/pentaho/pdc-docker-deployment/conf/ path and add or update the licensing URL in a line such as the following, and save the .env file.

   LICENSING_SERVER_URL=<your_license_server_url>

8. After updating the configurations, load the Docker images for PDC 10.2.1.

   ./pdc.sh load-images

9. Start PDC services with the new version:

   ./pdc.sh up

Result

You have successfully upgraded PDC from 10.2.0 to 10.2.1.