Data Catalog user features

This article describes the Pentaho Data Catalog user interface and the tasks non-admin users can perform. Before proceeding, make sure that a Data Catalog service user or administrator has set up your catalog for you. Data Catalog builds a complete inventory of data assets in a data warehouse, automatically and securely. It provides:

• exact data discovery and faster delivery to an authenticated user.
• improved and simplified understanding of data quality.
• an inventory of all assets for efficient data repository governance.

Data Catalog complements data visualization, data discovery, and data wrangling tools by streamlining the collection and initial data quality checks of the data repository, making the data repository available to those tools for further processing.

Data self-service

Data Catalog provides a rich interface for data self-service to help you find the best instances of the integrated data you are looking for. These services include:

Role-based access control

Data Catalog roles give users the ability to perform specific actions, especially edit actions, and allow lines of business to restrict access to sensitive or confidential information. You can create named communities, such as US Business Users or Commercial Lending Business Users to fine-tune the actions users can perform, as well as to allow access to a subset of glossaries and data sources.

Note: The number of data sources you can add is limited by your license.

Business Terms

You can discover metadata about files and fields and have Data Catalog associate fields to customer's business terms. You can associate business terms with data elements, business rules, related terms, and custom properties to form a comprehensive view of the organization’s business concepts and data landscape.

Data quality metrics and statistics

Data Catalog's profiling processes produce and present detailed data quality metrics and statistics that help you decide if the data is useful, valid, and complete, without having to write code to graph each field in the file.

User roles

User roles in Data Catalog are used for access control and permissions management. They help control who can view, edit, or delete items in the Data Catalog and ensure data security.

Note: The number of Expert user roles you can assign to users is limited by your license. The Expert user roles are:

• Business Steward
• Data Steward
• Admin
• Data Developer

Table creation

When you find a file you are interested in, you can create a table for that file. This ability is useful for non-technical users to find the files that contain the data that they need, create a table and use a tool like Grafana to visualize the data without any added efforts.

Data inventory

Behind Data Catalog’s self-service user interface is an engine that profiles the data repository and enriches it by propagating terms created by users. Data Catalog identifies the formats of the resources and profiles their contents, creating an inventory of data assets in the data warehouse securely.

Profiling

Most of the data-curation process entails writing code to profile and graph data. Data Catalog automates this process, improving the productivity of data engineers and data scientists.

Data profiling and discovery are the processes in which Data Catalog examines file data and gathers statistics about the data. It profiles data in the cluster, and uses its algorithms to compute detailed properties, including field-level data quality metrics, and data statistics. The resulting inventory includes rich metadata for delimited files, like JSON, and Parquet, and files compressed with supported compression algorithms such as gzip.

Note: The amount of data you can scan is limited by your license. Databases do not have a data scan quota.

Sensitive data discovery

Sensitive data residing in the data cluster presents a sizable liability if it is not protected and managed. The algorithms in Data Catalog identify sensitive data throughout the data clusters as a part of profiling with minimal additional processing overhead. Identification is the first step, and often the most difficult step in the process of protecting sensitive data. You cannot protect sensitive data unless you know where it resides. Data Catalog identifies sensitive data and facilitates the next step of protecting it through masking, encryption, or quarantine.

Data quality

You can discover data quality metrics automatically using large-scale profiling, such as discovering the number of nulls in a data column or cardinality. For example, you can assess the number of values that should be in a field versus the actual numbers that have been profiled.

Note: Data Catalog writes profiling process notifications to the log files.

Data governance

Data Catalog provides data governance by securing access to the data, by managing metadata creation, enrichment, and approval, and by linking physical data to business-related terminology.

Securing access to data

In Data Catalog, you can protect resources with secured access using glossaries. A glossary is a logical grouping of business terms that you can assign to a specific project user group. Once you have set up roles for glossaries, you can use them to limit access to data via specific roles and users.

Managing metadata creation, enrichment, and approval

By default, users with the Data Steward role can only associate terms with data, while users with the Business Steward role can create new metadata by adding business terms, term hierarchies, and custom properties. Users with the Business Steward role can also perform these functions.

Linking physical data to business-defined terms

In Data Catalog, with applicable permissions, you can manually tag data directly in the user interface. To learn more about this process, see the Administer Pentaho Data Catalog document.

Reporting and data visualization

Reporting in Data Catalog usually is done through dashboards using third-party BI tools. Dashboards further extend the visual discovery and relationship discovery capabilities of the Data Catalog in several ways. They also provide a means to add customized insight assets unique to the organization.

Business Intelligence Database

Data Catalog includes the Business Intelligence Database (BIDB), which stores aggregated metadata from connected sources. BIDB enables you to query, analyze, and create dashboards by connecting through standard BI tools using JDBC or ODBC.

PDC has two different implementations of BIDB depending on the version:

• PostgreSQL-based BIDB in PDC 10.2.5 and later.
• MongoDB-based BIDB in versions earlier than PDC 10.2.5

For more information on how to connect to BIDB, see Connect to Business Intelligence Database (BIDB) in the Advanced configuration section in Administer Pentaho Data Catalog guide.

BIDB in PDC 10.2.5 and later (PostgreSQL-based)

Starting with PDC 10.2.5, BIDB uses PostgreSQL as its underlying database. This simplifies connectivity because only standard PostgreSQL authentication is required. The services used in earlier versions (BIDB - MongoDB prior PDC 10.2.5), such as bi-mongo and mongo-bi-connector, are no longer part of the architecture. Instead, users connect directly to BIDB using JDBC or ODBC drivers for PostgreSQL. This streamlined setup provides easier integration while maintaining full compatibility with BI tools for reporting, data access, and dashboard creation.

The PostgreSQL-based BIDB contains a rich set of tables and views that organize aggregated metadata collected from data sources. These objects are structured into logical categories to simplify navigation and analysis. Each category focuses on a specific aspect of catalog operations, such as core entity definitions, relationships, cross-references, control and configuration, usage statistics, and analysis/monitoring.

Analysis and monitoring

The tables or views in the Analysis and monitoring category provides insights into data quality, duplication, file characteristics, and usage patterns within Pentaho Data Catalog (PDC). These views are designed to help administrators, data stewards, and analysts understand how data is distributed, identify anomalies, and monitor health at scale.

The following views and tables belong to this category:

• Checksum Aggregated View
• Duplicate Files View
• Entities Extension Count View
• Entities Temperature Count View

Checksum Aggregated View

The checksum_aggregated_view is a summarized view that reports, per entity, the number of duplicate files detected and their combined size. It’s populated from the entity-level checksums calculated during unstructured-content processing. With the checksum_aggregated_view you can:

• Quickly quantify duplicate-file impact by entity (for example, application, project, owner).
• Prioritize cleanup or tiering by targeting entities with the largest duplicate footprint.
• Feed storage dashboards or alerts to show rising duplication over time.

The following table shows the details of the data available in checksum_aggregated_view.

Field	Description	Data Type	Example Value
_id	Identifier of the entity the duplication stats are calculated for (derived from entities_master_view).	String	263481a48eb70e8e9b0cf983b0576b1c
duplicateFilesCount	Total number of duplicate files detected for the entity.	Integer	5
duplicateFilesSize	Sum of the sizes (in bytes) of all duplicate files for the entity.	Integer	9519205

• duplicateFilesSize is in bytes; convert to MB/GB when presenting to end users.
• There is one row per entity. Use the _id to join back to entity details (name, path, owner) from entities_master_view.

Duplicate Files View

The duplicate_files_view is a detailed view listing duplicate files detected in Data Catalog. Each row represents an individual duplicate file, including its unique identifier, the group of duplicates it belongs to, and file-level metadata such as size and Timestamps. The duplicate_files_view:

• Provides file-level visibility into duplicates, beyond aggregated stats.
• Allows investigation into which specific files are duplicates and how they are grouped.
• Supports actions such as cleaning up redundant files or analyzing duplicate storage consumption.
• Can be joined with entities_master_view or checksum_aggregated_view for entity-level reporting.

The following table shows the details of the data available in duplicate_files_view.

Field	Description	Data Type	Example Value
_id	Internal identifier for the duplicate file record.	String (UUID)	1
EntityId	Identifier of the entity (from entities_master_view) that the file belongs to.	String (UUID)	83d3a83c-0548-46ef-80f4-3000e18680ca
GroupId	Identifier for the duplicate group. All files with the same checksum share the same group ID.	String	263481a48eb70e8e9b0cf983b0576b1c
FileCount	Number of files in the duplicate group.	Integer	5
Size	Size of the duplicate file (in bytes).	Integer	670
CreatedAt	Timestamp when the record was created.	Timestamp	2025-02-25 10:59:50.000 +0530
ModifiedAt	Timestamp when the record was last updated.	Timestamp	2025-02-25 10:59:50.000 +0530

• Use GroupId to group files together and identify all duplicates of a given checksum.
• Combine with checksum_aggregated_view to compare group-level totals with individual file records.
• EntityId can be joined with entities_master_view to resolve details like entity name, type, and owner.

Entities Extension Count View

The entities_extension_count_view is a BIDB analysis table that summarizes how many files of each extension exist per entity. It’s built so you (or reporting tools) can quickly answer questions like “how many PDFs do we have for this application?” without rescanning raw files. The entities_extension_count_view you can:

• Spot risky or unwanted types (for example, too many .exe, .bat, .js).
• Prioritize cleanup or archiving based on the mix of extensions.
• Track normalization efforts (for example, converting many small .xls to .xlsx/.csv).
• Feed dashboards with extension distribution by entity, application, or business area.

The following table shows the details of the data available in duplicate_files_view.

Field	Description	Data Type	Example Value
entity_id	The PDC entity identifier this row summarizes (dataset/folder/report, and so on).	UUID (String in some tools)	83d3a83c-0548-46ef-80f4-3000e18680ca
extension	Normalized file extension (lowercase, no leading dot).	String	pdf
file_count	Count of files of this extension within the entity.	Integer	127
total_size_bytes*	Sum of sizes of those files (if available in your deployment).	Integer	15432987
created_at*	When this summary row was first created.	Timestamp with time zone	2025-02-25 10:59:48+05:30
updated_at*	When this summary row was last refreshed.	Timestamp with time zone	2025-03-06 15:44:46+05:30

* Optional columns, present in many installations, but not strictly required for using the view.

Entities Temperature Count View

The entities_temperature_count_view is a summary view that counts the number of files grouped by their temperature classification (for example, hot, warm, cold) for each entity. Temperature represents data access frequency or recency; hot files are frequently accessed, while cold files are rarely used. The entities_temperature_count_view:

• Gives visibility into how active or dormant data is within each entity.
• Helps prioritize storage optimization: cold data can be archived or tiered to cheaper storage.
• Supports governance and lifecycle management by quantifying “stale” vs. “active” content.
• Is useful for dashboards showing entity-level data temperature distribution.

The following table shows the details of the data available in entities_temperature_count_view.

Field	Description	Data Type	Example Value
entity_id	Identifier of the entity this temperature count belongs to.	UUID	83d3a83c-0548-46ef-80f4-3000e18680ca
temperature	Data temperature classification (for example, hot, warm, cold).	String	cold
file_count	Number of files in the entity that fall into this temperature.	Integer	1572
created_at*	When this summary row was created.	Timestamp with time zone	2025-02-25 10:59:48+05:30
updated_at*	When this summary row was last updated.	Timestamp with time zone	2025-03-06 15:44:46+05:30

^*Optional columns depending on your deployment.

Control and configuration

The tables or views in the Control and configuration category, manage system-level settings, reference mappings, and classification rules within Data Catalog. These objects provide supporting metadata that helps standardize costs, organize data sources, and enable user-defined categorizations.

The following views and tables belong to this category:

• Currency Exchange Rates
• Datasource Category Mapping
• Entities Custom Categorization

Currency Exchange Rates

The currency_exchange_rates is a reference table in BIDB that stores currency exchange rates used for normalizing costs, sizes, or policy values to a common standard (USD). This allows consistent reporting across entities or applications regardless of local currency usage. The currency_exchange_rates:

• Provides a single source of truth for currency conversions.
• Enables dashboards to show monetary values in a consistent currency.
• Supports analysis where datasets or policies originate from multi-currency environments.
• Helps with global governance, chargeback, or cost-allocation use cases.

The following table shows the details of the data available in currency_exchange_rates.

Field	Description	Data Type	Example Value
currency_symbol	Currency symbol or code identifying the currency.	String	$, €, ¥
ConversionRateToUSD	Conversion multiplier to USD. Always relative to 1 USD = 1.	Float	1, 1.08, 0.0064

Data Source Category Mapping

The datasource_category_mapping is a BIDB configuration table that maps each data source type (for example, FS, SMB, ORACLE) to a logical category. This categorization enables Data Catalog to organize diverse data sources into higher-level groups, such as File Servers, Databases, HDFS, and others. The datasource_category_mapping:

• Simplifies filtering and reporting by grouping similar data source types together.
• Supports governance and lineage views by showing data source categories instead of raw connection types.
• Enables dashboards and policy rules to apply at a category level (for example, treat all File Servers the same regardless of FS, SMB, or NFS).
• Provides flexibility to add or extend categories when integrating new technologies.

The following table shows the details of the data available in datasource_category_mapping.

Field	Description	Data Type	Example Value
DataSourceType	Internal identifier of the specific data source type.	String	FS, SMB, NFS, ORACLE
category	Logical grouping of the data source type.	String	File Servers (On-Prem / Cloud), Databases (On-Prem / Cloud)

Entities Custom Categorization

The entities_custom_categorization is a reference (master) table that lets you define your own high‑level categories for entities (for example, Temperature, PII) and map each category to one or more Business Glossaries or Glossary Categories by name. The entities_custom_categorization:

• Drives consistent, business‑friendly grouping in dashboards and views (for example, “Temperature terms across glossaries”).
• Enables lightweight governance, admins can steer which glossaries (or sub‑categories) roll up into organizational buckets like PII or Financials.
• Keeps the mapping externalized, so you can update categorizations without touching source entities.

The following table shows the details of the data available in entities_custom_categorization.

Field	Description	Data type	Example value
_id	Row identifier for the mapping record. Can be natural (e.g., “ec001”).	String	ec001
EntityCategory	Your business bucket that entities/terms should roll up to.	String	Temperature
GlossaryName	Name of the glossary or glossary category that participates in the category.	String	Business, Sensitive Data, Temperature Hierarchy
GlossaryType	What GlossaryName refers to: glossary (a whole glossary) or category (a category under a glossary).	String	glossary, category

• Keep GlossaryType accurate (glossary vs category) to avoid over/under‑matching.
• Treat _id as a stable key so downstream references don’t break when names change.
• Consider a unique constraint on (EntityCategory,GlossaryName,GlossaryType) to prevent duplicates.

Core entity tables

The tables or views in the Core entity tables category, are the foundational views that store and summarize metadata for all entities ingested into Pentaho Data Catalog (PDC). These tables provide the central reference point for entity definitions, attributes, statistics, and aggregations across structured and unstructured data sources.

The following tables and views belong to this category:

• Entities Aggregated View
• Entities Master View
• Entities Summary View

Entities Aggregated View

The entities_aggregated_view is a summary view in BIDB that provides aggregated statistics for both structured and unstructured entities. Instead of showing row-level detail, it consolidates information such as counts of objects, data sources, files, file formats, and size statistics. This view is designed to power dashboards and high-level reporting without repeatedly scanning raw entity data. With the entities_aggregated_view you can:

• Quickly compare structured vs. unstructured data volumes.
• Understand storage footprint (average and maximum resource sizes).
• Track discovery metrics (tables, parent paths, data sources, collections).
• Feed governance reports with overall trends instead of record-level data.
• Identify changes over time by monitoring “last modified” metadata.

The following table shows the details of the data available in entities_aggregated_view.

Field	Description	Data Type	Example Value
Attribute	The type of metric being tracked (for example, Object Ids, Files, DataSources).	String	Files
Type	Indicates whether the metric applies to Structured or Unstructured data.	String	Structured / Unstructured
Value	The aggregated value for the attribute and type.	Alphanumeric	4920, 1665124, "2025-08-19T12:09:14.000+00:00"

Following are the some of the common attributes present in the entities_aggregated_view:

• Object Ids: Count of unique object identifiers.
• DataSources: Number of connected data sources contributing entities.
• Parent Paths: Number of parent paths (directories, schemas).
• Tables: Number of structured database tables.
• Files: Count of files discovered.
• File Extensions: Number of unique file extensions.
• File Formats: Number of unique file formats.
• Discovered Collections: Number of PDC collections discovered automatically.
• Average Resource Size: Average size of resources within category.
• Maximum Resource Size: Maximum size observed for a resource.
• Last Created: Timestamp of last created unstructured resource.
• Last Modified: Timestamp of last modification for unstructured resource.
• Last Accessed: Timestamp of last access (if tracked).

Because values in this view are aggregated, they are best for dashboards and summaries. For detailed drill-down (e.g., which files or entities are biggest), join with entities_master_view or other detailed tables.

Entities Master View

The entities_master_view is the primary entity table in BIDB that stores comprehensive metadata about every discovered entity in PDC, including files, tables, schemas, and unstructured resources. It consolidates identifiers, data source info, profiling details, lineage-related attributes, and system metadata (size, Timestamps, ownership). This view is the foundation for all reporting, profiling, and lineage features in Data Catalog. The entities_master_view:

• Provides a single source of truth for all discovered entities across structured and unstructured data.
• Enables detailed profiling queries (row counts, null counts, min/max/avg values, etc.).
• Facilitates lineage and governance tracking with attributes like Parent, ParentPath, DataSourceId, and FQDN.
• Supports search, categorization, and monitoring in dashboards.
• Allows users to drill down from aggregated views (entities_summary_view, entities_aggregated_view) to entity-level detail.

The following table shows the details of the data available in entities_master_view.

Field Name	Description	Data Type	Example Value
_id	Unique identifier of the entity	UUID	e34d9ffb-5095-49c0-8545-f9d14c14c7d4
Name	Name of the entity (file, table, object)	String	Test_1MB_12659.dat
Type	Type of entity (FILE, TABLE, VIEW, etc.)	String	FILE
Parent	Parent entity identifier	UUID	af8064fa-af85-43fe-baf3-5fee11590301
ResourceType	Whether entity is Structured or Unstructured	String	Unstructured
DataSourceId	ID of the data source	UUID	68a567f3010fdaaed8384df4
DataSourceName	Name of the data source	String	AWS_S3_1
DataSourceType	Type of source system	String	AWS
DataSourceCostPerTbCurrency	Currency for cost tracking	String	$
DataSourceCostPerTbPrice	Cost per TB in source	Integer	0
DataSourceAffinityId	Affinity group for source	String	DEFAULT
DataProfileStatus	Status of profiling	String	Completed
DataProfiled	Indicates if profiling is done	Boolean	FALSE
LastUpdate	Last update Timestamp	Timestamp	2025-08-20 11:54:20.533 +0530
ProductName	Product associated	String	PDC
ProductVersion	Version of product	String	10.2
DriverName	Driver used	String	com.mysql.jdbc.Driver
Url	Connection URL	String	migration_020125191846
ParentName	Name of parent	String	RetailDB
TotalTables	Total tables (for DB entities)	Integer	5
TotalColumns	Total columns	Integer	23
SchemaName	Schema name (if structured)	String	COMMKTG
DatabaseName	Database name (if structured)	String	Demo_DB
LastUpdateStatistics	Last time stats updated	Timestamp	2025-08-20 11:54:20.533 +0530
RowCount	Number of rows (structured)	BigInteger	116
NullCount	Null value count	BigInteger	3
Cardinality	Distinct values count	BigInteger	0
Hll	HyperLogLog cardinality estimate	BigInteger	0
BlankCount	Count of blanks	BigInteger	5
Min	Minimum value (numeric/date)	String	24
Max	Maximum value	String	1145
AvgValue	Average value	NUMERIC	125
MinWidth	Minimum width (for strings)	Integer	13
MaxWidth	Maximum width	Integer	56
AvgWidth	Average width	Integer	34
ColumnsCount	Number of columns	Integer	12
CheckClause	Constraint details	String	CHECK (Age >= 18)
TableName	Table name (if structured)	String	DIM_CUSTOMER
DataType	Data type of column	String	String
TypeName	Database column type	String	nvarchar
ColumnSize	Size of column	Integer	11
BufferLength	Buffer length	Integer	18
DecimalDigits	Decimal precision	Integer	2
NumPrecRadix	Precision radix	Integer	0
IsNullable	Whether column accepts null	Boolean	FALSE
OrdinalPosition	Column position	Integer	-1
IsPrimaryKey	If column is primary key	Boolean	FALSE
IsForeignKey	If column is foreign key	Boolean	FALSE
Path	Path of entity (filesystem/db)	String	data-service-test/.../Test_1MB_12659.dat
ParentPath	Parent path	String	data-service-test/pentaho_migration/migration_020125191846
PathType	Path type (FILE/FOLDER)	String	FILE
FileExtension	File extension	String	dat
Size	File size	BigInteger	1064000
Flags	Flags if any	Integer	1
Owner	File/DB owner	String	dbo
Group	File group	String	finance-team
SymLinkTarget	Symlink target	String	/mnt/shared/finance/customers.csv
FileType	File type	String	dat
CreatedAt	Creation Timestamp	Timestamp	2025-01-03 00:49:02.000 +0530
ModifiedAt	Last modified Timestamp	Timestamp	2025-08-20 11:45:33.527 +0530
AccessedAt	Last accessed Timestamp	Timestamp	2025-08-21 11:45:33.527 +0530
ScannedAt	Time scanned by PDC	Timestamp	2025-08-20 11:45:33.527 +0530
IsSymlink	Whether entity is symlink	Boolean	TRUE
LinkType	Type of symlink	String	Soft Link
PhysicalLocation	Physical location path	String	/data/warehouse/customers.parquet
Title	Document title	String	Customer Profile Report
Author	Document author	String	John Doe
Subject	Document subject	String	Sutomer Segmentation
Application	Source application	String	Salesforce
Producer	Producer application	String	Informetica
Version	Document version	String	v2.1
DocumentSize	Document size	BigInteger	1245
PageSize	Page size	Integer	14
PageCount	Number of pages	Integer	15
Company	Company metadata	String	Company
Paragraphs	Count of paragraphs	Integer	78
Lines	Line count	Integer	178
Words	Word count	Integer	1567
Characters	Character count	Integer	16754
CharactersWithSpaces	Characters including spaces	Integer	1345
Language	Language detected	String	US
Checksum	Data checksum	String	f5a8d7e6c2b1a3d4
PropertiesChecksum	Property checksum	String	9c3b7d8a5f6e1c2d
ChildDirs	Number of child dirs	Integer	8
ChildFiles	Number of child files	Integer	24
ChildDirSize	Child directory size	BigInteger	1567
ChildFileSize	Child file size	BigInteger	1569
TotalChildDirs	Total child directories	Integer	134
TotalChildFiles	Total child files	Integer	1897
TotalChildDirSize	Total child dir size	BigInteger	1467
TotalChildFileSize	Total child file size	BigInteger	1897
LocationName	Location name	String	US
LocationStreetAddress	Street address	String	street address 1
LocationStreetAddress2