Processing collections
In Data Catalog, you can process collections to better understand their structure, quality, and consistency. Processing helps you prepare a collection for analysis, governance, sharing, or publication as a Data Product. Data Catalog supports two main processing jobs for collections: Data Profiling and Data Aggregation. Data Profiling is available for both Datasets and Data Collections, while Data Aggregation is available only for Datasets. These jobs help generate statistical summaries and quality insights for the selected collection.
Data Profiling is supported for both Datasets and Data Collections. Data Aggregation is supported only for Datasets.
Data Profiling
The Data Profiling job analyzes the structure and content of the selected collection and generates detailed statistics, such as data types, null value percentages, distinct counts, and pattern distributions. Use Data Profiling to validate the structure and quality of data before you share the collection, apply governance metadata, or publish it as a Data Product.
You can run Data Profiling for both Datasets and Data Collections. For a Dataset, Data Catalog profiles the files or tables in the dataset and helps you evaluate the consistency and quality of data across assets that share a common schema. For a Data Collection, Data Catalog profiles the entities in the collection so that you can inspect the structure and quality of heterogeneous assets grouped for a common business purpose.
When a collection is profiled, Data Catalog evaluates the included assets and captures key metrics for their columns. The resulting profile information helps you identify inconsistencies, anomalies, and potential data issues before further analysis or publication.
For Datasets, profiling also supports the columns view for common columns across the included files or tables. This helps you review the structure and quality information for the dataset at a unified level. To learn more about running a profiling job, see Run Data Profile job. Dataset processing can also include Data Aggregation, which summarizes statistics across common columns. Datasets support both profiling and aggregation.
Data Aggregation
The Data Aggregation job computes summary statistics across the common columns in a profiled Dataset. Unlike profiling, which evaluates individual files or tables, aggregation works at the dataset level to provide a consolidated view of the data. Use Data Aggregation when you want to understand the dataset more broadly by summarizing key statistics across all included files or tables.
Data Aggregation is supported only for Datasets. Because aggregation depends on common columns across homogeneous assets, it does not apply to Data Collections, which can contain heterogeneous assets with different structures.
Run Data Aggregation after the profiling process is complete. Aggregation is useful when you need summary insights across the entire dataset, especially when you want to identify patterns, trends, outliers, or structural differences across the included files or tables.
When you run this job, Data Catalog calculates summary values such as minimum, maximum, and average for each column that is common across the dataset. In addition to numerical summaries, aggregation helps you assess structural consistency across the dataset. You can identify whether the included files or tables follow the same column definitions and formats, or whether differences need attention. Aggregation results can also highlight trends and potential outliers across the dataset.
Run Data Profiling job
Data Profiling a collection helps you evaluate the structure and quality of the selected collection and identify inconsistencies, anomalies, and potential data issues before further analysis, governance, sharing, or publication as a Data Product.
When you run a Data Profiling job on a Dataset, it automatically runs the Data Aggregation job as well. You do not need to run it separately.
You can also use the PDC APIs to trigger a Data Profiling job for a collection. After the profiling job finishes, you can also use the API to retrieve profiling information for collections. For more information, see PDC API Documentation v3.
Perform the following steps to profile a Dataset or Data Collection in Data Catalog:
In the left navigation menu, click Collections.
The Collections page opens with the list of collections.
Select a Dataset or Data Collection that you want to profile.
The summary page of the selected collection appears.
Click Actions and then click Process.
The Choose Process page appears.
Under the Data Profiling card, click Start.
Data Catalog starts the data profiling job for the selected collection. You can monitor the job status on the Workers page.
The selected collection is profiled in Data Catalog. After the Data Profile job finishes, you can review the profile results in the content pane of the profiled collection.
After profiling finishes, you can continue working with the collection by reviewing the profile results and applying metadata such as business terms and sensitivity values. For more information, see Content pane.
Run Data Aggregation job
Aggregation helps you understand the dataset at a broader level by calculating summary statistics across common columns in the included files or tables. The aggregation results help you identify patterns, trends, structural differences, and potential outliers across the dataset.
Data Aggregation is supported only for Datasets. It is not available for Data Collections because aggregation depends on common columns across homogeneous assets.
You can also use the PDC APIs to trigger a Data Aggregation job for a dataset. For more information, see PDC API Documentation v3.
Perform the following steps to run a Data Aggregation job on a Dataset in Data Catalog:
In the left navigation menu, click Collections. The Collections page opens with the list of collections.
Select a Dataset for which you want to run the Data Aggregation job.
The summary page of the selected Dataset appears.
Click Actions and then click Process.
The Choose Process page appears.
Under the Data Aggregation card, click Start.
Data Catalog starts the Data Aggregation job for the Dataset. You can monitor the job status on the Workers page.
The selected dataset is aggregated in Data Catalog. After the Data Aggregation job finishes, you can review the aggregation results at the column level and use them to better understand the overall structure and quality of the dataset. The aggregation results can appear in areas such as the Statistics section of the Summary tab.
Last updated
Was this helpful?