> For the complete documentation index, see [llms.txt](https://docs.pentaho.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pentaho.com/pdc-use/pdc-collections/pdc-processing-collections.md).

# Processing collections

In Data Catalog, you can process collections to better understand their structure, quality, and consistency. Processing helps you prepare a collection for analysis, governance, sharing, or publication as a Data Product. Data Catalog supports two main processing jobs for collections: **Data Profiling** and **Data Aggregation**. **Data Profiling** is available for both **Datasets** and **Data Collections**, while **Data Aggregation** is available only for **Datasets**. These jobs help generate statistical summaries and quality insights for the selected collection.

{% hint style="info" %}
**Data Profiling** is supported for both **Datasets** and **Data Collections**. **Data Aggregation** is supported only for **Datasets**.
{% endhint %}

### Data Profiling

The **Data Profiling** job analyzes the structure and content of the selected collection and generates detailed statistics, such as data types, null value percentages, distinct counts, and pattern distributions. Use **Data Profiling** to validate the structure and quality of data before you share the collection, apply governance metadata, or publish it as a Data Product.

You can run Data Profiling for both Datasets and Data Collections. For a Dataset, Data Catalog profiles the files or tables in the dataset and helps you evaluate the consistency and quality of data across assets that share a common schema. For a Data Collection, Data Catalog profiles the entities in the collection so that you can inspect the structure and quality of heterogeneous assets grouped for a common business purpose.

When a collection is profiled, Data Catalog evaluates the included assets and captures key metrics for their columns. The resulting profile information helps you identify inconsistencies, anomalies, and potential data issues before further analysis or publication.

For Datasets, profiling also supports the columns view for common columns across the included files or tables. This helps you review the structure and quality information for the dataset at a unified level. To learn more about running a profiling job, see [Run Data Profile job](https://hv-eng.atlassian.net/wiki/spaces/PDC/pages/edit-v2/33912029233#Run-Data-Profile-job). Dataset processing can also include [Data Aggregation](https://hv-eng.atlassian.net/wiki/spaces/PDC/pages/edit-v2/33912029233#Data-Aggregation), which summarizes statistics across common columns. Datasets support both profiling and aggregation.

### **Data Aggregation**

The **Data Aggregation** job computes summary statistics across the common columns in a profiled **Dataset**. Unlike profiling, which evaluates individual files or tables, aggregation works at the dataset level to provide a consolidated view of the data. Use **Data Aggregation** when you want to understand the dataset more broadly by summarizing key statistics across all included files or tables.

{% hint style="info" %}
**Data Aggregation** is supported only for **Datasets**. Because aggregation depends on common columns across homogeneous assets, it does not apply to **Data Collections**, which can contain heterogeneous assets with different structures.
{% endhint %}

Run **Data Aggregation** after the profiling process is complete. Aggregation is useful when you need summary insights across the entire dataset, especially when you want to identify patterns, trends, outliers, or structural differences across the included files or tables.

When you run this job, Data Catalog calculates summary values such as minimum, maximum, and average for each column that is common across the dataset. In addition to numerical summaries, aggregation helps you assess structural consistency across the dataset. You can identify whether the included files or tables follow the same column definitions and formats, or whether differences need attention. Aggregation results can also highlight trends and potential outliers across the dataset.

### Run Data Profiling job

Data Profiling a collection helps you evaluate the structure and quality of the selected collection and identify inconsistencies, anomalies, and potential data issues before further analysis, governance, sharing, or publication as a Data Product.

{% hint style="info" %}

* When you run a **Data Profiling** job on a Dataset, it automatically runs the **Data Aggregation** job as well. You do not need to run it separately.
* You can also use the PDC APIs to trigger a **Data Profiling** job for a collection. After the profiling job finishes, you can also use the API to retrieve profiling information for collections. For more information, see [PDC API Documentation v3](https://docs.pentaho.com/pdc-api-docs).
  {% endhint %}

Perform the following steps to profile a Dataset or Data Collection in Data Catalog:

1. In the left navigation menu, click **Collections**.

   The **Collections** page opens with the list of collections.
2. Select a Dataset or Data Collection that you want to profile.

   The summary page of the selected collection appears.
3. Click **Actions** and then click **Process**.

   The **Choose Process** page appears.
4. Under the **Data Profiling** card, click **Start**.

   Data Catalog starts the data profiling job for the selected collection. You can monitor the job status on the **Workers** page.

The selected collection is profiled in Data Catalog. After the **Data Profile** job finishes, you can review the profile results in the content pane of the profiled collection.

After profiling finishes, you can continue working with the collection by reviewing the profile results and applying metadata such as business terms and sensitivity values. For more information, see [Content pane](https://hv-eng.atlassian.net/wiki/spaces/PDC/pages/33560428549/Collections+-+Content+pane#Content-pane).

### Run Data Aggregation job

Aggregation helps you understand the dataset at a broader level by calculating summary statistics across common columns in the included files or tables. The aggregation results help you identify patterns, trends, structural differences, and potential outliers across the dataset.

{% hint style="info" %}

* **Data Aggregation** is supported only for **Datasets**. It is not available for **Data Collections** because aggregation depends on common columns across homogeneous assets.
* You can also use the PDC APIs to trigger a **Data Aggregation** job for a dataset. For more information, see [PDC API Documentation v3](https://docs.pentaho.com/pdc-api-docs).
  {% endhint %}

Perform the following steps to run a **Data Aggregation** job on a **Dataset** in Data Catalog:

1. In the left navigation menu, click **Collections**.\
   The Collections page opens with the list of collections.
2. Select a Dataset for which you want to run the Data Aggregation job.

   The summary page of the selected Dataset appears.
3. Click **Actions** and then click **Process**.

   The **Choose Process** page appears.
4. Under the **Data Aggregation** card, click **Start**.

   Data Catalog starts the **Data Aggregation** job for the Dataset. You can monitor the job status on the **Workers** page.

The selected dataset is aggregated in Data Catalog. After the **Data Aggregation** job finishes, you can review the aggregation results at the column level and use them to better understand the overall structure and quality of the dataset. The aggregation results can appear in areas such as the **Statistics** section of the [**Summary** tab](https://hv-eng.atlassian.net/wiki/spaces/PDC/pages/33560428549/Collections+-+Content+pane#Summary-tab).


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.pentaho.com/pdc-use/pdc-collections/pdc-processing-collections.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.