> For the complete documentation index, see [llms.txt](https://docs.pentaho.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview/parquet-input/select-an-engine-parquet-input/using-parquet-input-on-spark-engine/general-parquet-input-spark/fields-parquet-input-spark-engine/using-get-fields-with-parquet-partitioned-datasets.md).

# Using Get Fields with Parquet partitioned datasets

This section explains how to use **Get Fields** and partitioned Parquet files in a Parquet Input step running under AEL-Spark.

When partitioning by column is used with Parquet in a Hadoop cluster, the data is stored in the file system in a structure where additional sub-directories hold the Parquet files with data. The field used as the partitioning column, along with its corresponding values, is used as the sub-directory name and is not actually stored within the Parquet file.

For example, if you had a Parquet dataset named `/tmp/sales_parquet` that is partitioned by a field called `year`, the directory structure looks like this:

`/tmp/sales.parquet/year=2019`

`/tmp/sales.parquet/year=2020`

The Parquet files with the year data are stored inside these "`year=`" sub-directories. Since the directory name already contains the year field and its value, this data is not stored within each Parquet file. Because **Get Fields** reads an actual Parquet file and not a Parquet Hadoop directory structure using this partitioning convention, **Get Fields** cannot parse the data, in this case, a year value, that is contained in the partitioned sub-directories.

If you are using Parque with partitioned datasets, use one of the following methods to add fields to the table instead of **Get Fields**.

* Manually edit the XML in the `.ktr` file using any text editor and add the partitioned fields.
* Use **Get Fields** to read a different, temporary Parquet file with the same schema and fields, but without the partitioning. After the fields are added to the table using this temporary file, change the file path to the target dataset. You can generate this non-partitioned Parquet file by using the Spark Shell with the code snippet `spark.read.parquet("*/tmp/dataset.parquet*").limit(1).coalesce(1).write.parquet("/tmp/dataset_unpartitioned.parquet")` where `*/tmp/dataset.parquet*` is your partitioned dataset.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://docs.pentaho.com/pdia-data-integration/9.3-data-integration/pdi-transformation-steps-reference-overview/parquet-input/select-an-engine-parquet-input/using-parquet-input-on-spark-engine/general-parquet-input-spark/fields-parquet-input-spark-engine/using-get-fields-with-parquet-partitioned-datasets.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.