Verify or edit core-site XML file

Note: If you plan to run MapReduce jobs on an Amazon EMR cluster, make sure you have read, write, and execute access to the S3 Buffer directories specified in the core-site.xml file on the EMR cluster.

You must edit the core-site.xml file to add information about your AWS Access Key ID, your Access key, and your LZO compression setting.

Perform the following steps to edit your core-site.xml:

  1. Navigate to the *<username>*/.pentaho/metastore/pentaho/NamedCluster/Configs/*<user-defined connection name>* directory and open the core-site.xml file.

  2. Add the following values:

Parameter

Values

fs.s3.awsAccessKeyId

​Value of your S3 AWS Access Key ID.```xml fs.s3.awsAccessKeyId [INSERT YOUR VALUE HERE]


</td></tr><tr><td>

**fs.s3.awsSecretAccessKey**

</td><td>

Value of your AWS secret access key.```xml
<property>   
   <name>fs.s3.awsSecretAccessKey</name>   
   <value>*\[INSERT YOUR VALUE HERE\]*</value>
</property>

3. If needed, enter the AWS Access Key ID and Access Key for S3N like this:

Parameter

Values

fs.s3n.awsAccessKeyId

Value of your S3N AWS Access Key ID.```xmlfs.s3n.awsAccessKeyId[INSERT YOUR VALUE HERE]


</td></tr><tr><td>

**fs.s3n.awsSecretAccessKey**

</td><td>

Value of your S3N AWS secret access key.```xml
<property>
   <name>fs.s3n.awsSecretAccessKey</name>
   <value>[INSERT YOUR VALUE HERE]</value>
</property>

4. Add the following values:

Parameter

Values

fs.s3n.impl

<property>
   <name>fs.s3n.impl</name>
   <value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>

fs.s3.impl

<property>
   <name>fs.s3.impl</name>
   <value>org.apache.hadoop.fs.s3.S3FileSystem</value>
</property>

fs.s3a.impl

<property>
   <name>fs.s3a.impl</name>
   <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>

5. LZO is a compression format that Amazon EMR supports. If you want to configure for LZO compression, you need to download a JAR file. If you do not, you need to remove a parameter from the `core-site.xml` file.

-   If you are not using LZO compression, remove any references to the **iocompression** parameter in the `core-site.xml` file: **com.hadoop.compression.lzo.LzoCodec**
-   If you are not using LZO compression, download the LZO JAR and add it to `pentaho-big-data-plugin/hadoop-configurations/emr3x/lib` directory. The LZO JAR can be found here: [http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/](http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/).

6. Save and close the file.

Last updated

Was this helpful?