Verify or edit core-site XML file
Note: If you plan to run MapReduce jobs on an Amazon EMR cluster, make sure you have read, write, and execute access to the S3 Buffer directories specified in the core-site.xml
file on the EMR cluster.
You must edit the core-site.xml
file to add information about your AWS Access Key ID, your Access key, and your LZO compression setting.
Perform the following steps to edit your core-site.xml
:
Navigate to the
*<username>*/.pentaho/metastore/pentaho/NamedCluster/Configs/*<user-defined connection name>*
directory and open thecore-site.xml
file.Add the following values:
Parameter
Values
fs.s3.awsAccessKeyId
Value of your S3 AWS Access Key ID.```xml fs.s3.awsAccessKeyId [INSERT YOUR VALUE HERE]
</td></tr><tr><td>
**fs.s3.awsSecretAccessKey**
</td><td>
Value of your AWS secret access key.```xml
<property>
<name>fs.s3.awsSecretAccessKey</name>
<value>*\[INSERT YOUR VALUE HERE\]*</value>
</property>
3. If needed, enter the AWS Access Key ID and Access Key for S3N like this:
Parameter
Values
fs.s3n.awsAccessKeyId
Value of your S3N AWS Access Key ID.```xmlfs.s3n.awsAccessKeyId[INSERT YOUR VALUE HERE]
</td></tr><tr><td>
**fs.s3n.awsSecretAccessKey**
</td><td>
Value of your S3N AWS secret access key.```xml
<property>
<name>fs.s3n.awsSecretAccessKey</name>
<value>[INSERT YOUR VALUE HERE]</value>
</property>
4. Add the following values:
Parameter
Values
fs.s3n.impl
<property>
<name>fs.s3n.impl</name>
<value>org.apache.hadoop.fs.s3native.NativeS3FileSystem</value>
</property>
fs.s3.impl
<property>
<name>fs.s3.impl</name>
<value>org.apache.hadoop.fs.s3.S3FileSystem</value>
</property>
fs.s3a.impl
<property>
<name>fs.s3a.impl</name>
<value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
</property>
5. LZO is a compression format that Amazon EMR supports. If you want to configure for LZO compression, you need to download a JAR file. If you do not, you need to remove a parameter from the `core-site.xml` file.
- If you are not using LZO compression, remove any references to the **iocompression** parameter in the `core-site.xml` file: **com.hadoop.compression.lzo.LzoCodec**
- If you are not using LZO compression, download the LZO JAR and add it to `pentaho-big-data-plugin/hadoop-configurations/emr3x/lib` directory. The LZO JAR can be found here: [http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/](http://maven.twttr.com/com/hadoop/gplcompression/hadoop-lzo/0.4.19/).
6. Save and close the file.
Last updated
Was this helpful?