Edit Mapred site XML file
If you are using MapReduce, edit the mapred-site.xml file to indicate where the job history logs are stored and to allow MapReduce jobs to run across platforms.
Perform the following steps to edit the mapred-site.xml file:
Navigate to the location where you have saved the previously edited site configuration files and open the
mapred-site.xmlfile.Verify that the mapreduce.jobhistory.address and mapreduce.job.hdfs-servers properties are in the
mapred-site.xmlfile. If they are not in the file, you can add them as follows:
Parameter
Value
mapreduce.jobhistory.address
Set this property to the place where job history logs are stored, as shown in the following example:```xmlmapreduce.jobhistory.address<active node name in the cluster>:10020
</td></tr><tr><td>
mapreduce.job.hdfs-servers
</td><td>
Add this property for YARN:
<property> <name>mapreduce.job.hdfs-servers</name> <value>hdfs://*&lt;active node name in the cluster&gt;*:8020</value></property></code></pre></td></tr></tbody></table>3\. Optionally, to allow YARN containers to launch on JDK11.x nodes, add the \*\*mapreduce.jvm.add-opens-as-default\*\* property to the \`mapred-site.xml\` file, as shown below. All MapReduce jobs require this property to be added in \`mapred-site.xml\` to run successfully on JDK11.x machines as JAVA 11 does not require \*\*ADD\\\_OPENS\*\* JVM arguments by default.
Note: Do not add this property to containers using JDK17.x nodes.
<table data-header-hidden><thead><tr><th></th><th></th></tr></thead><tbody><tr><td>Parameter</td><td>Value</td></tr><tr><td><strong>mapreduce.jvm.add-opens-as-default</strong></td><td><p>Add this property for YARN to launch on JDK11.x nodes:```xmlmapreduce.jvm.add-opens-as-default false</p><pre><code></td></tr></tbody></table>4. Save and close the file.</code></pre></td></tr></tbody></table>Note: You will have to edit the named connection and upload the edited site configuration file in order for this change to take effect.
Last updated
Was this helpful?