Data Optimizer configuration parameters
The Data Optimizer management interface distributes the configuration information to the Data Optimizer volume nodes for use by the Data Optimizer volume service.
CAUTION:
Never modify the BUCKET and MOUNT_POINT parameters in the Data Optimizer configuration file after the initial installation. Changing these values after installation breaks the instance because the Data Optimizer instance ID is calculated based on the values provided in these parameters.
Note: Do not include leading or trailing spaces if you copy and paste parameter values. Ambari and Cloudera Manager do not validate input.
Parameter
Requirement
Description
ENDPOINT
Required
Endpoint address for Hitachi Content Platform. If the ENDPOINT_TYPE is HCP, use the form tenant.hcp_dns_name
.
ENDPOINT_TYPE
Optional
Default endpoint type. Acceptable values are case sensitive. - If connecting to Hitachi Content Platform, use HCP.
If connecting to Virtual Storage Platform One Object, use HCPCS.
If connecting to Amazon S3, use AWS.
BUCKET
Required
Content Platform bucket name or a wildcard value of instance_id. You can use the unique ID generated by Content Platform (instance_id) as a wildcard to avoid name conflicts and to simplify configuration of the instances. Multiple instances can share a common configuration if you use the instance_id wildcard and all other values are identical. You cannot append or prepend the instance_id wildcard value to any other value. For example, bucket_instance_id is an invalid value. If Content Platform is properly configured, Data Optimizer creates its own bucket if the bucket does not already exist.
ACCESS_KEY
Required
S3 Access Key ID used to authenticate S3 requests to Content Platform.
SECRET_KEY
Required
S3 Secret Key used to authenticate S3 requests.
PROTOCOL
Optional
Protocol used to encrypt communication between Data Optimizer and Content Platform using TLS. The default value is https. Acceptable, case sensitive values are https and http.
VERIFY_SSL_CERTIFICATE
Optional
Value used to specify whether to verify certificates within Data Optimizer. Acceptable, case sensitive values are true and false. The default is value is true. If the VERIFY_SSL_CERTIFICATE parameter is set to false, certificate verification is disabled within Data Optimizer. Set this parameter to false when Content Platform is presenting a self-signed certificate, and you still want to use TLS to encrypt transmissions between Data Optimizer and Content Platform.
MOUNT_POINT
Required
HDFS DataNode local directory where Data Optimizer is mounted. The directory must exist and the HDFS user using Data Optimizer must have write permission for the directory. The directory must allow rwx
permissions for the owner and owner’s group. For example:```mkdir
MOUNT_POINT*<mount point>*chown
user:group MOUNT_POINT*<mount point>*chmod
770 MOUNT_POINT*<mount point>*
</td></tr><tr><td>
**BUCKET\_STORAGE\_LIMIT\_GB**
</td><td>
Required
</td><td>
Size in GB to report as the total capacity of the volume. **CAUTION:**
If the usage exceeds the quota, or upper limit, on the volume’s Content Platform bucket, writes to the volume fail. Data Optimizer does not prevent writing to the volume if the usage exceeds the capacity.
As a best practice, specify a value that is less than the bucket quota, so that HDFS stops choosing the volume for writes before the volume exceeds its quota on Content Platform.
</td></tr><tr><td>
**CACHE\_DIR**
</td><td>
Required
</td><td>
Directory that Data Optimizer uses to store temporary files associated with open file handles. If **MD\_STORE\_DIR** is not specified, Data Optimizer also uses this directory to store files associated with persisting the local metadata store. The directory must exist and the HDFS user using Data Optimizer must have write permission for the directory. The directory must allow `rwx` permissions for the owner and owner’s group. The **CACHE DIR** parameter must be a fully-qualified directory path starting at the system root \(`/`\). For example: ```
mkdir **CACHE\_DIR** *cache dir*
chown *user*:*group* **CACHE\_DIR** *cache dir*
chmod 770 **CACHE\_DIR** *cache dir*
MD_STORE_DIR
Optional
Local directory used to store files associated with persisting the Data Optimizer local metadata store. The MD_STORE_DIR parameter value must be a fully-qualified directory path starting at the system root (/
). If an MD_STORE_DIR value is not specified, the CACHE_DIR directory is used. Specify a value for MD_STORE_DIR when the CACHE_DIR directory is located is on volatile storage or if there is a more durable location for long term file persistence. Do not choose a volatile storage medium for this directory as it is intended to persist for the life of the Data Optimizer volume. For example, if you use transient storage for the CACHE_DIR directory such as RAM_DISK
, you should specify a more durable location for the MD_STORE_DIR directory. In addition, if you have a more durable location, such as a RAID partition, and there is room for the metadata store files (up to 2.5 GB), you should specify a MD_STORE_DIR directory on that partition. If the files associated with metadata store persistence are lost or corrupted, you can recover them as explained in Recovering from local metadata store failure or corruption.
RECOVERY_MODE
Optional
Value used to specify whether recovery mode is enabled. Do not set the RECOVERY_MODE parameter unless you have read and understood the section Recovering from local metadata store failure or corruption. The default value is false. Acceptable, case sensitive values are true and false.
LOG_LEVEL
Optional
Value used to specify how verbose the logging is for Data Optimizer. The default value is INFO. Acceptable, case-sensitive values are ALERT, ERR, WARNING, INFO, and DEBUG. See Data Optimizer logging for more details about logging and log levels.
METRICS_FILE
Optional
Local file that Data Optimizer writes metrics to when prompted by the ldoctl metrics collect
command. The METRICS_FILEvalue must be a fully-qualified file path starting at the system root (/
). If a METRICS_FILE value is not defined, Data Optimizer writes metrics to the system journal. The parent directory must exist and the HDFS user using Data Optimizer must have write permission for the directory. See Monitor Data Optimizer for more information.
LOG_SDK
Optional
Local directory where detailed AWS S3 logs are saved. If the LOG_SDK parameter is specified and if LOG_LEVEL is set to DEBUG, Data Optimizer volumes log details about the S3 communication between the Data Optimizer instance and Content Platform. The directory must exist, must be a fully-qualified directory path starting at the system root (/
), and the HDFS user using Data Optimizer must have write permission for the directory. See AWS S3 SDK logging for more information.
**Note:** The configuration file is located in the `/etc/ldo` directory on each HDFS DataNode on which both the Data Optimizer is installed, and the **ARCHIVE** volumes are configured.
Last updated
Was this helpful?