4. Configuring Services After Deployment

Configure Ambari

Setting up a Multiple-Node Cluster

Use the same OVA image to create all nodes for a multiple-node cluster, for example 5-nodes. On each node, perform the step Changing the Initial Password and Network Configuration.

Choose one node as Ambari Server. On this node, run the script:

/root/setup/ambari_setup.sh

This will prompt you to set the admin password. This will be the password of the Ambari admin. You can use these credentials to access the Ambari UI.

When this script finishes, it shows you the private key that has been set up for the node. Please save the key. You will need it for setting up the Ambari server private key.

You can always view the private key later by issuing the cat .ssh/id_rsa command, as illustrated in the example below:

On all other nodes, run the script:

/root/setup/node_setup.sh

This will prompt you for the public key of the Ambari server. Go to the Ambari server to retrieve the public key. This can be any of the two files:

• /root/.ssh/authorized_keys

• /root/.ssh/id_rsa.pub

Hadoop Services Installation Through Ambari

The Ambari server is configured to use TLS (https) at port 443 and using a self-signed certificate.

1. Navigate to Ambari GUI by using the hostname configured in the Set the Hostname. Log in as admin with the password that you set in the step Configure Ambari.

   Tip

   If you have troubles logging in, please refer to Accessing Ambari for further details.

2. Click LAUNCH INSTALL WIZARD.

0. Name the Cluster

Choose a name that you consider more appropriate and descriptive for your cluster:

1. Select Version

At this point, you should see the UI in this state:

In the Repositories screen, make sure that you select the Use Local Repository option and fill in the Base URL fields per the screenshot above. The only needed OS in the Ambari configuration is Redhat 7, with the string file:///var/repo/hdp as value for the three paths. Those fields should be already filled in, if you can see any other OS, you can remove it.

1. Click NEXT.

2. Install Options

In this screen the hosts are configured. Put the hostnames and private keys of all nodes that will be on your cluster. The private key is the one that you saved in the step Configure Ambari.

1. Click REGISTER AND CONFIRM.

3. Confirm Hosts

The installation will start for all nodes. This can take a few minutes.

1. Click NEXT → whenever the button gets enabled.

4. Choose Services

Here, we are going to select the services that will be available on our virtual machine. These are the required services for DDC:

• HDFS

• Yarn + MapReduce2

• HBase

• Spark

• Hive

• Tez

• Zookeeper

• Ambari Metrics

• Knox

In the list shown in the GUI, we only need to select those. In the next step, we'll add PQS and Livy to the cluster.

1. Click NEXT →.

5. Assign Masters

All the services will be automatically distributed between all the nodes. You may keep all the default settings.

Click NEXT →.

6. Assign Slaves and Clients

Select "Livy for Spark2 Server" and "Phoenix Query Server" in all nodes that you want these services running. You will need to remember which hosts you selected when you reach the "Update Livy configuration for Knox" step.

Click NEXT →.

7. Customize Services

At this point, you only need to provide credentials for different services and review the installation.

CREDENTIALS sub-step

1. Create the Knox master secret password and store it somewhere.

   Note

   You will need the Knox master secret password when you get to the step Updating and Exporting the Knox Server Certificate. It must have at least 6 characters.
   In the CREDENTIALS section you will also be asked to set passwords for Grafana and Hive.

2. Click NEXT →.

DATABASES sub-step

Click NEXT →.

DIRECTORIES sub-step

Click NEXT →.

ACCOUNTS sub-step

Click NEXT →.

ALL CONFIGURATIONS sub-step

1. HDFS tab, SETTINGS sub-tab

   1. Set NameNode directories in NameNode section to:

      /hadoop/hdfs/namenode,/hadoop/hadoop/hdfs/namenode

   2. Set DataNode directories in DataNode section to:

      /hadoop/hdfs/data,/hadoop/hadoop/hdfs/data

      

2. HDFS tab, ADVANCED sub-tab

   1. If the Advanced ssl-client section is red please expand it and re-confirm ssl.client.keystore.password password

3. Click NEXT →.

8. Review

1. Review the configuration created

2. Click DEPLOY →.

9. Install, Start and Test

When the installation completes, click NEXT.

10. Summary

Click COMPLETE.

In the next step you will continue with the ”Post Install” section.

For more detailed information about the above procedure, refer to the Hortonworks official webpage.

Steps to Set up Additional Nodes in the Hadoop Cluster

Optionally, you may add an additional node (or nodes) to the cluster after the above steps.

Use the same OVA image to create all nodes for a multiple-node cluster, for example 5-nodes. On each node, perform the step Changing the Initial Password and Network Configuration.

Then run the script:

/root/setup/node_setup.sh

This will prompt you for the public key of the Ambari server. Go to the Ambari server to retrieve the public key. This can be any of the two files:

• /root/.ssh/authorized_keys

• /root/.ssh/id_rsa.pub

Once the node is set up, log in to the Ambari server and add the node to the cluster with the following steps:

1. Go to the Ambari UI.

2. Click the Hosts button in the menu on the left. You should see the list of nodes already present on the cluster.

3. Click Actions -> Add New Hosts.

   Note

   • As in the host names field you can add more than one node, you can add more than one node at the same time. However, the previous steps are required on every node.

   • The Ambari server must solve the hostname of each node and the node must be accessible from it.

4. On the ssh private key field you need to paste the private key of the Ambari Server node (typically, the first node of your cluster).

5. Click Register and Confirm.

6. Now, you just have to follow the instructions and add the desired services on the new node.

If you are setting up an additional node in the TDP cluster upgraded from a previous version, rename the repo path on the new node to match the repo path of the existing nodes in the cluster and recreate the repo.

For example to add a new node in cluster upgrade from 3.1.5.1 to 3.1.6:

1. Rename the repo:

mv /var/repo/hdp /var/repo/hdp-316

2. Re-create the repo:

createrepo /var/repo/hdp-316