Ambari Configuration
Hadoop Services Installation Through Ambari
You have to install Livy for Spark2 Server on one or multiple node, which was optional in previous TDP version. Please follow the instructions below if you haven't select it in previous installation.
To execute the instructions below you will need to access Ambari, please refer to Accessing Ambari for further information.
Add Service
In the Ambari toolbar on the left, click the ellipsis next to Services.
Click Add Service.
Choose Services
Here you are going to select the services that will be available on our virtual machine.
Note
The following services are also required for DDC, but as they should be already installed, they are displayed in gray.
HDFS
Yarn + MapReduce2
HBase
Zookeeper
Ambari_metrics
Knox
These are the new required services for DDC:
Tez
Hive
Spark
If those services are already installed in your cluster (their checkbox are displayed in gray), please close the wizard and skip to the Reviewing Hosts Components section.
Tip
If any of the required services fails to start automatically, you may need to start it manually. In case of repeated problems, please check out the service log for additional troubleshooting information. Refer to the article "Viewing Service Logs" on the Hortonworks documentation pages for details.
Assign Masters
In our example, as we're only using one node, so this will be our master for all the services. In production environments, you can select other nodes as the service Master.
Assign Slaves and Clients
Select "Livy for Spark2 Server" in all nodes that you want this service running. You will need to remember which hosts you selected when you reach the "Update Livy configuration for Knox" step.
Caution
This step is really important. Make sure that you select "Livy for Spark2 Server" in at least one of your hosts because it is a hard requirement for DDC.
Note
It is recommended to not select "Spark2 thrift server".
Customize Services
Note
The services can be installed during the TDP installation or at any moment. If this is configured during the installation, you will not have to restart the installed services. It is possible to do so when a new service is installed, however in this case you will have to restart the services.
At this point, you only need to provide credentials for different services and review the installation.
For Hive you have to add a database.
Go to the Hive tab, then click Database.
Select Hive Database -> New MySQL.
Set the credentials for the database.
Then, keep clicking NEXT until the screen that shows DEPLOY. Click it.
When the installation completes, click NEXT and then click COMPLETE.
Reviewing Hosts Components
You have to install Livy for Spark2 Server on at least one node, which was optional in previous TDP version. In this section you are going to check if the new required services are properly configured, and if not, fix it.
To execute the instructions below you will need to access Ambari, please refer to Accessing Ambari for further information.
In the Ambari toolbar on the left, click Hosts.
In the Hosts list select to the node where you want to install Livy for Spark2 Server.
Check if Spark2 Client / Spark2 is listed in the Components page. If it is not there, click ADD and select Spark2 Client, then click Confirm Add on the following pop-up.
Check if Livy for Spark2 Server / Spark2 is listed in the Components page. If it is not there, click ADD and select Livy for Spark2 Server, then click Confirm Add on the following pop-up. Wait for the installation to finish.
In the Livy for Spark2 Server line, click on the elipsis on the right side and then Start if it is not yet started.
Repeat those steps for all nodes that needs Livy for Spark2 Server service activated.
For more detailed information about the above procedure, refer to the Hortonworks official webpage.
