Install Tez, Pig and Hive on HDP



 Check Support Matrix


Check Hortonworks support matrix to check which version of MySQL is supported :

https://supportmatrix.hortonworks.com/

In support matrix, I will choose :

  • Ubuntu 18.04 under Operating Systems
  • Ambari 2.7.3 and HDP 3.1.4 under Products.
And see which version of MySQL is supported , under Databases section.
If you need to confirm which versions of Ambari and HDP you have installed, here is how.

  • The following command will report your OS version

lsb_release -a

  • You can learn Ambari version by choosing “About” under user menu.



Finally, HDP version (and version of all services installed) can be found under Stack and Versions page, Versions tab.




So we have to install MySQL version 5.7. Let’s check what we have, in our Ubuntu repository :

root@dikanka:~# apt-cache showpkg mysql-server
Package: mysql-server
Versions:
5.7.27-0ubuntu0.18.04.1
Indeed we have 5.7.

mySQL 1/5 : Install MySQL Server

MySQL database server is needed to keep metastore of Hive. Before installing the services in Ambari console, we have to setup a MySQL server.
Following command will install mysql server on our Kubuntu.

sudo apt-get install mysql-server


mySQL 2/5 : Confirm MySQL Server Installation

After setup is finished, you may want to confirm mysql is up and running.
sudo mysql -u root
This will log you in mysql console as root. You can then
exit
or maybe
show databases;
before you exit this console.

mySQL 3/5 : Secure MySQL Server Installation

The following command will help us configure our newly installed MySQL server.
sudo mysql_secure_installation

Securing the MySQL server deployment.

Connecting to MySQL using a blank password.

VALIDATE PASSWORD PLUGIN can be used to test passwords
and improve security. It checks the strength of password
and allows the users to set only those passwords which are
secure enough. Would you like to setup VALIDATE PASSWORD plugin?

Press y|Y for Yes, any other key for No: N
Please set the password for root here.

New password: admin123

Re-enter new password: admin123
By default, a MySQL installation has an anonymous user,
allowing anyone to log into MySQL without having to have
a user account created for them. This is intended only for
testing, and to make the installation go a bit smoother.
You should remove them before moving into a production
environment.

Remove anonymous users? (Press y|Y for Yes, any other key for No) : Y
Success.


Normally, root should only be allowed to connect from
'localhost'. This ensures that someone cannot guess at
the root password from the network.

Disallow root login remotely? (Press y|Y for Yes, any other key for No) : N

... skipping.
By default, MySQL comes with a database named 'test' that
anyone can access. This is also intended only for testing,
and should be removed before moving into a production
environment.


Remove test database and access to it? (Press y|Y for Yes, any other key for No) : N

... skipping.
Reloading the privilege tables will ensure that all changes
made so far will take effect immediately.

Reload privilege tables now? (Press y|Y for Yes, any other key for No) : Y
Success.

All done!


mySQL 4/5 : Install mysql java connector

Download the latest mysql java connector for your OS as a deb package, from the following address :

https://dev.mysql.com/downloads/connector/j/



Double click the deb package and install it.



Check the location of mysql java connector
ls /usr/share/java/*mysql*
/usr/share/java/mysql-connector-java-8.0.17.jar
/usr/share/java/mysql-connector-java.jar


Stop ambari-server if it is running.
sudo ambari-server stop


Introduce mysql java connector to Ambari server
sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java.jar Using python /usr/bin/python
Using python /usr/bin/python
Setup ambari-server
Copying /usr/share/java/mysql-connector-java.jar to /var/lib/ambari-server/resources/mysql-connector-java.jar
If you are updating existing jdbc driver jar for mysql with mysql-connector-java.jar. Please remove the old driver jar
, from all hosts. Restarting services that need the driver, will automatically copy the new jar to the hosts.
JDBC driver was successfully initialized.
Ambari Server 'setup' completed successfully.




Start ambari-server
sudo ambari-server start






mySQL 5/5 : Get ready for Hive Metadata Storage



Ambari will create the metastore for Hive, in this newly installed MySQL database server.
But we need to provide a user with privileges to Ambari for this purpose.
Let’s open mysql console to prepare for Hive metadata storage. First we need to learn our computer’s FQDN (Fully Qualified Domain Name) It’s dikanka in my case. You can learn yours by typing :

hostname -f

Now let’s login the mysqlconsole.
sudo mysql -u root -p


We’ll create three users with same usernames, but different hosts : %, localhost and your FQDN.
mysql> CREATE USER 'hive'@'localhost' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.10 sec)

mysql> CREATE USER 'hive'@'%' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.00 sec)

mysql> CREATE USER 'hive'@'dikanka' IDENTIFIED BY 'hive';
Query OK, 0 rows affected (0.01 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'%';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'dikanka';
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON *.* TO 'hive'@'localhost';
Query OK, 0 rows affected (0.00 sec)

mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.02 sec)

Now we're ready to go to Ambari console.
exit

Install Hive, Pig & Tez



Now that MySQL is up and running, we can login to Ambari console on http://localhost:8080 and install the services.
Choose “Add Service” under Services menu.






Choose Tez, Hive and Pig. Click Next.



It shall not be difficult for Ambari to select our only namenode as master node, for these services. Click next.



Not too many choices for slaves / clients as well. Click Next.



Newly added services are marked with bold titles in service customization screen. Hive raises an alert, because it needs database configuration. Others shall be ok with default values.



Hive metastore database settings are as follows. We choose New MySQL as Hive database. This means Ambari will create a database named hive in our newly installed MySQL database server. Ambari will connect to this server with a user named hive, and the password is also hive. We created and granted this user after at step mySQL 5/5



Then we click deploy

Nothing went wrong with my installation, and I could start the services successfully. Good luck with yours!

No comments:

Post a Comment