Install a single-node Hortonworks Data Platform (HDP) version 3.1.4 on Kubuntu 18.04

This post will lead you through the setup process of a single-node Hortonworks Data Platform, working on a Kubuntu 18.04 workstation.

Of course you can easily download a Hortonworks Sandbox image here.
Having said this, one can also prefer to get their hands dirty; and build it from scratch.
So here we go.


  • Ambari 2.7.3
  • HDP 3.1.4
    • HDFS 3.1.1
    • YARN 3.1.1
    • MapReduce2 3.1.1
    • HBase 2.0.2
    • ZooKeeper 3.4.6
    • Ambari Metrics 0.1.0
    • SmartSense
We'll first install Ambari Server. 
Then we'll register an Ambari Agent, using Ambari console.
During this registration, we'll choose the services above to be installed. This is a subset - For now we'll leave some important services out of our scope, such as Hive, Pig, Spark, Kafka, etc. 
But we'll come back to them soon.

Decide : Check the support matrix

The support matrix for Hortonworks Data Platform is here.
Choose your OS version under Operating systems. See which Ambari and HDP versions are supported for your OS.

I have an Ubuntu 18.04,and I will install HDP 3.1.4 using Ambari 2.7.3.
Don’t assume a newer version of your favorite OS will be ok. Stick to the versions listed in the support matrix. Author of this blog lost some time trying to install HDP on Ubuntu 19 – which is too new to be supported.

Prepare 1/6 : Maximum Open Files Requirements

Upper limit for open file descriptors in your system shall not be less than 10,000.

First let's check the current settings. Sn stands for soft limits whereas Hn is for Hard limits.

ulimit -Sn

ulimit -Hn

And here's how we change them.

ulimit -n 10000

Hard limit was maybe even more than 10,000, but we don't have to worry about that. These changes will be lost with the next reboot, but we need this only during the setup process anyway.

Check here for some nice details.

Prepare 2/6 : Setup password-less SSH

As you may already know, Ambari helps us administer all the nodes in a Hadoop cluster. Each of these nodes shall have an Ambari agent installed.
Ambari server is able to install these agents in nodes. - Only if an SSH connection is built.
Here we have a single node - and it's also where Ambari Server is installed. But still we need this SSH connection.
Now let's register our laptop as a trusted ssh connection for ... our laptop!
  • Generate SSH keys. Leave the passphrase empty.

oguz@dikanka:~$ sudo -i
root@dikanka:~# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Created directory '/root/.ssh'.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/
The key fingerprint is:
  • Add the public key to authorized keys file. 
root@dikanka:~# cd .ssh
root@dikanka:~/.ssh# cat >> authorized_keys
  • Change permissions of .ssh folder and authorized keys file.
root@dikanka:~/.ssh# cd ..
root@dikanka:~# chmod 700 .ssh  
root@dikanka:~# chmod 600 .ssh/authorized_keys4 
  •  Check the results
root@dikanka:~# ssh dikanka  
ssh: connect to host dikanka port 22: Connection refused
  •  This happens when :
    •  openssh is not installled. Install it as follows :
sudo apt-get update
sudo apt-get install openssh-server
    •  port 22 is blockedby the firewall. Allow this port as below :
sudo ufw allow 22
Rules updated
Rules updated (v6)
  • Check the results. When asked, type “yes” and press enter to confirm adding the server to known hosts.
root@dikanka:~# ssh dikanka
The authenticity of host 'dikanka (' can't be established.
ECDSA key fingerprint is ...
Are you sure you want to continue connecting (yes/no)? yes

Warning: Permanently added 'dikanka' (ECDSA) to the list of known hosts.
Welcome to Ubuntu 18.04.3 LTS (GNU/Linux 5.0.0-25-generic x86_64)
  • All seems OK. Type “exit” to close the ssh connection.
root@dikanka:~# exit
Connection to dikanka closed.

Prepare 3/6 : Enable NTP on the Cluster and on the Browser Host

The clocks of all the nodes in your cluster and the machine that runs the browser through which you access the Ambari Web interface must be able to synchronize with each other.
First let's check if ntp is running :
oguz@dikanka:~$ ntpstat
Unable to talk to NTP daemon. Is it running?
It's not. The following command will start ntp service.
sudo service ntp start

And the following command will ensure that it gets automatically started during boot.
sudo update-rc.d ntp defaults

 Now it should be up and running :
oguz@dikanka:~$ ntpstat
synchronised to NTP server ( at stratum 2
    time correct to within 943 ms
    polling server every 64 s

Prepare 4/6 : Configuring iptables

For Ambari to communicate during setup with the hosts it deploys to and manages, certain ports must be open and available. The easiest way to do this is to temporarily disable iptables, as follows:

sudo -i
ufw disable
iptables -X
iptables -t nat -F
iptables -t nat -X
iptables -t nat -X
iptables -t mangle -F
iptables -t mangle -X
iptables -P INPUT ACCEPT

Prepare 5/6 : Umask

umask command is used to set the default permissions of newly created files and folders.
umask 022
umask with 022 value will set permissions of 755 for new files and folders.
If this numeric code does not ring a bell for you, I'd strongly suggest to learn more about linux file permissions. You can try wikipedia.

Prepare 6/6 : Repository Connection

Now we'll connect to Hortonworks software repository in order to install ambari server.

oguz@dikanka:~$sudo -i
root@dikanka:~#wget -O /etc/apt/sources.list.d/ambari.list
--2019-08-26 15:01:44--
Resolving (,,, . ...

 2019-08-26 15:01:44 (10,8 MB/s) - ‘/etc/apt/sources.list.d/ambari.list’ saved [187/187] 

apt-key adv --recv-keys --keyserver B9733A7A07513CAD 

Executing: /tmp/apt-key-gpghome.DmGQnBiPeG/ --recv-keys --keyserver B9733A7A07513CAD
gpg: key B9733A7A07513CAD: public key "Jenkins (HDP Builds) <>" imported
gpg: Total number processed: 1
gpg:               imported: 1

Let's update our software repository and see if we can locate ambari packages.

apt-get update

apt-cache showpkg ambari-server
apt-cache showpkg ambari-agent
apt-cache showpkg ambari-metrics-assembly

Ambari 1/3: Install Ambari Server

We'll now install the ambari server, and then configure and start ambari server. Should be easier than what you think.
apt-get install ambari-server

Success. You can now start the database server using:

   /usr/lib/postgresql/10/bin/pg_ctl -D /var/lib/postgresql/10/main -l logfile start

Ver Cluster Port Status Owner    Data directory              Log file
10  main    5432 down   postgres /var/lib/postgresql/10/main /var/log/postgresql/postgresql-10-main.log
update-alternatives: using /usr/share/postgresql/10/man/man1/postmaster.1.gz to provide /usr/share/man/man1/postmaster
.1.gz (postmaster.1.gz) in auto mode
Setting up postgresql (10+190) ...
Setting up ambari-server ( ...
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.24) ...

Ambari 2/3: Setup Ambari Server

 ambari-server setup

  • When asked, type 1 to download and install Oracle JDK
  • Type y to confirm Oracle licence agreement
  • Type n (or leave empty) when asked to downoad LZO packages
  • Type y when asked to enter advanced database configuration. (Default is using embedded PostgreSQL. Actually we’ll not change the defaults but this way we’ll be informed about db name, user name, password, etc.)

Using python  /usr/bin/python
Setup ambari-server
Checking SELinux...
WARNING: Could not run /usr/sbin/sestatus: OK
Customize user account for ambari-server daemon [y/n] (n)?
Adjusting ambari-server permissions and ownership...
Checking firewall status...
Checking JDK...
[1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8
[2] Custom JDK
Enter choice (1): 1
To download the Oracle JDK and the Java Cryptography Extension (JCE) Policy Files you must accept the license terms fo
und at and not accepting will cancel the Ambari
Server setup and you must install the JDK and JCE files manually.
Do you accept the Oracle Binary Code License Agreement [y/n] (y)? y

Downloading JDK from to /var/lib/ambari-serv
jdk-8u112-linux-x64.tar.gz... 100% (174.7 MB of 174.7 MB)
Successfully downloaded JDK distribution to /var/lib/ambari-server/resources/jdk-8u112-linux-x64.tar.gz
Installing JDK to /usr/jdk64/
Successfully installed JDK to /usr/jdk64/
Downloading JCE Policy archive from to /var/lib/ambari

Successfully downloaded JCE Policy archive to /var/lib/ambari-server/resources/
Installing JCE policy...
Check JDK version for Ambari Server...
JDK version found: 8
Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server.
Checking GPL software agreement...
GPL License for LZO:
Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? n
Completing setup...
Configuring database...
Enter advanced database configuration [y/n] (n)? y
Configuring database...
Choose one of the following options:
[1] - PostgreSQL (Embedded)
[2] - Oracle
[3] - MySQL / MariaDB
[4] - PostgreSQL
[5] - Microsoft SQL Server (Tech Preview)
[6] - SQL Anywhere
[7] - BDB
Enter choice (1):
Database admin user (postgres):
Database name (ambari):
Postgres schema (ambari):
Username (ambari):
Enter Database Password (bigdata):
Default properties detected. Using built-in database.
Configuring ambari database...
Checking PostgreSQL...
Configuring local database...
Configuring PostgreSQL...
Restarting PostgreSQL
Creating schema and user...
Creating tables...
Extracting system views...

Ambari repo file contains latest json url, updating stacks r
epoinfos with it...
Adjusting ambari-server permissions and ownership...
Ambari Server 'setup' completed successfully.

Ambari server will use a postgresql database for its repository. Note down the default database name and user credentials configured for this database.

Ambari 3/3: Start Ambari Server

ambari-server start 
Using python  /usr/bin/python
Starting ambari-server
Ambari Server running with administrator privileges.
Organizing resource files at /var/lib/ambari-server/resources...
Ambari database consistency check started...
Server PID at: /var/run/ambari-server/
Server out at: /var/log/ambari-server/ambari-server.out
Server log at: /var/log/ambari-server/ambari-server.log
Waiting for server start...................
Server started listening on 8080

DB configs consistency check: no errors and warnings were found.
Ambari Server 'start' completed successfully.

Register a cluster

Login to Ambari using a browser having root access. That's needed because we'll upload SSH key from root directory, during this registration.
oguz@dikanka~$ sudo chromium-browser --no-sandbox

Navigate to address http://localhost:8080

user: admin
password: admin

Click "Launch Install Wizard" to register our first and only cluster.

Name your cluster, click next.

HDP 3.1 is selected by default. “Use Public Repository” is also selected. Click Next.

Add your computer name in the list of target hosts.
Click “choose file”. Locate file “id_rsa” under path “/root/.ssh” (to be able to access hidden path “.ssh”, you may need to right click and choose option “Show hidden files”)
Cick “Register and confirm”
Ignore the warning about the computer name not being a valid FDQN.

You may need to do some problem solving here. Please remember that your OS shall be listed in the support matrix, client shall be reachable through passwordless SSH, and there should be no issues on connectivity, like a firewall blockage, or busy ports, etc.
If everything goes fine, you should see the screen below.

Time to install some services.
Following services will be installed. Deselect the others and click next.

  • YARN + MapReduce2
  • Hbase
  • ZooKeeper
  • Ambari Metrics
  • SmartSense

Ignore the warnings about limited functionally, due to skipping Apache Ranger and Apache Atlas.

All masters will be the same. Click Next. 

Only one slave is available. Click Next

All passwords are set as “admin123”

No database settings for selected services. Therefore the database settings are disabled. We left the directories as their default values. See them listed below.

DataNode directories
NameNode directories
SecondaryNameNode Checkpoint directories
NFSGateway dump directory
NameNode Backup directory
JournalNode Edits directory
NameNode Checkpoint Edits directory
Hadoop Log Dir Prefix
Hadoop PID Dir Prefix

YARN NodeManager Local directories
YARN Timeline Service Entity Group FS Store Active directory
YARN Node Labels FS Store Root directory
YARN NodeManager Recovery directory
YARN Timeline Service Entity Group FS Store Done directory
YARN NodeManager Log directories
YARN NodeManager Remote App Log directory
YARN Log Dir Prefix
YARN PID Dir Prefix

Mapreduce JobHistory Done directory
Mapreduce JobHistory Intermediate Done directory
YARN App Mapreduce AM Staging directory
Mapreduce Log Dir Prefix
Mapreduce PID Dir Prefix

HBase Java IO Tmpdir
HBase Bulkload Staging directory
HBase Local directory
HBase root directory
HBase tmp directory
ZooKeeper Znode Parent
HBase Log Dir Prefix
HBase PID Dir

ZooKeeper directory
ZooKeeper Log Dir
ZooKeeper PID Dir

Aggregator checkpoint directory
Metrics Grafana data dir
HBase Local directory
HBase root directory
HBase tmp directory
HBase ZooKeeper Property DataDir
Phoenix Spool directory
Phoenix Spool directory
Metrics Collector log dir
Metrics Monitor log dir
Metrics Grafana log dir
HBase Log Dir Prefix
Metrics Collector pid dir
Metrics Monitor pid dir
Metrics Grafana pid dir
HBase PID Dir

Accounts are left as default values. See them listed below.

Smoke User
Mapreduce User
Hadoop Group
Oozie User
Ambari Metrics User
Yarn ATS User
HBase User
Yarn User
ZooKeeper User
Proxy User Group


All settings in Advanced configuration tab are left as defaults, but the SSL client password setting under “HDFS / Advanced” might raise an error.

It’s a password setting issue. Type “admin123” in both password fields to fix the issue.

Click Deploy and pray to your preferred God.

Port issue for YARN

Yeah, it failed.
Luckily all the services were installed without issues. The failure happened during starting services. YARN fails with the following error : Problem binding to [dikanka:53] Address already in use; For more details see:
This happens because port 53 is not available. Solution is simple :

Under YARN / Configs / Advanced, locate the setting named “RegistryDNS Bind Port” and change from 53 to 553

Start Services

Since the services failed to start, we have to start them one by one. To start services, we’ll choose “Restart All” under “Actions” menu for each of the services.

Let’s start the services in the following order :
  • Zookeeper
  • HDFS
  • Hbase
  • YARN
  • MapReduce2

Check Files View

If all services are up and running, it’s time now to check what we have in hand. Click “Views” menu and choose “Files View”

This might result with the following error message :

Unauthorized connection for super-user: root from IP

In this case, apply the following steps to solve this issue :

  • In Ambari Web, browse to Services > HDFS > Configs.
  • Under the Advanced tab, navigate to the Custom core-site section.
  • Change the values of the following parameters to *


After these values are altered, you will need to restart all services. Retry opening files view. Confirm that the view looks like the screenshot below :


And this marks the end of the scope for this post. Soon we'll continue with other services like Pig, Tez and Hive.
Hope this was helpful for some of you. 



  1. helllo, really good article in the 3 step i have to confirm the host,
    i had this: E: Failed to fetch 401 Authentication required.
    how can i find the solution ?

  2. This cant be done anymore, re-install your ubuntu on your computer and next time choose 3.1.4 in ambari, otherwise you have to pay for cloudera credentials

  3. Hello, in the step 2 above, when I run the command
    ssh dikanka

    I get the error Permission denied (publickey). What can be the reason? Thanks.

    1. Hi Daega. dikanka was my computer name back when I did this installation. For you it should be your computer's name. I see thia was not clear enough in my post.


Post a Comment