HANDY 6

Automated Hadoop Cluster Deployment and Data Analysis

Features

• Beautiful, Easy to use, Responsive User Interface
• One Button Hadoop Cluster Deployment
• One Button Human Resource Data Analysis with Four in-built templates:
• Employee Satisfaction Analysis
• Total Number of Employees who left their jobs despite high salary
• Total Number of Employees who left their jobs due to medium salary
• Total Number of Employees who left their jobs due to low salary
• Add-on Utility #1: Docker Web Interface: Now Add/Delete/Start/Stop all of your docker containers with one button.
• Add-on Utility #2: Automated Web Server Deployment: Deploy Web Server on any system with one button.

Demonstration

User can log into the application using appropriate credentials.

Now this dashboard screen opens up. User can select any operation i.e. Setup Hadoop/Hadoop2, See Cluster Status, Manage Docker Containers, Setup Web Server etc.

If user selects 'Setup Hadoop', this screen opens up. Now there are two options to setup hadoop cluster. Manual and On-demand. In Manual, after entering number of datanodes, user will be asked IP addresses of namenode, jobtracker, client and all datanodes. Now this application will automatically install and setup hadoop on all the systems. In On-demand, user don't need to enter IP addresses of any server. This application will make docker containers for namenode, jobtracker, client, and all of the datanodes and configure hadoop in them.

When hadoop cluster setup is done, this screen opens up and shows Namspace ID of your hadoop cluster.

In this page, User can upload datasets and select the operation, user wants to perform on the dataset. If user wants to do employee satisfaction analysis (s)he can select the operation in the dropdown. Currently, there are four operations, as mentioned above in features. Here we have selected employee satisfaction analysis. This hrdata.csv file contains satisfaction score, salary type, experience, number of projects, job data of 14999 employees.

When analysis is done in MapReduce, Aplication fatches results from Hadoop File System and shows it right in the browser.

Add-on Utility #1: Manage Docker Containers page, User can manage all the docker containers running on server, User can add, delete, start, stop containers right from the browser.

Add-on Utility #2: Enter IP address and password of a system and make it a webserver. HANDY 6 automatically installs and configures httpd server on the system.

Technologies Used

• Bootstrap and Javascript: For Responsive UI
• Ansible: To automate tasks such as Install softwares, copy files from webserver to remote servers, and running commands on remote servers
• Docker: To run hadoop cluster in one system
• Hadoop: To handle and anlyse large datasets
• Python:To connect everything and Mapper, Reducer programs
• Splunk: For log management.

Target Audience

Human Resource Managers of an enterprise. This application is easy to use, HR Managers can upload and analyse huge datasets.

Made with ♥ by Invasive Analysts

HANDY 7 is coming!

• Almost 2 times faster deployment of Hadoop Cluster than HANDY 6
• Enhanced Security with use of Django
• Option to upload custom mapper and reducer programs so that user can analyse any type of data, which changes the target audience of this application to everyone who want to analyse huge data
• Extra templates
• Support for Mapper and Reducer programs written in R and Java
• Support for Machine Learning tools like Apache Mahout
• And Much More

© Priyansh Rastogi, 2017