Saturday, December 29, 2012

Integration of R, RStudio and Hadoop in a VirtualBox Cloudera Demo VM on Mac OS X


Motivation


I was inspired by Revolution's blog and step-by-step tutorial from Jeffrey Breen on the set up of a local virtual instance of Hadoop with R. However, this tutorial describes the implementation using VMware's application. One downside to using VMware is that it's not free. I know most of the people including me like to hear the words open-source and free, especially when it is a smooth ride. VirtualBox offers an open-source alternative and thenceforth, I chose this. Most of the trouble started after a hassle free installation of VirtualBox and creation of the cloudera's demo VM. I came across different hurdles when it came to addition of VirtualBox Guest Additions, which is intended to spruce up the virtual machine by offering such features as a shared folder with the host OS. Although there are solutions, the resources are scattered and obscure. I did manage to clear these hurdles and went on to installing R and RStudio along with RHadoop packages. I thought it would be useful to self-taught enthusiasts like me if I lay out the steps in a comprehensive manner, since I have spent some time dealing with the quirks in the process.


Description


Hadoop

Apache Hadoop is an open-source software framework that supports data-intensive distributed applications, licensed under the Apache v2 license. It supports the running of applications on large clusters of commodity hardware. The Hadoop framework transparently provides applications both reliability and data motion. Hadoop implements a computational paradigm named map/reduce, where the application is divided into many small fragments of work, each of which may be executed or re-executed on any node in the cluster. In addition, it provides a distributed file system that stores data on the compute nodes, providing very high aggregate bandwidth across the cluster. Both map/reduce and the distributed file system are designed so that node failures are automatically handled by the framework.


R and Hadoop

The most common way to link R and Hadoop is to use HDFS (potentially managed by Hive or HBase) as the long-term store for all data, and use MapReduce jobs (potentially submitted from Hive, Pig, or Oozie) to encode, enrich, and sample data sets from HDFS into R. Data analysts can then perform complex modeling exercises on a subset of prepared data in R.Revolution Analytics released RHadoop allowing integration of R and Hadoop. RHadoop is a collection of three R packages that allow users to manage and analyze data with Hadoop.

RHadoop consists of the following packages:
rmr - functions providing Hadoop MapReduce functionality in R
rhdfs - functions providing file management of the HDFS from within R
rhbase - functions providing database management for the HBase distributed database from within R


Cloudera Hadoop Demo VM

CDH is Cloudera’s 100% open source distribution of Hadoop and related projects, built specifically to meet enterprise demands. Cloudera created a set of virtual machines (VM) with everything we need to make it easy to get started with Apache Hadoop. Cloudera Hadoop's Demo VM provides everything you need to run small jobs in a virtual environment. The packages have been implemented and tested in Cloudera's distribution of Hadoop (CDH3) & (CDH4). and R 2.15.0. This offers a great way to get familiarized with Hadoop.


Steps...............


Platforms used in this tutorial:

Guest OS : Mac OS X 10.7.5 (Lion)
Virtualization software: VirtualBox 4.2.6
Cloudera Hadoop Demo VM: CDH 4.1.1
R: 2.15.2
RStudio server for : 0.97.248
RHadoop packages: rmr 2.0.2


1. Download and install the latest release of VirtualBox (Ver 4.2.6 at the time of this post) for your platform (Here OS X)
http://download.virtualbox.org/virtualbox/4.2.6/VirtualBox-4.2.6-82870-OSX.dmg

2. Download 'Cloudera's Hadoop Demo VM archive for CDH4
(Latest: Ver 4.1.1 runs CentOS 6.2 64 bit VM)
https://downloads.cloudera.com/demo_vm/virtualbox/cloudera-demo-vm-cdh4.1.1-virtualbox.tar.gz

3. Extract 'Cloudera's Hadoop Demo VM' archive
It extracts virtual machine image file: 'cloudera-demo-vm.vmdk'

4. Copy this virtual machine image to a desired folder (eg:- folder named 'Cloudera Hadoop'). This folder and image file has to be the permanent location of your Hadoop installation (not to be deleted!)

5. We will now create a virtual machine on VirtualBox.

Open application: 'VirtualBox'. Click on 'New'.


Give a name to the VM. Here: 'Cloudera Hadoop'. 
Pick type: Linux
Choose version: Linux 2.6 (64 bit)
Click 'Continue'.


It is generally recommended to allocate at least 2 GB of RAM. I recommend more, since I encountered problems installing the R package 'Rcpp' with about 2 GB of RAM. I allocated 4 GB and resolved the issue.

Click 'Continue'.



Choose the option 'Use an existing virtual hard drive file' and select the virtual image file ( 'cloudera-demo-vm.vmdk' ) saved in folder 'Cloudera Hadoop' (refer to step 4).

Click 'Create'.

6. Click 'Settings' to make a few recommended changes.


Click on tab 'Advanced' under 'General' category.



Pick 'Bidirectional' option for items: 'Shared Clipboard' and 'Drag'n Drop'


Click on 'System' category. Ensure option 'Enable IO APIC' under 'Extended Features' is checked on (This is default!)
Click on 'Network' category. Choose Adapter 1 option 'Attached to:' as 'Bridged Adapter' (This gives you access to physical wifi. Default is 'NAT')



Click on 'Shared Folders' category. You may choose to pick a folder to share with the host OS (here Mac OS X)

Click 'OK'.


Now click 'Start' to initiate the virtual machine. You will several pages of output on a black screen until you finally see the desktop of the virtual machine.

Once you launch the VM, you are automatically logged in as the cloudera user.

The account details are:
username: cloudera
password: cloudera
The cloudera account has sudo privileges in the VM.

7. For close integration and better performance we need to install "Guest additions" in the VM. 
There are some prerequisites to installation of 'Guest additions'.

Run console
Switch to root user: 

$ sudo bash
Update linux kernel: 
$ yum install kernel -y
Reboot
Run console
Open internet browser (Firefox) and download the following file (link below):
http://rpm.pbone.net/index.php3/stat/4/idpl/18259813/dir/scientific_linux_6/com/kernel-devel-2.6.32-220.23.1.el6.x86_64.rpm.html
Click on link for file: kernel-devel-2.6.32-220.23.1.el6.x86_64.rpm
Download and save file to folder 'Downloads' under 'home/cloudera' (Either create new folder using 'Save' dialog box or use console: mkdir /home/cloudera/Downloads

Run console
Install packages:
$ yum install kernel-devel-2.6.32-220.23.1.el6.x86_64.rpm -y
$ yum install gcc -y
Link the kernel sources to a standard location using the format:
'ln -s /usr/src/kernels/[current version] /usr/src/linux'
$ ln -s /usr/src/kernels/2.6.32-220.23.1.el6.x86_64 /usr/src/linux
Installation of package 'dkms' 
(It is important that you use the steps below to install 'dkms' in CentOS, which is the linux build for your Cloudera demo VM.)
Steps to install rpmforge-release package to enable rpmforge repository
Run Console
$ mkdir rpm (create folder 'rpm' under 'home/cloudera' : /home/cloudera/rpm)
$ cd rpm (change to 'rpm' folder: /home/cloudera/rpm)
$ wget http://packages.sw.be/rpmforge-release/rpmforge-release-0.5.2-2.el6.rf.x86_64.rpm
Install DAG's GPG key
$ rpm --import http://apt.sw.be/RPM-GPG-KEY.dag.txt
If you get an error message like the following the key has already been imported.
error: http://apt.sw.be/RPM-GPG-KEY.dag.txt: key 1 import failed.
Verify the package you have downloaded
$ rpm -K rpmforge-release-0.5.2-2.el6.rf.*.rpmInstall the package
$ rpm -i rpmforge-release-0.5.2-2.el6.rf.*.rpm
This will add a yum repository config file and import the appropriate GPG keys.
Now install package 'dkms' as root:
$ sudo yum install dkms
Now you are ready to download and install 'VirtualBox Guest Additions'. Download 'VirtualBox Guest Additions' .iso image file ('VBoxGuestAdditions_4.2.6.iso') corresponding to your version of 'VirtualBox' installation (in this case Ver 4.2.6).
(Note: The following steps are unlike what is described in most of the posts on this topic. 
I faced a lot of problems in making this happen using the steps described in these posts. 
I therefore, recommend this method to avoid those issues.)
Open internet browser (Firefox) and download the following file (link below). Save file to folder 
'Downloads' under 'home' (already created folder)
http://download.virtualbox.org/virtualbox/4.2.6/VBoxGuestAdditions_4.2.6.iso
Switch to 'root' user:
$ sudo bash
$ mkdir /mnt/ISO
Once your folder is created go to the folder where ISO image 'VBoxGuestAdditions_4.2.6.iso is stored.
$ cd /home/cloudera/Downloads
Use command: ls to list contents of the folder.
$ mount -t iso9660 -o loop VBoxGuestAdditions_4.2.6.iso /mnt/ISO
$ cd /mnt/ISO
$ ls (lists contents of the mounted 'VBoxGuestAdditions_4.2.6.iso' image)

32Bit        cert                    VBoxSolarisAdditions.pkg
64Bit        OS2                     VBoxWindowsAdditions-amd64.exe
AUTORUN.INF  runasroot.sh            VBoxWindowsAdditions.exe
autorun.sh   VBoxLinuxAdditions.run  VBoxWindowsAdditions-x86.exe
Now install 'Guest Additions' for Linux guest by running the following command.
$ sh VBoxLinuxAdditions.run
Reboot virtual machine. This completes installation of 'Guest Additions'.
(Note: Whenever you reboot, make sure there is network connection. 
Check the active network icon at the top right corner. 
If crossed out, click and enable network connection by clicking 'Auto eth0')
Enabling network connection at start-up of virtual machine (CentOS)
I learned that the network connection is not enabled automatically at start-up. 
This can be resolved by making the following changes.

Run console
Switch to root user
sudo bash
Create/edit the following file:
emacs /etc/sysconfig/network-scripts/ifcfg-eth0
This will open up this file in 'emacs' editor. (You will find it blank!)
Copy/paste the following into this file. (Hint: use Shift/Cntrl/V to paste!)
DEVICE="eth0"
HWADDR="08:00:27:FE:D5:10"
NM_CONTROLLED="yes"
ONBOOT="no"

Save the file.
Reboot and check if network connection is enabled at start-up (see above).
8. Installation of R
First add the EPEL repository, then intall git, wget and R. 
Find the latest release of the EPEL repository (http://fedoraproject.org/wiki/EPEL) and update the url accordingly.
$ sudo rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
$ sudo yum -y install git wget R
9. Set Hadoop environment variables so R can find them too!
The following is specific for CDH4 Demo VM.
$ sudo ln -s /etc/default/hadoop-0.20-mapreduce /etc/profile.d/hadoop.sh
$ cat /etc/profile.d/hadoop.sh | sed 's/export //g' > ~/.Renviron
10. Installation of Rstudio server
$ wget http://download2.rstudio.org/rstudio-server-0.97.248-x86_64.rpm
$ sudo yum install --nogpgcheck rstudio-server-0.97.248-x86_64.rpm
11. Access Rstudio from the browser (you may use any machine in the home network)
Check IP address by running command:
$ ifconfig

Access RStudio from browser by typing the address (uses port 8787) : e.g., http://10.0.1.15:8787/ Both username and password are 'cloudera' Username: cloudera Password: cloudera
12. Installation of RHadoop's rmr package
First install the pre-requisite packages. (Run R as root to install system-wide)
Run console.
$ sudo R
R> install.packages( c('RJSONIO', 'itertools', 
'digest', 'Rcpp', 'functional', 'plyr', 'stringr'),
repos='http://cran.revolutionanalytics.com')
R> q() (to quit 'R' session)
Download the latest stable release of rmr (2.0.2) from github.
Run console
$ wget --no-check-certificate https://github.com/downloads/RevolutionAnalytics/RHadoop/rmr2_2.0.2.tar.gz
$ sudo R CMD INSTALL rmr2_2.0.2.tar.gz
Test that 'rmr2' loads
$ R R> library(rmr2) Loading required package: Rcpp Loading required package: RJSONIO Loading required package: digest Loading required package: functional Loading required package: stringr Loading required package: plyr R> 13. Testing with a simple example
small.ints <- to.dfs(1:1000)
out <- mapreduce(input = small.ints, map = function(k, v) keyval(v, v^2))
df <- as.data.frame(from.dfs(out))
Screenshot showing execution of above lines on RStudio:

Other examples of mapreduce function are available at:
https://github.com/RevolutionAnalytics/RHadoop/blob/master/rmr2/docs/tutorial.md


That's it....................

11 comments:

  1. That is great! Thank you for very informative post!!
    Cheers,
    Marius

    ReplyDelete
  2. I followed the instruction but when accessing Rstudio, it keeps giving me prompts that says "cannot connect to service". Just wonder where I went wrong. PS, I cannot log in cloudera with username "cloudera" but username "admin"

    ReplyDelete
  3. Never mind. I unpdated the Rstudio manager and now it works well

    ReplyDelete
  4. Dear Sir,
    Thanks for the article.However when I tried to log into Rstudio it is giving "cannot connect to service " followed by the message R Studio initialization failed.Any idea on how to overcome this error would be greatly helpful

    Bingzie Zhang,
    can you provide inputs how you resolved this issue
    Regs
    Anand

    ReplyDelete
  5. This comment has been removed by the author.

    ReplyDelete
  6. hi after runing above test code I got this error.Please resove tte error

    Error in hadoop.streaming() :
    Please make sure that the env. variable HADOOP_STREAMING or HADOOP_HOME are set
    > out <- mapreduce(input = small.ints, map = function(k, v) keyval(v, v^2))
    Error in save(list = ls(all.names = TRUE, envir = envir), file = name, :
    object 'small.ints' not found
    > df <- as.data.frame(from.dfs(out))
    Error in to.dfs.path(input) : object 'out' not found
    > library(rmr2)
    > all.ints <- to.dfs(1:1000)
    Error in hadoop.streaming() :
    Please make sure that the env. variable HADOOP_STREAMING or HADOOP_HOME are set
    > out <- mapreduce(input = small.ints, map = function(k, v) keyval(v, v^2))
    Error in save(list = ls(all.names = TRUE, envir = envir), file = name, :
    object 'small.ints' not found
    > df <- as.data.frame(from.dfs(out))
    Error in to.dfs.path(input) : object 'out' not found

    ReplyDelete
  7. Really good piece of knowledge, I had come back to understand regarding your website from my friend Sumit, Hyderabad And it is very useful for who is looking for HADOOP.

    ReplyDelete
  8. Thanks for the wonderful information and it is useful for Hadoop Learners.123trainings provides best online Hadoop training.to see free demo class online Hadoop training class in india.

    ReplyDelete
  9. It was nice article it was very useful for me as well as useful for online Hadoop training learners.thanks for providing this valuable information.123trainings provides best Hadoop online training in uk

    ReplyDelete
  10. It was nice article it was very useful for me as well as useful foronline Hadoop training learners. thanks for providing this valuable information.

    ReplyDelete
  11. Good Well works.your article very useful for my projects Thanks a lot.


    Hadoop Training in Chennai

    ReplyDelete