Tips running Hadoop on Ubuntu

Below is some tips when running Hadoop on Ubuntu. If you find some errors running Hadoop on Ubuntu please comment the problem and how you solved it.

When you get this Warning: $HADOOP_HOME is deprecated

Solution: add “export HADOOP_HOME_WARN_SUPPRESS=”TRUE””  in the hadoop-env.sh.

Cannot create directory `/usr/local/hadoop/libexec/../logs

Solution: sudo chown -R hduser:hadoop /usr/local/hadoop/

Enter passphrase when running ./start-all.sh

Solution: ssh-keygen -t rsa -P “”     Create a ssh-key without a password.

Warning: <property>/<configuration> not set

Solution: make sure <property> and <configuration> tags are populated in core.site.xml, mapred.site.xml, hdfs.site.xml

Send or retrieve file to and from HDFS

Solution:

Send file to HDFS > bin/hadoop dfs -put /home/someone/interestingFile.txt /user/yourUserName/

Get file from HDFS > bin/hadoop dfs -get foo localFile

ssh: connect to host localhost port 22: Connection refused
Solution: By default Ubuntu will not have ssh installed so you will have to install and start it.

Install > sudo apt-get install ssh

Start > sudo service ssh start

hadoop Incompatible namespaceIDs in /app/hadoop/tmp/*

Solution: 

  1. Stop the cluster: ./stop-dfs.sh
  2. Delete the directory specified on the DataNode: rm -r /app/hadoop/tmp/*
  3. Reformat the NameNode: hadoop namenode -format

OR

  1. Stop the DataNode: ./stop.dfs.sh
  2. Edit the value of namespaceID in /current/VERSION to match the value of the current NameNode.
  3. Restart the DataNode: ./start.dfs.sh

hadoop java.net.UnknownHostException: ubuntu: ubuntu

Solution: 

1. Add ubuntu as your localhost IP to your /etc/hosts file: sudo vi /etc/hosts

2. Restart your network: sudo /etc/init.d/networking restart

So your /etc/hosts file on your master machine will look something like this:

172.16.62.152      master
172.16.62.151      slave
172.16.62.152      ubuntu

On your slave machine

172.16.62.152      master
172.16.62.151      slave
172.16.62.151      ubuntu

If none of it works then you can go and change the master/localhost hostname to the ipaddress in core-site.xml, mapred-site.xml

Install WebSphere Application Server V7 on Ubuntu

If you want to install WAS V7 on Ubuntu the IBM Installation Manager will show you the follow error:

Ubuntu Linux is not supported.

Follow the steps below to install:

Step 1. Change from dash to bash

sudo ln -sf /bin/bash /bin/sh

Step 2: Open /etc/lsb-release and change the following:

DISTRIB_ID=Ubuntu                         to                     DISTRIB_ID=LVMUbuntu

Step 3: Recheck the status of IBM Installation Manager and Install

Remeber to change DISTRIB_ID back to Ubuntu (DISTRIB_ID=Ubuntu )

Cloud computing in South Africa

cloud-computingThe world is wiating for the big reveal from big companies like IBM and Microsoft to reveal their big cloud solution, but will South Africa really feel the affect of the cloud technology or will we be left in the dark until our telco infrastructure improves?Is there a way we can create our own cloud and start selling it to the world. Thankfully there is already a open source cloud operating system that looks awesome but needs some time for all the bugs to be fixed, and it is know as EyeOS. It is a open source operating system solution that runs on the LAMP stack or WAMP if you are STILL using Windows. There is new applications being developed for the OS which is increasing the range of different applications for different jobs. Thus you can create different profiles for specific jobs, which means no more redundant apps on your PC. 

This fits all together with SaaS (Software as a Service) where software will run on any independant platform and execute a service which will be recieved by a user or another application. XML services is the best platform independant services that exist at the moment because it is only text and can be executed by almost any node. This means we can create a cloud OS that connects to these services to execute on-demand on different actions that is needed. This is the what SOA is all about!

There is also some  other Open Source Cloud solutions that is now being implemented by Netbooks. They include gOs and eeebuntu where you can access a cloud OS from your browser. This means you don’t need any OS just a plain browser and an internet connection. This is the way of the future.

We are looking for techies that wants to help with the development of a Open Source cloud solution or have any more information about what technolgies is outside that we can use to build a fully fledge cloud solution.