Working with HDFS Java Example

This is a java shows how we can work with the Hadoop File System.

Prerequisite for using the code in Eclipse is that you download and add the following jars to your project libraries:

•  hadoop-core-0.20.2.jar
• commons-logging-*.jar

See comments in code:

import java.io.IOException;
//hadoop imports
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;

/**
 * @author thysmichels
 *
 */
public class HDFSWordCounter {

	//change this to string arg in main
	public static final String inputfile = "hdfsinput.txt";
	public static final String inputmsg = "Count the amount of words in this sentence!\n";
	/**
	 * @param args
	 */
	public static void main(String [] args) throws IOException {
		// Create a default hadoop configuration
		Configuration config = new Configuration();
                // Parse created config to the HDFS 
		FileSystem fs = FileSystem.get(config);
		// Specifies a new file in HDFS.
		Path filenamePath = new Path(inputfile);

		try
		{
                        // if the file already exists delete it.
			if (fs.exists(filenamePath))
			{
				//remove the file
				fs.delete(filenamePath, true);
			}

                //FSOutputStream to write the inputmsg into the HDFS file 
		FSDataOutputStream fin = fs.create(filenamePath);
		fin.writeUTF(inputmsg);
		fin.close();

                //FSInputStream to read out of the filenamePath file
		FSDataInputStream fout = fs.open(filenamePath);
		String msgIn = fout.readUTF();
                //Print to screen
		System.out.println(msgIn);
		fout.close();
		}
		catch (IOException ioe)
		{
			System.err.println("IOException during operation " + ioe.toString());
			System.exit(1);
		}
	}
}</pre>

In this example we created a HDFS Configuration, specified a Path for our file, Read string to our file and read string out of our file using the HDFS library.

Play around with this to solve more intricate problems.