Working with HDFS Java Example

This is a java shows how we can work with the Hadoop File System.

Prerequisite for using the code in Eclipse is that you download and add the following jars to your project libraries:

  •  hadoop-core-0.20.2.jar
  • commons-logging-*.jar

See comments in code:

import java.io.IOException;
//hadoop imports
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;

/**
 * @author thysmichels
 *
 */
public class HDFSWordCounter {

	//change this to string arg in main
	public static final String inputfile = "hdfsinput.txt";
	public static final String inputmsg = "Count the amount of words in this sentence!\n";
	/**
	 * @param args
	 */
	public static void main(String [] args) throws IOException {
		// Create a default hadoop configuration
		Configuration config = new Configuration();
                // Parse created config to the HDFS 
		FileSystem fs = FileSystem.get(config);
		// Specifies a new file in HDFS.
		Path filenamePath = new Path(inputfile);

		try
		{
                        // if the file already exists delete it.
			if (fs.exists(filenamePath))
			{
				//remove the file
				fs.delete(filenamePath, true);
			}

                //FSOutputStream to write the inputmsg into the HDFS file 
		FSDataOutputStream fin = fs.create(filenamePath);
		fin.writeUTF(inputmsg);
		fin.close();

                //FSInputStream to read out of the filenamePath file
		FSDataInputStream fout = fs.open(filenamePath);
		String msgIn = fout.readUTF();
                //Print to screen
		System.out.println(msgIn);
		fout.close();
		}
		catch (IOException ioe)
		{
			System.err.println("IOException during operation " + ioe.toString());
			System.exit(1);
		}
	}
}</pre>

In this example we created a HDFS Configuration, specified a Path for our file, Read string to our file and read string out of our file using the HDFS library.

Play around with this to solve more intricate problems.

Advertisements

2 Comments on “Working with HDFS Java Example

  1. can you please explain, if the file has multiple lines will it read and store it in string array .

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: