Working with HDFS Java Example

This is a java shows how we can work with the Hadoop File System.

Prerequisite for using the code in Eclipse is that you download and add the following jars to your project libraries:

  •  hadoop-core-0.20.2.jar
  • commons-logging-*.jar

See comments in code:

//hadoop imports
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.FSDataInputStream;
import org.apache.hadoop.fs.FSDataOutputStream;
import org.apache.hadoop.fs.Path;

 * @author thysmichels
public class HDFSWordCounter {

	//change this to string arg in main
	public static final String inputfile = "hdfsinput.txt";
	public static final String inputmsg = "Count the amount of words in this sentence!\n";
	 * @param args
	public static void main(String [] args) throws IOException {
		// Create a default hadoop configuration
		Configuration config = new Configuration();
                // Parse created config to the HDFS 
		FileSystem fs = FileSystem.get(config);
		// Specifies a new file in HDFS.
		Path filenamePath = new Path(inputfile);

                        // if the file already exists delete it.
			if (fs.exists(filenamePath))
				//remove the file
				fs.delete(filenamePath, true);

                //FSOutputStream to write the inputmsg into the HDFS file 
		FSDataOutputStream fin = fs.create(filenamePath);

                //FSInputStream to read out of the filenamePath file
		FSDataInputStream fout =;
		String msgIn = fout.readUTF();
                //Print to screen
		catch (IOException ioe)
			System.err.println("IOException during operation " + ioe.toString());

In this example we created a HDFS Configuration, specified a Path for our file, Read string to our file and read string out of our file using the HDFS library.

Play around with this to solve more intricate problems.


2 Comments on “Working with HDFS Java Example

  1. can you please explain, if the file has multiple lines will it read and store it in string array .

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: