Using Spring XD to stream Tweets to Hadoop

Aman Kumar

1 – Download and Install Spring-XD

Spring-XD can be found at http://spring.io. This tutorial uses the 1.0.0.M3 version, so conventions may change in next release. Follow the install instructions and kick up Spring-XD with a test stream to make sure it’s looking good.

create stream --name ticktock --definition "Time | Log"

That simple instruction should begin showing output in the server terminal window similar to:

2013-10-12 17:18:09
2013-10-12 17:18:10
2013-10-12 17:18:11
2013-10-12 17:18:12
2013-10-12 17:18:13
2013-10-12 17:18:14

Congrats, Spring-XD is running.

2 – Download and Install Hadoop

The hadoop can be installed on the singlenode by using the steps mentioned in my previous blog at https://amankumarlabra.wordpress.com/2014/08/09/installing-single-node-hadoop-2-4-1-on-ubuntu/

Configuring Spring-XD to use Hadoop :

Step 1 – Edit the Hadoop.properties file Edit the file at XD_HOMExdconfighadoop.properties to enter the namenode config:

fs.default.name=hdfs://10.0.0.27:8020

Step 2 – Spin up the Spring-XD Service with Hadoop

In a terminal window get the server running from…

View original post 501 more words