Deploy Apache Flume NG (1.x.x)

In this tutorial I have explained how to install / deploy / configure Flume NG on single system, and how to configure Flume NG to copy data to HDFS, then configuration for copying data to HBase

Before going to configurations let’s understand what Flume NG (1.x.x) is:

Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log data. It has a simple and flexible architecture based on streaming data flows. It is robust and fault tolerant with tunable reliability mechanisms and many failover and recovery mechanisms. The system is centrally managed and allows for intelligent dynamic management. It uses a simple extensible data model that allows for online analytic application.

In this tutorial I have used:
Hadoop 0.20.X (Apache or Cloudera)
Flume 1.1.0 
HBase 0.90.4
Ubuntu 11.10

I am assuming that you have Hadoop cluster ready, if you don’t have Hadoop installed you can refer:

• For Hadoop in pseudo distributed mode please click here
• For Hadoop in distributed mode please click here

Let’s Deploy Flume NG:
Download Flume NG from 
https://www.apache.org/dyn/closer.cgi/incubator/flume/flume-1.1.0-incubating/
Or
http://archive.cloudera.com/cdh4/cdh/4/

Untar:

tar xzf flume-1.1.0-cdh4.0.0.tar.gz

Copy configuration file

cp conf/flume-conf.properties.template conf/flume.conf cp conf/flume-env.sh.template conf/flume-env.sh

Configuration:
Following are the Configurations for copy data from file on local file system to HDFS
Edit flume.conf and add following entries:

agent1.sources = tail agent1.channels = Channel-2 agent1.sinks = HDFS agent1.sources.tail.type = exec agent1.sources.tail.command = tail -F /var/log/apache2/access.log agent1.sources.tail.channels = Channel-2 agent1.sinks.HDFS.channel = Channel-2 agent1.sinks.HDFS.type = hdfs agent1.sinks.HDFS.hdfs.path = hdfs://localhost:9000/flume agent1.sinks.HDFS.hdfs.fileType = DataStream agent1.channels.Channel-2.type = memory

Start Flume to copy data to HDFS:

bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n agent1

Note that the agent name is specified by -n agent1 and must match a agent name given in -f conf/flume.conf

You can check your data in HDFS either from web console (http://localhost:50070) or command prompt


Now lets configure Flume NG to copy data to HBase:
Here I am assuming that you have HBase installed and ready

Create table in HBase in which you want to copy data

create 'myTab', 'cf'

Edit flume.conf and add following entries:

hbase-agent.sources=tail hbase-agent.sinks=sink1 hbase-agent.channels=ch1 hbase-agent.sources.tail.type=exec hbase-agent.sources.tail.command=tail -F /tmp/test05 hbase-agent.sources.tail.channels=ch1 hbase-agent.sinks.sink1.type=org.apache.flume.sink.hbase.HBaseSink hbase-agent.sinks.sink1.channel=ch1 hbase-agent.sinks.sink1.table=myTab hbase-agent.sinks.sink1.columnFamily=cf hbase-agent.sinks.sink1.column=c1 hbase-agent.sinks.sink1.serializer= org.apache.flume.sink.hbase.SimpleHbaseEventSerializer hbase-agent.sinks.sink1.serializer.payloadColumn=coll hbase-agent.sinks.sink1.serializer.incrementColumn=coll hbase-agent.sinks.sink1.serializer.rowPrefix=1+ hbase-agent.sinks.sink1.serializer.suffix=timestamp hbase-agent.channels.ch1.type=memory

Set Following variables in .bashrc

export HADOOP_HOME=YOUR-PATH export HADOOP_PREFIX=YOUR-PATH export PATH==YOUR-PATH:$PATH export FLUME_HOME=YOUR-PATH export FLUME_CONF_DIR=YOUR-PATH export HBASE_HOME=YOUR-PATH export CLASSPATH=$CLASSPATH:HBASE_HOME/conf:HADOOP_HOME/conf

Now start Flume to copy data to HBase:

bin/flume-ng agent --conf ./conf/ -f conf/flume.conf -n hbase-agent

Check whether data gets copy into HBase or not

scan 'myTab'

Note That SimpleHbaseEventSerializer specified in the configuration is just an example of Event Serializer you should write your own class depend on your requirements