This configuration defines a single agent named a1. a1 has a netcat source that listens for data on port 44444, a channel that buffers event data in memory, and a sink that logs event data to the console.
$ cd apache-flume-1.7.0-bin
$ bin/flume-ng agent --conf conf --conf-file conf/netcat_source.conf --name a1 -Dflume.root.logger=INFO,console
On local mac
$ flume-ng agent -f ./netcat_source.conf --name a1
From a separate terminal, we can then telnet port 44444 and send Flume an event:
$ telnet localhost 44444
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
hello
OK
world
OK
The original Flume terminal will output the event in a log message.
2017-01-13 10:59:19,955 (lifecycleSupervisor-1-0) [INFO - org.apache.flume.source.NetcatSource.start(NetcatSource.java:169)] Created serverSocket:sun.nio.ch.ServerSocketChannelImpl[/127.0.0.1:44444]
2017-01-13 10:59:57,913 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 68 65 6C 6C 6F 0D hello. }
2017-01-13 11:00:00,483 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.process(LoggerSink.java:95)] Event: { headers:{} body: 77 6F 72 6C 64 0D world. }
In this example, we use a exec source to cat content of a csv file, then write the lines as events to a hive table, partitioned by date.
$ cd apache-flume-1.7.0-bin
$ bin/flume-ng agent --conf conf --conf-file conf/hive.conf --name agent1 -Dflume.root.logger=INFO,console
Next start a hive shell, then
create table test(id string, message string)
PARTITIONED BY (day int)
ROW FORMAT DELIMITED
FIELDS TERMINATED BY ','
LOCATION '/hive/mydb.db/test';
hive> ALTER TABLE test
> ADD PARTITION (day = <today>);
put <today> like 20170113.
Then,
hive> select * from test;
You should see something like this:
1 AAAAAAA 20170113
2 BBBBBBB 20170113
3 CCCCCCCC 20170113
4 DDDDDD 20170113
5 EEEEEEE 20170113
6 FFFFFFFFFFF 20170113
7 GGGGGG 20170113
8 HHHHHHH 20170113
9 IIIIII 20170113
10 JJJJJJ 20170113
11 KKKKKK 20170113
12 LLLLLLLL 20170113
13 MMMMMMMMM 20170113
14 NNNNNNNNN 20170113
15 OOOOOOOO 20170113
16 PPPPPPPPPP 20170113
17 QQQQQQQ 20170113
18 RRRRRRR 20170113
19 SSSSSSSS 20170113
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent