flume 初步调研

flume

介绍

满足我们需求有
1，tail 监控日志
2，sdk支持传输输入日志
3，支持直接转换成hdfs
4，thrift数据源
等等

官方网站：http://flume.apache.org/
官方指导：http://flume.apache.org/FlumeUserGuide.html

支持数据源列表

Avro Source
Thrift Source
Exec Source
JMS Source
Converter
Spooling Directory Source
Event Deserializers
LINE
AVRO
BlobDeserializer
Twitter 1% firehose Source (experimental)
Kafka Source
NetCat Source
Sequence Generator Source
Syslog Sources
Syslog TCP Source
Multiport Syslog TCP Source
Syslog UDP Source
HTTP Source
JSONHandler
BlobHandler
Stress Source
Legacy Sources
Avro Legacy Source
Thrift Legacy Source
Custom Source
Scribe Source

测试

在10.10.5.11:/opt/apache-flume 目录下

收集数据

配置

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure tail -F source1
agent1.sources.source1.type = exec 
agent1.sources.source1.command = tail -F /data/log/jiemo-api/access.log#监控日志
agent1.sources.source1.channels = channel1
agent1.sources.source1.batchSize = 1
#configure host for source
agent1.sources.source1.interceptors = i1
agent1.sources.source1.interceptors.i1.type = host
agent1.sources.source1.interceptors.i1.hostHeader = hostname
# Describe avro  sink1  
agent1.sinks.sink1.type = avro
agent1.sinks.sink1.hostname = 10.10.5.11
agent1.sinks.sink1.port = 4545
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 0
agent1.channels.channel1.capacity = 500000
agent1.channels.channel1.transactionCapacity = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1

开启

bin/flume-ng agent -c conf -f conf/jiemo-api.properties  -n agent1 -Dflume.root.logger=DEBUG,console

接收方

配置【现在是文件保存】

agent1.sources = source1
agent1.sinks = sink1
agent1.channels = channel1
# Describe/configure avro source
agent1.sources.source1.type = avro
agent1.sources.source1.bind = 0.0.0.0
agent1.sources.source1.port = 4545
# Describe logger sink1
#agent1.sinks.sink1.type = logger
# Describe file sink1
agent1.sinks.sink1.type = file_roll
agent1.sinks.sink1.sink.directory = /var/log/flume #存放数据位置
# Use a channel which buffers events in memory
agent1.channels.channel1.type = memory
agent1.channels.channel1.keep-alive = 1
agent1.channels.channel1.capacity = 5000
agent1.channels.channel1.transactionCapacity = 100
# Bind the source and sink to the channel
agent1.sources.source1.channels = channel1
agent1.sinks.sink1.channel = channel1

开启

bin/flume-ng agent -c conf -f conf/jiemo-api-avro.properties  -n agent1 -Dflume.root.logger=DEBUG,console