flume+kafka+storm

flume kafka storm

flume+kafka+storm 实现大批量日志实时分析处理

部署

* apache 一系列 全面兼容，网址看着就爽 *

kafka，flume，storm 安装官方要求直接下载源码 直接可以运行

storm 安装暂时忽略 以前已经说明了

flume 收集日志配置[修改配置不用重启服务]

agent1.sources = source1
agent1.channels = channel1
##使用 tail命名监控日志 建议使用-F
agent1.sources.source1.type = exec 
agent1.sources.source1.command = tail -Fn 0 /data/log/jiemo-api/access.log
agent1.sources.source1.channels = channel1
agent1.sources.source1.batchSize = 1
##开启kafka传输
agent1.channels.channel1.type =  org.apache.flume.channel.kafka.KafkaChannel
agent1.channels.channel1.keep-alive = 15
agent1.channels.channel1.capacity = 500
agent1.channels.channel1.zookeeperConnect=zk.d.jiemoapp.com:2181
agent1.channels.channel1.brokerList=10.10.5.11:7092
agent1.chnnnels.channel1.topic= zkkafkaspout-topic
agent1.channels.channel1.transactionCapacity = 100

然后启动

bin/flume-ng agent -c conf -f conf/jiemo-api.properties  -n agent1 -Dflume.root.logger=DEBUG,console

开启kafka

其中server.properties配置修改如下[内网9092端口被我们自己用了，开始把我坑住了]

……
listeners=PLAINTEXT://:7092
zookeeper.connect=zk.d.jiemoapp.com:2181
……

开启服务

bin/zookeeper-server-start.sh config/server.properties

如果没有topic 需要新建

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic zkkafkaspout-topic 

对接storm

见 jiemo-storm develop 分支

其中config 变成了 flume-channel 这个有点郁闷 以为是自己去的名字zkkafkaspout_topic

开始测试

在10.10.5.11 运行2个flume agent 【其实一台服务器可以合并，这里做测试用】

bin/flume-ng agent -c conf -f conf/jiemo-api.properties  -n agent1 -Dflume.root.logger=DEBUG,console ##监控api access log
bin/flume-ng agent -c conf -f conf/jiemo-admin.properties  -n agent1 -Dflume.root.logger=DEBUG,console ##监控admin access log

然互开启kafka
运行 jiemo-storm 的 LogStatTopology 本地模式

在本机本地运行
打印log发现 是实时收集的api 和 admin log 如下 [开头乱码 稍后解决]

能收集到log 那么随后就可以分析了 嘎嘎嘎

该套日志分析优点是：与我们服务完全独立，我们可以随时的更改统计而不影响我们的在线服务

�2016-01-28 20:00:59.240|11268|POST|http://api798.jiemoapp.com/api/matchFriend/entity?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|20
�2016-01-28 20:01:03.124|11268|POST|http://api798.jiemoapp.com/api/chat/whisper/text?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|40
�2016-01-28 20:01:03.203|11268|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|4
�2016-01-28 20:01:03.345|11269|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|4
�2016-01-28 20:01:05.128|11269|POST|http://api798.jiemoapp.com/api/chat/whisper/text?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|30
�2016-01-28 20:01:05.211|11269|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|4
�2016-01-28 20:01:05.417|11268|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|4
�2016-01-28 20:01:06.274|1|GET|http://admin.jiemo.com/zookeeper|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36|10.10.7.200|null|1.1.6|null
�2016-01-28 20:01:07.104|1|GET|http://admin.jiemo.com/zookeeper|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36|10.10.7.200|null|1.1.6|null
�2016-01-28 20:01:07.964|1|GET|http://admin.jiemo.com/zookeeper|Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.111 Safari/537.36|10.10.7.200|null|1.1.6|null
�2016-01-28 20:01:10.058|11268|POST|http://api798.jiemoapp.com/api/chat/whisper/text?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|30
�2016-01-28 20:01:10.196|11268|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|4
�2016-01-28 20:01:10.220|11269|POST|http://api798.jiemoapp.com/api/chat/read/normal?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|4
�2016-01-28 20:01:12.133|11269|POST|http://api798.jiemoapp.com/api/chat/whisper/disappear/prep?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|5
�2016-01-28 20:01:12.592|11268|POST|http://api798.jiemoapp.com/api/chat/whisper/disappear/prep?appver=1.1.6&channel=0&p=7QL6Tx6ijYKM63IdbXtP2QLb-nFjsAKY0b7gT6a-V67ecWH_q2psfQkHtx1NGJ0X&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 8.0; Scale/2.00)|10.10.7.102|iOS|1.1.6|0|5
�2016-01-28 20:01:12.791|11269|POST|http://api798.jiemoapp.com/api/chat/whisper/disappear/prep?appver=1.1.6&channel=0&p=SGY1378yVpDB94wwTftBR0BXqP75KYA9E9AnRlrHaRrQTIsl006QGMIwahFqjg-R&platform=ios&v=jiemo|JiemoInHouse/1.1.6 (iPod touch; iOS 9.0.2; Scale/2.00)|10.10.7.98|iOS|1.1.6|0|4

```