Logstash Logspout

summary_2018/09 docker tools

1、日常

1.1、docker:ELK_Logstash

2.1、docker:Logspout

2、技术

2.1、docker:ELK_Logstash

2.1.1、简介

• Logstash是一个具有实时流水线功能的开源数据收集引擎，Logstash可以动态地统一来自不同数据源的数据，并将数据规范化到您所选择的目的地，对于各种高级的下游分析和可视化用例清理和统一化所有的数据。
• Logstash管道有两个必需的元素，输入和输出，以及一个可选的元素，过滤器。输入插件使用来自源的数据，过滤器插件根据您的指定修改数据，输出插件将数据写入目的地。
  

2.1.2、主要功能

• Inputs(inputs to get data into Logstash)
  
  • 支持多种输入方式，比如file、syslog、redis、beats等等
• Filters
  
  • 过滤器是Logstash管道中的中间处理设备，用于过滤或者替换修改，比如grok、mutate、drop、clone等等
• Outputs
  
  • 支持多种输出方式，比如es，file等等，可以输出到多个地方
• 具体参考文档

2.1.3、简单使用

• 简单使用：见下文

2.1.4、官方参考及详细文档

• 见文档

2.1、Logspout

2.1.1、简介

• Logspout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system.（负责日志转发，附着于所有的container上，可以仅转发指定container，也可以转发到多个目的地）
• Logspout is a log router for Docker containers that runs inside Docker. It attaches to all containers on a host, then routes their logs wherever you want. It also has an extensible module system.（无状态的，仅用于日志的转发）
• logspout will gather logs from other containers that are started without the -t option and are configured with a logging driver that works with docker logs (journald and json-file).（可以docker logs一起工作）
• Tips:Logspout依赖于Docker API来检索容器日志。API中的失败可能导致日志流挂起。

2.1.2、简单使用

• 官方参考

// 启动logspout，及两个http服务
docker run -d --name="logspout" \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    -p 8000:80 \
    gliderlabs/logspout
// 查看logspout收集到的日志 curl http://127.0.0.1:8000/logs
dazzling_hoover| * Serving Flask app "app" (lazy loading)
 dazzling_hoover| * Environment: production
 dazzling_hoover|   WARNING: Do not use the development server in a production environment.
 dazzling_hoover|   Use a production WSGI server instead.
 dazzling_hoover| * Debug mode: off
 dazzling_hoover| * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
 dazzling_hoover|172.17.0.1 - - [03/Sep/2018 12:09:10] "GET / HTTP/1.1" 200 -
 dazzling_hoover|172.17.0.1 - - [03/Sep/2018 12:09:10] "GET /favicon.ico HTTP/1.1" 404 -
 dazzling_hoover|172.17.0.1 - - [03/Sep/2018 12:11:16] "GET / HTTP/1.1" 200 -
compassionate_kalam| * Serving Flask app "app" (lazy loading)
compassionate_kalam| * Environment: production
compassionate_kalam|   WARNING: Do not use the development server in a production environment.
compassionate_kalam|   Use a production WSGI server instead.
compassionate_kalam| * Debug mode: off
compassionate_kalam| * Running on http://0.0.0.0:80/ (Press CTRL+C to quit)
    dazzling_hoover|172.17.0.1 - - [03/Sep/2018 12:28:34] "GET / HTTP/1.1" 200 -
compassionate_kalam|172.17.0.1 - - [03/Sep/2018 12:29:29] "GET / HTTP/1.1" 200 -
compassionate_kalam|172.17.0.1 - - [03/Sep/2018 12:29:30] "GET /favicon.ico HTTP/1.1" 404 -

2.1.2、转发到Logstash

• 官方Logstash docker 使用参考
• 使用Dockerfile的方式修改Logstash的配置

// Dockerfile修改pipeline/logstash.conf
FROM docker.elastic.co/logstash/logstash:6.4.0
#自定义输入、输出流
RUN rm -f /usr/share/logstash/pipeline/logstash.conf
ADD logstash.conf  /usr/share/logstash/pipeline/
// logstash.conf
input {
  udp {
  port => 5000
  type => syslog
  }
}
// 过滤筛选条件‘’
filter {
}
output {
  file {
    path => "~/logspout.log"
  }
}
// 启动loglogspout收集并转发日志
docker run --name="logspout" \
    --volume=/var/run/docker.sock:/var/run/docker.sock \
    gliderlabs/logspout \
    syslog://172.17.0.6:5000
# 172.17.0.6:5000是Logstash的地址,及监听端口
// logspout.log中的日志，还有一些ES检测的错误信息没有放上去
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:09:10] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.051Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:09:10] \"GET /favicon.ico HTTP/1.1\"404 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.052Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:11:16] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.052Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:28:34] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.052Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:45:33] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.052Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:45:54] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.052Z","type":"syslog","host":"172.17.0.3"}
{"message":"<11>1 2018-09-04T05:53:12Z 4f971db7b200 dazzling_hoover 8234 - - 172.17.0.1 - - [03/Sep/2018 12:46:31] \"GET / HTTP/1.1\" 200 -\n","@version":"1","@timestamp":"2018-09-04T05:53:12.053Z","type":"syslog","host":"172.17.0.3"}

2.1.3、参考及详细文档

• 官方文档