WaterDrop问题记录-2017.08.14

WaterDrop

2017.08.14

1. events遍历问题， 每个插件遍历一次events？

   // WaterDropMain.scala
   for (f <- filters) {
       val (processedEvents, isSuccess) = f.filter(events)
       events = processedEvents
   }
   // Split.scala
   for (i <- 0 until events.length) {
       for (v <- events(i).getField(srcField)) {
       }
   }

2. 提供支持删除source_field的配置

   // WaterDropMain.scala
   val parts = v.asInstanceOf[String].split(conf.getString("delimiter")).map(_.trim)
   val kvs = (keys zip parts).toMap
   events(i).setField(conf.getString("target_field") kvs)

3. beforeOutput和afterOutput为什么放在inputs模块里

   // WaterDropMain.scala
   inputs.head.beforeOutput
   eventRDD.foreachPartition { partitionOfRecords =>
       outputs.head.process(partitionOfRecords)
   }
   inputs.head.afterOutput

4. 多日志源支持

   // WaterDropMain.scala
   val dstream = inputs.head.getDstream.mapPartitions{ iter => ...

5. Interval配置问题

   // WaterDropMain.scala
   val ssc = new StreamingContext(sparkConf, Seconds(15))