@chyoo1991
2017-01-06T06:22:44.000000Z
字数 3189
阅读 945
入库任务整理
入库
优酷
整理目前的入库任务情况。
81 Machine
10.103.23.81
01. [No]AlgRecvInserter.sh
- sh /home/arc/insert/script/AlgRecvInserter.sh yk/td
- 从路径上看属于arc小队
- InterveInser.sh 最开始被执行
- 要Merge/CombineFile/然后查到过去的文件进行DIFF
- 入Hbase && MC
- 生成一些DIFF文件并存储
- 没有迁移 比较复杂
- hdfs://youkuDfs/workspace/dm/arc/project/insert/youku_recommend/workingtmp/common/comsimtable/COMBINE_CMT_DIFF/20170103/part-r-00079
- YK_REC_VIDEO family:v
- MC: 123~127 30006/30007/40006/40007
02. [No]AlgRecvInserterAssurance.sh yk
- sh /home/arc/insert/script/AlgRecvInserterAssurance.sh yk
- 从路径上看也是架构小队
- 执行过程同 AlgRecvInserter.sh
- 最近几天没有数据 没有入库
- /workspace/dm/arc/project/insert/youku_recommend/workingtmp/lr1_algidnew/update_20170103
03. [No]SingleAlgInserter.sh yk/td
04. [No]RecvRealtimeInserter.sh td
- 都是土豆的任务
- 挺复杂,中间调用了 SingleAlgInserter.sh 进行数据入库
- hdfs://youkuDfs/workspace/dm/arc/project/insert/tudou_recommend/workingtmp/common/comsimtable/COMBINE_CMT/cf026_2017010322/part-r-00004
- 入MC和Hbase
05. [No]RecvHourInserter.sh td
06. [Do]UploaderVideoTmpInserter.sh
- /workspace/dm/rec/result/youku_recommend/uservideo/20170104/
- YK_UPLOADER_VIDEO family:v
- 已经迁移,但目前用的是测试表 YK_UPLOADER_VIDEO_TEST
- 老任务还没停
- 在Uploader目录
07. [Do]EWMapInserter.sh yk cv/vc/cc
- /workspace/dm/rec/result/youku_recommend/easywatch/VideoCircleMap/ [长期]
- /workspace/dm/rec/result/youku_recommend/easywatch/CircleToCircle/ [已废弃]
- /workspace/dm/rec/result/youku_recommend/easywatch/CircleVideoMap/ [长期]
- 已经迁移
- 回头检查一下
08. [No]ProgramInserter.sh yk/td ...
09. [No]GameGuessTagidToTagInserter.sh yk ...
10. [No]GameInserter.sh yk gr002 tag_game 0430
- YK_REC_GAME family:v
- s/workspace/dm/arc/project/insert/youku_recommend/workingtmp/game_ins/gr002tag_game_algidnew201609300430/
- 还入 MC
- workspace/dm/rec/result/youku_recommend/common/simtable/game/GR002_CMT/tag_game/
- /workspace/dm/rec/result/youku_recommend/common/simtable/game/GR002_CMT/tag_game/201609300430
- /workspace/dm/arc/project/insert/youku_recommend/workingtmp/game_ins/gr002tag_game_algidnew201609300430
- /youkuDfs/workspace/dm/rec/result/youku_recommend/uploader/personal/ [先没找到]
12. [No]IdmInserter.sh td
- 土豆正副本数据
- /workspace/dm/rec/project/tudou_recommend/common/data/zfb/
13. [No]PushDevice.sh
- /workspace/dm/per/result/push/youku/wireless_token_guidOuid/
- 滕飞负责,不知道会不会停掉
68 Machine
PlayCard数据迁移
ProgramInserter.sh newshowshow
HotspotInserter.sh
有Python写的两个任务
优酷审核三个任务
6.VideoQualityInserter.sh td/yk
- YK_VIDEO_QUALITY_ALL family:v
- 土豆部分没有数据
- 优先迁移优酷
220 Machine
RecV3AlgsInserter.sh
HighQualityVideoMetaInserter.sh
RecVRInserter.sh
- /workspace/dm/rec/result/youku_recommend/common/simtable/VR_CMT/Relate/ 路径有点熟悉
- 找找
PlaycardVideoInserter.sh yk