一个spark streaming+Cassandra、分布式作业调度系统总结.pdf

发布时间：2022-05-29 发布人：admin 分类：说明书资料大小：4.51M 资料格式：pdf 举报版权申诉

tbanhui-9777320-4744302543452098914.pdf-第1页.png

第1页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第2页.png

第2页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第3页.png

第3页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第4页.png

第4页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第5页.png

第5页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第6页.png

第6页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第7页.png

第7页 / 共29页

tbanhui-9777320-4744302543452098914.pdf-第8页.png

第8页 / 共29页

文本预览

ShenCe Introduction Dec. 2016

1 2 3 ShenCe Introduction Web R&D’s New Plays Big Data R&D’s New Plays

General Introduction l Job Scheduling: GUI, cluster based, dynamic, distributed, relible job mgmt & scheduling service l Intrusion Detection: User account & data protection; Account platform protection, 3 levels l Real-time Monitor：Access count, success rate, performance of any Nginx based service , 3 levels Data Collect Login Data Distributed Scripts Web Server Log Data Analysis Job Dispatch Real-time Act Intrusion Detection Job Scheduling Real-time Monitor GUI ShenCe Web Automatic and Intelligent Platform p Main Advantages ü Good commonality (Nginx, Apache, …) ü No any influence to online service ü Good real-time (Not more than 5 seconds) ü 3 levels monitor (Cluster -> URL -> IP) p Current Status ü Has been deployed in production environment ü Scheduling jobs in 3 machine rooms ü Real-time detect, notify to intercept IP & protect data

Resources & Progress Total Human Resources: R&D: 17 in all, 8 interns, 2 formal employees, 7 part time employees Operations: 3 employees, part time Jul. Aug. ~ Sep. Oct. ~ Nov. Dec. l Evaluation, planning from middle July l 6 interns on July 20 l 2 interns and I started developing job scheduling part on July 23 l Finish job related R&D l Deployed in cloud, schedule job in two MRs l Begin focus on spark related parts in Sep. l 7 people joined in Sep., 1 internal l Set up R&D ENV with CLI + Maven l Finish spark related R&D and optimization l Deployed in cloud & physical machine ENV l Connect to other services in cloud ENV l 1 of 3 people 100% joined on Oct. 27 l Break through limitations of CN2 in proxy l Connect with other Dec. 21 services in PHYSICAL ENV l Requirements adjusting & adding l 1 intern joined on

3 Spark Apps Total Memory: 80Gb~120Gb NginxLogAnalyzer PopLoginAnalyzer WebLoginAnalyzer

2 Web Apps Intelligent tip & no SQL like* query

Job Mgmt, Scheduling, Execution Monitor

Real-time Intrusion Detection 1 Make Intrusion Intuitionistic

分享到：

赞收藏

资料库

一个spark streaming+Cassandra、分布式作业调度系统总结.pdf

相关推荐

开发技术

热门标签

最新资料