第1页 / 共167页
第2页 / 共167页
第3页 / 共167页
第4页 / 共167页
第5页 / 共167页
第6页 / 共167页
第7页 / 共167页
第8页 / 共167页
Adaptive real-time anomaly detection for safeguarding critical networks.pdf
001_003_lic-tfk-titelsid.en_kalbu.pdf
005_lic-tfk-abstract.en_kalbu.pdf
163_lic-tfk-bibblad.en_kalbu.pdf
165_Lic-sammanst.pdf
Linköping Studies in Science and Technology
Thesis No. 1231
Adaptive Real-time Anomaly Detection
for Safeguarding Critical Networks
by
Kalle Burbeck
Submitted to Linköping Institute of Technology at Linköping University in partial
fulfilment of the requirements for the degree of Licentiate of Engineering
Department of Computer and Information Science
Linköpings universitet
SE-581 83 Linköping, Sweden
Linköping 2006
Adaptive Real-time Anomaly Detection for Safeguarding
Critical Networks
by
Kalle Burbeck
February 2006
ISBN 91-85497-23-1
Linköping Studies in Science and Technology
Thesis No. 1231
ISSN 0280-7971
LiU-Tek-Lic-2006:12
ABSTRACT
Critical networks require defence in depth incorporating many different security technologies
including intrusion detection. One important intrusion detection approach is called anomaly
detection where normal (good) behaviour of users of the protected system is modelled, often
using machine learning or data mining techniques. During detection new data is matched
against the normality model, and deviations are marked as anomalies. Since no knowledge of
attacks is needed to train the normality model, anomaly detection may detect previously
unknown attacks.
In this thesis we present ADWICE (Anomaly Detection With fast Incremental Clustering)
and evaluate it in IP networks. ADWICE has the following properties:
(i) Adaptation - Rather than making use of extensive periodic retraining sessions on stored
off-line data to handle changes, ADWICE is fully incremental making very flexible on-line
training of the model possible without destroying what is already learnt. When subsets of the
model are not useful anymore, those clusters can be forgotten.
(ii) Performance - ADWICE is linear in the number of input data thereby heavily reducing
training time compared to alternative clustering algorithms. Training time as well as detection
time is further reduced by the use of an integrated search-index.
(iii) Scalability - Rather than keeping all data in memory, only compact cluster summaries are
used. The linear time complexity also improves scalability of training.
We have implemented ADWICE and integrated the algorithm in a software agent. The
agent is a part of the Safeguard agent architecture, developed to perform network monitoring,
intrusion detection and correlation as well as recovery. We have also applied ADWICE to
publicly available network data to compare our approach to related works with similar
approaches. The evaluation resulted in a high detection rate at reasonable false positives rate.
This work has been supported by the European project Safeguard IST-2001-32685 and
CENIIT (Center for Industrial Information Technology) at Linköping University.
Department of Computer and Information Science
Linköpings universitet
SE-581 83 Linköping, Sweden
Acknowledgement
First of all I would like to thank Simin Nadjm-Tehrani, my advisor. Without your
guidance and support, this work would not have been possible. I am also grateful
for all the fun we have had together during the Safeguard project. Too bad I did
not take a picture when we exited the subway in Barcelona. Or when the storm
forced us to sleep on the floor at a London airport and we experienced an overload
of a critical communication infrastructure first hand when everybody tried to call
home.
Thanks to all colleges at RTSLAB for discussions and support. Keep the
fika going or I will be forced to haunt you with my home made cakes. Special
thanks go to Anne Moe, for your support with administrative problems, travels and
organisation of events. Thanks also to Lillemor Wallgren, Britt-Inger Karlsson
and Inger Norén for administrative help. Thanks to TUS for help with technical
issues.
This work was financially supported by the European project Safeguard
IST-2001-32685 and CENIIT (Center for Industrial Information Technology) at
Linköping University. Taking part in a large international project has sometimes
been frustrating but most often instructive, challenging and fun. I am glad that I
got the opportunity to take part in Safeguard.
I would like to thank Tomas Lingvall, Thomas Dagonnier, Mikael Semling
and Stefan Burschka and their colleagues at Swisscom for fruitful discussions and
their many hours of work with the test network. Thanks also to Tomas for help
with the Preprocessor and data generation.
The Safeguard agent architecture has been developed with the input from all
the research nodes of the project, the cooperation of whom is gratefully acknowl-
edged. Special thanks to David Gamez and John Bigham from Queen Mary, Uni-
versity of London and Oleg Morajko at AIA in Spain. Thanks to Wes Carter, our
project coordinator.
Thanks to Daniel Garpe and Robert Jonasson for your work with the agent
platform evaluation. Thanks to Tobias Chyssler for your work with alert correla-
tion engines. Also thanks to Tobias and Daniel for your help with implementing
the correlation agent and for your company and many discussions during those
hectic months of implementation phase in the project. Thanks to Sara Garcia
Andrés for your first implementation of the simulation for our initial work on sur-
vivability modelling. I would like to thank Henrik Larsson and Karin Ring for
reading my thesis with fresh eyes.
Doing PhD-studies while being a father of two wonderful small girls is not
always easy. You have to learn to work very focused to get the maximum out of
those hours in your office, so that you also have time to spend with your family
at home. I would like to thank my dear wife and very best friend Malin for all
her help and support. Not the least for those weeks when conferences and project
meetings have taken me far away from home. I love you with all of my heart.
Thanks to Alva and Linnea for being such clever, cute and funny girls. Even
when life sometimes is harsh, you often manage to make me smile. Thanks to my
parents and Malin’s for your help with the girls and your support. Thanks also to
our cuddly black cats Mashlul and Korlash, for lying and purring in my lap while
I was writing those last hard chapters in the thesis.
In context of my family I also would like to give special thanks to my advisor
for her support not only with my licentiate studies, but also for supporting me in
my private situation. Thanks for helping me being home those months with my
girls in the middle of my studies.
In the end I would like to thank all my friends and my family for my years of
fun in Linköping. I will always remember those years as a very good time of my
life. I dedicate this work to you all.
Kalle Burbeck
CONTENTS
Contents
1 Introduction
.
.
.
.
.
.
.
1.1 Motivation .
1.2 Research challenges . .
.
1.3 Contribution . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
.
1.4 List of publications . . . .
1.5 Thesis outline . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
ix
1
1
3
5
6
7
2 Background
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2.2
.
.
.
.
.
.
2.1.1 Attack types
.
Intrusion detection .
.
2.2.1 Components
.
2.2.2 Taxonomy .
2.2.3 Evaluation metrics . . . .
2.1 Dependability and computer security .
.
.
.
.
9
9
.
. 11
. 12
. 12
. 14
. 18
2.3 Software agents . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
. 22
2.4 Data mining and machine learning . . . . . . . . . . . . . . . . . 23
. 24
. 25
2.4.1 Classification .
.
2.4.2 Clustering . . . .
.
.
.
.
.
.
.
.
.
. . . .
2.3.1 Agent platforms
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3 The Safeguard context
3.1 Critical infrastructures
3.2 Safeguard solutions .
.
.
.
.
.
.
.
.
.
.
.
.
.
3.1.1 Telecommunications vulnerabilities . .
3.1.2 Electricity vulnerabilities .
.
.
.
.
31
. 31
. 33
. 33
.
. 34
.
3.2.1 Agents for increased dependability .
. 35
3.2.2 The Safeguard agent platform . . . . . . . . . . . . . . . 36
3.2.3 The Safeguard agent architecture .
. 39
. . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
x
CONTENTS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. . . .
3.3 The Safeguard agents .
.
3.3.1 Wrapper agent
.
3.3.2 Hybrid detector agent . . . .
.
.
3.3.3 Topology agent . .
.
3.3.4 Correlation agent .
.
3.3.5 Human-machine interface agent
.
.
3.3.6 Action agent
.
3.3.7 Actuator agent
.
3.3.8 Negotiation agent .
.
.
.
3.4 Safeguard test beds . .
.
. 42
.
. 42
. . . . . . . . . . . . . . . . 43
. 44
.
.
. 44
. 49
. 52
. 53
. 53
. 54
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
4 ADWICE
4.1 Basic concepts
.
4.2 Training .
.
.
.
.
. .
.
.
. .
.
4.3 Detection .
4.4 Evaluation .
.
.
.
.
.
.
.
.
.
.
.
.
4.2.1 Using the original BIRCH index . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
. 57
. . . . . . . . . . . . . . . . . . . . . . . 58
. 60
. . . . . . . . . . . . . . . . . . . . . . 61
.
. . . . . . . . . . . . . . . . . . . . . . . 61
. . . . . . . . . . . . . . . . 63
. 66
. 67
. 68
. 70
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
. .
4.4.1 Determining parameters
4.4.2 Detection rate versus false positives rate .
.
4.4.3 Attack class results .
.
.
4.4.4 Aggregation for decreasing alert rate .
4.4.5
.
.
Safeguard scenarios
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5 ADWICE with grid index
.
.
.
.
.
.
.
.
. .
5.1.1
Influence of index errors . .
5.2 The grid-index .
5.3 Adaptation of the normality model
.
.
73
5.1 Problems of the original BIRCH index . . . . . . . . . . . . . . . 73
. 73
. . . . . . . . . . . . . . . . . . . . . . . 77
. 82
. 82
.
. 82
. . . . . . . . . . . . . . . . . . . . . . 84
. 84
. 84
. 86
5.4.1 Detection rate versus false positives rate .
.
5.4.2
5.4.3
.
Incremental training .
Forgetting . .
.
.
Incremental training .
Forgetting . .
.
5.4 Evaluation .
. . .
.
.
.
.
5.3.1
5.3.2
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6.1 Design .
6 Clustering hybrid detection agent
.
.
.
. .
6.1.1
Preprocessor
6.1.2 DataSource .
. .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
. 88
. 89
. 92
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
