Fuzzing for Software Security Testing and Quality Assurance, Second Edition
Contents
Foreword from the First Edition
Foreword to the Second Edition
Preface from the First Edition
Preface to the Second Edition
Chapter 1
Introduction
1.1 Software Security
1.1.1 Security Incident
1.1.2 Disclosure Processes
1.1.3 Attack Surfaces and Attack Vectors
1.1.4 Reasons Behind Security Mistakes
1.1.5 Proactive Security
1.1.6 Security Requirements
1.2 Software Quality
1.2.1 Cost-Benefit of Quality
1.2.2 Target of Test
1.2.3 Testing Purposes and Test Verdicts
1.2.4 Structural Testing
1.2.5 Functional Testing
1.2.6 Code Auditing
1.3 Introduction to Fuzzing
1.3.1 Brief History of Fuzzing
1.3.2 Fuzzing Overview
1.3.3 Vulnerabilities Found with Fuzzing
1.3.4 Fuzzer Types
1.3.5 Logical Structure of a Fuzzer
1.3.6 Fuzzing Process
1.3.7 Fuzzing Frameworks and Test Suites
1.3.8 Fuzzing and the Enterprise
1.4 Book Goals and Layout
Chapter 2
Software Vulnerability Analysis
2.1 Purpose of Vulnerability Analysis
2.1.1 Security and Vulnerability Scanners
2.2 People Conducting Vulnerability Analysis
2.2.1 Hackers
2.2.2 Vulnerability Analysts or Security Researchers
2.2.3 Penetration Testers
2.2.4 Software Security Testers
2.2.5 IT Security Engineers
2.3 Target Software
2.4 Basic Bug Categories
2.4.1 Memory Corruption Errors
2.4.2 Web Applications
2.4.3 Brute Force Login
2.4.4 Race Condition
2.4.5 Denial of Service
2.4.6 Session Hijacking
2.4.7 Man in the Middle
2.4.8 Cryptographic Attacks
2.5 Bug Hunting Techniques
2.5.1 Reverse Engineering
2.5.2 Source Code Auditing
2.6 Fuzzing
2.6.1 Basic Terms
2.6.2 Hostile Data
2.6.3 Number of Tests
2.7 Defenses
2.7.1 Why Fuzzing Works
2.7.2 Defensive Coding
2.7.3 Input Verification
2.7.4 Hardware Overflow Protection
2.7.5 Software Overflow Protection
2.8 Summary
Chapter 3
Quality Assurance and Testing
3.1 Quality Assurance and Security
3.1.1 Security in Software Development
3.1.2 Security Defects
3.2 Measuring Quality
3.2.1 Quality Is About Validation of Features
3.2.2 Quality Is About Finding Defects
3.2.3 Quality Is a Feedback Loop to Development
3.2.4 Quality Brings Visibility to the Development Process
3.2.5 End Users’ Perspective
3.3 Testing for Quality
3.3.1 V-Model
3.3.2 Testing on the Developer’s Desktop
3.3.3 Testing the Design
3.4 Main Categories of Testing
3.4.1 Validation Testing Versus Defect Testing
3.4.2 Structural Versus Functional Testing
3.5 White-Box Testing
3.5.1 Making the Code Readable
3.5.2 Inspections and Reviews
3.5.3 Code Auditing
3.6 Black-Box Testing
3.6.1 Software Interfaces
3.6.2 Test Targets
3.6.3 Fuzz Testing as a Profession
3.7 Purposes of Black-Box Testing
3.7.1 Conformance Testing
3.7.2 Functional Security Testing
3.7.3 Functional Safety Testing
3.7.4 Interoperability Testing
3.7.5 Performance Testing
3.7.6 Robustness Testing
3.8 Testing Metrics
3.8.1 Specification Coverage
3.8.2 Input Space Coverage
3.8.3 Interface Coverage
3.8.4 Code Coverage
3.9 Black-Box Testing Techniques for Security
3.9.1 Load Testing
3.9.2 Stress Testing
3.9.3 Security Scanners
3.9.4 Unit Testing
3.9.5 Fault Injection
3.9.6 Syntax Testing
3.9.7 Negative Testing
3.9.8 Regression Testing
3.10 Testing in Continuous Integration
3.11 Summary
Chapter 4
Fuzzing Metrics
4.1 Threat Analysis and Risk-Based Testing
4.1.1 Threat Trees
4.1.2 Threat Databases
4.1.3 Ad Hoc Threat Analysis
4.2 Transition to Proactive Security
4.2.1 Cost of Discovery
4.2.2 Cost of Remediation
4.2.3 Cost of Security Compromises
4.2.4 Cost of Patch Deployment
4.3 Defect Metrics and Security
4.3.1 Coverage of Previous Vulnerabilities
4.3.2 Expected Defect Count Metrics
4.3.3 Vulnerability Risk Metrics
4.3.4 Interface Coverage Metrics
4.3.5 Input Space Coverage Metrics
4.3.6 Code Coverage Metrics
4.3.7 Process Metrics
4.4 Test Automation for Security
4.5 Summary
Chapter 5
Building and Classifying Fuzzers
5.1 Fuzzing Methods
5.1.1 Paradigm Split: Random or Deterministic Fuzzing
5.1.2 Source of Fuzz Data
5.1.3 Fuzzing Vectors
5.1.4 Intelligent Fuzzing
5.1.5 Intelligent Versus Dumb (Nonintelligent) Fuzzers
5.1.6 Whitebox, Blackbox, and Graybox Fuzzing
5.2 Detailed View of Fuzzer Types
5.2.1 Single-Use Fuzzers
5.2.2 Fuzzing Frameworks
5.2.3 Using Prebuilt Protocol-Specific Fuzzers
5.2.4 Generic Fuzzers
5.2.5 Capture-Replay
5.2.6 In-Memory Fuzzing
5.3 Fuzzer Classification via Interface
5.3.1 Local Program
5.3.2 Network Interfaces
5.3.3 Files
5.3.4 APIs
5.3.5 Web Fuzzing
5.3.6 Client-Side Fuzzers
5.3.7 Layer 2 Through 7 Fuzzing
5.4 Summary
Chapter 6
Target Monitoring
6.1 What Can Go Wrong and What Does It Look Like?
6.1.1 Denial of Service (DoS)
6.1.2 File System–Related Problems
6.1.3 Metadata Injection Vulnerabilities
6.1.4 Memory-Related Vulnerabilities
6.2 Methods of Monitoring
6.2.1 Valid Case Instrumentation
6.2.2 System Monitoring
6.2.3 Remote Monitoring
6.2.4 Commercial Fuzzer Monitoring Solutions
6.2.5 Application Monitoring
6.3 Advanced Methods
6.3.1 Library Interception
6.3.2 Binary Simulation
6.3.3 Source Code Transformation
6.3.4 Compiler Instrumentation
6.3.5 Virtualization
6.4 Monitoring Overview
6.5 Deduplication
6.5.1 Test Case Generator Information
6.5.2 Operating System Logs
6.5.3 Stack Traces
6.5.4 Advanced Tools
6.6 A Test Program
6.6.1 The Program
6.6.2 Test Cases
6.6.3 Guard Malloc
6.6.4 Valgrind
6.6.5 Insure++
6.6.6 AddressSanitizer
6.7 Case Study: PCRE
6.7.1 Guard Malloc
6.7.2 Valgrind
6.7.3 Insure++
6.7.4 AddressSanitizer
6.8 Summary
Chapter 7
Advanced Fuzzing
7.1 Automatic Protocol Discovery
7.2 Automatic Generation of a Model-Based Fuzzer
7.2.1 Defensics SDK with Suite Wizard
7.2.2 Example Project Created with Suite Wizard
7.2.3 Fuzzer Modeling
7.2.4 Adding Rules to the Fuzzer
7.2.5 Settings to Configure the Fuzzer
7.2.6 Fuzzer Input and Output
7.2.7 Building and Packing the Fuzzer
7.2.8 Conclusion
7.3 Symbolic Execution with SAGE
7.4 Code Coverage in Fuzzing
7.4.1 Code Coverage Guided Fuzzing: American Fuzzy Lop
7.4.2 Code Coverage Guided Fuzzing: libFuzzer
7.5 Evolutionary Fuzzing
7.6 Evolutionary Fuzzing: EFS
7.6.1 EFS Overview
7.6.2 GPF + PaiMei + Jpgraph = EFS
7.6.3 EFS Data Structures
7.6.4 EFS Initialization
7.6.5 Session Crossover
7.6.6 Session Mutation
7.6.7 Pool Crossover
7.6.8 Pool Mutation
7.6.9 Running EFS
7.6.10 Test Case: Golden FTP Server
7.6.11 Results
7.6.12 Conclusions on EFS
7.7 In-Memory Fuzzing
7.7.1 Implementation of In-Memory Fuzzer
7.7.2 Instrumentation
7.7.3 Pin API
7.7.4 Register Example
7.7.5 Pros and Cons
7.7.6 In-Memory Fuzzing Improved by a Dynamic Symbolic Execution
7.8 Distributed Fuzzing
7.8.1 Distributed Fuzzing: Google’s ClusterFuzz
7.8.2 Distributed Fuzzing: DeMott’s ClusterFuzz
7.11 Summary
Chapter 8
Fuzzer Comparison
8.1 Fuzzing Life Cycle
8.1.1 Identifying Interfaces
8.1.2 Input Generation
8.1.3 Sending Inputs to the Target
8.1.4 Target Monitoring
8.1.5 Exception Analysis
8.1.6 Reporting
8.2 Evaluating Fuzzers
8.2.1 Retrospective Testing
8.2.2 Simulated Vulnerability Discovery
8.2.3 Code Coverage
8.2.4 Caveats
8.3 Introducing the Fuzzers
8.3.1 GPF
8.3.2 Taof
8.3.3 ProxyFuzz
8.3.4 Mu-4000
8.3.5 Codenomicon Defensics
8.3.6 beSTORM
8.3.7 Application-Specific Fuzzers
8.3.8 What’s Missing
8.4 The Targets
8.5 The Bugs
8.5.1 FTP Bug 0
8.5.2 FTP Bugs 2, 16
8.6 Results
8.6.1 FTP
8.6.2 SNMP
8.6.3 DNS
8.7 A Closer Look at the Results
8.7.1 FTP
8.7.2 SNMP
8.7.3 DNS
8.8 General Conclusions
8.8.1 The More Fuzzers, the Better
8.8.2 Generational-Based Approach Is Superior
8.8.3 Initial Test Cases Matter
8.8.4 Protocol Knowledge Helps
8.8.5 Real Bugs
8.8.6 Does Code Coverage Predict Bug Finding?
8.8.7 How Long to Run Fuzzers with Random Elements
8.8.8 Random Fuzzers Find Easy Bugs First
8.9 Summary
Chapter 9
Fuzzing Case Studies
9.1 Enterprise Fuzzing
9.1.1 Firewall Fuzzing
9.1.2 VPN Fuzzing
9.2 Carrier and Service Provider Fuzzing
9.2.1 VoIP Fuzzing
9.2.2 WiFi Fuzzing
9.3 Application Developer Fuzzing
9.3.1 Command-Line Application Fuzzing
9.3.2 File Fuzzing
9.3.3 Web Application Fuzzing
9.3.4 Browser Fuzzing
9.4 Network Equipment Manufacturer Fuzzing
9.4.1 Network Switch Fuzzing
9.4.2 Mobile Phone Fuzzing
9.5 Industrial Automation Fuzzing
9.6 Blackbox Fuzzing for Security Researchers
9.6.1 Select Target
9.6.2 Enumerate Interfaces
9.6.3 Choose Fuzzer/Fuzzer Type
9.6.4 Choose a Monitoring Tool
9.6.5 Carry Out the Fuzzing
9.6.6 Postfuzzing Analysis
9.7 Summary
About the Authors
Index