第1页 / 共277页
第2页 / 共277页
第3页 / 共277页
第4页 / 共277页
第5页 / 共277页
第6页 / 共277页
第7页 / 共277页
第8页 / 共277页
The Art of Application Performance Testing, 2nd Edition.pdf
Cover
Table of Contents
Preface
Audience
About This Book
Conventions Used in This Book
Glossary
Using Code Examples
Safari® Books Online
How to Contact Us
Acknowledgments
Why Performance Test?
What Is Performance? The End-User Perspective
Performance Measurement
Performance Standards
The World Wide Web and Ecommerce
Bad Performance: Why It’s So Common
The IT Business Value Curve
Performance Testing Maturity: What the Analysts Think
Lack of Performance Considerations in Application Design
Performance Testing Is Left to the Last Minute
Scalability
Underestimating Your Popularity
Performance Testing Is Still an Informal Discipline
Not Using Automated Testing Tools
Application Technology Impact
Summary
Choosing an Appropriate Performance Testing Tool
Performance Testing Tool Architecture
Choosing a Performance Testing Tool
Performance Testing Toolset: Proof of Concept
Proof of Concept Checklist
Summary
The Fundamentals of Effective Application Performance Testing
Making Sure Your Application Is Ready
Allocating Enough Time to Performance Test
Obtaining a Code Freeze
Designing a Performance Test Environment
Virtualization
Cloud Computing
Load Injection Capacity
Addressing Different Network Deployment Models
Environment Checklist
Software Installation Constraints
Setting Realistic Performance Targets
Consensus
Performance Target Definition
Network Utilization
Server Utilization
Identifying and Scripting the Business-Critical Use Cases
Use-Case Checklist
Use-Case Replay Validation
What to Measure
To Log In or Not to Log In
Peaceful Coexistence
Providing Test Data
Input Data
Target Data
Session Data
Data Security
Ensuring Accurate Performance-Test Design
Principal Types of Performance Test
The Load Model
Think Time
Pacing
Identifying the KPIs
Server KPIs
Network KPIs
Application Server KPIs
Summary
The Process of Performance Testing
Activity Duration Guidelines
Performance Testing Approach
Step 1: Nonfunctional Requirements Capture
Step 2: Performance Test Environment Build
Step 3: Use-Case Scripting
Step 4: Performance Test Scenario Build
Step 5: Performance Test Execution
Step 6: Post-Test Analysis and Reporting
Case Study 1: Online Banking
Application Landscape
Application Users
Step 1: Pre-Engagement NFR Capture
Step 2: Test Environment Build
Step 3: Use-Case Scripting
Step 4: Performance Test Build
Step 5: Performance Test Execution
Online Banking Case Study Review
Case Study 2: Call Center
Application Landscape
Application Users
Step 1: Pre-Engagement NFR Capture
Step 2: Test Environment Build
Step 3: Use-Case Scripting
Step 4: Performance Test Scenario Build
Step 5: Performance Test Execution
Call Center Case Study Review
Summary
Interpreting Results: Effective Root-Cause Analysis
The Analysis Process
Real-Time Analysis
Post-Test Analysis
Types of Output from a Performance Test
Statistics Primer
Response-Time Measurement
Throughput and Capacity
Monitoring Key Performance Indicators
Server KPI Performance
Network KPI Performance
Load Injector Performance
Root-Cause Analysis
Scalability and Response Time
Digging Deeper
Inside the Application Server
Looking for the Knee
Dealing with Errors
Baseline Data
Analysis Checklist
Pre-Test Tasks
Tasks During Test Execution
Post-Test Tasks
Summary
Performance Testing and the Mobile Client
What’s Different About a Mobile Client?
Mobile Testing Automation
Mobile Design Considerations
Mobile Testing Considerations
Mobile Test Design
On-Device Performance Not in Scope
On-Device Performance Testing Is in Scope
Summary
End-User Experience Monitoring and Performance
What Is External Monitoring?
Why Monitor Externally?
External Monitoring Categories
Active Monitoring
Output Metrics
ISP Testing Best Practices
Synthetic End-User Testing Best Practices
Passive Monitoring
How Passive Monitoring Works
Pros and Cons of Active Versus Passive Monitoring
Active Pros
Active Cons
Passive Pros
Passive Cons
Tooling for External Monitoring of Internet Applications
Tool Selection Criteria
Active Monitoring Tooling
Passive Monitoring Tooling
Creating an External Monitoring Testing Framework
Building Blocks of an Effective Testing Framework
Specific Design Aspects of Active Monitoring
Specific Design Aspects of Passive Monitoring
Isolating and Characterizing Issues Using External Monitoring
Monitoring Native Mobile Applications
Essential Considerations for CDN Monitoring
Performance Results Interpretation
Key Performance Indicators for Web-Based Ecommerce Applications
Setting KPI Values
The Application Performance Index (APDEX)
Management Information
Data Preparation
Statistical Considerations
Correlation
Effective Reporting
Competitive Understanding
Visitor performance map
Alerting
Gotchas!
Summary
Integrating External Monitoring and Performance Testing
Tooling Choices
Active and Passive Integration with Static Performance Testing
Passive and Performance Testing
RUM and APM
Integration of Active Test Traffic with APM Tooling
Active External Monitoring and Performance Testing
Test Approach
Test Scheduling
Performance Testing of Multimedia Content
End-User Understanding in Non-Internet Application Performance Tests
Useful Source Materials
Summary
Application Technology and Its Impact on Performance Testing
Asynchronous Java and XML (AJAX)
Push Versus Pull
Citrix
Citrix Checklist
Citrix Scripting Advice
Virtual Desktop Infrastructure
HTTP Protocol
Web Services
.NET Remoting
Browser Caching
Secure Sockets Layer
Java
Oracle
Oracle Two-Tier
Oracle Forms Server
Oracle Checklist
SAP
SAP Checklist
Service-Oriented Architecture
Web 2.0
Windows Communication Foundation and Windows Presentation Foundation
Oddball Application Technologies: Help, My Load Testing Tool Won’t Record It!
Before Giving Up in Despair . . .
Alternatives to Capture at the Middleware Level
Manual Scripting
Summary
Conclusion
Appendix A. Use-Case Definition Example
Appendix B. Proof of Concept and Performance Test Quick Reference
The Proof of Concept
POC Checklist
Performance Test Execution Checklist
Activity Duration Guidelines
Step 1: Pre-Engagement NFR Capture
Step 2: Test Environment Build
Step 3: Scripting
Step 4: Performance Test Build
Step 5: Performance Test Execution
Step 6 (Post-Test Phase): Analyze Results, Report, Retest If Required
Analysis Checklist
Pre-Test Analysis Tasks
Tasks During Test Execution
Post-Test Tasks
Appendix C. Performance and Testing Tool Vendors
Application Performance Management
End-User Experience and Website Monitoring
Functional Testing
Performance Testing
Open Source
SaaS Performance Testing
Requirements Management
Open Source
Appendix D. Sample Monitoring Templates: Infrastructure Key Performance Indicator Metrics
Generic KPI Templates
Windows OS : Generic KPI Template
Linux/Unix: Generic KPI Template
Application-Specific KPI Templates
Windows OS: MS SQL Server KPI Template
Appendix E. Sample Project Plan
Index
/ t he or y / i n / prac t ic e
The Art of Application Performance Testing
Because performance is paramount today, this thoroughly updated guide shows you how
to test mission-critical applications for scalability and performance before you deploy them—
whether it’s to the cloud or a mobile device. You’ll learn the complete testing process lifecycle
step-by-step, along with best practices to plan, coordinate, and conduct performance tests
on your applications.
Implement an effective application performance testing strategy
Interpret performance test results
Set realistic performance testing goals
Cope with different application technologies and architectures
Understand the importance of End User Monitoring (EUM)
Use automated performance testing tools
Test traditional local applications, web applications, and web services
Recognize and resolves issues often overlooked in performance tests
Written by a consultant with over 15 years’ experience with performance testing, The Art of
Application Performance Testing thoroughly explains the pitfalls of an inadequate testing strategy
and offers a robust, structured approach for ensuring that your applications perform well and
scale effectively when the need arises.
Ian Molyneaux, EMEA SME (Subject Matter Expert), is Head of Performance at Intechnica, a
software consultancy based in Manchester, UK. He specializes in performance assurance for
the enterprise with a strong focus on people, process, and tooling.
US $44.99
CAN $47.99
ISBN: 978-1-491-90054-3
Software Testing
Twitter: @oreillymedia
facebook.com/oreilly
oreilly.com
/ t he or y / i n / prac t ic e
The Art of Application
Performance Testing
From Strategy to Tools
Ian Molyneaux
2nd
Edition
T
h
e
A
r
t
o
f
A
p
p
l
i
c
a
t
i
o
n
P
e
r
f
o
r
m
a
n
c
e
T
e
s
t
i
n
g
l
M
o
y
n
e
a
u
x
www.it-ebooks.info
■
■
■
■
■
■
■
■
/ t he or y / i n / prac t ic e
The Art of Application
Performance Testing
From Strategy to Tools
Ian Molyneaux
2nd
Edition
/ t he or y / i n / prac t ic e
The Art of Application Performance Testing
Because performance is paramount today, this thoroughly updated guide shows you how
to test mission-critical applications for scalability and performance before you deploy them—
whether it’s to the cloud or a mobile device. You’ll learn the complete testing process lifecycle
step-by-step, along with best practices to plan, coordinate, and conduct performance tests
on your applications.
Implement an effective application performance testing strategy
Interpret performance test results
Set realistic performance testing goals
Cope with different application technologies and architectures
Understand the importance of End User Monitoring (EUM)
Use automated performance testing tools
Test traditional local applications, web applications, and web services
Recognize and resolves issues often overlooked in performance tests
Written by a consultant with over 15 years’ experience with performance testing, The Art of
Application Performance Testing thoroughly explains the pitfalls of an inadequate testing strategy
and offers a robust, structured approach for ensuring that your applications perform well and
scale effectively when the need arises.
Ian Molyneaux, EMEA SME (Subject Matter Expert), is Head of Performance at Intechnica, a
software consultancy based in Manchester, UK. He specializes in performance assurance for
the enterprise with a strong focus on people, process, and tooling.
US $44.99
CAN $47.99
ISBN: 978-1-491-90054-3
Software Testing
www.it-ebooks.info
Twitter: @oreillymedia
facebook.com/oreilly
oreilly.com
T
h
e
A
r
t
o
f
A
p
p
l
i
c
a
t
i
o
n
P
e
r
f
o
r
m
a
n
c
e
T
e
s
t
i
n
g
l
M
o
y
n
e
a
u
x
■
■
■
■
■
■
■
■
The Art of Application
Performance Testing
SECOND EDITION
Ian Molyneaux
www.it-ebooks.info
The Art of Application Performance Testing
by Ian Molyneaux
Copyright © 2015 Ian Molyneaux. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online
editions are also available for most titles (http://safaribooksonline.com). For more information,
contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editors: Andy Oram and Brian Anderson
Production Editor: Melanie Yarbrough
Copyeditor: Rachel Monaghan
Proofreader: Sharon Wilkey
Indexer: Wendy Catalano
Interior Designer: David Futato
Cover Designer: Ellie Volkhausen
Illustrator: Rebecca Demarest
December 2014:
Second Edition
Revision History for the Second Edition
2014-12-08: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491900543 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. The Art of Application Perfor-
mance Testing, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information
and instructions contained in this work are accurate, the publisher and the author disclaim all
responsibility for errors or omissions, including without limitation responsibility for damages
resulting from the use of or reliance on this work. Use of the information and instructions con-
tained in this work is at your own risk. If any code samples or other technology this work con-
tains or describes is subject to open source licenses or the intellectual property rights of others,
it is your responsibility to ensure that your use thereof complies with such licenses and/or
rights.
978-1-491-90054-3
[LSI]
www.it-ebooks.info
T A B L E O F C O N T E N T S
1
2
3
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Why Performance Test?. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
What Is Performance? The End-User Perspective 1
Performance Measurement 2
Performance Standards 3
The World Wide Web and Ecommerce 5
Bad Performance: Why It’s So Common 5
The IT Business Value Curve 5
Performance Testing Maturity: What the Analysts Think 6
Lack of Performance Considerations in Application Design 7
Performance Testing Is Left to the Last Minute 7
Scalability 8
Underestimating Your Popularity 8
Performance Testing Is Still an Informal Discipline 9
Not Using Automated Testing Tools 9
Application Technology Impact 10
Summary 10
Choosing an Appropriate Performance Testing Tool. . . . . . . . . . . . . . . . . . 11
Performance Testing Tool Architecture 12
Choosing a Performance Testing Tool 13
Performance Testing Toolset: Proof of Concept 16
Proof of Concept Checklist 17
Summary 19
The Fundamentals of Effective Application Performance Testing. . . . . . . 21
Making Sure Your Application Is Ready 23
Allocating Enough Time to Performance Test 24
Obtaining a Code Freeze 25
Designing a Performance Test Environment 26
Virtualization 27
www.it-ebooks.info
iii
Cloud Computing 29
Load Injection Capacity 31
Addressing Different Network Deployment Models 32
Environment Checklist 34
Software Installation Constraints 35
Setting Realistic Performance Targets 35
Consensus 35
Performance Target Definition 37
Network Utilization 41
Server Utilization 42
Identifying and Scripting the Business-Critical Use Cases 43
Use-Case Checklist 44
Use-Case Replay Validation 45
What to Measure 46
To Log In or Not to Log In 46
Peaceful Coexistence 47
Providing Test Data 47
Input Data 47
Target Data 48
Session Data 49
Data Security 49
Ensuring Accurate Performance-Test Design 49
Principal Types of Performance Test 50
The Load Model 51
Think Time 54
Pacing 54
Identifying the KPIs 59
Server KPIs 59
Network KPIs 62
Application Server KPIs 64
Summary 64
4
The Process of Performance Testing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
Activity Duration Guidelines 65
Performance Testing Approach 66
Step 1: Nonfunctional Requirements Capture 67
Step 2: Performance Test Environment Build 70
Step 3: Use-Case Scripting 71
iv
CONTENTS
www.it-ebooks.info
Step 4: Performance Test Scenario Build 72
Step 5: Performance Test Execution 74
Step 6: Post-Test Analysis and Reporting 75
Case Study 1: Online Banking 75
Application Landscape 76
Application Users 76
Step 1: Pre-Engagement NFR Capture 77
Step 2: Test Environment Build 78
Step 3: Use-Case Scripting 79
Step 4: Performance Test Build 80
Step 5: Performance Test Execution 81
Online Banking Case Study Review 81
Case Study 2: Call Center 83
Application Landscape 83
Application Users 85
Step 1: Pre-Engagement NFR Capture 85
Step 2: Test Environment Build 86
Step 3: Use-Case Scripting 86
Step 4: Performance Test Scenario Build 87
Step 5: Performance Test Execution 87
Call Center Case Study Review 88
Summary 89
5
Interpreting Results: Effective Root-Cause Analysis. . . . . . . . . . . . . . . . . . . 91
The Analysis Process 92
Real-Time Analysis 92
Post-Test Analysis 93
Types of Output from a Performance Test 93
Statistics Primer 93
Response-Time Measurement 96
Throughput and Capacity 99
Monitoring Key Performance Indicators 100
Server KPI Performance 102
Network KPI Performance 103
Load Injector Performance 104
Root-Cause Analysis 105
Scalability and Response Time 105
Digging Deeper 107
www.it-ebooks.info
CONTENTS
v
6
7
Inside the Application Server 108
Looking for the Knee 109
Dealing with Errors 110
Baseline Data 111
Analysis Checklist 111
Pre-Test Tasks 111
Tasks During Test Execution 112
Post-Test Tasks 114
Summary 115
Performance Testing and the Mobile Client. . . . . . . . . . . . . . . . . . . . . . . . . 117
What’s Different About a Mobile Client? 117
Mobile Testing Automation 118
Mobile Design Considerations 119
Mobile Testing Considerations 120
Mobile Test Design 120
On-Device Performance Not in Scope 121
On-Device Performance Testing Is in Scope 122
Summary 123
End-User Experience Monitoring and Performance. . . . . . . . . . . . . . . . . . 125
What Is External Monitoring? 126
Why Monitor Externally? 127
External Monitoring Categories 130
Active Monitoring 130
Output Metrics 132
ISP Testing Best Practices 133
Synthetic End-User Testing Best Practices 135
Passive Monitoring 136
How Passive Monitoring Works 138
Pros and Cons of Active Versus Passive Monitoring 140
Active Pros 140
Active Cons 141
Passive Pros 141
Passive Cons 141
Tooling for External Monitoring of Internet Applications 141
Tool Selection Criteria 142
Active Monitoring Tooling 144
vi
CONTENTS
www.it-ebooks.info
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
