第1页 / 共991页
第2页 / 共991页
第3页 / 共991页
第4页 / 共991页
第5页 / 共991页
第6页 / 共991页
第7页 / 共991页
第8页 / 共991页
BPF-Performance-Tools-by-Brendan-Gregg.pdf
Title Page
Copyright Page
Contents at a Glance
Table of Contents
Preface
Why Do You Need BPF Performance Tools?
About This Book
How This Book Is Structured
Intended Audience
Source Code Copyright
Supplemental Material and References
Part I: Technologies
Chapter 1. Introduction
What are BPF and eBPF?
What are Tracing, Snooping, Sampling, Profiling, and Observability?
What are BCC, bpftrace, and IO Visor?
A First Look at BCC: Quick Wins
BPF Tracing Visibility
Dynamic Instrumentation: kprobes and uprobes
Static Instrumentation: Tracepoints and USDT
A First Look at bpftrace: Tracing open()
Back to BCC: Tracing open()
Summary
Chapter 2. Technology Background
Illustrated
BPF
Extended BPF (eBPF)
Stack Trace Walking
Flame Graphs
Event Sources
Kprobes
Uprobes
Tracepoints
USDT
Dynamic USDT
PMCs
perf_events
Summary
Chapter 3. Performance Analysis
Overview
Performance Methodologies
Linux 60-Second Analysis
BCC Tool Checklist
Summary
Chapter 4. BCC
BCC Components
BCC Features
BCC Installation
BCC Tools
funccount
stackcount
trace
argdist
Tool Documentation
Developing BCC Tools
BCC Internals
BCC Debugging
Summary
Chapter 5. bpftrace
bpftrace Components
bpftrace Features
bpftrace Installation
bpftrace Tools
bpftrace One-Liners
bpftrace Documentation
bpftrace Programming
bpftrace Usage
bpftrace Probe Types
bpftrace Flow Control
bpftrace Operators
bpftrace Variables
bpftrace Functions
bpftrace Map Functions
bpftrace Future Work
bpftrace Internals
bpftrace Debugging
Summary
Part II: Using BPF Tools
Chapter 6. CPUs
Background
Traditional Tools
BPF Tools
BPF One-liners
Optional Exercises
Summary
Chapter 7. Memory
Background
Traditional Tools
BPF Tools
BPF One-liners
Optional Exercises
Summary
Chapter 8. File Systems
Background
Strategy
Traditional Tools
BPF Tools
BPF One-liners
BPF One-Liners Examples
Optional Exercises
Summary
Chapter 9. Disk I/O
Background
Traditional Tools
BPF Tools
BPF One-liners
BPF One-Liners Examples
Optional Exercises
Summary
Chapter 10. Networking
Background
Traditional Tools
BPF Tools
BPF One-liners
BPF One-Liners Examples
Optional Exercises
Summary
Chapter 11. Security
Background
BPF Tools
BPF One-Liners
BPF One-Liners Examples
Summary
Chapter 12. Languages
Background
C
Java
Bash Shell
Other Languages
Summary
Chapter 13. Applications
Background
BPF Tools
BPF One-liners
BPF One-Liners Examples
Summary
Chapter 14. Kernel
Background
Strategy
Traditional Tools
BPF Tools
BPF One-Liners
BPF One-Liners Examples
Challenges
Summary
Chapter 15. Containers
Background
Traditional Tools
BPF Tools
BPF One-liners
Optional Exercises
Summary
Chapter 16. Hypervisors
Background
Traditional Tools
Guest BPF Tools
Host BPF Tools
Summary
Part III: Additional Topics
Chapter 17. Other BPF Performance Tools
Vector and Performance Co-Pilot (PCP)
Grafana and Performance Co-Pilot (PCP)
Cloudflare eBPF Prometheus Exporter (with Grafana)
kubectl-trace
Other Tools
Summary
Chapter 18. Tips, Tricks, and Common Problems
1. Typical Event Frequency and Overhead
2. Sample at 49 or 99 Hertz
3. Yellow Pigs and Grey Rats
4. Write Target Software
5. Learn Syscalls
6. Keep it Simple
7. Missing Events
8. Missing Stacks Traces
9. Missing Symbols (Function Names) When Printing
10. Missing Functions When Tracing
11. Feedback Loops
12. Dropped Events
Part IV: Appendixes
A. bpftrace One-Liners [This content is currently in development.]
B. bpftrace Cheat Sheet [This content is currently in development.]
C. bcc Tool Development [This content is currently in development.]
D. C BPF [This content is currently in development.]
E. BPF Instructions [This content is currently in development.]
Glossary
Bibliography [This content is currently in development.]
BPF Performance
Tools
L
i
n
u
x
S
y
s
t
e
m
a
n
d
A
p
p
l
i
c
a
t
i
o
n
O
b
s
e
r
v
a
b
i
l
i
t
y
B
r
e
n
d
a
n
G
r
e
g
g
C
o
p
y
r
i
g
h
t
©
2
0
2
0
P
e
a
r
s
o
n
E
d
u
c
a
t
i
o
n
,
I
n
c
.
Contents at a Glance
P
r
e
f
a
c
e
P
a
r
t
I
:
T
e
c
h
n
o
l
o
g
i
e
s
C
h
a
p
t
e
r
1
.
I
n
t
r
o
C
h
a
p
t
e
r
2
.
T
e
c
h
n
o
l
o
g
y
B
a
c
k
g
r
o
u
n
d
C
h
a
p
t
e
r
3
.
P
e
r
f
o
r
m
a
n
c
e
A
n
a
l
y
s
i
s
C
h
a
p
t
e
r
4
.
b
c
c
C
h
a
p
t
e
r
5
.
b
p
f
t
r
a
c
e
P
a
r
t
I
I
:
U
s
i
n
g
B
P
F
T
o
o
l
s
C
h
a
p
t
e
r
6
.
C
P
U
s
C
h
a
p
t
e
r
7
.
M
e
m
o
r
y
C
h
a
p
t
e
r
8
.
F
i
l
e
S
y
s
t
e
m
s
C
h
a
p
t
e
r
9
.
D
i
s
k
I
/
O
C
h
a
p
t
e
r
1
0
.
N
e
t
w
o
r
k
i
n
g
C
h
a
p
t
e
r
1
1
.
S
e
c
u
r
i
t
y
C
h
a
p
t
e
r
1
2
.
L
a
n
g
u
a
g
e
s
C
h
a
p
t
e
r
1
3
.
A
p
p
l
i
c
a
t
i
o
n
s
C
h
a
p
t
e
r
1
4
.
K
e
r
n
e
l
C
h
a
p
t
e
r
1
5
.
C
o
n
t
a
i
n
e
r
s
C
h
a
p
t
e
r
1
6
.
H
y
p
e
r
v
i
s
o
r
s
P
a
r
t
I
I
I
:
A
d
d
i
t
i
o
n
a
l
T
o
p
i
c
s
C
h
a
p
t
e
r
1
7
.
O
t
h
e
r
B
P
F
T
o
o
l
s
C
h
a
p
t
e
r
1
8
.
T
i
p
s
a
n
d
T
r
i
c
k
s
P
a
r
t
I
V
:
A
p
p
e
n
d
i
x
e
s
A
.
b
p
f
t
r
a
c
e
O
n
e
-
L
i
n
e
r
s
B
.
b
p
f
t
r
a
c
e
C
h
e
a
t
S
h
e
e
t
C
.
b
c
c
T
o
o
l
D
e
v
e
l
o
p
m
e
n
t
D
.
C
B
P
F
E
.
B
P
F
I
n
s
t
r
u
c
t
i
o
n
s
G
l
o
s
s
a
r
y
B
i
b
l
i
o
g
r
a
p
h
y
Table of Contents
P
r
e
f
a
c
e
W
h
y
D
o
Y
o
u
N
e
e
d
B
P
F
P
e
r
f
o
r
m
a
n
c
e
T
o
o
l
s
?
A
b
o
u
t
T
h
i
s
B
o
o
k
H
o
w
T
h
i
s
B
o
o
k
I
s
S
t
r
u
c
t
u
r
e
d
I
n
t
e
n
d
e
d
A
u
d
i
e
n
c
e
S
o
u
r
c
e
C
o
d
e
C
o
p
y
r
i
g
h
t
S
u
p
p
l
e
m
e
n
t
a
l
M
a
t
e
r
i
a
l
a
n
d
R
e
f
e
r
e
n
c
e
s
P
a
r
t
I
:
T
e
c
h
n
o
l
o
g
i
e
s
C
h
a
p
t
e
r
1
.
I
n
t
r
o
d
u
c
t
i
o
n
W
h
a
t
a
r
e
B
P
F
a
n
d
e
B
P
F
?
W
h
a
t
a
r
e
T
r
a
c
i
n
g
,
S
n
o
o
p
i
n
g
,
S
a
m
p
l
i
n
g
,
P
r
o
fi
l
i
n
g
,
a
n
d
O
b
s
e
r
v
a
b
i
l
i
t
y
?
W
h
a
t
a
r
e
B
C
C
,
b
p
f
t
r
a
c
e
,
a
n
d
I
O
V
i
s
o
r
?
A
F
i
r
s
t
L
o
o
k
a
t
B
C
C
:
Q
u
i
c
k
W
i
n
s
B
P
F
T
r
a
c
i
n
g
V
i
s
i
b
i
l
i
t
y
D
y
n
a
m
i
c
I
n
s
t
r
u
m
e
n
t
a
t
i
o
n
:
k
p
r
o
b
e
s
a
n
d
u
p
r
o
b
e
s
S
t
a
t
i
c
I
n
s
t
r
u
m
e
n
t
a
t
i
o
n
:
T
r
a
c
e
p
o
i
n
t
s
a
n
d
U
S
D
T
A
F
i
r
s
t
L
o
o
k
a
t
b
p
f
t
r
a
c
e
:
T
r
a
c
i
n
g
o
p
e
n
(
)
B
a
c
k
t
o
B
C
C
:
T
r
a
c
i
n
g
o
p
e
n
(
)
S
u
m
m
a
r
y
C
h
a
p
t
e
r
2
.
T
e
c
h
n
o
l
o
g
y
B
a
c
k
g
r
o
u
n
d
I
l
l
u
s
t
r
a
t
e
d
B
P
F
E
x
t
e
n
d
e
d
B
P
F
(
e
B
P
F
)
S
t
a
c
k
T
r
a
c
e
W
a
l
k
i
n
g
F
l
a
m
e
G
r
a
p
h
s
E
v
e
n
t
S
o
u
r
c
e
s
K
p
r
o
b
e
s
U
p
r
o
b
e
s
T
r
a
c
e
p
o
i
n
t
s
U
S
D
T
D
y
n
a
m
i
c
U
S
D
T
P
M
C
s
p
e
r
f
_
e
v
e
n
t
s
S
u
m
m
a
r
y
C
h
a
p
t
e
r
3
.
P
e
r
f
o
r
m
a
n
c
e
A
n
a
l
y
s
i
s
O
v
e
r
v
i
e
w
P
e
r
f
o
r
m
a
n
c
e
M
e
t
h
o
d
o
l
o
g
i
e
s
L
i
n
u
x
6
0
-
S
e
c
o
n
d
A
n
a
l
y
s
i
s
B
C
C
T
o
o
l
C
h
e
c
k
l
i
s
t
S
u
m
m
a
r
y
C
h
a
p
t
e
r
4
.
B
C
C
B
C
C
C
o
m
p
o
n
e
n
t
s
B
C
C
F
e
a
t
u
r
e
s
B
C
C
I
n
s
t
a
l
l
a
t
i
o
n
B
C
C
T
o
o
l
s
f
u
n
c
c
o
u
n
t
s
t
a
c
k
c
o
u
n
t
t
r
a
c
e
a
r
g
d
i
s
t
T
o
o
l
D
o
c
u
m
e
n
t
a
t
i
o
n
D
e
v
e
l
o
p
i
n
g
B
C
C
T
o
o
l
s
相关推荐
2023年江西萍乡中考道德与法治真题及答案.doc
2012年重庆南川中考生物真题及答案.doc
2013年江西师范大学地理学综合及文艺理论基础考研真题.doc
2020年四川甘孜小升初语文真题及答案I卷.doc
2020年注册岩土工程师专业基础考试真题及答案.doc
2023-2024学年福建省厦门市九年级上学期数学月考试题及答案.doc
2021-2022学年辽宁省沈阳市大东区九年级上学期语文期末试题及答案.doc
2022-2023学年北京东城区初三第一学期物理期末试卷及答案.doc
2018上半年江西教师资格初中地理学科知识与教学能力真题及答案.doc
2012年河北国家公务员申论考试真题及答案-省级.doc
2020-2021学年江苏省扬州市江都区邵樊片九年级上学期数学第一次质量检测试题及答案.doc
2022下半年黑龙江教师资格证中学综合素质真题及答案.doc
