A demo: synthetic traffic, judged by DeployAngel's real engine. Back to the walkthrough
shop · production · shared release
DeployAngel checks each production deploy and says when it's safe to stop watching.
v182 failed: receipt.sent events dropped
Code release
· 9ea969edee39
· released Nov 1, 11:24 UTC
Promoted from staging v62, which was cleared.
- Released11:24 UTC
- Rolled out11:24 UTC
- Failed11:29 UTC
- Confidence
- Medium
- how strong the failing evidence is
- Coverage
- 100%
- of normally active routes and jobs, run at least once
- Critical flows
- 1 of 2
- run or checked since rollout
- Detected in
- 5 min
- Baseline
- v181
Why it failed
Before is the baseline (v181), after is v182 since rollout.
| Signal | Before | After | Recent releases | Samples | Status |
|---|---|---|---|---|---|
|
Checkpoint
Checkpoint · receipt.sent Rule: expected from the baseline, per request or per job as recorded; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50% p = 2.6e-16 receipt.sent ~36 expected → 0 |
~36 expected | 0 | – | Failing |
Other signals: 19 more (17 passing, 2 without enough data)
| Signal | Before | After | Recent releases | Samples | Status |
|---|---|---|---|---|---|
|
p95 latency
p95 latency · GET /account Rule: needs 50 requests in baseline and observation GET /account 218 ms → 290 ms · 24 samples |
218 ms | 290 ms | 24 |
Not enough data
needs 50 requests in baseline and observation |
|
|
p95 latency
p95 latency · POST /orders Rule: needs 50 requests in baseline and observation POST /orders 240 ms → 240 ms · 36 samples |
240 ms | 240 ms | 36 |
Not enough data
needs 50 requests in baseline and observation |
|
|
Checkpoint
Checkpoint · order.created Rule: expected from the baseline, per request or per job as recorded; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50% p = 0.55 order.created ~36 expected → 36 |
~36 expected | 36 | – | Pass | |
|
HTTP 5xx rate
HTTP 5xx rate Rule: failing: p < 0.001 and >= 2x and >= +0.5 pp; warning: p < 0.01 and >= 1.5x p = 1 0.13% → 0.00% · 300 samples |
0.13% | 0.00% | 300 | Pass | |
|
Job duration p95
Job duration p95 Rule: warning only: >= 2.0x and >= +1000 ms 753 ms → 685 ms · 91 samples |
753 ms | 685 ms | 91 | Pass | |
|
Job failure rate
Job failure rate Rule: failing: p < 0.001 and >= 2x and >= +1 pp; warning: p < 0.01 and >= 1.5x p = 0.46 0.67% → 1.10% · 91 samples |
0.67% | 1.10% | 91 | Pass | |
|
Job failure rate
Job failure rate · ImportProductsJob Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 ImportProductsJob 0.00% → 0.00% · 22 samples |
0.00% | 0.00% | 22 | Pass | |
|
Job failure rate
Job failure rate · SendReceiptJob Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 SendReceiptJob 0.95% → 0.00% · 33 samples |
0.95% | 0.00% | 33 | Pass | |
|
Job failure rate
Job failure rate · SyncAccountsJob Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 0.26 SyncAccountsJob 0.83% → 2.78% · 36 samples |
0.83% | 2.78% | 36 | Pass | |
|
p95 latency
p95 latency Rule: failing: >= 1.5x and >= +150 ms, sustained 3 evaluations; warning: >= 1.25x and >= +75 ms 240 ms → 264 ms · 300 samples |
240 ms | 264 ms | 300 | Pass | |
|
p95 latency
p95 latency · GET / Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms GET / 240 ms → 264 ms · 105 samples |
240 ms | 264 ms | 105 | Pass | |
|
p95 latency
p95 latency · GET /products Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms GET /products 264 ms → 218 ms · 75 samples |
264 ms | 218 ms | 75 | Pass | |
|
p95 latency
p95 latency · GET /products/:id Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms GET /products/:id 264 ms → 319 ms · 60 samples |
264 ms | 319 ms | 60 | Pass | |
|
Queue latency p95
Queue latency p95 Rule: warning only: >= 3.0x and >= +30000 ms 2148 ms → 1213 ms · 91 samples |
2148 ms | 1213 ms | 91 | Pass | |
|
Route 5xx rate
Route 5xx rate · GET / Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 GET / 0.10% → 0.00% · 105 samples |
0.10% | 0.00% | 105 | Pass | |
|
Route 5xx rate
Route 5xx rate · GET /account Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 GET /account 0.00% → 0.00% · 24 samples |
0.00% | 0.00% | 24 | Pass | |
|
Route 5xx rate
Route 5xx rate · GET /products Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 GET /products 0.20% → 0.00% · 75 samples |
0.20% | 0.00% | 75 | Pass | |
|
Route 5xx rate
Route 5xx rate · GET /products/:id Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 GET /products/:id 0.08% → 0.00% · 60 samples |
0.08% | 0.00% | 60 | Pass | |
|
Route 5xx rate
Route 5xx rate · POST /orders Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x p = 1 POST /orders 0.28% → 0.00% · 36 samples |
0.28% | 0.00% | 36 | Pass |
Next steps
-
v183 has already replaced this release, and was cleared, so rolling this one back no longer applies.
What production exercised
What normally runs during this window, from the last 14 days. Each item counts once however busy it is, and one run is enough. Clearing needs 50% coverage, and high confidence needs 80%.
Critical flows
Flows that matter more than their traffic suggests, counted since this release rolled out.
-
Ran 36 times
Checkout
POST /orderscheckpoint:receipt.sent -
Not run yet
Password reset
POST /password_resetsjob:PasswordsMailer
What changed
Since v181, from file digests.
3 changed files (3 modified)
Gemfile.lockmodifiedREADME.mdmodifiedapp/models/order.rbmodified
History
-
First telemetry from this release
Nov 1, 11:27 UTC
-
Confidence Medium → High
Nov 1, 11:28 UTC
-
Failed: regression detected
Nov 1, 11:29 UTC
