A demo: synthetic traffic, judged by DeployAngel's real engine. Back to the walkthrough

shop · production · shared release

DeployAngel checks each production deploy and says when it's safe to stop watching.

Try it free →
Failed

v182 failed: receipt.sent events dropped

Code release · 9ea969edee39 · released Nov 1, 11:24 UTC

Promoted from staging v62, which was cleared.

  1. Released11:24 UTC
  2. Rolled out11:24 UTC
  3. Failed11:29 UTC
Confidence
Medium
how strong the failing evidence is
Coverage
100%
of normally active routes and jobs, run at least once
Critical flows
1 of 2
run or checked since rollout
Detected in
5 min
Baseline
v181

Why it failed

Before is the baseline (v181), after is v182 since rollout.

20 signals
Signal Status
Checkpoint

Checkpoint · receipt.sent

Rule: expected from the baseline, per request or per job as recorded; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 2.6e-16

receipt.sent

~36 expected → 0

Failing
Other signals: 19 more (17 passing, 2 without enough data)
Signal Status
p95 latency

p95 latency · GET /account

Rule: needs 50 requests in baseline and observation

GET /account

218 ms → 290 ms · 24 samples

Not enough data

needs 50 requests in baseline and observation

p95 latency

p95 latency · POST /orders

Rule: needs 50 requests in baseline and observation

POST /orders

240 ms → 240 ms · 36 samples

Not enough data

needs 50 requests in baseline and observation

Checkpoint

Checkpoint · order.created

Rule: expected from the baseline, per request or per job as recorded; failing: p < 0.0001 and <= 20% of expected; warning: p < 0.001 and <= 50%

p = 0.55

order.created

~36 expected → 36

Pass
HTTP 5xx rate

HTTP 5xx rate

Rule: failing: p < 0.001 and >= 2x and >= +0.5 pp; warning: p < 0.01 and >= 1.5x

p = 1

0.13% → 0.00% · 300 samples

Pass
Job duration p95

Job duration p95

Rule: warning only: >= 2.0x and >= +1000 ms

753 ms → 685 ms · 91 samples

Pass
Job failure rate

Job failure rate

Rule: failing: p < 0.001 and >= 2x and >= +1 pp; warning: p < 0.01 and >= 1.5x

p = 0.46

0.67% → 1.10% · 91 samples

Pass
Job failure rate

Job failure rate · ImportProductsJob

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

ImportProductsJob

0.00% → 0.00% · 22 samples

Pass
Job failure rate

Job failure rate · SendReceiptJob

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

SendReceiptJob

0.95% → 0.00% · 33 samples

Pass
Job failure rate

Job failure rate · SyncAccountsJob

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 0.26

SyncAccountsJob

0.83% → 2.78% · 36 samples

Pass
p95 latency

p95 latency

Rule: failing: >= 1.5x and >= +150 ms, sustained 3 evaluations; warning: >= 1.25x and >= +75 ms

240 ms → 264 ms · 300 samples

Pass
p95 latency

p95 latency · GET /

Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms

GET /

240 ms → 264 ms · 105 samples

Pass
p95 latency

p95 latency · GET /products

Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms

GET /products

264 ms → 218 ms · 75 samples

Pass
p95 latency

p95 latency · GET /products/:id

Rule: failing: >= 2.0x and >= +300 ms, sustained 3 evaluations; warning: >= 1.5x and >= +150 ms

GET /products/:id

264 ms → 319 ms · 60 samples

Pass
Queue latency p95

Queue latency p95

Rule: warning only: >= 3.0x and >= +30000 ms

2148 ms → 1213 ms · 91 samples

Pass
Route 5xx rate

Route 5xx rate · GET /

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

GET /

0.10% → 0.00% · 105 samples

Pass
Route 5xx rate

Route 5xx rate · GET /account

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

GET /account

0.00% → 0.00% · 24 samples

Pass
Route 5xx rate

Route 5xx rate · GET /products

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

GET /products

0.20% → 0.00% · 75 samples

Pass
Route 5xx rate

Route 5xx rate · GET /products/:id

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

GET /products/:id

0.08% → 0.00% · 60 samples

Pass
Route 5xx rate

Route 5xx rate · POST /orders

Rule: failing: rate >= 20% and baseline < 5% and p < 0.0001; warning: p < 0.001 and >= 3x

p = 1

POST /orders

0.28% → 0.00% · 36 samples

Pass

Next steps

What production exercised

What normally runs during this window, from the last 14 days. Each item counts once however busy it is, and one run is enough. Clearing needs 50% coverage, and high confidence needs 80%.

Normally active routes 5 of 5 · 100%
Regular job classes 3 of 3 · 100%

Critical flows

Flows that matter more than their traffic suggests, counted since this release rolled out.

  • Checkout

    POST /orderscheckpoint:receipt.sent

    Ran 36 times
  • Password reset

    POST /password_resetsjob:PasswordsMailer

    Not run yet

What changed

Since v181, from file digests.

small change
3 changed files (3 modified)
  • Gemfile.lock modified
  • README.md modified
  • app/models/order.rb modified

History

  1. First telemetry from this release

    Nov 1, 11:27 UTC

  2. Confidence Medium → High

    Nov 1, 11:28 UTC

  3. Failed: regression detected

    Nov 1, 11:29 UTC