Worked-example report v1 / Delimit team / 2026-07-16

Catching a dropped response field in an AI-written pull request

What a merge gate catches that a diff skim does not.

Delimit ran against two published snapshots of stripe/openapi between commit fbd0091 (spec version 2026-05-27.dahlia) and f4ac6d9 (2026-06-24.dahlia). 89 changes, 84 additive, and 4 response fields quietly removed from the card payment-method schema. The merge gate isolated the 4.

4 breaking · 84 additive

Why a worked example. A recognizable public API is the cleanest way to show what the gate sees on real change volume.

stripe/openapi is public; the two snapshots are real published commits.
This is not a defect report and Stripe is not a Delimit customer. The removed fields are on a preview payment-records surface and are almost certainly intentional. We use the diff only to demonstrate the gate.
The result is reproducible byte-for-byte from the same two commits (command at the end).

The problem

An AI coding assistant is great at "tidy up this schema" and "regenerate the spec from the models." It is also great at dropping a field nobody asked it to drop. When that happens inside a 400-line spec diff, the removal is four lines in a wall of additions. The pull request looks like an expansion. Ship it and every consumer reading the removed field breaks in production, not in review.

This report is that exact shape, drawn from real Stripe history: 84 of 89 changes are safe additions, and 4 are removed response fields on one card object. A human reviewer skims 89 entries. A generic "review this PR" prompt summarizes the additions. The merge gate does the one thing both tend to miss: it classifies every change and stops on the removals.

The setup

The merge gate is one job in CI. It runs on every pull request that touches the spec, diffs the proposed spec against the base branch, classifies each change, and fails the check on a breaking change under the policy you pick. For a payment API, the strict preset is the right default.

# .github/workflows/api-merge-gate.yml
name: API merge gate
on: [pull_request]

jobs:
  delimit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: delimit-ai/delimit-action@v1
        with:
          spec: openapi/spec3.json
          policy: strict

Headline numbers

Total changes

Breaking

Additive

Semver bump

major

The four removals are all field_removed on a single schema. Because any breaking change forces the bump, the classifier calls the release major. Under the strict policy preset the gate returns a hard fail (exit code 1, 4 violations); under the default preset the removals surface as warnings. Either way the four lines are no longer buried.

What free Delimit detects

4 breaking changes, plus 2 representative additive findings so the signal ratio is visible. Every finding cites the change-type from the taxonomy and the exact JSON Pointer. This is mechanical diff output, annotated.

breakingfinding F1
change type: field_removed
surface: payments_primitives_payment_records_resource_payment_method_card_details.issuer
The issuer field was removed from the card payment-method details object. Any consumer that reads response.payment_method_details.card.issuer to display or reconcile the issuing bank now reads undefined. Nothing in the request shape changes, so this break is invisible until a live response comes back missing the field.
breakingfinding F2
change type: field_removed
surface: payments_primitives_payment_records_resource_payment_method_card_details.iin
The iin field (issuer identification number, the leading digits that identify the card network and bank) was removed from the same object. Fraud, routing, and BIN-lookup logic keyed on this field silently loses its input.
breakingfinding F3
change type: field_removed
surface: payments_primitives_payment_records_resource_payment_method_card_details.description
The description field was removed. Code that surfaces this string in receipts or admin tooling now renders an empty value.
breakingfinding F4
change type: field_removed
surface: payments_primitives_payment_records_resource_payment_method_card_details.stored_credential_usage
The stored_credential_usage field was removed. Consumers that branch on stored-credential metadata (recurring vs one-off) lose the signal. Four fields, one schema, all four flagged breaking.
additivefinding A1
change type: optional_field_added (58 across the surface)
surface: many component schemas
The bulk of the release is pure expansion: 58 new optional fields landed across the object graph. Every one is safe for existing clients, which is exactly why they are noise for a breaking-change review. A reviewer scrolling 89 diff entries spends almost all of their attention here.
additivefinding A2
change type: enum_value_added (26)
surface: many enum fields
26 new enum values were added across the spec. Additive for a reader of the enum; a producer that emits a new value is only a problem for a consumer with an exhaustive switch, which the taxonomy treats as non-breaking by default. More safe volume for the four breaking removals to hide behind.

Why a generic AI review misses it

"Review this PR for breaking changes" is a summarization task, and summarizers are drawn to volume. 84 additions dominate the token budget and the narrative; the model reports a healthy expansion and, often, does not enumerate the four removals at all. There is no ground-truth taxonomy behind the answer, so a missed removal reads exactly like a clean review.

The merge gate is not a summarizer. It compares two specs structurally, assigns every delta a change-type, and applies a deterministic policy. The same two files always produce the same verdict. A removed response field cannot hide in additive volume, because volume is not how the gate reads the diff.

The migration guide the gate generates

When the gate flags breaking changes it also emits a migration guide keyed to each one. This is the verbatim head of the generated output for this diff:

# Migration Guide: API (2026-05-27.dahlia -> 2026-06-24.dahlia)

This release contains 4 breaking change(s).
Follow the steps below to update your integration.

### Step 1: Field Removed
Change:   Optional field 'issuer' removed at
          payment_method_card_details
Action:   Remove any references to this field in your
          response parsers. Accessing it will return
          undefined/null.

### Step 2: Field Removed
Change:   Optional field 'iin' removed at
          payment_method_card_details
Action:   Remove any references to this field in your
          response parsers.

# ... steps 3-4 cover description and stored_credential_usage

The guide is generated from the classified diff, not written by hand. Its job is to turn "this is breaking" into "here is the consumer-side change per field."

Reproduce locally

Re-run the analysis against the same two commits and the same diff comes out:

# Install the CLI
npm install -g delimit-cli

# Clone the spec repo
git clone https://github.com/stripe/openapi
cd openapi

# Extract the two published snapshots
git show fbd0091:openapi/spec3.json > /tmp/old.json
git show f4ac6d9:openapi/spec3.json > /tmp/new.json

# Run the merge gate (strict = payment-grade policy)
delimit lint --policy strict /tmp/old.json /tmp/new.json

If your bytes differ from this report, that is itself a finding worth raising on the Delimit repo.

Free detects. Pro proves.

Free Delimit is the merge gate: it catches the four removed fields on every pull request and blocks the merge before the break ships. Install delimit-cli and run delimit lint <old> <new> against your own specs, or drop the delimit-ai/delimit-action into CI. Free for individual maintainers.

Pro turns the caught verdict into a signed, replayable attestation: a portable record that this pull request was checked, what was found, and how it was classified. Your reviewers, auditors, and downstream consumers can verify it without rerunning the gate. Pro tier $10/month for teams.