Skip to content

feat: automate weekly AI model price book refresh - #28146

Merged
evgeniy-scherbina merged 31 commits into
mainfrom
yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book
Aug 19, 2026
Merged

feat: automate weekly AI model price book refresh#28146
evgeniy-scherbina merged 31 commits into
mainfrom
yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book

Conversation

@evgeniy-scherbina

@evgeniy-scherbina evgeniy-scherbina commented Aug 13, 2026

Copy link
Copy Markdown
Contributor

Example of generated PR: #28228

Summary

The price book is regenerated by hand with make gen/aibridge-prices, so keeping it current depends on someone remembering to run it. Newly launched models stay unpriced and changed prices stay stale until the next manual run.

This adds a weekly workflow that regenerates both artifacts from live models.dev data and opens a pull request when the output changes.

For scale, a run against main today produced 32 models added, 11 removed, and 43 repriced.

What it does

  • Runs every Thursday at 09:00 UTC, plus workflow_dispatch. Exits without opening anything when regeneration produces no diff.
  • Reuses one branch and one pull request, force-pushing each week, so at most one refresh PR is open and it always carries the newest snapshot.
  • Ships prices.json and knownModelsGenerated.json together, since both come from a single upstream snapshot.
  • Never merges automatically. Prices are customer-visible cost numbers, so every change goes through human review.
  • Announces failures in Slack. The generator fails by design when upstream drops a model pinned in overrides.jq or curated in curation.json, and a silently red weekly cron would defeat the point.

scripts/aibridgepricesdiff renders the PR body summary: counts, then the models added, removed, and repriced. Exact figures stay in the diff rather than being restated in the body. It is deterministic and unit tested rather than AI-generated, so the model lists cannot drift from the file.

CODEOWNERS covers the two generated artifacts, which are exactly the files a refresh PR touches.

Verification

Ran on this branch via a temporary push trigger, since schedule and workflow_dispatch only fire from the default branch. Confirmed end to end: mise and pnpm setup, generation from live models.dev, change detection, summary and body assembly, commit under the bot identity, branch creation, force-push on the second run, and Slack delivery.

gh pr create and gh pr edit remain unverified: both runs hit HTTP 503 from api.github.com/graphql during a GitHub major outage. The push path itself succeeded, so the credential and permissions are proven.

Also verified locally: go test ./scripts/aibridgepricesdiff/, make lint/actions/actionlint, make lint/actions/zizmor, and make pre-commit.

Notes

  • The PR is opened with secrets.CDRCI_GITHUB_TOKEN. PRs opened with the default GITHUB_TOKEN do not trigger workflow runs, which would leave the refresh without CI signal on a file that feeds cost calculation.
  • Requires the AIGATEWAY_PRICES_SLACK_WEBHOOK repository secret. The notification step fails loudly if it is unset, since by then the refresh has already failed and nobody would be told.
  • The workflow installs pnpm dependencies because catalog generation formats its output with biome; without site/node_modules the formatting step silently no-ops and an unformatted file would be committed.
  • A refresh that lands after a release branch is cut can reach that release through the existing cherry-pick label, so no separate release-cut trigger is included. The label is not applied automatically; that stays a reviewer decision.
  • Identifiers keep the aibridge spelling where they name real paths and make targets (coderd/aibridge/prices, make gen/aibridge-prices, scripts/aibridgepricesgen). The workflow itself is named aigateway-prices-refresh, and prose says AI Gateway per the style guide.
  • The issue also asked for code owners on the generator. Only the generated artifacts are owned here, since those are the files a refresh PR touches. Ownership of curation.json and overrides.jq can be added separately if human edits to them should request review.
Implementation plan and decision log

Goal

Refresh make gen/aibridge-prices output on a schedule and open a human-reviewed PR when it changes, so releases never ship a stale price book.

Schedule

Weekly, Thursdays 09:00 UTC, leaving Thursday, Friday, and Monday for review before Tuesday releases. A monthly run was considered first. Release ships Tuesday morning and the freeze is the last Tuesday before EOM, so a monthly run on the 19th or 20th collapses to 1-2 usable business days in roughly 10 of 132 months: February, when the release Tuesday is the 22nd and the run day falls on a weekend. Measured across 2026-2036:

Cron day Worst case usable business days Months with 2 or fewer Median
20th 1 10 6
19th 1 10 6
16th 3 0 6
Weekly Thursday n/a, always a full week 0 n/a

The median is 6 days for every candidate, so moving to weekly costs nothing and removes the bad tail. The shipped book is never more than 7 days stale.

Branch strategy

One fixed branch, force-pushed, rather than a new branch and PR per run. prices.json is a full regeneration from the current upstream snapshot, not an incremental patch, so two open refresh PRs are the same file at two points in time and the older one is strictly wrong. A single PR makes merging a stale snapshot impossible, keeps reviewer load at one PR, and avoids a full CI run per week per stale PR.

Summary generation

Deterministic Go tool, not an AI summary. An LLM summarizing a JSON diff can drop or invent a row, and would be non-reproducible run to run. A jq implementation was considered and rejected as effectively untestable. The body lists which models moved; the diff in the Files tab remains the source of truth for figures.

An earlier version rendered full price tables with percentage deltas. That duplicated the diff, so it was reduced to model lists.

Token

secrets.CDRCI_GITHUB_TOKEN, the cdrci machine user already used for bot-authored PRs in release.yaml and cited for exactly this reason in the commented-out update-flake job in ci.yaml. A dedicated GitHub App scoped to this repo would be tighter, but needs org admin to create.

Commit identity

github-actions[bot] with its numeric noreply address, matching backport.yaml and cherry-pick.yaml. An invented address would leave refresh commits with an unlinked author.

Alerting

A dedicated webhook secret rather than reusing CI_FAILURE_SLACK_WEBHOOK. The repo already splits alerting per domain (docs, security, dependabot each have their own webhook), and the CI-failure channel carries a user mention plus vars.BLINK_CI_FAILURE_PROMPT that feeds an automated triage flow this payload does not belong in.

GitHub's built-in email notification was considered and rejected as the primary channel: for scheduled workflows it goes to a single user, the one who last modified the cron syntax, and cannot be routed to a team.

Out of scope

Triggering a refresh when a release branch is cut. The existing cherry-pick label already gets a late refresh into a release, so a separate mechanism is unnecessary.


Closes AIGOV-578.

Authored by Coder Agents on behalf of @evgeniy-scherbina.

The price book is regenerated by hand with `make gen/aibridge-prices`, so
newly launched models stay unpriced and changed prices stay stale until
someone remembers to run it.

Add a weekly workflow that regenerates both artifacts from live models.dev
data and opens a pull request when the output changes. The refresh is never
merged automatically: prices are customer-visible cost numbers, so every
change goes through human review. A generator failure, which happens by
design when upstream drops a pinned or curated model, is announced in Slack.

The pull request body carries a generated summary of models added, models
removed, and prices changed, rendered by a new deterministic
scripts/aibridgepricesdiff tool.

Add CODEOWNERS entries for the price book, the frontend catalog, and both
generators so the right reviewer is requested on every refresh.
@linear-code

linear-code Bot commented Aug 13, 2026

Copy link
Copy Markdown

AIGOV-578

The pull request diff already shows exact prices for every added, removed,
and changed row, so repeating them in the body duplicated what a reviewer
can read directly. Keep the counts and the model lists, which answer what
moved without restating the diff.
… fields

The per-field count did not match the model list below it and needed a
qualifier to reconcile. Counting models keeps every figure on the same
unit.
… fields

The summary lists models per category, so tracking which individual price
field moved produced detail nothing rendered. Compare rows as a unit and
carry model keys through, which removes the per-field change type, the
field accessor table, and the two name-mapping helpers.

Every category now holds at most one entry per model, so a single key
comparator gives a total order and the stable-sort requirement disappears.

Verified byte-identical output against the previous implementation on a
live upstream snapshot.
…ir own lines

Cases with more than one model packed both rows onto a single line, which
made the difference between the old and new snapshot hard to spot when
reviewing.
…ixtures

Struct literals packed two fields onto a line, and cases mixed an inline
old snapshot with a multiline new one, so the two sides did not line up
when read side by side. Inline both only when each snapshot holds a single
row.
…ror messages

Substring assertions left section order, spacing, and stray content
unverified: reordering the Removed and Changed sections passed the suite.
Compare the whole rendered summary for both the empty and populated cases.

Assert on error contents rather than mere failure, and cover a missing
input file, which the workflow hits if the snapshot step is ever skipped.
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch from 585b807 to 9d17c89 Compare August 17, 2026 16:29
The identity was a plausible-looking address that is not verified on any
GitHub account, so refresh commits would show an unlinked author. Use the
bot identity that backport.yaml and cherry-pick.yaml already use, whose
numeric noreply address associates the commit with a real profile.
The style guide's word-choice table lists AI Bridge as a form to avoid.
Identifiers stay as they are, since they name the paths and make targets
this workflow drives: coderd/aibridge/prices, make gen/aibridge-prices,
and scripts/aibridgepricesgen.
The refresh PR touches exactly these two files, so directory-level entries
on the generators never fired on it. The knownModels directory also holds
hand-written TypeScript, which would have requested review on unrelated
frontend work.
New artifacts follow the aigateway naming; a file named aibridge would be
invisible to anyone grepping the current name. The branch and concurrency
group move with it, since neither exists yet. Paths, make targets, and the
scripts keep their aibridge names, which match what they refer to.
…RICES_SLACK_WEBHOOK

Matches the workflow name and the <SCOPE>_SLACK_WEBHOOK convention. The
secret does not exist yet, so renaming now costs nothing.
DO NOT MERGE. Drop this commit after verifying the run.
The alert built its text in double quotes, so bash kept \n as two
characters and jq --arg escaped the backslash. Slack printed a literal \n
mid-sentence instead of breaking the line; build the text with printf.
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch from 4f5240c to 8ee6bd9 Compare August 17, 2026 19:01
The step only runs after the refresh has already failed, so exiting 0 hid
the more serious problem: nobody was told. Mark the step red and name the
secret, now that it exists and an empty value means a misconfiguration.
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch from a806862 to 643fee9 Compare August 17, 2026 19:43
Comment thread .github/workflows/aigateway-prices-refresh.yaml Outdated
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch from 640a76c to 643fee9 Compare August 17, 2026 20:06
gh pr list/create/edit go through GraphQL, whose queries resolve reviewer
and team fields that require the read:org scope. cdrci's token carries only
repo and workflow, so the calls succeed while no pull request exists and
fail once CODEOWNERS attaches a reviewer: the first refresh would open a PR
and every run after it would fail.

Observed on this branch: run 32058047624 created #28224, then 32058781222
and 32059086614 both failed on gh pr list with a read:org scope error. The
equivalent REST endpoints need only repo.
@evgeniy-scherbina
evgeniy-scherbina force-pushed the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch from 643fee9 to 5ae4a94 Compare August 17, 2026 20:14
…acts

Spreads review of the weekly refresh beyond one person, so a stale price
book is not blocked on a single reviewer's availability.
Comment thread .github/workflows/aigateway-prices-refresh.yaml
Comment thread scripts/aibridgepricesdiff/main_test.go
Comment thread scripts/aibridgepricesdiff/main_test.go
Comment thread scripts/aibridgepricesdiff/main.go
Comment thread scripts/aibridgepricesdiff/main.go
Comment thread .github/workflows/aigateway-prices-refresh.yaml Outdated
@evgeniy-scherbina

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

@coder-agents-review

coder-agents-review Bot commented Aug 18, 2026

Copy link
Copy Markdown
Contributor

Chat: Review posted | View chat
Requested: 2026-08-19 14:59 UTC by @evgeniy-scherbina

Review history
  • R1 (2026-08-18), 1 P1, COMMENT. Review
  • R2 (2026-08-19): 18 reviewers, 8 Nit, 1 Note, 1 P1, 2 P2, 2 P3, COMMENT. Review

deep-review v0.9.0 | Round 2 | d5bb35a..71197af

Last posted: Round 2, 14 findings (1 P1, 2 P2, 2 P3, 8 Nit, 1 Note), COMMENT. Review

Finding inventory

Finding inventory

Findings

# Sev Status Location Summary Round Reviewer Posted
CRF-1 P1 Author fixed (71197af) .github/workflows/aigateway-prices-refresh.yaml:26 TEMPORARY push trigger for personal branch still present at HEAD (commit ff7cf11 labelled "DO NOT MERGE") R1 Netero Yes
CRF-2 P2 Open scripts/aibridgepricesdiff/main.go:30 priceRow is a 4th verbatim copy of the price-seed schema; sync-note comments in the sibling files still say 3, so a new price column added later silently renders as "No price changes." R2 Netero P2, Robin P2, Zoro P2, Ryosuke P3, Razor P3, Hisoka P4 Yes
CRF-3 P3 Open .github/workflows/aigateway-prices-refresh.yaml:69 Change detector fires on either PRICES_FILE or CATALOG_FILE, but the summary tool reads only prices, so a catalog-only refresh opens a PR whose body reads "No price changes." R2 Hisoka P3, Kite P3, Meruem P3, Ryosuke P3, Mafuuu Nit, Pariston Note, Razor Note, Zoro Note Yes
CRF-4 P2 Open .github/workflows/aigateway-prices-refresh.yaml:168 Slack failure alert asserts one specific root cause (overrides.jq / curation.json) for every failure mode in the job R2 Leorio P2, Chopper P3, Mafuuu P3, Meruem P3, Pariston Note, Razor Nit Yes
CRF-5 Nit Open .github/workflows/aigateway-prices-refresh.yaml:113 Prose name drift: "AI model price book" in PR_TITLE and Slack failure text vs "AI Gateway price book" everywhere else (style guide) R2 Chopper, Gon, Leorio, Ryosuke Yes
CRF-6 P3 Open .github/workflows/aigateway-prices-refresh.yaml:45 Checkout has no explicit ref, so workflow_dispatch from a non-main branch would force-push that branch's history plus one price commit to the shared refresh PR R2 Hisoka P3, Ryosuke Note Yes
CRF-7 Note Open CODEOWNERS:49 New rule silently displaces @DanielleMaywood as owner of knownModelsGenerated.json (last-match-wins vs line 31) R2 Netero Yes
CRF-8 Nit Open scripts/aibridgepricesdiff/main.go:135 sort.Slice superseded by slices.SortFunc since Go 1.21; go.mod is on 1.26.5 R2 Ging-go, Meruem Yes
CRF-9 Nit Open scripts/aibridgepricesdiff/main.go:60 diff type doc missing terminating period; diverges from sibling docs in the same file R2 Gon, Leorio Yes
CRF-10 Nit Open .github/workflows/aigateway-prices-refresh.yaml:22 Trailing # allows manual runs for testing is bloat; narrows the intent of workflow_dispatch without evidence R2 Gon Yes
CRF-11 Nit Open scripts/aibridgepricesdiff/main.go:109 compare doc omits the deterministic-sort contract that TestCompareSortsDeterministically locks in R2 Leorio Yes
CRF-12 Nit Open .github/workflows/aigateway-prices-refresh.yaml:9 Header Behavior block doesn't mention the no-change Slack heartbeat added in commit 915c61b R2 Leorio Yes
CRF-13 Nit Open .github/workflows/aigateway-prices-refresh.yaml:32 PRICES_FILE / CATALOG_FILE hardcode the two Makefile output paths; a Makefile-side rename that isn't mirrored here silently drops that side of the refresh R2 Mafuuu Yes
CRF-14 Note Open .github/workflows/aigateway-prices-refresh.yaml:150 Slack step gates on failure() || changed == 'false'; cancellation is neither, so cancelled runs emit no signal on a workflow whose stated point is not being silent R2 Mafuuu Yes
CRF-15 Nit Open .github/workflows/aigateway-prices-refresh.yaml:168 Slack failure text spells scripts/aibridgepricesgen/overrides.jq with full path but curation.json without; both live in the same directory R2 Meruem Yes

Contested and acknowledged

None.

Round log

Round 1

Netero-only. 1 P1. Panel gated; will run after the P1 is addressed. Reviewed against d5bb35a..4cca010.

Round 2

Panel. CRF-1 (P1) verified fixed in 71197af. 2 P2, 3 P3, 8 Nit, 2 Note new. 18 reviewers (Netero + panel of 15 trigger-matched + 2 wildcards Meruem and Razor). Reviewed against d5bb35a..71197af.

About deep-review

CRF = Coder Review Finding (P0-P4, Nit, Note)

Reviewer Focus
Bisky tests
Chopper ops/errors
Churn-guard change verification
Ging language modernization
Gon naming
Hisoka edge cases
Killua perf
Kite change integrity
Knov contracts
Knuckle SQL
Komugi flake/determinism
Kurapika security
Law decomposition
Leorio docs
Luffy product
Mafu-san process
Mafuuu contracts
Melody dispatch/pairing
Meruem structural
Nami frontend
Netero mechanical checks
Pariston premise testing
Pen-botter product gaps
Razor verification
Robin duplication
Ryosuke Go arch
Takumi concurrency
Zoro shape

πŸ€– Managed by Coder Agents.

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First-pass review by Netero only. One mechanical finding blocks the panel: the workflow file still carries a push: trigger for a personal WIP branch, added by a commit whose own message says "DO NOT MERGE." The author already flagged the same block for removal at line 23. Drop that commit (or the block) and the full panel will review on the next round.

Out of scope for this round: the open thread suggestions (--force-with-lease, cmp.Diff, ${provider}/${model} vs struct, cron day) and the shell-vs-Go question are panel-domain calls. They will be evaluated when the panel runs.

No review-body signature.

πŸ€– This review was automatically generated with Coder Agents.

Comment thread .github/workflows/aigateway-prices-refresh.yaml Outdated

@ssncferreira ssncferreira left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM πŸš€ just a few nits and suggestions

Comment thread .github/workflows/aigateway-prices-refresh.yaml Outdated
// priceRow mirrors the seed file schema written by aibridgepricesgen. Pointer
// fields preserve the distinction between "not populated by upstream" (null)
// and "explicitly zero" (0).
type priceRow struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably move this to a shared package, as it is used by aibridgepricesgen and https://github.com/coder/coder/blob/main/coderd/aibridge/prices/prices.go#L26C6-L26C13

And possibly other structs/method. Can be done in a follow-up cleanup PR πŸ‘

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, I can fix it in a follow-up PR

Comment thread scripts/aibridgepricesdiff/main_test.go
Comment thread scripts/aibridgepricesdiff/main_test.go
Comment thread scripts/aibridgepricesdiff/main.go
Monday leaves only one business day before Tuesday releases and competes
with release preparation. Thursday leaves Thursday, Friday, and Monday for
review without changing the weekly freshness bound.
Complements the value-to-null case and verifies that an existing model whose
price becomes available is reported as changed, not added.
Complements zero-to-null and verifies that an explicit zero is treated as a
populated price in both directions.
Large upstream refreshes can add or reprice dozens of models, which buries
the review notes under a long body. Keep the counts visible and put each
model category in a native GitHub details block so reviewers can expand only
the sections they need.
GitHub wraps rendered prose to the viewer width, so hard-wrapping the source
at roughly 70 characters only made the raw Markdown and copied text look
cramped.
When generation is unchanged there is no refresh PR, so a successful no-op
and a scheduled workflow that never ran are otherwise indistinguishable.
Post a short heartbeat only for the no-change path; a refresh PR remains the
success signal when changes exist.
Select the failure alert or no-change heartbeat in one final step. This
removes duplicated webhook validation, JSON encoding, and curl setup, and
prevents a failed heartbeat delivery from triggering a second, contradictory
failure notification.
…ontext

JOB_STATUS makes it clear that the value comes directly from job.status and
is not a custom refresh result.
The workflow has now been exercised end to end on the feature branch,
including refresh PR creation, update, force-push, and Slack failure delivery.
Keep only the production schedule and manual dispatch triggers.
@evgeniy-scherbina

Copy link
Copy Markdown
Contributor Author

/coder-agents-review

…tomate-updates-to-the-shipped-ai-model-price-book

@ssncferreira ssncferreira left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM :shipit:

@coder-agents-review coder-agents-review Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Panel review, round 2. CRF-1 (temporary push trigger) fixed cleanly in 71197af.

Severity: 2 P2, 3 P3, plus 2 Notes and 8 Nits. Nothing shipping-blocking; two P2s worth resolving before merge.

Structural theme: the summary contract is narrower than the diff it fronts. The change detector at line 69 fires on either PRICES_FILE or CATALOG_FILE, but the diff tool reads only prices and the PR body renders "No price changes." whenever the catalog moved alone (CRF-3). The priceRow struct at scripts/aibridgepricesdiff/main.go:30 is a fourth verbatim copy of the price seed schema, and the sibling files at scripts/aibridgepricesgen/main.go:69 and coderd/aibridge/prices/prices.go:23 still carry the note "the JSON contract for the price seed lives in three places that must stay in sync," now inaccurate. When a new price column lands upstream, json.Unmarshal drops the field, samePrices returns true, and the weekly PR renders "No price changes." over a real reprice (CRF-2). Both failures land in the same place: the reviewer's first read tells them nothing moved. @ssncferreira's earlier thread on main.go:30 flagged the schema location as follow-up material; the panel is elevating it to P2 with a concrete customer-visible consequence, so it needs either a fix here, a linked ticket, or an explicit human decision to accept the gap.

Operational theme: the Slack failure alert at line 168 asserts one specific root cause ("the generator fails by design when upstream drops a model pinned in overrides.jq or curated in curation.json") but the if: fires on every failed step in the job. The PR body itself records that the author's end-to-end verification run hit HTTP 503 from api.github.com, which is exactly a non-generator failure that would land in Slack under this message. On-call would open overrides.jq first and lose time (CRF-4).

One security-adjacent risk: Checkout has no ref:, so workflow_dispatch from a non-main branch would build the refresh PR from that branch's history plus one price commit and force-push it to the shared bot branch. Pin the base with ref: main or a job-level if: github.ref == 'refs/heads/main' (CRF-6).

Process observation, not a finding: the PR description now names tooling the workflow no longer uses. The Verification section says gh pr create and gh pr edit are "unverified due to HTTP 503," but commit 5ae4a9494b replaced both with gh api REST calls whose message cites a run where gh pr create did succeed before subsequent gh pr list hit a read:org scope error. The "What it does" list still says only failures go to Slack, but commit 915c61be9c added a no-change heartbeat. The current REST create/update path isn't listed under "Confirmed end to end." Worth refreshing the description against the code that's actually shipping. Same class of drift the round-1 P1 lived in: verified once, not re-checked after subsequent commits.

Specialist coverage: Bisky, Luffy, Komugi, Kurapika returned no findings. Praise from Kite on the workflow security posture (permissions: {} at workflow and job, harden-runner, pinned SHAs, persist-credentials: false, privileged token scoped to one step's env). Bisky called out the test suite: "Every require.Equal I traced actually runs against the value produced by the function under test."

From Ryosuke, on the schema duplication: "Three-comments-in-sync is a rule; one imported type is a fact."

CI: Flake Check job failed (run 96114162391). Not surfaced by any reviewer as related to the diff; treating it as unrelated flake noise unless someone traces it to this PR.

πŸ€– This review was automatically generated with Coder Agents.

// priceRow mirrors the seed file schema written by aibridgepricesgen. Pointer
// fields preserve the distinction between "not populated by upstream" (null)
// and "explicitly zero" (0).
type priceRow struct {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-2] Fourth verbatim copy of the price-seed schema; two adjacent sync-note comments still say the schema lives in three places. (Netero P2, Robin P2, Zoro P2, Ryosuke P3, Razor P3, Hisoka P4)

coderd/aibridge/prices/prices.go:23-25 and scripts/aibridgepricesgen/main.go:68-71 both declare that "the JSON contract for the price seed lives in three places that must stay in sync": the seeder struct, the generator struct, and the batch SQL upsert in coderd/database/queries/aicostcontrol.sql. This PR adds priceRow here as a silent fourth. json.Unmarshal ignores unknown fields, so when a new price column lands upstream (say cache_hit_price), readRows will drop it, samePrices will not compare it, and compare will report 0 changed for a real repricing.

Consequence: the weekly refresh renders "No price changes." over a real reprice, and the summary the reviewer is asked to trust is quietly narrower than the diff below it. The tool exists precisely so reviewers see what moved; this bug hazard defeats its purpose on the first schema extension.

Two fixes:

  1. Minimal, matches existing pattern: update the two sibling NOTE comments to say four places and enumerate the diff tool.
  2. Structural: extract the row struct into a leaf schema package (e.g., coderd/aibridge/prices/priceseed) that owns only the type and JSON tags, import it from both scripts, and delete the drift-warning comments. go build then catches the drift the comment was asking humans to catch.

Ryosuke's framing: three-comments-in-sync is a rule; one imported type is a fact. @ssncferreira's earlier thread suggested the same structural move as a follow-up; the panel is elevating this to P2 because the consequence (silent "No price changes" on future column additions) is real and customer-visible, and there is no ticket linked to the follow-up commitment. Needs a fix here, a linked ticket, or an explicit human decision to accept the gap.

πŸ€–

id: detect
run: |
set -euo pipefail
if git diff --quiet -- "${PRICES_FILE}" "${CATALOG_FILE}"; then

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-3] Catalog-only refresh opens a PR titled "chore: refresh AI model price book" whose body reads "No price changes." (Hisoka P3, Kite P3, Meruem P3, Ryosuke P3, plus Mafuuu Nit, Pariston Note, Razor Note, Zoro Note)

Detect changes flips changed=true when either ${PRICES_FILE} or ${CATALOG_FILE} differs, yet Build pull request body runs aibridgepricesdiff against the two prices.json snapshots only. knownModelsGenerated.json carries contextLimit, maxOutputTokens, reasoningEffort, displayName, and aliases, none of which are in prices.json; when upstream changes one of those without touching prices, a PR opens whose body reads ## Price book changes\n\nNo price changes. The review note two lines further down claims "The summary above lists what moved", which is now false.

The class of bug is that "did anything change" lives in two places (the git-diff gate and the tool's price-only rendering) and the two can disagree. Two fixes eliminate the gap rather than paper over it:

  1. Snapshot ${CATALOG_FILE} too and teach aibridgepricesdiff to name catalog moves alongside prices; rename the heading (## Snapshot changes or similar) so the label does not promise a price-only view.
  2. Gate the PR-open decision on PRICES_FILE changing alone, letting a catalog-only drift wait for the next price move. Cheaper but delays catalog updates.

Option 1 is the more honest fix. Reviewers on this PR are the same for both artifacts, so the routing is fine; the mismatch is cognitive but predictable and repeated weekly.

πŸ€–

# printf, not a double-quoted literal: bash leaves \n as two
# characters, and jq --arg then escapes the backslash, so Slack
# would print \n as text instead of breaking the line.
text="$(printf ':warning: *AI model price book refresh failed.*\nThe generator fails by design when upstream drops a model pinned in scripts/aibridgepricesgen/overrides.jq or curated in curation.json. Logs: %s' "${RUN_URL}")"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 [CRF-4] Slack failure alert asserts one specific root cause for every failure mode in the job. (Leorio P2, Chopper P3, Mafuuu P3, Meruem P3, Pariston Note, Razor Nit)

At 2 AM:

⚠️ AI model price book refresh failed.
The generator fails by design when upstream drops a model pinned in scripts/aibridgepricesgen/overrides.jq or curated in curation.json. Logs: <run_url>

The step's if: is failure() || steps.detect.outputs.changed == 'false', so this branch runs when anything failed: Harden Runner, Checkout, Set up mise tools, Install pnpm dependencies, make gen/aibridge-prices for reasons unrelated to model drops, git push, the gh api calls (the PR description itself records that the verification runs hit HTTP 503 from api.github.com/graphql). The on-call reader is anchored to overrides.jq / curation.json before opening the run log, burns time confirming those files are fine, and only then reads the actual failure.

Diagnose, don't guess. Name the operation and run URL first; if a leading hypothesis is worth keeping, present it as one common cause, not the cause:

:warning: *AI Gateway price book refresh failed.* Logs: %s
A common cause is upstream dropping a model pinned in scripts/aibridgepricesgen/overrides.jq or scripts/aibridgepricesgen/curation.json. If the logs point elsewhere, that is the real cause; treat this hint as a starting point, not a diagnosis.

πŸ€–

with:
egress-policy: audit

- name: Checkout

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P3 [CRF-6] Checkout has no ref:, so a workflow_dispatch from a non-main branch smuggles that branch's history into the shared refresh PR. (Hisoka P3, Ryosuke Note)

Checkout has no ref:, so it defaults to github.ref. On schedule that is fine. On workflow_dispatch GitHub lets any actor with write pick a branch, and github.ref becomes that branch. Line 123 then git checkout -b "${REFRESH_BRANCH}" off that HEAD, git add only the two artifact paths, git commit, and git push --force refs/heads/bot/aigateway-prices-refresh. The remote refresh branch is now that topic branch's history plus one price commit. Line 142 creates the PR with base=main, so the diff shown to reviewers is main..<topic> for every file the topic branch touched, not just prices.

The implementation plan explicitly says workflow_dispatch "only fire from the default branch." It does not. Only maintainers can dispatch, so this is unlikely by accident, but the round-1 P1 (a temporary push: trigger for a personal branch, added because the author wanted to test from a branch) shows the pattern where testing from a non-main branch is real. A successor dispatching to test a change would open a refresh PR shipping unrelated commits. Reviewer opens a many-file diff titled "chore: refresh AI model price book" and either closes it in confusion or, worst case, merges it.

Two mechanical fixes, either sufficient:

  1. Pin the checkout ref: add ref: main under with: on the Checkout step, or git fetch origin main && git checkout -b "${REFRESH_BRANCH}" origin/main before staging.
  2. Guard the job: jobs.refresh.if: github.ref == 'refs/heads/main'. Costs a manual main re-dispatch to test.

πŸ€–

# with GITHUB_TOKEN do not trigger workflow runs, so the refresh
# would arrive without CI signal.
GH_TOKEN: ${{ secrets.CDRCI_GITHUB_TOKEN }}
PR_TITLE: "chore: refresh AI model price book"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-5] Prose name drift: "AI model price book" here and in the Slack failure text (line 168) versus "AI Gateway price book" in the workflow header (line 1) and the heartbeat text (line 170). (Chopper, Gon, Leorio, Ryosuke)

The PR description states the naming policy explicitly: identifiers stay aibridge, prose says "AI Gateway". The file header at line 1 follows that ("Refreshes the AI Gateway price book"), and the no-change Slack text at line 170 follows it too.

PR_TITLE: "chore: refresh AI model price book" produces the weekly commit message and PR title, so every future refresh commit lands in main's history spelling the object differently from the workflow that produced it. Line 168 has the same drift right next to the compliant heartbeat two lines down, which also fragments Slack search ("AI Gateway price book" misses the failure alerts).

Change both to "AI Gateway price book".

πŸ€–

--jq '.html_url'
fi

- name: Send Slack notification

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-14] Cancelled runs go silent because the notification step gates on failure() || changed == 'false', and failure() does not fire on cancellation. (Mafuuu)

If a scheduled run is manually cancelled or superseded by the concurrency group, no Slack signal is emitted. The workflow's stated intent is "Failures are announced in Slack"; cancellation is not a failure, but for a weekly cron whose whole point is to not be silent, on-call may want to know a run was killed mid-flight instead of finding out at the next heartbeat. Consider if: always() && steps.detect.outputs.changed != 'true' with a cancelled branch, or accept the gap and document it in the header comment.

πŸ€–

Comment thread CODEOWNERS
# models.dev data, and they carry customer-visible cost numbers, so every
# refresh needs a review from someone who owns them.
coderd/aibridge/prices/data/prices.json @evgeniy-scherbina @ssncferreira @johnstcn
site/src/pages/AgentsPage/components/ChatModelAdminPanel/knownModels/knownModelsGenerated.json @evgeniy-scherbina @ssncferreira @johnstcn

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note [CRF-7] New rule silently displaces @DanielleMaywood as an owner of knownModelsGenerated.json. (Netero)

CODEOWNERS uses last-match-wins. Line 31 (site/src/pages/AgentsPage/ @DanielleMaywood, unchanged by this PR) previously covered the frontend catalog. The new rule at line 49 lists only @evgeniy-scherbina @ssncferreira @johnstcn, so any change to that file (workflow or manual) now requests review only from those three. The PR body discusses ownership of curation.json and overrides.jq but not this narrowing.

Likely intentional (the file is generated and the new owners are the pricing-data reviewers), but the surrounding comment does not state the intent and the displacement is invisible in the diff. Confirm this is the intent, and add a line to the comment noting that the frontend owner is deliberately excluded from generated catalog updates.

πŸ€–

}

for _, keys := range [][]modelKey{d.added, d.removed, d.changed} {
sort.Slice(keys, func(i, j int) bool { return less(keys[i], keys[j]) })

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-8] sort.Slice is superseded by slices.SortFunc since Go 1.21; .claude/docs/GO.md names slices.SortFunc as the modern replacement. (Ging-go, Meruem)

go.mod is on go 1.26.5. Drop the less helper and the sort import:

slices.SortFunc(keys, func(a, b modelKey) int {
    return cmp.Or(cmp.Compare(a.provider, b.provider), cmp.Compare(a.model, b.model))
})

πŸ€–

return a.model < b.model
}

// diff is the full comparison between two snapshots

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-9] diff type doc is missing a terminating period. (Gon, Leorio)

// diff is the full comparison between two snapshots diverges from the sibling docs in the same file (priceRow at 27-29, modelKey at 39). Add the period.

πŸ€–

return rows, nil
}

// compare classifies every model as added, removed, or changed.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit [CRF-11] compare doc omits the deterministic-sort contract. (Leorio)

The comment says "classifies every model as added, removed, or changed." The function also sorts each output slice by (provider, model), and TestCompareSortsDeterministically locks that behavior in. render iterates each slice in order and depends on it for stable PR bodies. A reader working only from the doc would think the output order is iteration-dependent. Extend:

// compare classifies every model as added, removed, or changed, and sorts
// each category by (provider, model) so the rendered summary is stable.

πŸ€–

@evgeniy-scherbina
evgeniy-scherbina merged commit 05699c4 into main Aug 19, 2026
29 checks passed
@evgeniy-scherbina
evgeniy-scherbina deleted the yevhenii/aigov-578-automate-updates-to-the-shipped-ai-model-price-book branch August 19, 2026 15:40
@github-actions github-actions Bot locked and limited conversation to collaborators Aug 19, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants