feat(ai): LLM cost tracking and AI span inspector by ericallam · Pull Request #3213 · triggerdotdev/trigger.dev

ericallam · 2026-03-13T09:11:15Z

Automatic LLM cost enrichment for AI SDK spans (streamText, generateText, generateObject) with support for 145+ models from Langfuse defaults
New AI span inspector sidebar showing model, tokens, cost, messages, tool calls, and response text
LLM usage dual-write to ClickHouse llm_usage_v1 table for analytics
Provider cost fallback — uses gateway/OpenRouter reported costs from providerMetadata when registry pricing is unavailable
Prefix-stripping for gateway/OpenRouter model names (e.g. mistral/mistral-large-3 matches mistral-large-3 pricing)
Admin dashboard for managing LLM model pricing (list, create, edit, delete, search, test pattern matching)
Missing models detection page — queries ClickHouse for unpriced models with sample spans and Claude Code-ready prompts for adding pricing
AI span seed script (pnpm run db:seed:ai-spans) with 51 spans across 12 provider systems for local dev testing
UI fixes: completionTokens/promptTokens aliases, ai.response.object display for generateObject, cache read/write token breakdown

changeset-bot · 2026-03-13T09:11:21Z

🦋 Changeset detected

Latest commit: 463f972

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 29 packages

Name	Type
@trigger.dev/core	Patch
@trigger.dev/build	Patch
trigger.dev	Patch
@trigger.dev/python	Patch
@trigger.dev/redis-worker	Patch
@trigger.dev/schema-to-json	Patch
@trigger.dev/sdk	Patch
@internal/cache	Patch
@internal/clickhouse	Patch
@internal/llm-pricing	Patch
@internal/redis	Patch
@internal/replication	Patch
@internal/run-engine	Patch
@internal/schedule-engine	Patch
@internal/testcontainers	Patch
@internal/tracing	Patch
@internal/tsql	Patch
@internal/zod-worker	Patch
d3-chat	Patch
references-d3-openai-agents	Patch
references-nextjs-realtime	Patch
references-realtime-hooks-test	Patch
references-realtime-streams	Patch
references-telemetry	Patch
@internal/sdk-compat-tests	Patch
@trigger.dev/react-hooks	Patch
@trigger.dev/rsc	Patch
@trigger.dev/database	Patch
@trigger.dev/otlp-importer	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

coderabbitai · 2026-03-13T09:11:42Z

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

@coderabbitai resume to resume automatic reviews.
@coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

▶️ Resume reviews
🔍 Trigger review

Walkthrough

Adds end-to-end LLM cost tracking and UI. Introduces an internal llm-pricing package (types, registry, default prices, seeding, and tests), Prisma schema and migration for LLM pricing tables, and a ClickHouse llm_usage_v1 table plus insert helpers. Adds a llmPricingRegistry singleton, pricing enrichment that writes trigger.llm.* attributes and a side-channel _llmUsage, OTLP exporter changes (array handling and runTags), dual-write to ClickHouse, admin APIs/UIs for model management, and multiple React components/utilities to parse and display AI span data.

Estimated code review effort

🎯 5 (Critical) | ⏱️ ~120 minutes

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 26.56% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'feat(ai): LLM cost tracking and AI span inspector' directly aligns with the main objectives of the changeset, clearly summarizing the primary features being added (LLM cost tracking and an AI span inspector interface).
Description check	✅ Passed	The PR description addresses all required sections from the template except for the explicit Testing and Screenshots sections, which may be optional. The Changelog section is well-populated with a comprehensive list of features and changes. The description clearly documents the major additions and design decisions.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch feature/tri-7773-llm-cost-tracking

📝 Coding Plan

Generate coding plan for human review comments

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

coderabbitai

🧹 Nitpick comments (1)

apps/webapp/test/otlpExporter.test.ts (1)
436-438: Avoid undefined as any for registry cleanup.

The type assertion bypasses type safety. Consider exposing a dedicated reset/unload function from the module (e.g., resetLlmPricingRegistry()) or accepting undefined in the function signature if it's a valid state.
♻️ Suggested approach

Option 1 - Accept undefined in the function signature:
// In enrichCreatableEvents.server.ts
export function setLlmPricingRegistry(registry: LlmPricingRegistry | undefined): void
Option 2 - Add a dedicated reset function:
// In enrichCreatableEvents.server.ts  
export function resetLlmPricingRegistry(): void

// In test
afterEach(() => {
  resetLlmPricingRegistry();
});
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@apps/webapp/test/otlpExporter.test.ts` around lines 436 - 438, The test
currently calls setLlmPricingRegistry(undefined as any) which bypasses type
safety; update the module (enrichCreatableEvents.server.ts) to either allow
undefined in the setter signature (export function
setLlmPricingRegistry(registry: LlmPricingRegistry | undefined): void) or add a
dedicated reset function (export function resetLlmPricingRegistry(): void), then
change the test afterEach to call the new resetLlmPricingRegistry() or call
setLlmPricingRegistry(undefined) with the adjusted type so the cleanup is
type-safe and no longer uses undefined as any.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@apps/webapp/test/otlpExporter.test.ts`:
- Around line 436-438: The test currently calls setLlmPricingRegistry(undefined
as any) which bypasses type safety; update the module
(enrichCreatableEvents.server.ts) to either allow undefined in the setter
signature (export function setLlmPricingRegistry(registry: LlmPricingRegistry |
undefined): void) or add a dedicated reset function (export function
resetLlmPricingRegistry(): void), then change the test afterEach to call the new
resetLlmPricingRegistry() or call setLlmPricingRegistry(undefined) with the
adjusted type so the cleanup is type-safe and no longer uses undefined as any.

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: d7a96f6e-7769-4037-b7bf-7ff0bcdfef7c

📥 Commits

Reviewing files that changed from the base of the PR and between 4c0b521 and cbb946a.

📒 Files selected for processing (2)

apps/webapp/test/otlpExporter.test.ts
internal-packages/llm-pricing/package.json

🚧 Files skipped from review as they are similar to previous changes (1)

internal-packages/llm-pricing/package.json

📜 Review details

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (27)

GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (8, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (2, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (6, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (8, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (1, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (7, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (2, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (4, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (5, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (6, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (3, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (4, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (3, 8)
GitHub Check: units / internal / 🧪 Unit Tests: Internal (5, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (1, 8)
GitHub Check: units / webapp / 🧪 Unit Tests: Webapp (7, 8)
GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - pnpm)
GitHub Check: sdk-compat / Bun Runtime
GitHub Check: units / packages / 🧪 Unit Tests: Packages (1, 1)
GitHub Check: sdk-compat / Node.js 20.20 (ubuntu-latest)
GitHub Check: e2e / 🧪 CLI v3 tests (ubuntu-latest - pnpm)
GitHub Check: e2e / 🧪 CLI v3 tests (windows-latest - npm)
GitHub Check: sdk-compat / Cloudflare Workers
GitHub Check: sdk-compat / Node.js 22.12 (ubuntu-latest)
GitHub Check: sdk-compat / Deno Runtime
GitHub Check: typecheck / typecheck
GitHub Check: Analyze (javascript-typescript)

🧰 Additional context used

📓 Path-based instructions (14)

**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

**/*.{ts,tsx}: Use types over interfaces for TypeScript
Avoid using enums; prefer string unions or const objects instead

**/*.{ts,tsx}: Use task export syntax: export const myTask = task({ id: 'my-task', run: async (payload) => { ... } })
Use Run Engine 2.0 (@internal/run-engine) and redis-worker for all new work - avoid DEPRECATED zodworker (Graphile-worker wrapper)
Prisma 6.14.0 client and schema use PostgreSQL in internal-packages/database - import only from Prisma client

Files:

apps/webapp/test/otlpExporter.test.ts

{packages/core,apps/webapp}/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use zod for validation in packages/core and apps/webapp

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use function declarations instead of default exports

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.{test,spec}.{ts,tsx}

📄 CodeRabbit inference engine (.github/copilot-instructions.md)

Use vitest for all tests in the Trigger.dev repository

Files:

apps/webapp/test/otlpExporter.test.ts

apps/webapp/**/*.test.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

Test files should only import classes and functions from app/**/*.ts files and should not import env.server.ts directly or indirectly; pass configuration through options instead

In test files, do not import env.server.ts directly; pass configuration as constructor arguments or options instead for testable code

Files:

apps/webapp/test/otlpExporter.test.ts

apps/webapp/**/*.{ts,tsx}

📄 CodeRabbit inference engine (.cursor/rules/webapp.mdc)

apps/webapp/**/*.{ts,tsx}: When importing from @trigger.dev/core in the webapp, use subpath exports from the package.json instead of importing from the root path
Follow the Remix 2.1.0 and Express server conventions when updating the main trigger.dev webapp

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.ts

📄 CodeRabbit inference engine (.cursor/rules/otel-metrics.mdc)

**/*.ts: When creating or editing OTEL metrics (counters, histograms, gauges), ensure metric attributes have low cardinality by using only enums, booleans, bounded error codes, or bounded shard IDs
Do not use high-cardinality attributes in OTEL metrics such as UUIDs/IDs (envId, userId, runId, projectId, organizationId), unbounded integers (itemCount, batchSize, retryCount), timestamps (createdAt, startTime), or free-form strings (errorMessage, taskName, queueName)
When exporting OTEL metrics via OTLP to Prometheus, be aware that the exporter automatically adds unit suffixes to metric names (e.g., 'my_duration_ms' becomes 'my_duration_ms_milliseconds', 'my_counter' becomes 'my_counter_total'). Account for these transformations when writing Grafana dashboards or Prometheus queries

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.{js,ts,jsx,tsx,json,md,yaml,yml}

📄 CodeRabbit inference engine (AGENTS.md)

Format code using Prettier before committing

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.test.{ts,tsx,js,jsx}

📄 CodeRabbit inference engine (AGENTS.md)

**/*.test.{ts,tsx,js,jsx}: Test files should live beside the files under test and use descriptive describe and it blocks
Tests should avoid mocks or stubs and use the helpers from @internal/testcontainers when Redis or Postgres are needed
Use vitest for running unit tests

Files:

apps/webapp/test/otlpExporter.test.ts

**/*.test.{ts,tsx,js}