Metrics

ModelWorks tracks metrics at two levels: per version (aggregate across all runs) and per run (individual execution). These metrics help you monitor performance, diagnose issues, and understand costs.

Per-Version Metrics

Visible on the version details page, these aggregate metrics show how a circuit version is performing over time:

Metric	Description
Runs	Total number of times this version was executed.
Embeds	How many times this version was called by other circuits via the embed primitive.
Nodes	Number of logical steps (tasks) in the circuit.
Avg Response Time (ms)	Average execution duration in milliseconds.
Avg Tokens per Second	Average throughput across all fetch steps.
Avg Spend per Run	Average credits charged per run.
Input Cache Hit Rate	Percentage of input tokens served from cache vs total input tokens. Weighted across all runs by input volume. Higher is better — cached tokens are billed at a discount.
Error Rate	Percentage of runs that ended in an exception.

These metrics are also tracked over time in the trend charts (runs per day, average response time, error rate, tokens per second, tokens per run, and input cache hit rate).

If the error rate stays high over a 24-hour window, the version may be automatically moved to the EXCEPTION state. Versions in this state are still queryable but display a warning so consumers know something may be wrong.

Per-Run Metrics

Each completed run produces a result_blob containing billing and performance data. You can fetch it via GET /circuit-runs/{uid} (see Circuit Runs).

Billing Entries

The billing_entries array inside the result blob contains one entry per pricing type in your circuit. Each entry breaks down the cost:

Field	Description
`type`	Pricing type: `FLAT`, `TOKEN`, or `TOKEN_TIERED`.
`cost`	Total credits charged by this entry.
`total_input_tokens`	Input tokens consumed (TOKEN and TOKEN_TIERED only).
`total_output_tokens`	Output tokens generated (TOKEN and TOKEN_TIERED only).
`cached_input_tokens`	Input tokens served from cache at a discounted rate.
`cache_write_tokens`	Tokens written to cache.
`cost_per_input_token`	Rate applied to non-cached input tokens.
`cost_per_output_token`	Rate applied to output tokens.
`cost_per_input_cache_token`	Discounted rate applied to cached input tokens.
`cost_per_cache_write_token`	Rate applied to cache write tokens.
`cache_write_ttl`	Which cache write TTL tier was billed: `"5m"`, `"30m"`, or `"1h"`.
`tier_index`	Which pricing tier was matched (TOKEN_TIERED only, 0-indexed).
`note`	Human-readable summary, e.g. `"135 in, 1884 cached, 0 write, 10 out"`.

Performance Reports

The performance_reports object contains per-step execution statistics:

Field	Description
`time_taken`	Wall-clock execution time in milliseconds.
`tokens_per_sec`	Throughput — tokens generated per second.
`total_tokens`	Total tokens processed by this step.

Each key in performance_reports corresponds to a step name from your circuit definition. For example, a circuit with a step named chat_completion_response would have its performance data under performance_reports.chat_completion_response.

Example

JSON

{
  "billing_entries": [
    {
      "type": "TOKEN",
      "cost": 0.000119,
      "total_input_tokens": 19,
      "total_output_tokens": 21,
      "cached_input_tokens": 0,
      "cache_write_tokens": 0,
      "cost_per_input_token": 0.0000014,
      "cost_per_output_token": 0.0000044,
      "note": "19 in, 0 cached, 0 write, 21 out"
    }
  ],
  "performance_reports": {
    "chat_completion_response": {
      "time_taken": 1966,
      "tokens_per_sec": 346.31,
      "total_tokens": 841
    }
  }
}

Summary Metrics on the Run

The top-level run object also includes summary fields derived from the billing entries and performance reports:

Field	Description
`state`	Run lifecycle state: `PROVISIONED`, `RUNNING`, `COMPLETED`, or `EXCEPTION`.
`created_at`	When the run started.
`ended_at`	When the run finished.
`is_public`	Whether the run is publicly viewable.

The total cost of a run is the sum of all billing_entries[].cost values. This is the amount deducted from the caller's wallet.

Using Metrics

Monitor costs: Check billing_entries to see exactly how each run was billed, including the cache read/write breakdown.
Diagnose performance: Use performance_reports to identify which steps are slow or have low throughput.
Track cache efficiency: Watch the Input Cache Hit Rate — a high rate means the model provider is serving input from cache, reducing cost and latency. If it's consistently 0% on a model that supports caching, check your total_cached_input_tokens_selector configuration.
Track adoption: Watch the per-version Runs and Embeds counts to see how your circuit is being used.
Catch errors early: Monitor the Error Rate metric — a sudden spike may indicate a problem with your circuit or an upstream API.

Metrics

On this page