Skip to main content
Reference

Benchmarks

Reproduce cold-start and warm-execution measurements through the public Agent OS Exec API.

The checked benchmark at packages/runtime-benchmarks/coldstart.bench.ts uses JavaScriptRuntime.create() and execute() from the public package. It reports four timings:

  • runtime creation,
  • the first execution in that VM,
  • a second execution in the same VM,
  • runtime creation plus first execution as end-to-end cold start.

It compares an explicit fresh sidecar per VM with a named process-wide shared sidecar pool. Both placements still create an isolated kernel-owned VM for each runtime; placement changes process reuse and failure scope, not VM state isolation.

No historical result files are presented as Agent OS Exec numbers. Rebaseline with the command below before making latency claims about the current public surface.

Run the benchmark

Build the release sidecar and the JavaScript package first:

cargo build --release -p agentos-sidecar
pnpm --filter @rivet-dev/agentos build

Then run the public-API benchmark:

AGENTOS_SIDECAR_BIN="$PWD/target/release/agentos-sidecar" \
  pnpm --dir packages/runtime-benchmarks bench:coldstart

Use one measured iteration for a smoke run:

BENCH_ITERATIONS=1 BENCH_WARMUP=0 \
AGENTOS_SIDECAR_BIN="$PWD/target/release/agentos-sidecar" \
  pnpm --dir packages/runtime-benchmarks bench:coldstart

BENCH_SCENARIOS accepts fresh-sidecar, shared-sidecar, or both as a comma-separated list. Results are JSON on stdout and progress is written to stderr so they can be captured independently.

Interpret results

Cold start includes VM creation and the first language process. Warm execution still starts a fresh guest process; reusing a JavaScriptRuntime reuses the VM, filesystem, mounts, and sidecar placement, not JavaScript globals or module state. Use spawn() only when the application intentionally needs one live, stateful process such as a server or REPL.

Edit this page