# Traces Bento records every pipeline run across three layers: the **triggers** table as the entry point, the **trace** of the run with its envelope and per-phase tasks, and the **logs** of the agent as raw NDJSON or parsed turns. ## 1. Find a trigger Every webhook, cron tick, and MCP call enters the system as a trigger. List them directly from the database: ```bash PGPASSWORD=bento psql -h localhost -p 8421 -U bento -d bento \ -c "select id, kind, source, event_type, pipeline, lane, status, run_id, arrived_at from triggers order by arrived_at desc limit 10;" ``` ``` id | kind | source | event_type | pipeline | lane | status | run_id | arrived_at --------------+---------+---------+---------------------+-----------+------+--------+------------+-------------------- d990d0a0-5f1 | webhook | github | pull_request.opened | pr-review | host | done | XbSjXLrysC | 2026-05-26 11:35:21 e10552e5-7d1 | schedule| - | - | heartbeat | host | done | mecOey_7rx | 2026-05-26 11:40:00 ``` Filter by `kind`, `pipeline`, `status`, or `source` to narrow the list. The shortened identifier (first 12 characters) is enough for the trace command. `bento trace` accepts a prefix of an id. Give the prefix 6 or more characters after the kind (`trg_`, `run_`, `inv_`, or `tsk_`), and make sure it matches one trigger. If it matches more than one trigger, the command shows each candidate and stops. ## 2. Trace a trigger `bento trace ` follows one trigger through every layer it touched: ```bash bento trace d990d0a0-5f1 ``` ``` ══════════════════════════════════════════════════════════════════════ TRACE: d990d0a0-5f1 ══════════════════════════════════════════════════════════════════════ ┌─ Trigger │ id: d990d0a0-5f1 │ kind: webhook │ source: github/pull_request.opened │ pipeline: pr-review │ lane: host │ status: done │ arrived: 2026-05-26 11:35:21+00 │ completed: 2026-05-26 11:42:38+00 │ run_id: XbSjXLrysC └ ┌─ Workload │ id: 043cbc31-31ac-4726-862c-07511d286547 │ status: completed │ duration: 437.2s └ ┌─ Invocations (1) └── [a020a45e] reviewer runtime=claude model=claude-sonnet-4-6 status=completed duration=436.7s ✓ workspace-setup (setup) 1457ms [ok] ✓ setup[0] (setup) 812ms [ok] └─ {"command":"./scripts/index-symbols.sh","exit_code":0,"timeout_ms":360000} └─ stdout: ~/.bento/workspaces/9f2/runs/XbSjXLrysC/steps/setup_0_/stdout stderr: ~/.bento/workspaces/9f2/runs/XbSjXLrysC/steps/setup_0_/stderr ✓ task:reviewer (spawn) 220750ms [ok] ✓ task:reviewer (spawn) 232402ms [ok] ✓ spawn (spawn) 435163ms [ok] ┌─ Orchestrator Output │ └ ``` The trace includes these sections: * **Trigger** — what came in (HTTP webhook, cron, MCP), when, which pipeline matched, terminal status. A trigger replayed after a shutdown abort or by `bento trigger replay` adds a `replay_of` line naming the trigger it replayed. * **Workload** — the envelope that groups one or more invocations from a single trigger. * **Invocations** — one per agent run. Each one lists its phases (`workspace-setup`, `spawn`, hooks) with its duration and its outcome. An orchestrated run shows `task:` sub-spawns for each tool call the orchestrator made. A run that declares [`outputs:`](/pipelines/config#result-and-outputs) adds one `publish-` row per projection, for what the run wrote outside the sandbox. * **Orchestrator Output** — what the agent finally returned, after notes, citations, and post-back processing. * **Review Coverage** — for a run that returned a review verdict, each file the review declared it accounted for, beside whether the transcript shows a read of that file. A file declared `read` with no read call in the transcript is flagged. The reads come from the attempt that produced the result, so a failed attempt's reads never satisfy a declaration the winning candidate made. A runtime that never reports its reads renders as unknown rather than as a mismatch. ### Useful flags | Flag | When | |------|------| | `--json` | Pipe to `jq` for scripted inspection | | `--agent-output none\|final\|turns\|raw` | Agent output to render per invocation: nothing, final result text (default), parsed turns, or raw NDJSON. | | `--no-messages` | Deprecated alias for `--agent-output none`. | ## 3. Read the agent's log To see the full turn-by-turn stream — tool calls, intermediate reasoning — read the agent log: ```bash # NDJSON — one event per line bento daemon logs XbSjXLrysC --agent # Same data, collapsed to readable per-turn text bento daemon logs XbSjXLrysC --agent --render flat ``` Use `--render flat` for human-readable output. Use `raw` to pipe into a parser. ## 4. Watch a daemon in flight To tail the daemon log in real time: ```bash bento daemon logs # last 50 lines bento daemon logs -f # follow mode (tail -f) bento daemon logs -n 200 # widen the window ``` Set `name:` in `.bento/daemon.yaml` (default: `bento`) to control all service-level identifiers: | Thing | Path / label | |-------|-------------| | Log file | `~/.bento//daemon.log` | | PID file | `~/.bento//daemon.pid` | | launchd label | `dev.bento.` | | systemd unit | `.service` | | Process title | `-daemon` | On Linux (systemd), with `name: myproject`: ```bash journalctl --user -u myproject.service -f ``` Run-id-stamped lines correlate with `bento trace`: ``` [i] workload ▶ invocation running run_xbsjxq [i] workload ◐ invocation workspace ready run_xbsjxq 1.5s [i] workload ▶ pipeline [pr-review] executing reviewer/agentic-review run_xbsjxq └─ runtime=codex/gpt-5.6-terra/high timeout=15m sandbox=docker ``` The short id in the tag column is a real prefix of the full run id, so it pastes straight into `bento trace run_xbsjxq`. On an agent line, the name after `executing` is the agent, and the segment after the slash is the skill it ran under. ## 5. Troubleshoot **"Why did the agent fail?"** — `bento trace ` first. If the failure is in a hook, the failing step shows `outcome=errored` with the captured stderr. If the failure is in the agent itself, drop to `bento daemon logs --agent --render flat`. When runtime inspection succeeds on Docker or Podman, the failed attempt shows the sandbox exit code, out-of-memory kill state, and finish time. An absent value means unread. When a checkout exists, `bento trace` also shows `worktree kept: `. Inspect that directory to see the files at failure time. The hourly workspace sweep removes the directory after 24 hours. **"What did the agent actually see?"** — the run directory on disk: `~/.bento/workspaces//runs//`. It holds the assembled prompt (`meta.json`), the stdout and stderr files of the agent, and the per-step artifacts under `steps/`. **"Why did my webhook not match a pipeline?"** — list discarded triggers: ```bash PGPASSWORD=bento psql -h localhost -p 8421 -U bento -d bento \ -c "select id, source, event_type, note from triggers where status = 'discarded' order by arrived_at desc limit 10;" ``` The `note` column carries the verdict of the matcher, for example `no pipeline matched` or `filter rejected: `. **"Which candidate published, and what did it write?"** — a run with `fallback:` candidates shows one `spawn-attempt-` row per candidate, with the runtime, the model, the credential, and, when the route advanced, the failure category that advanced it. The `publish-` rows that follow show what the run then wrote. Each one names the candidate it published on and its attempt number, plus: * `effect_key` — the claim key, which is the logical identity of the publication. The daemon reserves it before the first request, so two candidates cannot publish the same thing twice. * `claim` — `claimed` when this run took the key, `resumed` when the key already stood with identical content and only unsettled deliveries were retried, `conflict` when another actor already owns the key and this run published nothing. * `deliveries` — the state each individual write settled in: `published`, `failed` (definitely not sent), `indeterminate` (may have been sent), or `blocked` (a delivery it depends on did not publish). A row with `claim=conflict` and no deliveries is the mechanism working: the candidate was beaten to the publication and wrote nothing. An `indeterminate` delivery is the one to act on — the daemon does not know whether that request reached the provider, so it never retries it. **"What did setup actually do?"** — the workspace-setup task is the first phase in every invocation. Its duration tells you whether a clone ran, in seconds, or the step was a no-op `mkdir`, in milliseconds. The pipeline's own [`setup:` steps](/pipelines/config#setup) follow it as `setup[0]`, `setup[1]`, … in run order, each one with the command it ran, its exit code, and the paths that captured its stdout and stderr. A step that soft-failed shows `outcome=errored` with its exit code; `cat` the stderr path to see why.