# Traces Every pipeline run is recorded across three layers: the **triggers** table (entry point), the run's **trace** (envelope and per-phase tasks), and the agent's **logs** (raw NDJSON / parsed turns). ## 1. Find a trigger Every webhook, cron tick, and MCP call enters the system as a trigger. List them directly from the DB: ```bash PGPASSWORD=bento psql -h localhost -p 8421 -U bento -d bento \ -c "select id, kind, source, event_type, pipeline, status, run_id, arrived_at from triggers order by arrived_at desc limit 10;" ``` ``` id | kind | source | event_type | pipeline | status | run_id | arrived_at --------------+---------+---------+---------------------+-----------+--------+------------+-------------------- d990d0a0-5f1 | webhook | github | pull_request.opened | pr-review | done | XbSjXLrysC | 2026-05-26 11:35:21 e10552e5-7d1 | schedule| - | - | heartbeat | done | mecOey_7rx | 2026-05-26 11:40:00 ``` Filter by `kind`, `pipeline`, `status`, or `source` to narrow down. The shortened id (first 12 chars) is enough for the trace command — `bento trace` accepts any unique prefix. ## 2. Trace a trigger `bento trace ` follows one trigger through every layer it touched: ```bash bento trace d990d0a0-5f1 ``` ``` ══════════════════════════════════════════════════════════════════════ TRACE: d990d0a0-5f1 ══════════════════════════════════════════════════════════════════════ ┌─ Trigger │ id: d990d0a0-5f1 │ kind: webhook │ source: github/pull_request.opened │ pipeline: pr-review │ status: done │ arrived: Tue May 26 2026 12:35:21 │ completed: Tue May 26 2026 12:42:38 │ run_id: XbSjXLrysC └ ┌─ Workload │ id: 043cbc31-31ac-4726-862c-07511d286547 │ status: completed │ duration: 437.2s │ ttl: 3600s └ ┌─ Invocations (1) └── [a020a45e] reviewer runtime=claude model=claude-sonnet-4-6 status=completed duration=436.7s ✓ workspace-setup (setup) 1457ms [ok] ✓ task:reviewer (spawn) 220750ms [ok] ✓ task:reviewer (spawn) 232402ms [ok] ✓ spawn (spawn) 435163ms [ok] ┌─ Orchestrator Output │ └ ``` The four sections, top to bottom: * **Trigger** — what came in (HTTP webhook, cron, MCP), when, which pipeline matched, terminal status. * **Workload** — the envelope that groups one or more invocations from a single trigger. * **Invocations** — one per agent run. Each lists its phases (`workspace-setup`, `spawn`, hooks) with duration + outcome. An orchestrated run shows `task:` sub-spawns for each tool call the orchestrator made. * **Orchestrator Output** — what the agent finally returned, after notes / citations / post-back processing. ### Useful flags | Flag | When | |------|------| | `--json` | Pipe to `jq` for scripted inspection | | `--agent-output none\|final\|turns\|raw` | Agent output to render per invocation: nothing, final result text (default), parsed turns, or raw NDJSON. | | `--no-messages` | Deprecated alias for `--agent-output none`. | ## 3. Read the agent's log To see the full turn-by-turn stream — tool calls, intermediate reasoning — read the agent log: ```bash # NDJSON — one event per line bento daemon logs XbSjXLrysC --agent # Same data, collapsed to readable per-turn text bento daemon logs XbSjXLrysC --agent --render flat ``` Use `--render flat` for human-readable output; `raw` for piping into a parser. ## 4. Watch a daemon in flight To tail the daemon log in real time: ```bash bento daemon logs # last 50 lines bento daemon logs -f # follow mode (tail -f) bento daemon logs -n 200 # widen the window ``` Set `name:` in `.bento/daemon.yaml` (default: `bento`) to control all service-level identifiers: | Thing | Path / label | |-------|-------------| | Log file | `~/.bento//daemon.log` | | PID file | `~/.bento//daemon.pid` | | launchd label | `dev.bento.` | | systemd unit | `.service` | | Process title | `-daemon` | On Linux (systemd), with `name: myproject`: ```bash journalctl --user -u myproject.service -f ``` Run-id-stamped lines correlate with `bento trace`: ``` [i] workload ▶ invocation running run_xbsjxq [i] workload ◐ invocation workspace ready run_xbsjxq 1.5s [i] workload ▶ pipeline [pr-review] executing reviewer/agentic-review run_xbsjxq └─ runtime=codex/gpt-5.6-terra/high timeout=15m sandbox=docker ``` The short id in the tag column is a real prefix of the full run id, so it pastes straight into `bento trace run_xbsjxq`. On an agent line, the name after `executing` is the agent, and the segment after the slash is the skill it ran under. ## 5. Troubleshoot **"Why did the agent fail?"** — `bento trace ` first. If the failure is in a hook, the failing step shows `outcome=errored` with the captured stderr. If the failure is in the agent itself, drop to `bento daemon logs --agent --render flat`. **"What did the agent actually see?"** — the run dir on disk: `~/.bento/workspaces//runs//`. It holds the assembled prompt (`meta.json`), the agent's stdout/stderr files, and per-step artifacts under `steps/`. **"Why didn't my webhook match a pipeline?"** — list discarded triggers: ```bash PGPASSWORD=bento psql -h localhost -p 8421 -U bento -d bento \ -c "select id, source, event_type, note from triggers where status = 'discarded' order by arrived_at desc limit 10;" ``` The `note` column carries the matcher's verdict (e.g. `no pipeline matched`, `filter rejected: `). **"What did setup actually do?"** — the workspace-setup task is the first phase in every invocation; its duration tells you whether a clone happened (seconds) or it was a no-op mkdir (milliseconds).