Repeatable repository audits and reviews for opencode

oy gives opencode users a bounded path from deterministic repository inputs to security audits, code-quality reviews, and focused remediation.

Why oy

Visible coverage

Gitignore-aware manifests and ordered chunks replace silent, model-selected sampling. Oversized runs fail closed.

Restricted review

Audit and review agents receive oy evidence tools, not generic shell, edit, search, or web tools.

Reports that survive chat

Markdown and SARIF reports carry stable finding IDs and statuses into reruns and one-finding remediation.

Quick start

curl -fsSL https://oy.adonm.dev/install.sh | sh
# Restart or activate your shell as instructed, then:
oy doctor
oy audit

The full installer uses mise to install oy, opencode, and optional local evidence helpers, then runs global setup. Review the installer before piping it to a shell.

For a minimal manual install:

mise use --global cargo-binstall cargo:oy-cli opencode
oy setup
oy audit

Continue with Getting started or go directly to the workflow guide.

One focused loop

1. Audit: oy audit writes ISSUES.md or SARIF.
2. Review: oy review main writes REVIEW.md for a target diff.
3. Remediate: oy enhance <finding-id> fixes and verifies one finding.
4. Confirm: rerun the originating audit or review to update its status.

A precise claim

Deterministic inputs, not deterministic conclusions.

oy owns collection, ordering, limits, and report rendering. opencode owns model execution, and model findings vary by model and prompt. “Every chunk” means every collected chunk; documented exclusions still apply.

Where to go next

• Install and configure oy
• Understand audit, review, and remediation
• See representative reports and CI integration
• Look up commands, MCP tools, and environment variables
• Check supported and tested environments
• Read the project direction
• Browse the Rust API

Getting started

Set up oy as a focused audit/review extension for an existing opencode installation.

Requirements

• opencode and a configured model provider
• git for target-diff reviews
• Rust 1.96+ only when building from source

oy does not store provider credentials or select a provider. Follow opencode’s provider setup, then use opencode once to verify the model works.

Install

Full mise installer

curl -fsSL https://oy.adonm.dev/install.sh | sh

The POSIX shell installer installs or updates mise, oy, opencode, tokei, and Universal Ctags, then runs oy setup. Restart your shell or use the activation command it prints.

Review install.sh before piping it to a shell. Set OY_SKIP_SETUP=1 to skip integration writes or OY_MISE_MINIMUM_RELEASE_AGE to change mise’s release-age filter. Optional Sighthound has no release binary; set OY_INSTALL_SIGHTHOUND=1 only when Rust 1.85+ is already installed and you want mise to build it from source.

Minimal manual install

mise use --global cargo-binstall cargo:oy-cli opencode
oy setup
oy doctor

You can also install from crates.io:

cargo install oy-cli --locked

Optional evidence helpers can be added later from the reference.

Choose setup scope

oy setup --dry-run        # preview global changes
oy setup                  # ~/.config/opencode/
oy setup --workspace      # .opencode/ in this repository

Global setup is convenient for personal use. Workspace setup is useful when one repository needs local overrides. Restart running opencode sessions after setup changes.

Config rewrite: oy replaces its documented config entries and pretty-serializes opencode.json. Other object keys remain, but JSONC comments and formatting do not. Back up a hand-edited config and preview first.

Create a first report

cd your-repository
oy doctor
oy audit

opencode runs the restricted auditor and writes ISSUES.md. Start with a small or medium repository so you can inspect the protocol and report before increasing scope.

For a code-quality review:

oy review             # collected workspace
oy review main        # git diff main

Continue with the workflow guide to understand focus text, path scope, SARIF, finding IDs, and reruns.

Compatibility

Prebuilt releases cover Linux x86_64/aarch64 with glibc and Apple Silicon macOS. Other Rust targets may build from source but are not release-tested. The curl installer assumes a Unix-like shell.

See the compatibility matrix for the distinction between CI-tested, release-built, and best-effort environments.

Next steps

• Run audit, review, and remediation loops
• Understand automatic refresh and owned config
• Inspect the deterministic MCP boundary
• Review report and CI examples
• Read the security guidance

Workflow guide

Use the same bounded evidence protocol for whole-repository audits, target-diff reviews, and report-driven remediation.

The workflow contract

1. oy inventories the requested scope or prepares git diff <target>.
2. It creates stable, ordered chunks and checks the maximum chunk budget.
3. A restricted opencode subagent reads each collected chunk in order.
4. The model produces candidates; the oy renderer normalizes the final report and findings block.
5. A later run reads the prior generated report once to carry forward only current findings.

Determinism stops at inference. Collection and rendering are deterministic; finding quality and prose depend on the selected model.

Security audit

oy audit
oy audit "authentication and authorization"
oy audit src/auth
oy audit --out reports/security.md --max-chunks 60
oy audit --format sarif --out oy.sarif

Free-form text is a lens. A focus that exactly names a workspace-relative file or directory becomes the collection scope. Markdown defaults to ISSUES.md; SARIF defaults to oy.sarif.

The auditor has access to repository collection, the previous audit report, optional Sighthound candidates, and the report renderer. It has no generic read, search, edit, shell, or web tools. Sighthound runs only when the focus explicitly requests it, for example oy audit "include Sighthound SAST"; its output is candidate evidence and still requires source confirmation.

Code-quality review

oy review
oy review main
oy review HEAD~3 --focus "error handling"
oy review main --out reports/change-review.md

Omit the target to review the collected workspace. Supply a branch, commit, tag, or other ref to review the current workspace against deterministic git diff <target> input. The default output is REVIEW.md.

The reviewer emphasizes high-conviction structural findings: unnecessary complexity, unclear state or ownership, weak boundaries/types, dependency cost, and files that need meaningful decomposition. It should report no major concern rather than fill space.

Reports and finding lifecycle

Rendered Markdown includes a human summary plus a machine-readable oy-findings JSON block. Each normalized finding has:

• a stable ID derived from source, severity, title, and primary location when the model did not provide one;
• a normalized status such as new, carried-forward, fixed?, or stale;
• path, line, and symbol evidence when available;
• short evidence and remediation context.

Each new audit/review supersedes the old generated report. Current findings may be carried forward; stale findings should be dropped. Keep reports under version control only if that matches your team’s disclosure policy.

Remediate one finding

oy enhance audit-0123456789abcdef
oy enhance "the highest-severity actionable finding"
oy enhance --review-target main review-0123456789abcdef

The focus is positional. The enhancer reads ISSUES.md/REVIEW.md, chooses one actionable finding, edits through opencode, runs focused verification when available, and summarizes the result. Edit and shell behavior follows opencode permissions.

Then rerun the originating workflow:

oy audit                 # confirm an audit finding
oy review main           # confirm a target-review finding

Coverage and failure limits

The default maximum is 80 chunks. If a successful input summary exceeds that limit, the restricted agent must fail closed rather than sample. You can increase --max-chunks, but path scope is usually cheaper and easier to evaluate.

The repository collector omits:

• gitignored and hidden paths;
• .git, target, node_modules, .venv, and .tmp content;
• common lockfiles, generated reports, likely secrets, and private-key formats;
• binary, non-UTF-8, empty, unreadable, and larger-than-512-KiB files.

These exclusions reduce accidental disclosure and context waste, but they also limit completeness. In particular, current lockfile exclusion means an oy audit is not a complete supply-chain audit.

Sighthound does not consume oy’s collected file list. It uses independent gitignore-aware discovery, common directory filters, and its own file-size limit (currently 10 MiB), so an explicitly requested scan may inspect supported hidden source or source files omitted by oy’s 512 KiB limit. Treat returned snippets as additional model-provider disclosure.

Practical guidance

• Use a model with strong tool use and enough context for a 64,000-token chunk plus prompt/report overhead.
• Start with default limits; narrow by path before raising limits on very large repositories.
• Do not put secrets under the workspace root solely because likely-secret names are skipped.
• Treat SAST and model findings as candidates until a human verifies the evidence and impact.
• Capture opencode version, model, oy version, target, and scope when comparing reports.

See Examples and CI for representative output and SARIF upload configuration.

Examples and CI integration

These shortened examples show report shape and handoff semantics. Paths, IDs, model names, and findings are illustrative; actual model output is nondeterministic.

Audit report with one finding

Command:

oy audit "authentication boundaries"

Representative ISSUES.md excerpt:

# Audit Issues

> Generated by oy audit; model: provider/model; focus: authentication boundaries; max chunks: 80.

## Findings summary

- `audit-2a71...` **High** `src/auth.rs:84` — Session lookup accepts an unscoped tenant ID _(status: new; fix: `oy enhance audit-2a71...`)_

## Detailed findings

### [High] Session lookup accepts an unscoped tenant ID

Evidence: `src/auth.rs:84` uses the caller-provided tenant before authorization.

```json oy-findings
[
  {
    "id": "audit-2a71...",
    "status": "new",
    "source": "audit",
    "severity": "High",
    "title": "Session lookup accepts an unscoped tenant ID",
    "locations": [{"path": "src/auth.rs", "line": 84}],
    "evidence": "Caller-provided tenant reaches session lookup before authorization.",
    "body": "Bind tenant scope to the authenticated principal before lookup.",
    "category": "access-control"
  }
]
```

Target-diff review

Command:

oy review main --focus "types and boundaries"

Representative REVIEW.md excerpt:

# Code Quality Review

## Verdict

Needs work.

## Findings summary

- `review-7bd1...` **Medium** `src/cli/config.rs:41` — Two structs represent the same persisted state _(status: new; fix: `oy enhance review-7bd1...`)_

## Detailed findings

### [Medium] Two structs represent the same persisted state

The diff introduces a second source of truth. Keep one persisted type and convert at the boundary.

```json oy-findings
[{"id":"review-7bd1...","status":"new","source":"review","severity":"Medium","title":"Two structs represent the same persisted state","locations":[{"path":"src/cli/config.rs","line":41}],"evidence":"Both structs are serialized independently.","body":"Keep one persisted representation.","category":"state-ownership"}]
```

No-findings report

The renderer still writes an explicit empty findings payload:

# Code Quality Review

## Verdict

No major structural concerns.

## Findings summary

No high-conviction findings.

```json oy-findings
[]
```

This is distinct from a failed or incomplete run. Check the report transparency line and command exit status as well.

One-finding remediation

oy enhance audit-2a71...
# Inspect the diff and verification output, then confirm:
oy audit "authentication boundaries"

The enhancer should change only one actionable finding and run focused verification. The next audit should omit the fixed finding or update its lifecycle state based on current evidence.

SARIF output

oy audit --format sarif --out oy.sarif

The output is SARIF 2.1.0 with normalized rule IDs, locations, severity mapping, and oy metadata. Validate and inspect it before upload, especially when repository paths or finding bodies are sensitive.

GitHub code scanning

An audit needs opencode plus provider credentials in the job. Use a protected environment/secret and avoid running untrusted pull-request code with privileged provider credentials.

permissions:
  contents: read
  security-events: write

steps:
  - uses: actions/checkout@v4
  - name: Install oy and opencode
    run: |
      # Pin versions in production CI rather than installing mutable latest releases.
      cargo install oy-cli --locked --version 0.11.15
      # Install/configure a compatible opencode release separately.
  - name: Run oy audit
    env:
      PROVIDER_API_KEY: ${{ secrets.PROVIDER_API_KEY }}
    run: oy audit --format sarif --out oy.sarif
  - name: Upload SARIF
    if: always() && hashFiles('oy.sarif') != ''
    uses: github/codeql-action/upload-sarif@v4
    with:
      sarif_file: oy.sarif

For other consumers, archive oy.sarif and ingest it with their SARIF 2.1.0 interface. oy does not upload reports itself.

CLI and MCP reference

Commands

Run oy <command> --help for all options and defaults. Unknown top-level actions and flags pass through to opencode unless they begin with a known oy command.

Safety modes

These modes apply to general launcher/remediation behavior. Restricted audit and review subagents define their own narrow tool permissions.

Setup ownership and refresh

Global setup writes ~/.config/opencode/; workspace setup writes .opencode/. It creates seven agents and two skills, then merges config.

oy owns and replaces:

• mcp.oy;
• command.oy-audit, command.oy-review, and command.oy-enhance;
• tool_output.max_bytes (262,144) and tool_output.max_lines (20,000);
• generated agent/skill files containing the oy marker.

Unknown sibling object keys are retained. A non-object mcp, command, or tool_output value is replaced with an object. The JSON/JSONC config is pretty-serialized, which removes comments and original formatting. Non-generated files at generated Markdown paths are not overwritten.

Launch-oriented commands refresh global integration and any detected workspace integration before opencode starts. Direct opencode use is not changed to the oy default agent; oy passes --agent only for its own launches.

MCP tools

Tool names are prefixed by the host’s MCP server name, for example oy_repo_manifest. Optional tools are advertised only after their executable passes a capability probe. Approximate token estimates currently use oy’s byte heuristic; the model value is workflow metadata, not tokenizer selection.

Optional helpers

mise use --global cargo:tokei
mise use --global github:universal-ctags/ctags
mise use --global cargo:https://github.com/Corgea/Sighthound@tag:1.0
# Homebrew alternative for two helpers:
brew install tokei universal-ctags

Helpers resolve to canonical absolute paths. Relative PATH entries are ignored. Calls use fixed arguments, closed stdin, timeouts, and output limits. Sighthound uses embedded rules, one worker, stable sorting, and finding/byte caps.

Sighthound has no release binary at the pinned 1.0 tag; its mise install builds from source and requires Rust 1.85+.

Environment

Filesystem and disclosure boundaries

Input paths must resolve under OY_ROOT or the current working directory. Output paths must be workspace-relative and may not escape through parent traversal or symlinks. The collector skips documented path/file categories and files over 512 KiB.

Repository text returned by MCP may become model input. Fixed external processes include read-only git and optional evidence helpers; oy MCP does not expose arbitrary shell, edit, web, network, or clone capabilities.

For implementation-level detail, read Tool safety and Architecture.

Compatibility

This matrix distinguishes what CI exercises from what release automation merely builds. It avoids implying support that the project does not currently test.

oy release targets

The curl installer requires a Unix-like POSIX shell. Rust 1.96 is the minimum source-build toolchain.

opencode host compatibility

oy currently integrates through opencode CLI flags, generated Markdown agents/skills, JSON config, and local stdio MCP. The installer selects the current opencode release through mise with a configurable minimum-release-age filter.

There is not yet an automated cross-version opencode matrix. The roadmap keeps that limitation explicit rather than claiming a minimum host version without evidence.

Optional evidence helpers

Relative PATH entries are ignored. Use OY_TOKEI, OY_CTAGS, or OY_SIGHTHOUND with an absolute path to override discovery.

Reporting compatibility problems

Include:

• oy --version;
• opencode --version;
• operating system and architecture;
• install method;
• oy doctor --json output with sensitive paths reviewed;
• whether setup is global or workspace-local.

Tool Safety

oy no longer exposes a native model-callable tool registry. The host owns the general tool surface: file reads/edits, bash, web fetches, task/subagent tools, questions, todos, repository cloning, and permissions.

This document covers only the deterministic tools served by oy mcp.

MCP Capability Matrix

Install optional local helper CLIs with:

mise use --global cargo:tokei
mise use --global github:universal-ctags/ctags
mise use --global cargo:https://github.com/Corgea/Sighthound@tag:1.0
# or: brew install tokei universal-ctags

The pinned Sighthound tag has no release binary; mise builds it from source and therefore requires Rust 1.85+.

Optional helpers are resolved once per oy mcp process to canonical absolute paths. Relative PATH entries are ignored; OY_TOKEI, OY_CTAGS, and OY_SIGHTHOUND can override discovery with an explicit absolute executable path. Probes verify successful version/capability output; calls close stdin and enforce tool-specific time and per-stream output limits. On Unix, helpers run in a dedicated process group that is terminated after direct-child exit or timeout so descendants cannot hold captured pipes open. Ctags option-file loading is disabled with --options=NONE. Sighthound is restricted to embedded rules and one worker; returned findings are stably sorted, string/array bounded, and capped below the generated host tool-output budget. Unsupported-language scopes return an empty status, and all-mode scans fall back to simple analysis when a language pack has no taint rules.

Run oy doctor to check whether the optional MCP tools are currently exposed.

What oy MCP Does Not Provide

oy mcp intentionally does not provide:

• arbitrary shell execution (it does run fixed, bounded git and optional helper commands)
• source edits
• arbitrary file reads beyond deterministic inputs
• web fetches
• repository cloning
• todo management
• model calls
• session persistence

Use built-in tools and permissions for those capabilities. When launched through oy, the generated oy, oy-plan, oy-edit, and oy-auto agents map old oy safety modes onto host permissions.

Filesystem Boundary

Workspace input tools resolve paths under the current workspace root (OY_ROOT or cwd). They accept workspace-relative paths and absolute paths that resolve inside the workspace. They reject parent traversal and resolved paths outside the workspace.

Report renderers use config::resolve_workspace_output_path, which rejects absolute paths, parent traversal, symlink ancestors that escape the workspace, and symlink final destinations.

When changing this boundary:

• validate before reading or writing,
• canonicalize existing paths,
• keep output writes inside the workspace,
• add tests for traversal and symlink cases,
• avoid broadening file collection without documenting the disclosure impact.

Disclosure Boundary

The host decides what to send to the selected model. oy mcp can return repository text chunks, so returned chunk content may become model input.

The collector skips gitignored and hidden paths, common dependency/build directories, lockfiles, generated reports, likely-secret file names, binary/non-UTF-8/empty files, unreadable files, and files larger than 512 KiB. “Every chunk” therefore means every collected chunk, not every repository byte. Keep this conservative and make exclusions visible when completeness matters.

Sighthound does not use that collected file list. It has independent gitignore-aware discovery, common directory exclusions, supported-language filtering, and a larger file limit (currently 10 MiB). It can therefore inspect supported hidden source or source files omitted by oy’s size limit. Generated auditors invoke it only when focus explicitly asks for Sighthound/SAST. Treat returned snippets as an additional disclosure boundary.

Permission Boundary

The host handles user approval for its own tools. oy mcp report-writing tools are exposed as MCP tools, so permission behavior follows the host MCP/tool configuration. Keep generated agents explicit about when they call report renderers.

Adding A New MCP Tool

Only add a tool if it is deterministic repo analysis or deterministic report rendering that the host does not already provide.

Checklist:

• no hidden LLM calls,
• no shell/process side effects unless read-only and documented,
• no network access,
• bounded external-process runtime and output,
• workspace path validation near entry,
• clear JSON schema in src/mcp.rs,
• generated agents/skills updated if the tool should be used by workflows,
• tests or smoke coverage for tools/list and tools/call behavior.

Project direction

oy exists to improve the audit → review → remediate loop in opencode, not to become another general AI coding runtime.

Mission and audience

Mission: make repository-wide audit, review, and remediation in opencode more repeatable, bounded, and reviewable.

Primary user: a maintainer already using opencode who wants better evidence coverage and durable reports without adopting another provider client, chat UI, or agent framework.

Primary artifact: a scoped, evidence-backed Markdown or SARIF report that can be rerun, compared, and handed to one-finding remediation.

Product scope

Decision principles

1. Own the evidence boundary, not the model. Deterministic collection and rendering are oy’s value.
2. Fail closed instead of sampling silently. Scope, limits, and exclusions should be inspectable.
3. Optimize for handoff artifacts. Reports and finding lifecycle matter more than chat features.
4. Make setup reversible and unsurprising. Owned writes and refresh behavior must be explicit.
5. Add native code only for deterministic value. Prefer host capabilities when they already solve the job.

Roadmap summary

Now: dependable core contract

• Preserve unrelated user config and validate setup ownership/idempotency.
• Add MCP transport, collection, diff, Markdown, and SARIF fixtures.
• Report included scope and skipped categories; resolve lockfile coverage.
• Publish tested opencode/platform compatibility and stronger doctor checks.
• Expand evaluated audit/review/remediation examples and quality canaries.

Next: lower friction

• Keep CLI/MCP reference material verified against source.
• Improve practical SARIF/CI integration and machine-readable preflight.
• Improve explicit monorepo scoping without hidden sampling.
• Retain optional helpers only when evaluation shows measurable value.

Later: stable host integration

Adopt stable opencode APIs/config changes when they reduce CLI coupling, while keeping the deterministic helper boundary small and migration straightforward.

Read the canonical roadmap and success criteria.

Non-goals

• Rebuild opencode’s provider routing, model loop, chat, sessions, editing, shell, web, or general search.
• Add arbitrary shell, edit, network fetch, or clone capabilities to oy MCP.
• Claim deterministic findings from model reasoning.
• Persist provider credentials, transcripts, model selection, or session state.
• Run paid/provider-backed evaluations in default CI.
• Chase every host prerelease before its integration contract stabilizes.

Contribute

Changes should improve evidence coverage, report usefulness, setup safety, or measured prompt quality without broadening the product into a second host.

Read the contributor guide, architecture, and evaluation playbook.

Architecture

oy is a focused audit/review integration layer for opencode. It does not own model execution, chat UI, sessions, provider routing, shell execution, file editing, web fetching, or a native LLM tool loop. The host owns those surfaces.

oy owns three things in support of the audit → review → remediate loop:

• setup and launch convenience
• a local stdio MCP server with deterministic repository helpers
• focused convenience wrappers plus passthrough to opencode for unknown top-level actions/flags

Runtime Flow

user argv/stdin
  -> src/main.rs
  -> oy::run in src/lib.rs
  -> cli::app in src/cli/app.rs
  -> opencode wrappers in src/opencode.rs
       -> oy setup writes ~/.config/opencode/* by default
       -> opencode --agent oy...

opencode
  -> starts local MCP command: oy mcp
  -> src/mcp.rs JSON-RPC stdio loop
  -> deterministic helpers in audit/tools modules

Main Modules

Deleted legacy modules include src/agent/, src/llm/, native chat/session handling, native provider routing, and the old model-callable tool registry.

Setup

oy setup writes global files under ~/.config/opencode/ by default:

~/.config/opencode/opencode.json
~/.config/opencode/agents/oy.md
~/.config/opencode/agents/oy-plan.md
~/.config/opencode/agents/oy-edit.md
~/.config/opencode/agents/oy-auto.md
~/.config/opencode/agents/oy-auditor.md
~/.config/opencode/agents/oy-reviewer.md
~/.config/opencode/agents/oy-enhancer.md
~/.config/opencode/skills/oy-audit/SKILL.md
~/.config/opencode/skills/oy-review/SKILL.md

oy setup --workspace writes the same generated files under .opencode/ for project-local overrides:

.opencode/opencode.json
.opencode/agents/oy.md
.opencode/agents/oy-plan.md
.opencode/agents/oy-edit.md
.opencode/agents/oy-auto.md
.opencode/agents/oy-auditor.md
.opencode/agents/oy-reviewer.md
.opencode/agents/oy-enhancer.md
.opencode/skills/oy-audit/SKILL.md
.opencode/skills/oy-review/SKILL.md

The config registers oy mcp as a local MCP server and adds commands for audit, review, and enhance workflows. Generated primary agents (oy, oy-plan, oy-edit, oy-auto) support launcher compatibility, but restricted audit/review agents are the product’s primary direction. oy passes --agent when launching instead of setting default_agent, so direct opencode usage keeps the user’s normal default. Running sessions need to be restarted after setup changes because config loads at startup.

Launch-oriented commands refresh the global integration before starting opencode and refresh workspace integration files when an existing oy workspace integration is detected. The config merge replaces oy-owned mcp.oy, command.oy-*, and tool_output.max_bytes/max_lines values. Unknown sibling object keys survive, but parsing and pretty-serialization remove JSONC comments and original formatting. Generated Markdown files refuse to replace files without the generated marker.

MCP Boundary

src/mcp.rs implements the small MCP subset needed by the host:

• initialize
• ping
• tools/list
• tools/call

The server is intentionally deterministic at the collection/rendering boundary. It returns manifests, chunks, diffs, existing reports, SLOC, outlines, optional Sighthound findings, and report-rendering results. It does not call an LLM. Audit/review conclusions remain model-dependent.

Tools exposed by oy mcp:

Sighthound uses its own gitignore-aware discovery and file-size limit rather than the manifest/chunk collector’s exact exclusions. The generated auditor invokes it only for explicit Sighthound/SAST focus, and its returned snippets form an additional disclosure boundary.

Trust Boundaries

Design Rules

• Do not reintroduce an LLM client or model tool loop in oy.
• Prefer host agents, commands, skills, and permissions for orchestration.
• Keep MCP tools deterministic and narrow.
• Keep workspace path validation close to reads/writes.
• If the host already has a tool, do not duplicate it in oy mcp.
• Update generated setup docs and schemas when changing MCP tool names or command templates.

LLM Evaluation

oy has two different quality bars:

1. Deterministic Rust/CI tests for the code oy owns: setup, config merging, MCP protocol behavior, path safety, repository chunking, and report rendering.
2. Live model evaluations for the behavior opencode owns: whether generated agents find useful findings, avoid noise, and follow the audit/review protocol with a real model.

Do not mix them. CI should stay deterministic and provider-free. Prompt changes should be judged with pinned public repositories, fixed model/provider settings, and a before/after scorecard.

Capability Inventory

Current generated capabilities:

Useful deterministic evidence already exposed by MCP: repo_manifest, repo_chunks, git_diff_input, existing_report, optional sloc, optional outline, optional sighthound, render_audit_report, and render_review_report. These tools do not clone repositories, fetch the web, edit source, run arbitrary shell commands, or call a model.

Evaluation Corpus

Use small, pinned, public open-source projects rather than only self-reviewing this repo. Keep clones and model outputs under .tmp/eval/ so they never enter the release artifact or git history.

The tracked seed corpus is docs/eval-corpus.toml. It was seeded from recent public GitHub activity for adonm: Rust/DataFusion/geospatial work in apache/sedona-db, tomtom215/quack-rs, datafusion-contrib/datafusion-ducklake, and adonm/zuko.

Start with three lanes:

Prefer tasks that fit within --max-chunks 80 at the default target_tokens. For larger projects, evaluate a documented path focus instead of raising chunk budgets until the task becomes impossible to compare.

For each task, record a rubric, not an exact output snapshot:

• repository URL, license, pinned commit/tag, and checked-out path
• command (audit, review, or enhance) and focus text
• expected issue classes, affected files/symbols, and unacceptable false-positive categories
• required report shape: transparency line, valid oy-findings JSON, path/line evidence where available, and a clear no-findings verdict when appropriate
• model/provider, opencode version, oy commit, date, elapsed time, and chunk count

Local Runner

Use the local runner for repeatability. just eval validates the corpus only; it does not clone repos or call a model.

just eval
python3 scripts/eval_runner.py list
python3 scripts/eval_runner.py run --dry-run
python3 scripts/eval_runner.py run --opencode-model openai/gpt-5.5 \
  --model-slug openai-gpt-5.5 \
  --task sedona-geoparquet-aws-allowlist-review
python3 scripts/eval_runner.py compare \
  .tmp/eval/runs/<baseline>/summary.json \
  .tmp/eval/runs/<candidate>/summary.json

The runner:

• reads docs/eval-corpus.toml
• clones/fetches pinned public refs under .tmp/eval/repos/
• builds the local oy binary unless --skip-build is passed
• prepends target/debug to PATH so opencode’s MCP command uses the candidate binary
• runs oy setup --workspace, then the configured oy audit or oy review (or opencode run --model ... --command oy-audit/oy-review when --opencode-model is supplied)
• copies reports and writes summary.json/summary.md under .tmp/eval/runs/
• checks report shape, oy-findings JSON, keyword/path scorecard hints, and unexpected source mutations outside .opencode/ and .oy-eval/

Full runs require opencode and a configured model provider. They are deliberately not part of default CI.

Manual Run Protocol

Evaluate the candidate oy binary that contains the prompt changes. The MCP server launched by opencode resolves oy from PATH, so put the local build first or install the candidate binary before running evals.

cargo build --locked
export PATH="$PWD/target/debug:$PATH"
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-<model-slug>"
mkdir -p .tmp/eval/repos .tmp/eval/runs/"$RUN_ID"

git clone --depth 1 --branch <tag-or-branch> \
  https://github.com/<owner>/<repo>.git \
  .tmp/eval/repos/<repo>

# For diff reviews, also fetch the target ref/SHA with enough history for
# `git diff <target-ref>` to work inside the clone.

(
  cd .tmp/eval/repos/<repo>
  OY_ROOT="$PWD" oy setup --workspace
  OY_ROOT="$PWD" oy audit --out ".oy-eval/$RUN_ID/ISSUES.md" \
    --max-chunks 80 "<focus>"
  # Omit the positional target for whole-repo precision reviews.
  OY_ROOT="$PWD" oy review <target-ref> \
    --out ".oy-eval/$RUN_ID/REVIEW.md" \
    --max-chunks 80 --focus "<focus>"
)

mkdir -p .tmp/eval/runs/"$RUN_ID"/<repo>
cp .tmp/eval/repos/<repo>/.oy-eval/"$RUN_ID"/*.md \
  .tmp/eval/runs/"$RUN_ID"/<repo>/

If the repo needs dependencies or tests for an oy enhance pass, install and run them only inside the clone. Do not point OY_ROOT at a parent directory that contains secrets or unrelated projects.

Scorecard

Score before and after a prompt change with the same model, opencode version, commands, focus, and refs.

Use a simple verdict per task: better, same, worse, or inconclusive. Accept prompt changes only when they improve at least one target lane without a material regression in the others.

Prompt Iteration Rules

1. Make one prompt change at a time.
2. Run the old and new prompts on the same pinned corpus.
3. Prefer shorter prompts unless longer text measurably improves the scorecard.
4. Preserve hard safety/protocol constraints in generated agent tests.
5. Put the scorecard summary in the PR or release notes; keep raw eval artifacts under .tmp/eval/.
6. Do not exact-match model prose. Match behavior: evidence, report schema, finding quality, and false-positive rate.

Planned work

ROADMAP.md is the canonical backlog for deterministic fixtures, corpus expansion, report examples, and workflow automation. Keep future work there so evaluation guidance describes the current method rather than maintaining a second roadmap.