AVCEL Engineering

Do AI coding agents follow company policy?

What happens when coding agents meet company rules nobody told them about.

AVAVCEL Engineering
5×
more policy-compliant changes than with no policy pointer
59% vs 12%, policy outside the repo
72%
of sealed-task changes follow policy with AVCEL, against 13% with no help
same 36 tasks, separate runs
4.0×
lower agent model cost per compliant completion
$1.31 → $0.33, policy outside the repo
3,800+
agent runs on 85 sealed tasks
two codebases, hidden tests

Apache Fineract and openIMIS · benchmark policies for a fictional bank and insurer · hidden feature and policy tests

Coding agents build what the ticket asks. The company rules nobody mentioned get missed. AVCEL hands the agent those rules.

One ticket, two runs

A real insurance ticket: let reviewers fix a typo in a claim line's justification.

Figure 1. Same ticket, same agent, two real runs
Coding agent alone31s · $0.07 · 8 turns
> Let reviewers fix the justification of a claim line: add edit_line_justification(line, text, user) to claim/services.py…
policy folder: never opened
●Searchgrep "^def |^class " services.py
●Searchgrep "def save|class Claim" models.py
●Searchgrep "ClaimItem" tests/tests_services.py
●Readcore/versioned_model.py
●Editservices.py, tests/tests_services.py
●Runclaim tests✓ pass
✓“I added edit_line_justification(line, text, user) … It raises ValidationError before changing anything if the line's validity_to is set.”
Hidden tests
Feature7 / 7
Policy0 / 7
Coding agent + AVCEL48s · $0.09 · 7 turns
> Let reviewers fix the justification of a claim line: add edit_line_justification(line, text, user) to claim/services.py…
⚑ AVCEL · 2 company rules for this change
ICS-4.1 An override … must carry a non-empty justification.
ICS-4.3 Overrides and their justifications … may not be edited after the claim is processed.
●Readservices.py
●Searchgrep "class ClaimDetail" models.py
●Searchgrep "qty_approved|price_approved" *.py
●Readmodels.py
●Editservices.py+2 rule checks
●Runclaim tests✓ pass
●Runfix a test fixture, claim tests again✓ pass
✓“Two extra refusals come from the business rules … it refuses when the claim is processed or valuated … it refuses an empty or blank text when the line has an approved quantity or price that differs from the default.”
Hidden tests
Feature7 / 7
Policy7 / 7

Claude Code (Sonnet 5.5), replayed at 10× speed from the recorded steps. Policy in a folder outside the repo; tests hidden from the agent. Both repeats of each setup ended the same way.

Both said “done”. Only one followed the rules.

This AVCEL run took longer and cost 2 cents more, because it did the work the other run skipped. Across the experiments, AVCEL's cost per compliant change was lower, because far fewer of its attempts failed.

How the rules reach the agent

Figure 2. Approved once, delivered with every task
Policy handbooks Regulations & circulars Internal procedures Product decisions Codebase AVCEL BlueprintApproved by policy ownerslimits, dates, versions, exceptions Coding agentThe rules for this change,in the developer's first message

Built once per codebase and resolved locally. AVCEL's server does not need to receive the developer's task, conversation or working code. The coding agent still talks to its configured model provider.

What happened across the experiments?

16 policy-risk tickets × 2 attempts per setup, policy in a folder outside the repo.

Figure 3. 32 attempts per setup: more compliant changes, for less
0%25%50%75%100%$0.25$0.50$1.00AGENT MODEL COST PER COMPLIANT CHANGE →COMPLIANT CHANGES ↑↖ betterNo policy pointer12% · $1.31Manual instructions50% · $0.43AVCEL59% · $0.33+47pp compliant4.0×lower cost

Compliant: 4/32 with no pointer, 16/32 with manual instructions, 19/32 with AVCEL (banking and insurance, 16 each). Manual instructions: the repo's CLAUDE.md and AGENTS.md point to the policy. AVCEL vs no pointer +47 points (95% CI +25 to +69); vs manual instructions +9 (95% CI −12 to +28), not established as significant.

Of course rules help. But does AVCEL beat a careful team's instruction file?

Figure 4. AVCEL vs a repo with the full policy and instruction files
AVCEL75%
Repo instruction file63%

28 policy-risk tasks from a 49-task cohort, 3 attempts each: 63/84 vs 53/84 compliant, +12 points (95% CI +4 to +20). The tasks were built after the design was fixed and never used to shape it; some ran in earlier evaluations. Banking 72% vs 61%, insurance 77% vs 65%. Cost per compliant change $0.26 vs $0.37.

What we learned

Rules can be followed too well. The bank's handbook: “A fee or penalty waiver made under an active hardship plan needs no checker, but it must reference the plan.”

Figure 5. Allowing what the rules allow
handled correctlyblocked work the rules allow
Manual instructions
5 / 127 sent to a checker anyway
AVCEL
11 / 121 sent to a checker

Three exception tasks (one banking, two insurance), 4 attempts each, policy in the repo.

AVCEL doesn't win everywhere yet. On one set of insurance tasks, a repo holding the full handbook still did better (78% vs 53%). Some rules depend on a definition or a related rule the agent also needs. What we are researching:

Stronger enforcementAdding the definitions and related rules a rule depends on, only where it needs them.
More complete coverageBringing more of the rules that govern a change, including cross-cutting ones, at enterprise scale.
Beyond policyAgents with the right rules already finish for less. Policy is the first thing the Blueprint gives them.

AVCEL Engineering. The institutions and policies in this benchmark are fictional, written for research and reviewed by AI, not by a domain expert.