Testing the AI legal-doc generator
Separate a demo from real engineering: read the PR to confirm your skill was used, test the AI chat like a user (and feel Cerebras fly), turn feedback into fixes, and ship every document type in PL-6.
PL-5 is built — an AI chat that interviews the user and fills the document. Now we do the two things that separate a demo from an engineer: read the code to confirm it used our skill, and test it like a user. Then we push straight on to PL-6 and support every document type.
01What PL-5 delivered
Claude reports success: an AI chat interface, a backend API, LiteLLM talking to Cerebras, structured outputs, and live preview updates — all in a PR of about 13 files and 456 lines. Impressive on paper. But two checks come first.
02Did it actually use our skill?
Open the PR and read the code. We are looking for the fingerprints of the Cerebras skill we wrote:
A Pydantic schema
A structured-output model for the fields — exactly what we asked for.
extra_body
Present — taken precisely from the way we wrote it in the skill. Proof the skill was read and used.
The right model
gpt-oss-120b via OpenRouter, pinned to Cerebras.
Seeing your own extra_body and provider routing appear verbatim in generated code is the whole
point: you encoded expertise once, and Claude applied it faithfully — and would do the same for every teammate who
clones the repo.
03Test it like a user
Start the server and open localhost:8000. A greeting appears: “I'll help you create a
mutual non-disclosure agreement. Let's start with the basics…” Answer its questions and watch the
right-hand preview fill in.
The responses come back instantly — the first chunk is basically the whole answer. That is Cerebras, and it is exactly why we argued streaming was unnecessary. A great demonstration of a deliberate design choice paying off.
It is not perfect, and that is fine — testing is building. Two honest pieces of feedback surface: the chat sometimes stops asking questions and the conversation dries up (it should always ask the next question if it needs more), and you can only download once you reach the end. We note these — they become fixes in the next ticket. Call it a 90% victory: real AI chat, over OpenRouter, using our new skill.
04Merge, record, reset
Same rhythm as before: merge the PR locally and push to main; make sure CLAUDE.md reflects the
new status; /clear; and re-authenticate Atlassian ready for the next ticket. Small, repeatable
discipline — it is what keeps a multi-ticket build from turning to mush.
05PL-6: every document type — plus those fixes
PL-6 expands the product to all the templates we curated. And because we can, we fold the two bits of feedback into the same request:
implement Jira ticket PL-6. Also please make a couple of fixes: after answering a question, return focus to the text input field, and always ask a follow-on question if more information is needed. Test, then make a PR.
It completes PL-6, applies the two fixes, adds some quality improvements from its own review, and raises a PR. Test it: ask for a Cloud Service Agreement — it recognises the type instantly, asks a clean run of questions (company names, dates, governing state, subscription period, pricing, payment terms), and fills the preview live on the right, in the right brand colours. Press Download PDF — a complete, polished document. Another 2,500-line PR, another feature shipped.
Notice how naturally it flows: build → test as a user → capture concrete feedback → fold it into the next ticket. You are not writing code; you are steering — noticing what a real user would, and turning it into precise instructions.
✓ Key takeaways
- Before trusting a PR, read the code — here we confirm
extra_body, the Pydantic schema, and the right model, proving our skill was used. - Test like a user. Cerebras makes responses feel instant — validating the “no streaming” decision.
- Real feedback (the chat drying up; download timing) is not failure — it becomes precise fixes in the next ticket.
- Keep the rhythm: merge → update
CLAUDE.md→/clear→ re-auth. - PL-6 adds all document types and the two fixes in one instruction — steering, not coding.