← All posts
How we test Docwake
The four test sets, what counts as a pass, and where the agent still fails.
FIG. 07
Type bars, a strike at a time.
Engineering
By A. Sample
Engineering
8
min
Sample post
Every name, figure and line here is sample copy.
Every week Docwake drafts edits to documentation it did not write. Before a draft reaches a person, it has passed four sets of tests. This post explains what each set checks, what counts as a pass, and where the agent still gets it wrong.
The four test sets
Stale page search: given a merge, did Docwake find every page that no longer agrees with the code?
Draft quality: does the edit change only the lines that were wrong, in the voice of the page around them?
Checks: do the links resolve, do the code samples run, and does the page follow its style rules?
Restraint: when a page is still correct, does Docwake leave it alone?
What counts as a pass
A draft passes when a reviewer would merge it without touching it. We do not score how close a draft comes. A draft that needs one word changed is a fail, because it still costs a person their attention.
A worked example
When a merge renamed the setting timeout_ms to request_timeout_ms, Docwake found four pages that used the old name. It drafted edits for three of them and flagged the fourth, a migration guide, for a person, because that page should mention both names.
The best draft is the one nobody has to think about.
Where the agent still fails
Pages that explain why, not just how. Docwake can update a value, but not a reason.
Screenshots. It can tell an image is out of date, but it cannot take a new one.
Changes spread over several merges, where no single merge tells the whole story.
We publish the pass rate for each set with every release. The figures on this site are samples; yours will come from your own repository.
Next post
One year of release notes →
New posts by email.
One email per post. Unsubscribe at any time. See the privacy notice.
