Review 3.6: severity tiers you can tune per practice area

Review 3.6 is available to all accounts from today. The headline change is that the single 0–100 risk score is gone. In its place, every flag carries one of three severities: High, Medium or Note. The thresholds that decide which tier a flag lands in are now set per practice area, not per firm.

This note explains what changed, why we changed it, what to check after upgrading, and what we have not yet solved.

What changed

• Three tiers replace the score. Each flag in the Word pane and in the summary report is labelled High, Medium or Note. The numeric score is no longer displayed. It still exists internally and can be exported through the API for firms that have built reporting on top of it.
• Thresholds per practice area. Playbook administrators can define a severity map for each practice group (for example M&A, commercial, employment, real estate). The same clause pattern can be High in one group and Note in another.
• Severity inherits from the playbook rule, then from the practice default, then from the firm default. A rule with no explicit severity falls through in that order. The effective severity is shown in the rule editor so there is no guessing.
• Filter and sort by tier. The review pane can hide Notes, show only Highs, or sort by tier and then by document position.
• Summary report restructured. The one-page summary now opens with a count of Highs, then Mediums, then Notes, followed by the Highs in full. Notes are listed by heading only.

Why we did it

The risk score was accurate and nobody used it.

We watched reviewers across roughly forty accounts work through drafts during the second half of 2024. A reviewer looking at a flag scored 71 and another scored 64 did not treat them differently. What they did do, almost universally, was triage: this one I must deal with, this one I should look at, this one I can leave. Three buckets. The score was being mentally rounded to a tier anyway, so we made the tier the primary object.

The second finding was that the buckets moved depending on who was reviewing. A limitation-of-liability cap set at twelve months’ fees is a serious issue in a long-term outsourcing MSA and a non-event in a one-off consultancy agreement. The same pattern, a different answer. A firm-wide threshold had to be either too noisy for commercial work or too quiet for M&A. Letting each practice group set its own lines removes the compromise.

The question a reviewer asks is not “how risky is this out of a hundred” but “do I have to do something about it before this goes back to the other side”.

What to check after upgrading

Existing playbooks have been migrated automatically. The migration maps the old score bands to tiers as follows: 70 and above becomes High, 40 to 69 becomes Medium, below 40 becomes Note. These are starting points, not recommendations.

We suggest three checks in the first week:

1. Open the severity map for each practice group and confirm the defaults are what the group head would expect. The most common adjustment we saw in the preview cohort was pulling governing-law and jurisdiction mismatches up to High for cross-border teams.
2. Run one recently reviewed draft through 3.6 and compare the tier labels against what the reviewer actually did with each flag. If a flag was fixed before the draft went back, it should probably be High or Medium. If it was ignored, it should probably be a Note.
3. Review any API integrations that read the numeric score. The field is still present but is now marked deprecated. It will remain for at least twelve months.

Rules created before 3.6 that had severity set explicitly keep that setting. Rules that relied on the firm default now resolve through the practice default first, which may change their effective tier. The rule editor shows a small “changed on migration” marker where this has happened.

Known limitations

• A flag can carry only one severity. A clause that is a Note for commercial reasons and High for regulatory reasons will show the higher tier, with the second reason in the explanation text. We are testing whether a secondary label is useful or merely clutter.
• Practice-area detection is driven by the matter’s tags in the firm’s document management system. Documents reviewed outside a tagged matter fall back to the firm default map. The pane shows which map is in effect, but the fallback is easy to miss. We are considering a more visible indicator.
• Severity maps cannot yet be copied between practice groups through the interface. Administrators who want a shared starting point will need to export and import. This is on the list.
• The summary report’s “Notes by heading only” layout can hide useful detail for very short agreements where everything is a Note. We may add a toggle.

Smaller changes in 3.6

• Redline proposals in Word now preserve the paragraph’s existing numbering style instead of resetting it.
• The explanation text under each flag quotes the playbook rule that triggered it, with the rule’s last-edited date.
• Reviewing a document that has already been reviewed under an older playbook version shows a one-line notice with the version difference.
• Load time for documents over 150 pages is reduced by roughly a third in our benchmark set.

If your group has not yet set its own severity map, set it this week while last quarter’s drafts are still fresh in mind.