Assessment · C.6 and B-4
Decide how you would know it failed, first
Every study in this literature is retrospective and non-randomized, which is why the evidence base is so weak. A pre-registered evaluation filed before the first device ships would be the best evidence in the field on day one - and it is the only kind this program can still choose to produce.
There are two ways a monitoring program reaches its reauthorization hearing with nothing to say. It can fail to work, or it can fail to be measurable. The second is the one this design document is at risk of, because the privacy architecture that makes it defensible is the same architecture that makes it hard to evaluate - and that is not a hypothetical, it is a documented outcome in the closest adjacent system.
MIT Lincoln Laboratory identified exactly this in exposure notification: the privacy-preserving aspects of the protocol make it difficult to assess the performance of the apps in real-world populations [15]. Part B of the assessment carries that forward as B-4, and it is the reason this page exists rather than a paragraph on operations.
Why a pre-registration is worth more here than anywhere else
The literature this design document keeps citing has a structural weakness that has nothing to do with the findings. Every study in it looks backwards at a program that was already running, with a comparison group assembled after the fact.
| Study | Design | What that costs the finding |
|---|---|---|
| Bales et al., NIJ 2010[22] | Retrospective, matched | Large sample, no randomization. The matching is on observable characteriztics, and supervision assignment is not made on observable characteriztics alone. |
| Gies et al., NIJ 2012[23] | Retrospective, matched | The favorable direction is reported alongside the DOJ SMART Office's statement that the differences are not statistically significant. |
| Turner et al., 2015[24] | Small matched design, one year | The study that most directly tested new offending, and the arms were 94 against 91 - which is the power problem this page is about, in the field. |
| Zgoba & Mitchell, 2021[25] | Meta-analysis of 18 retrospective studies | A meta-analysis inherits the design of what it pools. It can correct for sampling error; it cannot randomize after the fact. |
Not statistically significant is how the DOJ SMART Office reports both of the California GPS differences above.DOJ SMART Office, SOMAPI ch. 8, restating Sandler, Freeman & Socia (2008), Psychology, Public Policy, and Law 14(4)
The direction of the effect is favorable; the study could not establish that it is real.[4]
Nothing above is a criticism of the researchers, who evaluated the programs that existed. It is a description of an opportunity: a program that has not yet deployed can still pre-register, and a pre-registered evaluation with a named comparison group would be, on the day it published, the strongest piece of evidence in this field.
DOJ SMART Office, SOMAPI ch. 8, restating Sandler, Freeman & Socia (2008), Psychology, Public Policy, and Law 14(4)
The direction of the effect is favorable; the study could not establish that it is real.
C-6a · Pre-register before deployment
Pre-registration is only worth something if it happens before anyone knows how it went. The ordering below is the substance of the requirement, not a project plan.
- Name the primary outcomes, in a filed document, before the first device ships. Not a list of things that will be reported - a short list of things the program will be judged on, with the direction of the expected effect stated in advance.
- Name the comparison group and how people enter it. A comparison group chosen after the results are in is not a comparison group. If randomization is impossible, say why, and pre-specify the matching variables and the matching procedure before any of them are measured.
- File the analysis plan publicly. Which test, on which outcome, at which significance level, with which corrections for multiple comparisons, and what counts as a completed follow-up. Published, timestamped, and not amendable in silence.
- Pre-specify the stopping rules. What result would end the pilot early for harm, and what result would end it early for futility. A program with no futility rule cannot fail; it can only continue.
- Publish the whole thing whichever way it goes. The commitment to publish a null result is the part that makes the other four worth anything, and it is the part that gets quietly dropped.
The unglamorous reason this is priority five in the assessment's table
It is not the most valuable item. It is the item with a deadline attached to it. Compliance assist can be added late, the coverage model can be built late, the anchor security specification can be written late - a pre-registration cannot be filed late, because a pre-registration filed after deployment is not a pre-registration. The assessment's own note on the row says it: it must happen before deployment or it cannot happen at all.
C-6b · Name outcomes the system can actually move
The temptation is to name the outcome everybody cares about. The whole point of this page is to refuse that, and to name instead the outcomes this system has a mechanism for.
| Outcome | Status | Why it is on this list |
|---|---|---|
| False-alert rate, by tier | Primary | The core claim of the design is that graduated escalation makes alerts actionable. This is that claim, measured. Reported per tier, because a single pooled figure would hide the tier the ladder exists to protect. |
| Officer response rate | Primary | The failure the concept is aimed at is not detection, it is that nobody opens the alert. If response does not rise, the corroboration gates achieved nothing that matters. |
| Absconding | Primary | A replicated effect of electronic monitoring, on an outcome supervising agencies are already measured on. |
| Registration compliance | Primary | Same. These two are the effects the evidence supports, and they are worth money to the buyer on their own terms. |
| Warned-and-turned-back ratio | Primary | Warned and turned back, against warned and continued. It is the most useful operational metric the system can produce, it exists only because of compliance assist, and it is the number a reauthorization hearing will actually want. |
| Time from corroborated presence to human adjudication | Secondary | An operational outcome the system controls directly, and the one that decays first when volume rises. |
| Wrongful-flag rate and time to correction | Secondary | The harm the system does to the people it monitors, measured with the same seriousness as the harm it prevents. |
| New sexual offending | Secondary · underpowered | Reported, never relied on. The reason is arithmetic and it is below. |
The most important sentence on this page
Reoffending is a secondary outcome with honest power limits: base rates are low enough that no realistic pilot can detect a change in it.
That is not modesty and it is not hedging. It is a property of the arithmetic, it holds however well the system works, and it is true of every pilot any jurisdiction would actually fund. A program that names reoffending as a primary outcome has committed to reporting a number that cannot mean anything - and the reviewer who works that out will discount everything else in the report.
What a pilot of this size could detect
Two-proportion power, run here rather than asserted-
is what this pilot would have against the effect specified. The default base rate is the new sex crime conviction rate measured in the matched design that tested it, and the default arm size is a pilot a mid-sized jurisdiction could plausibly run.
2.1% against 9.9% was the absconding rate for GPS-supervised against comparison cases, and failure to register ran 3.2% against 13.2% - while new sex crime convictions were 1.1% in both groups.[24]
Move the base rate slider up towards the measured false-alert rate and watch the required sample collapse. That contrast is the entire argument for the primary outcome list: the same pilot is well powered for the things the system can move and hopeless for the thing it cannot. 95.50% of monthly GPS monitoring alerts in Cook County were false, up from 57.79% under radio frequency.[17]
The model is SPM.power() in /assets/coverage-model.js - an ordinary two-proportion calculation under the normal approximation, with the inverse normal implemented in the file rather than tabulated. It is re-run against its published vectors on the check page in the reader's own browser, including the two that matter: a comparison with no effect rejects at exactly half the significance level, and the required sample for the reoffending outcome exceeds any pilot a jurisdiction would run.
Turner, Chamberlain, Jannetta & Hess (2015), Victims & Offenders
A small matched design, 94 against 91, over one year. It is the study that most directly tested new offending, and it found no difference.
C-6c · Instrument measurability deliberately
The privacy architecture on isolation and identity is the best thing about this design. Routine operation produces no location history, the operator holds observations without keys, the custodian holds keys without observations, and no insider acting alone can produce an identity. It is also the reason the ordinary way of evaluating a monitoring program - query the movement database - is unavailable here, and unavailable by construction.
The failure this closes is specific: discovering at year three that the privacy design made the program unfalsifiable. By then the telemetry cannot be added retroactively, because the events it would have described were never recorded and cannot be recovered. So the minimum aggregate telemetry is specified up front, as part of the design, and it is specified as a maximum at the same time - anything not on this list is not collected.
| What is counted | Granularity | Why it does not rebuild a location database |
|---|---|---|
| Alerts raised, by tier | Per zone class, per day, counts only | A count by class carries no anchor, no subject and no time of day. It cannot be joined to anything. |
| Alerts adjudicated, and the adjudication outcome | Per tier, per day, counts only | The outcome is a category, not an event. No subject identifier enters the aggregate at any point. |
| Officer response, and time to first human look | Distribution per tier, per week | A latency distribution over a week describes the queue, not the people in it. |
| Warnings issued, and the disposition | Counts of cleared-by-leaving against escalated, per week | This is the warned-and-turned-back ratio. It is a pair of counters, and the warning path never learned who else was present, so the counter cannot leak it. |
| Loss-of-signal events, and their duration | Distribution per week | A duration histogram. The last known position is not part of the aggregate. |
| Identity resolutions, with the legal basis | Every one, individually, in the public audit record | This one is deliberately not aggregated. The audit record already exists, the split-custody design already writes it either way, and the count of resolutions is the single most important number an oversight board reads. |
| Absconding and registration compliance | Per subject, from the supervising agency | These come from records the agency already holds under existing authority. The monitoring system contributes nothing to them and does not need to. |
The rule that makes this safe rather than a slow leak
Every row above is a counter or a distribution, incremented at the edge and reported on a fixed schedule. No row is an event log with the identifiers removed, because a log with identifiers removed is a log, and re-identification of stripped movement traces is a solved problem for anyone who wants to do it. If a proposed metric cannot be expressed as a counter, it does not go on the list - and the evaluation is designed around what the counters can answer rather than the counters being expanded around the evaluation.
The consequence is worth stating rather than hiding: some questions become permanently unanswerable. Whether a particular subject changed their routine, whether alerts cluster around particular sites, whether an individual officer's response is worse than their colleagues' - none of those can be asked of this data, and asking them would require the database this design exists to avoid.
C-6d · Red-team before the pilot, and publish the result
An independent adversarial review, commissioned by someone other than the builder, completed before the first device ships, with the findings published whether or not they are flattering. The scope is named in the assessment and it is the right scope.
- The exigency path. Every safeguard in the isolation architecture has an exception written into it, and exceptions in this domain historically expand. The question for a red team is not whether the exception is correctly implemented but how many ways there are to reach it.
- Anchor spoofing. Anchors are unattended devices in public places holding the current-interval enrollment set. A hostile device introduced into a zone is the most accessible attack in the system - see anchor security for the countermeasures this review would be testing.
- Replay injection. A captured broadcast re-emitted elsewhere manufactures a presence that never happened, and the person it convicts was not there. This is the attack that turns the system into a weapon against the people it monitors.
- Stalking inversion. The child-side device turned into a locator, and the monitored person's device turned into a child detector. The threat model treats both as first-class adversaries; a red team is how that stops being an assertion.
Published, or it did not happen
An unpublished penetration test is a procurement document. It reassures the buyer, it changes nothing for the person wearing the device, and it cannot be checked by anyone with a reason to check it. The item on the list is with results published, and this design document treats the publication as the deliverable rather than the review.
The category has earned that skepticism. Vendors in this market have claimed their tamper technology virtually eliminates false alerts while the field data ran at 95.50% of monthly GPS monitoring alerts in Cook County were false, up from 57.79% under radio frequency.[17] Children's tracking devices with tens of millions of units in the field were found funnelling through three shared backends [35]. Both were discovered by independent researchers publishing, and by nothing else.
What would count as this system failing
This site states nowhere what evidence would falsify it, and that is a defect. A design document that cannot be falsified is advocacy. Each block below names a claim this design document makes and the observation that would end it.
The core claim - that graduated escalation makes alerts actionable
The whole concept rests on treating alerting as a signal-to-noise problem rather than a notification problem.
Falsified if: the false-alert rate at the officer-queue tier does not fall materially below the fielded baseline in a pre-registered comparison. 95.50% of monthly GPS monitoring alerts in Cook County were false, up from 57.79% under radio frequency.[17] If corroboration, dwell and zone class do not move that, the engineering answer was wrong and the rest of the design is decoration on it.
The operational claim - that officers will open these alerts
Fewer alerts is only useful if the remaining ones get read. 80% of alerts were never checked by officers in a Tennessee audit.[29]
Falsified if: the response rate to tier-one and tier-two alerts is statistically indistinguishable from the audited baseline. That would mean the problem was never volume, and this design solved something that was not the obstacle.
The compliance-assist claim - that warning prevents entries
The most valuable addition in Part C is justified by prevented violations, not recorded ones.
Falsified if: the warned-and-turned-back ratio is at or near zero over a full pilot. A warning that nobody acts on is a notification with extra steps, and its legal argument goes with it - a scheme cannot claim to be forward-looking and regulatory on the strength of a feature that changes no behavior.
The coverage claim - that anchor density is achievable
The coverage and cost model says the anchor count for a metropolitan area lands in the low thousands at plausible site densities.
Falsified if: a real site survey in a real jurisdiction returns a Class A site count an order of magnitude above the model, or an installed cost per anchor several times the assumed figure. Either result puts the program outside the monitoring line it would have to be funded from, and the assessment's judgment applies: the project does not work.
The privacy claim - that no location database exists
This is the structural property that lets the same proposal be defended to a child-safety committee and to a civil-liberties organization.
Falsified if: an identity is resolved outside the split-custody path even once, or an aggregate telemetry stream is found to be reconstructible into a movement trace. Not weakened - falsified. The claim is absolute, so a single counterexample ends it, and the audit record is designed so that a counterexample would be visible rather than deniable.
The safety claim - that the audible tier does not de-anonymize
The top tier makes a noise in a public place, and the threat model puts a number on what that costs.
Falsified if: a documented instance occurs of a monitored person being identified, confronted or harmed as a consequence of an alert. One is enough. The gates in front of the audible tier exist precisely because this outcome is not recoverable.
The measurability claim - that the privacy design is compatible with evaluation
This is B-4, and it is the failure mode with the longest fuse.
Falsified if: at the first scheduled analysis the pre-registered primary outcomes cannot be computed from the aggregate telemetry that was specified. If that happens the correct response is to stop, not to add collection - because a program that has to build a location database in order to prove it does not need one has already lost the argument it was built to win.
What is deliberately not on this list
A null result on new sexual offending is not a falsification of anything on this site, and it must not be offered as one. This design document does not claim an effect on reoffending, the published evidence does not support one [25], and the arithmetic above says no pilot could establish one either way. Treating an underpowered null as a refutation would be the same error as treating an underpowered favorable result as a vindication - and the second error is the one this category keeps making.
The honest position is the narrow one the assessment closes on. As a system that prevents child sexual abuse, this fails on contact with the evidence. As a system that makes existing monitoring alerts actionable, notifies the people actually present, and does it without building a location database, it is defensible and buildable inside current law. The falsification criteria above are for the second claim, because it is the only one this design document makes.
Karsten & West, Brookings (2017)
Zgoba & Mitchell (2021), Journal of Experimental Criminology. 18 studies, 474,640 individuals
A meta-analysis of 18 studies covering 474,640 people.
How this works
0:59 · narrated · captioned · nothing here is sent anywhere when you press play. Prefer to read it? The transcript is below.
Transcript
| At | Section | What it says |
|---|---|---|
| 0:02 | File it first | Every study in this literature is retrospective and non-randomised. An evaluation pre-registered before deployment would be the best evidence in the field on day one. |
| 0:12 | What gets filed | Name the primary outcomes and the comparison group before deployment. File the analysis plan, pre-specify the stopping rules, and commit to publishing whichever way it goes. |
| 0:23 | Outcomes it can move | The primary outcomes are the ones the system has a mechanism for. False alerts by tier, officer response, absconding, registration compliance, and warned and turned back. |
| 0:34 | Not reoffending | Reoffending is reported and never relied on. At the sizes a jurisdiction would fund, the arithmetic on the page shows no detectable change either way, and the calculator runs it. |
| 0:45 | What failure looks like | The page also says what would count as this failing. If the false alert rate does not fall below the fielded baseline, the engineering answer was wrong and the rest is decoration. |