What is the best way to create a district-wide edtech pilot program evaluation checklist in 2027?

The outcome you should expect

A district-wide edtech pilot evaluation checklist is not a satisfaction survey and it is not a procurement form. Done correctly, it produces a single defensible recommendation per tool — adopt, adopt with conditions, extend the pilot, or decline — supported by evidence a school board member could read in ten minutes and a technology director could defend under a public records request.

The concrete outcome you should expect from a well-built checklist is that the number of tools in active use across your district goes down, not up. Most districts that inventory their edtech for the first time find far more distinct applications touching student data than anyone in the central office believed existed — teacher-adopted free tools, building-level purchases, grant-funded platforms nobody renewed intentionally, and licenses that auto-renewed for years past their last real use. A checklist that only evaluates new candidates fails at this. The evaluation instrument has to be reusable against incumbents, because the most valuable decision it will ever produce is "stop paying for this."

The second outcome is speed with a paper trail. Before a checklist exists, the typical evaluation cycle looks like a vendor demo, a few enthusiastic teachers, a budget conversation, and a purchase. Afterward, the cycle looks like a standard intake form, a privacy and interoperability screen that either passes or kills the tool in days rather than months, a defined pilot window with pre-registered success measures, and a scored decision memo. The paperwork is heavier but the calendar is shorter, because the arguments happen once, at criteria-setting time, instead of repeatedly at each stage.

The third outcome is that you can say no without it being personal. The most common political failure in district edtech is that a declined tool reads as a rejection of the teacher or principal who championed it. A published, weighted rubric moves the conversation from "we didn't like your idea" to "it scored 2 out of 5 on interoperability and we can't roster it without manual CSV uploads every term." That is a fixable, factual objection — and often the vendor fixes it, which is a better outcome than either adoption or rejection.

Expect the checklist itself to be short at the point of use. The working instrument that a pilot teacher fills out should fit on two pages or roughly fifteen to twenty-five scored items. Everything else — the privacy review, the security questionnaire, the accessibility documentation, the cost model — belongs in a back-office packet completed by central staff, not by classroom educators. Districts that hand teachers a sixty-item instrument get sixty items of noise back.

Finally, expect the checklist to have a shelf life. Criteria that made sense before generative AI features shipped into every gradebook, assessment platform, and writing tool are no longer sufficient. Plan to revise the instrument annually, and treat the revision as a governance event with named approvers rather than a quiet document edit.

What drives that outcome

Four forces determine whether a district evaluation checklist produces real decisions or shelf-ware, and they are worth naming explicitly because each one has a specific countermeasure.

Pre-registration of success criteria. The single strongest driver is whether you wrote down what "success" means before the pilot ran. If the criteria are set afterward, the evaluation becomes a search for evidence that confirms the decision already made. Concretely: before day one of the pilot, the checklist should carry a filled-in line reading something like "we will consider this successful if at least 70% of enrolled teachers use it in a majority of instructional weeks, and if the benchmark assessment gap between pilot and comparison classrooms does not widen." Numbers can be modest. What matters is that they exist in writing, dated, before the data does.

Who holds the pen. Checklists authored solely by IT produce instruments heavy on security and integration and blind to instructional value. Checklists authored solely by curriculum produce the inverse. The instrument needs a named owner from teaching and learning, a named owner from technology, a named owner from special education or accessibility, a named owner from data privacy, and a finance reviewer for the cost model. Five signatures, not five committees.

Whether refusal is structurally possible. If your process has no defined path to "no" — no stage gate where a tool can be eliminated, no one empowered to eliminate it — the checklist is decorative. Build at least two hard gates: a privacy and interoperability screen that runs before any classroom pilot begins, and a go/no-go decision meeting with a calendar date set at pilot launch.

Teacher time cost as a first-class metric. Nearly every failed rollout is explained after the fact as "teachers didn't use it." Almost always the real cause is that the tool required time nobody budgeted — separate logins, manual rostering, duplicate gradebook entry, or a training model that assumed hours teachers did not have. Make time cost a scored, quantified domain: minutes to set up a class, minutes per week to maintain, whether it single-sign-on's, whether grades flow to the SIS automatically.

The diagram matters less than one property it encodes: the privacy and interoperability screens sit *before* the classroom pilot, not after. Districts routinely invert this, running an enthusiastic six-week pilot and then discovering the tool cannot sign a data privacy agreement or cannot roster from the SIS. At that point you have to take a beloved tool away from teachers, which costs far more political capital than declining it in week zero.

Benchmarks and realistic ranges

The following ranges reflect what is practical for a district-scale program rather than any single published standard — treat them as planning defaults to adjust against your own capacity.

Checklist length. Fifteen to twenty-five scored items in the teacher-facing instrument. Five to eight scored domains overall. Back-office packets (privacy questionnaire, security review, accessibility documentation) run longer, commonly thirty to sixty items, but are completed once by central staff per vendor, not per pilot site.

Pilot duration. Six to twelve weeks for a supplemental tool; a full semester or academic year for anything replacing a core curricular resource or an assessment system. Shorter than six weeks and you are measuring novelty; longer than a year without a decision gate and you have accidentally adopted the tool without deciding to.

Pilot scale. Three to six schools, or roughly five to fifteen percent of the relevant teacher population, chosen deliberately for variance rather than enthusiasm. A pilot composed only of volunteers at your best-resourced schools tells you almost nothing about district-wide viability. Include at least one high-poverty site, at least one school with weaker device or connectivity conditions, and at least one classroom with a meaningful population of students with IEPs and English learners.

Response rates. Expect fifty to seventy percent survey response from pilot teachers if you build the survey into a paid or contract-time obligation, and twenty to forty percent if it is voluntary. Below thirty percent, do not report the survey as a finding — report it as insufficient evidence and go get interviews instead. Six to ten structured teacher interviews usually surface more actionable detail than a low-response survey of a hundred.

Usage thresholds. A common practical bar is that a tool should reach active use by a majority of enrolled pilot teachers in a majority of instructional weeks. Tools that land under roughly forty percent weekly active teacher use during a supported pilot rarely improve after scaling, because the pilot is the period of maximum attention and support.

Cost modeling horizon. Model three years, not one. Include license cost at full district enrollment (not pilot pricing), implementation and integration labor, professional learning hours at your loaded hourly rate, ongoing support tickets, and the cost of the tool it replaces if any. Vendors often discount pilot pricing heavily; the checklist should require a written full-scale price before adoption, not after.

Scoring scale. Use a 0–4 or 1–5 scale with behaviorally anchored descriptions for each point, not a bare number. "3 = rosters automatically from the SIS but requires manual section mapping each term" is scorable by two different reviewers with similar results. "3 = adequate" is not.

Weighting. A defensible starting weight distribution for a supplemental instructional tool: instructional fit and evidence 30%, data privacy and security 20%, interoperability and rostering 15%, accessibility 15%, teacher time and usability 10%, total cost of ownership 10%. Adjust for context — an assessment platform should weight interoperability and privacy far higher — but publish the weights before scoring begins.

Decision timeline. From intake to decision, eight to sixteen weeks is realistic for a supplemental tool including a full pilot; a core resource adoption commonly runs a full year with board involvement. Publish the calendar at intake so vendors and champions both know the date.

Risks, edge cases, and failure modes

The rubric that only ratifies. The most frequent failure is a checklist filled out after the adoption decision has been socially made. Detect it by tracking your own outcome distribution: if in two years your process has never produced a "decline" on a tool that reached the pilot stage, the instrument is not doing evaluative work. Some districts address this by requiring the pilot report to state, in one sentence, what evidence would have changed the recommendation.

Confounded comparisons. Pilot teachers are volunteers, better supported, and observed. Any comparison to non-pilot classrooms is confounded by all three. Do not claim causal impact from a district pilot. Claim implementation feasibility, usage, teacher and student experience, and directional instructional signal — and say so explicitly in the report. Overclaiming here is how districts end up defending a growth number that does not survive the second year.

Privacy review as a rubber stamp. A signed data privacy agreement is necessary and insufficient. The checklist should require specifics: what student data elements are transmitted, whether the vendor uses data to train models, what subprocessors receive data, retention and deletion terms, breach notification timelines, and how a parent's deletion request is executed. Ask whether the vendor's AI features process student work, and whether that processing can be disabled district-wide rather than per-teacher. Confirm applicable obligations under FERPA, COPPA for students under thirteen, and any state student privacy statute — several states impose requirements stricter than federal law.

Accessibility deferred. Districts routinely treat accessibility as a post-adoption remediation item. That is both a legal exposure and an educational failure. Require a current accessibility conformance report from the vendor and, separately, run your own check with the assistive technology your students actually use — screen readers, switch access, magnification, captioning — on the two or three workflows students will use most. Vendor documentation and lived behavior diverge often enough that the independent check is not optional.

Integration debt. Every tool that does not roster automatically becomes recurring labor for someone, usually a school-level technology coordinator, every term forever. Score rostering and SSO strictly. A tool that needs a CSV upload per section per semester is a permanent tax, and the checklist should surface that tax in hours per year rather than as a checkbox.

AI feature drift. A tool evaluated in one school year may ship substantially different generative capabilities in the next without a new contract. Build a re-review trigger into the checklist and, ideally, into the contract: material changes to data processing or the addition of AI features that process student work require notification and re-evaluation. Without that clause, your evaluation is a snapshot of a product that no longer exists.

Champion departure. Roughly a third of pilots lose their primary champion — the instructional coach, the assistant principal, the one teacher who made it work — before the decision date. If the checklist has no named backup owner, the pilot quietly dies and the evaluation records nothing. Require two names per pilot site.

Sunk-cost renewal. The riskiest moment is not adoption; it is the second renewal, when the tool is embedded, nobody is measuring, and the invoice is routine. Set the annual re-evaluation date at the moment of adoption and put it on the same calendar as the budget cycle, so the question "is this still earning its line item?" is asked while there is still time to act on the answer.

Equity blind spots. A tool that performs well in pilot classrooms with strong devices and home connectivity can fail badly at scale. Score offline or low-bandwidth behavior, mobile usability, and language support explicitly, and require that pilot sites include the conditions where the tool is most likely to struggle.

A practical rollout plan

Treat building the instrument as its own small project with a six-to-eight week runway before the first pilot uses it.

Weeks 1–2: inventory and criteria. Pull the actual list of applications touching student data — from your SSO logs, your rostering platform, your accounts payable records, and building-level purchase cards. Reconcile the three; they will not match. In parallel, convene the five named owners and draft the domains and weights. Do not draft items yet. Agreeing on domains and weights first prevents the instrument from becoming a wish list.

Weeks 3–4: draft and anchor. Write fifteen to twenty-five teacher-facing items and the back-office packets. For every scored item, write the behavioral anchors for at least the top, middle, and bottom points of the scale. This is the tedious step everyone skips and it is what makes two reviewers agree.

Week 5: calibrate against a known tool. Score a tool you already use and have strong opinions about — ideally one you know is mediocre. Have three reviewers score it independently. If their totals diverge by more than roughly fifteen percent, your anchors are too vague. Rewrite the items that diverged most and re-score. This calibration pass is the highest-value hour in the entire build.

Week 6: pilot the instrument, not the product. Give the draft checklist to four or five teachers and ask how long it took and which items they could not answer honestly. Cut anything that takes a teacher more than fifteen minutes total or asks for information they do not have.

Weeks 7–8: publish and wire in. Put the checklist, the weights, the decision calendar, and the intake form on an internal page anyone in the district can reach. Publishing the weights before evaluations begin is what makes later decisions defensible. Wire the intake form to actually route — a form nobody monitors is worse than no form, because it creates the appearance of process.

Then run the first real cycle end to end on one tool, deliberately choosing something low-stakes, and hold a short retrospective on the process itself before you apply it to a high-visibility adoption.

Two governance details make this stick. First, put the decision memo template in the same folder as the checklist, so scoring flows directly into a one-page recommendation with the score table, the dissent if any, the conditions of adoption, and the renewal review date. Second, keep every completed evaluation, including declines. The archive becomes the fastest possible answer the next time a vendor returns — you already know what they scored and why, and the re-evaluation takes hours instead of weeks.

Related questions

How many tools should a district pilot at once?

Two to four concurrent pilots is manageable for most central teams. Beyond that, the same small group of staff supports every pilot, support quality degrades, and low usage gets misread as a product problem rather than a capacity problem.

Should teachers or IT own the evaluation checklist?

Neither alone. Instructional value and technical viability are both disqualifying if absent, so co-ownership with named leads in teaching and learning, technology, accessibility, privacy, and finance produces the only instrument that can legitimately say no on either axis.

Can the same checklist evaluate tools already in use?

Yes, and it should. Running incumbents through the identical rubric at renewal is where the checklist pays for itself — it surfaces duplicate functionality, unused licenses, and tools that no longer meet current privacy or accessibility expectations.

What evidence standard is realistic for a district pilot?

Implementation feasibility, usage, and experience data are credible. Causal learning-impact claims from a volunteer pilot are not. State the limitation in the report rather than letting a confounded comparison become the headline number.

How do you keep the checklist from becoming paperwork?

Cap the teacher-facing form at fifteen to twenty-five items and fifteen minutes, push everything else to central staff, and make sure the process has actually produced declines. An instrument that never changes an outcome will be filled out carelessly within a year.

FAQ

What belongs in the privacy section of a district edtech evaluation checklist?

Specifics, not attestations. List the exact student data elements transmitted, whether personally identifiable information leaves the district, subprocessor names, retention and deletion terms, breach notification timelines, whether student work is used for model training, and whether AI features can be disabled centrally. Confirm the signed agreement covers FERPA obligations, COPPA where students are under thirteen, and any applicable state student data privacy statute. A vendor unwilling to answer these in writing has answered the question.

How do you score interoperability without a technical background?

Ask four practical questions and score each: does it support single sign-on with our identity provider; does it roster automatically from our student information system; do grades or results flow back without manual entry; and if any answer is no, how many staff hours per term does the workaround cost? Converting integration gaps into an hours-per-year number makes the trade-off legible to non-technical reviewers and to finance.

Should the pilot include a comparison group?

Include one if it costs little — matched grade levels or courses not using the tool — but report it as context, not causation. Pilot classrooms differ from comparison classrooms in support, observation, and volunteer enthusiasm, and no district pilot design controls for that. The comparison is useful for spotting large negative signals, not for proving gains.

How often should the evaluation checklist itself be revised?

Annually, as a governance event with the same named approvers who built it. Product categories change fast enough that criteria written two years ago miss whole categories of risk, particularly around AI features processing student work. Version the document, date it, and record what changed and why, so past decisions remain interpretable against the criteria in force when they were made.

What is the minimum viable version if the district has no capacity for this?

One page: a privacy and rostering pass/fail gate, five weighted domains scored 1–5, a named owner per domain, a pre-registered usage target, and a dated go/no-go meeting. That version does most of the work of a mature process. Adding behavioral anchors and back-office packets improves consistency, but the gate, the weights, and the calendar date are what actually produce decisions.

How do you handle a tool teachers love that fails the checklist?

Name the specific failing criterion and its cost, and offer the vendor a path — most rostering and accessibility gaps are fixable and vendors fix them when a district purchase depends on it. If the failure is privacy-related and unfixable, decline plainly and help the champions find a comparable tool that passes. Transparency about which criterion failed is what keeps the process credible for the next request.

Sources

• https://studentprivacy.ed.gov/ — U.S. Department of Education Student Privacy Policy Office guidance on FERPA and school service providers
• https://www.ftc.gov/business-guidance/privacy-security/childrens-privacy — FTC guidance on COPPA compliance, including school-related consent
• https://www.w3.org/WAI/standards-guidelines/wcag/ — W3C Web Content Accessibility Guidelines, the reference standard for accessibility conformance
• https://ies.ed.gov/ncee/wwc/ — What Works Clearinghouse, evidence standards and reviews of education interventions
• https://www.imsglobal.org/ — 1EdTech (formerly IMS Global) standards for rostering, single sign-on, and learning data interoperability
• https://www.cosn.org/ — Consortium for School Networking, district technology leadership and privacy resources
• https://www.iste.org/ — ISTE standards and guidance for educational technology leadership
• https://www.ada.gov/resources/web-guidance/ — U.S. Department of Justice guidance on web accessibility obligations for public entities
• https://www.cisa.gov/topics/cybersecurity-best-practices — CISA cybersecurity best practices relevant to vendor security review

Related on PULSE

• How do you build a vendor security questionnaire that non-technical reviewers can actually score?
• What belongs in a three-year total cost of ownership model for a software purchase?
• How do you run a pilot program that produces a decision instead of a permanent trial?
• What does a defensible weighted scoring rubric look like for competitive vendor selection?
• How do you audit and consolidate an inventory of overlapping software licenses?
• How do you set pre-registered success criteria before a program launches?