Scoring & checks

A score answers a question under a named method. A check tells you whether specified requirements were met. Keep both separate from a researcher's conclusion about an observation.

Choose the result you need

Use Claims for attributed research conclusions. Use a score when you have a defined scoring method. A documentation checklist does not need a numerical grade, and a conforming score can remain unavailable.

Open the field reference → for policy, result and evaluation-request fields. For the public explanation, see Scoring.

Before you run

Published in npm 2.0.0-beta.4. Provider scoring, validation summaries and evaluation requests are included in this release. Document versions and scoring-method versions remain separate from the npm package version.

You need Node.js 22, npm and a ZIP extractor. Create a project and install the published APIs:

mkdir disclosureos-scoring-beta4
cd disclosureos-scoring-beta4
npm init -y
npm install --save-exact @disclosureos/scoring@2.0.0-beta.4 @disclosureos/records@2.0.0-beta.4
curl --fail --location https://os.disclosure.org/examples/provider-scoring.zip --output provider-scoring.zip
unzip provider-scoring.zip

1. Reproduce the example

The download contains the exact policies, source files and worksheets shown on the portal. Its filenames use Meridian for the fictional DisclosureOS example case. No physical collection or laboratory work occurred.

node provider-scoring/verify.mjs

This verifier checks all 21 retained files, the record and case links, four results and three comparisons. It also runs the example's known local calculation. It does not execute arbitrary provider code or decide whether the authored ratings are justified.

The documentation method gives 25 points to source traceability, 25 to specimen identity, 30 to method detail and 20 to stated limits. Each reviewer assigns 0, 0.5 or 1 for each criterion. Team A gives method detail half credit; team B gives it zero. The totals follow from those judgments, not from software independently evaluating the laboratory work.

2. Check and compare the exchanges

Download the walkthrough checker into your npm project:

curl --fail --location \
  https://os.disclosure.org/examples/v2/scoring/check-scoring.mjs \
  --output check-scoring.mjs
node check-scoring.mjs provider-scoring

The checker uses evaluateProviderScore for each result and compareProviderScores for each pair. It supplies exact policy bytes and separate maps for common inputs, provider derivation and method artifacts. Map keys are the declarations' ref values, not file paths. These APIs do not fetch files or run the method.

const checked = await evaluateProviderScore(document, {
  policyBytes,
  inputFiles,
  derivationFiles,
  methodFiles,
});

The display uses null for the unfinished review's value. Its retained result document omits value entirely. Both the available and unavailable documents pass validation; only the two complete documentation reviews are eligible for comparison.

Interpret the result

A provider-score evaluation returns three check stages, plus explicit limits:

Read success together with these stages. policyExecution, providerAuthentication, scientificValidity and subjectResolution remain not_checked. The result can contain parsed declarations even when validation fails; do not present them as verified scores.

Comparison additionally requires scored outcomes, the same exact policy, subject, input scope and common source-file references, media types and byte pins. Input ordering is ignored. Each provider's derivation is checked separately and can differ. A shared display scale alone does not make scores comparable.

• Same method: 85 and 70 are eligible for inspection together. The helper supplies no winner, average or consensus.
• Different method: 85 and 75 are ineligible with different_policy.
• Unfinished review: the pair is ineligible with unavailable_result and incomplete_inputs.

Shared files do not become independent corroboration because different providers scored them. statisticalIndependence remains not_checked.

3. Try three failure cases

Each command changes only the checker's in-memory inputs for team A. It exits with status 1 and prints the failed or unchecked stages. Retained files stay unchanged.

node check-scoring.mjs provider-scoring --missing-input

The checker withholds one common source file. Structure and semantics pass; external stays not_checked, with PROVIDER_SCORE.MISSING_FILE. A URL or digest is not a substitute for the bytes.

node check-scoring.mjs provider-scoring --changed-input

One byte changes without updating its pin. external fails with PROVIDER_SCORE.PIN_MISMATCH; the file still has the same length.

node check-scoring.mjs provider-scoring --wrong-grade

The result declares grade A for its unchanged score of 85. File checks pass, but semantic fails with PROVIDER_SCORE.GRADE. This method assigns B to that score.

Run the normal command again to recover; no file restoration is needed. A fully verified result still does not prove that its provider applied the method correctly. That requires the separate arithmetic reproduction and a review of the judgments.

Provider scores

Create a policy before exchanging a result. Define its question, applicable subject kinds, method artifacts, scale, missing-data handling and limits. Then retain each provider's result against that exact serialized policy.

Keep subject, sources and derivation separate

• subject identifies what is scored. The generic evaluator does not resolve this ID against a record.
• inputScope states which part was reviewed. Include the intended subject snapshot in inputs when the score concerns that snapshot.
• inputs holds the common source basis; derivation holds provider-specific worksheets. Different worksheets do not imply different observations.
• provider and producedAt are declarations, not authenticated identity or a trusted timestamp.

A result cannot silently change the policy. Preserve its ID, version, digest and length. Publish a new policy revision when the method or grade bands change.

Keep unavailable distinct from zero

A policy can withhold a score when required information is missing or explicitly allow partial results. Its method must explain the denominator and arithmetic. The comparison helper declines results with declared missing inputs, including permitted partial scores.

An unavailable result gives missing_inputs, not_applicable, undetermined or not_calculated and an explanation. A completed criterion can legitimately receive zero; an unrecorded judgment cannot be replaced with zero.

Comparing provider results

Call compareProviderScores(leftResult, rightResult, { left: leftOptions, right: rightOptions }). Retain its explicit reasons rather than combining scores anyway. The helper checks eligibility to inspect declared results on a shared basis, not statistical comparability or methodological validity.

Reproduce the teaching example

The provider comparison uses the same retained policies, worksheets and seven-file source scope as this walkthrough. Read the documentation policy and first worksheet to see where the displayed points originate. The canonical verifier and this guide's checker serve separate purposes: arithmetic reproduction and exchange validation.

Levels, grades and rankings

Numeric grades belong to a policy

providerScoreGrade(policy, value) finds a declared band or ordinal level. It does not calculate the score. Numeric grade bands are optional (the required grades array may be empty). When present, they cover the whole scale in order without gaps or overlaps.

Lower boundaries are inclusive. Upper boundaries are exclusive except the last maximum. These A–D grades belong only to the fictional documentation method. They are not universal DisclosureOS grades or the Disclosure Index application's A–F policy.

Ordinal levels are named categories

An ordinal policy supplies at least two stable level IDs, labels and descriptions, with an ordering direction. Its scored value is a level ID rather than a number. Do not infer numeric distances between levels. A numeric scale's unit and direction are also explicit; 0–100 points is not automatically a probability or percentage.

Support, outcomes and ranks answer other questions

Observable support labels (not_indicated, reported, documented, measured, confirmed) describe available support for a characteristic. They are distinct from a V2 claim's assessed outcome and cannot be converted automatically into anomaly strength or origin confidence. not_indicated does not prove absence. The observable guide explains that boundary.

A rank requires a declared collection, inclusion rules and handling of ties. The exchange format supplies no universal ranking algorithm, and an ordinal level is not a rank within a collection.

Read a check result

Other evaluators expose different stages. Read each result in the scope of its named evaluator; do not assume every API has the three provider-score stages above.

passed means the named check passed within that scope. failed means an attempted check found a problem. not_checked means the evaluator did not establish a result. Preserve these states; do not replace an unchecked result with a pass or a zero.

Read success together with the checks and explicit limits. For example, a history evaluator can return success: true after checking structure and references while leaving its profile unrun. A receipt can preserve a failed result. Neither case establishes that a scientific conclusion is correct.

The saved example checks cover three different records: specimen reviews, a material-to-field relationship and a calculation record. All three retain profile: not_checked. The packet also records scorePolicy: not_applied. Its matching file hashes and reproduced arithmetic do not supply an overall case grade.

Profiles and completion

A profile selects requirements for a particular task. Choose one because it matches the document and purpose, not because it produces a favorable result.

These functions are exported from @disclosureos/schema/experimental/v2. Use the input contract accepted by the selected function. In particular, the assessment-documentation and related completion paths use the original experimental claim-history 0.1.0 contract; they are not universal adapters for every later history type.

Documentation checklist states are satisfied, missing, failed, not_checked and not_applicable. The selected instrument-research checklist uses the first four. Counts refer to requirement groups, not independent observations, percentages of scientific certainty or weighted points. A requirement that is not applicable should not count as missing.

A checklist can tell a developer to obtain the source bytes, document the measurement frame or supply a review report. It cannot create that information. Retain rejected and inconclusive reviews, unknown timing and unavailable calibration. Do not adjust a declaration merely to make a check pass.

The research result still leaves scientific eligibility, calibration adequacy, timing-model adequacy and reviewer authenticity unchecked. A declared accepted review is an input, not an independent validation of that review. See the instrument example for the difference between a matching measurement binding and completed scientific review.

The document, testimony and physical-sample profiles have different requirements. In those profiles, external: passed can mean that the selected source bytes match their declared digests. Release authenticity, speaker authenticity and physical custody remain separate questions. The provenance examples retain those limits alongside their passing checks.

Assessment summaries

summarizeClaimHistory is exported from @disclosureos/scoring/experimental/v2. It accepts the original claim_history schema version 0.1.0, whose subjects are an observation or a measurement. It does not summarize the later archival, laboratory, contextual or case-history contracts. Validate those documents with their corresponding evaluators and retain their attributed results.

The summary groups current declarations by topic and subject. It keeps source assertions, unassessed entries and assessed outcomes separate, retains superseded references and collapses exact duplicate declarations. When comparing duplicates, it excludes the claim ID, recording time and supersession links; assessment input order is also normalized. It does not treat repeated copies as additional votes or merge differently attributed reviews.

import { summarizeClaimHistory } from "@disclosureos/scoring/experimental/v2";

// The caller supplies a claim_history document with schemaVersion "0.1.0".
export function inspectDeclaredAssessments(history: unknown) {
  const result = summarizeClaimHistory(history);
  if (!result.success) {
    return { checks: result.checks, issues: result.issues };
  }

  return {
    policy: result.policy,
    groups: result.groups,
    sharedInputs: result.sharedInputs,
    supersededClaimRefs: result.supersededClaimRefs,
    scientificEligibility: result.scientificEligibility,
  };
}

declaredOutcomes lists the outcomes present in a group. multiple_declared_outcomes describes differing declarations; it does not decide which is correct. Shared input references and equal declared artifact hashes indicate possible dependence. They do not establish statistical independence or verify the source bytes. The summary leaves scientific eligibility, artifact integrity, vocabulary membership, statistical independence and reproducibility unchecked.

Observable terms describe reported or investigated characteristics. The Origin Classification System organizes possible explanations. Use a shared vocabulary binding to identify the exact catalog revision and term separately from the subject and supporting inputs. A valid binding establishes identity and membership, not the assessment's scientific support. See Claims and Origins.

Receipts and reproduction

Evaluation-request fields describe the input manifests, not their generated receipts. A receipt preserves the request and computed result of a fixed, versioned workflow. The available wrappers serve different purposes:

Each workflow pins its declared input files and dependency artifacts using SHA-256 and byte length. It records built-in rule identities and hashes exact serialized request and output strings. These wrappers run fixed package evaluators; they do not accept arbitrary executable policies or caller-supplied passing results. They do not fetch source URLs or execute supplied implementation artifacts.

The receipt has requestJson and outputJson strings plus request and output pins. Hash the exact UTF-8 strings without adding a newline. Preserve the manifest, exact input bytes, dependency artifacts, rule versions, requestJson, outputJson and their hashes. Hashing the receipt strings checks agreement with those saved strings. Re-running the matching workflow with the retained inputs checks reproduction. These are distinct operations. A failed profile can be faithfully reproduced; a successful replay is not a passing profile.

Receipts are unsigned. They do not authenticate an author, establish a trusted timestamp or attest that a supplied implementation or environment artifact was the one previously executed. A vocabulary file pin does not by itself check the meaning or membership of its terms. Read the wrapper and nested profile results together: a wrapper's external: passed may only establish that its declared files match their pins, while the nested profile still fails a source check.

The showcase's saved verification report is an application report over named canonical evaluator results and a separate arithmetic check. Do not label it as one of these receipt contracts or infer that the wrapper workflows ran. Its original rule scope and unchecked fields remain visible in the verification workspace.

Numerical scores

There is no universal V2 case score. The current packages retain numerical utilities with narrower meanings:

• getPopulationCoverage counts populated paths in the legacy record schema. It does not validate scientific quality, determine which fields apply or run a research profile. getCompleteness is an older name for the same counter.
• The retained score function accepts origin-only inputs in the legacy observation shape. It rejects any nonempty descriptive observable-assessment list. Descriptive support such as “reported” or “measured” is not an anomaly-strength input. Rejection is not a zero score or a finding that a characteristic is absent.
• The origin calculation uses attributed origin confidence, the configured prosaic prefixes, relative weights and optional evaluator weighting. Its range describes the spread of supplied claims under that method; it is not a statistical confidence interval or a probability that an origin is true.
• calibrationTrust supplies an optional weighting hook based on declared calibration status resolved through sensor references. It does not review calibration adequacy. The hook changes the point estimate, not the reported range or contestedness.

The retained default weights still include technology and biologics components, although the scorer leaves those contributions at zero. Consequently an origin-only result is not automatically normalized to the full output range. Preserve the exact configuration and scoringVersion; do not present the number as a calibrated measure of anomaly strength.

No replacement numerical method for descriptive observable assessments has been established. Applications can exchange their own results through the optional provider-scoring format described above. A provider's policy must state its purpose, applicable inputs, method, scale, missing-data treatment and limits. Conforming to the exchange format does not make that policy an endorsed DisclosureOS method.

Source-file example

The Scoring page uses the retained fictional weather file, observation and source selections to demonstrate a bounded byte check. It narrows a copy of the selection document to the existing temperature-meter layout and its first temperature row, preserving the instrument partition and measurement reference, then calls evaluateProductSourceSelection with product-reference keys in its assets map.

The altered copy exists only for this demonstration; it does not overwrite the retained source or its digest. The same file can support separate products for the two weather instruments without turning them into independent files. A source-byte pass establishes neither physical calibration nor the truth of the synthetic measurements.

Continue building

Run the full showcase to inspect the case, or prepare a dataset with explicit availability and validation scope. Keep the method, exact source versions and remaining checks beside every exported result.