Data Migration

Map CSV, JSON, or database rows onto Observation records — a repeatable migration workflow

This guide turns an existing dataset — a CSV catalog, a legacy database, an API export — into valid DisclosureOS Observation records. The workflow is a loop: map your source fields, run validation, fix what fails, and rerun until clean.

Prerequisites

npm install @disclosureos/records @disclosureos/scoring @disclosureos/schema @disclosureos/cli

You also need a TypeScript runner. The examples below use tsx:

npm install -D tsx

The migration loop

Every migration follows the same cycle:

┌─────────────────────────────────────────────┐
│  1. Audit — understand your source schema   │
│  2. Map — write a row mapper function       │
│  3. Run — execute, write JSON to out/       │
│  4. Validate — disclosureos validate out/   │
│  5. Fix — update mapper, rerun from step 3  │
│  6. Measure — score what you produced       │
└─────────────────────────────────────────────┘

Below is each step in detail.

1. Audit your source

Know your source's reality before writing any mapping code:

Which fields are always present? Which are free text pretending to be structured?
How are dates encoded — and are "1947-07-08" dates actually day-precise, or catalog conventions?
Are coordinates real GPS fixes, or city-centroid lookups someone added later?
What is the source's own id scheme? You will need it for identifiers and dedup.

The answers determine your mapper's honesty, which matters more than its completeness.

2. Write a row mapper

One function from source row to ObservationInput, with guards at every enum boundary.

File structure

Use a consistent layout for every migration project:

my-migration/
  source.csv           # or source.json — your raw data
  migrate.ts           # the mapper script
  out/                 # validated output (one JSON file per record)
  quarantine.json      # rows that failed validation

The mapper

migrate.ts

import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
import { createObservation } from '@disclosureos/records/factories';
import { isObjectShape } from '@disclosureos/records/guards';
import type { Observation } from '@disclosureos/records';

// ------------------------------------------------------------------
// Define your source row shape
// ------------------------------------------------------------------
interface SourceRow {
  case_id: string;
  date: string;          // 'YYYY-MM-DD', sometimes 'YYYY-MM', sometimes 'unknown'
  city: string;
  state: string;
  lat?: number;
  lng?: number;
  shape?: string;
  duration_min?: number;
  summary?: string;
  witness_count?: number;
}

// ------------------------------------------------------------------
// Map one row to an Observation
// ------------------------------------------------------------------
function migrateRow(row: SourceRow): Observation {
  return createObservation(
    {
      temporal: mapTemporal(row.date, row.duration_min),
      location: {
        name: `${row.city}, ${row.state}`,
        country: 'United States',
        latitude: row.lat ?? 0,
        longitude: row.lng ?? 0,
        siteType: 'unknown',
        coordinatePrecision: row.lat ? 'locality' : 'region',
      },
      summary: row.summary,
      objectCharacteristics: isObjectShape(row.shape)
        ? { shape: row.shape }
        : undefined,
      witnesses: row.witness_count ? { count: row.witness_count } : undefined,
      dataSourceId: 'legacy-catalog-2019',
      extensions: {
        legacyCatalog: { rawRow: row.case_id, rawShape: row.shape },
      },
    },
    { id: `legacy-${row.case_id}`, status: 'draft' },
  );
}

Key decisions in that mapper:

Stable, prefixed ids — legacy-${case_id} keeps re-runs idempotent and provenance visible.
status: 'draft' — migrated records are not published records; review is a separate step.
dataSourceId + extensions — every record knows where it came from and carries the raw source for audit.
Unmappable enums degrade to undefined, never to a wrong enum value.

3. Map dates with their real precision

The temporal model exists so you do not have to lie. date is always a full YYYY-MM-DD anchor (machine-sortable); the real precision is declared in dateGranularity and dateCertainty:

import type { TemporalData } from '@disclosureos/records';

function mapTemporal(date: string, durationMin?: number): TemporalData {
  const base = durationMin ? { durationSeconds: durationMin * 60 } : {};

  if (/^\d{4}-\d{2}-\d{2}$/.test(date)) {
    return { ...base, date, dateCertainty: 'approximate' };
  }
  if (/^\d{4}-\d{2}$/.test(date)) {
    return { ...base, date: `${date}-01`, dateCertainty: 'approximate', dateGranularity: 'month' };
  }
  if (/^\d{4}$/.test(date)) {
    return { ...base, date: `${date}-01-01`, dateCertainty: 'estimated', dateGranularity: 'year' };
  }
  return { ...base, date: '1900-01-01', dateCertainty: 'unknown', dateGranularity: 'unknown' };
}

Reserve dateCertainty: 'exact' for records whose provenance actually establishes the date. Events spanning a period get a dateRange; dates known only relative to another event get a relativeDate.

4. Run the migration and write output

migrate.ts (continued)

// ------------------------------------------------------------------
// Load, migrate, quarantine, write
// ------------------------------------------------------------------
const raw: SourceRow[] = JSON.parse(readFileSync('source.json', 'utf-8'));

const migrated: Observation[] = [];
const quarantine: { row: SourceRow; error: string }[] = [];

for (const row of raw) {
  try {
    migrated.push(migrateRow(row));
  } catch (err) {
    quarantine.push({ row, error: String(err) });
  }
}

mkdirSync('out', { recursive: true });
for (const obs of migrated) {
  writeFileSync(join('out', `${obs.id}.json`), JSON.stringify(obs, null, 2));
}
writeFileSync('quarantine.json', JSON.stringify(quarantine, null, 2));

console.log(`migrated ${migrated.length}, quarantined ${quarantine.length}`);

Run it:

Terminal

npx tsx migrate.ts

5. Validate and fix

Validate the output files independently as a second gate:

Terminal

npx @disclosureos/cli validate ./out --recursive

For CI and scripted workflows, add --json for machine-readable output:

Terminal

npx @disclosureos/cli validate ./out --recursive --json

When validation reports errors:

Read the error paths and messages.
Update your mapper to handle the failing case.
Re-run from step 4.
Repeat until validate is clean.

Never drop failures silently — the quarantine list is your data-quality report.

6. Do not fabricate evaluation

A row that says explanation: "probably Venus" is not an origin claim by you — it is the source's assessment. Either carry it in extensions for later review, or record it as an attributed claim from the source:

import { createOriginClaim } from '@disclosureos/origins';

const origin = [
  createOriginClaim('1.1.1.1.2', 0.6, {
    rationale: 'Catalog disposition: "probably Venus".',
    evaluatedBy: 'legacy-catalog-2019',
    evaluatedAt: '2019-01-01T00:00:00Z',
  }),
];

The same goes for observables: migration produces records; claims come from evaluators. Resist the urge to bulk-generate observableAssessments from keyword matching — it poisons scoring for every downstream consumer.

7. Measure what you produced

import { getCompleteness } from '@disclosureos/scoring';

const coverage = migrated.map((o) => getCompleteness(o).percentage);
const mean = coverage.reduce((a, b) => a + b, 0) / coverage.length;
console.log(`mean completeness: ${mean.toFixed(1)}%`);

The missing-fields lists from getCompleteness tell you what a future enrichment pass should target.

Quick reference

Convention	Value
Source file	`source.csv` or `source.json`
Mapper script	`migrate.ts`
Output directory	`out/` (one `.json` per record)
Quarantine file	`quarantine.json`
Record id scheme	`{source-prefix}-{source-id}`
Initial status	`draft`

Checklist

Stable, source-prefixed ids; re-runs are idempotent
dataSourceId set; raw rows preserved in extensions
Date/coordinate precision honestly declared
Enum mappings guarded; unmappable values kept as text
Failures quarantined and reported, not dropped
Source dispositions attributed to the source, not to you
disclosureos validate clean on the output

Runnable examples

Two complete, working examples you can run locally:

JSON source — examples/migration-path.ts converts three JSON rows end-to-end:

Terminal

pnpm --filter @disclosureos/examples migration-path

CSV source — examples/migration-csv-path.ts parses a real CSV file with csv-parse and produces Observations:

Terminal

pnpm --filter @disclosureos/examples migration-csv-path

Data Migration

Map CSV, JSON, or database rows onto Observation records — a repeatable migration workflow

Prerequisites

npm install @disclosureos/records @disclosureos/scoring @disclosureos/schema @disclosureos/cli

You also need a TypeScript runner. The examples below use tsx:

npm install -D tsx

The migration loop

Every migration follows the same cycle:

┌─────────────────────────────────────────────┐
│  1. Audit — understand your source schema   │
│  2. Map — write a row mapper function       │
│  3. Run — execute, write JSON to out/       │
│  4. Validate — disclosureos validate out/   │
│  5. Fix — update mapper, rerun from step 3  │
│  6. Measure — score what you produced       │
└─────────────────────────────────────────────┘

Below is each step in detail.

1. Audit your source

Know your source's reality before writing any mapping code:

Which fields are always present? Which are free text pretending to be structured?
How are dates encoded — and are "1947-07-08" dates actually day-precise, or catalog conventions?
Are coordinates real GPS fixes, or city-centroid lookups someone added later?
What is the source's own id scheme? You will need it for identifiers and dedup.

The answers determine your mapper's honesty, which matters more than its completeness.

2. Write a row mapper

One function from source row to ObservationInput, with guards at every enum boundary.

File structure

Use a consistent layout for every migration project:

my-migration/
  source.csv           # or source.json — your raw data
  migrate.ts           # the mapper script
  out/                 # validated output (one JSON file per record)
  quarantine.json      # rows that failed validation

The mapper

migrate.ts

import { readFileSync, writeFileSync, mkdirSync } from 'node:fs';
import { join } from 'node:path';
import { createObservation } from '@disclosureos/records/factories';
import { isObjectShape } from '@disclosureos/records/guards';
import type { Observation } from '@disclosureos/records';

// ------------------------------------------------------------------
// Define your source row shape
// ------------------------------------------------------------------
interface SourceRow {
  case_id: string;
  date: string;          // 'YYYY-MM-DD', sometimes 'YYYY-MM', sometimes 'unknown'
  city: string;
  state: string;
  lat?: number;
  lng?: number;
  shape?: string;
  duration_min?: number;
  summary?: string;
  witness_count?: number;
}

// ------------------------------------------------------------------
// Map one row to an Observation
// ------------------------------------------------------------------
function migrateRow(row: SourceRow): Observation {
  return createObservation(
    {
      temporal: mapTemporal(row.date, row.duration_min),
      location: {
        name: `${row.city}, ${row.state}`,
        country: 'United States',
        latitude: row.lat ?? 0,
        longitude: row.lng ?? 0,
        siteType: 'unknown',
        coordinatePrecision: row.lat ? 'locality' : 'region',
      },
      summary: row.summary,
      objectCharacteristics: isObjectShape(row.shape)
        ? { shape: row.shape }
        : undefined,
      witnesses: row.witness_count ? { count: row.witness_count } : undefined,
      dataSourceId: 'legacy-catalog-2019',
      extensions: {
        legacyCatalog: { rawRow: row.case_id, rawShape: row.shape },
      },
    },
    { id: `legacy-${row.case_id}`, status: 'draft' },
  );
}

Key decisions in that mapper:

Stable, prefixed ids — legacy-${case_id} keeps re-runs idempotent and provenance visible.
status: 'draft' — migrated records are not published records; review is a separate step.
dataSourceId + extensions — every record knows where it came from and carries the raw source for audit.
Unmappable enums degrade to undefined, never to a wrong enum value.

3. Map dates with their real precision

The temporal model exists so you do not have to lie. date is always a full YYYY-MM-DD anchor (machine-sortable); the real precision is declared in dateGranularity and dateCertainty:

import type { TemporalData } from '@disclosureos/records';

function mapTemporal(date: string, durationMin?: number): TemporalData {
  const base = durationMin ? { durationSeconds: durationMin * 60 } : {};

  if (/^\d{4}-\d{2}-\d{2}$/.test(date)) {
    return { ...base, date, dateCertainty: 'approximate' };
  }
  if (/^\d{4}-\d{2}$/.test(date)) {
    return { ...base, date: `${date}-01`, dateCertainty: 'approximate', dateGranularity: 'month' };
  }
  if (/^\d{4}$/.test(date)) {
    return { ...base, date: `${date}-01-01`, dateCertainty: 'estimated', dateGranularity: 'year' };
  }
  return { ...base, date: '1900-01-01', dateCertainty: 'unknown', dateGranularity: 'unknown' };
}

4. Run the migration and write output

migrate.ts (continued)

// ------------------------------------------------------------------
// Load, migrate, quarantine, write
// ------------------------------------------------------------------
const raw: SourceRow[] = JSON.parse(readFileSync('source.json', 'utf-8'));

const migrated: Observation[] = [];
const quarantine: { row: SourceRow; error: string }[] = [];

for (const row of raw) {
  try {
    migrated.push(migrateRow(row));
  } catch (err) {
    quarantine.push({ row, error: String(err) });
  }
}

mkdirSync('out', { recursive: true });
for (const obs of migrated) {
  writeFileSync(join('out', `${obs.id}.json`), JSON.stringify(obs, null, 2));
}
writeFileSync('quarantine.json', JSON.stringify(quarantine, null, 2));

console.log(`migrated ${migrated.length}, quarantined ${quarantine.length}`);

Run it:

Terminal

npx tsx migrate.ts

5. Validate and fix

Validate the output files independently as a second gate:

Terminal

npx @disclosureos/cli validate ./out --recursive

For CI and scripted workflows, add --json for machine-readable output:

Terminal

npx @disclosureos/cli validate ./out --recursive --json

When validation reports errors:

Read the error paths and messages.
Update your mapper to handle the failing case.
Re-run from step 4.
Repeat until validate is clean.

Never drop failures silently — the quarantine list is your data-quality report.

6. Do not fabricate evaluation

import { createOriginClaim } from '@disclosureos/origins';

const origin = [
  createOriginClaim('1.1.1.1.2', 0.6, {
    rationale: 'Catalog disposition: "probably Venus".',
    evaluatedBy: 'legacy-catalog-2019',
    evaluatedAt: '2019-01-01T00:00:00Z',
  }),
];

7. Measure what you produced

import { getCompleteness } from '@disclosureos/scoring';

const coverage = migrated.map((o) => getCompleteness(o).percentage);
const mean = coverage.reduce((a, b) => a + b, 0) / coverage.length;
console.log(`mean completeness: ${mean.toFixed(1)}%`);

The missing-fields lists from getCompleteness tell you what a future enrichment pass should target.

Quick reference

Convention	Value
Source file	`source.csv` or `source.json`
Mapper script	`migrate.ts`
Output directory	`out/` (one `.json` per record)
Quarantine file	`quarantine.json`
Record id scheme	`{source-prefix}-{source-id}`
Initial status	`draft`

Checklist

Stable, source-prefixed ids; re-runs are idempotent
dataSourceId set; raw rows preserved in extensions
Date/coordinate precision honestly declared
Enum mappings guarded; unmappable values kept as text
Failures quarantined and reported, not dropped
Source dispositions attributed to the source, not to you
disclosureos validate clean on the output

Runnable examples

Two complete, working examples you can run locally:

JSON source — examples/migration-path.ts converts three JSON rows end-to-end:

Terminal

pnpm --filter @disclosureos/examples migration-path

CSV source — examples/migration-csv-path.ts parses a real CSV file with csv-parse and produces Observations:

Terminal

pnpm --filter @disclosureos/examples migration-csv-path

Data Migration

Prerequisites

The migration loop

1. Audit your source

2. Write a row mapper

File structure

The mapper

3. Map dates with their real precision

4. Run the migration and write output

5. Validate and fix

6. Do not fabricate evaluation

7. Measure what you produced

Quick reference

Checklist

Runnable examples

See also

On this page

Data Migration

Prerequisites

The migration loop

1. Audit your source

2. Write a row mapper

File structure

The mapper

3. Map dates with their real precision

4. Run the migration and write output

5. Validate and fix

6. Do not fabricate evaluation

7. Measure what you produced

Quick reference

Checklist

Runnable examples

See also

On this page