Pipeline

Every message goes through the same stages. Each stage is recorded in the ingest result under stages, with timing and a short note.

  1. Parse. The raw RFC 822 message is parsed: headers, addresses, subject, date, text and HTML parts, attachments. Authentication-Results headers give SPF, DKIM and DMARC outcomes.
  2. Sanitize. HTML is rendered to the text a person would see. Text that was hidden (display none, zero font size, foreground equal to background, hidden attribute) is kept apart and never enters extraction. Scripts and external resources are noted as flags. Attachments pass an allowlist (documents, images, plain data); executables, macro-enabled office files, and archives are refused and the refusal recorded. PDFs yield a text layer under size and page caps. A From header that yields no address is flagged sender_unparsed and the sender fields are left empty. An HTML body whose tags arrived entity-escaped is flagged escaped_markup and read as text.
  3. Classify. Enabled buckets are evaluated in priority order; the first whose rule matches wins. The result names the exact predicate that decided. No match means unrouted and, by default, a task.
  4. Extract. If the bucket has a schema, each field's extractors run in order; the first that yields a value wins. Values are coerced to the field type and carry a confidence (1 for a clean deterministic match, lower when coercion fails) and a source (which text, and character offsets).
  5. Validate. Required, type, enum, pattern, min and max. Issues are listed, never silently dropped.
  6. Route. Policy decides: applied, task, or held. Fields marked requires_human always create a task when they carry a value, whatever the confidence.
  7. Deliver. Applied captures are delivered to the bucket's destinations, signed. The outcome is recorded.

Every state change is an event with an actor: system, rule, agent (a key), or user.

Reprocessing

Change a rule or an extractor, then reprocess_message re-runs stages 3 to 7 for a stored message against the current configuration. Earlier captures are kept, so you can compare.

Dry runs

dry_run_rule and dry_run_schema run a rule or a schema against stored messages, or an inline email, and return results without storing anything. dry_run_schema accepts expected values per message and reports pass or fail per field, which is how an agent builds a test set for a bucket.

This page as Markdown.