Pipeline

Every message goes through the same stages. Each stage is recorded in the ingest result under stages, with timing and a short note.

  1. Parse. The raw RFC 822 message is parsed: headers, addresses, subject, date, text and HTML parts, attachments. Authentication-Results headers give SPF, DKIM and DMARC outcomes.
  2. Sanitize. HTML is rendered to the text a person would see. Text that was hidden (display none, visibility hidden, opacity zero, zero font size, foreground equal to background or transparent, a box collapsed to nothing with its overflow hidden, text pushed off the page or clipped away, Outlook's mso-hide, the hidden attribute) is kept apart under hidden_text and never enters extraction; script, style and comment contents and image alt text are kept apart under removed. Delimited attachments whose cells would open as formulas in a spreadsheet are flagged csv_formula and kept as text. Scripts and external resources are noted as flags. Attachments pass an allowlist (documents, images, plain data); executables, macro-enabled office files, and archives are refused and the refusal recorded. PDFs yield a text layer under size and page caps. A From header that yields no address is flagged sender_unparsed and the sender fields are left empty. An HTML body whose tags arrived entity-escaped is flagged escaped_markup and read as text.
  3. Classify. Enabled buckets are evaluated in priority order; the first whose rule matches wins. The result names the exact predicate that decided. No match means unrouted and, by default, a task.
  4. Extract. If the bucket has a schema, each field's extractors run in order; the first that yields a value wins. Values are coerced to the field type and carry a confidence (1 for a clean deterministic match, lower when coercion fails) and a source (which text, and character offsets).
  5. Validate. Required, type, enum, pattern, min and max. Issues are listed, never silently dropped.
  6. Route. Policy decides: applied, task, or held. Fields marked requires_human always create a task when they carry a value, whatever the confidence.
  7. Deliver. Applied captures are delivered to the bucket's destinations, signed. The outcome is recorded.

Every state change is an event with an actor: system, rule, agent (a key), or user.

Reprocessing

Change a rule or an extractor, then reprocess_message re-runs stages 3 to 7 for a retained message against the current configuration. Earlier captures are kept, so you can compare. A delivered message is deleted within the hour (Data); ingest_message the original again to reprocess it.

Dry runs

dry_run_rule and dry_run_schema run a rule or a schema against stored messages, or an inline email, and return results without storing anything. dry_run_schema accepts expected values per message and reports pass or fail per field, which is how an agent builds a test set for a bucket.

This page as Markdown.