Pipeline
Every message goes through the same stages. Each stage is recorded in the ingest result under stages, with timing and a short note.
- Parse. The raw RFC 822 message is parsed: headers, addresses, subject, date, text and HTML parts, attachments. Authentication-Results headers give SPF, DKIM and DMARC outcomes.
- Sanitize. HTML is rendered to the text a person would see. Text that was hidden (display none, visibility hidden, opacity zero, zero font size, foreground equal to background or transparent, a box collapsed to nothing with its overflow hidden, text pushed off the page or clipped away, Outlook's mso-hide, the hidden attribute) is kept apart under
hidden_textand never enters extraction; script, style and comment contents and image alt text are kept apart underremoved. Delimited attachments whose cells would open as formulas in a spreadsheet are flaggedcsv_formulaand kept as text. Scripts and external resources are noted as flags. Attachments pass an allowlist (documents, images, plain data); executables, macro-enabled office files, and archives are refused and the refusal recorded. PDFs yield a text layer under size and page caps. A From header that yields no address is flaggedsender_unparsedand the sender fields are left empty. An HTML body whose tags arrived entity-escaped is flaggedescaped_markupand read as text. - Classify. Enabled buckets are evaluated in priority order; the first whose rule matches wins. The result names the exact predicate that decided. No match means
unroutedand, by default, a task. - Extract. If the bucket has a schema, each field's extractors run in order; the first that yields a value wins. Values are coerced to the field type and carry a confidence (1 for a clean deterministic match, lower when coercion fails) and a source (which text, and character offsets).
- Validate. Required, type, enum, pattern, min and max. Issues are listed, never silently dropped.
- Route. Policy decides:
applied,task, orheld. Fields markedrequires_humanalways create a task when they carry a value, whatever the confidence. - Deliver. Applied captures are delivered to the bucket's destinations, signed. The outcome is recorded.
Every state change is an event with an actor: system, rule, agent (a key), or user.
Reprocessing
Change a rule or an extractor, then reprocess_message re-runs stages 3 to 7 for a retained message against the current configuration. Earlier captures are kept, so you can compare. A delivered message is deleted within the hour (Data); ingest_message the original again to reprocess it.
Dry runs
dry_run_rule and dry_run_schema run a rule or a schema against stored messages, or an inline email, and return results without storing anything. dry_run_schema accepts expected values per message and reports pass or fail per field, which is how an agent builds a test set for a bucket.
This page as Markdown.