Skip to content

Allyanonimiser

Australian-focused PII detection and anonymization for the insurance industry with support for stream processing of very large files.

PyPI version Python Versions Tests Release Check License: MIT

Overview

Allyanonimiser detects and anonymizes personally identifiable information (PII) in text, with first-class support for Australian formats (TFN, ABN, Medicare, AU phone, etc.) and insurance-industry identifiers (policy numbers, claim references, vehicle rego, VIN).

What's new in v3.7.0

Detection release; see the changelog for details.

  • Much better recall on realistic insurance text — on a new held-out test set, fully masked entities rose from 68% to 86% (pattern-only) and from 74% to 85% (en_core_web_lg).
  • Identifiers — hyphenated TFN/ABN/ACN/Medicare, form-style labels ("Worker TFN for PAYG withholding:"), licences, two-letter passports, account numbers, and insurer policy/claim references in any format.
  • Dates and names — written and ISO dates; reversed, all-caps, initialled and accented names; names after titles, greetings, role labels and self-introductions.
  • Addresses and phones — unit/level prefixes, PO Box and Locked Bag addresses, "+61 ..." numbers, and mobiles read aloud.
  • Bug fixes — dates after words such as "Incident" or "Payment" were dropped; names ending in "id" or "ref" (Reid, Al-Rashid) could vanish; labelled names were hidden when spaCy missed them.
  • New AU challenge corpus — 480 synthetic documents with a held-out test split and per-variant results (bench/make_au_challenge.py).

Behaviour change: PERSON patterns you add yourself now run when spaCy is loaded; they were previously ignored.

v3.6.0 (prior)

Correctness release. Detection quality on the AU corpus is unchanged from v3.5.1; see the changelog for details.

  • Streaming works on real CSV files — stream_from_file() crashed on any real file in 3.5.x. It now reads ordered row batches with correct counts and row indices, and writes entity output incrementally.
  • Exported configuration reproduces behaviour — exports now include custom patterns, analyzer settings and priorities, and reload into identical detections in JSON or YAML.
  • Opt-in sensitive-output controls — minimal_output=True drops original values, source columns, caches and reports. require_spacy_model=True fails at start-up instead of silently running without NER.
  • State isolation fixes — cached results can no longer be altered by callers or earlier calls, cache limits are enforced, custom pattern scores are honoured, and duplicate DataFrame index labels keep their own rows' output.
  • Reports belong to their instance — instances no longer share one report. Pass a shared ReportingManager to combine them.
  • Offline detection evaluation — bench/eval_ally.py plus a pattern-only regression gate in the test suite. Detection quality on the AU corpus is unchanged from 3.5.1.

Compatibility: the stream extra needs Polars 1.37 or later, and streaming reads CSV columns as strings by default.

v3.5.1 (prior)

Bug-fix and precision release — AU bench unchanged from v3.5.0 (still beats openai/privacy-filter on 5 of 6 categories, see Benchmarks).

  • Per-call options no longer leak — analyze(active_entity_types=...) / min_score_threshold=... are now true per-call parameters instead of sticky setters that silently restricted every later call (also fixed in the DataFrame and stream processors). The explicit set_active_entity_types() / set_min_score_threshold() methods remain the persistent knobs.
  • analyze_batch() now matches analyze() exactly — the batch path previously skipped the PERSON/LOCATION/ORG false-positive filtering and entity-type mapping, so DataFrame/CSV processing missed the v3.5.0 precision improvements.
  • Label-context disambiguation — "TFN is 123 456 789" now resolves to AU_TFN instead of the bare 9-digit CRN shape; an explicit label immediately before a value beats shape priority and checksum validation, and the labelled regexes tolerate filler words (TFN is, ABN number:).
  • Span-containment absorption — DATE fragments inside DOB spans (the lg-model fragmentation), stray NUMBER triplets inside identifiers, and CRN-tail-of-ABN matches are absorbed by their containing span. Postcodes inside addresses are never absorbed (keep_postcode depends on them).

v3.5 (prior)

  • 5 new entity types loaded by default — PHONE_INTL, US_SSN, CREDIT_CARD (Luhn-validated), ISO_DATETIME, TIME — see International Patterns.
  • Validate-then-pick conflict resolution — the resolver walks candidates from highest priority down and returns the first that passes per-type validation.
  • PERSON precision overhauled — city / state-postcode / date-shape / acronym / label-word rejection and trailing-label trim. AU bench PERSON F1 0.836 → 0.954.

Key Features

  • Australian-focused PII: TFN (with checksum), ABN (with checksum), Medicare, AU_PHONE, driver's license, Centrelink CRN, passport, postcode
  • Insurance domain: policy numbers, claim references, vehicle registration, VIN
  • Flexible anonymization: replace, mask, redact, hash (SHA-256), age-bracket, consistent-replacement
  • Stream processing: memory-efficient chunked processing for very large files via Polars
  • DataFrame support: pandas with optional PyArrow backing; expand_acronyms wiring for preprocessing
  • Reporting: session-level statistics, entity histograms, Jupyter-native rendering

Quick example

from allyanonimiser import create_allyanonimiser

ally = create_allyanonimiser()  # defaults to en_core_web_sm

results = ally.analyze(
    "Customer John Smith (TFN: 123 456 782) called about policy POL-987654."
)
for r in results:
    print(f"{r.entity_type}: {r.text!r} (score={r.score:.2f})")

out = ally.anonymize(
    "Customer John Smith (TFN: 123 456 782) called about policy POL-987654.",
    operators={
        "PERSON": "replace",
        "AU_TFN": "mask",
        "INSURANCE_POLICY_NUMBER": "hash",
    },
)
print(out["text"])

Choosing a spaCy model

SPACY_MODEL_FAST (en_core_web_sm) SPACY_MODEL_ACCURATE (en_core_web_lg)
Default in v3.3+? yes no
Download size 44 MB 587 MB
Cold start ~0.5s 2–5s
Pattern detection (TFN, ABN, MEDICARE, AU_PHONE, EMAIL, dates) identical identical
PERSON / LOCATION / ORG recall medium high
Serverless friendliness (Azure Functions, Lambda) good poor

Opt into the accurate model when a missed name is expensive in your downstream workflow:

from allyanonimiser import create_allyanonimiser, SPACY_MODEL_ACCURATE

ally = create_allyanonimiser(spacy_model=SPACY_MODEL_ACCURATE)

Next steps

License

MIT — see LICENSE.