Skip to content
Wasl al-Athar
Donate to the project

Preservation method

Repeatable, auditable steps and precise definitions for every figure we publish.

1. Catalogue captures through the CDX index

We query the Internet Archive’s CDX index with one worker and at least six seconds between requests, within declared limits for request count, data size and runtime. We paginate and save a checkpoint after each page so work can resume without duplication.

  • A 403 response or access denial means we stop and review manually; we do not try to bypass it.
  • For 429, 503 or a Temporarily Offline page, we wait with conservative backoff, then stop and record the result.
  • Every request is logged with its URL, status, size, duration and result.

2. What we count

We keep distinct measures separate:

  • Capture rows: stored CDX records; one capture is a URL and timestamp.
  • Distinct urlkeys: URLs normalized by the archive, including query-string variants.
  • Canonical items: one page path after normalizing protocol, www subdomain and port.
  • Distinct digests: content that differs according to the archive’s digest.
  • Extracted and published are different stages. Extraction is internal; publication requires permission.

Item identifiers such as /mat/107647 are identifiers, not counts. We do not infer an item count from them.

3. Select a capture for each URL

We assess each capture’s status, HTML type, size against the median and whether it predates the end of original content. We choose the newest sound capture, then fall back to an older one if extraction fails. Selection and exclusion reasons are recorded on each item page.

4. Extraction and safety

  • We preserve the raw response unchanged with its SHA-256 digest in a private content-addressed store.
  • A conservative extractor reads the title, author, section and text, and removes scripts, frames, styles and event handlers.
  • A capture date is not treated as a publication date. If a page gives no date, we say so.
  • Reader comments and forum sections are excluded because they are user-generated content.
  • Archived code is never executed on our domain.

5. Original text and edited editions

An extracted original is immutable; another extraction creates a new record. A scholarly edition is a separate layer with a version number and editorial note. It is published only after rights review and approval by two independent scholars other than its editor.

6. Storage and backups

Our 3-2-1-1-0 policy is a target: three verified copies, on two media, one off-site, one immutable or under separate administration, and zero errors in restore tests. We publish the actual state and do not treat replication as a backup.

7. Artificial intelligence

Automated tags, transcripts or summaries are suggestions awaiting human review. They are not attributed to the source or used to issue a fatwa or ruling.