Skip to content

Pipeline — Download

Module: src/lib/pipeline/download.py
Input: resolved pdf_url from a crawl row + caller-supplied doc_meta
Output: data/raw/<adapter>/pdf/<year>/<filename>.pdf + <filename>.meta.json

Design

Content and metadata have different change rates:

  • .pdf / .md — immutable once a bill is published; never changes.
  • .meta.json — mutable; status fields (passed_at, second_reading) update retroactively as the government updates the live site.

download_pdf() writes both on the first crawl. On re-crawl (cron), it refreshes only the doc_meta fields without re-downloading the PDF. crawl_date is always updated on every pass, which distinguishes a stale null (never crawled) from a confirmed null (crawled, field is genuinely absent).

API

from lib.pipeline.download import download_pdf, read_meta

meta = download_pdf(
    url,
    dest,                        # Path — where to write the PDF
    verify=True,                 # TLS verification
    cookies=None,                # session cookies for authenticated requests
    doc_meta=None,               # adapter-supplied admin fields (pre-filtered by clean_*_meta)
    timeout=60,
    headers=None,
)

Callers are responsible for pre-filtering doc_meta — use the adapter's clean_bill_meta() or clean_hansard_meta() helpers, which strip crawl-internal keys (pdf_url, source_tree_node_id, etc.) before passing the row to download_pdf().

.meta.json schema

{
  "url": "https://www.parlimen.gov.my/files/billindex/pdf/2024/DR 17 BI.pdf",
  "sha256": "abc123...",
  "bytes": 204800,
  "status": "ok",
  "crawl_date": "2026-05-26",

  "year": "2024",
  "dr_label": "D.R.17/2024",
  "summary": "Rang Undang-undang ...",
  "first_reading": "18 November 2024",
  "second_reading": "25 November 2024",
  "presented_by": "Menteri ...",
  "passed_at": ""
}

The top-level infrastructure fields (url, sha256, bytes, status, crawl_date) are always written by download_pdf(). Everything else is merged from doc_meta — these are the admin fields produced by the adapter's clean_*_meta() helper.

Idempotency

If the .pdf already exists, the download is skipped. Only doc_meta fields and crawl_date are refreshed in the existing .meta.json. This supports retroactive status updates (e.g. passed_at populated months after the bill was first downloaded) without re-fetching large PDFs.

Output layout

data/raw/parliament_my/pdf/2024/
  DR 17 BI.pdf
  DR 17 BI.meta.json
data/raw/parliament_my/pdf/2024/
  DN-16122025.pdf
  DN-16122025.meta.json

Adapter helpers — parliament_my

from lib.sources.my.parliament_my.parse import clean_bill_meta, clean_hansard_meta

These strip adapter-specific infra keys and keep only serialisable admin fields:

Helper Strips Keeps
clean_bill_meta(row) pdf_url, pdf_url_embedded, pdf_resolve_status, pdf_filename, source_tree_node_id, ajax_url, seed_url year, bill_item_id, summary, dr_label, first_reading, second_reading, presented_by, passed_at
clean_hansard_meta(row) pdf_url, pdf_filename, source_tree_node_id, seed_url house, sitting_date, sitting_date_text, parliament_no, penggal_no, mesyuarat_no, mesyuarat_text

Known edge cases

Case Handling
TLS failure Pass verify=False; the caller decides the fallback policy
Soft-200 HTML error page Detected by error_pages.py upstream; these rows have no valid pdf_url
Network timeout Returns {"status": "error", "error": "..."} — PDF not written
Session cookie required Pass cookies=bill_cookies from the crawl session
Re-crawl / cron Skip download; refresh doc_meta + crawl_date in existing .meta.json