Full-Text Ingestion Pipeline - Philippine Congress Data Analysis

The API currently exposes bill metadata and source/download links, not extracted bill body text. The ingestion pipeline in scripts/ingest_full_text.py turns those links into analysis-ready local datasets.

Inputs¶

Open Congress API endpoint: /api/documents
Optional filters: congress, type
Source files: download_url_sources from each bill record

The pipeline uses the hosted API by default: https://open-congress-api.bettergov.ph/api.

Override the API URL with:

export OPEN_CONGRESS_API_BASE_URL=https://open-congress-api.bettergov.ph/api

Run¶

Metadata only:

python -m scripts.ingest_full_text --metadata-only

Small smoke test:

python -m scripts.ingest_full_text --congress 20 --limit 25

Full ingestion:

python -m scripts.ingest_full_text

Useful filters:

python -m scripts.ingest_full_text --congress 20 --type HB
python -m scripts.ingest_full_text --congress 19 --type SB --force

Outputs¶

All generated files are written under data/ingested/, which is ignored by git.

Path	Purpose
`raw/api/documents.jsonl`	Raw API bill metadata
`raw/pdf/*.pdf`	Downloaded source PDFs
`processed/text/*.txt`	Extracted text per source PDF
`processed/tables/bill_metadata.csv`	Flattened bill metadata
`processed/tables/bill_texts.csv`	Metadata plus `full_text` and `analysis_text`
`processed/tables/bill_texts.jsonl`	JSONL version for downstream NLP
`processed/tables/bill_texts.parquet`	Parquet version when `pyarrow` is available
`manifests/downloads.csv`	Download status, content type, byte size, hash, errors
`manifests/text_extraction.csv`	Extraction status, page count, character count, errors
`manifests/summary.json`	Run summary

Robustness Features¶

Resumable by default: existing PDFs and text files are reused.
--force re-downloads and re-extracts files.
HTTP retries for rate limits and transient server errors.
Atomic writes for JSONL, CSV, PDF, and text outputs.
SHA-256 hashes for downloaded PDFs.
Per-file manifests capture failures without stopping the whole run.
analysis_text falls back to title, abstract, subjects, and raw authors when full text is unavailable.

Limitations¶

The pipeline extracts text from PDFs with pypdf. Scanned/image-only PDFs may produce little or no text.
OCR is not included by default because it requires system dependencies such as Tesseract and Poppler.
Download links may expire, redirect, or block automated access. The manifest is the source of truth for what succeeded.

Recommended Next Step¶

Add an OCR fallback for records where has_full_text = false but a PDF was downloaded successfully. A practical stack is ocrmypdf plus Tesseract for OCR and pypdf for final text extraction.