Technical Tools for Automating Obituary Collection in Philadelphia Suburbs

The integration of these tools enables researchers, genealogists, and archivists to efficiently compile comprehensive obituary datasets while minimizing manual effort. Below are structured approaches for implementation, including technical specifications, workflow diagrams, and database schemas.

Python Libraries and APIs for Web Scraping Obituary Data

Legal Disclaimers and Rate-Limiting Techniques
Scraping public websites must comply with robots.txt directives, Terms of Service, and copyright laws (e.g., Fair Use exceptions for research). Always:

Recommended Python Libraries for Scraping

Python libraries for scraping must handle:

Static HTML (BeautifulSoup, lxml)

Dynamic JavaScript content (Selenium, Playwright)

API-based data (Requests, aiohttp)

Large-scale crawling (Scrapy)

• Newspaper3k
  A high-level library for extracting articles from URLs, including metadata (title, author, publish date). Ideal for newspaper archives like The Inquirer or The Times Herald-Record.
  • Example Use Case:
    from newspaper import Article
url = "https://www.phillytrib.com/obituaries/..."
article = Article(url)
article.download()
article.parse()
print(article.title, article.publish_date)

  • Limitations: Struggles with JavaScript-heavy sites; requires manual adjustments for non-standard HTML structures.
• Scrapy
  A full-fledged framework for large-scale scraping, supporting item pipelines for data cleaning and storage. Suitable for scraping multiple funeral home websites (e.g., Dignity Memorial, Landmark Memorial).
  • Key Features:
    • Built-in request throttling via `DOWNLOAD_DELAY`.
    • Item pipelines for processing scraped data (e.g., deduplication, NLP tagging).
    • Middleware for handling JavaScript (e.g., `scrapy-splash`).
  • Example Pipeline for Obituaries:
    class ObituaryPipeline:
    def process_item(self, item, spider):
    

    Extract names using regex (e.g., "[A-Z][a-z]+ [A-Z][a-z]+")

    item['name'] = re.search(r'\b[A-Z][a-z]+ [A-Z][a-z]+\b', item['text'])
    return item
    
• BeautifulSoup (with Requests)
  Lightweight solution for parsing static HTML, often used in tandem with `lxml` for speed. Example:
  from bs4 import BeautifulSoup
import requests
response = requests.get("https://www.legacy.com/obituaries/...")
soup = BeautifulSoup(response.text, 'lxml')
obituaries = soup.find_all('div', class_='obituary-entry')

• Selenium/Playwright
  Required for sites relying on JavaScript (e.g., interactive obituary filters). Playwright is preferred for headless browsing:
  from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch()
page = browser.new_page()
page.goto("https://www.legacy.com/obituaries/...")
obituary_text = page.inner_text('div.obituary-content')

• APIs for Structured Data
  Some funeral homes (e.g., Social Security Death Index, Find a Grave) offer APIs. Example using `requests`:
  import requests
response = requests.get(
"https://api.findagrave.com/api",
params={'name': 'Smith', 'state': 'PA', 'apikey': 'YOUR_KEY'}
)
data = response.json()


Workflow Diagram: Semi-Automated Obituary Collection System

Workflow Stages:
1. Data Acquisition

Scrape target websites (newspapers, funeral homes) using Scrapy/Newspaper3k.

Schedule crawls via `cron` or `APScheduler` (e.g., daily at 3 AM).

2. Data Processing

Clean HTML/text (remove ads, boilerplate) using `BeautifulSoup` or `goose3`.

Extract metadata (dates, names) via regex/NLP (detailed in subsequent section).

3. Alerting System

Send email/SMS alerts for new obituaries matching keywords (e.g., "Philadelphia suburbs") via `smtplib` or Twilio.

Example alert trigger:

if "Montgomery County" in obituary_text and "2023" in obituary_text:
send_alert(obituary_text, recipient_email)
4. Database Storage

Insert processed data into SQLite/MySQL (structured in next section).

Use `SQLAlchemy` for ORM or raw `sqlite3` for lightweight setups.

5. Periodic Maintenance

Deduplicate entries (e.g., same name across sources).

Archive old data to cold storage (e.g., AWS S3).

┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Web Scraping │───▶│ Data Cleaning │───▶│ Metadata │
│ (Scrapy/ │ │ (BeautifulSoup) │ │ Extraction │
│ Newspaper3k) │ └─────────────────┘ │ (NLTK/spaCy) │
└─────────────────┘ ┌─────────────────┴────┘ └─────────────┬─┐
▲ │ │
│ ▼ ▼
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
│ Email Alerts │◀───┤ Database │◀───┤ Deduplication │
│ (Twilio/smtplib)│ │ (SQLite/MySQL) │ │ (Fuzzy Matching)│
└─────────────────┘ └─────────────────┘ └─────────────────┘

Key Components:

Database Setup for Obituary Metadata Storage