Sources and Databases for Compiling Free Compliance Lists

Authoritative Sources for Free Compliance Data

• U.S. Government Publishing Office (GPO) – Official federal regulations and laws.

  API: Federal Digital System (FDsys)

• European Union Open Data Portal – EU directives, standards, and compliance frameworks.

  API: EU Open Data API

• World Health Organization (WHO)API: WHO Global Health Observatory (GHO)
• International Organization for Standardization (ISO) – ISO standards and compliance documentation.

  API: ISO Online Browsing Platform

• U.S. Environmental Protection Agency (EPA) – Environmental regulations and compliance data.

  API: EPA Data Finder

• International Labour Organization (ILO) – Labor laws and occupational safety standards.

  API: ILOSTAT Database

• Food and Agriculture Organization (FAO) – Food safety and agricultural compliance standards.

  API: FAOSTAT API

• OpenCorporates – Corporate compliance and registration data.

  API: OpenCorporates API (Free tier available)

• Global Reporting Initiative (GRI) – Sustainability reporting standards.

  API: GRI Database

• World Bank Open Data – Economic and regulatory compliance indicators.

  API: World Bank API

• GitHub (Open Compliance Repositories) – Publicly available compliance tools and datasets.

  Example: Open Compliance Projects

• Wikidata – Structured compliance-related metadata (e.g., regulatory bodies, standards).

  API: Wikidata Query Service

Extracting Structured Compliance Data from Open-Source Repositories

• Accessing APIs with `requests`:
  Most authoritative sources provide REST APIs for structured data retrieval. The `requests` library simplifies API interactions.

  import requests
  
  response = requests.get("https://api.europa.eu/data/v1/euodp/en/datasets")
  
  data = response.json()
  
  print(data["results"])  # Extract relevant fields
  
              

• Web Scraping with `BeautifulSoup`:
  For static or non-API-hosted data, `BeautifulSoup` extracts HTML content.

  from bs4 import BeautifulSoup
  
  import requests
  url = "https://www.iso.org/obp/ui/#iso:code:ISO+9001"
  
  response = requests.get(url)
  
  soup = BeautifulSoup(response.text, 'html.parser')
  
  tables = soup.find_all('table')
  
  for table in tables:
  
      rows = table.find_all('tr')
  
      for row in rows:
  
          cells = row.find_all('td')
  
          print([cell.text.strip() for cell in cells])
  
              

• Querying Wikidata with SPARQL:
  Wikidata’s structured metadata can be queried using SPARQL for compliance-related entities.

  from SPARQLWrapper import SPARQLWrapper, JSON
  sparql = SPARQLWrapper("https://query.wikidata.org/sparql")
  
  sparql.setQuery("""
  
      SELECT ?standard ?label WHERE {
  
          ?standard wdt:P31 wd:Q12426273.  # Instance of "standard"
  
          SERVICE wikibase:label { bd:serviceParam wikibase:language "[AUTO_LANGUAGE],en". }
  
      }
  
  """)
  
  sparql.setReturnFormat(JSON)
  
  results = sparql.query().convert()
  
  for result in results["results"]["bindings"]:
  
      print(result["label"]["value"])
  
              

• Storing Extracted Data:
  Use Pandas to organize scraped data into a structured format for further processing.

  import pandas as pd
  data = {
  
      "Source": ["ISO", "EPA", "WHO"],
  
      "Data Type": ["Standard", "Regulation", "Guideline"],
  
      "Access Method": ["API", "Web Scraping", "API"],
  
      "Last Updated": ["2023-10", "2023-09", "2023-11"]
  
  }
  
  df = pd.DataFrame(data)
  
  df.to_csv("compliance_sources.csv", index=False)
  
              

Organizing Scraped Data into a Responsive HTML Table

Validation of Free Compliance Lists Against Primary Sources

1. Reg

Tools and Methods for Automating Free Compliance List Updates

Python Script for API-Driven Compliance List Updates

Example Workflow for EPA’s EnviroAtlas (JSON API):

import requests
import json
import pandas as pd
from datetime import datetime

# API endpoint and parameters
url = "https://enviroatlas.epa.gov/api/v1/regulatory-data"
params = {"limit": 1000, "format": "json"}
headers = {"Authorization": "Bearer YOUR_API_TOKEN"}

# Fetch and parse data
response = requests.get(url, params=params, headers=headers)
data = response.json()

# Convert to DataFrame and save as CSV
df = pd.DataFrame(data["features"])
df.to_csv(f"compliance_list_{datetime.now().strftime('%Y%m%d')}.csv", index=False)

Key Considerations:

Parsing XML/JSON Feeds for Compliance Data

JSON Parsing Example (FDA’s OpenFDA):

import json

# Sample JSON response from OpenFDA
fda_data = {
"results": [
{
"facility_name": "Example Corp",
"inspection_type": "FDA Inspection",
"violation_description": "Non-compliance with GMP",
"jurisdiction": "US"
}
]
}

# Extract and structure data
compliance_records = []
for record in fda_data["results"]:
compliance_records.append({
"facility": record["facility_name"],
"regulation": "FDA GMP",
"jurisdiction": record["jurisdiction"]
})

# Save to JSON
with open("fda_compliance.json", "w") as f:
json.dump(compliance_records, f, indent=4)

XML Parsing Example (EPA’s EnviroAtlas):

import xml.etree.ElementTree as ET

# Sample XML response
xml_data = """
Green Energy Inc Clean Air Act Non-compliant """

root = ET.fromstring(xml_data)
records = []
for record in root.findall("record"):
records.append({
"facility": record.find("facility").text,
"regulation": record.find("regulation").text,
"status": record.find("status").text
})

# Convert to CSV
import csv
with open("epa_compliance.csv", "w", newline="") as f:
writer = csv.DictWriter(f, fieldnames=["facility", "regulation", "status"])
writer.writeheader()
writer.writerows(records)

Best Practices:

Integration with a Simple Compliance Dashboard

HTML/CSS Structure:

Compliance Status Dashboard

JavaScript (`dashboard.js`) for Data Loading and Filtering:

// Sample compliance data (loaded from CSV/JSON)
const complianceData = [
{ facility: "Alpha Corp", regulation: "OSHA", jurisdiction: "US", status: "Compliant" },
{ facility: "Beta Ltd", regulation: "REACH", jurisdiction: "EU", status: "Non-compliant" }
];

// Initialize chart
const ctx = document.getElementById('complianceChart').getContext('2d');
let chart = new Chart(ctx, {
type: 'bar',
data: {
labels: complianceData.map(d => d.facility),
datasets: [{
label: 'Compliance Status',
data: complianceData.map(d => d.status === "Compliant" ? 1 : 0),
backgroundColor: complianceData.map(d => d.status === "Compliant" ? 'green' : 'red')
}]
}
});

// Filter data on dropdown change
document.getElementById('jurisdiction-filter').addEventListener('change', function() {
const jurisdiction = this.value;
const filteredData = jurisdiction === "all"
? complianceData
: complianceData.filter(d => d.jurisdiction === jurisdiction);

chart.data.labels = filteredData.map(d => d.facility);
chart.data.datasets[0].data = filteredData.map(d => d.status === "Compliant" ? 1 : 0);
chart.data.datasets[0].backgroundColor = filteredData.map(d => d.status === "Compliant" ? 'green' : 'red');
chart.update();
});

Enhancements:

Open-Source Tools for Processing and Enriching Compliance Datasets

1. Apache NiFi

2. Talend Open Studio

Responsive HTML Charts for Compliance Violation Trends