# Webscraper CRM Integration Architecture | AGM Network

## Current System Snapshot
The repository webscraper-main includes Flask applications, API rotation, validation logic, export paths, and a production deployment profile. It is well suited to operate as lead-intelligence ingestion for CRM and off-page operations.

## Integration Goals
- Convert discovered contacts into campaign-ready outreach records
- Feed lead quality signals into Leads Admin workflows
- Provide campaign intelligence to command center dashboards
- Align webscraper keyword discovery with on-page content and off-page outreach priorities

## Reference Integration Pattern

### 1) Ingestion Layer
- Source: webscraper app exports (sqlite, xlsx, csv)
- Adapter: normalized ingestion script writes canonical outreach records
- Destination: data/offpage/outreach_database.csv and optional sqlite campaign table

### 2) Orchestration Layer
- Trigger options:
  - Scheduled daily ingestion
  - Manual refresh from command center
  - Event-based ingestion after major campaign runs
- Primary controls:
  - duplicate detection
  - contact format validation
  - source quality scoring

### 3) CRM and Admin Layer
- Leads Admin consumes normalized records for qualification and assignment.
- Chat Admin endpoints remain independent for support conversations, while status rollups surface in command center.

### 4) Optimization Layer
- Extract high-performing keywords and industries from webscraper results.
- Feed these into guest-post and PR topic planning.
- Route qualified opportunities to partnership outreach queue.

## Immediate Build Steps
1. Build export normalizer from webscraper output to outreach csv format.
2. Add source_quality, lead_score, and next_action fields.
3. Add ingestion run log and delta count report.
4. Expose last_ingestion timestamp in command center KPIs.

## Data Contract (Canonical Fields)
- site
- contact
- method
- acceptance_rate
- status
- notes
- source_system
- lead_score
- next_action_date

## Quality Gate
- Reject empty contact and site fields.
- Flag non-business domains and low-confidence scraped contacts.
- Maintain dedup key by site + contact.

## Outcome
The integration turns webscraper-main from an isolated lead finder into a production-grade CRM intelligence feeder connected directly to off-page campaign execution.