Programmatic SEO Competitor Spy

Before building your own programmatic SEO pages, know what your competitors are already doing. This skill crawls competitor sitemaps, detects which URL patterns are programmatic, estimates which ones actually drive traffic, assesses template quality, and finds the gaps they're missing.

Core principle: The best pSEO strategy starts with competitive intelligence. If 3 competitors have /vs/ pages and none have /for-{industry}/ pages, that's a signal. This skill turns competitor site structures into strategic intelligence.

When to Use

• "What programmatic SEO are competitors doing?"
• "Reverse-engineer competitor SEO pages"
• "Which pSEO patterns actually work in our space?"
• "Find pSEO gaps our competitors are missing"
• "Analyze competitor URL structure for SEO patterns"

Tool Enhancement (Optional)

This skill works with existing capabilities but is dramatically better with domain analytics data that shows actual traffic per URL pattern.

Agent Prompt to User

"I can analyze competitor site structures with our existing crawling tools. For the most accurate results — especially knowing which patterns actually drive traffic vs. just existing — I'd recommend connecting a domain analytics API."

Recommended: DataForSEO (pay-per-use, ~$0.02-0.05 per domain analysis)

• Sign up at dataforseo.com → get API login + password
• Set DATAFORSEO_LOGIN and DATAFORSEO_PASSWORD env vars

Alternatives that also work:

• SEMrush API (if you have a subscription) → set SEMRUSH_API_KEY
• Ahrefs API (if you have a subscription) → set AHREFS_API_TOKEN
• SimilarWeb API (if you have access) → set SIMILARWEB_API_KEY

"Want to use one of these, or should I proceed with baseline mode? Baseline will detect patterns and assess template quality, but traffic estimates will be inferred rather than measured."

Mode Selection

• Enhanced mode — Domain analytics API provides organic keywords and estimated traffic per URL. Enables confident "this pattern drives X traffic" conclusions. Can show which specific pages within a pattern rank well.
• Baseline mode — Uses site-content-catalog for URL crawling, seo-domain-analyzer for domain-level metrics, web_search for spot-checking rankings. Pattern detection is equally good. Traffic attribution is directional, based on page count × domain authority × keyword indicators.

Phase 0: Intake

1. Competitors — 1-5 competitor URLs to analyze
2. Your product URL — So we can compare coverage
3. Category — What market/category are these competitors in?
4. Specific interest? — Looking for anything specific (vs/ pages, integrations, etc.) or full analysis?
5. Tool preference — Enhanced mode with domain analytics API, or baseline? (see Tool Enhancement above)

Phase 1: Sitemap Crawl

For each competitor, run site-content-catalog:

python3 skills/site-content-catalog/scripts/catalog_content.py \
  --url "<competitor_url>" \
  --output json

Collect:

• All URLs from sitemap and internal crawl
• Page titles
• URL structure/hierarchy
• Last modified dates (if available)

Phase 2: Pattern Detection

2A: URL Clustering

Group URLs by structural patterns using regex matching:

For each detected pattern:

• Page count — How many pages follow this pattern?
• Data axis — What varies per page? (competitor name, tool name, industry, etc.)
• URL consistency — Are URLs cleanly structured or messy?
• Coverage completeness — Have they covered the obvious variations?

2B: Programmatic vs. Editorial Classification

Not every URL pattern is programmatic. Classify each cluster:

• Programmatic — Clearly templated: consistent URL structure, similar page titles following a pattern, high page count, structured content
• Semi-programmatic — Template with heavy manual customization
• Editorial — Individually written, no template pattern
• Auto-generated — Thin/low-quality programmatic (tag pages, archive pages)

Focus analysis on Programmatic and Semi-programmatic clusters only.

Phase 3: Traffic & Performance Analysis

Enhanced Mode (DataForSEO / SEMrush / Ahrefs)

For each competitor domain, pull organic keywords data:

# DataForSEO example
POST /v3/dataforseo_labs/google/ranked_keywords/live
{
  "target": "competitor.com",
  "filters": [
    ["ranked_serp_element.serp_item.url", "contains", "/vs/"]
  ]
}

Per pattern cluster:

• Total ranking keywords across all pages in the pattern
• Estimated monthly organic traffic to the pattern
• Average ranking position for pages in this pattern
• Top-performing pages within the pattern (which variations rank best?)
• Keyword difficulty distribution (are they targeting easy or hard keywords?)

Baseline Mode

Estimate traffic indicators:

• Run seo-domain-analyzer for overall domain metrics
• Spot-check 3-5 pages per pattern via web_search — do they appear in top 10?
• Check if pages are indexed (site:competitor.com/vs/)
• Infer relative traffic: pattern page count × indexation rate × domain authority proxy
• Note: these are directional estimates, not exact numbers

Phase 4: Template Quality Assessment

For each programmatic pattern, fetch 3-5 sample pages via fetch_webpage:

# Pick pages from the pattern — one high-variation, one low-variation, one middle

Evaluate template quality on:

Quality tiers:

• 4-5 avg: Well-executed pSEO — hard to beat without significant differentiation
• 3-4 avg: Decent but improvable — opportunity to out-template them
• 1-3 avg: Thin/low-quality — easy to dominate with better content

Phase 5: Gap Analysis

5A: Pattern Coverage Matrix

5B: Variation Gaps

Within each pattern competitors use, find missing variations:

• Competitors they haven't written vs/ pages for
• Integrations they support but haven't built pages for
• Industries they serve but haven't targeted
• These are your quick-win entries

5C: Pattern White Space

Entire pattern types no competitor has built:

• Highest opportunity — no one to outrank, first-mover advantage
• Validate search demand exists (Phase 1D of programmatic-seo-planner)

5D: Quality Gaps

Patterns where competitors have pages but they're low quality:

• Thin content (< 500 words, no unique data)
• Outdated (old screenshots, wrong pricing)
• Missing schema markup
• No visual differentiation between pages
• These are "out-template" opportunities

Phase 6: Output

# Programmatic SEO Competitive Landscape — [Category] — [DATE]
 
## Competitors Analyzed
- [Competitor A] — [domain metrics, total pages crawled]
- [Competitor B] — [domain metrics, total pages crawled]
- [Competitor C] — [domain metrics, total pages crawled]
 
## Executive Summary
- [N] programmatic patterns detected across [M] competitors
- Strongest competitor pSEO: [Competitor X] with [pattern] ([N] pages, [quality])
- Biggest opportunity: [pattern type] — [reasoning]
- Quick wins: [N] variation gaps in existing competitor patterns
 
---
 
## Pattern-by-Pattern Analysis
 
### Pattern: [vs/ Comparisons]
**Who's doing it:** [Competitor A (25 pages, ★★★★), Competitor B (10 pages, ★★★)]
**Traffic estimate:** [X monthly visits across pattern] (enhanced) / [directional estimate] (baseline)
**Template quality:** [summary of strengths/weaknesses]
**Top-performing pages:** [specific pages that rank well]
**Variation gaps:** [competitors/variations they're missing]
**Your opportunity:** [specific recommendation]
 
### Pattern: [integrations/]
...
 
---
 
## Opportunity Ranking
 
| Priority | Pattern | Opportunity Type | Effort | Expected Impact |
|----------|---------|-----------------|--------|----------------|
| P0 | [pattern] | White space — no competitors | Medium | High |
| P0 | [pattern] | Quality gap — beat weak templates | Low | Medium |
| P1 | [pattern] | Variation gap — fill missing pages | Low | Medium |
| P2 | [pattern] | Head-to-head — outrank strong competitor | High | High |
 
---
 
## Recommended Action Plan
 
1. **Immediate (Week 1-2):** Build [pattern] — [rationale]
2. **Short-term (Month 1):** Build [pattern] — [rationale]
3. **Medium-term (Month 2-3):** Build [pattern] — [rationale]
 
---
 
## Raw Data
[Link to crawl data, pattern clusters, sample pages analyzed]

Save to the current working directory or wherever the user prefers.

Cost

Tools Required

• Apify API token — APIFY_API_TOKEN env var
• Upstream skills: site-content-catalog, seo-domain-analyzer, fetch_webpage
• Optional (enhanced): DataForSEO (DATAFORSEO_LOGIN + DATAFORSEO_PASSWORD), SEMrush (SEMRUSH_API_KEY), Ahrefs (AHREFS_API_TOKEN), or SimilarWeb (SIMILARWEB_API_KEY)

Trigger Phrases

• "What pSEO are competitors doing?"
• "Reverse-engineer competitor programmatic pages"
• "Analyze competitor URL structure"
• "Find pSEO gaps in our space"
• "What comparison pages do competitors have?"