Skip to main content

​
Lead Enrichment Pipeline

Collect LinkedIn profiles from a search, enrich them with LLM analysis, and load results into PostgreSQL.
1

Create the pipeline

dataset.yaml
name: lead-enrichment
description: Find and qualify engineering leads

sources:
  - id: search
    endpoint: /api/linkedin/search/users
    params:
      keywords: "Head of Engineering"
      count: 200
    parallel: 1

  - id: profiles
    endpoint: /api/linkedin/user
    dependency:
      from_source: search
      field: urn.value
    input_key: user
    parallel: 5
    rate_limit: "10/s"
    on_error: skip

  - id: qualified
    type: llm
    dependency:
      from_source: profiles
      field: name
    llm:
      - type: classify
        categories: "high_priority,medium,low"
        output_column: lead_score
      - type: enrich
        add:
          - "seniority:junior/mid/senior/lead/executive"
          - "is_technical:boolean"
          - "team_size:small/medium/large"
      - type: summarize
        max_length: 30
        output_column: quick_bio

storage:
  format: parquet
  path: ./data/
2

Run the collection

anysite dataset collect dataset.yaml
3

Load into PostgreSQL

anysite dataset load-db dataset.yaml -c pg
4

Query the results

anysite db query pg --sql "
  SELECT name, headline, lead_score, seniority, quick_bio
  FROM qualified
  WHERE lead_score = 'high_priority' AND is_technical = true
  ORDER BY name
" --format table

​
Competitor Monitoring

Track competitor companies, their employees, and recent posts on a weekly schedule.
1

Create a competitors file

competitors.txt
anthropic
openai
google-deepmind
mistral-ai
2

Define the pipeline

competitor-monitor.yaml
name: competitor-monitor
description: Weekly competitor intelligence

sources:
  - id: companies
    endpoint: /api/linkedin/company
    from_file: competitors.txt
    input_key: company
    parallel: 2
    refresh: always

  - id: recent_posts
    endpoint: /api/linkedin/company/posts
    dependency:
      from_source: companies
      field: urn.value
    input_key: company
    parallel: 3
    rate_limit: "10/s"
    refresh: always

  - id: key_employees
    endpoint: /api/linkedin/company/employees
    dependency:
      from_source: companies
      field: urn.value
    input_key: companies
    parallel: 3
    rate_limit: "10/s"
    refresh: auto

  - id: post_analysis
    type: llm
    dependency:
      from_source: recent_posts
      field: text
    llm:
      - type: classify
        categories: "product_launch,hiring,partnership,thought_leadership,other"
        output_column: post_type
      - type: summarize
        max_length: 30
        output_column: summary

storage:
  format: parquet
  path: ./data/

schedule:
  cron: "0 9 * * 1"

notifications:
  on_complete:
    - url: "https://hooks.slack.com/services/xxx"
3

Start the scheduled collection

anysite dataset schedule competitor-monitor.yaml --incremental --load-db pg
4

Analyze the data

# What are competitors posting about?
anysite dataset query competitor-monitor.yaml --sql "
  SELECT c.name as company, pa.post_type, COUNT(*) as count
  FROM post_analysis pa
  JOIN companies c ON pa.company_id = c.urn_value
  GROUP BY c.name, pa.post_type
  ORDER BY c.name, count DESC
" --format table

# New hires this week
anysite dataset query competitor-monitor.yaml --sql "
  SELECT name, headline, company_name
  FROM key_employees
  ORDER BY collected_at DESC
  LIMIT 20
" --format table

​
Multi-Platform Research

Collect data from LinkedIn, Twitter, and GitHub for a set of people, merge the results, and export a unified dataset.
1

Define the pipeline

research.yaml
name: multi-platform-research
description: Cross-platform person research

sources:
  - id: linkedin_profiles
    endpoint: /api/linkedin/user
    from_file: people.txt
    input_key: user
    parallel: 3
    rate_limit: "10/s"
    on_error: skip

  - id: twitter_profiles
    endpoint: /api/twitter/user
    from_file: twitter_handles.txt
    input_key: user
    parallel: 3
    rate_limit: "10/s"
    on_error: skip

  - id: github_profiles
    endpoint: /api/github/user
    from_file: github_users.txt
    input_key: user
    parallel: 3
    on_error: skip

  - id: all_profiles
    type: union
    sources: [linkedin_profiles, twitter_profiles, github_profiles]

storage:
  format: parquet
  path: ./data/
2

Collect and query

# Collect all sources
anysite dataset collect research.yaml

# Query the unified dataset
anysite dataset query research.yaml --sql "
  SELECT * FROM all_profiles
" --format csv --output unified_research.csv

# Get statistics
anysite dataset stats research.yaml

​
Quick One-Liners

Common tasks that don’t need a full pipeline: 
# Enrich a single profile and save to database
anysite api /api/linkedin/user user=satyanadella -q --format jsonl | \
  anysite db insert mydb --table profiles --stdin --auto-create

# Batch process a CSV of companies
anysite api /api/linkedin/company --from-file companies.csv --input-key company \
  --parallel 5 --rate-limit "10/s" --on-error skip \
  --format csv --output company_profiles.csv

# Search and export in one command
anysite api /api/linkedin/search/users keywords="AI researcher" count=100 \
  --format csv --output ai_researchers.csv

# Quick database query
anysite db query pg --sql "SELECT name, headline FROM profiles WHERE headline LIKE '%CEO%'" \
  --format table