Lead-Bot v1.0

Lead-Bot is a powerful, automated suite of tools designed for lead generation, site enrichment, and data sanitization. It leverages browser automation (Playwright), AI (Google Gemini), and robust data processing to transform search terms into high-quality, enriched lead lists.

🚀 Features

The bot consists of three specialized tools that work together in a pipeline:

1. Domain Collection Tool

• Google Search Automation: Automatically iterates through search terms in keywords.txt.
• Negative Filtering: Appends exclusions from excluded_domains.txt (e.g., -site:yelp.com) to every query.
• Root Domain Extraction: Intelligently extracts base domains (e.g., example.com), filtering out subdomains and noise.
• Persistence: Deduplicates against leads.csv in real-time to ensure unique results across sessions.
• Human-in-the-loop: Detects CAPTCHAs and pauses for manual resolution.

2. Site Enrichment Tool (AI-Powered)

• Deep Crawling: Visits each domain from leads.csv to find contact info, licenses, and business details.
• Gemini 2.0 Flash AI: Uses advanced AI to analyze page content and extract structured data accurately.
• Automated Age Gate Bypassing: Handles common "18+/21+" age verification popups automatically.
• Contact Extraction: Captures up to 3 emails and 3 phone numbers (validated/formatted) per site.
• Schema & Meta Analysis: Parses JSON-LD and meta tags for precise business names and addresses.
• Smart Navigation: Automatically explores "Contact," "About," and "License" pages if data is missing from the homepage.

3. Lead CSV Cleanup Tool

• CSV Alignment: Fixes broken CSV rows by stripping rogue newlines and sanitizing fields.
• Strict Domain Validation: Removes malformed URLs and non-domain entries.
• Directory Blacklist: Automatically filters out common directories (Yelp, Weedmaps, Facebook, etc.).
• Deduplication: Final pass to ensure every lead in the final list is unique.

---

🛠️ Setup

Prerequisites

• Node.js (v16 or higher)
• Playwright Browsers (installed automatically via npx playwright install or during first run)

Configuration

1. Install Dependencies:

   npm install

2. Environment Variables: Create a .env file in the root directory and add your Google Gemini API Key:

   GEMINI_API_KEY=your_gemini_api_key_here

3. Input Files:
   • keywords.txt: Add search terms (one per line).
   • excluded_domains.txt: Add domains to ignore (one per line).

---

📖 Usage

Run the main entry point:

npm start

You will be presented with an interactive menu to select the tool you wish to run:

1. Domain Collection: Scrapes Google for new leads.
2. Site Enrichment: Visits websites to find emails, phones, and business data.
3. Lead CSV Cleanup: Sanitizes and deduplicates the final output.

---

📂 File Reference

File	Description
keywords.txt	Input: Search terms for the Domain Collection tool.
excluded_domains.txt	Input: Domains to exclude from search results.
leads.csv	Output: Raw list of domains and search terms found.
enriched_leads.csv	Output: Detailed business data from the Enrichment tool.
cleaned_leads.csv	Output: Final, sanitized, and deduplicated lead list.

---

⚠️ Important Notes

• CAPTCHAs: When running the Domain Collection tool, if Google presents a CAPTCHA, the bot will pause. Solve it in the browser window and press Enter in the terminal to continue.
• AI Usage: The Enrichment tool uses the Google Gemini API. Monitor your usage and quotas in the Google AI Studio.
• Headless Mode: Tools are currently configured to run in non-headless mode so you can monitor progress.