**🧠 The data layer for AI systems.** Skill Seekers turns documentation sites, GitHub repos, PDFs, videos, notebooks, wikis, and more — **18 source types** — into structured knowledge assets, ready to power AI Skills (Claude, Gemini, OpenAI), RAG pipelines (LangChain, LlamaIndex, Pinecone), and AI coding assistants (Cursor, Windsurf, Cline). Prepare once, export to **22 targets**. ## 💛 Sponsors **Launch Partner**

[Atlas Cloud](https://www.atlascloud.ai/) — A full-modal, OpenAI-compatible AI inference platform. Skill Seekers supports it as a packaging/enhancement target via `--target atlas` with `ATLAS_API_KEY`. **[Become a sponsor](SPONSORSHIP.md)** · [GitHub Sponsors](https://github.com/sponsors/yusufkaraaslan) --- ## 🚀 Quick Start ```bash # 1. Install pip install skill-seekers # 2. Create a skill from any source skill-seekers create https://docs.djangoproject.com/ # 3. Package it for your AI platform skill-seekers package output/django --target claude ``` You now have `output/django-claude.zip`, ready to use. ```bash # Pick a different AI agent for enhancement (default: claude) skill-seekers create https://docs.djangoproject.com/ --agent kimi skill-seekers create https://docs.djangoproject.com/ --agent-cmd "my-custom-agent run" ``` ### 🛰️ AI-driven project scan Point `scan` at a project and an AI agent reads its manifests, README, Dockerfile/CI and sampled source imports — then emits one config per detected framework, plus a `-codebase.json` for your own code: ```bash skill-seekers scan ./my-react-app --out ./configs/scanned/ # → react.json, vite.json, tailwind.json, jest.json, my-react-app-codebase.json skill-seekers create ./configs/scanned/react.json ``` If a detection has no existing preset, the AI generates a fresh config; on exit you can optionally publish it back to the [community registry](https://github.com/yusufkaraaslan/skill-seekers-configs). ### All 18 source types ```bash skill-seekers create facebook/react # GitHub repository skill-seekers create ./my-project # Local codebase skill-seekers create manual.pdf # PDF skill-seekers create report.docx # Word skill-seekers create book.epub # EPUB skill-seekers create notebook.ipynb # Jupyter skill-seekers create openapi.yaml # OpenAPI/Swagger skill-seekers create presentation.pptx # PowerPoint skill-seekers create guide.adoc # AsciiDoc skill-seekers create page.html # Local HTML (or a whole dir) skill-seekers create feed.rss # RSS/Atom skill-seekers create curl.1 # Man page # Video (YouTube, Vimeo, or local — needs skill-seekers[video]) skill-seekers create --video-url https://www.youtube.com/watch?v=... --name mytutorial skill-seekers create --setup # auto-install GPU-aware visual deps skill-seekers create --space-key TEAM --name wiki # Confluence skill-seekers create --database-id ... --name docs # Notion skill-seekers create --chat-export-path ./slack-export --name team-chat # Slack/Discord ``` See the [Scraping Guide](docs/user-guide/02-scraping.md) for every source type and its options. --- ## 📦 Installation ```bash pip install skill-seekers # Core: scraping, GitHub, PDF, packaging pip install skill-seekers[all-llms] # + every LLM platform pip install skill-seekers[mcp] # + MCP server pip install skill-seekers[all] # Everything ``` **Not sure what you need?** Run the wizard: `skill-seekers-setup`

All installation extras

| Install | Adds | |---------|------| | `skill-seekers[gemini]` | Google Gemini support | | `skill-seekers[openai]` | OpenAI ChatGPT support | | `skill-seekers[all-llms]` | All LLM platforms | | `skill-seekers[mcp]` | MCP server for Claude Code, Cursor, etc. | | `skill-seekers[video]` | YouTube/Vimeo transcript & metadata extraction | | `skill-seekers[video-full]` | + Whisper transcription & visual frame extraction | | `skill-seekers[jupyter]` | Jupyter Notebook support | | `skill-seekers[pptx]` | PowerPoint support | | `skill-seekers[confluence]` | Confluence wiki support | | `skill-seekers[notion]` | Notion pages support | | `skill-seekers[rss]` | RSS/Atom feed support | | `skill-seekers[chat]` | Slack/Discord chat export support | | `skill-seekers[asciidoc]` | AsciiDoc support | | `skill-seekers[all]` | Everything | > **Video visual deps (GPU-aware):** after installing `skill-seekers[video-full]`, run `skill-seekers create --setup` to auto-detect your GPU and install the matching PyTorch variant + easyocr.

**Prerequisites:** Python 3.10+, Git. New here? → **[Bulletproof Quick Start](docs/getting-started/BULLETPROOF_QUICKSTART.md)** 🎯 --- ## 📚 Documentation | I want to... | Read this | |--------------|-----------| | **Get started quickly** | [Quick Start](docs/getting-started/02-quick-start.md) — 3 commands to your first skill | | **Understand the concepts** | [Core Concepts](docs/user-guide/01-core-concepts.md) | | **Scrape sources** | [Scraping Guide](docs/user-guide/02-scraping.md) — all 18 source types | | **Enhance skills with AI** | [Enhancement Guide](docs/user-guide/03-enhancement.md) · [Enhancement Modes](docs/features/ENHANCEMENT_MODES.md) | | **Export skills** | [Packaging Guide](docs/user-guide/04-packaging.md) | | **Build workflows** | [Workflows](docs/user-guide/05-workflows.md) | | **Look up a command** | [CLI Reference](docs/reference/CLI_REFERENCE.md) — all 19 commands | | **Configure** | [Config Format](docs/reference/CONFIG_FORMAT.md) · [Environment Variables](docs/reference/ENVIRONMENT_VARIABLES.md) | | **Set up MCP** | [MCP Setup](docs/guides/MCP_SETUP.md) · [MCP Reference](docs/reference/MCP_REFERENCE.md) | | **Integrate with RAG / IDEs** | [LangChain](docs/integrations/LANGCHAIN.md) · [RAG Pipelines](docs/integrations/RAG_PIPELINES.md) · [Cursor](docs/integrations/CURSOR.md) · [Windsurf](docs/integrations/WINDSURF.md) · [Cline](docs/integrations/CLINE.md) | | **Handle huge doc sets** | [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) — 10K–40K+ pages | | **Understand the architecture** | [UML Architecture](docs/UML_ARCHITECTURE.md) — 14 diagrams | | **Fix a problem** | [Troubleshooting](docs/user-guide/06-troubleshooting.md) | **Complete documentation index:** [docs/README.md](docs/README.md) --- ## 🎯 What you get | Use case | Output | Powers | |----------|--------|--------| | **AI Skills** | Comprehensive `SKILL.md` + reference files | Claude Code, Gemini, GPT | | **RAG pipelines** | Chunked documents with rich metadata | LangChain, LlamaIndex, Haystack | | **Vector databases** | Pre-formatted data ready for upsert | Pinecone, Chroma, Weaviate, FAISS, Qdrant | | **AI coding assistants** | Context files your IDE AI reads automatically | Cursor, Windsurf, Cline, Continue.dev | ### Export targets (22) ```bash skill-seekers package output/react --target claude # → Claude Skill (ZIP + YAML) skill-seekers package output/react --target langchain # → LangChain Documents skill-seekers package output/react --target llama-index # → LlamaIndex TextNodes skill-seekers package output/react --target ibm-bob # → IBM Bob skill directory ``` **LLM platforms (12):** `claude` · `gemini` · `openai` · `minimax` · `opencode` · `kimi` · `deepseek` · `qwen` · `openrouter` · `together` · `fireworks` · `markdown` **RAG & vector (8):** `langchain` · `llama-index` · `haystack` · `chroma` · `faiss` · `weaviate` · `qdrant` · `pinecone` **Other (2):** `atlas` · `ibm-bob` See the [Feature Matrix](docs/reference/FEATURE_MATRIX.md) for per-platform support details. ### Why it matters - ⚡ **99% faster** — days of manual data prep → 15–45 minutes - 🎯 **Real skill quality** — 500+ line `SKILL.md` files with examples, patterns, and guides - 📊 **RAG-ready chunks** — smart chunking preserves code blocks and context - 🔄 **Multi-source** — combine docs + GitHub + PDFs + videos into one knowledge asset - 🌐 **One prep, every target** — export to 22 targets without re-scraping - ✅ **Battle-tested** — 3,900+ tests, 68 workflow presets, production-ready --- ## ✨ Key capabilities

Documentation scraping — SPA discovery, llms.txt, smart categorization

Three-layer discovery for JavaScript SPA sites (`sitemap.xml` → `llms.txt` → headless browser rendering), automatic `llms.txt` detection (10× faster when present), smart topic categorization, and a lenient HTML parser fallback so broken markup still scrapes. → [Scraping Guide](docs/user-guide/02-scraping.md) · [llms.txt Support](docs/reference/LLMS_TXT_SUPPORT.md)

GitHub & codebase analysis (C3.x) — AST parsing, pattern detection, how-to guides

Three-stream architecture: code analysis (AST, design patterns, tests), documentation (README, `docs/`, wiki), and community (issues, PRs, metadata). The C3.x pipeline adds 10 GoF pattern detectors across 9 languages, usage examples extracted from tests, AI-written how-to guides, config extraction, and architecture overviews. ```bash skill-seekers create ./my-project --preset quick # 1–2 min, surface level skill-seekers create ./my-project --preset standard # balanced (default) skill-seekers create ./my-project --preset comprehensive # deep, exhaustive ``` → [Pattern Detection](docs/features/PATTERN_DETECTION.md) · [How-To Guides](docs/features/HOW_TO_GUIDES.md) · [Test Example Extraction](docs/features/TEST_EXAMPLE_EXTRACTION.md)

AI enhancement — API or local agents, 68 workflow presets

Every AI call runs through one transport, in **API mode** (Anthropic, Google Gemini, OpenAI, Moonshot/Kimi, MiniMax) or **LOCAL mode** (Claude Code, Kimi Code, Codex, Copilot, OpenCode, custom agents — no API costs). Control depth with `--enhance-level 0-3` and pick an agent with `--agent`. → [Enhancement Guide](docs/user-guide/03-enhancement.md) · [Enhancement Modes](docs/features/ENHANCEMENT_MODES.md) · [Multi-Agent Setup](docs/guides/MULTI_AGENT_SETUP.md)

Unified multi-source scraping — combine many sources into one skill

One config can pull documentation, GitHub, PDFs, videos, and more into a single knowledge asset, with conflict detection and pairwise synthesis across sources. → [Unified Scraping](docs/features/UNIFIED_SCRAPING.md)

Video extraction — transcripts, frames, on-screen code

YouTube, Vimeo, and local files. Three-tier transcript fallback (subtitles → YouTube transcript API → local Whisper), plus optional visual extraction that OCRs on-screen code from sampled frames. → [Video Guide](docs/VIDEO_GUIDE.md)

Quality, sync & scale

Quality scoring with a gate (`skill-seekers quality output/react/ --threshold 7`), doc-change detection with scheduled re-scrapes and notifications, streaming ingestion for very large doc sets, and incremental updates. → [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) · [Code Quality](docs/reference/CODE_QUALITY.md)

--- ## 🔌 MCP Integration (40 tools) Skill Seekers ships an MCP server for Claude Code, Cursor, Windsurf, VS Code + Cline, and IntelliJ IDEA. ```bash # stdio mode (Claude Code, VS Code + Cline) python -m skill_seekers.mcp.server_fastmcp # HTTP mode (Cursor, Windsurf, IntelliJ) python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765 ``` Then just ask your assistant: *"Package and upload the React skill."* → [MCP Setup](docs/guides/MCP_SETUP.md) · [MCP Reference](docs/reference/MCP_REFERENCE.md) · [HTTP Transport](docs/guides/HTTP_TRANSPORT.md) --- ## 🤖 Installing to AI agents Skills install automatically into **19 AI coding agents**: ```bash skill-seekers install-agent output/react/ --agent cursor skill-seekers install-agent output/react/ --agent all # every detected agent skill-seekers install-agent output/react/ --agent cursor --dry-run ``` | Agent | Path | Scope | |-------|------|-------| | Claude Code | `~/.claude/skills/` | Global | | Cursor | `.cursor/skills/` | Project | | VS Code / Copilot | `.github/skills/` | Project | | Amp | `~/.amp/skills/` | Global | | Goose | `~/.config/goose/skills/` | Global | | OpenCode | `~/.opencode/skills/` | Global | | Letta | `~/.letta/skills/` | Global | | Aide | `~/.aide/skills/` | Global | | Windsurf | `~/.windsurf/skills/` | Global | | Neovate | `~/.neovate/skills/` | Global | | Roo Code | `.roo/skills/` | Project | | Cline | `.cline/skills/` | Project | | Aider | `~/.aider/skills/` | Global | | Bolt | `.bolt/skills/` | Project | | Kilo Code | `.kilo/skills/` | Project | | Continue | `~/.continue/skills/` | Global | | Kimi Code | `~/.kimi/skills/` | Global | | IBM Bob | `.bob/skills/` | Project | ### Uploading to Claude ```bash export ANTHROPIC_API_KEY=sk-ant-... skill-seekers package output/react/ --upload # package + upload skill-seekers upload output/react.zip # upload an existing zip ``` No API key? Package it and upload `output/react.zip` manually at [claude.ai/skills](https://claude.ai/skills). → [Upload Guide](docs/guides/UPLOAD_GUIDE.md) --- ## ⚙️ How it works ```mermaid graph LR A[Documentation Website] --> B[Skill Seekers] B --> C[Scraper] B --> D[AI Enhancement] B --> E[Packager] C --> F[Organized References] D --> F F --> E E --> G[AI Skill .zip] G --> H[Upload to AI Platform] ``` 1. **Scrape** — extract every page (checking `llms.txt` first) 2. **Categorize** — organize content into topics (API, guides, tutorials, …) 3. **Enhance** — AI writes a comprehensive `SKILL.md` with examples 4. **Package** — bundle into a platform-ready artifact 5. **Upload** — ship it to your AI platform (optional) ### Architecture **8 core modules + 5 utility modules** (~200 classes): | Module | Purpose | |--------|---------| | **CLICore** | Git-style command dispatcher, source auto-detection | | **Scrapers** | 18 source-type extractors on a shared build layer | | **Adaptors** | 22 output platform formats behind one `SkillAdaptor` ABC | | **Analysis** | C3.x codebase pipeline, 10 GoF pattern detectors | | **Enhancement** | AI improvement via a single `AgentClient` transport | | **Packaging** | Package, upload, and install skills | | **MCP** | FastMCP server (40 tools, 10 tool modules) | | **Sync** | Doc change detection and notification | → [UML Architecture](docs/UML_ARCHITECTURE.md) · [API Reference](docs/reference/API_REFERENCE.md) · [Skill Architecture](docs/reference/SKILL_ARCHITECTURE.md) --- ## 🆕 New in v3.9.0 - **HTML parser fallback for broken markup** (#96) — severely malformed pages no longer scrape as empty; well-formed pages are byte-identical. - **Transient-failure retries** — the doc scraper (#97) and MCP `fetch_config` (#92) now retry connection blips and 5xx with backoff; 4xx still fails fast. - **Whisper transcription fallback** (#420) — local videos without subtitles finally get a real transcript. - **MiniMax image OCR + registry-driven multimodal providers** (#423) — providers declare their wire protocol and image capability; China-issued keys work against the right endpoint. - **Token-lean GitHub issue defaults** (#169) — GitHub skills no longer bundle full closed-issue history by default. - **Env-driven CORS across all three servers** (#422, #424) — no more wildcard origins with credentials. Full history: **[CHANGELOG.md](CHANGELOG.md)** --- ## 📈 Performance | Documentation size | Time | Output | |---|---|---| | Small (< 100 pages) | 5–10 min | ~2 MB | | Medium (100–500 pages) | 15–30 min | ~10 MB | | Large (500–2,000 pages) | 30–60 min | ~40 MB | | Huge (10K–40K+ pages) | Use `stream` | See [Large Documentation](docs/reference/LARGE_DOCUMENTATION.md) | --- ## 🐛 Troubleshooting ```bash skill-seekers doctor # diagnose installation & environment skill-seekers sync-config # detect config drift ``` Common issues and fixes: **[Troubleshooting Guide](docs/user-guide/06-troubleshooting.md)** · [TROUBLESHOOTING.md](docs/TROUBLESHOOTING.md) --- ## 🤝 Contributing Contributions are welcome — see **[CONTRIBUTING.md](CONTRIBUTING.md)**. - 📋 **[Development Roadmap & Tasks](https://github.com/users/yusufkaraaslan/projects/2)** — pick any task - 💬 **[Discussions](https://github.com/yusufkaraaslan/Skill_Seekers/discussions)** — questions and ideas - 🐛 **[Issues](https://github.com/yusufkaraaslan/Skill_Seekers/issues)** — bugs and feature requests --- ## 📝 License MIT — see [LICENSE](LICENSE). ## 🔒 Security --- ## 🌐 Ecosystem Skill Seekers is a multi-repo project: | Repository | Description | Links | |-----------|-------------|-------| | **[Skill_Seekers](https://github.com/yusufkaraaslan/Skill_Seekers)** | Core CLI & MCP server (this repo) | [PyPI](https://pypi.org/project/skill-seekers/) | | **[skillseekersweb](https://github.com/yusufkaraaslan/skillseekersweb)** | Website & documentation | [Live](https://skillseekersweb.com/) | | **[skill-seekers-configs](https://github.com/yusufkaraaslan/skill-seekers-configs)** | Community config repository | | | **[skill-seekers-action](https://github.com/yusufkaraaslan/skill-seekers-action)** | GitHub Action for CI/CD | | | **[skill-seekers-plugin](https://github.com/yusufkaraaslan/skill-seekers-plugin)** | Claude Code plugin | | | **[homebrew-skill-seekers](https://github.com/yusufkaraaslan/homebrew-skill-seekers)** | Homebrew tap for macOS | | > **Want to contribute?** The website and configs repos are great starting points for new contributors!