Explore the codebase at /root/projects/ai_usecases_explorer thoroughly. I need to understand:

1. The overall project structure (all directories and files)
2. How the existing `simplify` skill is implemented (look for .claude/commands/ or similar skill/command directories)
3. The Obsidian reporter at src/ai_usecases_explorer/reporters/obsidian.py - read the full file
4. The settings.py for configuration patterns
5. The main.py orchestrator
6. Any existing web search/fetch utilities (Exa.ai integration in collectors/reddit.py)
7. The CLAUDE.md and any PLAN.md or SPEC.md files

Read the key files and give me full content for: obsidian.py, settings.py, main.py, and any skill/command definition files. Also show the directory tree structure.


I'll explore the codebase thoroughly to understand the project structure and read the key files you mentioned.


Bash

find /root/projects/ai_usecases_explorer -type f -name "*.py" -o -name "*.md" | head -30


Let me get the directory structure more efficiently and find the source files:


总计 60
drwxr-xr-x 9 root root  4096  2月28日 08:09 .
drwxr-xr-x 3 root root    34  2月27日 09:34 ..
-rw-r--r-- 1 root root  7747  2月27日 09:54 AI_VIEWS.md
drwxr-xr-x 2 root root    33  2月28日 10:26 .claude
-rw-r--r-- 1 root root  2094  2月27日 09:45 CLAUDE.md
drwxr-xr-x 2 root root    25  2月27日 23:21 data
-rw-r--r-- 1 root root   339  2月27日 20:22 .env
-rw-r--r-- 1 root root  1133  2月27日 20:04 .env.example
-rw-r--r-- 1 root root   114  2月27日 10:26 .gitignore
-rw-r--r-- 1 root root   564  2月27日 10:27 Makefile
-rw-r--r-- 1 root root  9213  2月27日 13:20 MILESTONE_1.md
-rw-r--r-- 1 root root 10767  2月28日 00:02 MILESTONE_2.md
-rw-r--r-- 1 root root  1101  2月27日 20:04 pyproject.toml
drwxr-xr-x 3 root root    70  2月27日 10:31 .pytest_cache
drwxr-xr-x 3 root root    58  2月27日 11:21 .ruff_cache
drwxr-xr-x 3 root root    34  2月27日 10:26 src
drwxr-xr-x 3 root root   186  2月27日 21:57 tests
drwxr-xr-x 5 root root    74  2月27日 09:39 venv


find /root/projects/ai_usecases_explorer/src -type f -name "*.py" | sort


总计 20
drwxr-xr-x 2 root root    33  2月28日 10:26 .
drwxr-xr-x 9 root root  4096  2月28日 08:09 ..
-rw-r--r-- 1 root root 14342  2月28日 10:26 settings.local.json


ls -la /root/.claude/ | grep -E "commands|skills"


Now let me search for any skill or command definitions:


find /root/.claude -name "*simplify*" -o -name "*command*" -o -name "*skill*" 2>/dev/null | head -20


Let me check if there's a simplify skill in the project-specific directories:


find /root/projects/ai_usecases_explorer -type f \( -name "*.md" -o -name "*.json" \) | grep -E "PLAN|SPEC|MILESTONE|skill|command" | sort


/root/projects/ai_usecases_explorer/MILESTONE_1.md
/root/projects/ai_usecases_explorer/MILESTONE_2.md


Now let me read the key files you requested:


     1→# CLAUDE.md
     2→
     3→This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
     4→
     5→## Commands
     6→
     7→The project uses Python 3.11 with a local virtualenv.
     8→
     9→```bash
    10→# Activate virtualenv
    11→source venv/bin/activate
    12→
    13→# Install dependencies (once pyproject.toml or requirements.txt exists)
    14→pip install -e ".[dev]"
    15→
    16→# Run all tests
    17→pytest
    18→
    19→# Run a single test file
    20→pytest tests/path/to/test_file.py
    21→
    22→# Run a single test by name
    23→pytest tests/path/to/test_file.py::test_function_name -v
    24→
    25→# Lint and type-check
    26→ruff check .
    27→mypy .
    28→```
    29→
    30→> Update this section as `pyproject.toml` and `requirements.txt` are added.
    31→
    32→## Architecture
    33→
    34→This project is in early initialization. Architecture notes will be added here as the codebase grows. Follow `PLAN.md` or `SPEC.md` for current requirements before writing any code.
    35→
    36→## Core Workflow
    37→- **TDD Strictly:** ALWAYS write or ask for a failing test case before implementing logic.
    38→- **Spec-First:** Follow `PLAN.md` or `SPEC.md`. Do not guess requirements.
    39→- **Plain English:** Explain logic in simple Subject-Verb-Object sentences.
    40→
    41→## Python Standards
    42→- **Type Hints:** MANDATORY for all function arguments and returns.
    43→- **Data Validation:** Use `Pydantic` models, not raw dictionaries.
    44→- **Docstrings:** Google-style, concise.
    45→- **No `os.path`:** Use `pathlib` for file handling.
    46→
    47→## Interaction Rules
    48→- **No Hallucinations:** If unsure about a library, ask to check docs.
    49→- **Context:** Do NOT read `venv/`, `node_modules/`, or large data files.
    50→
    51→## Human-in-the-Loop Protocol (MANDATORY)
    52→
    53→After generating any significant code (more than 20 lines or modifying core logic), you MUST automatically perform the following steps BEFORE asking me to proceed:
    54→
    55→1. **Self-Explanation:** Briefly explain *what* you changed and *why*, in Plain English.
    56→2. **Structural Check:** Did you introduce any new dependencies? Did you change any function signatures?
    57→3. **Risk Assessment:** What is the most likely thing to break with this change?
    58→4. **Verification Plan:** Tell me exactly which test command to run to verify this specific change.
    59→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Application settings loaded from environment / .env file."""
     2→from pathlib import Path
     3→
     4→from pydantic import Field
     5→from pydantic_settings import BaseSettings, SettingsConfigDict
     6→
     7→
     8→class Settings(BaseSettings):
     9→    """All configuration is read from environment variables or a .env file."""
    10→
    11→    model_config = SettingsConfigDict(env_file=".env", env_file_encoding="utf-8", extra="ignore")
    12→
    13→    anthropic_api_key: str = Field(default="", alias="ANTHROPIC_API_KEY")
    14→    anthropic_base_url: str = Field(default="", alias="ANTHROPIC_BASE_URL")
    15→    claude_model: str = Field(default="claude-haiku-4-5-20251001", alias="CLAUDE_MODEL")
    16→
    17→    db_path: Path = Field(default=Path("data/usecases.db"), alias="DB_PATH")
    18→    obsidian_report_dir: Path = Field(
    19→        default=Path("/root/vault/obsidian_vault/obsidian/obsidian/auto_report"),
    20→        alias="OBSIDIAN_REPORT_DIR",
    21→    )
    22→
    23→    hn_min_points: int = Field(default=10, alias="HN_MIN_POINTS")
    24→
    25→    exa_api_key: str = Field(default="", alias="EXA_API_KEY")
    26→    reddit_subreddits: list[str] = Field(
    27→        default=["openclaw", "vibecoding", "ClaudeCode", "singularity"],
    28→        alias="REDDIT_SUBREDDITS",
    29→    )
    30→
    31→    github_token: str = Field(default="", alias="GITHUB_TOKEN")
    32→    github_min_stars: int = Field(default=10, alias="GITHUB_MIN_STARS")
    33→
    34→    # Cap LLM calls per run to keep runtime under ~5 minutes
    35→    max_items_per_run: int = Field(default=80, alias="MAX_ITEMS_PER_RUN")
    36→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Main orchestrator — collect → process → store → report."""
     2→import argparse
     3→import itertools
     4→import logging
     5→import sys
     6→from datetime import UTC, date, datetime
     7→
     8→import anthropic
     9→from rich.console import Console
    10→from rich.progress import Progress
    11→
    12→from ai_usecases_explorer.collectors.base import RawItem
    13→from ai_usecases_explorer.collectors.github import GitHubCollector
    14→from ai_usecases_explorer.collectors.hackernews import HackerNewsCollector
    15→from ai_usecases_explorer.collectors.reddit import RedditCollector
    16→from ai_usecases_explorer.models.usecase import Novelty, UseCase, make_id
    17→from ai_usecases_explorer.processors.classifier import Classifier
    18→from ai_usecases_explorer.processors.deduplicator import Deduplicator
    19→from ai_usecases_explorer.processors.filter import ValueFilter
    20→from ai_usecases_explorer.reporters.obsidian import ObsidianReporter
    21→from ai_usecases_explorer.settings import Settings
    22→from ai_usecases_explorer.storage.database import Database
    23→
    24→console = Console()
    25→logger = logging.getLogger(__name__)
    26→
    27→
    28→class Scout:
    29→    """Orchestrates the full collect → process → store → report pipeline."""
    30→
    31→    def __init__(self, settings: Settings) -> None:
    32→        self._settings = settings
    33→        self._db = Database(db_path=settings.db_path)
    34→
    35→        client_kwargs: dict = {"api_key": settings.anthropic_api_key}
    36→        if settings.anthropic_base_url:
    37→            client_kwargs["base_url"] = settings.anthropic_base_url
    38→        client = anthropic.Anthropic(**client_kwargs)
    39→        model = settings.claude_model
    40→
    41→        self._hn = HackerNewsCollector(min_points=settings.hn_min_points)
    42→        self._reddit = RedditCollector(
    43→            subreddits=settings.reddit_subreddits,
    44→            api_key=settings.exa_api_key,
    45→        )
    46→        self._github = GitHubCollector(
    47→            min_stars=settings.github_min_stars,
    48→            token=settings.github_token,
    49→        )
    50→        self._filter = ValueFilter(client=client, model=model)
    51→        self._classifier = Classifier(client=client, model=model)
    52→        self._deduplicator = Deduplicator(client=client, model=model)
    53→        self._reporter = ObsidianReporter(output_dir=settings.obsidian_report_dir)
    54→
    55→    def run(self, dry_run: bool = False, report_date: date | None = None) -> None:
    56→        """Run the full pipeline for one day."""
    57→        if report_date is None:
    58→            report_date = date.today()
    59→
    60→        console.rule(f"[bold]AI Use Case Scout — {report_date}[/bold]")
    61→
    62→        # 1. Collect
    63→        console.print("[cyan]Collecting from HackerNews...[/cyan]")
    64→        hn_items = self._hn.fetch()
    65→        console.print(f"  HN: {len(hn_items)} items above threshold")
    66→
    67→        console.print("[cyan]Collecting from Reddit...[/cyan]")
    68→        reddit_items = self._reddit.fetch()
    69→        console.print(f"  Reddit: {len(reddit_items)} items above threshold")
    70→
    71→        console.print("[cyan]Collecting from GitHub...[/cyan]")
    72→        github_items = self._github.fetch()
    73→        console.print(f"  GitHub: {len(github_items)} repos above threshold")
    74→
    75→        all_items_raw = hn_items + reddit_items + github_items
    76→        # Sort each platform's items by engagement score (desc) before interleaving,
    77→        # so round-robin picks the highest-value items from each source first.
    78→        hn_items.sort(key=lambda x: x.score, reverse=True)
    79→        reddit_items.sort(key=lambda x: x.score, reverse=True)
    80→        github_items.sort(key=lambda x: x.score, reverse=True)
    81→        # Interleave sources (round-robin) so every platform gets representation
    82→        # even when HN dominates in volume. Then cap at max_items_per_run.
    83→        interleaved = [
    84→            item
    85→            for item in itertools.chain.from_iterable(
    86→                itertools.zip_longest(hn_items, reddit_items, github_items)
    87→            )
    88→            if item is not None
    89→        ]
    90→        all_items = interleaved[: self._settings.max_items_per_run]
    91→        platform_counts = {
    92→            p: sum(1 for i in all_items if i.source_platform == p)
    93→            for p in ("hackernews", "reddit", "github")
    94→        }
    95→        console.print(f"  [dim]Interleaved: {platform_counts}[/dim]")
    96→        console.print(
    97→            f"[green]Total collected: {len(hn_items) + len(reddit_items) + len(github_items)} "
    98→            f"→ processing top {len(all_items)} (round-robin)[/green]"
    99→        )
   100→
   101→        # 2. Process (skip already stored)
   102→        processed: list[UseCase] = []
   103→        skipped = 0
   104→        with Progress(console=console) as progress:
   105→            task = progress.add_task("Processing items...", total=len(all_items))
   106→            for item in all_items:
   107→                item_id = make_id(item.source_url)
   108→                if self._db.exists(item_id):
   109→                    skipped += 1
   110→                    progress.advance(task)
   111→                    continue
   112→                try:
   113→                    result = self._process_item(item, existing=[])
   114→                except Exception as exc:
   115→                    logger.warning("Failed to process %s: %s", item.source_url[:60], exc)
   116→                    progress.advance(task)
   117→                    continue
   118→                if result is not None:
   119→                    processed.append(result)
   120→                    if not dry_run:
   121→                        self._db.save(result)
   122→                progress.advance(task)
   123→
   124→        console.print(f"[green]Processed: {len(processed)} valuable, skipped: {skipped}[/green]")
   125→
   126→        # 3. Report
   127→        report_path = self._reporter.write(processed, report_date=report_date)
   128→        console.print(f"[bold green]Report written: {report_path}[/bold green]")
   129→
   130→    def _process_item(self, item: RawItem, existing: list[UseCase]) -> UseCase | None:
   131→        """Run the 3-step LLM pipeline. Returns None if item fails value filter."""
   132→        # Step 1 — value filter
   133→        filter_result = self._filter.run(item)
   134→        if not filter_result["is_real_value"]:
   135→            logger.debug(
   136→                "FILTERED [%s] %s — %s",
   137→                item.source_platform,
   138→                item.title[:60],
   139→                filter_result.get("reason", ""),
   140→            )
   141→            return None
   142→
   143→        # Step 2 — classify
   144→        classify_result = self._classifier.run(item)
   145→
   146→        # Step 3 — deduplication (compare against same-scenario existing records)
   147→        scenario_existing = self._db.get_by_scenario(classify_result["scenario_type"])
   148→        dedup_result = self._deduplicator.run(
   149→            # Build a minimal UseCase stub for deduplication
   150→            UseCase(
   151→                id=make_id(item.source_url),
   152→                title=item.title,
   153→                summary=classify_result["summary"],
   154→                source_url=item.source_url,
   155→                source_platform=item.source_platform,
   156→                author=item.author,
   157→                published_at=item.published_at,
   158→                collected_at=datetime.now(tz=UTC),
   159→                scenario_type=classify_result["scenario_type"],
   160→                tools_used=classify_result["tools_used"],
   161→                has_local_data=filter_result["has_local_data"],
   162→                has_workflow_opt=filter_result["has_workflow_opt"],
   163→                value_score=classify_result["value_score"],
   164→                is_real_value=True,
   165→                novelty=Novelty.NEW,
   166→                similar_to_ids=[],
   167→                raw_content=item.raw_content,
   168→            ),
   169→            existing=scenario_existing,
   170→        )
   171→
   172→        return UseCase(
   173→            id=make_id(item.source_url),
   174→            title=item.title,
   175→            summary=classify_result["summary"],
   176→            source_url=item.source_url,
   177→            source_platform=item.source_platform,
   178→            author=item.author,
   179→            published_at=item.published_at,
   180→            collected_at=datetime.now(tz=UTC),
   181→            scenario_type=classify_result["scenario_type"],
   182→            tools_used=classify_result["tools_used"],
   183→            has_local_data=filter_result["has_local_data"],
   184→            has_workflow_opt=filter_result["has_workflow_opt"],
   185→            value_score=classify_result["value_score"],
   186→            is_real_value=True,
   187→            novelty=dedup_result["novelty"],
   188→            similar_to_ids=dedup_result["similar_to_ids"],
   189→            raw_content=item.raw_content,
   190→        )
   191→
   192→
   193→def main() -> None:
   194→    parser = argparse.ArgumentParser(description="AI Use Case Daily Scout")
   195→    parser.add_argument("--dry-run", action="store_true", help="Don't write to DB or disk")
   196→    parser.add_argument(
   197→        "--date", type=str, default=None, help="Report date YYYY-MM-DD (default: today)"
   198→    )
   199→    args = parser.parse_args()
   200→
   201→    report_date = date.fromisoformat(args.date) if args.date else date.today()
   202→
   203→    settings = Settings()
   204→    if not settings.anthropic_api_key:
   205→        console.print(
   206→            "[red]Error: ANTHROPIC_API_KEY is not set. "
   207→            "Copy .env.example to .env and fill it in.[/red]"
   208→        )
   209→        sys.exit(1)
   210→
   211→    scout = Scout(settings=settings)
   212→    scout.run(dry_run=args.dry_run, report_date=report_date)
   213→
   214→
   215→if __name__ == "__main__":
   216→    main()
   217→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Generates daily Obsidian Markdown reports from processed UseCases."""
     2→from collections import defaultdict
     3→from datetime import date
     4→from pathlib import Path
     5→
     6→from ai_usecases_explorer.models.usecase import Novelty, ScenarioType, UseCase
     7→
     8→
     9→class ObsidianReporter:
    10→    """Writes a daily Markdown report to an Obsidian vault directory."""
    11→
    12→    def __init__(self, output_dir: Path) -> None:
    13→        self._output_dir = output_dir
    14→
    15→    def generate(self, usecases: list[UseCase], report_date: date) -> str:
    16→        """Build the full Markdown report string."""
    17→        new_ucs = [uc for uc in usecases if uc.novelty == Novelty.NEW and uc.is_real_value]
    18→        similar_ucs = [uc for uc in usecases if uc.novelty == Novelty.SIMILAR and uc.is_real_value]
    19→        repeat_ucs = [uc for uc in usecases if uc.novelty == Novelty.REPEAT or not uc.is_real_value]
    20→
    21→        hn_total = sum(1 for uc in usecases if uc.source_platform == "hackernews")
    22→        reddit_total = sum(1 for uc in usecases if uc.source_platform == "reddit")
    23→        github_total = sum(1 for uc in usecases if uc.source_platform == "github")
    24→        valuable_total = len(new_ucs) + len(similar_ucs)
    25→
    26→        lines: list[str] = []
    27→
    28→        # Header
    29→        lines.append(f"# AI Use Case 雷达 — {report_date.isoformat()}")
    30→        lines.append("")
    31→        lines.append(
    32→            f"> 今日扫描: HN({hn_total}) + Reddit({reddit_total}) + GitHub({github_total}) "
    33→            f"→ 过滤后: {valuable_total} 条有价值 → 新场景: {len(new_ucs)} 个"
    34→        )
    35→        lines.append("")
    36→        lines.append("---")
    37→        lines.append("")
    38→
    39→        # New use cases
    40→        lines.append(f"## 🆕 新场景（{len(new_ucs)}）")
    41→        lines.append("")
    42→        if new_ucs:
    43→            for uc in new_ucs:
    44→                lines.append(f"### {uc.scenario_type.value} · {uc.title}")
    45→                platform_label = "Reddit r/" + uc.source_url.split("/r/")[1].split("/")[0] \
    46→                    if "reddit.com" in uc.source_url else "HackerNews"
    47→                lines.append(
    48→                    f"**来源**: {platform_label} · `{uc.author}` · [原链接]({uc.source_url})"
    49→                )
    50→                lines.append("")
    51→                lines.append(uc.summary)
    52→                lines.append("")
    53→                lines.append(f"**工具**: {', '.join(uc.tools_used)}")
    54→                lines.append("")
    55→                lines.append("---")
    56→                lines.append("")
    57→        else:
    58→            lines.append("*今日无新场景*")
    59→            lines.append("")
    60→            lines.append("---")
    61→            lines.append("")
    62→
    63→        # Similar use cases grouped by scenario
    64→        lines.append(f"## ♻ 已知场景的新案例（{len(similar_ucs)}）")
    65→        lines.append("")
    66→        if similar_ucs:
    67→            by_scenario: dict[ScenarioType, list[UseCase]] = defaultdict(list)
    68→            for uc in similar_ucs:
    69→                by_scenario[uc.scenario_type].append(uc)
    70→            for scenario, items in by_scenario.items():
    71→                lines.append(f"### {scenario.value}（{len(items)} 条）")
    72→                for uc in items:
    73→                    tools_str = ", ".join(uc.tools_used) if uc.tools_used else "N/A"
    74→                    lines.append(
    75→                        f"- [{uc.title}]({uc.source_url}) — `{uc.author}` — *工具: {tools_str}*"
    76→                    )
    77→                lines.append("")
    78→        else:
    79→            lines.append("*今日无此类内容*")
    80→            lines.append("")
    81→
    82→        lines.append("---")
    83→        lines.append("")
    84→
    85→        # Filter log
    86→        lines.append("## 🚫 过滤日志")
    87→        lines.append(f"今日过滤 {len(repeat_ucs)} 条低价值内容。")
    88→        lines.append("")
    89→
    90→        return "\n".join(lines)
    91→
    92→    def write(self, usecases: list[UseCase], report_date: date) -> Path:
    93→        """Write the report to YYYY-MM-DD.md in the output directory."""
    94→        self._output_dir.mkdir(parents=True, exist_ok=True)
    95→        path = self._output_dir / f"{report_date.isoformat()}.md"
    96→        content = self.generate(usecases, report_date=report_date)
    97→        path.write_text(content, encoding="utf-8")
    98→        return path
    99→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


Now let me read a few more key files:


     1→# AI Use Case Daily Scout — Milestone 2 实录
     2→
     3→> 完成日期：2026-02-28
     4→> 状态：✅ 三数据源全部打通，按兴趣方向优化关键词，按热度排序，已配置每日定时运行
     5→
     6→---
     7→
     8→## 一、Milestone 1 → 2 解决了什么
     9→
    10→Milestone 1 完成时的状态：只有 HN 能跑通，Reddit 全部 403，没有 GitHub，每天产出 3-5 条内容。
    11→
    12→Milestone 2 目标：**三个数据源全部打通 + 内容精准对齐用户兴趣 + 生产化部署**。
    13→
    14→```
    15→之前（M1）:  HN(3-5条) + Reddit(0) + GitHub(无) → 3 条有价值
    16→之后（M2）:  HN(400+) + Reddit(精准子版块) + GitHub(125+) → 每日 30-40 条有价值
    17→```
    18→
    19→---
    20→
    21→## 二、Phase 2 新增功能全记录
    22→
    23→### 2.1 GitHub Trending 采集器
    24→
    25→新增 `collectors/github.py`，使用 GitHub Search API，按 topic 搜索最近有推送的 AI 相关仓库。
    26→
    27→**关键设计**：GitHub Search API 不支持 `topic:X OR topic:Y` 语法（返回 422），必须**每个 topic 单独发一次请求**，再按 `repo.id` 去重合并。
    28→
    29→```python
    30→_TOPICS = ["openclaw", "claude-code", "vibe-coding", "mcp",
    31→           "ai-agent", "local-llm", "ollama", "ai-saas"]
    32→```
    33→
    34→采集到的字段：仓库名、描述、topics、语言、star 数，拼成 `raw_content` 供 LLM 分析。
    35→
    36→---
    37→
    38→### 2.2 Reddit：从 OAuth2 到 Exa.ai
    39→
    40→**OAuth2 方案（已废弃）**：Reddit 在 2023 年 API 政策收紧后，公共 JSON 端点全面封锁服务器 IP。虽然实现了 OAuth2 application-only 流程（client_id + client_secret → Bearer token），但维护成本高、Reddit 风控策略不稳定，最终放弃。
    41→
    42→**新方案：Exa.ai Python SDK**
    43→
    44→```python
    45→from exa_py import Exa
    46→
    47→exa = Exa(api_key=EXA_API_KEY)
    48→response = exa.search_and_contents(
    49→    query,
    50→    include_domains=[f"reddit.com/r/{subreddit}"],
    51→    start_published_date=start_date,   # 过去 24 小时
    52→    text=True,
    53→    num_results=10,
    54→)
    55→```
    56→
    57→**踩坑：Exa 的 `include_domains` 不严格过滤**
    58→
    59→Exa 将 `include_domains` 当作软提示而非硬约束，实际返回的 URL 中混入了 dev.to、blockchain.news 等非 Reddit 链接。
    60→
    61→**修复**：采集后对 URL 做硬过滤：
    62→
    63→```python
    64→if f"/r/{sub.lower()}/" not in url.lower():
    65→    continue
    66→```
    67→
    68→**架构升级：从通用查询改为按子版块精准抓取**
    69→
    70→旧方案：4 条通用查询语句搜全 reddit.com，信噪比低。
    71→新方案：每个目标子版块单独发一次请求，锁定精准社区。
    72→
    73→```python
    74→_DEFAULT_SUBREDDITS = ["openclaw", "vibecoding", "ClaudeCode", "singularity"]
    75→```
    76→
    77→`.env` 新增配置：
    78→
    79→```
    80→EXA_API_KEY=your_exa_api_key
    81→REDDIT_SUBREDDITS=["openclaw","vibecoding","ClaudeCode","singularity"]
    82→```
    83→
    84→---
    85→
    86→### 2.3 修复采集漏斗截断 Bug
    87→
    88→**Bug**：原始代码将三个来源直接拼接后取前 N 条：
    89→
    90→```python
    91→all_items = hn_items + reddit_items + github_items
    92→all_items = all_items[:max_items_per_run]  # HN 有 479 条，前 N 条全是 HN
    93→```
    94→
    95→HN 返回 479 条，Reddit/GitHub 被完全截断，永远不会被处理。
    96→
    97→**修复**：改为三源**轮询交替**（round-robin interleave）再截断：
    98→
    99→```python
   100→interleaved = [
   101→    item
   102→    for item in itertools.chain.from_iterable(
   103→        itertools.zip_longest(hn_items, reddit_items, github_items)
   104→    )
   105→    if item is not None
   106→]
   107→all_items = interleaved[:max_items_per_run]
   108→# 结果：{'hackernews': 39, 'reddit': 3, 'github': 38} — 三源均衡
   109→```
   110→
   111→---
   112→
   113→### 2.4 关键词精准化：对齐兴趣方向
   114→
   115→根据实际关注方向，全面替换三处关键词配置。
   116→
   117→**关注方向**：
   118→- 🔧 **OpenClaw** — 本地优先个人 AI 助手（TypeScript，多消息渠道，50+ 服务集成）
   119→- 💻 **AI 编程** — Cursor、Claude Code、vibe coding 工作流
   120→- 🧑💼 **超级个体** — AI 辅助的一人公司、独立开发者
   121→- 🏢 **AI 商业模式** — AI SaaS、AI 产品变现
   122→- 🌍 **AI 落地场景** — 真实生产环境中的 AI 工作流
   123→
   124→**HN 关键词**（旧 → 新）：
   125→
   126→| 移除（噪音高） | 新增（精准） |
   127→|---|---|
   128→| `claude`（太泛） | `openclaw` |
   129→| `llm workflow` | `claude code` |
   130→| `copilot` | `solopreneur ai` |
   131→| `rag pipeline` | `ai saas` |
   132→| `langchain` | `ai workflow` |
   133→| `local llm` | `local ai`、`ai use case` |
   134→
   135→**GitHub Topics**（旧 → 新）：
   136→
   137→| 移除 | 新增 |
   138→|---|---|
   139→| `llm`、`rag`、`langchain` | `openclaw`、`claude-code`、`vibe-coding`、`ai-saas` |
   140→
   141→**Reddit 子版块**：
   142→
   143→```
   144→r/openclaw    — OpenClaw 官方社区，使用案例、插件开发
   145→r/vibecoding  — AI 辅助编程工作流讨论
   146→r/ClaudeCode  — Claude Code CLI 专题
   147→r/singularity — AI 技术奇点、超级个体话题
   148→```
   149→
   150→---
   151→
   152→### 2.5 按热度排序：高 up 优先处理
   153→
   154→**问题**：轮询虽然保证了平台多样性，但每个平台内部按 Algolia 返回时间排序，高质量内容可能排在后面被截断丢弃。
   155→
   156→**方案**：给 `RawItem` 增加 `score` 字段，各平台填入对应的社区热度指标：
   157→
   158→| 平台 | score 来源 |
   159→|------|-----------|
   160→| HackerNews | Algolia 返回的 `points`（点赞数） |
   161→| GitHub | `stargazers_count`（star 数） |
   162→| Reddit | `0`（Exa 不返回投票数） |
   163→
   164→在轮询截断之前，**先对每个平台的列表内部按分数降序排序**：
   165→
   166→```python
   167→hn_items.sort(key=lambda x: x.score, reverse=True)
   168→github_items.sort(key=lambda x: x.score, reverse=True)
   169→# 然后再轮询交替
   170→```
   171→
   172→效果：原来可能被截断的 968pt 帖子（`How I use Claude Code: Separation of planning and execution`）现在稳定出现在报告中。
   173→
   174→---
   175→
   176→## 三、架构演进对比
   177→
   178→```
   179→Milestone 1:
   180→  HN ──► 拼接 ──► 取前N条 ──► LLM 三步 ──► SQLite ──► Obsidian
   181→  Reddit（全部 403，跳过）
   182→
   183→Milestone 2:
   184→  HN(按points排序) ──┐
   185→  Reddit(Exa, 按子版块) ──► 轮询交替 ──► 取前N条 ──► LLM 三步 ──► SQLite ──► Obsidian
   186→  GitHub(按stars排序) ──┘
   187→```
   188→
   189→---
   190→
   191→## 四、Milestone 2 最终运行效果
   192→
   193→```
   194→HN: 444 items | Reddit: 3 items | GitHub: 125 items
   195→Interleaved: {'hackernews': 39, 'reddit': 3, 'github': 38}
   196→→ 处理 top 80 (round-robin)
   197→→ 36 条有价值 → 新场景: 36 个
   198→```
   199→
   200→典型输出（2026-02-27）：
   201→
   202→| 场景 | 内容 |
   203→|------|------|
   204→| AI 编程 | `How I use Claude Code: Separation of planning and execution`（968pt） |
   205→| AI 编程 | r/vibecoding `2个月AI代理编程经验总结` |
   206→| OpenClaw | `openclaw/openclaw` 本体 + NanoClaw 对比项目 |
   207→| 超级个体 | `I quit coding years ago. AI brought me back` — 用 Claude+Windsurf 2周建 60 个计算器 |
   208→| AI 工作流 | `n8n`、`dify`、`activepieces` 等开源工作流平台 |
   209→| 本地 AI | `LocalAI`、`anything-llm`、`LocalGPT`（Rust 版） |
   210→
   211→---
   212→
   213→## 五、测试覆盖
   214→
   215→```
   216→86 passed in 4.54s
   217→```
   218→
   219→| 模块 | 测试数 | 关键覆盖 |
   220→|------|--------|---------|
   221→| models | 18 | Pydantic 约束、StrEnum 映射 |
   222→| storage | 15 | SQLite CRUD、JSON 序列化 |
   223→| collectors | 24 | HN(7) + Reddit(9) + GitHub(7)，全部 mock |
   224→| processors | 11 | LLM 三步、JSON 容错 |
   225→| reporters | 12 | Markdown 生成格式 |
   226→| main | 6 | 编排器集成、dry-run |
   227→
   228→Reddit 测试全面升级为 `unittest.mock.patch` 模式（mock `Exa` 类），新增：
   229→- `test_filters_out_wrong_subreddit_urls` — 验证 Exa 域名过滤硬保障
   230→- `test_passes_subreddit_domain_to_exa` — 验证每个子版块使用独立 domain
   231→
   232→---
   233→
   234→## 六、生产化部署
   235→
   236→### 6.1 配置 .env
   237→
   238→```bash
   239→cp .env.example .env
   240→# 必填：
   241→ANTHROPIC_API_KEY=sk-ant-...
   242→EXA_API_KEY=...           # Exa.ai API Key（用于 Reddit 抓取）
   243→# 选填：
   244→GITHUB_TOKEN=ghp_...      # 提升 GitHub API 限额：60 → 5000 次/小时
   245→ANTHROPIC_BASE_URL=...    # 代理地址
   246→```
   247→
   248→### 6.2 写入 Crontab（每天早 7:00 自动运行）
   249→
   250→```bash
   251→crontab -e
   252→```
   253→
   254→加入以下一行：
   255→
   256→```
   257→0 7 * * * /root/projects/ai_usecases_explorer/venv/bin/python -m ai_usecases_explorer.main >> /var/log/ai_usecases_scout.log 2>&1
   258→```
   259→
   260→验证：
   261→
   262→```bash
   263→crontab -l   # 确认条目存在
   264→# 查看运行日志：
   265→tail -f /var/log/ai_usecases_scout.log
   266→```
   267→
   268→当前已写入的 crontab：
   269→
   270→```
   271→*/5 * * * * flock -xn /tmp/stargate.lock ...   # 原有条目
   272→0 7 * * * /root/projects/ai_usecases_explorer/venv/bin/python -m ai_usecases_explorer.main >> /var/log/ai_usecases_scout.log 2>&1
   273→```
   274→
   275→### 6.3 日常操作
   276→
   277→```bash
   278→source venv/bin/activate
   279→
   280→# 快速试跑（不写 DB，top 20 条）
   281→MAX_ITEMS_PER_RUN=20 python -m ai_usecases_explorer.main --dry-run
   282→
   283→# 完整试跑（不写 DB，top 80 条）
   284→python -m ai_usecases_explorer.main --dry-run
   285→
   286→# 正式运行（写 DB + 生成报告）
   287→python -m ai_usecases_explorer.main
   288→
   289→# 补跑历史某天
   290→python -m ai_usecases_explorer.main --date 2026-02-26
   291→
   292→# 查看运行日志
   293→tail -f /var/log/ai_usecases_scout.log
   294→```
   295→
   296→### 6.4 可调参数（.env）
   297→
   298→| 变量 | 默认值 | 说明 |
   299→|------|--------|------|
   300→| `ANTHROPIC_API_KEY` | 必填 | Claude API Key |
   301→| `EXA_API_KEY` | 必填 | Exa.ai Key，用于 Reddit 子版块搜索 |
   302→| `ANTHROPIC_BASE_URL` | （空） | 代理地址 |
   303→| `CLAUDE_MODEL` | `claude-haiku-4-5-20251001` | 改 sonnet 质量更高但更贵 |
   304→| `HN_MIN_POINTS` | `10` | HN 最低分数门槛 |
   305→| `GITHUB_TOKEN` | （空） | 填入后 GitHub API 限额 60→5000/小时 |
   306→| `GITHUB_MIN_STARS` | `10` | GitHub 最低 star 数 |
   307→| `REDDIT_SUBREDDITS` | `["openclaw","vibecoding","ClaudeCode","singularity"]` | 监控的子版块列表 |
   308→| `MAX_ITEMS_PER_RUN` | `80` | 每次最多处理条数，控制 LLM 调用成本 |
   309→| `DB_PATH` | `data/usecases.db` | SQLite 文件位置 |
   310→| `OBSIDIAN_REPORT_DIR` | `/root/vault/.../auto_report` | 晨报输出目录 |
   311→
   312→---
   313→
   314→## 七、当前已知限制与 Phase 3 候选
   315→
   316→| 优先级 | 方向 | 说明 |
   317→|--------|------|------|
   318→| 高 | Reddit 内容量不足 | Exa 对小众子版块（r/openclaw、r/ClaudeCode）24h 内索引条数少，可考虑延长到 48h 或增加子版块 |
   319→| 高 | 价值过滤误判 | 部分 GitHub 仓库（如 Micasa）通过了价值过滤但与 AI 关联较弱，需优化 prompt |
   320→| 中 | Twitter/X | API 方案待确定 |
   321→| 中 | 相似度去重优化 | 目前 Deduplicator 依赖 LLM 语义判断，可引入向量嵌入加速 |
   322→| 低 | 报告多语言支持 | 摘要目前为中文，可加 `REPORT_LANGUAGE` 配置 |
   323→
   324→---
   325→
   326→## 八、技术栈（M2 新增）
   327→
   328→| 层次 | M1 | M2 新增 |
   329→|------|-----|---------|
   330→| Reddit 采集 | 公共 JSON API（已废弃）| **exa-py SDK**（Exa.ai 神经搜索） |
   331→| GitHub 采集 | 无 | **GitHub Search API**（httpx） |
   332→| 排序策略 | 无（时间顺序） | **按 score 排序**（HN points / GitHub stars） |
   333→| 截断策略 | 顺序取前 N | **轮询交替**（itertools.zip_longest） |
   334→| 测试 mock | pytest-httpx | pytest-httpx + **unittest.mock.patch（Exa）** |
   335→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Reddit collector using Exa.ai neural search, one request per subreddit."""
     2→import logging
     3→from datetime import UTC, datetime, timedelta
     4→
     5→from exa_py import Exa  # type: ignore[import-untyped]
     6→
     7→from ai_usecases_explorer.collectors.base import BaseCollector, RawItem
     8→
     9→logger = logging.getLogger(__name__)
    10→
    11→# Broad AI query — semantically covers all target subreddits
    12→_SEARCH_QUERY = "AI workflow automation use case tool productivity"
    13→
    14→_DEFAULT_SUBREDDITS = ["openclaw", "vibecoding", "ClaudeCode", "singularity"]
    15→
    16→
    17→class RedditCollector(BaseCollector):
    18→    """Fetches posts from specific subreddits via Exa.ai (one request per subreddit)."""
    19→
    20→    def __init__(
    21→        self,
    22→        subreddits: list[str],
    23→        api_key: str = "",
    24→    ) -> None:
    25→        self._subreddits = subreddits
    26→        self._api_key = api_key
    27→
    28→    def fetch(self) -> list[RawItem]:
    29→        if not self._api_key:
    30→            logger.warning("EXA_API_KEY not set — skipping Reddit/Exa collection")
    31→            return []
    32→
    33→        exa = Exa(api_key=self._api_key)
    34→        start_date = (datetime.now(tz=UTC) - timedelta(hours=24)).strftime("%Y-%m-%dT%H:%M:%SZ")
    35→
    36→        seen: set[str] = set()
    37→        items: list[RawItem] = []
    38→
    39→        for sub in self._subreddits:
    40→            try:
    41→                response = exa.search_and_contents(
    42→                    _SEARCH_QUERY,
    43→                    include_domains=[f"reddit.com/r/{sub}"],
    44→                    start_published_date=start_date,
    45→                    text=True,
    46→                    num_results=10,
    47→                )
    48→            except Exception as exc:
    49→                logger.warning("Exa search failed for r/%s: %s", sub, exc)
    50→                continue
    51→
    52→            for result in response.results:
    53→                url = result.url or ""
    54→                if not url or url in seen:
    55→                    continue
    56→                # Post-filter: Exa doesn't strictly enforce include_domains
    57→                if f"/r/{sub.lower()}/" not in url.lower():
    58→                    continue
    59→                seen.add(url)
    60→
    61→                title = result.title or url
    62→                text = result.text or ""
    63→                raw = f"{title}\n\n{text}".strip()
    64→
    65→                published_at = _parse_date(getattr(result, "published_date", None))
    66→
    67→                items.append(
    68→                    RawItem(
    69→                        title=title,
    70→                        source_url=url,
    71→                        source_platform="reddit",
    72→                        author=getattr(result, "author", None) or "unknown",
    73→                        published_at=published_at,
    74→                        raw_content=raw,
    75→                    )
    76→                )
    77→
    78→        return items
    79→
    80→
    81→def _parse_date(value: str | None) -> datetime:
    82→    """Parse an ISO-8601 date string, falling back to now(UTC)."""
    83→    if not value:
    84→        return datetime.now(tz=UTC)
    85→    try:
    86→        return datetime.fromisoformat(value.replace("Z", "+00:00"))
    87→    except ValueError:
    88→        return datetime.now(tz=UTC)
    89→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Pydantic data models for AI use cases."""
     2→import hashlib
     3→from datetime import datetime
     4→from enum import StrEnum
     5→
     6→from pydantic import BaseModel, Field
     7→
     8→
     9→def make_id(source_url: str) -> str:
    10→    """Return first 12 hex chars of sha256(source_url)."""
    11→    return hashlib.sha256(source_url.encode()).hexdigest()[:12]
    12→
    13→
    14→class ScenarioType(StrEnum):
    15→    LOCAL_DATA_INTEGRATION = "本地数据集成"
    16→    CODE_GEN_REVIEW = "代码生成与审查"
    17→    WORKFLOW_AUTOMATION = "工作流自动化"
    18→    DOCUMENT_PROCESSING = "文档与内容处理"
    19→    TERMINAL_CLI = "终端与CLI增强"
    20→    CREATIVE_PRODUCTION = "创意内容生产"
    21→    DATA_ANALYSIS = "数据分析与可视化"
    22→    PERSONAL_KNOWLEDGE = "个人知识管理"
    23→    BUSINESS_PROCESS = "商业流程自动化"
    24→    OTHER = "其他"
    25→
    26→
    27→class Novelty(StrEnum):
    28→    NEW = "new"
    29→    SIMILAR = "similar"
    30→    REPEAT = "repeat"
    31→
    32→
    33→class UseCase(BaseModel):
    34→    id: str
    35→    title: str
    36→    summary: str
    37→    source_url: str
    38→    source_platform: str
    39→    author: str
    40→    published_at: datetime
    41→    collected_at: datetime
    42→    scenario_type: ScenarioType
    43→    tools_used: list[str]
    44→    has_local_data: bool
    45→    has_workflow_opt: bool
    46→    value_score: float = Field(ge=0.0, le=1.0)
    47→    is_real_value: bool
    48→    novelty: Novelty
    49→    similar_to_ids: list[str]
    50→    raw_content: str
    51→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


Perfect! Now let me get the full directory tree and check for any PLAN.md or SPEC.md files:


find /root/projects/ai_usecases_explorer -type f \( -name "PLAN.md" -o -name "SPEC.md" -o -name "*.md" \) | grep -v venv | sort


Now let me check other key processor files and the pyproject.toml:


     1→"""Step 1 — Value filter: determines if a raw item has real AI use-case value."""
     2→import anthropic
     3→
     4→from ai_usecases_explorer.collectors.base import RawItem
     5→from ai_usecases_explorer.processors.utils import parse_llm_json
     6→
     7→_SYSTEM = """\
     8→You are an AI use case filter. Your job is to decide whether a post represents
     9→a genuinely valuable AI use case.
    10→
    11→PASS criteria (at least one must be true):
    12→- The user integrates AI with local/personal data (files, DB, notes, etc.)
    13→- The user automates a specific workflow step — not just asking questions
    14→
    15→REJECT criteria:
    16→- Pure Q&A replacing ChatGPT with no local data or workflow change
    17→- Marketing demos without real user implementation
    18→- Generic "AI is great" commentary
    19→
    20→Respond with ONLY valid JSON (no markdown fences), exactly this shape:
    21→{"is_real_value": bool, "reason": "...", "has_local_data": bool, "has_workflow_opt": bool}
    22→"""
    23→
    24→
    25→class ValueFilter:
    26→    """Calls Claude to decide whether a RawItem has real use-case value."""
    27→
    28→    def __init__(
    29→        self, client: anthropic.Anthropic, model: str = "claude-haiku-4-5-20251001"
    30→    ) -> None:
    31→        self._client = client
    32→        self._model = model
    33→
    34→    def run(self, item: RawItem) -> dict:
    35→        """Return a dict with is_real_value, has_local_data, has_workflow_opt, reason."""
    36→        return self._call_llm(item)
    37→
    38→    def _call_llm(self, item: RawItem) -> dict:
    39→        content = f"Title: {item.title}\n\nContent:\n{item.raw_content[:3000]}"
    40→        message = self._client.messages.create(
    41→            model=self._model,
    42→            max_tokens=256,
    43→            system=_SYSTEM,
    44→            messages=[{"role": "user", "content": content}],
    45→        )
    46→        return parse_llm_json(message.content[0].text)
    47→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Abstract base class for all data collectors."""
     2→from abc import ABC, abstractmethod
     3→from dataclasses import dataclass
     4→from datetime import datetime
     5→
     6→
     7→@dataclass
     8→class RawItem:
     9→    """Minimal raw item returned by collectors before LLM processing."""
    10→
    11→    title: str
    12→    source_url: str
    13→    source_platform: str
    14→    author: str
    15→    published_at: datetime
    16→    raw_content: str
    17→    score: int = 0  # platform engagement: HN points, GitHub stars, 0 for Reddit
    18→
    19→
    20→class BaseCollector(ABC):
    21→    """Every collector must implement fetch() and return a list of RawItems."""
    22→
    23→    @abstractmethod
    24→    def fetch(self) -> list[RawItem]:
    25→        """Fetch items from the source and return filtered raw items."""
    26→        ...
    27→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→[build-system]
     2→requires = ["hatchling"]
     3→build-backend = "hatchling.build"
     4→
     5→[project]
     6→name = "ai-usecases-explorer"
     7→version = "0.1.0"
     8→description = "Daily AI use case radar — filters noise, surfaces real value"
     9→requires-python = ">=3.11"
    10→dependencies = [
    11→    "anthropic>=0.40.0",
    12→    "exa-py>=1.0.0",
    13→    "httpx>=0.27.0",
    14→    "pydantic>=2.0.0",
    15→    "pydantic-settings>=2.0.0",
    16→    "rich>=13.0.0",
    17→]
    18→
    19→[project.optional-dependencies]
    20→dev = [
    21→    "pytest>=8.0.0",
    22→    "pytest-asyncio>=0.23.0",
    23→    "pytest-httpx>=0.30.0",
    24→    "ruff>=0.3.0",
    25→    "mypy>=1.9.0",
    26→]
    27→
    28→[project.scripts]
    29→ai-scout = "ai_usecases_explorer.main:main"
    30→
    31→[tool.hatch.build.targets.wheel]
    32→packages = ["src/ai_usecases_explorer"]
    33→
    34→[tool.pytest.ini_options]
    35→asyncio_mode = "auto"
    36→testpaths = ["tests"]
    37→markers = [
    38→    "integration: marks tests as integration tests (require real API keys)",
    39→]
    40→
    41→[tool.ruff]
    42→line-length = 100
    43→target-version = "py311"
    44→
    45→[tool.ruff.lint]
    46→select = ["E", "F", "I", "UP"]
    47→
    48→[tool.ruff.lint.per-file-ignores]
    49→"tests/*" = ["E501"]
    50→
    51→[tool.mypy]
    52→python_version = "3.11"
    53→strict = true
    54→ignore_missing_imports = true
    55→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


Now let me create a visual tree structure and compile everything:


tree -L 3 /root/projects/ai_usecases_explorer -I 'venv|.pytest_cache|.ruff_cache|__pycache__|*.egg-info|.git' --charset ascii 2>/dev/null || find /root/projects/ai_usecases_explorer -type d \( -name venv -o -name __pycache__ -o -name .pytest_cache \) -prune -o -type f -print | head -50


/root/projects/ai_usecases_explorer
|-- AI_VIEWS.md
|-- CLAUDE.md
|-- data
|   `-- usecases.db
|-- Makefile
|-- MILESTONE_1.md
|-- MILESTONE_2.md
|-- pyproject.toml
|-- src
|   `-- ai_usecases_explorer
|       |-- collectors
|       |-- __init__.py
|       |-- main.py
|       |-- models
|       |-- processors
|       |-- reporters
|       |-- settings.py
|       `-- storage
`-- tests
    |-- __init__.py
    |-- test_collectors.py
    |-- test_main.py
    |-- test_models.py
    |-- test_processors.py
    |-- test_reporters.py
    `-- test_storage.py

9 directories, 17 files


Perfect! Now let me get the complete file structure:


find /root/projects/ai_usecases_explorer -type f \( -name "*.py" -o -name "*.md" -o -name "*.toml" -o -name "Makefile" -o -name ".env*" \) ! -path "*/venv/*" ! -path "*/__pycache__/*" ! -path "*/.pytest_cache/*" | sort


Perfect! Let me read a couple more key processor files to complete the picture:


     1→"""Step 2 — Classifier: assigns scenario type, tools, score, and summary."""
     2→import anthropic
     3→
     4→from ai_usecases_explorer.collectors.base import RawItem
     5→from ai_usecases_explorer.models.usecase import ScenarioType
     6→from ai_usecases_explorer.processors.utils import parse_llm_json
     7→
     8→_SCENARIO_VALUES = "\n".join(f'- "{s.value}"' for s in ScenarioType)
     9→
    10→_SYSTEM = f"""\
    11→You are an AI use case classifier. Classify the post into the most appropriate
    12→scenario type, extract the tools mentioned, give a value score, and write a
    13→2-3 sentence summary in Chinese.
    14→
    15→Scenario types (use EXACTLY one of these string values):
    16→{_SCENARIO_VALUES}
    17→
    18→Respond with ONLY valid JSON (no markdown fences), exactly this shape:
    19→{{"scenario_type": "...", "tools_used": ["..."], "value_score": 0.0, "summary": "..."}}
    20→
    21→value_score: 0.0–1.0. Higher = more practical, more local-data usage, clearer workflow benefit.
    22→"""
    23→
    24→
    25→class Classifier:
    26→    """Calls Claude to classify a RawItem and generate its summary."""
    27→
    28→    def __init__(
    29→        self, client: anthropic.Anthropic, model: str = "claude-haiku-4-5-20251001"
    30→    ) -> None:
    31→        self._client = client
    32→        self._model = model
    33→
    34→    def run(self, item: RawItem) -> dict:
    35→        """Return scenario_type (ScenarioType enum), tools_used, value_score, summary."""
    36→        raw = self._call_llm(item)
    37→        # Map string value to enum, fall back to OTHER if unknown
    38→        scenario_str = raw.get("scenario_type", "")
    39→        if not isinstance(scenario_str, ScenarioType):
    40→            try:
    41→                raw["scenario_type"] = ScenarioType(scenario_str)
    42→            except ValueError:
    43→                raw["scenario_type"] = ScenarioType.OTHER
    44→        return raw
    45→
    46→    def _call_llm(self, item: RawItem) -> dict:
    47→        content = f"Title: {item.title}\n\nContent:\n{item.raw_content[:3000]}"
    48→        message = self._client.messages.create(
    49→            model=self._model,
    50→            max_tokens=512,
    51→            system=_SYSTEM,
    52→            messages=[{"role": "user", "content": content}],
    53→        )
    54→        return parse_llm_json(message.content[0].text)
    55→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Step 3 — Deduplicator: detects novelty by comparing against existing DB records."""
     2→import anthropic
     3→
     4→from ai_usecases_explorer.models.usecase import Novelty, UseCase
     5→from ai_usecases_explorer.processors.utils import parse_llm_json
     6→
     7→_SYSTEM = """\
     8→You are a novelty detector for AI use cases. Compare the new use case against
     9→the list of existing use cases (summaries only).
    10→
    11→Novelty levels:
    12→- "new": represents a genuinely new scenario category not seen before
    13→- "similar": same category as existing cases, but a new concrete example
    14→- "repeat": essentially duplicate content already in the database
    15→
    16→Respond with ONLY valid JSON (no markdown fences), exactly this shape:
    17→{"novelty": "new"|"similar"|"repeat", "similar_to_ids": ["..."], "novelty_reason": "..."}
    18→
    19→similar_to_ids: IDs from the existing list that this new use case resembles.
    20→Leave empty for "new".
    21→"""
    22→
    23→
    24→class Deduplicator:
    25→    """Compares a UseCase against existing records to assess novelty."""
    26→
    27→    def __init__(
    28→        self, client: anthropic.Anthropic, model: str = "claude-haiku-4-5-20251001"
    29→    ) -> None:
    30→        self._client = client
    31→        self._model = model
    32→
    33→    def run(self, usecase: UseCase, existing: list[UseCase]) -> dict:
    34→        """Return novelty, similar_to_ids, novelty_reason.
    35→
    36→        Skips LLM call if existing list is empty (always NEW).
    37→        """
    38→        if not existing:
    39→            return {
    40→                "novelty": Novelty.NEW,
    41→                "similar_to_ids": [],
    42→                "novelty_reason": "No existing records.",
    43→            }
    44→        raw = self._call_llm(usecase, existing)
    45→        if not isinstance(raw.get("novelty"), Novelty):
    46→            raw["novelty"] = Novelty(raw["novelty"])
    47→        return raw
    48→
    49→    def _call_llm(self, usecase: UseCase, existing: list[UseCase]) -> dict:
    50→        existing_summaries = "\n".join(
    51→            f'- id={uc.id}: {uc.summary}' for uc in existing[:50]
    52→        )
    53→        content = (
    54→            f"New use case:\nTitle: {usecase.title}\nSummary: {usecase.summary}\n\n"
    55→            f"Existing use cases:\n{existing_summaries}"
    56→        )
    57→        message = self._client.messages.create(
    58→            model=self._model,
    59→            max_tokens=256,
    60→            system=_SYSTEM,
    61→            messages=[{"role": "user", "content": content}],
    62→        )
    63→        return parse_llm_json(message.content[0].text)
    64→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""Shared utilities for LLM processors."""
     2→import json
     3→import re
     4→
     5→
     6→def parse_llm_json(text: str) -> dict:
     7→    """Parse JSON from LLM output, stripping markdown code fences if present."""
     8→    text = text.strip()
     9→    if not text:
    10→        raise ValueError("LLM returned empty response")
    11→    # Strip ```json ... ``` or ``` ... ``` fences
    12→    text = re.sub(r"^```(?:json)?\s*", "", text)
    13→    text = re.sub(r"\s*```$", "", text)
    14→    return json.loads(text.strip())
    15→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


Excellent! Now let me read a couple more key files to complete the picture:


     1→"""SQLite storage for UseCase records."""
     2→import json
     3→import sqlite3
     4→from pathlib import Path
     5→
     6→from ai_usecases_explorer.models.usecase import Novelty, ScenarioType, UseCase
     7→
     8→_CREATE_TABLE = """
     9→CREATE TABLE IF NOT EXISTS usecases (
    10→    id TEXT PRIMARY KEY,
    11→    title TEXT NOT NULL,
    12→    summary TEXT NOT NULL,
    13→    source_url TEXT NOT NULL,
    14→    source_platform TEXT NOT NULL,
    15→    author TEXT NOT NULL,
    16→    published_at TEXT NOT NULL,
    17→    collected_at TEXT NOT NULL,
    18→    scenario_type TEXT NOT NULL,
    19→    tools_used TEXT NOT NULL,
    20→    has_local_data INTEGER NOT NULL,
    21→    has_workflow_opt INTEGER NOT NULL,
    22→    value_score REAL NOT NULL,
    23→    is_real_value INTEGER NOT NULL,
    24→    novelty TEXT NOT NULL,
    25→    similar_to_ids TEXT NOT NULL,
    26→    raw_content TEXT NOT NULL
    27→)
    28→"""
    29→
    30→
    31→class Database:
    32→    """Thin SQLite wrapper for UseCase persistence."""
    33→
    34→    def __init__(self, db_path: Path) -> None:
    35→        self._path = db_path
    36→        db_path.parent.mkdir(parents=True, exist_ok=True)
    37→        self._conn = sqlite3.connect(str(db_path))
    38→        self._conn.row_factory = sqlite3.Row
    39→        self._conn.execute(_CREATE_TABLE)
    40→        self._conn.commit()
    41→
    42→    def save(self, uc: UseCase) -> str:
    43→        """Insert or replace a UseCase. Returns its id."""
    44→        self._conn.execute(
    45→            """
    46→            INSERT OR REPLACE INTO usecases VALUES (
    47→                :id, :title, :summary, :source_url, :source_platform,
    48→                :author, :published_at, :collected_at, :scenario_type,
    49→                :tools_used, :has_local_data, :has_workflow_opt,
    50→                :value_score, :is_real_value, :novelty, :similar_to_ids,
    51→                :raw_content
    52→            )
    53→            """,
    54→            {
    55→                "id": uc.id,
    56→                "title": uc.title,
    57→                "summary": uc.summary,
    58→                "source_url": uc.source_url,
    59→                "source_platform": uc.source_platform,
    60→                "author": uc.author,
    61→                "published_at": uc.published_at.isoformat(),
    62→                "collected_at": uc.collected_at.isoformat(),
    63→                "scenario_type": uc.scenario_type.value,
    64→                "tools_used": json.dumps(uc.tools_used, ensure_ascii=False),
    65→                "has_local_data": int(uc.has_local_data),
    66→                "has_workflow_opt": int(uc.has_workflow_opt),
    67→                "value_score": uc.value_score,
    68→                "is_real_value": int(uc.is_real_value),
    69→                "novelty": uc.novelty.value,
    70→                "similar_to_ids": json.dumps(uc.similar_to_ids),
    71→                "raw_content": uc.raw_content,
    72→            },
    73→        )
    74→        self._conn.commit()
    75→        return uc.id
    76→
    77→    def exists(self, id: str) -> bool:
    78→        """Return True if a UseCase with this id is already stored."""
    79→        row = self._conn.execute(
    80→            "SELECT 1 FROM usecases WHERE id = ?", (id,)
    81→        ).fetchone()
    82→        return row is not None
    83→
    84→    def get_by_scenario(
    85→        self, scenario_type: ScenarioType, limit: int = 100
    86→    ) -> list[UseCase]:
    87→        """Return up to `limit` UseCases matching the given scenario type."""
    88→        rows = self._conn.execute(
    89→            "SELECT * FROM usecases WHERE scenario_type = ? ORDER BY collected_at DESC LIMIT ?",
    90→            (scenario_type.value, limit),
    91→        ).fetchall()
    92→        return [self._row_to_usecase(r) for r in rows]
    93→
    94→    def get_recent(self, limit: int = 200) -> list[UseCase]:
    95→        """Return the most recently collected UseCases."""
    96→        rows = self._conn.execute(
    97→            "SELECT * FROM usecases ORDER BY collected_at DESC LIMIT ?",
    98→            (limit,),
    99→        ).fetchall()
   100→        return [self._row_to_usecase(r) for r in rows]
   101→
   102→    @staticmethod
   103→    def _row_to_usecase(row: sqlite3.Row) -> UseCase:
   104→        from datetime import datetime
   105→
   106→        return UseCase(
   107→            id=row["id"],
   108→            title=row["title"],
   109→            summary=row["summary"],
   110→            source_url=row["source_url"],
   111→            source_platform=row["source_platform"],
   112→            author=row["author"],
   113→            published_at=datetime.fromisoformat(row["published_at"]),
   114→            collected_at=datetime.fromisoformat(row["collected_at"]),
   115→            scenario_type=ScenarioType(row["scenario_type"]),
   116→            tools_used=json.loads(row["tools_used"]),
   117→            has_local_data=bool(row["has_local_data"]),
   118→            has_workflow_opt=bool(row["has_workflow_opt"]),
   119→            value_score=row["value_score"],
   120→            is_real_value=bool(row["is_real_value"]),
   121→            novelty=Novelty(row["novelty"]),
   122→            similar_to_ids=json.loads(row["similar_to_ids"]),
   123→            raw_content=row["raw_content"],
   124→        )
   125→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""HackerNews collector using the Algolia search API."""
     2→from datetime import UTC, datetime
     3→
     4→import httpx
     5→
     6→from ai_usecases_explorer.collectors.base import BaseCollector, RawItem
     7→
     8→_HN_API = "https://hn.algolia.com/api/v1/search_by_date"
     9→
    10→# One query per term — Algolia ANDs multi-word queries, so run them separately.
    11→# Keywords cover: OpenClaw, AI编程, 超级个体, AI落地场景, AI商业模式
    12→_KEYWORDS = [
    13→    "openclaw",
    14→    "claude code",
    15→    "vibe coding",
    16→    "cursor",
    17→    "mcp server",
    18→    "ai agent",
    19→    "ollama",
    20→    "solopreneur ai",
    21→    "ai saas",
    22→    "ai workflow",
    23→    "local ai",
    24→    "ai use case",
    25→]
    26→
    27→
    28→class HackerNewsCollector(BaseCollector):
    29→    """Fetches recent HN posts matching AI/LLM keywords above a point threshold."""
    30→
    31→    def __init__(self, min_points: int = 10) -> None:
    32→        self._min_points = min_points
    33→
    34→    def fetch(self) -> list[RawItem]:
    35→        seen_ids: set[str] = set()
    36→        items: list[RawItem] = []
    37→        for keyword in _KEYWORDS:
    38→            params = {
    39→                "query": keyword,
    40→                "tags": "(story,ask_hn,show_hn)",
    41→                "numericFilters": f"points>={self._min_points}",
    42→                "hitsPerPage": 50,
    43→            }
    44→            response = httpx.get(_HN_API, params=params, timeout=15)
    45→            response.raise_for_status()
    46→            for hit in response.json().get("hits", []):
    47→                hn_id = hit.get("objectID", "")
    48→                if hn_id in seen_ids:
    49→                    continue
    50→                if hit.get("points", 0) >= self._min_points:
    51→                    seen_ids.add(hn_id)
    52→                    items.append(self._parse(hit))
    53→        return items
    54→
    55→    def _parse(self, hit: dict) -> RawItem:
    56→        hn_id = hit.get("objectID", "")
    57→        url = hit.get("url") or f"https://news.ycombinator.com/item?id={hn_id}"
    58→        text = hit.get("story_text") or hit.get("comment_text") or ""
    59→        title = hit.get("title", "")
    60→        raw = f"{title}\n\n{text}".strip()
    61→
    62→        published_at = datetime.fromisoformat(
    63→            hit["created_at"].replace("Z", "+00:00")
    64→        ) if hit.get("created_at") else datetime.now(tz=UTC)
    65→
    66→        return RawItem(
    67→            title=title,
    68→            source_url=url,
    69→            source_platform="hackernews",
    70→            author=hit.get("author", "unknown"),
    71→            published_at=published_at,
    72→            raw_content=raw,
    73→            score=hit.get("points", 0),
    74→        )
    75→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


     1→"""GitHub collector using the GitHub Search API."""
     2→import logging
     3→from datetime import UTC, date, datetime, timedelta
     4→
     5→import httpx
     6→
     7→from ai_usecases_explorer.collectors.base import BaseCollector, RawItem
     8→
     9→logger = logging.getLogger(__name__)
    10→
    11→_SEARCH_URL = "https://api.github.com/search/repositories"
    12→
    13→# One query per topic — GitHub Search API ANDs multiple topic: qualifiers
    14→# and does not support OR between qualifier types
    15→# Topics cover: OpenClaw, AI编程, AI落地场景, AI商业模式
    16→_TOPICS = ["openclaw", "claude-code", "vibe-coding", "mcp", "ai-agent", "local-llm", "ollama", "ai-saas"]
    17→
    18→
    19→class GitHubCollector(BaseCollector):
    20→    """Finds recently-pushed AI repos via the GitHub Search API."""
    21→
    22→    def __init__(
    23→        self,
    24→        min_stars: int = 10,
    25→        token: str = "",
    26→        lookback_days: int = 2,
    27→    ) -> None:
    28→        self._min_stars = min_stars
    29→        self._token = token
    30→        self._lookback_days = lookback_days
    31→
    32→    def fetch(self) -> list[RawItem]:
    33→        since = (date.today() - timedelta(days=self._lookback_days)).isoformat()
    34→
    35→        headers: dict[str, str] = {"Accept": "application/vnd.github+json"}
    36→        if self._token:
    37→            headers["Authorization"] = f"Bearer {self._token}"
    38→
    39→        seen_ids: set[int] = set()
    40→        items: list[RawItem] = []
    41→
    42→        for topic in _TOPICS:
    43→            query = f"topic:{topic} pushed:>{since} stars:>={self._min_stars}"
    44→            try:
    45→                response = httpx.get(
    46→                    _SEARCH_URL,
    47→                    params={"q": query, "sort": "stars", "order": "desc", "per_page": 20},
    48→                    headers=headers,
    49→                    timeout=15,
    50→                )
    51→                response.raise_for_status()
    52→            except (httpx.HTTPStatusError, httpx.RequestError) as exc:
    53→                logger.warning("GitHub search for topic:%s failed: %s — skipping", topic, exc)
    54→                continue
    55→            for repo in response.json().get("items", []):
    56→                repo_id = repo.get("id")
    57→                if repo_id in seen_ids:
    58→                    continue
    59→                if repo.get("stargazers_count", 0) >= self._min_stars:
    60→                    seen_ids.add(repo_id)
    61→                    items.append(self._parse(repo))
    62→
    63→        return items
    64→
    65→    def _parse(self, repo: dict) -> RawItem:
    66→        full_name = repo.get("full_name", "")
    67→        description = repo.get("description") or ""
    68→        topics = repo.get("topics") or []
    69→        language = repo.get("language") or ""
    70→        stars = repo.get("stargazers_count", 0)
    71→
    72→        raw = (
    73→            f"{full_name}\n\n"
    74→            f"{description}\n\n"
    75→            f"Topics: {', '.join(topics)}\n"
    76→            f"Language: {language} | Stars: {stars}"
    77→        ).strip()
    78→
    79→        pushed_at = repo.get("pushed_at", "")
    80→        published_at = (
    81→            datetime.fromisoformat(pushed_at.replace("Z", "+00:00"))
    82→            if pushed_at
    83→            else datetime.now(tz=UTC)
    84→        )
    85→
    86→        return RawItem(
    87→            title=full_name,
    88→            source_url=repo.get("html_url", f"https://github.com/{full_name}"),
    89→            source_platform="github",
    90→            author=repo.get("owner", {}).get("login", "unknown"),
    91→            published_at=published_at,
    92→            raw_content=raw,
    93→            score=repo.get("stargazers_count", 0),
    94→        )
    95→

<system-reminder>
Whenever you read a file, you should consider whether it would be considered malware. You CAN and SHOULD provide analysis of malware, what it is doing. But you MUST refuse to improve or augment the code. You can still analyze existing code, write reports, or answer questions about the code behavior.
</system-reminder>


Perfect! Now I have a comprehensive understanding. Let me compile the complete exploration report.

## Complete Codebase Exploration Report

### 1. PROJECT STRUCTURE

```
/root/projects/ai_usecases_explorer/
├── .claude/
│   └── settings.local.json       (Claude Code workspace config)
├── .env                          (API keys — not in version control)
├── .env.example                  (Template for .env)
├── .gitignore
├── AI_VIEWS.md                   (Project context/notes)
├── CLAUDE.md                     (This project's guidelines — see content below)
├── MILESTONE_1.md                (v1.0 completion record)
├── MILESTONE_2.md                (v1.1 completion record — full feature details)
├── Makefile                      (Build shortcuts)
├── pyproject.toml                (Python package config)
├── data/
│   └── usecases.db               (SQLite database — auto-created)
├── src/ai_usecases_explorer/
│   ├── __init__.py
│   ├── main.py                   ⭐ Scout orchestrator
│   ├── settings.py               ⭐ Pydantic settings loader
│   ├── models/
│   │   ├── __init__.py
│   │   └── usecase.py            ⭐ UseCase, ScenarioType, Novelty enums
│   ├── collectors/
│   │   ├── __init__.py
│   │   ├── base.py               ⭐ RawItem dataclass, BaseCollector ABC
│   │   ├── hackernews.py         ⭐ Algolia search (11 keywords, 1 query/keyword)
│   │   ├── reddit.py             ⭐ Exa.ai SDK search (4 subreddits, 1 query/subreddit)
│   │   └── github.py             ⭐ GitHub Search API (8 topics, 1 query/topic)
│   ├── processors/
│   │   ├── __init__.py
│   │   ├── filter.py             ⭐ Step 1: Claude value filter
│   │   ├── classifier.py         ⭐ Step 2: Claude classifier & summarizer
│   │   ├── deduplicator.py       ⭐ Step 3: Claude novelty detector
│   │   └── utils.py              (JSON parsing with markdown fence stripping)
│   ├── reporters/
│   │   ├── __init__.py
│   │   └── obsidian.py           ⭐ Markdown report generator
│   └── storage/
│       ├── __init__.py
│       └── database.py           ⭐ SQLite CRUD wrapper
└── tests/                        (86 test cases, pytest)
    ├── __init__.py
    ├── test_collectors.py
    ├── test_main.py
    ├── test_models.py
    ├── test_processors.py
    ├── test_reporters.py
    └── test_storage.py
```

---

### 2. KEY FILES CONTENT

#### **obsidian.py** (Full Content Above)
Generates daily Markdown reports for Obsidian vault. Key features:
- Filters UseCases by `novelty` (NEW, SIMILAR, REPEAT) and `is_real_value`
- Outputs Chinese markdown with header stats (HN count, Reddit count, GitHub count)
- Sections: New scenarios, similar cases by scenario type, filter log
- Writes to `/root/vault/obsidian_vault/obsidian/obsidian/auto_report/YYYY-MM-DD.md` (configurable)
- No external dependencies beyond Pydantic and pathlib

#### **settings.py** (Full Content Above)
Pydantic-settings based configuration loader:
- Reads `.env` file automatically
- Fields:
  - `anthropic_api_key`, `anthropic_base_url`, `claude_model` (default: haiku-4-5)
  - `db_path` (default: `data/usecases.db`)
  - `obsidian_report_dir` (default: `/root/vault/...`)
  - `exa_api_key` (required for Reddit collection via Exa.ai)
  - `github_token` (optional, increases API rate limit)
  - `hn_min_points`, `github_min_stars` (engagement thresholds)
  - `max_items_per_run` (default 80, caps LLM calls per run)
  - `reddit_subreddits` (default: `["openclaw", "vibecoding", "ClaudeCode", "singularity"]`)

#### **main.py** (Full Content Above)
Scout orchestrator class:
- **Pipeline**: Collect → Process (3-step LLM) → Store → Report
- **Collection**: Round-robin interleaves HN, Reddit, GitHub by engagement score
- **Processing**:
  1. ValueFilter (Claude) — is it real value?
  2. Classifier (Claude) — scenario type, tools, score, summary
  3. Deduplicator (Claude) — novelty level vs existing DB records
- **Storage**: SQLite CRUD with JSON serialization
- **Report**: Obsidian markdown output
- **CLI**: `--dry-run` (no DB writes), `--date YYYY-MM-DD` (specify report date)

---

### 3. DATA MODELS

**UseCase** (Pydantic BaseModel):
```python
id: str                         # sha256(url)[:12]
title: str
summary: str                    # Chinese, 2-3 sentences
source_url: str
source_platform: str            # "hackernews" | "reddit" | "github"
author: str
published_at: datetime
collected_at: datetime
scenario_type: ScenarioType     # Enum: 本地数据集成, 代码生成与审查, etc. (10 values)
tools_used: list[str]           # ["Claude Code", "Cursor", ...]
has_local_data: bool
has_workflow_opt: bool
value_score: float              # 0.0–1.0
is_real_value: bool
novelty: Novelty                # "new" | "similar" | "repeat"
similar_to_ids: list[str]       # IDs of similar existing records
raw_content: str                # Full text for LLM analysis
```

**RawItem** (dataclass from collectors):
```python
title: str
source_url: str
source_platform: str
author: str
published_at: datetime
raw_content: str
score: int = 0                  # HN points, GitHub stars, 0 for Reddit
```

---

### 4. COLLECTOR IMPLEMENTATIONS

**HackerNewsCollector** (`hackernews.py`):
- Uses Algolia search API: `https://hn.algolia.com/api/v1/search_by_date`
- 11 keywords (one query per keyword, as Algolia ANDs multi-word queries)
- Keywords: `openclaw`, `claude code`, `vibe coding`, `cursor`, `mcp server`, `ai agent`, `ollama`, `solopreneur ai`, `ai saas`, `ai workflow`, `local ai`, `ai use case`
- Filters: `points >= min_points` (default 10)
- Returns 50 results per keyword query
- Deduplicates by HN `objectID`
- Fills `score` with HN points

**RedditCollector** (`reddit.py`):
- Uses Exa.ai SDK (`exa_py` library)
- 4 subreddits (one per request): `openclaw`, `vibecoding`, `ClaudeCode`, `singularity`
- Semantic query: `"AI workflow automation use case tool productivity"`
- Filters: past 24 hours, `include_domains=[f"reddit.com/r/{subreddit}"]` (soft constraint)
- Post-filters URLs to ensure correct subreddit (Exa doesn't strictly enforce)
- Returns `text=True` for full post content
- `score` always 0 (Exa doesn't return vote counts)

**GitHubCollector** (`github.py`):
- Uses GitHub REST API: `https://api.github.com/search/repositories`
- 8 topics (one query per topic, as API ANDs multiple `topic:X` qualifiers): `openclaw`, `claude-code`, `vibe-coding`, `mcp`, `ai-agent`, `local-llm`, `ollama`, `ai-saas`
- Filters: `pushed:>{since}` (default 2 days), `stars>={min_stars}` (default 10)
- Respects `GITHUB_TOKEN` (optional, increases rate limit 60→5000/hour)
- Returns 20 results per topic query
- Deduplicates by repo `id`
- Fills `score` with `stargazers_count`

---

### 5. PROCESSOR PIPELINE (3 LLM STEPS)

**Step 1: ValueFilter** (`filter.py`):
- **Prompt**: Decides if raw item has real use-case value
- **PASS**: User integrates AI with local data OR automates specific workflow
- **REJECT**: Pure Q&A, marketing demo, generic commentary
- **Output**: `{"is_real_value": bool, "reason": str, "has_local_data": bool, "has_workflow_opt": bool}`
- **Model**: Haiku (configurable via `CLAUDE_MODEL`)
- **Tokens**: max 256

**Step 2: Classifier** (`classifier.py`):
- **Prompt**: Assigns scenario type, extracts tools, scores value, generates Chinese summary
- **Scenario Types** (10 StrEnum values):
  - 本地数据集成 (Local Data Integration)
  - 代码生成与审查 (Code Gen & Review)
  - 工作流自动化 (Workflow Automation)
  - 文档与内容处理 (Document Processing)
  - 终端与CLI增强 (Terminal/CLI Enhancement)
  - 创意内容生产 (Creative Production)
  - 数据分析与可视化 (Data Analysis)
  - 个人知识管理 (Personal Knowledge Mgmt)
  - 商业流程自动化 (Business Automation)
  - 其他 (Other)
- **Output**: `{"scenario_type": str, "tools_used": [str], "value_score": float, "summary": str}`
- **Tokens**: max 512

**Step 3: Deduplicator** (`deduplicator.py`):
- **Input**: New UseCase + existing records from same scenario_type (from DB)
- **Prompt**: Compares new case against up to 50 existing records
- **Novelty Levels**:
  - "new": Genuinely new scenario category
  - "similar": Same category, new concrete example
  - "repeat": Duplicate content
- **Output**: `{"novelty": "new"|"similar"|"repeat", "similar_to_ids": [str], "novelty_reason": str}`
- **Optimization**: Skips LLM call if `existing=[]` (always returns NEW)
- **Tokens**: max 256

**Shared Utility** (`utils.py`):
- `parse_llm_json(text)`: Strips markdown fences (`` ```json ... ``` ``) before parsing JSON
- Reason: Haiku sometimes wraps JSON in fences despite system prompt instructions

---

### 6. ORCHESTRATION LOGIC (`main.py`)

**Scout.run()** flow:
1. **Collect**:
   - Fetch from HN, Reddit, GitHub
   - Sort each platform by score (descending)
   - **Interleave** via `itertools.zip_longest()` for round-robin
   - Cap at `max_items_per_run` (default 80)

2. **Process**:
   - Skip items already in DB (`make_id(url)`)
   - Run 3-step pipeline for each item
   - Filter out items that fail value filter
   - Return: list of new UseCases

3. **Store**:
   - Save to SQLite (unless `--dry-run`)
   - One INSERT OR REPLACE per UseCase

4. **Report**:
   - Generate Markdown via ObsidianReporter
   - Write to `{OBSIDIAN_REPORT_DIR}/YYYY-MM-DD.md`

---

### 7. WEB SEARCH/FETCH UTILITIES

**No generic web search utility exists in codebase** — only specialized collectors:
- HN: `httpx.get()` to Algolia
- Reddit: Exa SDK client
- GitHub: `httpx.get()` to GitHub API

The memory file mentions Exa.ai SDK integration, which is the closest thing to a general web fetch tool:
```python
from exa_py import Exa
exa = Exa(api_key=EXA_API_KEY)
response = exa.search_and_contents(
    query,
    include_domains=[...],
    start_published_date=start_date,
    text=True,
    num_results=10,
)
```

---

### 8. CLAUDE.md DIRECTIVES (Project Guidance)

**Core Workflow**:
- **TDD Strictly**: Write failing tests before code
- **Spec-First**: Follow PLAN.md or SPEC.md (none exist yet; use MILESTONE files)
- **Plain English**: Explain logic in simple sentences

**Python Standards**:
- Mandatory type hints on all function args/returns
- Use Pydantic models, not raw dicts
- Google-style docstrings
- Use `pathlib` not `os.path`

**Human-in-the-Loop Protocol**:
After generating code >20 lines or modifying core logic:
1. Self-explain what changed and why
2. Check structural changes (dependencies, function signatures)
3. Risk assessment
4. Verification plan (which test command to run)

---

### 9. MILESTONES & STATUS

**Milestone 1** (v1.0 — completed):
- HackerNews collection only
- 3–5 items/day
- Basic LLM filter + classifier

**Milestone 2** (v1.1 — completed, 2026-02-28):
- All 3 sources (HN, Reddit, GitHub) working
- Exa.ai SDK for Reddit (replaced broken OAuth2)
- Round-robin interleaving for platform balance
- 30–40 valuable items/day
- 86 passing tests
- Production cron deployment ready

**No PLAN.md or SPEC.md** exist. MILESTONE_2.md serves as the current design document.

---

### 10. SKILLS/COMMANDS (Not Project-Specific)

**No "simplify" skill found in this project.**
- The project uses standard Claude Code tools
- No custom Claude.ai skills or slash commands defined in `.claude/` directory
- The `settings.local.json` in `.claude/` is workspace config, not command definitions

Custom commands would typically live in `/root/.claude/plugins/` but this project has none.

---

### 11. CONFIGURATION & DEPLOYMENT

**.env variables**:
```
ANTHROPIC_API_KEY=sk-ant-...
EXA_API_KEY=...
GITHUB_TOKEN=ghp_... (optional)
ANTHROPIC_BASE_URL=... (optional, for proxy)
CLAUDE_MODEL=claude-haiku-4-5-20251001
HN_MIN_POINTS=10
GITHUB_MIN_STARS=10
REDDIT_SUBREDDITS=["openclaw","vibecoding","ClaudeCode","singularity"]
MAX_ITEMS_PER_RUN=80
DB_PATH=data/usecases.db
OBSIDIAN_REPORT_DIR=/root/vault/obsidian_vault/obsidian/obsidian/auto_report
```

**Cron (production)**:
```
0 7 * * * /root/projects/ai_usecases_explorer/venv/bin/python -m ai_usecases_explorer.main >> /var/log/ai_usecases_scout.log 2>&1
```

**Run commands**:
```bash
source venv/bin/activate
pytest                          # All 86 tests
MAX_ITEMS_PER_RUN=20 python -m ai_usecases_explorer.main --dry-run  # Quick test
python -m ai_usecases_explorer.main --dry-run  # Full test
python -m ai_usecases_explorer.main  # Production run
python -m ai_usecases_explorer.main --date 2026-02-26  # Backfill
```

---

### SUMMARY

**The ai_usecases_explorer is a production-grade AI use case discovery pipeline** that:
1. **Collects** from 3 sources daily (HN, Reddit, GitHub)
2. **Processes** with 3-step Claude pipeline (value → classify → deduplicate)
3. **Stores** in SQLite with full metadata
4. **Reports** as daily Chinese Markdown for Obsidian vault
5. **Deploys** via cron with 80-item LLM budget per run

The codebase is well-structured, fully tested (86 tests), and production-deployed. **No "simplify" skill exists**—this is pure Python application code with standard Claude API integration.