٦i?dZddlZddlZddlZddlmZmZddlmZddlm Z m Z ddl m Z ddl mZddlmZdd lmZmZmZeeZGd d e ZGd d ZdS)z6Note ingestion engine for the Second Brain RAG system.N)datetimetimezone)Path)ListOptional) BaseModel) setup_logger) GeminiLLM) ChunkMetadataChromaVectorStoresplit_md_by_headersceZdZUdZeed<eed<eeed<eeed<eed<dZeeed<d Z eed <dS) ExtractedNotez2LLM-extracted structured note from raw input text.titlesummary key_pointsentities cleaned_bodyN source_urlzhlanguage) __name__ __module__ __qualname____doc__str__annotations__rrrr9/root/projects/butler/slack_bot/obsidian/note_ingester.pyrrsz<< JJJ LLLS 3i $J $$$HcrrceZdZdZdededdfdZdedefdZd ede efd Z d d hZ d Z dedefdZ dedefdZdedefdZdedefdZdedefdZdede edefdZdede edefdZdedefdZdededdfdZdS) NoteIngesterzBIngests raw text or URLs into the Obsidian vault and vector store. vault_path vector_storereturnNc||_|dz |_|jdd||_t |_dS)zInitialize the note ingester. Args: vault_path: Root path of the Obsidian vault. vector_store: ChromaDB vector store for indexing chunks. notesT)parentsexist_okN)r# notes_dirmkdirr$r llm)selfr#r$s r __init__zNoteIngester.__init__"sI%#g- TD999(;;r user_inputc ||}d}|rrtd|d ||}|}n=#t$r.}td|dd|dcYd}~Sd}~wwxYw|} |||}n:#t$r-}td|d |d cYd}~Sd}~wwxYw||}| ||d d |j dd D}| |j } d|jd|jd|d| d S)zIngest text or a URL into the vault and vector store. Args: user_input: Raw text or a URL to ingest. Returns: Slack-formatted confirmation message. NzDetected URL: z. Fetching content...zURL fetch failed ()u+⚠️ 无法自动抓取该链接内容（uM） 请将文章正文直接粘贴发送给我，我来帮你保存入库。zLLM extraction failed: u/❌ 内容提取失败（LLM 解析出错）：u2 请重试，或检查原始内容是否正常。 c3"K|] }d|dV dS)z[[z]]Nr).0es r z&NoteIngester.ingest..Ss*FFq Q FFFFFFru✅ *Note saved:*  z _u_ 📁 ``) _detect_urlloggerinfo _fetch_url Exceptionwarning_extract_with_llmerror_write_md_file_chunk_and_indexjoinr relative_tor#rr) r-r/urlrraw_textr5note file_path entity_linksrel_paths r ingestzNoteIngester.ingest/s z**$(  " KKCCCC D D D ??3//    8A888999`!``` "H }))(J??DD } } } LL6166 7 7 7|Q||| | | | | | | }''--  dI...xxFFDM"1"4EFFFFF ((99 ! ! !| ! ! ! ! ! ! ! s;A B #B>B B B&& C0"CCCtextctjd|}|sdS|ddS)awExtract the first HTTP/HTTPS URL from text, handling Slack's format. Slack wraps URLs as ```` or ````. The angle brackets and pipe are excluded from the match so they don't get appended to the URL. Args: text: Input text to scan. Returns: First clean URL found, or None. zhttps?://[^\s<>|]+Nrz.,;:)researchgrouprstrip)r-rMmatchs r r:zNoteIngester._detect_url\sA /66 4{{1~~$$V,,,rzmp.weixin.qq.comz weixin.qq.comrFctjd|}|r|dnd}||jvr|j|j|jg}n|j|j|jg}td}|D]} ||}t||j kr6t dt|d|j |cSt|j dt|d#t$r0}t|j d ||}Yd }~d }~wwxYwtd |) uFetch URL content using multiple strategies with automatic fallback. Strategy order: - WeChat / known blocked domains: direct js_content parse → Tavily → Playwright - All other URLs: Tavily → Jina → Playwright Args: url: URL to fetch. Returns: Extracted text content (>= _MIN_CONTENT_LEN chars). Raises: RuntimeError: If all strategies fail. zhttps?://([^/]+)zNo strategies availablezFetched z chars via z returned too little content (z chars)z failed: Nu所有抓取策略均失败: )rOrPrQ_JINA_FIRST_DOMAINS_fetch_direct_wechat _fetch_tavily_fetch_playwright _fetch_jina RuntimeErrorlen_MIN_CONTENT_LENr;r<rdebugr>) r-rF host_matchhost strategieslast_errstrategycontentr5s r r=zNoteIngester._fetch_urlss Y2C88 &08z"""b 4+ + +3T5GI_`JJ,d.>@VWJ*+DEE"  H "(3--w<<4#888KK W3w<< W WHDU W WXXX"NNN  1ffQTU\Q]Q]fffgggg     1??A??@@@ E8EEFFFs5AD2D D; &D66D;cddl}d|}ddd}tjd}|rd||d <|||d d }||jS) uFetch via Jina Reader (r.jina.ai) — works well for WeChat and most sites. Args: url: Target URL. Returns: Markdown text extracted by Jina. rNzhttps://r.jina.ai/z text/plainmarkdown)AcceptzX-Return-Format JINA_API_KEYzBearer AuthorizationTheaderstimeoutfollow_redirects)httpxosenvirongetraise_for_statusrM)r-rFrqjina_urlrnjina_keyresps r r\zNoteIngester._fetch_jinas  ---#/JOO:>>.11  <';';';GO $yy7BQUyVV yrc2ddl}ddd}|||dd}||j}g}t jd |tj}|rUt jd d |d  }|r| d |t jd|} | r| d| d z} gd} t|} | D]*} || | }d|cxkr| krn(|} +|| | }t jd d|}t jdd|}t jdd |}t jdd| }|r| ||s>t jd d|}t jdd| Sd|S)aDirectly fetch a WeChat public article by parsing js_content from HTML. WeChat public articles (mp.weixin.qq.com) embed the full body in
. This method extracts title + body without needing a headless browser or external proxy. Args: url: WeChat article URL. Returns: Plain text of title + article body. rNzoMozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36zzh-CN,zh;q=0.9)z User-AgentzAccept-LanguageTrmz>class=["\'][^"\']*rich_media_title[^"\']*["\'][^>]*>(.*?)z<[^>]+>rWrVz# zid=["\']js_content["\']>)zid="js_bottom_bar"zid="js_related_posts"zid="js_msg_card"zid="js_sg_bar"zid="js_pc_qr_code"zclass="qr_code_pc_outer"r2z z &[a-zA-Z]+;z\s+ )rqrtrurMrOrPDOTALLsubrQstripappendindexendr^findrD)r-rFrqrnrxhtmlpartstitle_m title_textstart_m content_start end_markers content_endmarkeridx body_htmlbodyrMs r rYz!NoteIngester._fetch_direct_wechatsC  1 0   yygrDyQQ y) M ")    0 B a0@0@AAGGIIJ 0 .*..///)6==  # JJsGKKMM::Q>MKd))K% & &ii 66s(((([((((("%K];67I6*c955D6)S$//D6."d33D6&#t,,2244D # T""" 56*c400D6&#t,,2244 4{{5!!!rcDddlm}tjd}|st d||}||g}|r7|dr"|dddd St d ) zFetch via Tavily extract API. Args: url: Target URL. Returns: Raw content string from Tavily. Raises: RuntimeError: If API key missing or no content returned. r) TavilyClientTAVILY_API_KEYz(TAVILY_API_KEY is not set in environment)api_key)urlsresults raw_contentrWuTavily 未能提取到内容)tavilyrrrrsrtr]extract)r-rFrrclientresults r rZzNoteIngester._fetch_tavilys ('''''*..!122 KIJJ Jg...cU++  ?fjj++ ?)$Q'++M2>> >9:::rc ddlm}n#t$rtdwxYw ddlm}n#t$rtdwxYw|5}|jd} |}||| |dd | d | d  }td t|d|||cdddS#|wxYw#1swxYwYdS)aFetch via headless Chromium with stealth mode (last resort). Renders the full page in a real browser, bypassing JS-heavy SPAs and most anti-bot fingerprinting. Slower (~3-5s) but most comprehensive. Args: url: Target URL. Returns: Visible body text extracted from the rendered page. Raises: ImportError: If playwright or playwright-stealth is not installed. RuntimeError: If the page loads but body text is too short. r)sync_playwrightzWplaywright is not installed. Run: pip install playwright && playwright install chromium)StealthzHplaywright-stealth is not installed. Run: pip install playwright-stealthT)headlessi0udomcontentloaded)ro wait_untilirzPlaywright fetched z chars from N)playwright.sync_apir ImportErrorplaywright_stealthrchromiumlaunchnew_pageapply_stealth_syncgotowait_for_timeout inner_textrr;r<r^close)r-rFrrpbrowserpagerfs r r[zNoteIngester._fetch_playwrights   ; ; ; ; ; ; ;   .     2 2 2 2 2 2 2   Z   _   !j'''66G '')) ,,T222 #u9K LLL%%d+++//&117799 Q#g,,QQCQQRRR    s< #.AE2B3E%EEEE#&E#rGrcz|rd|dnd}d|d|ddd}|j|g\}}|}|d r,t jd d|}t jd d|}t j|}t |S) acUse LLM to extract structured note data from raw text. Args: raw_text: Raw content to process (truncated to 8000 chars). source_url: Optional source URL for context. Returns: Validated ExtractedNote model. Raises: ValueError: If LLM output cannot be parsed as valid JSON. z Source URL: r8rWz^You are a knowledge extraction assistant. Extract structured information from the text below. z === TEXT === Ni@a === TASK === Return a JSON object with EXACTLY these fields: - "title": A concise, descriptive title (max 80 chars) - "summary": 2-3 sentence summary of the core content - "key_points": Array of up to 7 key points (strings) - "entities": Array of important people, concepts, projects, or tools mentioned (max 10) - "cleaned_body": The full text, cleaned and reorganized. Wrap entity names in [[double brackets]] like [[entity name]]. - "language": "zh" if the content is primarily Chinese, "en" otherwise CRITICAL: - Output ONLY valid JSON, no markdown code blocks, no explanation - Do NOT include tags like #writing_sample or #reply_sample - Entities should be wrapped as [[entity]] in cleaned_body z```z^```(?:json)?\n?z\n?```$) r,generate_responser startswithrOr~jsonloadsrmodel_validate)r-rGr url_contextpromptresponse_datas r r@zNoteIngester._extract_with_llm;s8BI3Z3333r   %4%&h00<< !>>##   u % % 8v12x@@Hvj"h77Hz(##++D111rc |dddddz}t||ddgg||dS) zCreate a minimal note when LLM extraction fails. Args: raw_text: Raw content. source_url: Optional source URL. Returns: Basic ExtractedNote with raw text as body. N<r8r2z...rTr)rrrrrrr)replacerr)r-rGrrs r _fallback_notezNoteIngester._fallback_notegsf" %%dC0066885@TcTN!!    rrHctjtjd}t jdd|jddd}|d|d}|j |z }|j r d |j d nd }d d |j D}tjtj }d|jd|d|jd |d|d|d|jd|jd } || dt$d||S)zWrite note to vault as a Markdown file with YAML frontmatter. Args: note: Extracted note data. Returns: Path to the written file. tzz%Y-%m-%dz[^\w\u4e00-\u9fff-]-N2rz.mdz source_url: "z" rWr8c3"K|] }d|dV dS)z - ""Nr)r4kps r r6z.NoteIngester._write_md_file..s*#N#NOROOO#N#N#N#N#N#Nrz --- title: "z" date: z# tags: - auto_ingested language: z key_points: z ingested_at: z --- ## Summary  ## Content zutf-8)encodingzNote written: )rnowrutcstrftimerOr~rrr*rrDr isoformatrrr write_textr;r<) r-rHdate_strslugfilenamerI source_linekey_points_yaml ingested_atrfs r rBzNoteIngester._write_md_file|s<8<00099*EEv,c4:>>ssCII#NN*****NX- @DWz1NoteIngester._chunk_and_index..s    5  NN272B2B32G2GSEKK%%a(#..V:%&'       rzIndexed z chunks for note: )rrr rrrrr enumerater$ add_chunksr;r<r^r)r-rHrIrchunks metadatasrs `` @r rCzNoteIngester._chunk_and_indexsT SS@QSS$T** J|f,t/@@%4%HIFlhl333==??        &f--     $$VY777 Js6{{JJdjJJKKKKKr)rrrrrr r.rrLrr:rXr_r=r\rYrZr[rr@rrBrCrrr r"r"s LL 4 7H T    + + + + + + Z-- ----(.?%Gc%Gc%G%G%G%GNss*F"F"F"F"F"F"P;;;;;;0* S* S* * * * X*2#*28C=*2]*2*2*2*2X s   -    *&=&T&&&&PL]LtLLLLLLLrr")rrrrrOrrpathlibrtypingrrpydanticrhealth.utils.logging_configr slack_bot.llm.geminir slack_bot.obsidian.vector_storer r r rr;rr"rrr rs;<< ''''''''!!!!!!!!444444******aaaaaaaaaa h       I   cLcLcLcLcLcLcLcLcLcLr