domain-skills: add feishu docx scraping - #595
Open
bianjiefilm wants to merge 1 commit into
Open
Conversation
Covers the four non-obvious traps when scraping *.feishu.cn/docx: scroll container is an inner div (window.scrollBy is a no-op), content is virtualized so innerText only returns the rendered slice, tables degrade to undelimited text, and image URLs behind internal-api-drive-stream need the browser session. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
2 issues found across 1 file
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="agent-workspace/domain-skills/feishu/docx-scraping.md">
<violation number="1" location="agent-workspace/domain-skills/feishu/docx-scraping.md:29">
P2: The `m:'win'` fallback branch is broken in practice. When the FIND probe finds no scrollable `div` it returns `{m:'win', sh:..., ch:...}`, but it also leaves `window.__sc` as `null`. Every subsequent step in 陷阱 2 unconditionally dereferences `window.__sc.scrollTop` / `window.__sc.clientHeight`, so on any doc whose container doesn't pass the `scrollHeight > clientHeight+200` filter the loop throws instead of falling back to window scrolling. Consider setting `window.__sc = document.scrollingElement` (or `window`) in the `win` branch, or making the loop branch on `m` before scrolling.</violation>
<violation number="2" location="agent-workspace/domain-skills/feishu/docx-scraping.md:48">
P3: The runnable snippet diverges from its own guidance. The loop hardcodes a 500px step instead of the recommended `clientHeight * 0.8`, and it omits the documented stall counter, leaving only the `top+ch >= sh-5` break plus a ~6-minute 500-iteration cap as backstops. For a smaller viewport, 500px can exceed the intended overlap and skip a slice; and if the container's `scrollHeight` grows while virtualizing, the end condition may never trigger and the tail spins to the cap. Consider applying `step = max(50, int(clientHeight * 0.8))` in the snippet and adding the stall counter so the example matches the bullets.</violation>
</file>
Reply with feedback, questions, or to request a fix.
Fix all with cubic | Re-trigger cubic
| if((s.overflowY==='auto'||s.overflowY==='scroll') && el.scrollHeight>el.clientHeight+200){ | ||
| if(el.scrollHeight>bh){bh=el.scrollHeight;best=el;} | ||
| }}); | ||
| window.__sc=best; |
Contributor
There was a problem hiding this comment.
P2: The m:'win' fallback branch is broken in practice. When the FIND probe finds no scrollable div it returns {m:'win', sh:..., ch:...}, but it also leaves window.__sc as null. Every subsequent step in 陷阱 2 unconditionally dereferences window.__sc.scrollTop / window.__sc.clientHeight, so on any doc whose container doesn't pass the scrollHeight > clientHeight+200 filter the loop throws instead of falling back to window scrolling. Consider setting window.__sc = document.scrollingElement (or window) in the win branch, or making the loop branch on m before scrolling.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agent-workspace/domain-skills/feishu/docx-scraping.md, line 29:
<comment>The `m:'win'` fallback branch is broken in practice. When the FIND probe finds no scrollable `div` it returns `{m:'win', sh:..., ch:...}`, but it also leaves `window.__sc` as `null`. Every subsequent step in 陷阱 2 unconditionally dereferences `window.__sc.scrollTop` / `window.__sc.clientHeight`, so on any doc whose container doesn't pass the `scrollHeight > clientHeight+200` filter the loop throws instead of falling back to window scrolling. Consider setting `window.__sc = document.scrollingElement` (or `window`) in the `win` branch, or making the loop branch on `m` before scrolling.</comment>
<file context>
@@ -0,0 +1,104 @@
+ if((s.overflowY==='auto'||s.overflowY==='scroll') && el.scrollHeight>el.clientHeight+200){
+ if(el.scrollHeight>bh){bh=el.scrollHeight;best=el;}
+ }});
+ window.__sc=best;
+ return best?{m:'el',sh:best.scrollHeight,ch:best.clientHeight}
+ :{m:'win',sh:document.documentElement.scrollHeight,ch:window.innerHeight};
</file context>
| s = ln.strip() | ||
| if s and s not in seen: | ||
| seen.add(s); lines.append(s) | ||
| st = js("(()=>{window.__sc.scrollTop+=500; return {top:window.__sc.scrollTop,sh:window.__sc.scrollHeight,ch:window.__sc.clientHeight}})()") |
Contributor
There was a problem hiding this comment.
P3: The runnable snippet diverges from its own guidance. The loop hardcodes a 500px step instead of the recommended clientHeight * 0.8, and it omits the documented stall counter, leaving only the top+ch >= sh-5 break plus a ~6-minute 500-iteration cap as backstops. For a smaller viewport, 500px can exceed the intended overlap and skip a slice; and if the container's scrollHeight grows while virtualizing, the end condition may never trigger and the tail spins to the cap. Consider applying step = max(50, int(clientHeight * 0.8)) in the snippet and adding the stall counter so the example matches the bullets.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At agent-workspace/domain-skills/feishu/docx-scraping.md, line 48:
<comment>The runnable snippet diverges from its own guidance. The loop hardcodes a 500px step instead of the recommended `clientHeight * 0.8`, and it omits the documented stall counter, leaving only the `top+ch >= sh-5` break plus a ~6-minute 500-iteration cap as backstops. For a smaller viewport, 500px can exceed the intended overlap and skip a slice; and if the container's `scrollHeight` grows while virtualizing, the end condition may never trigger and the tail spins to the cap. Consider applying `step = max(50, int(clientHeight * 0.8))` in the snippet and adding the stall counter so the example matches the bullets.</comment>
<file context>
@@ -0,0 +1,104 @@
+ s = ln.strip()
+ if s and s not in seen:
+ seen.add(s); lines.append(s)
+ st = js("(()=>{window.__sc.scrollTop+=500; return {top:window.__sc.scrollTop,sh:window.__sc.scrollHeight,ch:window.__sc.clientHeight}})()")
+ time.sleep(0.75)
+ if st['top']+st['ch'] >= st['sh']-5: break
</file context>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
agent-workspace/domain-skills/feishu/docx-scraping.md.Filed after scraping a 7.8k-line Feishu doc plus four sibling docs. Four things cost me steps that shouldn't cost the next agent anything:
window.scrollYstays 0 andwindow.scrollBy()is a no-op. Includes the probe snippet that finds the real container.document.body.innerTextreturns only the currently rendered slice, so a single read silently truncates the doc. Needs small-step scrolling with line-level dedupe, plus a stall counter so the tail doesn't spin.internal-api-drive-stream.feishu.cn; a barehttp_getgets nothing.Also lists the fixed UI noise strings that leak into extracted text (
AI QuickView,Backlinks (0), …) and the zero-width characters Feishu injects throughout body text and titles.No secrets, cookies, or user-specific state. No pixel coordinates.
🤖 Generated with Claude Code
Summary by cubic
Adds a Feishu Docs scraping guide showing how to reliably extract text and images from
*.feishu.cn/docxusing browser-driven scrolling and screenshots. Prevents truncated text, preserves tables via screenshots, and avoids auth errors on images.agent-workspace/domain-skills/feishu/docx-scraping.md.Written for commit d9a2980. Summary will update on new commits.