Skip to content

feat(xhs): add "collect" crawler type to fetch collected/收藏 notes#933

Open
dake6767 wants to merge 2 commits into
NanmiCoder:mainfrom
dake6767:feat/xhs-collected-notes
Open

feat(xhs): add "collect" crawler type to fetch collected/收藏 notes#933
dake6767 wants to merge 2 commits into
NanmiCoder:mainfrom
dake6767:feat/xhs-collected-notes

Conversation

@dake6767

@dake6767 dake6767 commented Jul 4, 2026

Copy link
Copy Markdown

What

Adds a new CRAWLER_TYPE="collect" for Xiaohongshu that crawls the
logged-in user's collected (收藏) notes. Previously the xhs crawler only
supported search / detail / creator, so there was no way to export
your own 收藏.

How

  • client.py: get_self_user_id() resolves the numeric user_id via
    /api/sns/web/v2/user/me (note: selfinfo only returns red_id, not the
    numeric user_id the collect API needs), plus get_notes_by_collect() /
    get_all_notes_by_collect() paging /api/sns/web/v2/note/collect/page.
  • core.py: new dispatch branch + get_collected_notes(), reusing the
    existing note-detail / media / comment pipeline — so ENABLE_GET_COMMENTS,
    CRAWLER_MAX_NOTES_COUNT, and all storage backends work unchanged.
  • config: XHS_COLLECT_USER_ID_LIST (empty = own collection; or list
    bare user_ids / full profile URLs with xsec_token for others' public
    collections) and CrawlerTypeEnum.COLLECT in cmd_arg (otherwise the
    config value silently falls back to search).

Usage

# config/base_config.py
PLATFORM = "xhs"
CRAWLER_TYPE = "collect"
# config/xhs_config.py
XHS_COLLECT_USER_ID_LIST = []   # empty = the logged-in user's own collection
python main.py            # or: python main.py --platform xhs --type collect

Output goes to the usual data/xhs/... sink (jsonl/csv/db/...), written as
collect_contents_* / collect_comments_*.

Testing

Verified end-to-end against a live logged-in account: resolves the user_id,
pages the collection, fetches note details + comments, and writes the output
files. Only xhs implements collect; other platforms treat the type as a
no-op via the existing else branch, so nothing else is affected.

Adds a new CRAWLER_TYPE="collect" for Xiaohongshu that crawls the
logged-in user's collected (收藏) notes. Previously only search, detail
and creator modes were supported, so there was no way to export one's
own collection.

- client: get_self_user_id() via /api/sns/web/v2/user/me, plus
  get_notes_by_collect() / get_all_notes_by_collect() hitting
  /api/sns/web/v2/note/collect/page
- core: dispatch branch + get_collected_notes(), reusing the existing
  note-detail + comment pipeline
- config: XHS_COLLECT_USER_ID_LIST (empty = own collection; may also
  list bare user_ids or full profile URLs for others' public collections)
- cmd_arg: add CrawlerTypeEnum.COLLECT
@dake6767 dake6767 requested a review from NanmiCoder as a code owner July 4, 2026 00:29
@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. enhancement New feature or request labels Jul 4, 2026
- tests/test_xhs_collect.py: `--type collect` is accepted (not downgraded
  to search), empty XHS_COLLECT_USER_ID_LIST resolves the logged-in user's
  own id, and a configured profile URL is parsed into user_id + xsec_token
- README: document `--type collect` and XHS_COLLECT_USER_ID_LIST usage
@NanmiCoder

Copy link
Copy Markdown
Owner

hello 感谢贡献,能把其他平台也一并支持吗,单独支持xhs和到main分支比较怪

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants