Giter VIP home page Giter VIP logo

spider_xhs's Introduction

🎀Spider_XHS

最新signSvn55 AES | xs xt xscommom sign a1 qsignature源码

联系作者 全部协议接口、小红书pc、小红书创作者、小红书专业号、小红书千帆、小红书蒲公英,定制需求

小红书exe程序, 网页程序定制

小红书个人主页无水印图片、无水印视频、个人信息和搜索爬取。

🎨效果图

处理后的所有用户

image

某个用户所有的笔记

image

某个笔记具体的内容

image

图形化界面

image

⛳运行环境

Python环境>=3.7 NodeJS环境>=16

🎯运行方法

多用户下载(下载用户列表所有的笔记)
python home.py
多笔记下载(下载笔记列表里所有的笔记)
python one.py
下载搜索内容
python search.py

🍥日志

日期 说明
23/08/09 - 首次提交。
23/09/13 - api更改params增加两个字段,修复图片无法下载,有些页面无法访问导致报错。
23/09/16 - 较大视频出现编码问题,修复视频编码问题,加入异常处理。
23/09/18 - 代码重构,加入失败重试。
23/09/19 - 新增下载搜索结果功能。
23/10/05 - 新增跳过已下载功能,获取更详细的笔记和用户信息。
23/10/08 - 上传代码☞Pypi,可通过pip install安装本项目。
23/10/17 - 搜索下载新增排序方式选项(1、综合排序 2、热门排序 3、最新排序)。
23/10/21 - 新增图形化界面,上传至release v2.1.0。
23/10/28 - Fix Bug 修复搜索功能出现的隐藏问题。

🧸注意事项

本项目仅供学习与交流,侵权必删

  1. home处理的是个人主页 https://www.xiaohongshu.com/user/profile/6185ce66000000001000705b
  2. one处理的是笔记详细页 https://www.xiaohongshu.com/explore/64d06670000000000800fb4a
  3. search处理的是搜索结果

🛹额外说明

  1. 自行将cookie放到static目录下cookies.txt中,去设置里的应用程序里找或者网络请求里找,需要哪些可以参考cookies.txt文件。

  2. 可采用以下方法获取cookie,并运行对应文件,只有登陆后的cookies是有用的。 image image

  3. 感谢star⭐!不时更新。

  4. 有问题可以加QQ(992822653)或者微信(CVZC15751076989)交流!

  5. 感谢赞助!如果此项目对您有帮助,请作者喝一杯奶茶~~ (开心一整天😊😊)

  6. thank you~~~

mm_facetoface_collect_qrcode_16968399159071696832397

spider_xhs's People

Contributors

byyuchun avatar cv-cat avatar juore avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

spider_xhs's Issues

feed接口好像不能用了

/api/sns/web/v1/search/notes接口可以正常拿到笔记列表,但是/api/sns/web/v1/feed接口今天突然报错了,之前都是好的,报错的返回:
status: 461 status code 461, text: {"code":0,"success":true,"msg":"成功","data":{}}

一直返回461的响应,即便是换代理、换账号貌似还是不行,莫非小红书直接给该接口上了新的buf么。不知道各位道友是不是好的

保存的图片损坏

小红书原图是webp格式的,直接保存为jpg和png后在Windows上都显示不了,双击打开显示位图损坏,保存为webp就没问题

有bug Spider_XHS/xhs_utils/xhs_util.py

File "/home/xxxx/tmp/Spider_XHS/xhs_utils/xhs_util.py", line 74, in handle_profile_info
info = re.findall(r'<script>window.INITIAL_STATE=(.*?)</script>', html_text)[0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

one-note.py老是报语法错误

Traceback (most recent call last):
File "d:\develop\pythonWorkspace\Spider_XHS\one-note.py", line 193, in
main()
File "d:\develop\pythonWorkspace\Spider_XHS\one-note.py", line 188, in main
handle_note(note_id_)
File "d:\develop\pythonWorkspace\Spider_XHS\one-note.py", line 115, in handle_note
ret = js.call('get_xs', '/api/sns/web/v1/feed', note_id, cookies['a1'])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_abstract_runtime_context.py", line 37, in call
return self._call(name, *args)
^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_external_runtime.py", line 92, in _call
return self.eval("{identifier}.apply(this, {args})".format(identifier=identifier, args=args))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_external_runtime.py", line 78, in eval
return self.exec
(code)
^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_abstract_runtime_context.py", line 18, in exec

return self.exec(source)
^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_external_runtime.py", line 88, in exec
return self._extract_result(output)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\admin\AppData\Roaming\Python\Python311\site-packages\execjs_external_runtime.py", line 167, in _extract_result
raise ProgramError(value)
execjs._exceptions.ProgramError: SyntaxError: 语法错误

关于反爬

请问作者大佬,这俄格反爬是针对用户还是针对ip啊,什么速度以内不会被封号啊

配置正确的情况下,用户查询失败

大概两天前能够成功爬取内容,今天再用就用户查询失败了。cookie和用户主页网址没有问题。尝试通过源码中打印来寻找排查错误,但是也没看出问题所在。
附修改的部分代码片段以及终端截图,其他地方没改过。
已经尝试过重新克隆项目运行例子,但是也是用户查询失败,猜测可能是小红书更新了json内容导致列表出错?
e4817a75a04daa73b743024db1dee88
修改位置为home.py中的save_all_note_info函数中的循环体

nodejs黑框

image
每开始运行一个任务,都会跳出nodejs黑框,然后消失
很影响使用体验

画质问题

我在网页版上手动保存的大小比用spider下载的图片大不少
画质不一样 不知道能让它和网页版上手动保存的画质一致么

更新cookie失效

我尝试更新cookie但是仍然无法运行;所有我尝试着直接把cookie写入代码,运行显示cookie失效,但是这个cookie是我刚更新的。。。

`def check_cookies():
more_url = 'https://edith.xiaohongshu.com/api/sns/web/v1/user_posted'
params = get_params()
headers = get_headers()

cookies_local = {
    "sec_poison_id": "2b74c391",
    "gid": "y",
    "a1": "18",
    "websectiga": "6",
    "webId": "5",
    "web_session": "0",
    "xsecappid": "xhs-pc-web",
    "webBuild": "4.27.7"
}

try:
    test_user_id = '5ad2ede14eacab146f865fe9'
    params['user_id'] = test_user_id
    params['cursor'] = ''
    api = f"/api/sns/web/v1/user_posted?num=30&cursor=&user_id={test_user_id}&image_scenes="
    a1 = cookies_local['a1']
    
    try:
        ret = js.call('get_xs', api, '', a1)
    except Exception as e:
        print(f'缺少nodejs环境: {e}')
        return
    
    headers['x-s'], headers['x-t'] = ret['X-s'], str(ret['X-t'])
    response = requests.get(more_url, headers=headers, cookies=cookies_local, params=params)
    res = response.json()
    
    if not res.get("success", False):
        raise Exception("cookie失效")
    else:
        print("cookie有效")
        return cookies_local

Snipaste_2024-07-30_21-36-52
Snipaste_2024-07-30_21-24-15
Snipaste_2024-07-30_21-24-21
Snipaste_2024-07-30_21-35-21

except Exception as e:
    print(f"cookie失效,请手动更改cookies.txt文件: {e}")
    sys.exit(1)`

windows电脑下登录cookie不可用,mac却可以

windows下Chrome v126 登录xhs,然后通过editThisCookie插件将Cookie拷贝出来,发现程序在执行checkCookie()方法时response = requests.get(more_url, headers=headers, cookies=cookies_local, params=params) 得到的返回虽然是True,但data却是空的。

res:{'code': 0, 'success': True, 'msg': '成功', 'data': {}}

有大佬知道什么情况吗

功能建议

之前下了某个博主的所有内容放在datas里面,博主更新了几篇内容,想再次把博主的所有内容下下来(即更新datas),但是不想把以前下过的再重新下一遍,建议增加获取博主文章列表后,检测是否为已下载内容,是则跳过此篇文章的下载

下载视频的时候会报错,大佬遇到过这个问题吗

这块代码
elif type == 'video':
print(f"{name}开始下载, {media_url}")
start_time = time.time()
res = requests.get(media_url, stream=True)
size = 0
chunk_size = 1024 * 1024
content_size = int(res.headers["content-length"])
with open(path + '/' + name + '.mp4', mode="wb") as f:
for data in res.iter_content(chunk_size=chunk_size):
f.write(data)
size += len(data) # 已下载文件的大小
# 已完成的百分比
percentage = size / content_size
# 打印进度条
print(f'\r视频:%.2fMB\t' % (content_size / 1024 / 1024),
'下载进度:[%-50s%.2f%%]耗时:%.1fs' % ('>' * int(50 * percentage), percentage * 100, time.time() - start_time),
end='')
print(f"{name}下载完成")

在下载某些视频的时候,会报这个错误。网上搜了好像也没啥解决方法

urllib3.exceptions.ProtocolError: ('Connection broken: IncompleteRead(10485760 bytes read, 280808 more expected)', IncompleteRead(10485760 bytes read, 280808 more expected))

search报错

Traceback (most recent call last):
File "/Users/tinghuang/Spider_XHS/search.py", line 95, in
search.main(info)
File "/Users/tinghuang/Spider_XHS/search.py", line 79, in main
self.handle_note_info(query, number, sort, need_cover=True)
File "/Users/tinghuang/Spider_XHS/search.py", line 55, in handle_note_info
ret = js.call('get_xs', api, data, self.cookies['a1'])
TypeError: 'NoneType' object is not subscriptable

关于反爬

楼主好
1、小红书一个关键词搜索最多获取两百多条笔记吗,
2、有没有什么办法能够解决这个搜索关键词获取笔记太少的问题啊,
3、如果设置多个账号或者iP代理会不会好点

Encountering "缺少nodejs环境" Error When Using Source Code

Description: When running *.py files using the source code, I consistently encounter the "缺少nodejs环境" error. I have provided the necessary cookie.txt, and the exe file works fine (though the terminal window keeps appearing continuously).
Packages installed before running:

pip install xhs-spider -i https://pypi.org/simple
npm i jsdom
Current node version: nodejs v18.17
Operating System: Windows 11
20240104_175540

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.