zouzanyan / douyin_crawl Goto Github PK
View Code? Open in Web Editor NEW抖音视频批量爬取
License: GNU General Public License v3.0
抖音视频批量爬取
License: GNU General Public License v3.0
Traceback (most recent call last):
File "/Users/apple/Downloads/douyin_crawl-main/CrawlHome.py", line 157, in
c.get_home_video(user_in)
File "/Users/apple/Downloads/douyin_crawl-main/CrawlHome.py", line 79, in get_home_video
cursor = json_str["max_cursor"] # 当页页码
KeyError: 'max_cursor'
execjs._exceptions.ProgramError: TypeError: Cannot read property 'JS_MD5_NO_COMMON_JS' of null
首先非常感谢您的代码给予我的帮助!本人不太会这个 lssues 的格式。。🤣
CrawlMajority.py 文件中 get_video_info 方法:
pageurl = f'https://www.douyin.com/aweme/v1/web/aweme/post/?aid=6383&sec_user_id={self.sec_uid}&count=10&max_cursor={max_cursor}&publish_video_strategy_type=2'
在您给出的以上获取作者主页信息的链接中,我发现您的代码是通过判断响应数据中 has_more
是否等于1来进行下一次请求的,而下一次请求变更的 max_cursor
参数也是从响应中取得,但您使用的该 v1 版本链接中 max_cursor
抖音其实每次都是返回固定的(一个时间戳,可能是作者的注册日期之类的🤔),也就是说当请求执行到第二次的时候,您的代码会固定 break,这就导致获取到的用户视频信息永远只会有10条。我有尝试搜索,发现以往使用的 v2 版本链接中的确是通过 max_cursor
来进行分页的,本人不善于爬虫分析,所以提了一个 lssues,提醒一下,也想关注一下作者后续的分析思路
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.