This project aims to crawl videos from www.ixigua.com, a short video publishing website. This website uses AJAX web pages and they are dynamic in structure. After trying several methods(urllib, request, beatifulsoup, selenium,etc), I suggest using selenium framework and python for web crawling. The selenium package is used to automate web browser interaction from Python. Required knowledge: HTML, CSS, Javascript(need to learn), Python, Selenium package Web to learn selenium: https://www.bilibili.com/video/av19430527/?p=1 Selenium Offical documentation:https://www.seleniumhq.org/docs/
2018-08-16 version 1 In this version, I have built a foundamental framework which can automatically loading main pages and get video urls shown on main page. Main Function:
- Open main page www.ixigua.com and load videos as much as possible(by scroll down to the end of page).
- Collect all video page url(only those loaded will show address)
- For each video, open video page, use find_element_by_xpath to get title, publisher, publishing time, video url, comments and user_id.
- Filter ads.
- Download videos.(urllib.request.urlretrieve)
Problems to be solved:
- Each time scroll down to the end of page, the web may not load. Sometimes it has to wait for a few seconds(2-10 seconds). When too many videos are loading, the pages loads even slower. So through this method I can't control the number of videos I want to download. The only thing I can control is the number of scrolls. Each time I run, the page show different number of videos.
- The pages don't show publishing time and I can't find it in the source code.
- Some video pages loads too slow, then will return noelementexception and timeoutexception.
- Some comments are reply to other comments, which is not displayed in the page. Try to use actionchian to click to show the comments.