Giter VIP home page Giter VIP logo

tiktok-scraper's Introduction

TikTok Scraper & Downloader

NPM npm Codacy grade CI

Scrape and download useful information from TikTok.

No login or password are required

This is not an official API support and etc. This is just a scraper that is using TikTok Web API to scrape media and related meta information.


Buy Me A Coffee

Discord Server


Content

Important notes

  • As of right now it is NOT possible to download video without the watermark

Features

  • Download unlimited post metadata from the User, Hashtag, Trends, or Music-Id pages
  • Save post metadata to the JSON/CSV files
  • Download media with and without the watermark and save to the ZIP file
  • Download single video without the watermark from the CLI
  • Sign URL to make custom request to the TikTok API
  • Extract metadata from the User, Hashtag and Single Video pages
  • Save previous progress and download only new videos that weren't downloaded before. This feature only works from the CLI and only if download flag is on.
  • View and manage previously downloaded posts history in the CLI
  • Scrape and download user, hashtag, music feeds and single videos specified in the file in batch mode

To Do

  • CLI: save progress to avoid downloading same videos
  • Rewrite everything in TypeScript
  • Improve proxy support
  • Add tests
  • Download video without the watermark
  • Indicate in the output file(csv/json) if the video was downloaded or not
  • Build and run from Docker
  • CLI: Scrape and download in batch
  • CLI: Load proxies from a file
  • CLI: Optional ZIP
  • Renew API
  • Set WebHook URL (CLI)
  • Add new method to collect music metadata
  • Add Manual Pagination
  • Improve documentation
  • Download audio files
  • Web interface

Contribution

  • Don't forget about tests
yarn test
yarn build

Installation

tiktok-scraper requires Node.js v10+ to run.

Install from NPM

npm i -g tiktok-scraper

Install from YARN

yarn global add tiktok-scraper

USAGE

In Terminal

$ tiktok-scraper --help

Usage: tiktok-scraper <command> [options]

Commands:
  tiktok-scraper user [id]     Scrape videos from username. Enter only username
  tiktok-scraper hashtag [id]  Scrape videos from hashtag. Enter hashtag without #
  tiktok-scraper trend         Scrape posts from current trends
  tiktok-scraper music [id]    Scrape posts from a music id number
  tiktok-scraper video [id]    Download single video without the watermark
  tiktok-scraper history       View previous download history
  tiktok-scraper from-file [file] [async]  Scrape users, hashtags, music, videos mentioned
                                in a file. 1 value per 1 line

Options:
  --version            Show version number                             [boolean]
  --session            Set session cookie value. Sometimes session can be
                       helpful when scraping data from any method  [default: ""]
  --session-file       Set path to the file with list of active sessions. One
                       session per line!                           [default: ""]
  --timeout            Set timeout between requests. Timeout is in Milliseconds:
                       1000 mls = 1 s                               [default: 0]
  --number, -n         Number of posts to scrape. If you will set 0 then all
                       posts will be scraped                        [default: 0]
  --since              Scrape no posts published before this date (timestamp).
                       If set to 0 the filter is deactived          [default: 0]
  --proxy, -p          Set single proxy                            [default: ""]
  --proxy-file         Use proxies from a file. Scraper will use random proxies
                       from the file per each request. 1 line 1 proxy.
                                                                   [default: ""]
  --download, -d       Download video posts to the folder with the name input
                       [id]                           [boolean] [default: false]
  --asyncDownload, -a  Number of concurrent downloads               [default: 5]
  --hd                 Download video in HD. Video size will be x5-x10 times
                       larger and this will affect scraper execution speed. This
                       option only works in combination with -w flag
                                                      [boolean] [default: false]
  --zip, -z            ZIP all downloaded video posts [boolean] [default: false]
  --filepath           File path to save all output files.
      [default: "/Users/karl.wint/Documents/projects/javascript/tiktok-scraper"]
  --filetype, -t       Type of the output file where post information will be
                       saved. 'all' - save information about all posts to the`
                       'json' and 'csv'
                               [choices: "csv", "json", "all", ""] [default: ""]
  --filename, -f       Set custom filename for the output files    [default: ""]
  --noWaterMark, -w    Download video without the watermark. NOTE: With the
                       recent update you only need to use this option if you are
                       scraping Hashtag Feed. User/Trend/Music feeds will have
                       this url by default            [boolean] [default: false]
  --store, -s          Scraper will save the progress in the OS TMP or Custom
                       folder and in the future usage will only download new
                       videos avoiding duplicates     [boolean] [default: false]
  --historypath        Set custom path where history file/files will be stored
                   [default: "/var/folders/d5/fyh1_f2926q7c65g7skc0qh80000gn/T"]
  --remove, -r         Delete the history record by entering "TYPE:INPUT" or
                       "all" to clean all the history. For example: user:bob
                                                                   [default: ""]
  --webHookUrl         Set webhook url to receive scraper result as HTTP
                       requests. For example to your own API       [default: ""]
  --method             Receive data to your webhook url as POST or GET request
                                      [choices: "GET", "POST"] [default: "POST"]
  --help               Show help                                       [boolean]

Examples:
  tiktok-scraper user USERNAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper trend -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper hashtag HASHTAG_NAME -d -n 100 --session sid_tt=dae32131231
  tiktok-scraper music MUSIC_ID -d -n 50 --session sid_tt=dae32131231
  tiktok-scraper video https://www.tiktok.com/@tiktok/video/6807491984882765062 -d
  tiktok-scraper history
  tiktok-scraper history -r user:bob
  tiktok-scraper history -r all
  tiktok-scraper from-file BATCH_FILE ASYNC_TASKS -d

Output File Example

Demo

Docker

By using docker you won't be able to use --filepath and --historypath , but you can set volume(host path where all files will be saved) by using -v

Build
docker build . -t tiktok-scraper
Run

Example 1: All files including history file will be saved in the directory($pwd) where you running the docker from

docker run -v $(pwd):/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Example 2: All files including history file will be saved in /User/blah/downloads

docker run -v /User/blah/downloads:/usr/app/files tiktok-scraper user tiktok -d -n 5 -s

Module

Methods

.user(id, options) //Scrape posts from a specific user (Promise)
.hashtag(id, options) //Scrape posts from hashtag section (Promise)
.trend('', options) // Scrape posts from a trends section (Promise)
.music(id, options) // Scrape posts by music id (Promise)

.userEvent(id, options) //Scrape posts from a specific user (Event)
.hashtagEvent(id, options) //Scrape posts from hashtag section (Event)
.trendEvent('', options) // Scrape posts from a trends section (Event)
.musicEvent(id, options) // Scrape posts by music id (Event)

.getUserProfileInfo('USERNAME', options) // Get user profile information
.getHashtagInfo('HASHTAG', options) // Get hashtag information
.signUrl('URL', options) // Get signature for the request
.getVideoMeta('WEB_VIDEO_URL', options) // Get video meta info, including video url without the watermark
.getMusicInfo('https://www.tiktok.com/music/original-sound-6801885499343571718', options) // Get music metadata

Options

const options = {
    // Number of posts to scrape: {int default: 20}
    number: 50,

    // Scrape posts published since this date: { int default: 0}
    since: 0,

    // Set session: {string[] default: ['']}
    // Authenticated session cookie value is required to scrape user/trending/music/hashtag feed
    // You can put here any number of sessions, each request will select random session from the list
    sessionList: ['sid_tt=21312213'],

    // Set proxy {string[] | string default: ''}
    // http proxy: 127.0.0.1:8080
    // socks proxy: socks5://127.0.0.1:8080
    // You can pass proxies as an array and scraper will randomly select a proxy from the array to execute the requests
    proxy: '',

    // Set to {true} to search by user id: {boolean default: false}
    by_user_id: false,

    // How many post should be downloaded asynchronously. Only if {download:true}: {int default: 5}
    asyncDownload: 5,

    // How many post should be scraped asynchronously: {int default: 3}
    // Current option will be applied only with current types: music and hashtag
    // With other types it is always 1 because every request response to the TikTok API is providing the "maxCursor" value
    // that is required to send the next request
    asyncScraping: 3,

    // File path where all files will be saved: {string default: 'CURRENT_DIR'}
    filepath: `CURRENT_DIR`,

    // Custom file name for the output files: {string default: ''}
    fileName: `CURRENT_DIR`,

    // Output with information can be saved to a CSV or JSON files: {string default: 'na'}
    // 'csv' to save in csv
    // 'json' to save in json
    // 'all' to save in json and csv
    // 'na' to skip this step
    filetype: `na`,

    // Set custom headers: user-agent, cookie and etc
    // NOTE: When you parse video feed or single video metadata then in return you will receive {headers} object
    // that was used to extract the information and in order to access and download video through received {videoUrl} value you need to use same headers
    headers: {
        'user-agent': "BLAH",
        referer: 'https://www.tiktok.com/',
        cookie: `tt_webid_v2=68dssds`,
    },

    // Download video without the watermark: {boolean default: false}
    // Set to true to download without the watermark
    // This option will affect the execution speed
    noWaterMark: false,

    // Create link to HD video: {boolean default: false}
    // This option will only work if {noWaterMark} is set to {true}
    hdVideo: false,

    // verifyFp is used to verify the request and avoid captcha
    // When you are using proxy then there are high chances that the request will be
    // blocked with captcha
    // You can set your own verifyFp value or default(hardcoded) will be used
    verifyFp: '',

    // Switch main host to Tiktok test enpoint.
    // When your requests are blocked by captcha you can try to use Tiktok test endpoints.
    useTestEndpoints: false
};

Don't forget to check the examples folder

Promise

const TikTokScraper = require('tiktok-scraper');

// User feed by username
(async () => {
    try {
        const posts = await TikTokScraper.user('USERNAME', {
            number: 100,
            sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']
        });
        console.log(posts);
    } catch (error) {
        console.log(error);
    }
})();

// User feed by user id
// Some TikTok user id's are larger then MAX_SAFE_INTEGER, you need to pass user id as a string
(async () => {
    try {
        const posts = await TikTokScraper.user(`USER_ID`, {
            number: 100,
            by_user_id: true,
            sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']
        });
        console.log(posts);
    } catch (error) {
        console.log(error);
    }
})();

// Trending feed
(async () => {
    try {
        const posts = await TikTokScraper.trend('', {
            number: 100,
            sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']
        });
        console.log(posts);
    } catch (error) {
        console.log(error);
    }
})();

// Hashtag feed
(async () => {
    try {
        const posts = await TikTokScraper.hashtag('HASHTAG', {
            number: 100,
            sessionList: ['sid_tt=58ba9e34431774703d3c34e60d584475;']
        });
        console.log(posts);
    } catch (error) {
        console.log(error);
    }
})();

// Get single user profile information: Number of followers and etc
// input - USERNAME
// options - not required
(async () => {
    try {
        const user = await TikTokScraper.getUserProfileInfo('USERNAME', options);
        console.log(user);
    } catch (error) {
        console.log(error);
    }
})();

// Get single hashtag information: Number of views and etc
// input - HASHTAG NAME
// options - not required
(async () => {
    try {
        const hashtag = await TikTokScraper.getHashtagInfo('HASHTAG', options);
        console.log(hashtag);
    } catch (error) {
        console.log(error);
    }
})();


// Get single video metadata
// input - WEB_VIDEO_URL
// For example: https://www.tiktok.com/@tiktok/video/6807491984882765062
// options - not required
(async () => {
    try {
        const videoMeta = await TikTokScraper.getVideoMeta('https://www.tiktok.com/@tiktok/video/6807491984882765062', options);
        console.log(videoMeta);
    } catch (error) {
        console.log(error);
    }
})();

Event

const TikTokScraper = require('tiktok-scraper');

const users = TikTokScraper.userEvent("tiktok", { number: 30 });
users.on('data', json => {
    //data in JSON format
});
users.on('done', () => {
    //completed
});
users.on('error', error => {
    //error message
});
users.scrape();

const hashtag = TikTokScraper.hashtagEvent("summer", { number: 250, proxy: 'socks5://1.1.1.1:90' });
hashtag.on('data', json => {
    //data in JSON format
});
hashtag.on('done', () => {
    //completed
});
hashtag.on('error', error => {
    //error message
});
hashtag.scrape();

Get Set Session

NOT REQUIRED

Very common problem is when tiktok is blacklisting your IP/PROXY and in such case you can try to set session and there will be higher chances for success

Get the session:

  • Open https://www.tiktok.com/ in any browser
  • Login in to your account
  • Right click -> inspector -> networking
  • Refresh page -> select any request that was made to the tiktok -> go to the Request Header sections -> Cookies
  • Find in cookies sid_tt value. It usually looks like that: sid_tt=521kkadkasdaskdj4j213j12j312;
  • sid_tt=521kkadkasdaskdj4j213j12j312; - this will be your authenticated session cookie value that should be used to scrape user/hashtag/music/trending feed

Set the session:

  • CLI:

    • Set single session by using option --session. For example --session sid_tt=521kkadkasdaskdj4j213j12j312;
    • Set path to the file with the list of sessions by using option --session-file. For example --session-file /var/bob/sessionList.txt
      • Example content /var/bob/sessionList.txt:
      sid_tt=521kkadkasdaskdj4j213j12j312;
      sid_tt=521kkadkasdaskdj4j213j12j312;
      sid_tt=521kkadkasdaskdj4j213j12j312;
      sid_tt=521kkadkasdaskdj4j213j12j312;
      
  • In the MODULE you can set session by setting the option value sessionList . For example sessionList:["sid_tt=521kkadkasdaskdj4j213j12j312;", "sid_tt=12312312312312;"]

Download Video

This part is related to the MODULE usage (NOT THE CLI)

The {videoUrl} value is binded to the cookie value {tt_webid_v2} that can contain any value

Method 1: default headers

When you extract videos from the user, hashtag, music, trending feed or single video then in response besides the video metadata you will receive headers object that will contain params that were used to extract the data. Here is the important part, in order to access/download video through {videoUrl} value you need to use same {headers} values.

    headers: {
        "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36",
        "referer": "https://www.tiktok.com/",
        "cookie": "tt_webid_v2=689854141086886123"
    },

Method 2: custom headers

You can pass your own headers with the {options}.

const headers = {
    "user-agent": "BOB",
    "referer": "https://www.tiktok.com/",
    "cookie": "tt_webid_v2=BOB"
}
getVideoMeta('WEB_VIDEO_URL', {headers})
user('WEB_VIDEO_URL', {headers})
hashtag('WEB_VIDEO_URL', {headers})
trend('WEB_VIDEO_URL', {headers})
music('WEB_VIDEO_URL', {headers})
// And after you can access video through {videoUrl} value by using same custom headers

Json Output Example

Video Feed

Example output for the methods: user, hashtag, trend, music, userEvent, hashtagEvent, musicEvent, trendEvent

{
    headers: {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36',
        referer: 'https://www.tiktok.com/',
        cookie: 'tt_webid_v2=689854141086886123'
    },
    collector:[{
        id: 'VIDEO_ID',
        text: 'CAPTION',
        createTime: '1583870600',
        authorMeta:{
            id: 'USER ID',
            name: 'USERNAME',
            following: 195,
            fans: 43500,
            heart: '1093998',
            video: 3,
            digg: 95,
            verified: false,
            private: false,
            signature: 'USER BIO',
            avatar:'AVATAR_URL'
        },
        musicMeta:{
            musicId: '6808098113188120838',
            musicName: 'blah blah',
            musicAuthor: 'blah',
            musicOriginal: true,
            playUrl: 'SOUND/MUSIC_URL',
        },
        covers:{
            default: 'COVER_URL',
            origin: 'COVER_URL',
            dynamic: 'COVER_URL'
        },
        imageUrl:'IMAGE_URL',
        videoUrl:'VIDEO_URL',
        videoUrlNoWaterMark:'VIDEO_URL_WITHOUT_THE_WATERMARK',
        videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 },
        diggCount: 2104,
        shareCount: 1,
        playCount: 9007,
        commentCount: 50,
        mentions: ['@bob', '@sam', '@bob_again', '@and_sam_again'],
        hashtags:
        [{
            id: '69573911',
            name: 'PlayWithLife',
            title: 'HASHTAG_TITLE',
            cover: [Array]
        }...],
        downloaded: true
    }...],
    //If {filetype} and {download} options are enbabled then:
    zip: '/{CURRENT_PATH}/user_1552963581094.zip',
    json: '/{CURRENT_PATH}/user_1552963581094.json',
    csv: '/{CURRENT_PATH}/user_1552963581094.csv'
}
getUserProfileInfo
{
    secUid: 'MS4wLjABAAAAv7iSuuXDJGDvJkmH_vz1qkDZYo1apxgzaxdBSeIuPiM',
    userId: '107955',
    isSecret: false,
    uniqueId: 'tiktok',
    nickName: 'TikTok',
    signature: 'Make Your Day',
    covers: ['COVER_URL'],
    coversMedium: ['COVER_URL'],
    following: 490,
    fans: 38040567,
    heart: '211522962',
    video: 93,
    verified: true,
    digg: 29,
}
getHashtagInfo
{
    challengeId: '4231',
    challengeName: 'love',
    text: '',
    covers: [],
    coversMedium: [],
    posts: 66904972,
    views: '194557706433',
    isCommerce: false,
    splitTitle: ''
}
getVideoMeta
{
    headers: {
        'user-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36',
        referer: 'https://www.tiktok.com/',
        cookie: 'tt_webid_v2=689854141086886123'
    },
    collector:[{
        id: '6807491984882765062',
        text: 'We’re kicking off the #happyathome live stream series today at 5pm PT!',
        createTime: '1584992742',
        authorMeta: { id: '6812221792183403526', name: 'blah' },
        musicMeta:{
            musicId: '6822233276137213677',
            musicName: 'blah',
            musicAuthor: 'blah'
        },
        imageUrl: 'IMAGE_URL',
        videoUrl: 'VIDEO_URL',
        videoUrlNoWaterMark: 'VIDEO_URL_WITHOUT_THE_WATERMARK',
        videoMeta: { width: 480, height: 864, ratio: 14, duration: 14 },
        covers:{
            default: 'COVER_URL',
            origin: 'COVER_URL'
        },
        diggCount: 49292,
        shareCount: 339,
        playCount: 614678,
        commentCount: 4023,
        downloaded: false,
        hashtags: [],
    }]
}
getMusicInfo
{
    music: {
        id: '6882925279036066566',
        title: 'doja x calabria',
        playUrl: 'dfdfdfdf',
        coverThumb:
            'dfdfdf',
        coverMedium:
            'dfdfdf',
        coverLarge:
            'fdfdf',
        authorName: 'bryce',
        original: true,
        playToken:
            'ffdfdf',
        keyToken: 'dfdfdfd',
        audioURLWithcookie: false,
        private: false,
        duration: 46,
        album: '',
    },
    author: {
        id: '6835300004094166021',
        uniqueId: 'mashupsbybryce',
        nickname: 'bryce',
        avatarThumb:
            'dfdfd',
        avatarMedium:
            'dfdfdf',
        avatarLarger:
            'dfdfdf',
        signature: 'hi ily :)\n70k sounds cool tbh\n👇follow my soundcloud & insta👇',
        verified: false,
        secUid: 'MS4wLjABAAAA1_5bjLAamayD4rv3q49qJGa_7dZ5jzExTO0ozOybqIwwhw5TAg_iM25lkO94DM3K',
        secret: false,
        ftc: false,
        relation: 0,
        openFavorite: false,
        commentSetting: 0,
        duetSetting: 0,
        stitchSetting: 0,
        privateAccount: false,
    },
    stats: { videoCount: 361700 },
    shareMeta: {
        title: 'bryceyouloser | ♬ doja x calabria | on TikTok',
        desc: '361.0k videos - Watch awesome short ' + 'videos created with ♬ doja x calabria',
    },
};

Buy Me A Coffee


License


MIT

Free Software

tiktok-scraper's People

Contributors

43081j avatar arjanpronk avatar asportnoy avatar bwmp avatar codacy-badger avatar daawesomerazor avatar davispuh avatar dbellotti avatar dependabot[bot] avatar devbiansor avatar drawrowfly avatar goldenangel2 avatar jcpalacios avatar khoroshcho avatar kmkramer avatar lu3do avatar lucasintel avatar onzu-cb avatar pterko avatar tbm avatar thibaultjanbeyer avatar tnoetzel avatar trodiz avatar vasylkhoroshcho avatar vjchem avatar weavewillg avatar yonas avatar zvictor avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

tiktok-scraper's Issues

fs.promises API is experimental

I see the following message when I try to run the scraper and it exits without downloading any data.
⠋ TikTok Scraper Started(node:19861) ExperimentalWarning: The fs.promises API is experimental

Have you bumped into this?

Can't download all user videos

_Even when setting -n 0, i can only scrape 20-30 videos depending on each users. Redownloading the same user would get the same number of videos.
_I did scraped different users during 24h window, here's just an example:
Username: chiyuu
Total videos: 446.
Downloaded: 28.
In older version i've got 415.
Current version 1.0.12

Collector is empty when using module. Fine with cli

When using the module example, the collector returns an empty array.
Module not working:

(async () => {
    try {
        const posts = await TikTokScraper.user({ USER_ID: SOME_USER_ID_STRING }, { number: 20, by_user_id: true });
        console.log(posts); // { collector: [] }
    } catch (error) {
        console.log(error);
    }
})();

CLI works:

tiktok-scraper user lebeats_211

Any idea what could be causing the difference?

Search by keyword

Hi @drawrowfly

Currently, your node module supports for searching by hashtag, user, trend, music. Do you know a way to search by keyword? Are you planing to add "search by keyword" to this module?

Best,
-Young

Integration into php page

Hello drawrowfly,
I want to integrate your scraper into php website, can you give me your email? Are you available for a custom paid project? Thank you!

Support for Hebrew alphabet?

The scraper doesn't seem to work when the keyword (e.g. the hashtag I search for) is in Hebrew script. The word does show up in Hebrew alphabet in Terminal and I have Unicode (UTF-8) text encoding enabled, so I don't think the issue is with Terminal. Can you advise? Thanks!

collector[]

can you update the signature? seems like not working again.

Nothing to be downloaded

As of 9:50am CET, I can't download video's anymore. What worked the day before, doesn't work now, I'm affraid.

command use: tiktop-scraper user thisisanexampleuser -n 9999 -d -store

commands used to download by music association does not work, either.

Some problems found during use.

Thank you for providing such a useful tool, but when I used it, I found that the videos are watermarked. There are two suggestions to see if it can be achieved:

  1. Can you put the video link without watermark in it?
  2. Can you provide a query to get the video information based on the video id (including the video download link without watermark)

Not scraping videos w/ "This sound isn't available." message.

When viewing TikTok videos via the mobile app, sometimes there is a

This sound isn't available.

message above the video. I noticed that these videos do not get scraped. Is it possible to include them?

IDK if it helps in figuring out the issue, but I've been able to share them to Instagram via the mobile app and download/hear the sound that way.

Love the scraper, thanks!

Scraping is Returning Empty Collector

Scraping started returning an empty collector. I'm guessing the signature is out of data again.

(Btw, how are you reverse engineering the signature? Happy to help maintain if you point me in the right direction.)

videos without watermark

hi, i have a idea/solution for downloading video without watermark
i sniffing in app tiktok in android and i found new api.
but i dont know how to get video id

example :
https://www.tiktok.com/@vikaberk/video/6721248223035411718 >> video id right
but how to convert to
v09044310000bl3ammlqa7lrlgrvj9tg

this is old api to get no wm https://api2.musical.ly/aweme/v1/play/?video_id=v09044310000bl3ammlqa7lrlgrvj9tg&line=0&ratio=720p&watermark=0&media_type=4&vr_type=0&test_cdn=None&improve_bitrate=0&logo_name=tiktok

but right now, not work again with videoid 6721248223035411718

download video

is it possible to download videos without a watermark?

Scrapper doesn't work for Trending

just tried today and it doesn't work - doesn't even create cvs file for trending.
It works perfectly for other API - get hashtag info etc.

Delete stored download history per user?

This isn't really an issue, I'm just not sure where to put questions.

Is it possible to easily delete stored download history per user?

For instance, I want to re-download a user's videos and store the download history.

How do I delete their old download history?

I'm using ubuntu. I can navigate to the tmp folder but I'm not seeing how to identify which history file belongs to which user.

Thanks

unrecognized arguments

can someone explain to me how to fix this?

unrecognized arguments: fyp -n 100 -d -t all

screenshot

Get user profile

Thank you for such amazing library.
is there any way to get user profile info ? Like following/followers count ?

Scrape discover users & hashtags

Trending hashtags are a good way to seed the scraper. These hashtags appear to be different than the hashtags present in the trending posts. The app itself seems to have an infinite scroll of these trending hashtags, but the web interface only loads a fixed number.

https://www.tiktok.com/discover

A feature request, something to investigate when you have time. :)

Scrapping waring

I`m just curious should we be worried from this warning we see every time we use the scrapper:

(node:61341) ExperimentalWarning: The fs.promises API is experimental

Fail to run

I've got Nodejs and installed from NPM, run the example in terminal but didn't work. Am i missing something?

image

Error: Data should not be empty or the "fields" option should be included

I'm getting following error:

Error: Data should not be empty or the "fields" option should be included
    at JSON2CSVParser.preprocessData (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1484:15)
    at JSON2CSVParser.parse (/Users/ehsan/.config/yarn/global/node_modules/json2csv/dist/json2csv.cjs.js:1450:32)
    at Bluebird.fromCallback.cb (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:102)
    at tryCatcher (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/util.js:16:23)
    at Function.Promise.fromNode.Promise.fromCallback (/Users/ehsan/.config/yarn/global/node_modules/Bluebird/js/release/promise.js:209:30)
    at Promise (/Users/ehsan/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:225:44)
    at processTicksAndRejections (internal/process/next_tick.js:81:5)

Cannot find module Bluebird

When runnning with nodejs v10.18.1 and installed with Yarn or npm, I receive this error when trying $tiktok-scraper --help

internal/modules/cjs/loader.js:638
throw err;
^

Error: Cannot find module 'Bluebird'
at Function.Module._resolveFilename (internal/modules/cjs/loader.js:636:15)
at Function.Module._load (internal/modules/cjs/loader.js:562:25)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/index.js:13:18)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/lib/instance.js:3:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)
at Object.Module._extensions..js (internal/modules/cjs/loader.js:789:10)
at Module.load (internal/modules/cjs/loader.js:653:32)
at tryModuleLoad (internal/modules/cjs/loader.js:593:12)
at Function.Module._load (internal/modules/cjs/loader.js:585:3)
at Module.require (internal/modules/cjs/loader.js:692:17)
at require (internal/modules/cjs/helpers.js:25:18)
at Object. (/home/me/.config/yarn/global/node_modules/tiktok-scraper/bin/cli.js:4:23)
at Module._compile (internal/modules/cjs/loader.js:778:30)

_getUserId

i run this project in RDP AWS
but i getting result

collector[]

and i test _getUserId() but getting result {"statusCode":200,"contentType":"text/html","content":""}
how to fix that?

Error when using proxy

Hello,

my setup :
Ubuntu 18.04
NodeJS : I tried version 12 and 13
tiktok scraper : 1.0.12

I keep getting an error message when I'm trying to use a proxy-server . Tried different proxy-servers.
Only appears when I'm trying to use a proxy.

tiktok-scraper trend -d -n 5 -p https://115.23.28.168:8080
Error: Error: [object Object]
at TikTokScraper.extractTac (/usr/local/lib/node_modules/tiktok-scraper/lib/index.js:311:19)

Post Likes

The result does not include the likes count of a post.

How are new posts handled?

Hey there, when downloading posts, how does the program handle new posts for an already downloaded user? Ideally, I'd be able to just run the script again for a user and it would download said user's new posts and their metadata without having to redownload everything. Is this a possibility?

Apologies for opening an issue for this, I couldn't see a way to contact you, drawrowfly.

Getting error 'RequestError: Error: ESOCKETTIMEDOUT'

Code
const TikTokScraper = require('tiktok-scraper');

let posts = TikTokScraper.hashtag("singing", { event: true, number: 10});

posts.on('data', (json) => {
console.log(json)
})

posts.on('done', () => {
console.log("done")
})

posts.on('error', (error) => {
console.log(error)
})

posts.scrape();

Version:
"tiktok-scraper": "^1.0.14

Error:
Error: RequestError: Error: ESOCKETTIMEDOUT
at TikTokScraper.scrapeData (*/node_modules/tiktok-scraper/lib/index.js:616:19)
done

Followers Required

Hi there,
Can you please add number of followers in your api reponse.

Getting error on Windows

Script
const TikTokScraper = require('tiktok-scraper');

(async () => {
try {
const posts = await TikTokScraper.hashtag('singing', { number: 100 });
console.log(posts);
} catch (error) {
console.log(error);
}
})();

Version:
"tiktok-scraper": "^1.0.13"

Error:
...\node_modules\jsdom\lib\jsdom\living\helpers\dates-and-times.js:235
} catch {
^

SyntaxError: Unexpected token {
at createScript (vm.js:80:10)
at Object.runInThisContext (vm.js:139:10)
at Module._compile (module.js:607:28)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\form-controls.js:18:5)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (...\node_modules\jsdom\lib\jsdom\living\helpers\focusing.js:5:24)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)

not showing result

Hi i tested today,
i cant scraping profile, result collector length is 0

Getting TypeError when scraping more than 50

Code app.js:
const TikTokScraper = require('tiktok-scraper');

let posts = TikTokScraper.hashtag("singing", { event: true, number: 200});

posts.on('data', (json) => {
console.log(json)
})

posts.on('done', () => {
console.log("done")
})

posts.on('error', (error) => {
console.log(error)
})

posts.scrape();

Version:
"dependencies": {
"tiktok-scraper": "^1.0.9"
}

Error:
TypeError: Cannot read property 'itemListData' of undefined
at TikTokScraper.mainLoop (/node_modules/tiktok-scraper/lib/index.js:394:53)
at async TikTokScraper.scrape (
/node_modules/tiktok-scraper/lib/index.js:297:9)

Error when downloading by hastags and store

this is the error:

the command:
tiktok-scraper hashtag tiktokchefs -d -n 30 -store

C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\multipleBar.js:42
        this.stream.clearLine();
                    ^

TypeError: this.stream.clearLine is not a function
    at Multibar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\help                                                                                           ers\multipleBar.js:42:21)
    at ProgressBar.bar.terminate (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l                                                                                           ib\helpers\multipleBar.js:33:22)
    at ProgressBar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\node_modul                                                                                           es\progress\lib\node-progress.js:103:10)
    at Multibar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\helpers\m                                                                                           ultipleBar.js:56:17)
    at ProgressBar.bar.tick (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\lib\he                                                                                           lpers\multipleBar.js:28:18)
    at Request.r.get.on.on.chunk (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper\l                                                                                           ib\index.js:117:34)
    at Request.emit (events.js:198:13)
    at IncomingMessage.<anonymous> (C:\Users\_my_user\AppData\Roaming\npm\node_modules\tiktok-scraper                                                                                           \node_modules\request\request.js:1073:12)
    at IncomingMessage.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at IncomingMessage.Readable.push (_stream_readable.js:224:10)
    at HTTPParser.parserOnBody (_http_common.js:122:22)
    at TLSSocket.socketOnData (_http_client.js:442:20)
    at TLSSocket.emit (events.js:198:13)
    at addChunk (_stream_readable.js:288:12)
    at readableAddChunk (_stream_readable.js:269:11)
    at TLSSocket.Readable.push (_stream_readable.js:224:10)
    at TLSWrap.onStreamRead [as onread] (internal/stream_base_commons.js:94:17)

It downloads a file(a corrupt zip file) with one video in it, but just only that.

Download user working from CLI but not from node

Firstly, thanks for all your awesome work on this!

I'm using the latest version (0.0.16), so I do not think it's a duplicate of this issue #44

CLI works just fine. e.g.

tiktok-scraper user USERNAME -n 20 -d -store

Using the node method returns 0 results (was previously working). Any ideas?

const posts = await TikTokScraper.user(USER_ID, {
      by_user_id: true,
      user_data: true,
      number: 10,
    });

"TikTok Scraper Started" message and then program exits.

This morning I tried to scrape a profile and the program immediately exited with no other message after displaying the "TikTok Scraper Started" message.

I updated the program and repeatedly receive the same message. Could it be that my IP is banned/limited?

Thanks

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.