Giter VIP home page Giter VIP logo

parse-play's Introduction

parse-play

Library for fetching and parsing select data on Android apps from the Google Play Store via undocumented internal APIs.

This library is able to fetch and parse data from undocumented internal API endpoints of the Google Play Store. Currently, it has the following features:

  • Fetch the charts of the most popular apps, including filtering by category and chart.
  • Fetch an app's metadata including data safety labels.
  • Search for apps.

I'll extend the supported API endpoints over time, as per what I need for my projects. The focus will likely be on functions useful for research into mobile privacy and data protection.

As all the used endpoints are undocumented, I had to resort to reverse-engineering the Play Store website, which involved some amount of guessing as to which values mean what. It is possible that I have misinterpreted some of them. It is also entirely possible that some or all of the endpoints will stop working out of the blue at some point, or change their request and/or response formats.

Installation

You can install parse-play using yarn or npm:

yarn add parse-play
# or `npm i parse-play`

API reference

A full API reference can be found in the docs folder.

Usage examples

Fetch app top charts

The following example fetches the current top 100 free apps across all categories for Germany:

import { fetchTopCharts } from 'parse-play';

(async () => {
    const topChart = await fetchTopCharts(
        { category: 'APPLICATION', chart: 'topselling_free', count: 100 },
        { country: 'DE', language: 'EN' }
    );

    console.log(topChart?.length); // 100
    console.log(topChart?.[0]?.app_id, topChart?.[0]?.name); // com.amazon.mShop.android.shopping Amazon Shopping
})();

You can also request multiple top charts at once. These will all be fetched in a single API request. Note that country and language apply to all requests in a batch.

This example fetches the top 5 free education apps, as well as the top 1000 paid adventure game apps, both for the UK:

const topCharts = await fetchTopCharts(
    [
        { category: 'EDUCATION', chart: 'topselling_free', count: 5 },
        { category: 'GAME_ADVENTURE', chart: 'topselling_paid', count: 1000 },
    ],
    { country: 'GB', language: 'EN' }
);

console.log(topCharts[0]?.length); // 5
console.log(topCharts[0]?.[0]?.app_id, topCharts?.[0]?.[0]?.name); // cn.danatech.xingseus PictureThis - Plant Identifier
console.log(topCharts[1]?.length); // 660
console.log(topCharts[1]?.[0]?.app_id, topCharts?.[1]?.[0]?.name); // com.MOBGames.PoppyMobileChap1 Poppy Playtime Chapter 1

Note that despite us trying to fetch 1000 apps for the second chart, only 660 apps were returned. This is a server-side limit.

Fetch app details

The following example fetches the metadata of the Facebook app:

import { fetchAppDetails } from 'parse-play';

(async () => {
    const appDetails = await fetchAppDetails({ appId: 'com.facebook.katana' }, { language: 'EN', country: 'DE' });
    console.log(appDetails.name, 'costs', appDetails.price, 'and was last updated on', appDetails.updated_on);
    // Facebook costs €0.00 and was last updated on 2024-06-13T04:58:13.000Z
})();

Through this endpoint, you can also fetch an app's data safety labels:

const appDetails = await fetchAppDetails({ appId: 'com.facebook.katana' }, { language: 'EN', country: 'DE' });

console.log('Data shared:', appDetails.data_shared);
console.log('Data collected:', appDetails.data_collected);
console.log('Security practices:', appDetails.security_practices);
console.log('Privacy policy URL:', appDetails.privacy_policy_url);
Data safety label response The result looks like this:
Data shared: [
  {
    category: 'Personal info',
    type: 'Name',
    purposes: [ 'Fraud prevention, security, and compliance' ],
    optional: false
  },
  // …
]
Data collected: [
  {
    category: 'Personal info',
    type: 'Name',
    purposes: [
      'App functionality',
      'Analytics',
      'Developer communications',
      'Advertising or marketing',
      'Fraud prevention, security, and compliance',
      'Personalization',
      'Account management'
    ],
    optional: false
  },
  // …
]
Security practices: {
  data_encrypted_in_transit: true,
  can_request_data_deletion: true,
  committed_to_play_families_policy: undefined,
  independent_security_review: undefined
}
Privacy policy URL: https://www.facebook.com/about/privacy/

Search for apps

The following example searches for the term "education":

import { searchApps } from 'parse-play';

(async () => {
    const searchResult = await searchApps({ searchTerm: 'education' }, { language: 'EN', country: 'DE' });
    console.dir(searchResult, { depth: null });
})();
Search apps response The response looks like this:
[
  {
    position: 1,
    app_id: 'de.easysoft.app.education',
    icon_url: 'https://play-lh.googleusercontent.com/KZ19KJw8vrNy6gpRtyzLAGichfxShCU9L2kZdJbnKs6mrKblKqcWBvM5v9QdgEW-SGFR',
    screenshot_urls: [
      'https://play-lh.googleusercontent.com/Bh0sDOl-oOcOtmjKTIXL4eE_vIcDqntnrwqvoi9qylQjptmPnMtZyMkUxUh4JnC0hQ',
      'https://play-lh.googleusercontent.com/vlOZjzYHjRZEwBTWYVWxkWvXMEjtJGJ2tbJQJuNuB89wgXA-MVLM5MwaJOhRMdY7vA',
      'https://play-lh.googleusercontent.com/zEiBcIIuY6LP_BbNZQ5PxxilZMmkf6dOn2XsYCNET5GumPOktuhZPo438QiasoVv5g4l',
      'https://play-lh.googleusercontent.com/XP02HcK1hsyCUdrt9abKiy-KdF0ATB3W5jVVW5StHkxsmrlz22DFXfPbovZhyYjLiqI',
      'https://play-lh.googleusercontent.com/c3pmHB-DkHZ6j3g3LfmgWgdHlIK18jOt-2oFGkh9GTtQwY2aay7C9VO70XnZPX3qJas',
      'https://play-lh.googleusercontent.com/8Pj29QXYfhFlmPrMhNvgXdWeCj4X2n3vubIxoHGgd_w4h4MsE04TftKskB53BHp01XU',
      'https://play-lh.googleusercontent.com/mnyR06BYAQQ66ONQrYMluqALsdpKIV1_M2pKEIYurLlpEdRsE0Yu-AMsOmuPNYk-a8jP'
    ],
    name: 'easySoft App Education',
    rating: 2.739726,
    category: 'Business',
    price: '€0.00',
    buy_url: 'https://play.google.com/store/apps/details?id=de.easysoft.app.education&rdid=de.easysoft.app.education&feature=md&offerId',
    store_path: '/store/apps/details?id=de.easysoft.app.education',
    trailer_url: undefined,
    description: 'With the easySoft App Education, […]',
    developer: 'easySoft. GmbH',
    downloads: '10,000+',
    cover_image_url: 'https://play-lh.googleusercontent.com/mnyR06BYAQQ66ONQrYMluqALsdpKIV1_M2pKEIYurLlpEdRsE0Yu-AMsOmuPNYk-a8jP'
  },
  // …
]

Fetch an app's data safety labels

Warning

The separate function for fetching data safety labels is deprecated and will be removed in a future release. Instead, you can use fetch an app's metadata, which includes the data safety label.

The following example fetches the data safety labels for TikTok in English:

import { fetchDataSafetyLabels } from 'parse-play';

(async () => {
    const labels = await fetchDataSafetyLabels([{ app_id: 'com.zhiliaoapp.musically' }], { language: 'EN', });
    console.dir(labels, { depth: null });
})();
Data safety label response The response looks like this:
{
  name: 'TikTok',
  app_id: 'com.zhiliaoapp.musically',
  developer: {
    name: 'TikTok Pte. Ltd.',
    path: '/store/apps/developer?id=TikTok+Pte.+Ltd.',
    website_url: 'https://www.tiktok.com/',
    email: '[email protected]',
    address: '201 Henderson Road,\n#06-22 Apex@Henderson,\nSingapore 159545 Singapore'
  },
  icon_url: 'https://play-lh.googleusercontent.com/iBYjvYuNq8BB7EEEHktPG1fpX9NiY7Jcyg1iRtQxO442r9CZ8H-X9cLkTjpbORwWDG9d',
  privacy_policy_url: 'https://www.tiktok.com/legal/privacy-policy',
  data_shared: [],
  data_collected: [
    {
      category: 'Location',
      type: 'Approximate location',
      purposes: [
        'App functionality',
        'Analytics',
        'Advertising or marketing',
        'Personalization'
      ]
    },
    // …
  ],
  security_practices: {
    data_encrypted_in_transit: true,
    can_request_data_deletion: true,
    committed_to_play_families_policy: undefined,
    independent_security_review: undefined
  }
}

You can also request the labels for multiple apps at once by adding corresponding objects to the first parameter, they will all be fetched in a single API request.

License

parse-play is licensed under the MIT license, see the LICENSE file for details.

Issues and pull requests are welcome!

parse-play's People

Contributors

baltpeter avatar mal-tee avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

parse-play's Issues

"require() of ES Module not supported"

I'm getting an error when trying to run this:

"Error [ERR_REQUIRE_ESM]: require() of ES Module...not supported."

According to Stack Overflow, node-fetch v3 removed support for require() in favor of ES Modules.

Top charts: Server-side limit seems to have changed

Whereas previously the top charts would have up to 660 entries, I'm now only getting 200 apps per category. It seems like the server-side limit has changed.

We should check whether that's something we can influence through the request or whether we'll have to accept that (and update the documentation accordingly).

App details

In parse-tunes, we got app details/metadata for free along with the privacy labels, but here that is a separate endpoint.

Search suggestions

I'm not going to implement this not, but it would be nice to have support for getting search suggestions.

Here's the corresponding request from the website:

await fetch("https://play.google.com/_/PlayStoreUi/data/batchexecute?rpcids=teXCtc&source-path=%2Fstore%2Fapps&f.sid=9122600153491381881&bl=boq_playuiserver_20240515.05_p0&hl=en-US&authuser&soc-app=121&soc-platform=1&soc-device=1&_reqid=956358&rt=c", {
    "credentials": "include",
    "headers": {
        "User-Agent": "Mozilla/5.0 (Windows NT 10.0; rv:126.0) Gecko/20100101 Firefox/126.0",
        "Accept": "*/*",
        "Accept-Language": "en-US,en;q=0.7,de;q=0.3",
        "X-Same-Domain": "1",
        "Content-Type": "application/x-www-form-urlencoded;charset=utf-8",
        "Sec-GPC": "1",
        "Sec-Fetch-Dest": "empty",
        "Sec-Fetch-Mode": "cors",
        "Sec-Fetch-Site": "same-origin"
    },
    "referrer": "https://play.google.com/",
    "body": "f.req=%5B%5B%5B%22teXCtc%22%2C%22%5Bnull%2C%5B%5C%22facebook%5C%22%5D%2C%5B10%5D%2C%5B2%2C1%5D%2C4%5D%22%2Cnull%2C%22generic%22%5D%5D%5D&",
    "method": "POST",
    "mode": "cors"
});

Response:

)]}'

923
[["wrb.fr","teXCtc","[[[\"facebook\",null,[null,null,null,null,[null,null,\"/store/search?q\\u003dfacebook\\u0026c\\u003dapps\"]],null,null,[\"CCO6AgQIARAA\"]],[\"facebook lite\",null,[null,null,null,null,[null,null,\"/store/search?q\\u003dfacebook+lite\\u0026c\\u003dapps\"]],null,null,[\"CCO6AgQIARAA\"]],[\"facebook messenger\",null,[null,null,null,null,[null,null,\"/store/search?q\\u003dfacebook+messenger\\u0026c\\u003dapps\"]],null,null,[\"CCO6AgQIARAA\"]],[\"facebook video downloader\",null,[null,null,null,null,[null,null,\"/store/search?q\\u003dfacebook+video+downloader\\u0026c\\u003dapps\"]],null,null,[\"CCO6AgQIARAA\"]],[\"facebook whatsapp\",null,[null,null,null,null,[null,null,\"/store/search?q\\u003dfacebook+whatsapp\\u0026c\\u003dapps\"]],null,null,[\"CCO6AgQIARAA\"]]],[\"CAhKAggD\"],[[null,[[172800],null,[604800]]]]]",null,null,null,"generic"],["di",28],["af.httprm",27,"-7204844960192752240",99]]
25
[["e",4,null,null,959]]

image

Only 200 positions?

Now it's possible to take only 200 pos from the top. Maybe you know solutions to take 660 like before?

Searching

For the tweasel platform, we'll need to be able to search for apps.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.