Giter VIP home page Giter VIP logo

scarf-js's Introduction

scarf-js

npm version Join the Scarf Community Slack

Scarf is like Google Analytics for your npm packages. By sending some basic details after installation, this package can help you can gain insight into how your packages are used and by which companies. Scarf aims to help open-source developers fund their work when it is used commercially.

To read more about why we wrote this library, check out this post on the topic.

Features

  • No dependencies.
  • Fully transparent to the user. Scarf will log its behavior to the console during installation. It will never silently report analytics for someone that hasn't explictly given permission to do so.
  • Never interrupts your package installation. Reporting is done on a best effort basis.

Installing

You'll first need to create a library entry on Scarf. Once created, add a dependency on this library to your own:

npm i --save @scarf/scarf

Once your library is published to npm with this change, Scarf will automatically collect stats on install, no additional code is required!

Head to your package's dashboard on Scarf to see your reports when available.

Configuring

Users of your package will be opted in by default and can opt out by setting the SCARF_ANALYTICS=false environment variable. If you'd like Scarf analytics to instead be opt-in, you can set this by adding an entry to your package.json

// your-package/package.json

{
  // ...
  "scarfSettings": {
    "defaultOptIn": false
  }
  // ...
}

Scarf will now be opt-out by default, and users can set SCARF_ANALYTICS=true to opt in.

Regardless of the default state, Scarf will log what it is doing to users who haven't explictly opted in or out.

By default, scarf-js will only trigger analytics when your package is installed as a dependency of another package, or is being installed globally. This ensures that scarf-js analytics will not be triggered on npm install being run within your project. To change this, you can add:

// your-package/package.json

{
  // ...
  "scarfSettings": {
    "allowTopLevel": true
  }
  // ...
}

Full Configuration Example

// your-package/package.json

{
  // ...
  "scarfSettings": {
    // Toggles whether Scarf is enabled for this package
    "enabled": true,
    // Enables Scarf when users run npm install directly in your repository
    // Scarf will try to report the Git commit SHA of your repository if it can
    // be obtained.
    "allowTopLevel": true,
    // Users will be opted into analytics by default
    "defaultOptIn": true,
    // By default, Scarf searches for its own location in your build's dependency
    // graph to ensure reporting can be done for all packages using Scarf.
    // For large projects with lots of dependencies, generating that dependency
    // graph takes more time than Scarf allots for its entire process, so Scarf
    // will always time out. `skipTraversal` is an optional flag for large
    // applications to skip that traversal entirely. Use this flag with caution and
    // care, as it will break Scarf analytics for all other packages you depend
    // on in your build.
    "skipTraversal": false
  }
  // ...
}

FAQ

What information does scarf-js provide me as a package author?

  • Understanding your user-base
    • Which companies are using your package?
    • Is your project growing or shrinking? Where? On which platforms?
  • Which versions of your package are being used?

As a user of a package using scarf-js, what information does scarf-js send about me?

Scarf does not store personally identifying information. Scarf aims to collect information that is helpful for:

  • Open Source package maintainence
  • Open Source commercialization

Specifically, scarf-js sends:

  • The operating system you are using
  • Your IP address will be used to look up any available company information. Scarf does not store the actual IP address
  • Limited dependency tree information. Scarf sends the name and version of the package(s) that directly depend on scarf-js. Additionally, scarf-js will send SHA256-hashed name and version for the following packages in the dependency tree:
    • Packages that depend on a package that depends on scarf-js.
    • The root package of the dependency tree. This allows Scarf to provide information for maintainers about which public packages are using their own, without exposing identifying details of non-public packages.

You can have scarf-js print the exact JSON payload it sends by setting SCARF_VERBOSE=true in your environment.

As a user of a package using scarf-js, how can I opt out of analytics?

Scarf's analytics help support developers of the open source packages you are using, so enabling analytics is appreciated. However, if you'd like to opt out, you can add your preference to your project's package.json:

// your-package/package.json

{
  // ...
  "scarfSettings": {
    "enabled": false
  }
  // ...
}

Alternatively, you can set this variable in your environment:

export SCARF_ANALYTICS=false

You can also set this variable in accordance to the Console Do Not Track standard:

export DO_NOT_TRACK=1

Either route will disable Scarf for all packages.

I distribute a package on npm, and scarf-js is in our dependency tree. Can I disable the analytics for my downstream dependents?

Yes. By opting out of analytics via package.json, any package upstream will have analytics disbabled.

// your-package/package.json

{
  // ...
  "scarfSettings": {
    "enabled": false
  }
  // ...
}

Installers of your packages will have scarf-js disabled for all dependencies upstream from yours.

Developing

Setting the environment variable SCARF_LOCAL_PORT=8080 will configure Scarf to use http://localhost:8080 as the analytics endpoint host.

Future work

Future releases of scarf-js will provide a module of utility functions to collect usage analytics in addition to the current installation analytics.

Community

Join the Scarf-Community workspace on Slack and find us in the #scarf-js channel. We'll keep an eye out for your questions and concerns.

scarf-js's People

Contributors

aviaviavi avatar dependabot[bot] avatar justinwoo avatar havi avatar dysinger avatar nshipman-io avatar philderbeast avatar yashpandey06 avatar

Stargazers

Merlin Moelter avatar Stefano Faieta avatar Vitaly Zadorozhny avatar  avatar Himanshu Gilani avatar Nithin R avatar Patrick Roos avatar  avatar Leonardo Dimarchi avatar Silvio Ronaldo avatar Tokarev Igor avatar Nliver avatar Ankesh Bharti avatar Simon Zhou avatar Jan Žák avatar Alex avatar Noorullah Ahmadzai avatar Adnan Ahmed avatar Rodinei Costa avatar Matthew McAchran avatar Siddharth avatar Vlad avatar Dmitriy  avatar Ivica Čardić avatar Ben Blackmore avatar  avatar  avatar George Nurijanian avatar Marquel Waites avatar Christian Gill avatar Alex Pineda avatar Joohun, Maeng avatar Raju avatar Peter Hedenskog avatar Justin Dorfman avatar Jeeyune avatar Yasin ATEŞ avatar Joe avatar Samet Karakus avatar André Mazayev avatar Lukas Elmer avatar Marcus R. Brown avatar Nicholas Meyers avatar Rowland I. Ekemezie avatar Hans Chan avatar Jim Buck avatar NealST avatar Simon Fridlund avatar Jadhiel Vélez avatar Sebastian Sobociński avatar Dharmesh Kakadia avatar Omar Magoury avatar Komil Sobitov avatar Subhan Bakh avatar Jonatan E. Salas avatar Henry avatar Gabin Aureche avatar Max Stoiber avatar Johannes Schickling avatar JJ avatar Sang Nguyen avatar Luiz Fernando da Silva Cieslak avatar of avatar Nikita avatar Greg Bergé avatar  avatar Dan avatar Eka Prasetia avatar Sora Morimoto avatar Ali Gasymov avatar  avatar Pedro Filho avatar Ifiok Jr. avatar savi2w avatar Deep Sheth avatar Matt Love avatar Aleksandr Shreyner avatar Emily Marigold Klassen avatar Rafael Escobar avatar Tyler Maran avatar Anjolaoluwa avatar GuangTao Zhang avatar David Johnson avatar Jacky avatar Evan Tahler avatar Eliza Zhang avatar Sunny Singh avatar Azim Kurt avatar Sergey Glazov avatar Florian Treml avatar Erlend Sogge Heggen avatar Hyan Mandian avatar Eric Clemmons avatar fhuel avatar António Santos avatar Ahsan Sher avatar Michael Liquori avatar Alexander Jeurissen avatar Eyal Cohen avatar David Tran avatar

Watchers

James Cloos avatar Brady Ouren avatar  avatar  avatar Alexander Biehl avatar  avatar Holly Springsteen avatar Fabio Luz avatar  avatar Nliver avatar

scarf-js's Issues

Some questions

Hi @aviaviavi scarf is an awesome package, i hope it will be soon widely adapted by os libs.

I am having some questions related to scarf, mainly in context of disabling it for projects :) due to security policies :-

  1. I have gone through docs and saw scarf can be disabled through packages json, but is there any way to switch off the same for multiple projects like monoreps etc..

  2. Is it possible to have switch based on apps, for eg. say i have lib LibA that uses scarf, and this lib is used in AppA and AppB, can i switch it off for one of them ?

  3. Say this the deps tree, LibD -> LibC -> LibB -> LibA -> scarf , and scarf is disabled through package.json in LibC will it be disabled for any application which uses LibD

Fail silently when API JSON response parsing failed

I use react-table in a secure build environment. API requests from the build gets blocked. So when scarfJS tried to send same, it got blocked. I tried adding disable package.json config but didn't work. Following is the error observed

16:26:52 
16:26:52 SyntaxError: Unexpected end of JSON input
16:26:52     at JSON.parse (<anonymous>)
16:26:52     at /local/apps/test/frontend/node_modules/@scarf/scarf/report.js:65:27
16:26:52 npm ERR! code ELIFECYCLE
16:26:52 npm ERR! errno 1
16:26:52 npm ERR! @scarf/[email protected] postinstall: `node ./report.js`
16:26:52 npm ERR! Exit status 1
16:26:52 npm ERR! 
16:26:52 npm ERR! Failed at the @scarf/[email protected] postinstall script.
16:26:52 npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
16:26:52 

Yarn support

Right now Scarf uses npm ls output to determine dependency info for finding scarfSettings in a package.json. We'll need parity for users that are using yarn.

Validate behavior with non-node runtimes

Scarf should send runtime information, including when the runtime is not node. We have only tested Scarf-js with node so far. I'm not sure how it will behave with Deno, and I believe Bun currently surpresses postInstall hooks altogether if you're not in the top N packages.

Flag to circumvent npm ls to avoid timeouts for projects with a high number of depencies.

Background

When scarf-js runs, it needs to figure out where in the dependency tree it lives. IE, which package is importing Scarf?

In order to achieve this, we run a shell command in order to find the path to Scarf in the current project dependency tree:

npm ls @scarf/scarf --json --long

However, on a project with lots and lots of dependencies, this command is quite slow:

~/d/f/s/superset-frontend ❯❯❯ time npm ls @scarf/scarf --json --long

npm ls @scarf/scarf --json --long  7.00s user 1.62s system 130% cpu 6.585 total

7 seconds on my mac! 😱

The runtime of this single command is way beyond the 3-second timeout scarf-js gives itself to run before quitting, in order to never block npm install.

The proposed solution below is what we decided to do with the Superset team, as other options (eg, bumping the timeout) all had significant drawbacks.

You are need to know IP ...

Why?

"Your IP address will be used to look up any available company information"

But it's the lie =)

Pls, explain me, how is it works technically.
How are you to grab a company info with the ip's knowledge?

Why wasn't I warned about this?

This package was installed without my Consent.
I'd like to ask that you warn users prior to collecting any Analytics on them.

Furthermore, could you please provide me with your analytics endpoint server?
I wish to permanently prevent my computer from sending any further unsolicited Data to you.

Discussion around Scarf's analytics

TL;DR: I created a gist which can patch Scarf Analytics (@scarf/scarf on npm) with this Gist that replaces this lib on postinstall because NPM/Yarn gives you an ability to install any gist or repository like any package to node_modules folder.

Before you install it, delete @scarf/scarf folder in your local project node_modules folder if present (probably you are using some of this packages) or if you are going to install any of those packages in your current project.

Install mock version from Gist instead of local project version which sends your data with this commands depending of what package manager you use:

npm install --save-dev gist:bd5c18861b76eb34f068bf2ed7de903e
yarn add gist:bd5c18861b76eb34f068bf2ed7de903e --dev
pnpm install --save-dev gist:bd5c18861b76eb34f068bf2ed7de903e

It will install mocked version of Scarf Analytics in devDependencies.
The mock just gives you clear vision on what is running on postinstall. It just console.logs that scarf was mocked.

You can opt out from running postinstall scripts using npm install --ignore-scripts, yarn install --ignore-scripts or, if you use pnpm, pnpm install --ignore-scripts but be careful because some your packages will not work, like node-sass because they require compiling it or they just ask you to fund them which is good because you support good open-source developers and their works like the main JS polyfill on which works babel - core-js. Better support its developer for his job for JS ecosystem.
You probably can manually run postinstall hooks for each package you need by running it in node_modules/dependency_that_needs_building_or_compiling

npm/yarn/pnpm run postinstall

You are abke to globally disable running postinstall hooks via npm/yarn/pnpm config to be more secure.
https://blog.npmjs.org/post/141702881055/package-install-scripts-vulnerability
Don't forget about vulnerabilities like:
https://www.zdnet.com/article/microsoft-spots-malicious-npm-package-stealing-data-from-unix-systems/
https://www.zdnet.com/article/cryptocurrency-startup-hacks-itself-before-hacker-gets-a-chance-to-steal-users-funds/
https://www.zdnet.com/article/hacker-backdoors-popular-javascript-library-to-steal-bitcoin-funds/
https://eslint.org/blog/2018/07/postmortem-for-malicious-package-publishes
https://blog.npmjs.org/post/173526807575/reported-malicious-module-getcookies
https://twitter.com/o_cee/status/892306836199800836)
https://medium.com/@jsoverson/how-two-malicious-npm-packages-targeted-sabotaged-one-other-fed7199099c8
https://blog.sonatype.com/sonatype-2020-0003-npm-malicious-package-1337qq-js

I am working on scarf patcher which will on library blacklist patch your package.json with this gist instead of running scarf:
https://www.npmjs.com/package/itssummernoscarfs

Well, let me explain a bit why it is bad idea.
So, you say this is "Google Analytics-like" for OSS libs, but NPM gives you ability to see how much users installed per week.
BTW, this package has ~300k downloads per week only because of redux-form, final-form, react-table.
Why do you need to see which OS users install lib?
There is 3 main OS: Windows (+Windows Subsystem for Linux), Linux, MacOS.
Look at the most popular from StackOverflow Survey 2020:
https://insights.stackoverflow.com/survey/2020#technology-platforms
Also, users when create issue tell you what OS they use.
Next, the OS architecture. Are you sure that architecture value will break lib? There is 3 most popular architectures: x64, x32 (x86) and ARM.
Library users can talk via issues if anybody has some problems even without scarfjs.
Also as I heard library sends IP adresses, what means also country and city to your server and, maybe, deletes it or not, but we don't know yet, we don't see the server code.
Also you are not protected from hacking your database, server or other and leaking data.
Somebody mentioned later that this library not GDPR-friendly and don't forget about NDA, because it's can be private or etc.
There is variant of creating issue like "Who uses *?" and there is a choice to say about company or no. What if my project is private and I doesn't want to leak any data about it?
Why this library gets data about my dependency list? You tell that is gets info about most used lib combo with those who has @scarf/scarf in their dependencies but then you say "Limited dependency tree information. Scarf sends the name and version of the package(s) that directly depend on scarf-js". What I think as lib user I mentioned above.
It sends data on EACH install and doesn't opt-out by default and there is already 300k download and it is used in ~1300 repos on GitHub regarding with "Used by" section on https://github.com/scarf-sh/scarf-js.
Also it takes some time to run every time it's postinstall script and can take up to minute depending on project.
Also, I forget it WILL run on CI every time on build!

I think this is spyware/rootkit/trojan which cannot be predicted in future, it looks like cancer of JS ecosystem and like any disease has to be cure at the very start/has to be deleted by antivirus before it could be worse.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.