Comments (5)
pyscan is very slow for a repo with 429 3rd-party packages
pyscan has 10 dependencies/crates, as visible in the cargo.toml. These dependencies have their own dependencies, which results in:
160 crates for Linux,
167 crates for windows.
This is not uncommon for a Rust project. I'm not sure where you are picking up 429 "packages" but regardless of it, pyscan uses the most common crates any Rust dev would have used before. It might seem hard for you to acclimatize to, but it gets better over time.
Thanks for letting me know about the batch query, it is better, and I figured sooner or later it would have been the better choice. It is currently being developed and will be released in the next version.
BTW: What is the added value of Rust in app that just parses a text file, makes a HTTP call and formats the results?
Compiling the pyscan-rs takes ages and perhaps a pure Python code could be fast enough?
I understand the sentiment. Pyscan is in its alpha stage and hasn't been through the necessary optimizations any established project would have. The main idea behind it was having a single binary capable of executing what Pyscan claims to do, instead of having to depend on the user having a python runtime. It's useful in terms of a CI where you want to minimize the number of things you install, and I provide releases just for that and other cases.
I don't think language here has any relevance, only implementation. Changing to batch request is a priority and appreciate the concern.
from pyscan.
By the 429 3rd-party packages I meant the number of lines in my project's requirements.txt
-- sorry for the confusion.
So the (runtime) slowness of pyscan I encountered was most likely caused by the 429 remote API calls when scanning for the vulnerabilities in my project.
from pyscan.
I get the point -- I just would not be that afraid of the things installed into the CI image because once a project starts using something like pre-commit, it becomes necessary to install Python and some Python packages into your CI image anyway.
At least that's where we ended up at work. Where I really appreciate Rust's speed is when running the pre-commit hooks (flake8 -> Ruff is a huge step forward - AST parsing is CPU-bound).
from pyscan.
I understand that Python is probably on every developer's system, but I don't think it's worth changing languages now, lol.
This project is experimental atm, and It's my first time messing with Rust as well, so improvements are expected over time as I get better. It's nice to have input from someone who might need it for work, though. Appreciate that.
And yeah, those API calls should be replaced ASAP. Working on it right now. Hopefully I'll be able to optimize the speed to a better level in the future.
from pyscan.
Hey @sarimak batched API is the default way of doing things now. Can you test it out on your big requirements file again? I tried it out with 230+ packages, and it took about 20 seconds to complete. Figure it might take double on yours, curious to see the result though
from pyscan.
Related Issues (13)
- Incorrect version detection of requests package HOT 4
- cannot install via pip on Ubuntu 20.04 HOT 3
- Parsing of dependencies from different build systems HOT 11
- error querying deps with a version qualifier HOT 1
- pyscan seems to depend on pip HOT 7
- Fails to parse ```--hash=``` values embedded in requirements.txt HOT 2
- Exit with zero in case of vulnerability found HOT 4
- prompt/default to when dependency conflict occurs HOT 12
- Add support for constraints.txt HOT 2
- Crashing on my machine HOT 8
- Cannot install pyscan `v0.1.4` on Mac with an older rust compiler (`< v1.70`) HOT 3
- CI fails due to an OpenSSL building issue. HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pyscan.