Comments (3)
Thanks again for another good idea @Seb0
We will talk about this internally. Our framework is targeted towards making NLP as easy as possible to run and deploy on your infastructure. So pipenv could be a good way to overcome the python dependency mess, especially for production environments.
Some questions pop up though:
Are there any downsides to using pipenv? Are there alternatives? What about other big ML frameworks: what are they using or why are/aren't they using pipenv?
Having answers to these questions would greatly speed up our decision process. So if you want to go ahead and give it the answers a try that would be fantastic.
from farm.
Potential Downsides: pipenv's philosophy is, that it wants to establish a deterministic reproducible application. So if you wanted python 2 and python 3 environments -- for example to run as 2 separate environments (e.g. for tests) -- you're out of luck with pipenv.
Also, diving deeper into this, I realized that there seems to be a divide between applications and libraries in terms of how to handle dependency issues in python:
Pipenv is designed solely for packaging applications, per the maintainers' admission. It's not suited for libraries and is not designed to be. If you want to replace pip, you should have a look at Poetry.
Another alternative would be the classical way of pinning dependencies and pip freeze, also explained in article from realpython (see above).
The way you're handling it with setup.py is also a way that some people seem to like.
As to why this is not the case for some of the bigger libs? I don't know. But Data Scientists aren't the best developers sometimes :D
Checkout this article (and maybe even the associated reddit discussion that is linked)
https://medium.com/@DJetelina/pipenv-review-after-using-in-production-a05e7176f3f0
from farm.
Thanks for your evaluation and the articles!
It's not suited for libraries and is not designed to be.
As I understand this, pipenv is a great project but probably not the best choice for FARM. Since we also didn't face any problems yet due to dependency versioning, this doesn't have highest priority in our backlog. We will therefore (for now) keep the standard approach via setup.py and pin versions when issues pop up.
from farm.
Related Issues (20)
- MTL Processor QA + Classification HOT 1
- Querying API Docker examples HOT 1
- Should be possible to use the proper aggregated loss for early stopping HOT 3
- AdaptiveModel.convert_to_onnx does not save float16 model conversion to output_path HOT 1
- ONNXAdaptiveModel causes NameError: name 'onnxruntime' is not defined HOT 1
- Error reporting using other pre training models HOT 2
- how to predict on single data points for classification problem.? HOT 2
- Error Importing Inferencer HOT 4
- Retreiver Fine Tuning : Are language models like roberta, gpt2 supported to use in retreiver? HOT 3
- Can't train a language model HOT 4
- Max token size? HOT 2
- summarization HOT 1
- Combine several models into one with several prediction heads HOT 2
- Need a guidance on Multi label Classification HOT 2
- Extract embedding while using parameter "extraction_strategy="per_token"" HOT 1
- Which pytorch (and other package) versions are actually required HOT 4
- Current version `0.8.1-snapshot` is not valid according to PEP 440 and causes installation problems HOT 1
- Columns and DataType Not Explicitly Set on line 147 of wordembedding_utils.py
- Installation error
- IndexError: too many indices for array: array is 2-dimensional, but 3 were indexed
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from farm.