Comments (2)
loubnabnl
commented on July 20, 2024
Hi, could give more details on the command you're trying to run with the full stack trace. Also did you follow all the DS100 installation instructions here it has more requirements that the other tasks (it could also be that you have a module named lm_eval from another library like the LLM harness so they are getting confused I would suggest having a separate environement)
from bigcode-evaluation-harness.
dhingratul
commented on July 20, 2024
This is solved, i was working in an offline manner, and had to manage the DS1000 dependencies correctly.
from bigcode-evaluation-harness.
Related Issues (20)
- The results are different from those of the codellama paper HOT 6
- solved
- code HOT 1
- Multi card operation large model HOT 2
- Is it possible to run the harness against API hosted models? HOT 3
- Phi 1.5 evaluation problem HOT 2
- Humanevalpack-c++ issue HOT 1
- Evaluation of instruct model HOT 8
- Score discrepancy with humaneval HOT 2
- A common interface for APIs and Models. HOT 9
- exact commandline to reproduce leaderboard HOT 2
- Anyway to access to the public leaderboard? HOT 1
- Inconsistencies in results in Mistral, CodeLlama and some strange behavior. HOT 11
- Reproduction Issues of Code Llama HOT 3
- OOM when running Multi-gpu inference using A100 40GB HOT 1
- Evaluation speed with multi-gpu HOT 2
- Try to reproduce the leaderboard result of codellama-instruct-34B for Rust HOT 6
- Human_eval .yaml HOT 1
- Why doesn't mbpp use few-shot? HOT 1
- Use trust_remote_code for dataset load HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigcode-evaluation-harness.