Comments (3)
Quite strange, I got 40.2% with the same script using another conda environment.
{
"humaneval": {
"pass@1": 0.4024390243902439
},
"config": {
"prefix": "",
"do_sample": false,
"temperature": 0.2,
"top_k": 0,
"top_p": 0.95,
"n_samples": 1,
"eos": "<|endoftext|>",
"seed": 0,
"model": "/mnt/models/CodeLlama-7b-Python",
"modeltype": "causal",
"peft_model": null,
"revision": null,
"use_auth_token": true,
"trust_remote_code": true,
"tasks": "humaneval",
"instruction_tokens": null,
"batch_size": 1,
"max_length_generation": 512,
"precision": "bf16",
"load_in_8bit": false,
"load_in_4bit": false,
"limit": null,
"limit_start": 0,
"postprocess": true,
"allow_code_execution": true,
"generation_only": false,
"load_generations_path": null,
"load_data_path": null,
"metric_output_path": "/root/ch/bigcode-evaluation-harness/outputs/metrics/humaneval_codellama_7b_python.json",
"save_generations": true,
"save_generations_path": "/root/ch/bigcode-evaluation-harness/outputs/generations/humaneval_codellama_7b_python.json",
"save_references": false,
"prompt": "prompt",
"max_memory_per_gpu": "auto",
"check_references": false
}
}
What happened? The former environment is created using the requirement.txt with python=3.7.10, while the next one has python of 3.10.0
from bigcode-evaluation-harness.
I haven't tried execution with python 3.7 but it could be that the execution of the generated solution failed in that env. Try evaluationg the generations you got with python 3.10 in your python 3.7 env using evaluation only mode by providing this flag
--load_generations_path /root/ch/bigcode-evaluation-harness/outputs/generations/humaneval_codellama_7b_python.json
https://github.com/bigcode-project/bigcode-evaluation-harness/tree/main#evaluation-only
from bigcode-evaluation-harness.
Thx @loubnabnl, I've figured it out :>
from bigcode-evaluation-harness.
Related Issues (20)
- If I want to add my own designed prompts before each question, how should I modify the code HOT 1
- Make `bigcode_eval` pip installable HOT 1
- The result of llama2-13b-chat(pass@1 18) is worse than paper(pass@1 37) HOT 2
- API-based evaluation support (humanevalpack_openai.py is too old)
- Run 70b evaluation.
- Fix MBPP bug with transformers 4.38+ HOT 6
- How can I pass num_beams? HOT 2
- Some questions about APPS HOT 1
- Following submission documentation fails to save generated model outputs for our(Jdoodle) private HF model HOT 3
- Value error regarding to "--max_length_generation" HOT 1
- Using custom prompts and postprocessing HOT 1
- MBPP Llama3-8B-Instruct lower pass@1 score expected HOT 1
- Hi
- Using the humanevalpack to test the ChatGLM3 model results in an abnormal score.
- The evaluation results are inconsistent across different GPUs
- Need some context for certain args for Instruct Human Eval HOT 2
- [Possibly system specific] Wild (12% vs 20%) run-to-run swings in `multiple-cpp` reported scores HOT 1
- `multiple-cs`, `multiple-go`, and `multiple-d` are broken (`FileNotFound`) HOT 2
- HumanEval evaluation results mismatch for Codegemma-2b HOT 1
- Humaneval and MBPP results of deepseek-6.7b-coder-instruct are lower than offical report of Deepseek team
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bigcode-evaluation-harness.