Comments (3)
Just reproduced the paper results! Thank you very much for your help!
temperature 0.2: {‘pass@1’: 0.2910060975609756, ‘pass@10’: 0.4311547000349374, ‘pass@100’: 0.5379728166388384}
temperature 0.6: {‘pass@1’: 0.26408536585365855, ‘pass@10’: 0.5354897815147874, ‘pass@100’: 0.7727023953428368}
temperature 0.8: {‘pass@1’: 0.23518292682926834, ‘pass@10’: 0.528405455080335, ‘pass@100’: 0.7764527488844969}
from codegen.
Thanks for your comment. The results look oddly uncorrelated with the temperatures.
From the snippet, here are the differences:
- The temperature is specified with
temperature
, nottemp
. This may cause ignoring specified temperature values. - We did not use canonical solutions to determine the maximum length. For HumanEval experiments, we used
input_ids_len + 512
.
Additionally, please verify that you are using the corresponding tokenizer for the model.
If you could share your sampling script somewhere (e.g. gist), I'd be happy to take a look.
from codegen.
If it is possible, please share you evaluation script @boblee22
from codegen.
Related Issues (20)
- What is the hardware requirement for fine tuning codegen 2B and higher models?
- memory out of error. Hardware requirements HOT 1
- A question about the detail of data preprocessing
- Limit of code generation HOT 1
- instruct dataset
- Using LoRA with CodeGen 2B mono HOT 2
- How to use infills sampling?
- What is min loss in CodeGen1B while finetuning.
- Clarity on training data for each of the codegen versions
- How to use gpu to accelerate inference? HOT 1
- How much VRAM do I need if I want to enable GPU acceleration? codegen25-7B-instruct
- Set different temperature
- fine tunning : data format
- AttributeError: 'CodeGen25Tokenizer' object has no attribute 'encoder' HOT 3
- What is the context window for Codegen2? HOT 1
- Defect detection
- Error calling tokenizer.get_vocab() (Codegen2.5) HOT 1
- Atrribute Error: 'AlignConfig' object has no attribute 'encoder', 'PoolFormerConfig' object has no attribute 'encoder'. HOT 1
- Which dataset is used for fine-tuning CodeGen25-7B-multi resulting in CodeGen25-7B-mono?
- AttributeError: 'CodeGenTokenizer' object has no attribute 'encoder'. Did you mean: 'encode'? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from codegen.