Comments (4)
Hi @chris-aeviator ,
a) the labeled (training) data is not split at all. In case of fewglue, this means that TRAIN_FILE_NAME
should point to a file containing all 32 examples, whereas DEV_FILE_NAME
and TEST_FILE_NAME
should point to files containing the original dev/test examples. Note that the dev examples are not at all used during training or for hyperparameter optimization; just like the test examples, they are only used for evaluation. If you have no dev examples, you can simply set def get_dev_examples(self, data_dir: str)
to return an empty list.
b) yes, but only for the individual models and not for the final distilled classifier. If you need predictions for the unlabeled data, you can simply set TEST_FILE_NAME = UNLABELED_FILE_NAME
. The result is then stored in a file predictions.jsonl
where each line is of the form {"idx": <IDX>, "label": "<LABEL>"}
where <IDX>
is the index of the example in the test file and <LABEL>
is the predicted label.
from pet.
I'm closing this issue for now. Feel free to reopen it if you have further questions.
from pet.
Hi @timoschick ,
I am running PET for a custom task with --model_type bert . In the --data_dir I have 4 files train.csv, test.csv, dev.csv, unlabeled.csv.
In the shell script, I have:
--do_train
--do_eval
Now in the output, I always get the predictions.jsonl file. The UNLABELED_FILE_NAME = "unlabeled.csv", so it is not set to other datasets. However, in the predictions file I thought I was getting model predictions of dev.csv. I tested it with different number of samples for each file train/test/dev/unlabeled and the number of rows in predictions.jsonl matched with that of dev set. Is it by default predictions file (located in the final folder) showing the predictioins of dev.csv?
from pet.
More info on what I said earlier...
@timoschick I run two similar experiments on the same dataset (playing with unlabeled sample size)
Experiment A settings:
- Balanced Dataset
- train (50 samples per class)
- test (150 samples per class)
- dev (150 samples per class)
- unlabeled (10 samples per class)
predictions.jsonl
file ha 300 predicted labels in total
Experiment A predictions.jsonl
has predictions labels (300 samples) of only one class
Experiment B settings:
- Balanced Dataset
- train (50 samples per class)
- test (150 samples per class)
- dev (150 samples per class)
- unlabeled (100 samples per class)
predictions.jsonl
file has 300 predicted labels in total
Experiment B predictions.jsonl
file has predictions labels from both classes
My task is a classification problem with two labels but I don't understand what's the role of unlabeled data in this case and why is it impacting the result.
from pet.
Related Issues (20)
- OSError: Model name 'clue/albert_chinese_tiny' was not found in tokenizers model name list HOT 2
- TypeError: expected str, bytes or os.PathLike object, not NoneType HOT 1
- ZeroDivisionError when reduction is set to 'wmean' while training iPET HOT 1
- Stuck at 'There are 0 Examples for Label [my_label]' During iPETs Example Selection for Next Generation
- There is no softmax
- Training Time Issue HOT 4
- GenPET commands with GPT-2 HOT 2
- RuntimeError on eval method HOT 2
- Token indices sequence length is longer than the specified maximum sequence length for this model HOT 2
- PET and iPET parameters
- Random seed parameter for iterations
- Data format for few-shot text classification
- Roberta-large using BPE tokenizer generates multi tokens. HOT 6
- API Usage
- How much test data do you use in your experiments?
- How to training PET model uses xlm-roberta with byte-level Byte-Pair-Encoding?
- How to reproduce results of the paper? HOT 1
- Clarification on how to interpret PET's results HOT 2
- A question about “Automatically Identifying Words That Can Serve as Labels for Few-Shot Text Classification”
- RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)` HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pet.