Comments (4)
Sorry for late reply. The hyper-parameters are a little sensitive and required to be tuned on the given dataset (because it is a unsupervised task). We tune these hyper-parameters according to the model performance on the validation set.
A simple rule is: a larger
from nast.
@hzhwcmhf Thank you for your kind reply, I've followed your instruction to tune and then it comes to evaluation step. And I'm wondering how you select and preprocess the data for the classifier and the ppl model. Could you please share the detailed setting and the script? Thank you so much.
from nast.
In your readme.md, you mentioned that in the classifier training, some samples from the original test set are in the train.txt, and some of them are in the test.txt. Could you please give me more hints why you apply this setting and share more details.
from nast.
@FayeXXX We randomly sample some data from the original training and test set to form the training set for the classification and language models. The final training set is "yelp/classifier|languagemodel/train.txt", and the test set is "yelp/classifier|languagemodel/test.txt".
Here is the reason why we sample some data from the original test set:
Since this is an unsupervised task, the training set only contains unparallel data, so the classifiers never see two sentences having similar contents but different sentiments. Therefore, it can be difficult for the classifier to distinguish the transferred samples (written by human or generated by models) from the original ones. For example, "they will even deliver to the school !" and "they will not deliver to the school !" has opposite sentiments, but a small classifier could easily ignore the not
word and label them with the same sentiment. It is also called as an "adversarial" sample. (Here is a paper to construct adversarial samples http://aclanthology.lst.uni-saarland.de/2020.emnlp-main.498.pdf)
The simplest way to tackle the problem is to do adversarial training (A survey paper). That is, train the model with some adversarial samples. In our case, we add some human-written "adversarial" samples (from the test set) to improve the performance against "the adversarial attack", leading to a better correlation with human evaluation.
from nast.
Related Issues (7)
- Code Release ETA HOT 1
- How to generate data? HOT 1
- Running with Custom Dataset HOT 4
- a problem when running run.py HOT 1
- TypeError HOT 3
- Validation takes very long time (10x) HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nast.