Comments (9)
What do you mean different points in time? And to what model are you referring? What is a run?
from ranking.
@eggie5 - I am reading in data and training my ranker as such:
seed = 1234
config = tf.estimator.RunConfig(tf_random_seed=seed)
hparams = tf.contrib.training.HParams(learning_rate=0.05)
ranker = get_estimator(hparams, config)
ranker.train(input_fn=lambda: input_fn(_TRAIN_DATA_PATH)[0], steps=100)
I then evaluate my model as such:
ranker.evaluate(input_fn=lambda: input_fn(_TEST_DATA_PATH)[0], steps=100)
When I run this command (cell?) multiple times in a row, each time I will obtain a different result for the metric I care about, e.g.:
metric/ndcg@1- run1 - 0.5576208
metric/ndcg@1 - run2 - 0.5582908
metric/ndcg@1 - run3 - 0.55627906
The same applies when I repeatedly call
ranker.predict(input_fn=lambda: input_fn_prediction(_TEST_DATA_PATH_2)[0]))
(I edited the input_fn_prediction
thanks to your help yesterday!)
Each time I only train with the same list of data all belonging to the same qid
.
from ranking.
You didn't provide the actual implementation of your input function but I'll assume you're using the example in the notebook.
My guess is that since you're not evaluating on the whole dataset and since the notebook input function shuffles the queries, you are not evaluating on the same set of queries for each run.
Either remove steps or remove shuffle when you evaluate.
from ranking.
Hi @eggie5 , here's my input_fn_prediction
:
def input_fn_prediction(path):
train_dataset = tf.data.Dataset.from_generator(
tfr.data.libsvm_generator(path, _NUM_FEATURES, _LIST_SIZE),
output_types=(
{str(k): tf.float32 for k in range(1,_NUM_FEATURES+1)},
tf.float32
),
output_shapes=(
{str(k): tf.TensorShape([_LIST_SIZE, 1])
for k in range(1,_NUM_FEATURES+1)},
tf.TensorShape([_LIST_SIZE])
)
)
return train_dataset.make_one_shot_iterator().get_next()
As you'll see, I have removed the shuffling
and the repeat
you mentioned.
Any other ideas? Thank you so much!
from ranking.
did you turn off the shuffle in libsvm_generator
?
from ranking.
Just did that - thank you so much @eggie5 - you rock!
Feel free to close now! :)
from ranking.
Thanks for resolving this issue, @eggie5 ! Much appreciated.
Totally agree with Stefan, you rock!
from ranking.
Hi @sjhermanek @eggie5 ,
I am facing the same issue as @sjhermanek but I am unable to find a shuffle
parameter in the function of libsvm_generator
. I tried looking at other post by @eggie5. However, I tried commenting the np.random.shuffle(doc_list)
in data.py however, it does not seem to be working. Can you let me know how do I solve this issue. This is preventing me from getting the prediction in the order in which they were present in the input file and is causing a change in output each time predict function is running with all else being equal. Please help me out. Thanks!
from ranking.
I have solved it just by commenting out the lines of np.random.shuffle(doc_list)
. Looks like, there was some other issue. Please let me know if there are any other nicer ways to get around this issue
from ranking.
Related Issues (20)
- tfr MRR metric for sparse tensors crashes
- Example deprecated
- Multi-MLP document representation
- How to create tf records for a custom dataset and run TFR Ranking on it?
- package Bert
- is_label_valid in utils.py lacks support for integer targets
- Cannot import functions and classes directly
- how can i use the context features like movie title text in listwise ranking
- Extra loss not added to the overall model loss
- Subscores in neural ranking GAM
- [Feature Request] add some multivariate examples (i.e. DLCM, SetRank)
- MRR=1 for movie lens example, how is it possible?
- Custom loss function with MultiTaskPipeline HOT 2
- v0.5.3 not available on pypi HOT 2
- Negative labels are silently masked
- Inputs are not scaled according to the temperature in pair wise losses
- submitted issue by error
- Clarifying global floating point policy
- Landing page example broken HOT 1
- Inconsistent pairwise hinge loss values when using masked labels
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ranking.