Are predictions stable over time? about ranking HOT 9 CLOSED

What do you mean different points in time? And to what model are you referring? What is a run?

from ranking.

@eggie5 - I am reading in data and training my ranker as such:
seed = 1234
config = tf.estimator.RunConfig(tf_random_seed=seed)
hparams = tf.contrib.training.HParams(learning_rate=0.05)
ranker = get_estimator(hparams, config)
ranker.train(input_fn=lambda: input_fn(_TRAIN_DATA_PATH)[0], steps=100)

I then evaluate my model as such:
ranker.evaluate(input_fn=lambda: input_fn(_TEST_DATA_PATH)[0], steps=100)

When I run this command (cell?) multiple times in a row, each time I will obtain a different result for the metric I care about, e.g.:
metric/ndcg@1- run1 - 0.5576208
metric/ndcg@1 - run2 - 0.5582908
metric/ndcg@1 - run3 - 0.55627906

The same applies when I repeatedly call
ranker.predict(input_fn=lambda: input_fn_prediction(_TEST_DATA_PATH_2)[0])) (I edited the input_fn_prediction thanks to your help yesterday!)
Each time I only train with the same list of data all belonging to the same qid.

from ranking.

You didn't provide the actual implementation of your input function but I'll assume you're using the example in the notebook.

My guess is that since you're not evaluating on the whole dataset and since the notebook input function shuffles the queries, you are not evaluating on the same set of queries for each run.

Either remove steps or remove shuffle when you evaluate.

from ranking.

Hi @eggie5 , here's my input_fn_prediction:

def input_fn_prediction(path):
    train_dataset = tf.data.Dataset.from_generator(
        tfr.data.libsvm_generator(path, _NUM_FEATURES, _LIST_SIZE),
        output_types=(
            {str(k): tf.float32 for k in range(1,_NUM_FEATURES+1)},
            tf.float32
        ),
        output_shapes=(
            {str(k): tf.TensorShape([_LIST_SIZE, 1])
             for k in range(1,_NUM_FEATURES+1)},
            tf.TensorShape([_LIST_SIZE])
        )
    )

    return train_dataset.make_one_shot_iterator().get_next()

As you'll see, I have removed the shuffling and the repeat you mentioned.

Any other ideas? Thank you so much!

from ranking.

did you turn off the shuffle in libsvm_generator?

from ranking.

Just did that - thank you so much @eggie5 - you rock!
Feel free to close now! :)

from ranking.

Thanks for resolving this issue, @eggie5 ! Much appreciated.

Totally agree with Stefan, you rock!

from ranking.

Hi @sjhermanek @eggie5 ,

I am facing the same issue as @sjhermanek but I am unable to find a shuffle parameter in the function of libsvm_generator. I tried looking at other post by @eggie5. However, I tried commenting the np.random.shuffle(doc_list) in data.py however, it does not seem to be working. Can you let me know how do I solve this issue. This is preventing me from getting the prediction in the order in which they were present in the input file and is causing a change in output each time predict function is running with all else being equal. Please help me out. Thanks!

from ranking.

I have solved it just by commenting out the lines of np.random.shuffle(doc_list). Looks like, there was some other issue. Please let me know if there are any other nicer ways to get around this issue

from ranking.