Comments (10)
ciprian-chelba
commented on May 26, 2024
Hi,
The training data is:
1-billion-word-language-modeling-benchmark/training-monolingual.tokenized.shuffled/news.en-000??-of-00100
As you will notice, the fileglob expansion is missing the
news.en-00000-of-00100 file which is used as held-out data:
1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en-00000-of-00100
Since that is a bit large, we sharded it 50-way, giving us 50 smaller sets
for evaluation, parameter tuning, etc. The test set on which we reported
results in the paper is:
1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050
You can find more details on all this in the README files at:
https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark
Hope this answers your questions,
-Ciprian
…On Mon, Sep 10, 2018 at 11:38 AM bjascob ***@***.***> wrote:
While using the preprocessed data from http://www.statmt.org/lm-benchmark/
<http://url> I noticed that some of the training data was duplicated in
the heldout (aka test). This is in addition to
*train/news.en-00000-of-00100* which appears to be a complete copy of all
the heldout data.
Using a simple python script to put the sentences into a *dict*, I see
303,465 unique heldout sentences and 3,223 duplicates to sentences in the
training directory. Attached is a file bw_duplicates.txt
<https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/files/2367910/bw_duplicates.txt>
with the duplicates. You can easily verify this by grep'ing for them in the
training directory.
Is this a known issue? My concern is that many people use this data for
benchmarking language models and the test data has about 1% of the training
data mixed into it. That's probably not going to change the results much
but it isn't desirable either.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
<#5>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AK_u-2mmb3N5IkQVSLOqb1FwMxBxrNRRks5uZq_PgaJpZM4Wh8H7>
.
--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
bjascob
commented on May 26, 2024
Yes, that was my understanding.
I'm saying that sentences in the attached bw_duplicates.txt file show up in both,
training-monolingual.tokenized.shuffled/news.en-00000-of-00100 and
training-monolingual.tokenized.shuffled/news.en-000xx-of-00100 (where xx is 1 to 99)
For instance, the first duplicate sentence in the list "Bush is remembered by many Haitians -- " shows up in training/news.en-00000-of-00100 and training/news.en-00056-of-00100 exactly the same.
(Note that shards 59 and 75 also contain close to the same sentence but do differ by the "--" )
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
On Mon, Sep 10, 2018 at 3:21 PM bjascob ***@***.***> wrote:
Yes, that was my understanding.
Sorry, I thought your concern was that somehow the entire held-out set is
also part of the training set.
I'm saying that sentences in the attached bw_duplicates.txt file show up
in both,
training-monolingual.tokenized.shuffled/news.en-00000-of-00100 and
training-monolingual.tokenized.shuffled/news.en-000xx-of-00100 (where xx
is 1 to 99)
For instance, the first duplicate sentence in the list "Bush is remembered
by many Haitians -- " shows up in training/news.en-00000-of-00100 and
training/news.en-00056-of-00100 exactly the same.
(Note that shards 59 and 75 also contain close to the same sentence but do
differ by the "--" )
Well, as I mention at:
https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/scripts/split-input-data.perl#L17
after tokenization I ran:
# $ sort -u --parallel=10 news.20XX.en.shuffled.tokenized
# --output=news.20XX.en.shuffled.tokenized.sorted
# to get the input data.
#
# A sample command line for running this:
# ./scripts/split-input-data.perl
#
--output_file_base="$PWD/training-monolingual.tokenized.shuffled.perl/news.en"
# --num_shards=100
#
--input_file=./training-monolingual.tokenized/news.20XX.en.shuffled.tokenized.sorted
So the problem is in the Unix sort then?! Hard to believe. Perhaps what
starts as different UTF-8 character sequences at sort time gets later
normalized to the same sequence?! Not sure where the duplicates could come
from...
This was originally done in MapReduce internally, but that data could not
be released for legal concerns; I could only release code. As I explained
at 3 in README.corpus_generation
<https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/README.corpus_generation>,
we went through a few iterations in making sure the results I was getting
on my machine were the same as the ones that Tony got on his; I guess some
bugs survived. :)
How much of the test
set 1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050
overlaps with the training data?
Generally speaking, it was pointed out to me by users of this data that
some level of overlap between training and test is to be expected in
practice, especially at the short sentence end. So in that sense de-duping
the data is not ideal either... But we started from a far worse situation.
…--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
On Mon, Sep 10, 2018 at 4:14 PM Ciprian Chelba (personal account) <
[email protected]> wrote:
On Mon, Sep 10, 2018 at 3:21 PM bjascob ***@***.***> wrote:
> Yes, that was my understanding.
>
Sorry, I thought your concern was that somehow the entire held-out set is
also part of the training set.
> I'm saying that sentences in the attached bw_duplicates.txt file show up
> in both,
> training-monolingual.tokenized.shuffled/news.en-00000-of-00100 and
> training-monolingual.tokenized.shuffled/news.en-000xx-of-00100 (where xx
> is 1 to 99)
> For instance, the first duplicate sentence in the list "Bush is
> remembered by many Haitians -- " shows up in
> training/news.en-00000-of-00100 and training/news.en-00056-of-00100 exactly
> the same.
> (Note that shards 59 and 75 also contain close to the same sentence but
> do differ by the "--" )
>
Well, as I mention at:
https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/scripts/split-input-data.perl#L17
after tokenization I ran:
# $ sort -u --parallel=10 news.20XX.en.shuffled.tokenized
# --output=news.20XX.en.shuffled.tokenized.sorted
# to get the input data.
#
# A sample command line for running this:
# ./scripts/split-input-data.perl
#
--output_file_base="$PWD/training-monolingual.tokenized.shuffled.perl/news.en"
# --num_shards=100
#
--input_file=./training-monolingual.tokenized/news.20XX.en.shuffled.tokenized.sorted
So the problem is in the Unix sort then?! Hard to believe. Perhaps what
starts as different UTF-8 character sequences at sort time gets later
normalized to the same sequence?! Not sure where the duplicates could come
from...
Nevermind, here is probably the reason:
Reading through:
https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/scripts/get-data.sh#L40
the unique sort of the data was done before running punctuation
normalization and tokenization, so that explains the origin of the
duplicates.
… This was originally done in MapReduce internally, but that data could not
be released for legal concerns; I could only release code. As I explained
at 3 in README.corpus_generation
<https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/README.corpus_generation>,
we went through a few iterations in making sure the results I was getting
on my machine were the same as the ones that Tony got on his; I guess some
bugs survived. :)
How much of the test
set 1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050
overlaps with the training data?
Generally speaking, it was pointed out to me by users of this data that
some level of overlap between training and test is to be expected in
practice, especially at the short sentence end. So in that sense de-duping
the data is not ideal either... But we started from a far worse situation.
--
-Ciprian
--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
On Mon, Sep 10, 2018 at 4:21 PM Ciprian Chelba (personal account) <
[email protected]> wrote:
On Mon, Sep 10, 2018 at 4:14 PM Ciprian Chelba (personal account) <
***@***.***> wrote:
>
>
> On Mon, Sep 10, 2018 at 3:21 PM bjascob ***@***.***> wrote:
>
>> Yes, that was my understanding.
>>
> Sorry, I thought your concern was that somehow the entire held-out set is
> also part of the training set.
>
>> I'm saying that sentences in the attached bw_duplicates.txt file show up
>> in both,
>> training-monolingual.tokenized.shuffled/news.en-00000-of-00100 and
>> training-monolingual.tokenized.shuffled/news.en-000xx-of-00100 (where xx
>> is 1 to 99)
>> For instance, the first duplicate sentence in the list "Bush is
>> remembered by many Haitians -- " shows up in
>> training/news.en-00000-of-00100 and training/news.en-00056-of-00100 exactly
>> the same.
>> (Note that shards 59 and 75 also contain close to the same sentence but
>> do differ by the "--" )
>>
> Well, as I mention at:
>
> https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/scripts/split-input-data.perl#L17
> after tokenization I ran:
>
> # $ sort -u --parallel=10 news.20XX.en.shuffled.tokenized
> # --output=news.20XX.en.shuffled.tokenized.sorted
> # to get the input data.
> #
> # A sample command line for running this:
> # ./scripts/split-input-data.perl
> #
> --output_file_base="$PWD/training-monolingual.tokenized.shuffled.perl/news.en"
> # --num_shards=100
> #
> --input_file=./training-monolingual.tokenized/news.20XX.en.shuffled.tokenized.sorted
>
> So the problem is in the Unix sort then?! Hard to believe. Perhaps what
> starts as different UTF-8 character sequences at sort time gets later
> normalized to the same sequence?! Not sure where the duplicates could come
> from...
>
Nevermind, here is probably the reason:
Reading through:
https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/scripts/get-data.sh#L40
the unique sort of the data was done before running punctuation
normalization and tokenization, so that explains the origin of the
duplicates.
P.p.s. Looking at the history on get-data.sh, it was the last commit
<3780330?diff=split>
that moved the sorting of the data before the normalization/tokenization.
The description explains the decision:
Sort the date before doing the perl pre-processing.
This will make sure the training/held-out partitioning of the data is
the same irrespective of how Perl handles Unicode peculiarities in the
raw text.
So it seems that (other than typing "data" instead of "date") we could not
have done it better. I am now relieved. :-)
Thanks for pointing this out! It would be great to know what percentage of
the test set sentences are observed as such in the training data.
…-Ciprian
So it seems that
>
> This was originally done in MapReduce internally, but that data could not
> be released for legal concerns; I could only release code. As I explained
> at 3 in README.corpus_generation
> <https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/blob/master/README.corpus_generation>,
> we went through a few iterations in making sure the results I was getting
> on my machine were the same as the ones that Tony got on his; I guess some
> bugs survived. :)
>
> How much of the test
> set 1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050
> overlaps with the training data?
>
> Generally speaking, it was pointed out to me by users of this data that
> some level of overlap between training and test is to be expected in
> practice, especially at the short sentence end. So in that sense de-duping
> the data is not ideal either... But we started from a far worse situation.
>
> --
> -Ciprian
>
--
-Ciprian
--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
bjascob
commented on May 26, 2024
For all 50 holdout shards, I see 303,465 unique sentences and 3,223 duplicates to sentences in the training directory, so roughly 1%.
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
On Mon, Sep 10, 2018 at 4:48 PM bjascob ***@***.***> wrote:
For all 50 holdout shards, I see 303,465 unique sentences and 3,223
duplicates to sentences to those in the training directory, so roughly 1%.
Would it hard to get the exact number for
1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en-00000-of-00100
only?
I could do it too, but if you have the script handy and are willing to
re-run it...
…--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
On Mon, Sep 10, 2018 at 4:55 PM Ciprian Chelba (personal account) <
[email protected]> wrote:
On Mon, Sep 10, 2018 at 4:48 PM bjascob ***@***.***> wrote:
> For all 50 holdout shards, I see 303,465 unique sentences and 3,223
> duplicates to sentences to those in the training directory, so roughly 1%.
>
Would it hard to get the exact number for
1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en-00000-of-00100
only?
Wrong copy/paste, sorry:
I meant:
1-billion-word-language-modeling-benchmark/heldout-monolingual.tokenized.shuffled/news.en.heldout-00000-of-00050
…
I could do it too, but if you have the script handy and are willing to
re-run it...
--
-Ciprian
--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
bjascob
commented on May 26, 2024
For news.en.heldout-00000-of-00050 there were 6,005 unique sentences and 70 that were duplicates of training data. The duplicates sentences are listed in the following file: bw_dup_shard0.txt.
from 1-billion-word-language-modeling-benchmark.
ciprian-chelba
commented on May 26, 2024
Thanks!
…On Mon, Sep 10, 2018 at 5:14 PM bjascob ***@***.***> wrote:
For news.en.heldout-00000-of-00050 there were 6,005 unique sentences and
70 that were duplicates of training data. The duplicates sentences are
listed in the following file: bw_dup_shard0.txt
<https://github.com/ciprian-chelba/1-billion-word-language-modeling-benchmark/files/2368867/bw_dup_shard0.txt>
.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<#5 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AK_u-4kX1z0wYO8mfSFqCp8ALe5Ytsqgks5uZwBTgaJpZM4Wh8H7>
.
--
-Ciprian
from 1-billion-word-language-modeling-benchmark.
Related Issues (7)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from 1-billion-word-language-modeling-benchmark.