We might want to exclude certain dates from our datasets. Something like:

date_range = Date(2018, 1, 1):Day(1):Date(2020, 1, 1)

# Normal date selection
selector = RandomSelector(42, 0.10, Day(3))
validation, holdout = partition(date_range, selector)

# But we don't want these dates
bad_dates =[Date(2019, 1, 1), Date(2019, 12, 1)]

# Do selection as usual, just don't return any of the bad dates
selector_exc = DateExclusionSelector(selector, bad_dates)
validation_exc, holdout_exc = partition(date_range, selector_exc)

# The following should hold:
@assert validation_exc == setdiff(validation, bad_dates)
@assert holdout_exc == setdiff(holdout, bad_dates)

There is a long standarding TODO in the code to remove do this,
now that invenia/Intervals.jl#141 is done we can.

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!

from @glennmoy

The idea behind randomly/periodically selecting weekly blocks is that it allows use to adequately sample a 2 year period with some statistical guarantees about the proportional representation of weekdays/weekends and seasons within the validation and holdout sets.

The implication is that this provides (albeit somewhat weaker) guarantees about the distribution of the underlying grid state, seasonality effects, and our performance over the period.

In the ensembling squad this assumption was undermined by one Problem as our model performance varied a lot between years and one year happened to be sampled more than the other.

The current RandomSelector is therefore not robust enough to provide any guarantees about the statistics of our returns which are necessary to provide a reliable baseline against which we can compare optimised models.

This issue is more to document the concern and some possible avenues for taking this forward with different, and more sophisticated, selectors in future. Namely:

• Systematic Sampling
• Stratified Sampling
• Cluster Sampling
• Multistage Sampling

As a simple remedy to the above, we might have instead done something like

• Cluster the dates by season
• Within each cluster, sort the dates by their some difficulty measure
• Systematically select dates (e.g every second date or in blocks of 7) to ensure (roughly) proportional statistics

This would retain the same seasonal guarantees as before, somewhat weakened the weekday/weekend guarantee, but at the benefit of more similar return statistics. This is just a simple example, perhaps there's an easier/better way to doing it.

Moreoever, if we ever wish to discriminate by other criteria, e.g. grid regimes, the example gets more complicated but the same principle applies.

The example link in this package's README.md redirected to 404.

Is it supposed to link to the DateSelectors.jl/docs/src/examples.md file ? If so, I could try to submit a PR

Best

It would be useful to be able to select the seed for (e.g.) the RandomSelector based on user-specified criteria. This would allow an implementation of more sophisticated sampling methods (cf. #7 ) via rejection sampling, with a compact representation (the seed) of the final date set. Something like the following:

function acceptance(validation_dates, holdout_dates)
    # We want 1st of January to be in the validation dates
    return DateTime(2021, 1, 1) in validation_dates
end

# Gives a seed with the desired properties to use with RandomSelector
seed = rejection_sample(RandomSelector, date_range, acceptance)

It currently has 3 positional args,. it is hard to remember which is which
PeriodicSelector(Week(1), Day(2), Day(1))

An idea @glennmoy and I were discussing

@glennmoy

> could we not generate a random bit string once, to represent whether a block will go in the validation/holdout?
> 
> This is just pseduo code but the idea would be e.g. 
> - take a large enough date range to capture any reasonable dates we would consider: date_range = `Date(1900, 1, 1):block_size:Date(2100, 1, 1)`
> - Generate a bit string of `length(date_range)` once with equal number of 0s/1s 
> - Mask the `date_range` using the bit string in global_validation/global_holdout
> - Intersect with the dates we provide to get the validation/holdout we care about
> 
> * only uses one random number 
> * no loops
> * exact fraction gets allocated (modulo block size)

@oxinabox replied

If we could actually fit it into some thing like a UInt1024 (which would from https://github.com/rfourquet/BitIntegers.jl)
that would cover us for 1024 blocks. which if doing block size of 1 day is not really enough.

But it would be nice because one could generate the bitstring in seperately to constructing the DateSelector, and pass it into the DateSelector as instead of the seed.
And that would be safe against the random number selector changing.
Such a 1024 bit string would look like 0x328d_8a1e_86f8_5c55_3cfe_223a_5904_02fd

Complicated bit would be that to achieve invarience to start date we probably need to do 200 years of days.
So would need to be a Int65536 which would not work great with BitIntegers.
But we could make it work.
Probably would want to Base64 encode it.

Disclaimer: I am not familiar with the design of this package so I might be missing something obvious.

That said, why are the inputs to partition restricted to Intervals and ranges of dates? IIUC allowing a vector of dates would solve #14 and remove the need for #17 since we could just filter the input dates.