Right now, we have files in the cachedmetrics folder in the form:

{dataset}_cachemedtrics.txt
{dataset}_TrainTestMetrics.txt

It would be highly beneficial to simply have one single file for each dataset, thus representing the actual architecture performance with one single line (in which one would have, in this order):

Arch_Idx, NASWOT, logSynflow, PortionSkipped, Test-Accuracy, TrainingTime(xEpoch), TrainingTime(total)

Ideally, this file would still be a .txt file one could import to memory via np.loadtxt(...).
Having one file only would make it easier to access the information.

@darin65 will present us a initial reading of Diagonality Measures of Hermitian Positive-Definite Matrices with Application to the Approximate Joint Diagonalization Problem, a paper investigating measures of diagonality (alternative to logdet) for matrices such as the one defined in NASWOT. With this, we aim at finding an alternative measure of diagonality to logdet which would correlate more with test accuracy.

Rush: 10/03

1. Add support to ImageNet16-120 (using pickled subsets of the dataset instead of the whole one).
2. Add NAS-Bench 101 to considered search space.
3. Figure out why we do not reach SOTA in NATS-Bench (check number of generations and implementation details on normalization coefficients for scores).

Read:

1. A Comprehensive Survey of Neural Architecture Search: Challenges and Solutions
2. HW-NAS-Bench:Hardware-Aware Neural Architecture Search Benchmark

Write down reviews and possibly sketch out a set of libraries that can help accessing hardware-related measures.

1. Re-run experiments logging on wandb to figure out what normalization style is best suited to tackle the problem at hand (once and for all). Styles shall be compared on mean value but also relatively to the variance of the obtained solution.
2. Fix standard normalization style adjusting prior knowledge on mean and std as new architectures are discovered.