It will be useful to uniquely identify each person providing labels. For example, this would allow conflicting labels for people with different levels of expertise to be optimally combined.

This issue is to implement a simple one-time sign in process, with the user name cached for subsequent visits to the page.

Spinning off from #9: thumbnail images are currently written using the default jpeg quality of 95, which seems un-necessarily high, and pushes the client memory usage to ~800Mb.

This issue is to replace plt.imsave in prepare/extract.py with:

fig = Figure(dpi=dpi, frameon=False)
FigureCanvas(fig)
fig.figimage(arr, cmap=cmap, vmin=vmin, vmax=vmax, origin=origin, resize=True)
fig.savefig(fname, dpi=dpi, format=format, transparent=True, quality=80)

where the new quality has been tuned for a reasonable size / appearance tradeoff.

The prototype uses histogram equalization to scale each image individually (code here).

This issue is to investigate alternative schemes that use the same scaling for all images of the same band.

The prototype uses all 121,123 exposures listed in:

/global/project/projectdirs/cosmo/work/legacysurvey/dr8b/image-lists/image-list-decam-dr8.txt

This issue is to use a different list that has known bad exposures filtered out, so the expert labeling can focus on identifying bad exposures that are not already being automatically filtered by the existing quality cuts.

@schlafly Where is the final list of DECaLs exposures that we should run this tool on?

Clicking a label on the prototype now just displays a message to the javascript console.

This issue is to POST the label to a backend server, e.g. a google form.

The purpose of this issue is to discuss the main classes of bad exposure and select the labels to use in visual inspection and for machine learning classification.

The ideal number of classes (not including "good") is 4, since that is easy to implement (one per thumbnail corner) and still few enough to allow quick manual classification of ~6K images.

A suggestion from Eddie on [decam-chatter]:

It would be great to be able to conveniently get higher resolution versions. The viewer makes individual chips available as follows http://legacysurvey.org/viewer/ccd/decals-dr7/decam-723820-S22-r

I am trying to process ~120K compressed FITS files (.fits.fz) from the DESI legacy imaging survey, e.g.

/global/project/projectdirs/cosmo/work/legacysurvey/dr8b/images/decam/DECam_CP/CP20140810_g_v2/c4d_140815_235550_ooi_g_v2.fits.fz

Each compressed input file is ~300Mb.

I have tested a python script (extract.py in this repo) that processes 10 files in < 10 mins, and estimate its RSS memory requirement is ~4Gb (using ps -p PID -o time,rss,vsz).

I have also tested a slurm script (extract.slurm in this repo) that works with a stub python script.

However, when I run with the real python script, jobs are killed with:

Starting slurm script at Fri Mar 29 10:28:57 PDT 2019
slurmstepd: error: Detected 1 oom-kill event(s) in step 12810111.0 cgroup. Some of your processes may have been killed by the cgroup out-of-memory handler.
srun: error: nid06081: tasks 0-23: Out Of Memory
srun: Terminating job step 12810111.0

real	0m35.528s
user	0m0.040s
sys	0m0.032s
done with slurm script at Fri Mar 29 10:29:33 PDT 2019

Is there a way to increase the default memory available per process?

Any suggestions for decreasing the memory footprint of the script? I am already using memmap=True to open the FITS file and only keep 2 HDUs in memory at once (~2 x 32Mb), so I don't understand why the memory usage is so much bigger than the whole (compressed) input file.

The prototype currently loads 100 images for each band.

This issue is to implement progressive loading of ~2K images per band, so that the help tab can be read immediately and images can be labeled as soon as they have loaded.

@tskisner Could you have a look over this slurm script that I used on edison and give me some pointers for migrating to cori?