Tiny example dataset about monitor HOT 19 OPEN

I think this issue is the roadblock for Monitor development (which needs to happen fast if we are to be able to make plots of @TomGlanzman and @tony-johnson 's Twinkles outputs before March 7. Does anyone have a `ForcedSource catalog (or catalogs) that can be stripped down to 2 objects in 3 filters observed over N epochs for development purposes? @SimonKrughoff? @jbkalmbach?

from monitor.

I agree we need this very quickly, for understanding how to build the light curve class. I was going through @SimonKrughoff's plot_mags code as suggested by @jchiang87 and it would help to be able to see the data.

from monitor.

@rbiswas4 Do you have a slac account? If so, you can try using the following output_data directory with Simon's code:

bash-4.1$ pwd -P
/nfs/slac/g/ki/ki18/jchiang/DESC/Twinkles/work/output_data
bash-4.1$ ls
_parent@                  forced/
calexp/                   forcedPhotCcd_metadata/
config/                   icMatch/
deepCoadd/                icSrc/
deepCoadd-results/            processEimage_metadata/
deep_makeCoaddTempExp_metadata/       schema/
deep_safeClipAssembleCoadd_metadata/  src/
bash-4.1$ 

Unfortunately, to use that code with the butler etc., the full output directory structure is probably needed, i.e., some sample forced source catalogs won't be sufficient.

from monitor.

@jchiang87

@rbiswas4 Do you have a slac account?
This is in the works, but not yet. Your reply just cleared up a bunch of things I was confused about and was composing questions about: things like what the strings like 'output_data' were doing in that script.

Since I do not have an account yet, could you please upload that example dataset to nersc? I can download it from there to start looking at it. We can think about stripping it down to size for an example_data set after that. Thanks!

from monitor.

Ok. I can try. I have a NERSC account, but I haven't tried using it yet.

from monitor.

Thank you!

On Sat, Feb 27, 2016 at 6:26 PM, James Chiang [email protected]
wrote:

Ok. I can try. I have a NERSC account, but I haven't tried using it yet.

—
Reply to this email directly or view it on GitHub
#4 (comment)
.

from monitor.

I wasn't able to access my NERSC account, so I uploaded the data to lsst-dev.ncsa.illinois.edu instead. The tarball, output_data.tgz, should be read-accessible:

[jchiang@lsst-dev Twinkles]$ pwd -P
/nfs/home/jchiang/Twinkles
[jchiang@lsst-dev Twinkles]$ ls
output_data  output_data.tgz
[jchiang@lsst-dev Twinkles]$ ls output_data/
calexp                               forcedPhotCcd_metadata
config                               icMatch
deepCoadd                            icSrc
deepCoadd-results                    _parent
deep_makeCoaddTempExp_metadata       processEimage_metadata
deep_safeClipAssembleCoadd_metadata  schema
forced                               src
[jchiang@lsst-dev Twinkles]$ ls -l output_data.tgz 
-rw-rw-r-- 1 jchiang jchiang 1260134730 Feb 27 22:28 output_data.tgz
[jchiang@lsst-dev Twinkles]$ 

HTH.

from monitor.

I do have an account on lsstdev, but I cannot access this directory because your home directory blocks permissions. But thanks for putting this up on lsstdev!

[rbiswas@lsst-dev ~]$ cd /nfs/home/jchiang/Twinkles
-bash: cd: /nfs/home/jchiang/Twinkles: Permission denied
rbiswas@lsst-dev home]$ cp /nfs/home/jchiang/Twinkles/output_data.tgz .
cp: cannot stat `/nfs/home/jchiang/Twinkles/output_data.tgz': Permission denied
[rbiswas@lsst-dev ~]$ cd /home/
[rbiswas@lsst-dev home]$ ls -ltr|grep jchiang
drwx------  11 jchiang       jchiang        22 Feb 27 22:07 jchiang

from monitor.

Hi @rbiswas4, Can you try again? I just changed the directory permissions. I think you should be able to do cp /nfs/home/jchiang/Twinkles/output_data.tgz <some place local>

from monitor.

OK. I tried it now and still have that problem:

rbiswas@lsst-dev ~]$ cp /nfs/home/jchiang/Twinkles/output_data.tgz .
cp: cannot stat `/nfs/home/jchiang/Twinkles/output_data.tgz': Permission denied

from monitor.

Sorry for my silence on this. Is there anything I can do to help? It sounds like @rbiswas4 is looking into it.

from monitor.

@SimonKrughoff Can you provide @rbiswas4 with an output_data collection? You probably have the gri run handy. The one I put at lsst-dev is just the older r band analysis.

from monitor.

O.K. I'll put this together tomorrow.

from monitor.

Ok, I've got Simon's data, but it's quite large so I'm working on making the smallest, most basic but useful dataset out of it that I can.

from monitor.

So, the output_data folder provided by @SimonKrughoff only had visits from the same MJD and the same band. @jchiang87 are yours any different? If not, would it possible for anybody to run dm processing on the new sprinkled outputs to help create a better test set? Or at the very least have any output_data folders lying around that span a few days/bands?

from monitor.

Mine is just r-band for 9 visits, They might span more than one night but they won't be significantly different from Simon's, if at all. @tony-johnson should have the workflow engine going soon on the multiband/multiday data that are being generated with phosim. I can look into running the Twinkles cookbook on those data tomorrow to get some files for you sooner.

from monitor.

Yeah, anything from the actual twinkles data would be great if we can that!

from monitor.

Ok, small test set of forced photometry output available in 'data/test/forced'. ObjectIds run from 0-375. Now, by using one of those numbers in build_lightcurve(objectId), anybody can run the example notebook on their own.

from monitor.

I just realized we still need to get the true light curves associated with this test data. So, I reopened this issue until that gets done.

from monitor.