This module should be able to plug in to mkrefs and will be used to create gain reference files.

Need to get persistence maps from Jarron to go into the initial delivery of persistence map reference files.

Also need to turn hard saturation maps into persistence saturation reference files.

The wrapper script that does the high-level work

Need to determine if darks should be mean of observed darks, or zeros with only hot pixels populated.

Appropriate reference files must be created for full frames and subarrays.

I was looking at the jwst_pipeline_walkthrough.ipynb notebook (which is great!) and in section 11, you read in the source catalog using astropy.io.ascii.read:

# below is an example source catalog product
from astropy.io import ascii
catalog = ascii.read("mosaic_A1-A4_obs1-8_cat.ecsv")

I just wanted to point out that the preferred way to read in an ECSV file is to use astropy.table.Table (or astropy.table.QTable). That will preserve the metadata stored in the table, e.g. units on column values (e.g. flux) and you will also get back any serialized astropy objects stored in the table (e.g. Time objects or SkyCoord objects (instead of separate columns for RA and Dec):

from astropy.tables import Table
catalog = Table.read('mosaic_A1-A4_obs1-8_cat.ecsv')

CC: @hcferguson

For @aliciacanipe
e.g. sloper pipeline is now calwebb_detector1.

Also would be helpful to show how to override individual step values when calling a pipeline. Here's an example I put together the other day after talking with Howard and James.

from jwst.pipeline import calwebb_detector1
m = calwebb_detector1.Detector1Pipeline(config_file='calwebb_detector1.cfg')
m.saturation.override_saturation = 'mysatfile.fits'
m.superbias.override_superbias = 'mysuperbias.fits'
m.refpix.odd_even_rows = False
m.group_scale.skip = True
m.ipc.skip = True
m.dark_current.skip = True
m.persistence.skip = True
m.output_file = outfile
m.run('myrawfile_uncal.fits')

I wrote this for WSS but it really belongs to NIRCam. Are you interested?

https://github.com/STScI-JWST/wss_tools/blob/master/wss_tools/utils/mosaic.py

I need to add the pipeline testing scripts to this repository.

James Davies tells me that the only way to use config files (or more accurately, the only way that the pipeline pays attention to config files) is when you call the pipeline from the command line (strun) or when you use the call method. The run method just looks for default values and values set by the user, but ignores cfg files.

I've only looked at the level 3 pipeline notebook, but in there the config files are used with run, which we saw at a recent hack day didn't work well.

We should check the existing notebooks and make updates anywhere they are needed.

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

Some future enhancements for the GSEG Data Validation Notebook:

• Verify header APERNAME based on APT Subarray (currently APERNAME is only checked to be consistent with DETECTOR, but not with the APT file, so this has to be checked manually)
• Modify adjust_exptype() to work with more than EXP_TYPE=NRC_IMAGE and NRC_GRISM (e.g. NRC_TSIMAGE, NRC_TSGRISM)
• Include checks for dark.fits file for files with EXP_TYPE=NRC_DARK, rather than assuming all uncal files have corresponding rate files
• Fix EFFEXPTM estimate calculation, which is giving wrong answers for some files
• Remove assumption that all detectors will be used for a given observation (currently missing files have to be checked for manually because of this)
• Remove dependence on MAST Portal (astroquery, read access to MAST directories)?
• Clean up unnecessary messages in output (i.e. only show warnings and mismatches), and save output to a text file
• Check TFRAME values to make sure they are computed for the correct number of outputs (see JIRA ticket JWSTDMS-19)
• enhancement (nice to have): check for more than just *rate.fits files (e.g., exposures with NINTS>1 will also have a *rateints.fits file, data products are here: https://jwst-pipeline.readthedocs.io/en/latest/jwst/data_products/product_types.html)
• Check that any associations (ASNTABLE) exist and that its members are assigned the correct designation

I don't see any way to install this package. This issue is just a friendly reminder that it needs to be done.

These scripts were tweaked to work for OTIS data, but they need to be tested for reference files and could use some cleaning up.

Create the code that will take a list of file base names and required SSB calibration pipeline steps, and generate strun commands to convert the input base files to the proper data reduction state.

Some of the older functions and modules need to be updated for python 3. mkref.py is one that I came across. I'm sure there are others.

Need to decide which gain values we are going to use, U of A's or ours? The values are different by ~20%.

The chosen gain values need to be packaged up as reference files and delivered. Also, there are several dependencies with their own deadlines.

Gain values will impact full well values in electrons, which are present in JDox.
Gain values are used in the production of the flux calibration reference files.
The flux calibration reference files include filter-dependent zeropoints, which are present on JDox.

Deadline for all reference files is the end of January.
JDox deadline is ?

Karl still has not delivered the F070W flat (and maybe also the F090W) flat. He was having trouble removing all of the illumination pattern effects and wanted to keep working on it.

Also, a check of the other flat field reference files in CRDS show that the error and DQ extensions that Karl provided have been zeroed out. Most likely this happened when converting Karl's files to SSB format and the extensions were never checked. These files will have to be remade at some point.

Deadline for everything is 31 January 2018.