Create a class or function that will take an .asn file and return a table object / list of datasets / other useful info in a single function call.

Remove Median background per read from the IMA step. The algorithm is describe in the notebook here:
https://github.com/mackjenn/wfc3/blob/Nonlinear_bkg/nonlinear_bkg.ipynb

This would correct for Helium I emission (flat pedestal of extra signal) by subtract median per read and rerun calwf3 with ramp-fitting turned on. 

Requirements for completion:

• Documentation
• Functionality
• Testing

Following the methodology described here, the WFC3 team has written a function to remove time-variable background from IR images and improve the ramp fitting.
https://github.com/shannnnnyyy/wfc3tools/blob/ir_tools/wfc3tools/wfc3ir_tools.py

The procedure will partially process a RAW exposure through calwf3 (with CRCORR=OMIT), remove the median signal per read from the IMA, and then resume calwf3 on the corrected IMA file, this time with CRCORR=PERFORM, in order to produce an improved FLT image.

The problem is that calwf3 fails the last step when the file is not part of an association. (eg. ASN_ID=NONE). This appears to be a bug in calwf3, which hopefully could be a simple fix?? We need this in order to run our pipeline to generate sky flats using exposures impacted by non-linear background. Tagging individuals involved in this effort @shannnnnyyy @heatherkurtz

Create a tool in the wfc3tools package that users can examine their data with - basically do a header keyword check on the files to alert users to problems that might have occured with the data.
For example: Now that TDFTRANS is ignored, check this key in the imsets for IR data and alert
Get a list of other important keys from the group, consider letting users supply their own list.

When making final, public releases of this software please use the Github release feature (https://github.com/spacetelescope/wfc3tools/releases/new). This will trigger the build and upload of the corresponding package at the tag specified to PyPI.

@sosey Myself and others (@cmartlin, @mrafelski, @ivastar) are running into an issue where wfc3tools.util.py crashes when attempting to return an error code, e.g.:

  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 212, in wrapped
    func(*a, **kw)
  File "/grp/hst/wfc3h/bourque/repository/pyql/pyql/logging/logging_functions.py", line 175, in wrapped
    func(*a, **kw)
  File "cal_uvis_make_darks.py", line 276, in process_darks_main
    preprocess_data_main(paths, postflash)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/preprocess_data.py", line 87, in preprocess_data_main
    make_blvs_main(directory)
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 175, in make_blvs_main
    run_calwf3()
  File "/grp/hst/wfc3h/bourque/repository/automated_scripts/cal_uvis_make_darks/make_blvs.py", line 126, in run_calwf3
    calwf3(input='dark_asn.fits', save_tmp=True)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/calwf3.py", line 77, in calwf3
    ec = error_code(return_code)
  File "/grp/hst/wfc3p/bourque/envs/anaconda3/envs/astroconda3/lib/python3.5/site-packages/wfc3tools/util.py", line 64, in error_code
    return codes[code]
KeyError: -7

Digging into the valid error codes in wfc3tools.util.py, it does't look like -7 is a choice:

def error_code(code=None):
    """ return the error code text, or all if code is None."""

    codes = {2: "ERROR_RETURN",
             111: "OUT_OF_MEMORY",
             114: "OPEN_FAILED",
             115: "CAL_FILE_MISSING",
             116: "NOTHIN_TO_DO",
             117: "KEYWORD_MISSING",
             118: "ALLOCATION_PROBLEM",
             119: "HEADER_PROBLEM",
             120: "SIZE_MISMATCH",
             130: "CAL_STEP_NOT_DONE",
             141: "TABLE_ERROR",
             142: "COLUMN_NOT_FOUND",
             143: "ELEMENT_NOT_FOUND",
             144: "ROW_NOT_FOUND",
             151: "NO_GOOD_DATA",
             152: "NO_CHIP_FOUND",
             171: "REF_TOO_SMALL",
             999: "INTERNAL_ERROR",
             1001: "INVALID_EXPTIME",
             1011: "INVALID_FILENAME",
             1020: "WRITE_FAILED",
             1021: "INVALID_TEMP_FILE",
             1023: "FILE_NOT_READABLE",
             1025: "COPY_NOT_POSSIBLE",
             1111: "INVALID_VALUE",
             1030: "UNSUPPORTED_APERTURE",
             }

    if code is None:
        return codes
    else:
        return codes[code]

Any idea what might be going on here?

This prompted me to dig into why my call to calwf3 might be triggering an error in the first place. Upon digging, I noticed that though I explicitly set the input variable to point to an association table (which contains the list of files I want to process) e.g ...

from wfc3tools.calwf3 import calwf3
calwf3(input='dark_asn.fits', save_tmp=True)

... calwf3 appears to be gathering any FITS files in the CWD and setting them as the infiles, e.g.:

        infiles, dummy_out = parseinput.parseinput(input)
        logging.info(infiles)
        for image in infiles:
            if not os.path.exists(image):
                raise IOError('Input file not found: ' + image)

        call_list.append(','.join(infiles))

['id0k2wkrq_raw.fits', 'id0k2vi6q_raw.fits', 'id0k2wklq_raw.fits', 'id0k2yn7q_raw.fits', 'id0k2xmiq_raw.fits', 'id0k2vi0q_raw.fits', 'id0k2uclq_raw.fits', 'id0k2zsmq_raw.fits', 'id0k2yn3q_raw.fits', 'id0k2zskq_raw.fits', 'id0k3aupq_raw.fits', 'id0k2xmhq_raw.fits', 'id0k3auoq_raw.fits', 'id0k3fujq_raw.fits', 'id0k3fuiq_raw.fits', 'id0k3ifnq_raw.fits', 'id0k3hcxq_raw.fits', 'id0k3hcwq_raw.fits', 'id0k3ga5q_raw.fits', 'id0k3jfsq_raw.fits', 'id0k3ga1q_raw.fits', 'id0k3jfqq_raw.fits', 'id0k3lq6q_raw.fits', 'id0k3lq5q_raw.fits', 'id0k3qhyq_raw.fits', 'id0k3qhxq_raw.fits', 'id0k3pfhq_raw.fits', 'id0k3tbkq_raw.fits', 'id0k3tbgq_raw.fits', 'id0k3obsq_raw.fits', 'id0k3obrq_raw.fits', 'id0k4areq_raw.fits', 'id0k4ardq_raw.fits', 'id0k4dc1q_raw.fits', 'id0k4ec9q_raw.fits', 'id0k4ec5q_raw.fits', 'id0k4jknq_raw.fits', 'id0k4jkiq_raw.fits', 'id0k4ikbq_raw.fits', 'id0k4ijpq_raw.fits']

When I bypass this code and explicitly set call_list, I can avoid an error and calwf3 works fine:

call_list = ['calwf3.e', 'dark_asn.fits', '-s']
proc = subprocess.Popen(
    call_list,
    stderr=subprocess.STDOUT,
    stdout=subprocess.PIPE,
)

Could you advise?

On ReadTheDocs I see this warning for wfc3tools:

Your project(s) "stistarg, wfc3tools, stsci-env" don't have a configuration
 file. Configuration files will soon be required by projects, and will no 
longer be optional. 
[Read our blog post to create one](https://blog.readthedocs.com/migrate-configuration-v2/) 
and ensure your project continues building successfully.

Tyler Desjardins mentions that we should consider moving emails from help[at]stsci.edu to point to the web portal where possible and appropriate. For HST (or any non-JWST), it is https://hsthelp.stsci.edu . For JWST, it is https://jwsthelp.stsci.edu . Please update info in setup.py, setup.cfg, documentation, etc as appropriate.

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

xref spacetelescope/hstcal#317

Create a WFC3 Photometry Tools subpackage within WFC3Tools. Tools for UVIS/IR photometry calculation and analysis.

The setting of the calibration step switches to 'OMIT' or 'PERFORM' seem to be ignored. Instead, the *raw.fits file is processed according to the setting of the calibration switch step in the primary header of the file. This error needs to be investigated. It may be the actual problem is in the underlying C code.

The parameter Version in calwf3.py was removed in PR #64 but the docstring still lists it as a valid parameter. We need to remove it from the docstring and instead show an example of how to check the calwf3 version (i.e. !calwf3.e -r or !calwf3.e --version)

Some of the wf3tools (Python) present a different interface than their C counterparts. The Python wrappers which exhibit this behavior should be updated to reflect the underlying C interface.
For example, wf3rej.py has explicit checking for input lists and raises an exception if a list is passed in as this shows here.

if len(parseinput.irafglob(input)) > 1:
        raise IOError("wf3rej can only accept 1 file for"
"input at a time: {0}".format(infiles))

The documentation implies input lists are allowed.

input : str, Name of input files

      - a single filename (``iaa012wdq_raw.fits``)
      - a Python list of filenames                            <== HERE
      - a partial filename with wildcards (``\*raw.fits``)
      - filename of an ASN table (``\*asn.fits``)
      - an at-file (``@input``)

This can be fixed by removing the check and exception. The underlying C executable, wf3rej.e, can handle input lists, but it requires the input to be a CSV string. This needs to be fixed in the Python wrapper (e.g., here in wf3rej.py)

call_list.append(input)

should be

input_string_list = ','.join(input)
call_list.append(input_string_list)

This was introduced in #13.

Code Changes

List is also evident in other tools and should be addressed.

• wf3rej.py TESTING https://github.com/spacetelescope/wfc3tools/compare/master...mdlpstsci:issue22_inputlist?expand=1
• calwf3.py
• TBD

Doc Changes

• calwf3
• wf3ref
• TBD

It would be useful for other functions that call calwf3 (or that run it on organized data) to be able to input file paths, rather than needing to change directories in a script because it only allows file names.

In pstack, the input column and row parameters appear to be put in to the array indexing incorrectly. Since numpy's convention is row major order, this should be flipped. @mackjenn tested this and finds that pstack selects the wrong pixel (the pixel at the flipped coordinates) compared to doing a similar process with IRAF.

For working examples, see hstcal, acstools, stistools, or calcos. When Jenkins is working here, you can retire Travis.

Add real tests first. Preferably using Github Action.

http://www.stsci.edu/hst/wfc3/analysis/grism_obs/wfc3-grism-faq.html#4.3_Answer is "distributing" a script (http://www.stsci.edu/hst/wfc3/analysis/grism_obs/embed_subs.py) that still uses deprecated pyfits. It should be officially distributed as part of wfc3tools (and also modernized).

Allow the user to select lower/upper limits of data (to exclude sources and sample the sky background)

The current on-line documentation for calwf3, as well as its components which can be run in standalone mode, is incorrect and out-of-date. The documentation and any associated diagrams need to be updated to reflect the true state of the pipeline software. This issue is related to Git PR#61. This issue is scheduled as JIRA HLA-759.

I dont think the documentation has been updated in a while, it would be good to read through it and make sure it is consistent with the current functionality of the package.

Allow the user to overplot 'signal' vs 'time' for different subsections of the detector (eg. LHS vs RHS allows the user to diagnose reads impacted by bright earth limb, see notebook below)

Python 2 will not be maintained past Jan 1, 2020 (see https://pythonclock.org/). Please remove all Python 2 compatibility and move this package to Python 3 only.

For conda recipe (including astroconda-contrib), please include the following to prevent packaging it for Python 2 (https://conda.io/docs/user-guide/tasks/build-packages/define-metadata.html?preprocessing-selectors#skipping-builds):

build: 
  skip: true  # [py2k] 

Please close this issue if it is irrelevant to your repository. This is an automated issue. If this is opened in error, please let pllim know!

(Related to trac #985) Create a tool which will allows users to process IR ramps while leaving out specified reads. Previously users could set TDFTRANS so that calwf3 ignored the reads, but now calwf3 ignores that key (trac #980)

When verbose is set to True, it sets the '-v' flag rather than setting the '-q' flag when verbose=False

Enable PR build for RTD. Need RTD admin to do this.

Should we remove the teal support at some point in the future?

I hope this is the correct repository where my issue should be reported.

I have noticed in some WFC3 images that PHTRATION has 0 as value. This does not seem to be correct:

SCI,1 = CHIP2:

PHOTFLAM= 1.9811659E-19 / inverse sensitivity, ergs/cm2/Ang/electron
PHTFLAM1= 1.981165900000E-19 / PHOTFLAM on chip 1 when scales are not uniform
PHTFLAM2= 2.0070588E-19 / PHOTFLAM on chip 2 when scales are not uniform
PHTRATIO= 1.013069526383E+00 / Ratio of PHTFLAM1/PHTFLAM2

SCI,2 = CHIP1:

PHOTFLAM= 1.9811659E-19 / inverse sensitivity, ergs/cm2/Ang/electron
PHTFLAM1= 1.9811659E-19 / PHOTFLAM on chip 1 when scales are not uniform
PHTFLAM2= 2.0070588E-19 / PHOTFLAM on chip 2 when scales are not uniform

PHTRATIO=0.0 / Ratio of PHTFLAM1/PHTFLAM2

Obviously PHTRATIO cannot be 0 (unless PHTFLAM2=inf).

The WFC3 team is working on an early 2020 delivery which would require calwf3 to interpret time-dependent photometry keywords from the IMPHTTAB photometry reference file and write them to the image headers.

ACS already does this via their IMPHTTAB, which has different columns for various epochs. These define the endpoints with which calacs performs a linear interpolation of the sensitivity based on observation date .

For UVIS, the PHOTMODE values in the image header apply to all epochs: PHOTMODE= ‘WFC3 UVIS1 F475W' and with the new software would need to look something more like the ACS syntax: PHOTMODE= 'ACS WFC1 F475W MJD#54113.9559'

Does calwf3 already have the ability to handle time-dependent sensitivity (eg. is it just commented out), or do you think this is code that will need to be developed?
If so, does it only support a linear interpolation, or could one apply a higher order fit?

One difference from ACS is that our IMPHTTAB contains chip-specific sensitivity values, so we have more extensions than usual. For example, below compares ACS/WFC and WFC3/UVIS:

EXT# FITSNAME FILENAME EXTVE DIMENS BITPI OBJECT
0 2391649pj_imp 2391649pj_imp.fits 8
1 BINTABLE PHOTFLAM 1 13Fx780R
2 BINTABLE PHOTPLAM 1 13Fx780R
3 BINTABLE PHOTBW 1 13Fx780R

EXT# FITSNAME FILENAME EXTVE DIMENS BITPI OBJECT
0 0bi2206ti_imp 0bi2206ti_imp.fits 16
1 BINTABLE PHOTFLAM 1 5Fx256R
2 BINTABLE PHOTPLAM 1 5Fx256R
3 BINTABLE PHOTBW 1 5Fx256R
4 BINTABLE PHTFLAM1 1 5Fx256R
5 BINTABLE PHTFLAM2 1 5Fx256R

The timing of this will be important, as we would like to deliver the solutions in early 2020 and we would need to be able to do initial testing in late-2019.

Allow the user to provide a mask as input (to exclude sources prior to computing the median)

I was running calwf3 on a set of raw files called by an association table file (dark_asn.fits). calwf3 did not run which I later found was due to the fact that the CRREJTAB key word in the raw file header was a directory to the calibration file crr_for_dark.fits that was too long and was truncated. It would be really helpful that if calwf3 needs a calibration file but the path is invalid, to show this as a warning. It showed no warning in the python version calwf3 or in the calwf3.e executable from the command line

from wfc3tools import pstat
pstat('/grp/hst/wfc3a/GO_Links/11597/Visit14/ib5l14x4q_ima.fits',extname="sci",units="counts",stat="midpt",plot=True)

This used to produce a plot, but currently only prints the arrays to the screen.
(array([ 102.934334, 77.933823, 52.933311, 27.932802, 2.932291,
0. ]), array([ 52.19914665, 39.69794801, 26.85327161, 14.15401637,
1.33199539, 0. ]))

• update the package versions being tested against

WFC3 grism background subtraction relies on being able to process WFC3 IR RAW data using calwf3 first by setting FLATCORR on and CRCORR off. The resulting IMA files data are modified and are then passed to calwf3 again after setting CRCORR on. This method was recommended to use a few years ago and It used to work perfectly but now calwf3 fails with the error "RuntimeError: wf3ir.e exited with code -6"