Once we verify the phase calculation we will need @jkmacc-LANL to verify he is happy with the structure of compute_psd.py (currently awaiting a pull-request).

We can let @jkmacc-LANL when we are ready.

We need to check the following things. I will do 4, 5, and 6, once Rob checks 1, 2, and 3 again.

1. I am seeing the pckls are a third smaller than the txt files. Around 128K while the text are 302K. Can you verify this is what you are finding and in the right ballpark? What would the new size calculation be?

2. I am finding the data request to be about a factor of 100 slower than the PSD calculation. Can you confirm this?

3. We should check the timing with reorganized code. Do my changes speed things up?

4. We need to deal with multiple location codes (rare, but could throw off calculations).

5. We should parallelize this (I can do this once you verify 1, 2, and 3).

6. We should include a log file for problematic pulls from IRIS (I can do this).

Hi @reanthony-usgs and @aringler-usgs . I wanted to update you both on progress.

I ran the spectral calculation last week and now have all the corrected PSDs and CSDs for the TA in Alaska for 2018. This doesn't include retrieval time from IRIS, which was another 4-5 hours on a much smaller cluster (20 nodes). Estimating from the work we'd done previously, the whole TA could be ingested in about 3 days if we pursued it.

I'm working on how best to share this with you, or at least to show you what the results look like. I think I could post a Jupyter notebook somewhere that illustrates some of this. Full access to the data themselves is not currently possible, however.

Please let me know how you'd like to proceed. I'll work on getting approval to share the notebook(s).

Thanks for your patience.

EDIT: PSDs for Alaska took 4-5 hours.

We need to add code for doing a re-request based on reading the log files. This is just a parsing of the log file and then request. We might want to move some of the functionality into the utils.

1. Add 50% overlap of PSD Windows - Done

2. Add a dead channel QC metric - Rob suggests using 4-8 s microseism and ignoring time periods when PSD falls X dB (3?) below the NLNM in this range. Rob can work on this

3. Dave suggests using the INV to possibly cull sensor types used in the analysis (e.g. only uses STS-2, T120, T240, etc. data)

4. Verify the 90% completeness threshold (?) Is this worth our time?

5. Verify that we want to use a "Constant" detrend (line 102) before zero padding time-series data (there was discussion about making the starts and ends of these data segments "0" prior to zero padding)

Rob,

Can you verify that the new PSD calculation is what we want? I switched to cross-powers so that when the channels aren't equal we can get the phase.