The NO2 segmentation exists, but the plot doesn't show it.

Comparing the figures of two consecutive orbits, we find that the cloud pressure of the previous seems wrong. The left part is mostly around 400 hPa.

The mask is wrong. We need to write a notebook to dig more.

The default input lightning data is point data (time, longitude, and latitude). We can add the support of gridded density data, such as count=XXX in 10 minutes and 0.1*0.1 degree grid.

It can give us the property of cloud which has lightning.

When we finish #24, we can modify s5p_lnox_plot_tracks.py to plot figures by linking order.

Sometimes one orbit has several lightning masks, add the label text as subtitle is useful for users to compare and select cases.

The mask by the re-segmentation masked NO2 should be saved to output for checking relationships among lightning, lightning NO2, lightning NO2 mask, LNO2, and NO2VIS.

The current code just reads the necessary variables for analyzing NO2 and cloud.

Users may use our previous tool (S5P-WRFChem) to calculate AMF based on our products.

Sometimes users don't want to download all large CTMANA files, it's better to support reading one file containing dates generated by their own program.

The sensitivity of lifetime, alpha, crf_min, and peak_width is useful for calculating the uncertainty of estimated lightning NO2 production efficiency.

It should be st = pd.to_datetime(ds.attrs['time_coverage_start']).

The changes in the stratospheric NO2 column between consecutive orbits can affect the tropospheric NO2 calculation. Thus, it's better to add the output of stratospheric variables: nitrogendioxide_stratospheric_column and SCD_Strato (has been added).

The DBSCAN lightning labels have both clean and polluted lightning, while the lightning mask has distinguished them and grouped them using wind data:

Because the number of lighting labels represents the lightning count, it should be updated with the lightning mask. Otherwise, we don't know how much lightning happened in each lightning mask.

This method could lead to memory error if the data is large and the computer's cores are few. It's safe to set it to 2 or 3 and let users make their own choice.

Problem

The observed lightning flashes are clustered first and then classified into clean and polluted clusters.

However, that would split storms using the default 40 km

This method works well for isolated storms like this:
The LNO2 were produced and transported with wind:

However, there're some storms that occurred closely but were still far from 40 km. The LNO2 produced by them could mix together:

Solution

It's better to calculated the transported air containing LNO2 first, and then cluster and classify it.

Although #26 supports resampling the lightning mask values to lightning location (lightning_label), there're some missing values.

Lightning mask

lightning label

The lightning mask is wrong ... We need to fix it.

There's one lightning threshold at the current stage:
At least 20 lightning in the cluster

We can add another one: at least xx lightning during the 100-min period before the TROPOMI overpass time. This can make sure there's enough fresh lightning happens before TROPOMI overpasses for the calculation of lightning NO2.

Because pandarallel creates embedded Pools, it is not supported in the multiprocessing pool.

That's why we use ProcessPoolExecutor instead:

However, the Exception isn't printed at all ...

It's better to switch from the local computer to the cloud.

I found the free method of combing repl.it and uptimerobot.com.

The cloud slicing method is useful to get NO2 above the cloud. This can benefit the lightning NOx studies.

The size global CTMANA file is large. It's better to let users crop it to their interested region.

The threshold is hardcoded. A better method is to use the monthly average NO2 as the minimum value of the threshold.

Because we are developing S5P-LNO2 for Arctic lightning NO2 (background NO2 is quite low), this doesn't matter at the current stage.

Please feel free to PR for making it work for polluted regions!

For convection, there're many pixels with missing values. What's the exact problem?

Sometimes, the error will raise if any lightning happened before the first day of a month:

 File "/student/zhangxin/papers/Arctic/S5P_LNOx/main/s5p_lnox_functions.py", line 216, in calc_wind
    u_interp = interpn(coords, u, interp_points)
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/scipy/interpolate/interpolate.py", line 2698, in interpn
    raise ValueError("One of the requested xi is out of bounds "
ValueError: One of the requested xi is out of bounds in dimension 0

So, we need to load the correct ERA5 data here:

Two options:

• Load two months' ERA5 data
• Append the last day of the previous month to the current ERA5 data

See the tobac issue

To make #24 easier and quicker, we can find the consecutive filenames of outputs from s5p_lnox_main.py.

There are two applications:

• Extracting the TM5 profile data for useful swath data instead of all data (s5p_lnox_tm5_extract.py)
• Grouped swaths can be used directly for analysis of lightning NO2 production (s5p_lnox_main.py)

The CTMANA NO2 and Temperature profile data are useful for manual AMF calculation. It's better to modify the PyCAMA TM5_profile.py script to add the profiles in S5P group of output product.

We focus on LNO2 produced by storms occurring over 70N at this stage, but some convection can cross the line.

Here's an example:

Because we need the TROPOMI pixel area info for calculating the summation of NO2, it's useful to support the accurate area instead of using the first pixel's. Note that the pixel area is constant for different swaths, though

Along-track there are 3245 or 3246 scanlines (4172 or 4173 after the along-track pixel size reduction) in regular radiance orbits.

As mentioned in Marais et al. (2021):

they impose a modest threshold to only use TROPOMI tropospheric columns > 4 × 10^13 molec. cm−2 to mimic the detection limits of the instruments (Gomez et al., 2014)

It's 6.6^-7 mol/m2. It's better to change the constant 0~4e-5 colorbar to 0-max_value for tracked NO2 VCD.

Although I use Pool for predicting the cluster of lightning, it's still slow.

Because the next time step's location relies on previous location and u/v wind, I can't accelerate it (at least for my current Python ability) ... Leave it here, in case someone or I come up with a better idea.

The looping will recreate the pandarallel worker. That will waster some time.

Because the production efficiency is based on consecutive swaths, we need to finish #24 first.

Using SCD_bkgd only works when SCD_bkgd / AMF_trop = VCD_bkgd. But I suppose it's not true. We need to try to use VCD_bkgd directly and compare the results.

The Scene object doesn't have attrs anymore. So, the end_time should be passed from DataArray's attrs.

It's better to use tobac to link the masks automatically instead of using the plot function to check it by ourselves.

np.delete() will remove the element by the index, not the value.

There's another option: using TROPOMI CO data instead of SNPP VIIRS fire data. But, it will occupy more storage space.

After the discussion with Ronald van der A, the correct/accurate way to calculate VCD NO2 is using cloud pressure.

Besides, the lightning NO2 retrieval should use the customed AMF_LNO2 instead of using the ratio of integration.

Some users may use global attributes, such as time_coverage_start, time_coverage_end, product_version, etc.

I got this error when extract TM5 profiles to products generated by s5p_lnox_main.py

Adding NO2 and Temperature profiles to  /student/zhangxin/papers/Arctic/S5P_LNOx/main/output_gld360/clean_lightning/202107/S5P_PAL__L2__NO2____20210712T000148_20210712T014317_19403_02_020301.nc
Traceback (most recent call last):
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 180, in _nc4_require_group
    ds = ds.groups[key]
KeyError: 'S5P'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/student/zhangxin/papers/Arctic/S5P_LNOx/main/s5p_lnox_tm5_extract.py", line 48, in <module>
    with xr.open_dataset(file, group='S5P') as ds:
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/api.py", line 495, in open_dataset
    backend_ds = backend.open_dataset(
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 553, in open_dataset
    store = NetCDF4DataStore.open(
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 382, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 330, in __init__
    self.format = self.ds.data_model
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 391, in ds
    return self._acquire()
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 386, in _acquire
    ds = _nc4_require_group(root, self._group, self._mode)
  File "/public/home/zhangxin/new/miniconda3/envs/knmi_arctic/lib/python3.9/site-packages/xarray/backends/netCDF4_.py", line 186, in _nc4_require_group
    raise OSError(f"group not found: {key}", e)
OSError: [Errno group not found: S5P] 'S5P'

Two solutions:

1. Skip saving the product if it's empty
2. Skip extracting if it's empty

Because the duplicate of lightning density data doesn't affect the generation of lightning_mask, it's better to move it just before save_data().

It's useful to calculate the geo center which can be used to know where the linked convection is. Then, we can distinguish the calculated PEs is over land or ocean.

Saving a subset of original data takes up unnecessary spaces. It's better to create an empty file because we won't use that.