ERROR: No GIPP of type GIP_CKEXTL has been detected for the Mission <SENTINEL-2A> about maja HOT 21 CLOSED

Hello Olivier,
I totally missed that error indeed. When downloading the altered GIPP files from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master , I see that you updated GIPP_S2_MAJA_3_TM for MAJA 3.1 standalone version. I am currently using MAJA 3.3 and manually added the dashes in the four GIPP files in my Maja-3.3.0-noTM, but this unfortunately resulted in another error:

"ERROR - First backward processing was unsuccessful, check MAJA installation"

Interestingly, my .log file ends with:
"Process OK ;-) [code return: 0]"
and I have a variety of .TIF, .HDR and a .JPEG as outputs.

The .log file is the following:
S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

Could you give me any insights about this?
Thank you,
Daria

from maja.

Hi Daria,
I am lacking information. Please send the command line, folders.txt and full log.
Did you get the GIPP directory from gitlab, which one ?
Thanks,
Olivier

from maja.

Thanks for your quick reply, Olivier.

The command line used in the Start-MAJA folder is
./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t T31TFJ -s avignon -d 20190314 -e 20190326

The GIPP directory used is GIPP_S2_MAJA_3.3_TM
(https://github.com/CNES/Start-MAJA/files/3349574/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.txt)
downloaded from http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM

Here is the folders.txt and the log file.
folders.txt
S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

Thanks,
Daria

from maja.

Thanks Daria,
there was an error in 4 GIPP files which must contain <Mission>SENTINEL-2A</Mission> or <Mission>SENTINEL-2B</Mission> but contained <Mission>SENTINEL2A</Mission> or <Mission>SENTINEL2B</Mission>

S2A_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF S2B_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF

While the other GIPP files must not contain the dashes. I hope we will be able to remove this unneeded complexity in MAJA 4.0.

You can download again the GIPP folder, I have updated it.
Sorry for this error.
Olivier

from maja.

Hi,
I am not sure, but the parameters you provided : "-d 20190314 -e 20190326" indicate a very short period of time. MAJA is a multi-temporal processor, and needs to process time series. It is possible that in that period there is no cloud free pixel.
Could you please try to download more dates and process a longer time period ? Usually we use a minimum period of one month.
Olivier

from maja.

Hello Olivier,

I re-ran MAJA 3.3 noTM without CAMS with a month of input data and got the same result:
Again, the .log file gives me
"Process OK ;-) [code return: 0]"
as the last line while the terminal returns the error of
"ERROR - First backward processing was unsuccessful, check MAJA installation".

As far as the outputs are concerned, they are (apart of the folder structure) consistent with the expected native format published here: http://www.cesbio.ups-tlse.fr/multitemp/?page_id=10464#English .

(To be specific what I mean when I talk about the difference in folder structure is that the masks (QLT, CLD, MSK) are provided in the same directory as FRE, SRE, and ATB; while the MSK_DETFOO folder contains one .gml file per band and there's an additional PRIVATE folder whose use I don't really understand.)

Do you think the outputs are correct despite the error in the terminal? I loaded the TIFFs in a GIS and they all have the correct spatial resolution, spatial extent, and the right number of bands.

Thanks,
Daria

from maja.

Hi Daria,

I am sorry for the long process, I have too many meetings, but it's good to comeback to coding at the end of the week.

Yes the log you sent clearly shows that MAJA works fine, and the L2A product you have is fine.
But start_maja.py is supposed to process the whole time series for you, and it seems it stops after the first production in backward mode, because it does not detect the L2A product.
The problem is that I cannot reproduce the error, it works on my side.

Did you find the l2a product in this folder ? /home/darialuedtke/MAJA/data/sentinel-2/l2a_maja
Could you also give me the log which is printed on your terminal ?

I am also running a new tests, result in 30 minutes...
Olivier

from maja.

Oups, backward is two hours, result on Monday...
Olivier

from maja.

My test was unsuccessful in reproducing what you observe.
Please send the information requested above.
Olivier

from maja.

Hello Olivier,

Sorry for the late reply. I am on my vacation until Friday and didn't bring my computer. I will provide you what you need first thing Friday morning and ask my colleague who had the same observation if he can share with you today.

Best, Daria

from maja.

All right, enjoy your holidays !

I will be on holidays myself at the end of the week, but I'll see if someone else in the team can help you.
Olivier

from maja.

Hello Olivier,
I am Gonzalo, Daria's colleague.
I have reproduced the error that Daria had, with the same setup.

We have installed Maja-3.3.0-noTM and executed "./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t T31TFJ -s avignon -d 20190314 -e 20190326" (I have excuted maja with only 3 different L1C files for the sake of speed, seeing that the error I have is the same as Daria's with a full month of data).

The GIPP files and LUTs are the following (downloaded from Cesbio's repositories and after applying the corrections to the mission SENTINEL-2A).
GIPP downloaded from: http://tully.ups-tlse.fr/olivier/gipp_maja/tree/master/GIPP_S2_MAJA_3.3_TM
LUTs downloaded from: https://zenodo.org/record/2636694

My folders.txt file is the following:
folders.txt

The output log file from the latest execution is:
S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log

The result on my L2 folder (/mnt/data/sentinel2/L2) is the following:
.
└── Avignon
└── 31TFJ
└── GIPP_S2_MAJA_3.3_TM
├── S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log
├── S2A_OPER_PMC_L2REPT_31TFJ____20190314.EEF
├── S2A_OPER_SSC_L2VALD_31TFJ____20190314.DBL
├── S2A_OPER_SSC_L2VALD_31TFJ____20190314.DBL.DIR
│   ├── MSK_DETFOO
│   │   ├── MSK_DETFOO_B01.gml
│   │   ├── MSK_DETFOO_B02.gml
│   │   ├── MSK_DETFOO_B03.gml
│   │   ├── MSK_DETFOO_B04.gml
│   │   ├── MSK_DETFOO_B05.gml
│   │   ├── MSK_DETFOO_B06.gml
│   │   ├── MSK_DETFOO_B07.gml
│   │   ├── MSK_DETFOO_B08.gml
│   │   ├── MSK_DETFOO_B09.gml
│   │   ├── MSK_DETFOO_B10.gml
│   │   ├── MSK_DETFOO_B11.gml
│   │   ├── MSK_DETFOO_B12.gml
│   │   └── MSK_DETFOO_B8A.gml
│   ├── PRIVATE
│   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ____20190314_LTC.DBL.DIR
│   │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ____20190314_LTC.DBL.mha
│   │   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ____20190321_LTC.DBL.mha
│   │   │   └── S2B_OPER_SSC_LUTANX_L2VALD_31TFJ____20190326_LTC.DBL.mha
│   │   ├── S2A_OPER_SSC_LUTANX_L2VALD_31TFJ____20190314_LTC.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_NDT.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_NDT.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_PXD.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_PXD.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RCR.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RCR.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RTA.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RTA.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RTC.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_RTC.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_STO.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_STO.HDR
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_WAM.DBL.TIF
│   │   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_WAM.HDR
│   │   └── S2A_OPER_TEC_L2VALD_31TFJ____20190314.EEF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R1.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_ATB_R2.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD_R1.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_CLD_R2.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_MSK_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_MSK_R1.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_MSK_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_MSK_R2.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R1.HDR
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTANX_L2VALD_31TFJ____20190314_QLT_R2.HDR
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_FRE_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_FRE_R1.HDR
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_FRE_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_FRE_R2.HDR
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_SRE_R1.DBL.TIF
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_SRE_R1.HDR
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_SRE_R2.DBL.TIF
│   ├── S2A_OPER_SSC_PDTIMG_L2VALD_31TFJ____20190314_SRE_R2.HDR
│   ├── S2A_OPER_SSC_PDTQLK_L2VALD_31TFJ____20190314.DBL.JPG
│   └── S2A_OPER_SSC_PDTQLK_L2VALD_31TFJ____20190314.HDR
└── S2A_OPER_SSC_L2VALD_31TFJ____20190314.HDR
(Here in image format, it may be easyer to read:

)

The full log from terminal is the following:
root@610433098ef8:/Start-MAJA# ./start_maja.py -f folders.txt -g GIPP_S2_MAJA_3.3_TM -l LUT_MAJA_3_TM_CAMS -t 31TFJ -s Avignon -d 20190314 -e 20190326

2019-07-08 11:03:07,396 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2019-07-08 11:03:07,396 - Start-Maja - INFO - => processing date 20190314
2019-07-08 11:03:07,399 - Start-Maja - INFO - dates to process in backward mode :
2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190314 : /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE
2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190321 : /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190321T103021_N0207_R108_T31TFJ_20190321T173032.SAFE
2019-07-08 11:03:07,400 - Start-Maja - INFO - -- 20190326 : /mnt/data/sentinel2/L1/Avignon/S2B_MSIL1C_20190326T103029_N0207_R108_T31TFJ_20190326T155314.SAFE
2019-07-08 11:03:07,415 - Start-Maja - INFO - #################################
2019-07-08 11:03:07,415 - Start-Maja - INFO - #################################
2019-07-08 11:03:07,415 - Start-Maja - INFO - processing /mnt/data/sentinel2/L1/Avignon/S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE in backward mode
2019-07-08 11:03:07,415 - Start-Maja - INFO - Initialisation mode with backward is longer
2019-07-08 11:03:07,415 - Start-Maja - INFO - MAJA logfile: /mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log
2019-07-08 11:03:07,416 - Start-Maja - INFO - #################################
2019-07-08 12:03:48,299 - Start-Maja - INFO - => processing date 20190321
2019-07-08 12:03:48,320 - Start-Maja - INFO - MAJA command failed : /maja/bin/maja -i /mnt/data/sentinel2/work/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//in -o /mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM/ -m L2BACKWARD -ucs /mnt/data/sentinel2/work/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//userconf --TileId 31TFJ >/mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM//S2A_MSIL1C_20190314T104021_N0207_R008_T31TFJ_20190314T160209.SAFE.log 2>&1
2019-07-08 12:03:48,321 - Start-Maja - ERROR - First backward processing was unsuccessful, check MAJA installation

I hope this information is all you needed and it is clear enough, but feel free to ask for any other test or complementary information you may need.

Thanks in advance,
Gonzalo

from maja.

Hi, thanks for providing more information !
I still do not understand the cause of failure. Everything seems fine in the log file, the L2 is generated, but Start_Maja does not find the L2A product to produce the next one. I have made a few modifications within start_maja.py to print in the console log the L2A file name MAJA is searching for. If you could update your start_maja.py version and run again the command line.

The best would be to erase this directory before running again:
/mnt/data/sentinel2/L2/Avignon/31TFJ/GIPP_S2_MAJA_3.3_TM

Thanks, Olivier

from maja.

Hi!
I have not had the time to test with the new version of Start-MAJA. But I have tested installing and using Start-MAJA with Maja-3.3.0-TM and this time the process is able to finish and it creates all the L2 files

I will retest with MAJA-noTM and the new version of Start-MAJA as soon as possible and let you know the results.
Thanks, Gonzalo

from maja.

Thanks for the good news.
I had not noticed you were using the no-TM version (although Daria had written it). The problem is that you are using TM GIPP and LUTS with the No-TM version of MAJA.

My advice is too use the TM plugin. The no-TM is only provided for compatibility for previous users of MAJA. It will be probably abandoned at the end of 2019.

Olivier

from maja.

Hello Olivier,

I hope you are enjoying your vacation. Gonzalo and I have had a chat to update me what happened during my absence. MAJA is still causing several issues for us.

One of them is the processing time and the space required. I tried to run it on one month's worth of data (9 images) and we provided around 30GB of disc space, which wasn't sufficient. It produced an error after over 1 hour running time due to lack of space. Could you give us an estimation how much disk space and time is needed for processing one tile? Alternatively, is there a way to reduce the disk space needed for the processing?

Secondly, since we want to investigate a larger region, we aim at reusing the flat surface reflectances so we don't have to go through the entire multi-temporal processing chain every time a new image for a tile comes in. Could you please give us an idea how we can reuse it and how it will affect the processing parameters (disk space/ time)?

Thanks for your help!
Daria

from maja.

Hello everyone,

I have got the same error (No GIPP of type GIP_CKEXTL has been detected for the Mission ). However, in my case I am running MAJA-3.2.2_TM.run using GIPP_S2_MAJA_3_TM.

Olivier, you have commented that is already fixed and when I open the file I can see the change so I don't understand what is happening.

The .log file is the following:
S2B_MSIL1C_20180803T112109_N0206_R037_T29TPH_20180803T152545.SAFE.log

Thank you very much
Ruben

from maja.

Hello Ruben,
We have found out after testing the noTM and TM versions that there is a discrepancy between what they expect in the <Mission> field on the GIP_CKEXTL files.

For the TM version, Start-MAJA expect to have the <Mission> parameter without a dash. In order to correct your problem, you can modify the following files:
S2A_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF S2A_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF S2B_TEST_GIP_CKEXTL_S_31TJF____10001_20150703_21000101.EEF S2B_TEST_GIP_CKQLTL_S_31TJF____10005_20150703_21000101.EEF

You will have to change the parameter so it shows <Mission>SENTINEL2A</Mission> or <Mission>SENTINEL2B</Mission>. With this changes, MAJA should work.

Olivier, I don't think you where aware of this problem, we just found it out.

I hope this helps!

from maja.

Dear Ruben,
Unless you have very specific reason to use version 3.2.2, I would recommend you to download the latest 3.3.0[TM] version and re-try. You may well keep version 3.2.2 if needed and still have 3.3.0 installed beside, provided that you point to the proper version of the executable in your 'folder.txt' file.

Please let us know if you succeed,
Best,
Jerome

from maja.

Hello both,

@alvarezWegaw Thanks. I have changed the parameter and it seems to be working now.
@jerome-colin I will test the new version 3.3.0 in coming weeks, for now I will just change the parameter.

Thanks
Ruben

from maja.

I like that when users help each other (thank you @alvarezWegaw !)
Sorry for the mistake, this error in the parameters keeps coming back although I have chased it. Thanks for telling us, I will try to correct it.
Olivier

from maja.