Error running start_maja.py / Doubts about folder-file structure about maja HOT 10 CLOSED

Hi, this one is easy, start_maja is meant for python 2.7, and what you observe is typically an error when running it witth python3.
Come on, you're almost there !
Olivier

from maja.

Hi Olivier,
Thanks a lot. I want to get there soon :)

I installed a python 2.7 environment with conda:

conda create --name py2 python=2.7
conda activate py2

and the run the code again:

python start_maja.py -f ~/maja/Start-MAJA/folders.txt -g ~/maja/Start-MAJA/GIPP/SENTINEL2_TM -l ~/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS -t 16PDU -s ser -d 20190801 -e 202000320

I get the following error of missing folders/files

However I do have the files.
Here is the tree of folder/files in html for easy visualization:
https://www.dropbox.com/s/t5glnhrht7zcm6f/tree_folders.html?dl=0

Thanks again!!!! :D

from maja.

Hi Inti, the message means that some of the folders you defined in folders.txt are not- existing. Could you please check them ?
repCode = ~/maja/bin/Start-MAJA
repMaja = ~/maja/bin/maja

One possibility is that python does not recognise "~", maybe try to replace by the whole path.

(it's now time for me to disconnect, but I can try to answer new questions tomorrow)
Olivier

from maja.

Dear Olivier,

I managed to get MAJA script finding correctly the paths and files. It took some tries to realize that when running the start-maja.py we should think of a additive path (folders.txt path + additional path for GIPP, LUT and site name)

Full path to LUT:
"/home/ernesto/maja/Start-MAJA/20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs"
Full path to GIP:"/home/ernesto/maja/Start-MAJA/GIPP/SENTINEL2_TM"
Full path to L1C data: "/media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA"

In my case, the folders.txt is like this now:

repCode= /home/ernesto/maja/Start-MAJA
repWork= /media/ernesto/cuatrot/maja_procesos/temporal
repL1 = /media/ernesto/cuatrot/maja_procesos/S2/L1C
repL2 = /media/ernesto/cuatrot/maja_procesos/S2/L2A
repMaja= /home/ernesto/maja/bin/maja
repCAMS= /media/ernesto/cuatrot/maja_procesos/cams

So, when running the script I just have add the rest of the path like this:
python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

Later I run it but I get some others errors:
`
ernesto@ernestomacpro:~/maja/Start-MAJA$ python start_maja.py -f folders.txt -g GIPP/SENTINEL2_TM -l 20190626_LUT_MAJA_SENTINEL2_TM_NOCAMS/LUTs -t 16PDU -s CHINANDEGA -d 20190801 -e 202000320

2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190805.DBL.DIR not found
2020-04-02 11:48:46,254 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190805-_T16PDU_C_V not found
2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190810.DBL.DIR not found
2020-04-02 11:48:46,255 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190810-_T16PDU_C_V not found
2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20190815.DBL.DIR not found
2020-04-02 11:48:46,258 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20190815-_T16PDU_C_V not found
....cut some rows just like the above part but for another dates....

2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20191223.DBL.DIR not found
2020-04-02 11:48:46,319 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191223-_T16PDU_C_V not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2?_OPER_SSC_L2VALD_16PDU____T20191228.DBL.DIR not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//SENTINEL2?_20191228-_T16PDU_C_V not found
2020-04-02 11:48:46,321 - Start-Maja - INFO - No existing L2 product, we start with backward mode
2020-04-02 11:48:46,322 - Start-Maja - INFO - => processing date 20190805
2020-04-02 11:48:46,337 - Start-Maja - INFO - dates to process in backward mode :
2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190805 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE
2020-04-02 11:48:46,337 - Start-Maja - INFO - -- 20190810 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190810T160911_N0208_R140_T16PDU_20190810T212513.SAFE
2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190815 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190815T160829_N0208_R140_T16PDU_20190815T211555.SAFE
2020-04-02 11:48:46,338 - Start-Maja - INFO - -- 20190820 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190820T160901_N0208_R140_T16PDU_20190820T212247.SAFE
2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190825 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190825T160829_N0208_R140_T16PDU_20190825T193312.SAFE
2020-04-02 11:48:46,339 - Start-Maja - INFO - -- 20190830 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190830T160901_N0208_R140_T16PDU_20190830T212354.SAFE
2020-04-02 11:48:46,340 - Start-Maja - INFO - -- 20190904 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190904T160829_N0208_R140_T16PDU_20190904T193256.SAFE
2020-04-02 11:48:46,341 - Start-Maja - INFO - -- 20190909 : /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2A_MSIL1C_20190909T160901_N0208_R140_T16PDU_20190909T212411.SAFE
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02 11:48:46,362 - Start-Maja - INFO - processing /media/ernesto/cuatrot/maja_procesos/S2/L1C/CHINANDEGA/S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE in backward mode
2020-04-02 11:48:46,362 - Start-Maja - INFO - Initialisation mode with backward is longer
2020-04-02 11:48:46,362 - Start-Maja - INFO - MAJA logfile: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log
2020-04-02 11:48:46,362 - Start-Maja - INFO - #################################
2020-04-02T11:48:47.644243 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:GetListOfGippFilenames:1699]

2020-04-02T11:48:47.645796 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.647014 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] PreProcessing: vns::Data::ERROR: DataApplicationHandler(0x1a655d0): In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! [vnsDataApplicationHandler.cxx:1699] [MAJA Data Exception] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.648199 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] [vnsMajaMainProcessor.cxx:main:129]

2020-04-02T11:48:47.649346 ernestomacpro maja-processing-3.3.2 3.3 [000000009310] [E] ****************************************************************************************************** [vnsMajaMainProcessor.cxx:main:129]

2020-04-02 11:48:47,666 - Start-Maja - INFO - #######################################
2020-04-02 11:48:47,666 - Start-Maja - INFO - Error detected, see: /media/ernesto/cuatrot/maja_procesos/S2/L2A/CHINANDEGA/16PDU/GIPP/SENTINEL2_TM//S2B_MSIL1C_20190805T160839_N0208_R140_T16PDU_20190805T193305.SAFE.log
2020-04-02 11:48:47,666 - Start-Maja - INFO - #######################################
(py2) ernesto@ernestomacpro:~/maja/Start-MAJA$
`
What is this file type (S2?_OPER_SSC_L2VALD_16PDU____T20190805.DBL.DIR not found)missing?

In my GIP folder I have:

and in my LUT folder:

Thank you for your help,
Inti

from maja.

Hi Inti,
The message "In the input directory, there is no GIPP input data matching with the 'AUX_REFDE2' File_Type pattern! " means that the DTM was not found.

It has to be stored in the DTM folder, where you launch start_maja from.
Olivier

from maja.

Hi Olivier,
It is strange because I do have a DTM folder inside star-maja folder in which I run the script as it can be seen here:

and inside the DTM folder I have the corresponding DTM files (2 files per tile) created with the preparation tools:

As you can see there are the files required for this tile:
S2__TEST_AUX_REFDE2_16PDU_0001.DBL.DIR
S2__TEST_AUX_REFDE2_16PDU_0001.HDR

What else could it be?
Thank you!

from maja.

Hi Olivier,
I have to say that the DTM files were created using the dtm preparation tools suggested by Peter (petket-5) in the previous post:#50 that are located in https://github.com/CNES/Start-MAJA/tree/reprog-rc1/prepare_mnt.

Afterwards, I just moved the created dtm files to the DTM folder inside original start-maja folder (coming from https://github.com/CNES/Start-MAJA). I hope this information can help find the error cause.

Thanks,

from maja.

Hi inti,
I checked, and (sorry to have forgotten this part), the DBL.DIR and HDR files are not stored in DTM folder. There is another folder in between:

[hagolle@visu02 Start_Maja]$ ls -1 DTM
S2__TEST_AUX_REFDE2_T11SPC_0001 
S2__TEST_AUX_REFDE2_T29TPH_0001  
S2__TEST_AUX_REFDE2_T31TFJ_0002  
S2__TEST_AUX_REFDE2_T32TNS_0001  
S2__TEST_AUX_REFDE2_T35JPM_0001
[hagolle@visu02 Start_Maja]$ ls DTM/S2__TEST_AUX_REFDE2_T11SPC_0001
S2__TEST_AUX_REFDE2_T11SPC_0001.DBL.DIR  S2__TEST_AUX_REFDE2_T11SPC_0001.HDR

I hope it's the last one !
Olivier

from maja.

Hi Olivier,
Thanks, that last hint of the folder in between the DTM and the DTM files (.DBL.DIR .HDR) was what was missing and now it worked fine and I was able to process images for 5 months of one tile since yesterday.

Thanks a lot,
Inti

from maja.

I told you you were close to succeed, but that's a good news, thanks for letting us know !
Peter @petket-5 has developped a new version of start_maja, which should be more user friendly. It is in its final tests.

from maja.