jwely / ledaps Goto Github PK

LT40230281982346XXX04 atmospheric opacity layer is totally fill, and the 
fill_qa layer disagrees with that.  And I've never seen a totally filled 
atmopacity layer before.  Worth looking into to make sure the layer can be 
generated for L4TM data, or if we should stop making atmospheric opacity for 
L4TM

Hello.
I am having immense difficulty finding the correct dependent libraries listed 
in step one of the the ledaps installation guide:

1. Install dependent libraries - HDF-EOS GCTP, HDF4, HDF-EOS2, TIFF, GeoTIFF

Google has not been my friend on this. I have spent virtually an entire day 
installing and uninstalling several different libraries. A google search of ' 
"HDF-EOS GCTP" library' turns up nothing other than the original install 
documentation for ledaps I cited above. 

Below is a list of what I am guessing are the correct libraries but a nod from 
you guys would go along way to me keeping my sanity.

HDF4 is the correct version  
http://hdf-eos4.sourcearchive.com/downloads/2.17v1.00.dfsg.1-3/

HDF-EOS2 is this the correct version  
ftp://edhs1.gsfc.nasa.gov/edhs/hdfeos/latest_release/HDF-EOS2.18v1.00.tar.Z

TIFF is this the correct version:  tiff-4.0.3.tar.gz from 
ftp://ftp.remotesensing.org/pub/libtiff 

GeoTIFF is this the correct version: libgeotiff-1.3.0.tar.gz from here 
ftp://ftp.remotesensing.org/pub/geotiff/libgeotiff/ (I couldn't get 1.4.0 to 
build) 

Thanks in advance.
Geordie Hobart
[email protected]

LE70390372008210EDC00 has a ton of nonveg bright areas next to water, not 
LEDAPS' favorite place, but the diffs b/n SR values before and after ESPA212 
and 213 are concerning. Gail has my before and after files.

LT40230281982346XXX04 pixels out of bounds (I like to call them OOBs), in the 
L4TM TOA Band 4, 84 pixels have values over 16000.

Updated 6S code to do error checking in the event that a file is not available 
to be opened/created.  The error is caught and a message is printed.  Updated 
lndsr code to use the bash shell instead of the sh shell to invoke the 6s 
command file.  These mods were made in an attempt to prevent the temporary 
file error which occurs intermittently in ESPA.

What steps will reproduce the problem?
I thought I had everything installed correctly, but when trying to make lndsrbm:
/home/jgrn/code/src/ledapsSrc/src/lndsrbm 
@taub307> make
gfortran -Wall -O2 -fno-range-check -o cmrbv1.0 cmrbv1.0.f -I. 
-I/usr/apps/oa/include -I -L/usr/apps/oa/lib -lmfhdf -ldf -L -ljpeg -lz -lm
/usr/bin/ld: cannot find -lmfhdf
collect2: ld returned 1 exit status
make: *** [cmrbv1.0] Error 1

What is odd, is that lndsr also relies on lmfhdf and compiled correctly, and I 
confirmed that in my lib directory I have:
libmfhdf.a
libmfhdf.la
libmfhdf.so
libmfhdf.so.0
libmfhdf.so.0.0.0

My LD_LIBRARY_PATH has the correct path to this:
@taub307> echo $LD_LIBRARY_PATH
/usr/local/intel/intel-13.1.1/lib/intel64:/usr/local/intel/intel-13.1.1/mkl/lib/
intel64:/usr/local/openmpi-1.4-gcc/lib:/usr/local/torque-4.2.5.h2/lib:/usr/local
/lib64:/usr/local/lib:/lib:/lib64:/usr/lib64:/usr/lib:/usr/apps/oa/lib

(/usr/apps/oa/lib has the .so above).

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
1.3.1, *nix

Please provide any additional information below.

I am having an issue checking out the most recent version of ledaps.
I am running Apache Subversion 1.7 on a windows 7 64 bit machine through 
windows powershell.

When I run the command 
 svn checkout http://ledaps.googlecode.com/svn/trunk/ ledaps-read-only

I get the following output and error.

Restored 'ledaps-read-only\ledapsAncSrc'
Restored 'ledaps-read-only\ledapsAncSrc\ncep_reanalysis_surface_repackage.c'
Restored 'ledaps-read-only\ledapsAncSrc\convert_ozone.c'
Restored 'ledaps-read-only\ledapsAncSrc\updatetoms.py'
Restored 'ledaps-read-only\ledapsAncSrc\Makefile'
Restored 'ledaps-read-only\ledapsAncSrc\updatencep.py'
Restored 'ledaps-read-only\ledapsAncSrc\Makefile.static'
Restored 'ledaps-read-only\ledapsSrc'
Restored 'ledaps-read-only\ledapsSrc\release.note'
Restored 'ledaps-read-only\ledapsSrc\doc'
Restored 'ledaps-read-only\ledapsSrc\doc\LEDAPS_User_Ref_SR.txt'
Restored 'ledaps-read-only\ledapsSrc\doc\lndsr.fs'
Restored 'ledaps-read-only\ledapsSrc\src'
Restored 'ledaps-read-only\ledapsSrc\src\lndpm'
Restored 'ledaps-read-only\ledapsSrc\src\lndpm\lndpm.c'
Restored 'ledaps-read-only\ledapsSrc\src\lndpm\Makefile'
Restored 'ledaps-read-only\ledapsSrc\src\lndpm\HISTORY.txt'
Restored 'ledaps-read-only\ledapsSrc\src\lndpm\Makefile.static'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\OCEA.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\HAPKALBE.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\OXYG6.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\CHAND.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\WAVA3.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\main.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\SPLIE2.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\MODISALBE.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\SPLINT.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\AATSR.f'
Restored 'ledaps-read-only\ledapsSrc\src\6sV-1.0B\MIE.f'
svn: E720003: Can't open file 
'V:\Landsat\ledaps\ledaps-read-only\.svn\pristine\be\be25d26afa803faec491f5a9307
085e9f4694
446.svn-base': The system cannot find the path specified.

Am I doing something wrong?

Thanks
Geordie Hobart, BEng
Research Assistant
Natural Resources Canada / Ressources naturelles Canada
Canadian Forest Service / Service canadien des forêts
Pacific Forestry Centre / Centre de foresterie du Pacifique
506 W. Burnside Road / 506 rue Burnside Ouest
Victoria, BC  V8Z 1M5 / Victoria, C-B V8Z 1M5

Tel: (250) 298-2403    Fax: (250) 363-0775
mailto:[email protected]

What steps will reproduce the problem?
1. Run ncep_reanalysis_surface_repackage, e.g.
$ ./ncep_reanalysis_surface_repackage pr_wtr.eatm.2007.nc 
REANALYSIS/RE_2007/REANALYSIS_2007334.hdf 334 

What is the expected output? 
The REANALYSIS_2007334.hdf should have   base_date = 2007s, 1s, 1s ;

What do you see instead?
The REANALYSIS_2007334.hdf   base_date = 2006s, 1s, 1s ;

What version of the product are you using? On what operating system?
Linux....

Please provide any additional information below.
See minimal patched file for more info. Note, patch build requires an extra 
link with -ludunits2

Cheers,
Andy

The lndcal change specifically is to include all the science data sets in the 
global attributes.  Currently they do not list the qa layer (DataFieldName="qa 
bitmap"), which would be added as DataField_6.  I'm not sure what the DataType 
value would be exactly, but it is an unsigned 8-bit field.  

        GROUP=DataField  
            OBJECT=DataField_0  
                DataFieldName="band1"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_0  
            OBJECT=DataField_1  
                DataFieldName="band2"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_1  
            OBJECT=DataField_2  
                DataFieldName="band3"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_2  
            OBJECT=DataField_3  
                DataFieldName="band4"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_3  
            OBJECT=DataField_4  
                DataFieldName="band5"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_4  
            OBJECT=DataField_5  
                DataFieldName="band7"  
                DataType=DFNT_INT16  
                DimList=("YDim","XDim")  
            END_OBJECT=DataField_5  
        END_GROUP=DataField  

Scenes which are highly cloudy have snow vs. cloud transitions which are quite 
abrupt, even linear.

Here are a few scenes which show these delineations, with LE70280312004362EDC00 
being the most obvious.

LE70280311999348EDC00  LT50280312001025XXX02
LE70280312004362EDC00  LT50280312010114GNC01

What steps will reproduce the problem?
1. Build lndsr with http://edcftp.cr.usgs.gov/pub/software/gctpc/gctpc20.tar
2.
3.

What is the expected output? What do you see instead?

I get the error:

INFO: From Compiling third_party/ledaps/lndsr/space.c:
third_party/ledaps/lndsr/space.c:101:5: error: conflicting types for 'for_init'
int for_init( int outsys, int outzone, double *outparm, int outdatum,
    ^
third_party/gctpc/source/proj.h:164:6: note: previous declaration is here
void for_init(long outsys, long outzone, double *outparm, long outspheroid,
     ^
third_party/ledaps/lndsr/space.c:104:5: error: conflicting types for 'inv_init'
int inv_init( int insys, int inzone, double *inparm, int indatum,
    ^
third_party/gctpc/source/proj.h:197:6: note: previous declaration is here
void inv_init(long insys, long inzone, double *inparm, long inspheroid,
     ^
2 errors generated.


What version of the product are you using? On what operating system?

LEDAPS (r122) on 64bit Linux. 


Please provide any additional information below.

The land/water mask flags the top and bottom of the scene as water, even when 
it isn't.

Use LT50260272011279PAC01 as a good example when tracking this issue.

What steps will reproduce the problem?
run do_leadps.csh (1.3.0)on LT50060221989206PAC00 and LT50060231989206PAC00

What is the expected output? What do you see instead?
The ledaps product file run on LT50060221989206PAC00 and LT50060231989206PAC00 
has an opacity layer that is all no data. Both images are very hazy.


What version of the product are you using? 

PARAMETER_FILE
DEM_FILE = /finland/ledaps/LedapsAnc/CMGDEM.hdf
OZON_FIL = /finland/ledaps/LedapsAnc/EP_TOMS/ozone_1989/TOMS_1989206.hdf
PRWV_FIL = /finland/ledaps/LedapsAnc/REANALYSIS/RE_1989/REANALYSIS_1989206.hdf
REF_FILE = lndcal.LT50060221989206PAC00.hdf
TEMP_FILE = lndth.LT50060221989206PAC00.hdf
SR_FILE = lndsr.LT50060221989206PAC00.hdf
META_FILE = LT50060221989206PAC00_MTL.txt
LEDAPSVersion = 1.3.0
END

On what operating system?
Linux version 2.6.32-358.2.1.el6.x86_64 ([email protected]) (gcc version 
4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) ) 

Please provide any additional information below.
I have run LEDAPS 1.3.0 on over 40,000 images with no issues like this 
occurring before to my knowledge.

Create an ADD using the ADD template for the LEDAPS application.

I'm thinking L7 polar stereographic scenes probably don't get a lot of ground 
control input, which is probably the reason there are no GCP files associated 
with LE72101172013061ASN00, but thought I'd mention it.

We've been forgetting to change the SDS name "fmask" to "cfmask."  Not sure if 
that happens in the cfmask code or in ledaps, so entering issue in both places.

Add command line include_xxx options

such as
include_sr
include_toa
include_bt

Default to all/include_sr, which will do all.

But at times maybe only toa and or bt are needed.
This would reduce the processing time required to get those two products, since 
today all three are generated all the time.

Initiated by Junchang Ju's observations, and maybe not a bad idea to consider 
for future implementation.

Ideally in the one-byte QA mask bands, 0 can be for “absence”, and 1 for 
“presence”, and 255 for fill value

Update the building of Ledaps to utilize an automated build tool such as cmake 
or the autoconf set of tools.

I'm noting negative (out of bounds) values in the Band 5 and Band 7 TOA and 
Surface Reflectance products from LEDAPS 1.3.1.  They occur in difficult areas, 
such as the western edge of the image, and especially in water and in terrain 
shadows. I'm not suggesting an error in calculation, but perhaps the min/max 
range is struggling to know what to do in these odd landscapes?  Reference 
scenes include LT40230281982346XXX04, LT50230281984344XXX02, and 
LE72101172013061ASN00

The Landsat metadata values will be changing due to the LDCM project merges 
with Landsat.  These changes need to be assessed and determine if/how they 
affect the LEDAPS processing.

* What steps will reproduce the problem?

1.
svn co ledaps to /opt/ledaps
Debian Wheezy

2.
compile hdfeos in /opt/hdfeos from these source:
hdf-4.2.6.tar.gz
szip-2.1.tar.gz
hdf5-1.8.8.tar.gz
libgeotiff-1.4.0.tar.gz
hdf-4.2.10-linux-x86_64.tar.gz
jpegsrc.v6b.tar.gz
zlib-1.2.5.tar.gz
tiff-4.0.3.tar.gz
HDF-EOS2.18v1.00_TestDriver.tar.Z
hint to do this: http://hdfeos.org/forums/showthread.php?p=1339
(was about the gfortran bug in installer)

3.
set env.sh and make


* What is the expected output? What do you see instead?

Expected: Done ;-)

Result:
***
root@landsat:/opt/ledaps/ledapsSrc# ./env.sh
make[1]: Entering directory `/opt/ledaps/ledapsSrc/src/lndpm'
cc -g -D_BSD_SOURCE -Wall -O2 -I. -I -I lndpm.c -L -l_espa_raw_binary 
-l_espa_common -L -lxml2 -lz -lm  -o lndpm
In file included from lndpm.c:38:0:
lndpm.h:29:27: fatal error: espa_metadata.h: Datei oder Verzeichnis nicht 
gefunden
compilation terminated.
***


* What version of the product are you using? On what operating system?
Ledaps 1.3.1 on Linux landsat 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 
GNU/Linux with gcc (and automake, bison, flex...)


* Please provide any additional information below.
Even find /opt -name "espa*" does not find the file

Peer review of the LEDAPS source code have identified the following two items 
that should be addressed:
1) The 6S part of code and calls to them are not very well commented.
2) There are lots of commented out code in different applications, for example 
in "lndsr" and "lndcal" parts of code, maybe the unused code there are better 
kept in the release for some purposes?

The lndsr metadata does not contain information for the internal QA bits.  This 
information for bits 8-15 should be included in the metadata.

Investigated an issue flagged by one of our users.  He indicated the QA bands 
from the GeoTiff files are basically listing all the non-cloud, non-snow, etc. 
pixels as NoData in ArcGIS.  The pixels are flagged appropriately as 
cloud/no-cloud, snow/no-snow, etc.  However, I believe his ArcGIS tool is 
picking up the NoData value in the GeoTiff metadata and then using it to flag 
the "no-whatever" pixels as NoData.  I would like to simply remove the NoData 
tag from these bands.  I think it would be cleaner to simply have a single 
fill/no-fill band (which we currently have), and then not address NoData in the 
other QA bands.  If the user wants to know what is fill/NoData, then they can 
look at the Fill QA band.  I think the NoData tag simply makes this information 
more confusing.

I am having an issue wrt the results of Ledaps generated by my build of 1.2.1 
and the products ordered through  https://espa.cr.usgs.gov.
The values differ (for some images) in bands 1, 2,3 and 4 (only bands I have 
looked at) by between 5 and 100 which seems to be beyond what I should be 
expecting due to differences in architecture between machines. Should I be 
concerned with this? In most case the difference is less than 1%.

I was wondering if this might be due to differences in the ancillary files 
being used? I downloaded my ancillary files from 
http://landsat.usgs.gov/espa/files/. 

I started to investigate and attempt to build the files within the directory 
ledapsAncSrc. Is this useful or necessary? If no please ignore the following 
paragraph. :)

I am having issues as there appears to be no documentation to supplement this 
package. I tried to run the makefile, but failed with the message 
"convert_ozone.c:5:25: error: hdf4_netcdf.h: No such file or directory". A 
search of my system and the dependent libraries (installed for ledaps) for 
"hdf4_netcdf.h" was unsuccessful. I did however find a "netcdf.h" file and 
changed the include statements to netcdf.h instead in both the "convert_ozone.c 
and nep_reanalysis_...c as well as including the szlib in the library and 
include declarations in the makefile. And volia it compiled and installed, and 
the executables were copied into the ledapsSrc/bin file as expected.

I reran ledaps with the new executables - from ledapsAncSrc in place - but the 
results still did not jive with the NASA generated images. Even stranger still 
is the fact that the SR result of ledaps for LT50330242010197EDC00 matched the 
expected results within 1 BUT for LT50360232010218EDC00 the results are as 
stated above. Confounding as both images are from the same sensor and year. 

Can you give me any suggestions or insight as to why this discrepancy is 
occurring and whether I should be concerned?

As an aside. I should note that when I built ledaps I needed to edit several of 
the C files (ar.c, lndsr.c, myhdf.c, output.c and run_sixs.c) and expunge all 
the old school // commented code ( as opposed to the /* comment */ style) since 
my compiler would not accept these // commented lines.

I look forward to your response. Thanks for your assistance.

When running the updatencep script, the script is unable to read the NCEP 
Reanalysis products.  This just occurred recently.

What steps will reproduce the problem?
1. Download a GeoTIFF Landsat product (including MTL.txt)
2. Try to run lndpm from an external directory: 
 lndpm /projects/oa/ledaps/tutorial/input_data/LT50090602007277CHM01/LT50090602007277CHM01_MTL.txt 

Running lndpm ...
TIFFOpen: LT50090602007277CHM01_B1.TIF: Cannot open.
FAIL lndpm.c:2294 : Can't open base GEOTIFF file LT50090602007277CHM01_B1.TIF

ls /projects/oa/ledaps/tutorial/input_data/LT50090602007277CHM01/
LT50090602007277CHM01.metadata.txt  LT50090602007277CHM01_MTL.txt
LT50090602007277CHM01_B1.TIF        LT50090602007277CHM01_VER.jpg
LT50090602007277CHM01_B2.TIF        LT50090602007277CHM01_VER.txt
LT50090602007277CHM01_B3.TIF        LogReport
LT50090602007277CHM01_B4.TIF        README.GTF
LT50090602007277CHM01_B5.TIF        lndcal.LT50090602007277CHM01.hdf.hdr
LT50090602007277CHM01_B6.TIF        lndcal.LT50090602007277CHM01.txt
LT50090602007277CHM01_B7.TIF        lndsr.LT50090602007277CHM01.hdf.hdr
LT50090602007277CHM01_GCP.txt       lndth.LT50090602007277CHM01.hdf.hdr

3. Move all the Landsat files to the ancillary folder:
cp /projects/oa/ledaps/tutorial/input_data/LT50090602007277CHM01/* $ANC_PATH
cd $ANC_PATH
lndpm LT50090602007277CHM01_MTL.txt 

Running lndpm ...
using DEM : ./CMGDEM.hdf
using TOMS : ./TOMS_2007277.hdf
using REANALYSIS : ./REANALYSIS_2007277.hdf
lndpm complete.


ALSO:
I had to move all .hdf files out of their folders into the top level of the 
$ANC_PATH -- lndpm didn't do any recursive searching or use your pre-determined 
folder structure.

What is the expected output? What do you see instead?


What version of the product are you using? On what operating system?
1.3.1.

Please provide any additional information below.
While dropping all the HDF files in a single directory is ok with me (although 
perhaps it would make sense to at least have each basic dataset in its own 
folder, e.g. $ANC_PATH/ozone for the ozone data), have the Landsat data in a 
separate folder is key since the ancillary files, on my system, are in a shared 
folder (e.g. /usr/etc/ledaps_Anc) whereas a given user should be able to 
download their own Landsat file and run the algorithm without needing to modify 
the root dataset.