noaa-gfdl / fms Goto Github PK
View Code? Open in Web Editor NEWGFDL's Flexible Modeling System
License: Other
GFDL's Flexible Modeling System
License: Other
Default behavior of slurm has impacted the ability of the affinity handling code to properly place MPI-tasks and OpenMP threads. Updates are needed to work within the slurm environment that assigns MPI-tasks to groups of processors using cpusets.
Associated source file will be moved to its own directory within FMS and a function to handle affinity will be added. This will necessitate a change in NOAA-GFDL/FMScoupler (see FMSCoupler issue #7)
When setting do_diag_field_log == .true., the model will print a log or lines:
Module|Field|Long Name|Units|Number of Axis|Time Axis|Missing Value|Min Value|Max Value|AXES LIST
and will then fail with the error:
FATAL from PE 486: diag_axis_mod::get_diag_axis_name: Illegal value for axis_id used (value 0).
For fun I packaged up an FMS dist tarball and tried to build it in theia.
Here's how I tried to build:
CC=mpiicc FC=mpiifort CPPFLAGS='-I/home/Edward.Hartnett/local/netcdf-4.6.3-development_mpiicc/include -I/home/Edward.Hartnett/local/netcdf-fortran-4.4.5_c_4.6.3-development_mpiifort/include' LDFLAGS='-L/home/Edward.Hartnett/local/netcdf-4.6.3-development_mpiicc/lib -L/home/Edward.Hartnett/local/netcdf-fortran-4.4.5_c_4.6.3-development_mpiifort/lib' ./configure
Here's how it failed:
mpiifort -DPACKAGE_NAME=\"FMS\" -DPACKAGE_TARNAME=\"fms\" -DPACKAGE_VERSION=\"2.0-development\" -DPACKAGE_STRING=\"FMS\\
2.0-development\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"fms\" -DVERSION=\"2.0-development\" -DSTDC_H\
EADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_\
H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -Duse_netCDF=1 -Dus\
e_libMPI=1 -I. -I../include -I../mpp/include -I/home/Edward.Hartnett/local/netcdf-4.6.3-development_mpiicc/include -I/\
home/Edward.Hartnett/local/netcdf-fortran-4.4.5_c_4.6.3-development_mpiifort/include -g -c -o mpp.o mpp.F90
fms_platform.h(114): #warning: macro redefined: QUAD_KIND
mpp_comm_mpi.inc(1386): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1387): #warning: macro redefined: MPI_TYPE_
mpp_comm_mpi.inc(1390): #warning: macro redefined: MPP_TYPE_CREATE_
mpp_comm_mpi.inc(1391): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1392): #warning: macro redefined: MPI_TYPE_
mpp_comm_mpi.inc(1395): #warning: macro redefined: MPP_TYPE_CREATE_
mpp_comm_mpi.inc(1396): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1397): #warning: macro redefined: MPI_TYPE_
mpp_comm_mpi.inc(1400): #warning: macro redefined: MPP_TYPE_CREATE_
mpp_comm_mpi.inc(1401): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1402): #warning: macro redefined: MPI_TYPE_
mpp_comm_mpi.inc(1405): #warning: macro redefined: MPP_TYPE_CREATE_
mpp_comm_mpi.inc(1406): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1407): #warning: macro redefined: MPI_TYPE_
mpp_comm_mpi.inc(1410): #warning: macro redefined: MPP_TYPE_CREATE_
mpp_comm_mpi.inc(1411): #warning: macro redefined: MPP_TYPE_
mpp_comm_mpi.inc(1412): #warning: macro redefined: MPI_TYPE_
Using 8-byte addressing
Using pure routines.
Using allocatable derived type array members.
Using cray pointers.
../mpp/include/mpp_comm_mpi.inc(258): error #6285: There is no matching specific subroutine for this generic subroutine\
call. [MPP_MIN]
tmin = t; call mpp_min(tmin)
-----------------------^
../mpp/include/mpp_comm_mpi.inc(259): error #6285: There is no matching specific subroutine for this generic subroutine\
call. [MPP_MAX]
tmax = t; call mpp_max(tmax)
-----------------------^
../mpp/include/mpp_comm_mpi.inc(260): error #6285: There is no matching specific subroutine for this generic subroutine\
call. [MPP_SUM]
tavg = t; call mpp_sum(tavg); tavg = tavg/mpp_npes()
-----------------------^
It is only a warning but nevertheless I think it is bad practice to alias arguments with conflicting intents. The compiler could legitimately do exactly the wrong thing and overwrite file_out before reading file_in.
../../../../../MOM6-examples/src/FMS/fms/fms_io.F90:7385.33:
call get_mosaic_tile_file(actual_file, actual_file, is_no_domain, domain, tile_count)
1
Warning: Same actual argument associated with INTENT(IN) argument 'file_in' and INTENT(OUT) argument 'file_out' at (1)
@underwoo , @bensonr, @Zhi-Liang and other FMS code leaders, I suggest that you do a versioned release of this package immediately, and then do them periodically thereafter.
I understand that you also have a larger versioning scheme, which applies to your entire model stack, in which you tag every repo involved to mark the exact software used in a particular model build. This meets the versioning needs of the users of that model, but not of other FMS users (like me).
What I am suggesting is that, in addition to that, you release versions of FMS, so that external users can just install a version of FMS and use it until its next release.
If you do a versioned release, the current master and all branches are all still available for anyone using them in their workflow. The ability to create an over-all tag for all software is still there, and all your existing workflows still work.
But new workflows, in accordance with general practice, can be followed. Users and researchers will have a canonical version number for the FMS package. Different FMS users will be able to understand what version of the software they are using, and be able to manage upgrades properly.
For example, when I add PIO capability to FMS, I will be testing it with various versions of netcdf-c, netcdf-fortran, parallel-netcdf, and HDF5. This is so I can say: "version 2.0 of FMS has been tested with netcdf-c 4.6.2, 4.6.3, etc." Without this information, how do sysadmins know what packages they need to upgrade to support your software?
Providing a well-understood versioned release is part of having a library that others are relying upon. It's necessary so we all can manage our dependencies.
To do a versioned release of FMS, there are a few quick steps that must be followed. It's about a 5-minute process to generate a versioned, tested, portable, and stand-alone tarball for the FMS package.
Please let me know your thinking, and I can guide you through the process the first time. In particular, there are some shared library version-info numbers that must be updated properly, so that shared libraries will work with this code correctly.
We have the preprocessor macro use_netCDF, which we are always setting.
Does FMS build without netCDF? Or is netcdf always required?
Here's a note that I just put into monin_obukhov/Makefile.am:
# Note that the name of the mod is different from the name of the F90
# code for monin_obukhov_kernel.F90. Also note that the mod file for
# this one does not have "_mod" in the name.
So this module is not named after its code, and also does not have the "_mod" in the name of the mod file (which I think is fine, but does not match the rest of the code base).
Axes%num_attributes is not initialised. Since it can take any value this can cause problems such as array bounds overruns.
With your mkmf build system, there does not seem to be a "make install" target. In the autotools system, of course there is. ;-)
Right now "make install" installs only the libFMS library (static and shared). It is installed in the lib directory under whatever was provided to the configure script prefix argument. For example:
./configure --prefix=/home/ed/test_fms && make install
will build the FMS library and install it in /home/ed/test_fms/lib, creating directories as needed.
With netcdf-fortran and some other fortran packages I work with, we also install the .mod files. With netcdf-fortran we also install a netcdf.inc file, and some man pages.
So the question is, do we know of anything we want to install other than libFMS?
Looking at the mpp/affinity.c
code, the get_cpu_affinity
function has two return lines.
if (last_cpu != -1) {return (first_cpu);}
return (last_cpu == -1) ? first_cpu : -1;
However, when looking at all the possible ways the two if
statements can be evaluated, there is only one true return value: first_cpu
. This should be reduced to only one return statement.
Further evaluation shows the function sets a default value for first_cpu = -1
, and this should be safe.
fms_platform.h is included a lot by users of FMS. Due to some ifdefs, it generates a warning every time it is used. This can be easily fixed.
/usr/local/fms/include/fms_platform.h:114:0:
#define QUAD_KIND DOUBLE_KIND
Warning: "QUAD_KIND" redefined
/usr/local/fms/include/fms_platform.h:39:0:
#define QUAD_KIND 16
note: this is the location of the previous definition
I used MOM6 tagged "om4/v1.0.1", and dumped regional oceanic diagnostics in Carribean Windward Passage. The variables: "vmo" (Ocean Mass Y Transport) looked unreasonable when I used certain ocean layout, for example,
ocn ranks="4671" threads="1" layout = "90,72" io_layout = "1,4" mask_table="mask_table.1809.90x72".
The untarred history file is
gfdl:/archive/oar.gfdl.cmip6/CM4/warsaw_201710_om4_v1.0.1/CM4_piControl_C/gfdl.ncrc4-intel16-prod-openmp/history/tmp_650/06500101.ocean_Windward_Passage.nc
The stdout is gfdl:/archive/oar.gfdl.cmip6/CM4/warsaw_201710_om4_v1.0.1/CM4_piControl_C/gfdl.ncrc4-intel16-prod-openmp/ascii/06500101.ascii_out.tar
Could you please take a look? Please let me know if any other info. is needed.
Thanks.
Some hiccups on our system's wrapper scripts revealed that FMS models are still doing #include <mpif.h>
to import the MPI symbols, even though the standard recommends the module (use mpi
).
Currently only the sgi_mipspro
flag enables use mpi
. The default is to use the CPP #include
statement.
To be annoyingly pedantic (sorry!), here are some quotes from the MPI 3.1 standard.
p605:
INCLUDE ’mpif.h’: This method is described in Section 17.1.4. The use of the include file
mpif.h
is strongly discouraged starting with MPI-3.0, because this method neither guarantees compile-time argument checking nor provides sufficient techniques to solve the optimization problems with nonblocking calls, and is therefore inconsistent with the Fortran standard. It exists only for backwards compatibility with legacy MPI applications.
p611:
The use of the
mpif.h
include file is strongly discouraged and may be deprecated in a future
version of MPI.
The comments above refer to the Fortran include statement (include 'mpif.h'
); the standard doesn't even consider using the CPP include, which is probably even more strongly discouraged.
Rather than send a patch, I figured that you guys might be more aware of any odd platforms that still require this and would be in a better position to make the change.
15 years ago, when I put together the netcdf-fortran build system, there were some fortran compilers that would upper-case mod file names.
In other words, where gfortran produces fms_mod.mod, some compilers would yield FMS_MOD.mod.
This was annoying but there is a way to handle it in the build system. However I would like to avoid it if I can.
I don't know if any modern fortran compilers still do this. If you try to build and see an error from this, let me know and I will put the goop into the autotools system that will handle it.
The openMPI library used in the trusty travis-ci image does not have mpi_comm_create_group
, which was added to libFMS in the latest release. The .travis.yml
file needs to be updated to either use mpich2, or to a newer (xenial) image.
Is the code expected to compile without MPI?
When I try a gfortran build, without use libMPI defined, then the build fails for me in the mpp directory like this:
libtool: compile: gfortran -DPACKAGE_NAME=\"FMS\" -DPACKAGE_TARNAME=\"fms\" -DPACKAGE_VERSION=\"2.0-development\" "-DPACKAGE_STRING=\"FMS 2.0-development\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"fms\" -DVERSION=\"2.0-development\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -Duse_netCDF=1 -I. -I../include -Iinclude -I/usr/local/netcdf-c-4.6.2/include -I/usr/local/netcdf-fortran-4.4.5_c_4.6.2/include -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check -c mpp.F90 -fPIC -o .libs/mpp.o
../include/fms_platform.h:114:0:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
include/mpp_comm_nocomm.inc:1113:42:
call mpp_error(NOTE, 'MPP_TYPE_FREE: ' &
1
Error: Syntax error in argument list at (1)
include/mpp_alltoall_nocomm.h:63:30:
rbuf, rsize, rdispl, rtype, pelist)
1
Error: Array ‘rbuf’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:63:45:
rbuf, rsize, rdispl, rtype, pelist)
1
Error: Array ‘rdispl’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:63:37:
rbuf, rsize, rdispl, rtype, pelist)
1
Error: Array ‘rsize’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:63:52:
rbuf, rsize, rdispl, rtype, pelist)
1
Error: Array ‘rtype’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:62:30:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Array ‘sbuf’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:62:45:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Array ‘sdispl’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:62:37:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Array ‘ssize’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:62:52:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Array ‘stype’ at (1) cannot have a deferred shape
include/mpp_alltoall_nocomm.h:63:30:
rbuf, rsize, rdispl, rtype, pelist)
1
I am about 90% of the way to compiling the FMS and MOM6 libraries with gcc 8.2 in debug mode on my personal machine running MacOS Mojave (v 10.14.2). Following the steps on the MOM6 wiki for compiling the ocean-only configuration here, I am able to compile FMS and MOM6, but get the following error during the link stage:
Undefined symbols for architecture x86_64:
"_mpi_abort_", referenced from:
___mpp_mod_MOD_mpp_error_basic in mpp.o
"_mpi_allreduce_", referenced from:
___mpp_mod_MOD_mpp_sum_int4 in mpp.o
___mpp_mod_MOD_mpp_sum_int8 in mpp.o
___mpp_mod_MOD_mpp_sum_real8 in mpp.o
___mpp_mod_MOD_mpp_min_int4_1d in mpp.o
___mpp_mod_MOD_mpp_min_int4_0d in mpp.o
___mpp_mod_MOD_mpp_min_int8_1d in mpp.o
___mpp_mod_MOD_mpp_min_int8_0d in mpp.o
...
"_mpi_alltoall_", referenced from:
___mpp_mod_MOD_mpp_alltoall_logical8 in mpp.o
___mpp_mod_MOD_mpp_alltoall_logical4 in mpp.o
___mpp_mod_MOD_mpp_alltoall_real8 in mpp.o
___mpp_mod_MOD_mpp_alltoall_real4 in mpp.o
___mpp_mod_MOD_mpp_alltoall_int8 in mpp.o
___mpp_mod_MOD_mpp_alltoall_int4 in mpp.o
"_mpi_alltoallv_", referenced from:
___mpp_mod_MOD_mpp_alltoall_logical8_v in mpp.o
___mpp_mod_MOD_mpp_alltoall_logical4_v in mpp.o
___mpp_mod_MOD_mpp_alltoall_real8_v in mpp.o
___mpp_mod_MOD_mpp_alltoall_real4_v in mpp.o
___mpp_mod_MOD_mpp_alltoall_int8_v in mpp.o
___mpp_mod_MOD_mpp_alltoall_int4_v in mpp.o
"_mpi_alltoallw_", referenced from:
___mpp_mod_MOD_mpp_alltoall_logical8_w in mpp.o
___mpp_mod_MOD_mpp_alltoall_logical4_w in mpp.o
___mpp_mod_MOD_mpp_alltoall_real8_w in mpp.o
___mpp_mod_MOD_mpp_alltoall_real4_w in mpp.o
___mpp_mod_MOD_mpp_alltoall_int8_w in mpp.o
___mpp_mod_MOD_mpp_alltoall_int4_w in mpp.o
"_mpi_barrier_", referenced from:
___mpp_mod_MOD_mpp_sync in mpp.o
"_mpi_bcast_", referenced from:
___mpp_mod_MOD_mpp_broadcast_logical4 in mpp.o
___mpp_mod_MOD_mpp_broadcast_logical8 in mpp.o
___mpp_mod_MOD_mpp_broadcast_int4 in mpp.o
___mpp_mod_MOD_mpp_broadcast_int8 in mpp.o
___mpp_mod_MOD_mpp_broadcast_real4 in mpp.o
___mpp_mod_MOD_mpp_broadcast_real8 in mpp.o
___mpp_mod_MOD_mpp_broadcast_char in mpp.o
...
"_mpi_comm_create_group_", referenced from:
___mpp_mod_MOD_get_peset in mpp.o
"_mpi_comm_group_", referenced from:
___mpp_mod_MOD_mpp_init in mpp.o
"_mpi_comm_rank_", referenced from:
___mpp_mod_MOD_mpp_init in mpp.o
"_mpi_comm_size_", referenced from:
___mpp_mod_MOD_mpp_init in mpp.o
"_mpi_finalize_", referenced from:
___mpp_mod_MOD_mpp_exit in mpp.o
"_mpi_get_count_", referenced from:
___mpp_mod_MOD_mpp_transmit_logical4 in mpp.o
___mpp_mod_MOD_mpp_transmit_logical8 in mpp.o
___mpp_mod_MOD_mpp_transmit_int4 in mpp.o
___mpp_mod_MOD_mpp_transmit_int8 in mpp.o
___mpp_mod_MOD_mpp_transmit_real4 in mpp.o
___mpp_mod_MOD_mpp_transmit_real8 in mpp.o
___mpp_mod_MOD_mpp_sync_self in mpp.o
...
"_mpi_group_incl_", referenced from:
___mpp_mod_MOD_get_peset in mpp.o
"_mpi_init_", referenced from:
___mpp_mod_MOD_mpp_init in mpp.o
"_mpi_initialized_", referenced from:
___mpp_mod_MOD_mpp_init in mpp.o
"_mpi_irecv_", referenced from:
___mpp_mod_MOD_mpp_transmit_logical4 in mpp.o
___mpp_mod_MOD_mpp_transmit_logical8 in mpp.o
___mpp_mod_MOD_mpp_transmit_int4 in mpp.o
___mpp_mod_MOD_mpp_transmit_int8 in mpp.o
___mpp_mod_MOD_mpp_transmit_real4 in mpp.o
___mpp_mod_MOD_mpp_transmit_real8 in mpp.o
"_mpi_isend_", referenced from:
___mpp_mod_MOD_mpp_transmit_logical4 in mpp.o
___mpp_mod_MOD_mpp_transmit_logical8 in mpp.o
___mpp_mod_MOD_mpp_transmit_int4 in mpp.o
___mpp_mod_MOD_mpp_transmit_int8 in mpp.o
___mpp_mod_MOD_mpp_transmit_real4 in mpp.o
___mpp_mod_MOD_mpp_transmit_real8 in mpp.o
"_mpi_recv_", referenced from:
___mpp_mod_MOD_mpp_transmit_logical4 in mpp.o
___mpp_mod_MOD_mpp_transmit_logical8 in mpp.o
___mpp_mod_MOD_mpp_transmit_int4 in mpp.o
___mpp_mod_MOD_mpp_transmit_int8 in mpp.o
___mpp_mod_MOD_mpp_transmit_real4 in mpp.o
___mpp_mod_MOD_mpp_transmit_real8 in mpp.o
"_mpi_type_commit_", referenced from:
___mpp_mod_MOD_mpp_type_create_logical8 in mpp.o
___mpp_mod_MOD_mpp_type_create_logical4 in mpp.o
___mpp_mod_MOD_mpp_type_create_real8 in mpp.o
___mpp_mod_MOD_mpp_type_create_real4 in mpp.o
___mpp_mod_MOD_mpp_type_create_int8 in mpp.o
___mpp_mod_MOD_mpp_type_create_int4 in mpp.o
"_mpi_type_create_subarray_", referenced from:
___mpp_mod_MOD_mpp_type_create_logical8 in mpp.o
___mpp_mod_MOD_mpp_type_create_logical4 in mpp.o
___mpp_mod_MOD_mpp_type_create_real8 in mpp.o
___mpp_mod_MOD_mpp_type_create_real4 in mpp.o
___mpp_mod_MOD_mpp_type_create_int8 in mpp.o
___mpp_mod_MOD_mpp_type_create_int4 in mpp.o
"_mpi_type_free_", referenced from:
___mpp_mod_MOD_mpp_type_free in mpp.o
"_mpi_wait_", referenced from:
___mpp_mod_MOD_mpp_sync_self in mpp.o
"_mpi_wtick_", referenced from:
___mpp_mod_MOD_system_clock_mpi in mpp.o
"_mpi_wtime_", referenced from:
___mpp_mod_MOD_system_clock_mpi in mpp.o
"_sched_setaffinity", referenced from:
_set_cpu_affinity in affinity.o
ld: symbol(s) not found for architecture x86_64
collect2: error: ld returned 1 exit status
make: *** [MOM6] Error 1
Note that I had to replace linux-specific macros with MAC equivalents in mpp/affinity.c. Please see the file in my fork for additional details here.
I used the following script to configure the environment and build FMS+MOM6 :
#!/usr/bin.sh
# Set environment variables
export PKG_CONFIG_PATH="/opt/local/lib/pkgconfig"
export LD_LIBRARY_PATH="/usr/local/mpich-install/lib/"
buildDir="~/gfdl_model"
execDir="${buildDir}/gnu8_debug/exec"
makeTemplate="WKoD_gnu8_2.mk"
mkdir -p ${execDir}
cd ${execDir}
# create fms directory if it does not exist
if [ ! -d "fms" ]; then
mkdir "fms"
fi
cd "fms"
# remove the path_names file if it exists
pathFileList="path_names"
if [ -f "${pathFileList}" ]; then
rm -f ${pathFileList}
fi
#create new list_paths file for shared files
${buildDir}/mkmf/bin/list_paths -l ${buildDir}/MOM6_unmodified/src/shared
# create the make file for fms
${buildDir}/mkmf/bin/mkmf -t ${buildDir}/mkmf/templates/${makeTemplate} -o '-I/usr/local/include -I/usr/local/mpich-install/include' -p libfms.a -c '-Duse_libMPI -Duse_netCDF -DSPMD' path_names
# run Make
make NETCDF=3 DEBUG=1 libfms.a -j 4 2>&1 | tee make.out
# Build MOM6
cd ${execDir}
# create fms directory if it does not exist
if [ ! -d "mom6" ]; then
mkdir "mom6"
fi
cd "mom6"
if [ -f "${pathFileList}" ]; then
rm -f ${pathFileList}
fi
#create new list_paths file for MOM6 solo_ocean configuration and the source code
${buildDir}/mkmf/bin/list_paths -l ${buildDir}/MOM6_unmodified/src/MOM6/{config_src/dynamic,config_src/solo_driver,src/{*,*/*}} ${buildDir}/MOM6_unmodified/src/shared/{mpp,mpp/include,include}
# create the makefile for MOM6
${buildDir}/mkmf/bin/mkmf -t ${buildDir}/mkmf/templates/${makeTemplate} -o '-I../fms -I/usr/local/mpich-install/include' -p MOM6 -l '-L../fms -lfms' -c '-Duse_libMPI -Duse_netCDF -DSPMD' path_names
# Run make
make NETCDF=3 DEBUG=1 MOM6 -j 4 2>&1 | tee make.out
Other details:
# Template for the GNU Compiler Collection on Linux systems
#
# Typical use with mkmf
# mkmf -t WKoD_gnu8.mk -c"-Duse_libMPI -Duse_netCDF" path_names /usr/local/include
############
# Commands Macros
FC = gfortran
CC = gcc
CXX = g++
LD = gfortran $(MAIN_PROGRAM)
#######################
# Build target macros
#
# Macros that modify compiler flags used in the build. Target
# macrose are usually set on the call to make:
#
# make REPRO=on NETCDF=3
#
# Most target macros are activated when their value is non-blank.
# Some have a single value that is checked. Others will use the
# value of the macro in the compile command.
DEBUG = # If non-blank, perform a debug build (Cannot be
# mixed with REPRO or TEST)
REPRO = # If non-blank, erform a build that guarentees
# reprodicuibilty from run to run. Cannot be used
# with DEBUG or TEST
TEST = # If non-blank, use the compiler options defined in
# the FFLAGS_TEST and CFLAGS_TEST macros. Cannot be
# use with REPRO or DEBUG
VERBOSE = # If non-blank, add additional verbosity compiler
# options
OPENMP = # If non-blank, compile with openmp enabled
NO_OVERRIDE_LIMITS = # If non-blank, do not use the -qoverride-limits
# compiler option. Default behavior is to compile
# with -qoverride-limits.
NETCDF = # If value is '3' and CPPDEFS contains
# '-Duse_netCDF', then the additional cpp macro
# '-Duse_LARGEFILE' is added to the CPPDEFS macro.
INCLUDES = # A list of -I Include directories to be added to the
# the compile command.
SSE = # The SSE options to be used to compile. If blank,
# than use the default SSE settings for the host.
# Current default is to use SSE2.
COVERAGE = # Add the code coverage compile options.
# Need to use at least GNU Make version 3.81
need := 3.81
ok := $(filter $(need),$(firstword $(sort $(MAKE_VERSION) $(need))))
ifneq ($(need),$(ok))
$(error Need at least make version $(need). Load module gmake/3.81)
endif
# REPRO, DEBUG and TEST need to be mutually exclusive of each other.
# Make sure the user hasn't supplied two at the same time
ifdef REPRO
ifneq ($(DEBUG),)
$(error Options REPRO and DEBUG cannot be used together)
else ifneq ($(TEST),)
$(error Options REPRO and TEST cannot be used together)
endif
else ifdef DEBUG
ifneq ($(TEST),)
$(error Options DEBUG and TEST cannot be used together)
endif
endif
# Get number of CPUs
#MAKEFLAGS += --jobs=$(shell grep '^processor' /proc/cpuinfo | wc -l)
MAKEFLAGS += --jobs=$(shell sysctl -n hw.ncpu)
# Macro for Fortran preprocessor
FPPFLAGS := $(INCLUDES)
# Fortran Compiler flags for the NetCDF library
FPPFLAGS += $(shell nf-config --fflags)
# Fortran Compiler flags for the MPICH MPI library
#FPPFLAGS += $(shell pkg-config --cflags-only-I mpich2)
FPPFLAGS += $(shell pkg-config --cflags-only-I mpich)
# Base set of Fortran compiler flags
FFLAGS := -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check
# Flags based on perforance target (production (OPT), reproduction (REPRO), or debug (DEBUG)
FFLAGS_OPT = -O3
FFLAGS_REPRO = -O2 -fbounds-check
FFLAGS_DEBUG = -O0 -g -W -fbounds-check -fbacktrace -ffpe-trap=invalid,zero,overflow
# Flags to add additional build options
FFLAGS_OPENMP = -fopenmp
FFLAGS_VERBOSE =
FFLAGS_COVERAGE =
# Macro for C preprocessor
CPPFLAGS = $(INCLUDES)
# C Compiler flags for the NetCDF library
CPPFLAGS += $(shell nc-config --cflags)
# C Compiler flags for the MPICH MPI library
# Output the effective list of Cflags from the -I group for all the requested packages
# there's also an 'openpa.pc' package in the same directory, so this flag must ignore it
#CPPFLAGS += $(shell pkg-config --cflags-only-I mpich2)
CPPFLAGS += $(shell pkg-config --cflags-only-I mpich)
# Base set of C compiler flags
CFLAGS := -D__IFC
# Flags based on perforance target (production (OPT), reproduction (REPRO), or debug (DEBUG)
CFLAGS_OPT = -O2
CFLAGS_REPRO = -O2
CFLAGS_DEBUG = -O0 -g
# Flags to add additional build options
CFLAGS_OPENMP = -fopenmp
CFLAGS_VERBOSE =
CFLAGS_COVERAGE =
# Optional Testing compile flags. Mutually exclusive from DEBUG, REPRO, and OPT
# *_TEST will match the production if no new option(s) is(are) to be tested.
FFLAGS_TEST = $(FFLAGS_OPT)
CFLAGS_TEST = $(CFLAGS_OPT)
# Linking flags
LDFLAGS := -L/usr/local/mpich-install/lib -lmpich
LDFLAGS_OPENMP := -fopenmp
LDFLAGS_VERBOSE :=
LDFLAGS_COVERAGE :=
# Start with a blank LIBS
LIBS =
# NetCDF library flags
LIBS += $(shell nf-config --flibs)
# MPICH MPI library flags
#LIBS += $(shell pkg-config --libs mpich2-f90)
LIBS += $(shell pkg-config --libs mpich)
# Get compile flags based on target macros.
ifdef REPRO
CFLAGS += $(CFLAGS_REPRO)
FFLAGS += $(FFLAGS_REPRO)
else ifdef DEBUG
CFLAGS += $(CFLAGS_DEBUG)
FFLAGS += $(FFLAGS_DEBUG)
else ifdef TEST
CFLAGS += $(CFLAGS_TEST)
FFLAGS += $(FFLAGS_TEST)
else
CFLAGS += $(CFLAGS_OPT)
FFLAGS += $(FFLAGS_OPT)
endif
ifdef OPENMP
CFLAGS += $(CFLAGS_OPENMP)
FFLAGS += $(FFLAGS_OPENMP)
LDFLAGS += $(LDFLAGS_OPENMP)
endif
ifdef SSE
CFLAGS += $(SSE)
FFLAGS += $(SSE)
endif
ifdef NO_OVERRIDE_LIMITS
FFLAGS += $(FFLAGS_OVERRIDE_LIMITS)
endif
ifdef VERBOSE
CFLAGS += $(CFLAGS_VERBOSE)
FFLAGS += $(FFLAGS_VERBOSE)
LDFLAGS += $(LDFLAGS_VERBOSE)
endif
ifeq ($(NETCDF),3)
# add the use_LARGEFILE cppdef
ifneq ($(findstring -Duse_netCDF,$(CPPDEFS)),)
CPPDEFS += -Duse_LARGEFILE
endif
endif
ifdef COVERAGE
ifdef BUILDROOT
PROF_DIR=-prof-dir=$(BUILDROOT)
endif
CFLAGS += $(CFLAGS_COVERAGE) $(PROF_DIR)
FFLAGS += $(FFLAGS_COVERAGE) $(PROF_DIR)
LDFLAGS += $(LDFLAGS_COVERAGE) $(PROF_DIR)
endif
LDFLAGS += $(LIBS)
This involves code on head of branch: coupler_type_reform_rwh (@Hallberg-NOAA)
Calls to data_override()
from CT_data_override_2d()
and CT_data_override_3d()
are updating values inside a dummy argument (var
) that has intent(in)
.
mpiifort on NCEP's wcoss platform reports:
coupler_types.F90(3740): error #6780: A dummy argument with the INTENT(IN) attribute shall not be defined nor become undefined. [VAR]
call data_override(gridname, var%bc(n)%field(m)%name, var%bc(n)%field(m)%values, Time)
Solution is the dummy argument var
should have intent(inout)
.
Relevent code:
Lines 3731 to 3757 in 3d8f68c
Issue first reported by @jiandewang
Programmers of Fortran packages usually assume that the compiler flags should be set by the build system, but this is a peculiarity of Fortran programmers. Most build systems leave the setting of flags entirely to the user, and this is the wisest course.
In many cases, flags are necessary because programmers are not writing portable Fortran, and this is the case with FMS as well. For example, FMS requires the -ffree-line-length-none is required because some of the code has lines longer than 132 characters without a continuation line. This could and should easily be avoided in code. The first step to a robust build system is that all programmers must write their code to be portable.
So we should not need to set flags, or, if we do, we should deal with the minimum possible number of them by coding for portability.
However, the current FMS build system does do a lot of flag setting, and I know that FMS users would miss this feature, so I have a set of changes ready to add to my branch which lets configure set Fortran flags (but also provides the capability to turn setting of flags off).
What would be nice would be to see a move away from dependence on fortran flags to get code to compile.
In PR #117 with new 32-bit compiler flags added to the .travis.yml file, the travis-CI test (https://travis-ci.org/NOAA-GFDL/FMS/jobs/571517159#L1153) fails after building the diag_integral code at line 1153 with the following error:
Fatal Error: Can't open module file ‘ensemble_manager_mod.mod’ for reading at (1): No such file or directory
This might be more of a feature request or a question, rather than an issue, but we noticed that when calling mpp_sum
for a vector, the length
argument is required. Something like this:
integer :: x
integer :: T(5,5,5)
call mpp_sum(x)
call mpp_sum(T, size(T))
But it seems that length
is almost always going to equal the size of the first argument. If that's correct (which is my question), would it be possible to have length
default to the variable length when unset?
Also, I wasn't able to think of a good case where one would actually want this length to be a different value. Maybe it is actually worth omitting the argument altogether?
We (NASA GMAO) would like to build GEOS using the NAG Fortran compiler to perform various aggressive compile-time and runtime checks provided by that compiler. Unfortunately, NAG does not support Cray pointers, so a workaround is needed for FMS.
Now that there is a firm integration path, I will happy to attempt the necessary changes and then issue a pull-request.
I'm surprised that this hasn't come up before, maybe there's something unusual about my setup. I'm doing processor masking on a 1 degree config and getting segfaults within FMS on this line of code:
https://github.com/NOAA-GFDL/FMS/blob/master/mpp/include/mpp_do_global_field.h#L146
There is no check in this code to see whether from_pe is NULL_PE (-3), i.e. it may have been masked when setting up the domains.
The diag_manager may not be using the correct domain when trying to find the array indices for the regional subsetting. This problem can be demonstrated using the Pacific_undercurrent umo
diagnostic in the OM4_025 test case. Essentially what seems to be happening is that the vector of longitudes at the corner (q) points (which are ostensibly placeholders and not representative of the tripolar grid) are incorrectly being used to find the point closest to the requested longitudinal point. Instead the diagnostic manager should use the full 2D lat/lon arrays to try to find the correct indices for the requested subregion.
A notebook demonstrating the case where this was found in more detail can be found https://gist.github.com/ashao/77a8cdc20eb18dd4adb7f388e816bd6a. To use for yourself, any ocean_static.nc
file from an OM4_025 run can be used, other model outputs can be found in a run that @nikizadehgfdl did:
/lustre/f2/scratch//oar.gfdl.ogrp-account/work/nnz/xanadu_mom6_devgfdl_20190412/OM4p25_IAF_BLING_csf_rerun_ShaoDiags_1x1m0d_1756x1o.o268462119/output.stager/lustre/f2/scratch/oar.gfdl.ogrp-account/nnz/xanadu_mom6_devgfdl_20190412/OM4p25_IAF_BLING_csf_rerun_ShaoDiags/ncrc4.intel16-prod/archive/1x1m0d_1756x1o/history/17080101.nc/
This means, for example, that one cannot run a two member ensemble on 2 cpus.
I am trying to get the mkmf build working. I have read all the documentation I could find, as well as the wiki, but no luck
Here's one of the problems:
mpicc -DNDEBUG -D__IFC -I/usr/local/netcdf-c-4.6.2_mpich-3.2/include -O2 -c -I./mosaic -I/usr/local/netcdf-c-4.6.2/include ./mosaic/read_mosaic.c
./mosaic/read_mosaic.c: In function ‘handle_netcdf_error’:
./mosaic/read_mosaic.c:38:43: warning: implicit declaration of function ‘nc_strerror’; did you mean ‘strerror’? [-Wimplicit-function-declaration]
sprintf( errmsg, "%s: %s", msg, (char *)nc_strerror(status) );
^~~~~~~~~~~
strerror
./mosaic/read_mosaic.c:38:35: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
sprintf( errmsg, "%s: %s", msg, (char *)nc_strerror(status) );
^
./mosaic/read_mosaic.c: In function ‘get_var_data’:
./mosaic/read_mosaic.c:232:3: error: unknown type name ‘nc_type’
nc_type vartype;
^~~~~~~
./mosaic/read_mosaic.c: In function ‘get_var_data_region’:
./mosaic/read_mosaic.c:292:3: error: unknown type name ‘nc_type’
nc_type vartype;
^~~~~~~
Makefile:151: recipe for target 'read_mosaic.o' failed
make: *** [read_mosaic.o] Error 1
make: *** Waiting for unfinished jobs...
Seems to not be finding netcdf.h, though the include directories are provided on the command line...
Currently, if I forget to set FC to mpifort, then the build system tries to use gfortran, and fails in drifters_comm.F90, because it can't find mpif.h.
This should be detected by configure, so the user gets a nice error message instead of a rude compile failure.
Hello all. I currently maintain the FMS instance inside of the GEOS model here at Goddard. While we have some plans to perhaps just use your Github as a submodule, for now it's sort of walled off and merged into our base.
But, until that point, I did have a request from my work with FMS. It would be to let all compilers be F2000 compliant. At present include/fms_platform.h
only allows Intel as such:
#if defined(__INTEL_COMPILER)
#define _F95
#define _F2000
#endif
when in truth, GCC and PGI are quite easily so and NAG should be as well (I haven't gotten NAG to build FMS yet because it is fundamentally impossible at the moment; I've emailed Raymond Menzel about that). So, if anything, the ALLOCATABLE component version of FMS should be default and the pointer one the option.
In my local copy, I've just removed the test, and I could easily do a pull request for that off a fork, but I didn't know if you had a better thought on how to handle it.
The version of gcc used by travisCI is old. It needs to be updated to handle newer fortran features.
The built libFMS library is not currently used when running make check
. Although the initial example test is just an example, it would be good if it did link in the libFMS library.
master is up-to-date with dev/master via #103 , so dev/master is ready to be deleted. The branch is protected, and the repository owner needs to remove it.
@menzel-gfdl
MOM6 already includes the .res. in the base restart file name. Since fms2_io::open_file routine calls a routine that automatically appends .res. to the file name when is_restart = .true.
, this means that MOM_restart has to incorporate several additional restart name string checks and additional variables to hold two versions of the restart file name: one with .res. and one without it.
It would be simpler if:
a) fms2_io::open_file requires the user to specify .res. in restart file names, or
b) fms2_io::restart_file_path_mangle subroutine also checks for whether .res. is already present in the file name passed to the routine if is_restart = .true.
, and only appends .res. if it is not.
A user requested adding the ensemble number to the end of the diag_integral.out filename to distinguish between ensembles.
As background, the reason I'm interested in this is that I've used this
file to figure out the radiative imbalance of the model in the past with
single member runs. Now that I have ensembles to play with I'd like to
be able to do the same thing for each ensemble member and see the spread.
I still get a single diag_integral.out file from my runs, but I have no
idea which member the data belongs to or whether the data gets written
by the first, last, or "last to get to the routine" member. I don't even
know if different timesteps are from the same member or an average of
the entire ensemble.
I see in various directories what seems to be test code. For example, in mpp there is a program test_mpp.F90.
When I build this and run it I don't get good results. Am I doing something wrong? How do you build this? How do you run the tests?
The mkmf system seems to lump these test files into the build of the library. Is that expected?
I had told you that make -j would work on this new build system, but that's not quite true yet, though I hope it will be soon.
Until further notice, don't use make -j, it will not work well.
The issue is that some of the mod files depend on other mod files being built first. I have to record these dependencies in the makefiles before parallel builds will work.
contrary to C code, APPLE is not defined in Fortran and used here:
Line 302 in db20e25
this requires the user to pass explicitly the cpp at compile time (e.g. ' -D__APPLE__')
The mpp_io_stack is used by write_record() as scratch space when writing a diagnostic with non-standard data packing. However the mpp_io_stack is increased to be the same size as the data even when no packing is used.
This results in large unnecessary memory usage. For example I recently had a case where the mpp_io_stack was being increased to 200 Mb even though it was not needed. In some cases this can cause a process to run out of memory and exit.
I suggest that we only increase the IO stack when it is needed.
How does building with openmp work?
Do we just provide the -openmp flag to the Fortran compiler?
Are there any pre-processor macros that need to be set?
In
https://github.com/NOAA-GFDL/MOM6-examples/blob/dev/gfdl/ice_ocean_SIS2/OM4_025/diag_table.MOM6
we have two sections (below) that sample MOM fields completely south of the equator. A handful of the southern latitudes have 0 for the latitude values, rather than -71 or the like. So the latitudes are not monotonic and that messes with the plotting, yielding spurious latitude values altogether for Ferret.
We (Adrcroft and Griffies) think there is a bug in diag manager.
Here are the sections
"ocean_model_z", "volcello", "volcello", "ocean_Drake_Passage", "all", "mean", "-70. -70. -71. -54.5 -1 -1",2
"ocean_model_z", "volcello", "volcello", "ocean_Agulhas_section", "all", "mean", "20. 20. -71.0 -34 -1 -1",2
An example output is
/archive/ogrp/CMIP6/OMIP/warsaw_201803_mom6_2018.04.06/OM4p25_IAF_fc0/gfdl.ncrc3-intel16-prod/pp/ocean_Drake_Passage/ts/monthly/20yr/ocean_Drake_Passage.198801-200712.umo.nc
The diag_manager_nml has a max_files options however once this is over 1024 it has no effect. This is because the maximum number of netcdf files is hard-coded in mpp/include/mpp_io_misc.inc:42 to 1024.
When doing MOM6 diagnostic testing we currently write each diagnostic to a separate file. We have reached this limit.
If we write many diags to the same file then we exceed MAX_FIELDS_PER_FILE which is hard-coded to 300 at diag_data.F90:78
I am getting lots of these errors with the mkmf build I am trying:
.```
/mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_comm_nocomm.inc:1113:42:
call mpp_error(NOTE, 'MPP_TYPE_FREE: ' &
1
Anyone know what is causing these?
I will put in a Fortran test in the autotools builds. The tests will simply return 0 (i.e. pass). But that's where you can insert some test code for your library.
To do this I will add a test directory, which will be compiled after libFMS, which is the directory where the combined libFMS library is built. The library must be built before the test is run, so the tests must be in their own directories. This also makes it easy to organize tests. (For example in the netcdf-c build we have test directories nctest, nc_test, nc_test4, dap_test, hdf4_test, etc.)
I will add a directory test_fms, and put a simple test in there. This test will be run by "make check" so it's a starting point for your continuous integration.
PIO (https://github.com/NCAR/ParallelIO) is a HPC I/O library that has been used in CESM at NCAR, and has recently become an output option for WRF.
PIO allows the same code to use netCDF classic, netCDF-4 parallel, netCDF-4 sequential/compressed, or pnetcdf for input/output, on a file-by-file basis. It also allows designation of an arbitrary number of I/O processors to handle all I/O. It will also allow use of new technologies being added to the netCDF C library, like zarr, and to the HDF5 library, including some other forms of cloud storage. Applications like FMS will not have to change their code to use these new features.
PIO users can easily switch from netcdf classic to pnetcdf to HDF5, to zarr - this could be a run-time decision, or changed at compile-time by changing the mode flag in nf_create/nf_open. Furthermore, PIO allows codes to scaled to thousands of processors or more, while still providing the performance of parallel I/O from a much more reasonable number of I/O processors.
Right now I am finishing a PIO-netCDF integration project. PIO will become available to existing netCDF code (C or Fortran), via the use of a mode flag in nc_create/nc_open. (Some code changes are required to set up the I/O system, define data decomposition, and do reads/writes of the distributed arrays.) The netCDF-PIO integration will be available for the next released versions of netcdf-c and PIO.
Using this integrated code, I can convert the FMS code without changing most netCDF calls. I see that you already have a data decomposition scheme and I will have to map this on to PIO's data decomposition functions.
In order to execute these changes, of course I need the mpp directory to be fully tested, so I will add tests to cover untested code, until we can all be confident that no changes will break existing code.
My plan is to submit these changes as a pull request to FMS. I hope that you will consider it for merging to master, once I have demonstrated its value.
I hope that PIO will allow users to easily try out a variety of different strategies, and use what works best in each case.
My work is taking me around to FMS again! ;-) I will need to do some work on the IO code for NOAA to try using PIO.
Before I start, I would love to get some automatic tests for the code in the mpp directory. I notice there are a lot of fortran test codes in the directory. Can we turn these on as tests?
The return value from (FMS ensemble interface) get_ensemble_size() is not complete until ensemble_pelist_setup() has been called. It will return bad values. A check for this would be good as it has caused problems in the past.
There's one module I can't build, oda_core_ecda.F90.
Where is xbt_adjust?
For now I just have this module commented out of the autotools build.
/bin/bash ../libtool --tag=FC --mode=compile mpifort -DPACKAGE_NAME=\"FMS\" -DPACKAGE_TARNAME=\"fms\" -DPACKAGE_VERSION=\"2.0-development\" -DPACKAGE_STRING=\"FMS\ 2.0-development\" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"fms\" -DVERSION=\"2.0-development\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../include -I../fms -I../mpp -I../time_manager -I../axis_utils -I../horiz_interp -I../constants -I/usr/local/netcdf-c-4.6.2_mpich-3.2/include -I/usr/local/netcdf-fortran-4.4.5_c_4.6.2_mpich-3.2/include -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check -c -o oda_core_ecda.lo oda_core_ecda.F90
libtool: compile: mpifort -DPACKAGE_NAME=\"FMS\" -DPACKAGE_TARNAME=\"fms\" -DPACKAGE_VERSION=\"2.0-development\" "-DPACKAGE_STRING=\"FMS 2.0-development\"" -DPACKAGE_BUGREPORT=\"\" -DPACKAGE_URL=\"\" -DPACKAGE=\"fms\" -DVERSION=\"2.0-development\" -DSTDC_HEADERS=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_SYS_STAT_H=1 -DHAVE_STDLIB_H=1 -DHAVE_STRING_H=1 -DHAVE_MEMORY_H=1 -DHAVE_STRINGS_H=1 -DHAVE_INTTYPES_H=1 -DHAVE_STDINT_H=1 -DHAVE_UNISTD_H=1 -DHAVE_DLFCN_H=1 -DLT_OBJDIR=\".libs/\" -Duse_netCDF=1 -Duse_libMPI=1 -I. -I../include -I../fms -I../mpp -I../time_manager -I../axis_utils -I../horiz_interp -I../constants -I/usr/local/netcdf-c-4.6.2_mpich-3.2/include -I/usr/local/netcdf-fortran-4.4.5_c_4.6.2_mpich-3.2/include -fcray-pointer -fdefault-double-8 -fdefault-real-8 -Waliasing -ffree-line-length-none -fno-range-check -c oda_core_ecda.F90 -fPIC -o .libs/oda_core_ecda.o
oda_core_ecda.F90:49:6:
use xbt_adjust, only : xbt_drop_rate_adjust
1
Fatal Error: Can't open module file ‘xbt_adjust.mod’ for reading at (1): No such file or directory
compilation terminated.
Makefile:415: recipe for target 'oda_core_ecda.lo' failed
make[1]: *** [oda_core_ecda.lo] Error 1
make[1]: Leaving directory '/home/ed/tmp/f3/oda_tools'
Makefile:399: recipe for target 'all-recursive' failed
make: *** [all-recursive] Error 1
I am getting answer changes when running with the new io in land because the river restart is not read correctly. I think it is because code is setting every rank as the root pe and is using the same starting and ending index in the read, so they are all reading the same part of the file.
In L322-324 of fms_netcdf_domain_io.F90, when opening the file the code gets the pelist from the io_domain
pelist_size = mpp_get_domain_npes(io_domain)
allocate(pelist(pelist_size))
call mpp_get_pelist(io_domain, pelist)
The pelist ends up been just a one element array with the current pe
Then, in L433-442 of netcdf_io.F90, the code assigns the root pe as the first element in the pelist to the fileobj
In domain_read.inc, in L133-140 (for a 2D read), it gets the starting index and length of each pe list
call mpp_get_compute_domains(io_domain, xbegin=pe_isc, xsize=pe_icsize, position=xpos)
call mpp_get_compute_domains(io_domain, ybegin=pe_jsc, ysize=pe_jcsize, position=ypos)
do i = 1, size(fileobj%pelist)
c(xdim_index) = pe_isc(i) - pe_isc(1) + 1
c(ydim_index) = pe_jsc(i) - pe_jsc(1) + 1
e(xdim_index) = pe_icsize(i)
e(ydim_index) = pe_jcsize(i)
c and e end up being the same for all pe, so I get the same data.
This is my work dir /lustre/f2/scratch/gfdl/Uriel.Ramirez/work/esm2mb-luh2/esm2mb-luh2_1x0m0d72h_64x1a.o201466261
In this case, there is only 1 tile.
The object field_manager/libfm_util.o
is not included in the libFMS library built by autotools.
It would be great if there was a standard build system, such that I could do ./configure && make.
Do you guys have any interest in such a system?
I am getting this error with my mkmf build:
./mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_alltoall_nocomm.h:62:25-25:
subroutine MPP_ALLTOALLW_(sbuf, ssize, sdispl, stype, &
1
Error: Procedure ‘mpp_alltoallw_’ at (1) is already defined at (2)
./mpp/include/mpp_comm_nocomm.inc:1113:42:
call mpp_error(NOTE, 'MPP_TYPE_FREE: ' &
1
Any idea what might be causing this?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.