e.g. ArNEST

Following the split of sbncode/sbnana I can't build sbncode on its own.

I'm getting a few errors like this:

[57/127] Building CXX object sbncode/sbncode/CAFMaker/CMakeFiles/CAFMaker.dir/FillFlashMatch.cxx.o
FAILED: sbncode/sbncode/CAFMaker/CMakeFiles/CAFMaker.dir/FillFlashMatch.cxx.o
/cvmfs/larsoft.opensciencegrid.org/products/gcc/v8_2_0/Linux64bit+3.10-2.17/bin/g++  -DCAFMaker_EXPORTS -DNDEBUG -I/sbnd/app/users/icaza/work/sbncode/srcs -I/sbnd/app/users/icaza/work/sbncode/srcs/sbncode -Isbncode -I/cvmfs/larsoft.opensciencegrid.org/products/cppunit/v1_14_0/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/clhep/v2_4_1_2/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/cetlib_except/v1_04_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/sqlite/v3_26_00_00/Linux64bit+3.10-2.17/include -I/cvmfs/larsoft.opensciencegrid.org/products/tbb/v2019_3/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/hep_concurrency/v1_04_01/slf7.x86_64.e19.prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/cetlib/v3_10_00/slf7.x86_64.e19.prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/fhiclcpp/v4_11_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/messagefacility/v2_06_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/range/v3_0_4_0/include -I/cvmfs/larsoft.opensciencegrid.org/products/canvas/v3_09_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/root/v6_18_04d/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/larcoreobj/v09_01_00/include -I/cvmfs/larsoft.opensciencegrid.org/products/larcorealg/v09_00_02/include -I/cvmfs/larsoft.opensciencegrid.org/products/art/v3_05_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/canvas_root_io/v1_05_02/include -I/cvmfs/larsoft.opensciencegrid.org/products/art_root_io/v1_03_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/larcore/v09_01_00/include -I/cvmfs/larsoft.opensciencegrid.org/products/dk2nudata/v01_08_00g/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/nusimdata/v1_21_05/include -I/cvmfs/larsoft.opensciencegrid.org/products/lardataobj/v09_00_07/include -I/cvmfs/larsoft.opensciencegrid.org/products/lardataalg/v09_03_06/include -I/cvmfs/larsoft.opensciencegrid.org/products/postgresql/v9_6_11b/Linux64bit+3.10-2.17-p372/include -I/cvmfs/larsoft.opensciencegrid.org/products/fftw/v3_3_8a/Linux64bit+3.10-2.17-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/lardata/v09_01_09/include -I/cvmfs/larsoft.opensciencegrid.org/products/libwda/v2_28_0/Linux64bit+3.10-2.17/include -I/cvmfs/larsoft.opensciencegrid.org/products/larevt/v09_01_06/include -I/cvmfs/larsoft.opensciencegrid.org/products/marley/v1_1_1e/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/nutools/v3_07_05/include -I/cvmfs/larsoft.opensciencegrid.org/products/xerces_c/v3_2_2/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/nug4/v1_05_04/include -I/cvmfs/larsoft.opensciencegrid.org/products/genie/v3_00_06i/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/dk2nugenie/v01_08_00l/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/nugen/v1_10_12/include -I/cvmfs/larsoft.opensciencegrid.org/products/nurandom/v1_04_05/include -I/cvmfs/larsoft.opensciencegrid.org/products/log4cpp/v1_1_3b/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/larsim/v09_08_00/include -I/cvmfs/larsoft.opensciencegrid.org/products/eigen/v3_3_5/include/eigen3 -I/cvmfs/larsoft.opensciencegrid.org/products/larreco/v09_04_10/include -I/cvmfs/larsoft.opensciencegrid.org/products/nuevdb/v1_02_09/include -I/cvmfs/larsoft.opensciencegrid.org/products/lareventdisplay/v09_00_20/include -I/cvmfs/larsoft.opensciencegrid.org/products/larexamples/v09_00_19/include -I/cvmfs/larsoft.opensciencegrid.org/products/larana/v09_00_20/include -I/cvmfs/larsoft.opensciencegrid.org/products/protobuf/v3_11_2a/Linux64bit+3.10-2.17-e19/include -I/cvmfs/larsoft.opensciencegrid.org/products/tensorflow/v1_12_0c/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/trtis_clients/v19_11b/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/larrecodnn/v09_04_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/pandora/v03_16_00/Linux64bit+3.10-2.17-e19-prof/include -I/cvmfs/larsoft.opensciencegrid.org/products/larpandoracontent/v03_22_04/include -I/cvmfs/larsoft.opensciencegrid.org/products/larpandora/v09_04_01/include -I/cvmfs/larsoft.opensciencegrid.org/products/TRACE/v3_16_00/include -I/cvmfs/larsoft.opensciencegrid.org/products/artdaq_core/v3_06_03/include -I/cvmfs/sbnd.opensciencegrid.org/products/sbnd/sbnobj/v09_10_00/include -I/sbnd/app/users/icaza/work/sbncode/srcs/sbncode/sbncode/CAFMaker -I/cvmfs/larsoft.opensciencegrid.org/products/ifdhc/v2_5_7/Linux64bit+3.10-2.17-e19-p372-prof/inc -I/cvmfs/larsoft.opensciencegrid.org/products/ifdh_art/v2_10_07/slf7.x86_64.e19.prof/inc -isystem /cvmfs/larsoft.opensciencegrid.org/products/boost/v1_70_0/Linux64bit+3.10-2.17-e19-prof/include -g -gdwarf-4 -O3 -fno-omit-frame-pointer -std=c++98 -Werror -Wno-error=deprecated-declarations -pedantic -Wno-unused-local-typedefs -Wno-undefined-var-template -I /include/c++/v1 -std=c++17 -Wall -Werror=return-type -fPIC -MD -MT sbncode/sbncode/CAFMaker/CMakeFiles/CAFMaker.dir/FillFlashMatch.cxx.o -MF sbncode/sbncode/CAFMaker/CMakeFiles/CAFMaker.dir/FillFlashMatch.cxx.o.d -o sbncode/sbncode/CAFMaker/CMakeFiles/CAFMaker.dir/FillFlashMatch.cxx.o -c /sbnd/app/users/icaza/work/sbncode/srcs/sbncode/sbncode/CAFMaker/FillFlashMatch.cxx
In file included from /sbnd/app/users/icaza/work/sbncode/srcs/sbncode/sbncode/CAFMaker/FillFlashMatch.cxx:7:
/sbnd/app/users/icaza/work/sbncode/srcs/sbncode/sbncode/CAFMaker/FillFlashMatch.h:14:10: fatal error: sbnanaobj/StandardRecord/StandardRecord.h: No such file or directory
 #include "sbnanaobj/StandardRecord/StandardRecord.h"
          ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.

where crucially the path to StandardRecord.h is not included.

I've noticed that the units of energy whenever dealing with a truth quantity are in GeV, whilst reconstructed quantities; and actually maybe only rec.reco.slc.shw are in MeV.

Not a deal breaker, but it's a minor nuisance that seems unnecessary to me.

Make a nicely documented site for listing standard fcl files. Needs to have an avenue for being updated as needed, with as little maintenance that we can image.

Could also include "common configuration" set.

CI builds using ninja were failing, and could be fixed by removing wildcards from the cmake files (see branch feature/wketchum_NoWildcard). For now we'll have CI not use ninja, but in the future would be good to arrange things such that ninja can work (while also including everything properly).

Thanks to @vitodb and @cjbackhouse for talking through this.

Pending merge of LArSoft/larsim#44.

When a pull-request is issued trigger a CI validation and have the results of that validation be reported back on the pull-request to enable a rapid validation of code changes

For use of both data in ICARUS, and cosmic simulation in SBND.

Need to do refactoring maintenance on the CRT reco pieces that had to be overlooked for 2020A production. Includes getting CRT matching into the CAFs.

Go through the files from the recent production, identify the largest data-products, determine methods to reduce them or eliminate them if they are not needed downstream

I got

%MSG-s ArtException:  FlashPredict:fmatchREDO@Construction  26-Mar-2021 08:33:46 CDT ModuleConstruction
cet::exception caught in art
---- FatalRootError BEGIN
  Fatal Root Error: TFile::TFile
  file name is not specified
  ROOT severity: 3000
---- FatalRootError END

which didn't give me very much to go on. It turns out the problem is I hadn't setup the sbnd_data product. It would be good to improve the message to something like

"the_flashmatch_file.root" not found in $FW_SEARCH_PATH

plus maybe some hint that this is likely about sbnd_data

Need an sbn object repository for sbn-specific objects to be added. sbncode should depend on it.

A repository sbnobj is needed hosting data products.
This issue is for the creation of the empty GIT/cetbuildtools based repository.

I think the dependencies should be either on lardataobj or with the same as nusimdata.

Both experiments depend on GENIE cross section, and there is value in having them share the same version of those.
Technically, the dependency can be moved into sbncode.

A continuous integration system might be useful for sbncode itself.

We need to figure out what would be the content of the integration tests, which may be at different levels, e.g.:

• no actual test, but just a way to make sure the code still compiles
• minimal unit tests (e.g. for data products and libraries)
• actual tests

Contact people for the infrastructure are @vitodb (SBND) and Vladimir Podstavkov.
Deciding on the content is to us.

Some CRT data products are being moved from experiment code to sbncode, with change in namespace.
"Legacy` versions are available as well.
We need to verify that this is sufficient to be able to correctly read the old data files.

Layout step by step our expectation of what is needed for a pull-request to be approved and merged into our code

The PR for the LArContent fix is here: PandoraPFA/LArContent#146

The fix should be going into LArSoft next week, pending some other urgent pandora improvements that have to take precedence.

While testing SBND CI workflow using sbndcode from CVMFS, I found out that sbncode is missing the c7 build.
In the updated CI workflow the CI will build sbncode for c7 itself, so this is no more an issue for the CI, but users could still require it.

to go along with 2D drift simulation

Develop utilities for doing detector variations/systematic evaluation

sbncode also depends on uboonecode.
The feasibility and consequences of removing that dependency should be investigated.

Get CAF files as part of standard production workflows.

CI build error log here

https://dbweb5.fnal.gov:8443/LarCI/app/ns:sbnd/storage/docs/2020/10/20/buildSBND%23lodHpSd.log

Finalize evaluation of semi-analytical or library-based optical simulation for use in 2021A production

Building icaruscode v09_15_00 I encountered the following CMake error:

CMake Error at /cvmfs/larsoft.opensciencegrid.org/products/cetbuildtools/v7_17_01/Modules/CheckUpsVersion.cmake:142 (message):
  Bad Version: sbnobj 1.70.0 is less than minimum required version 1.50
Call Stack (most recent call first):
  /cvmfs/larsoft.opensciencegrid.org/products/cetbuildtools/v7_17_01/Modules/FindUpsPackage.cmake:161 (_check_version)
  /cvmfs/sbn.opensciencegrid.org/products/sbn/sbncode/v09_15_00/slf7.x86_64.e19.prof/lib/sbncode/cmake/sbncodeConfig.cmake:192 (fi
nd_ups_product)
  /cvmfs/larsoft.opensciencegrid.org/products/cetbuildtools/v7_17_01/Modules/FindUpsPackage.cmake:38 (find_package)
  /cvmfs/larsoft.opensciencegrid.org/products/cetbuildtools/v7_17_01/Modules/FindUpsPackage.cmake:157 (_use_find_package)
  icaruscode/CMakeLists.txt:63 (find_ups_product)

I can't fully understand what's happening, but I drop here a point: as far as I know, specifying a minimum version in find_ups_products() macro (like in SBNSoftware/sbncode:CMakeLists.txt) is not necessary; now I find it actually deleterious.
I have solved it locally by checking out sbncode in my working area and removing version requirements.

P.S. That CMakeLists.txt appears to also be missing the inclusion of $ENV{ART_DIR}/Modules.

Today it is v08_57_00.

Perform a census of each module run during production to determine its resource usage to identify places where we can improve memory usage and CPU time usage

When one tries to build only sbncode the build crashes here:

/sbnd/app/users/jaz8600/ChrisTest/srcs/sbncode/sbncode/LArRecoProducer/CRTHitConverter_module.cc:22:10: fatal error: sbnobj/Common/CRT/CRTHit.hh: No such file or directory #include "sbnobj/Common/CRT/CRTHit.hh" ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~ compilation terminated. make[2]: *** [sbncode/sbncode/LArRecoProducer/CMakeFiles/sbncode_LArRecoProducer_CRTHitConverter_module.dir/CRTHitConverter_module.cc.o] Error 1 make[1]: *** [sbncode/sbncode/LArRecoProducer/CMakeFiles/sbncode_LArRecoProducer_CRTHitConverter_module.dir/all] Error 2 make[1]: *** Waiting for unfinished jobs....

this is overcome by also checking out sbnobj, but it would be nice if that wasn't necessary.

What is the motivation for sbnanalysis to need an independent build step to the rest of sbncode? Can it just be included in the main mrb build?

@cjbackhouse brought this up in a slack channel: we don't have an obvious place to put common SBN software productions, like a SBN CVMFS area? For the moment, can request to publish all areas, but we should define the proper area(s) for this as part of the repository restructuring.

Build a computing model that enables us to forecast production timelines and resources requirements to meet our physics goals

The example scripts in anademo do not run. I get the following error:

/sbnd/app/users/etyley/cafs/localProducts_larsoft_v09_09_01_e19_prof/sbncode/v09_09_01/include//sbncode/CAFAna/Core/Var.h:12:10: fatal error: 'StandardRecord/Proxy/FwdDeclare.h' file not found
#include "StandardRecord/Proxy/FwdDeclare.h"
         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error in <ACLiC>: Dictionary generation failed!

Develop/verify ability to do systematic uncertainty evaluations with CAF output files

As part of the "normal hierarchy" work on sbncode, I notice that sbncode/sbnanalysis depends on various icaruscode and sbndcode paths.

What is the plan for all of this code?

This is a useful tool. Fernanda has a working version that should be cleaned up and committed.

setup gdb and setup valgrind get you much more modern versions of these tools than you get from the base OS, versions that actually work. We should give these to users by default, but it's not quite clear to me where to put this, since they can't strictly be said to be dependencies of any of our packages.

Integrate reductions identified in issue #72 into the standard workflows

In commit 9659ce2, an explicit call to abort was added into CAFMaker if there are no weights present.

This is a mistake -- CAFMaker is intended to fail gracefully if any data products are not present in the input file.

Test and make configurations for reducing the overall file sizes.

Determine the standard set of workflows for both experiments that are supported, document which fcls are run to achieve this, and layout instructions for how to modify these workflows

The duplication of the trk and shw branches within the slc branch (added here) is incomplete as the shower energy and dEdx are not filled whilst they are filled in the original rec branch. The daughters branch is filled in neither the trk or shower branches.

Current code is a bit clunky with a if(SBND) statement. Think it's better to implement the MapAlg as a tool, and have ICARUS/SBND define ones to use.

(Ideally this should be coming from larsoft geometry somehow? Not sure why this is really needed here...but can figure that out later.)

We have a little bit of a mess with the flatcafs that needs to be straightened out.

Due to my oversight, convert_to_flat wasn't propagating the POT information until this morning, so Gray will have to remake the MCP2020A flatcaf file. Sorry!

While looking into doing that myself, I ran into a bit of a complication with reco.shw.energy. Due to the nested-vectors fiasco, this array was inaccessible straight from ROOT. One of the motivations for flatcafs is to get access to it, and things like it.

As a temporary workaround, Diana added 3 non-vector fields containing this information, and likewise with dEdx, and commented out the vector.

Unfortunately, this means that code running from develop expects these fields in Gray's file, which predates them. Even though they would be able to read just fine from the vector in the flatcaf. Similarly, convert_to_flat in development is going to choke on the absence of these fields. It would have to be run in whatever revision was used to make the original CAF. But the resulting file will still be hard to read from development.

I think the best solution is to reinstate the proper array, and also leave the individual variables in place, duplicating the information, alongside for now, and then regenerate CAFs and FlatCAFs. That gives all versions the best chance to access what they're expecting. If/when we wind up in a world where the flatcafs are universal then we can safely remove the duplicates.

NOvA originated some scripts, which are now also available in duneutil, which I think are very useful. Perhaps SBN already has equivalents, in which case please point me to them, but if not I think we should look into adopting these.

setup_fnal_security - takes care of getting all the various tickets you need to work with the grid, xrootd, etc. This is all very confusing, and the advice keeps changing, so having one script to do everything, that someone else maintains, is very useful.

samweb2xrootd - This is a combination of pnfs2xrootd (which SBN already has a version of as pnfsToXRootD from the larsoft product itself) and samweb locate-file. The upshot is you give it the filename of something in SAM, and it gives back an xrootd URL, allowing you to open the file in one step.

latest - finds the most recent version of a ups product and prints it in a form appropriate for passing to ups setup

cache_state.py - reports whether a file, or what fraction of a dataset, is available in cache vs only on tape

ack - this is a better (more code-focused) version of grep. https://beyondgrep.com/ This one is so good I have gone to the effort of installing it in my home directory (it's just a perl script).

There was talk of moving these to some more central larsoft location, but now that dune has its itch scratched I imagine there will be no further movement unless we start making requests.

Testing automating add to project ...