key4hep / k4fwcore Goto Github PK

Core Components for the Gaudi-based Key4hep Framework

License: Apache License 2.0

CMake 5.59% C++ 70.35% Python 23.71% Shell 0.36%

k4fwcore's Introduction

k4FWCore (key4hep FrameWork Core)

k4FWCore is a Gaudi package that provides the PodioDataService, which allows to use podio-based event data models like EDM4hep in Gaudi workflows.

k4FWCore also provides the k4run script used to run Gaudi steering files.

Components

Basic I/O

k4DataSvc

Component wrapping the PodioDataService to handle PODIO types and collections.

PodioInput

Algorithm to read data from one or multiple input file(s) on disk.

PodioOutput

Algorithm to write data to an output file on disk.

k4run

$ k4run --help
usage: k4run [-h] [--dry-run] [-v] [-n NUM_EVENTS] [-l] [--gdb] [--ncpus NCPUS] [config_files ...]

Run job in the Key4HEP framework

positional arguments:
  config_files          Gaudi config (python) files describing the job

options:
  -h, --help            show this help message and exit
  --dry-run             Do not actually run the job, just parse the config files
  -v, --verbose         Run job with verbose output
  -n NUM_EVENTS, --num-events NUM_EVENTS
                        Number of events to run
  -l, --list            Print all the configurable components available in the framework and exit
  --gdb                 Attach gdb debugger
  --ncpus NCPUS         Start Gaudi in parallel mode using NCPUS processes. 0 => serial mode (default), -1 => use all CPUs

When supplied with a Gaudi steering file k4run --help file.py also shows the settable properties of the Gaudi algorithms used in the file. Additionally, it is possible to add further arguments and use them in the steering file by using the Python argparse.ArgumentParser shared by k4run.

from k4FWCore.parseArgs import parser
parser.add_argument("-f", "--foo", type=int, help="hello world")
my_opts = parser.parse_known_args()
print(my_opts[0].foo)

Dependencies

ROOT
PODIO
Gaudi

Installation and downstream usage.

k4FWCore is a CMake project. After setting up the dependencies (use for example source /cvmfs/sw.hsf.org/key4hep/setup.sh)

mkdir build
cd build
cmake ..
make install

Implementing algorithms

k4FWCore uses Gaudi::Functional for executing algorithms. There are several types of algorithms, depending on your use case:

The Consumer takes inputs but no outputs; can be used for reading
The Producer takes outputs but no inputs; can be used for generating collections or events
The Transformer is the more general one (both the Consumer and the Producer are a particular case of this one) and takes both inputs and outputs

A more complete list of algorithms can be found in https://lhcb.github.io/DevelopKit/03a-gaudi/, in the Gaudi::Functional section.

In all cases the implementation process is the same: we'll create a new class that will inherit from one of the previous algorithms. Then, we implement operator(), where our algorithm will be. This operator() will return either a single type (including void) or a tuple with multiple types. It will take one parameter per input. Simple examples can be found in the test folder for each one of the above-mentioned algorithms. In addition, there are tests that have either multiple inputs and / or multiple outputs (like ExampleFunctionalProducerMultiple) that can be used as a template for the more typical case when working with multiple inputs or outputs.

GaudiAlg is deprecated and will be removed in future versions of Gaudi.

k4fwcore's People

Contributors

Stargazers

Watchers

k4fwcore's Issues

[CI] tests not run on github actions

locally, spack dev-build --test=all k4fwcore@master runs the tests fine, but on github actions it looks like no tests are found: https://github.com/key4hep/k4FWCore/runs/1411791420#step:6:439

Merge `k4DataSvc` and `PodioDataSvc`

As discussed in #100 k4DataSvc is effectively just a very thin wrapper around PodioDataSvc and can be removed. This issue is mainly for keeping track of the removal and for related discussions.

Issues with EDM4hep

I have created an example in https://github.com/ihep-sft-group/K4FWCore to write and read the EDM4hep data using K4FWCore and Gaudi v33r1. One problem is that the contents of the branches are not shown due to the dictionary are not linked to the module.

Following is how to repeat the problem:

$ git clone [email protected]:ihep-sft-group/K4FWCore.git K4FWCoreIHEPSFT
$ cd K4FWCoreIHEPSFT
$ mkdir build && cd build
$ cmake .. -DHOST_BINARY_TAG=${BINARY_TAG}
$ make

Then, run the script and file test.root is created:

$ ./run gaudirun.py ../Examples/options/edm4hep_write.py

Using root to explore the content:

$ root -l test.root
Warning in <TInterpreter::ReadRootmapFile>: class  podio::CollectionBase found in libpodioDict.so  is already in libpodio.so
Warning in <TInterpreter::ReadRootmapFile>: class  podio::CollectionIDTable found in libpodioDict.so  is already in libpodio.so
Warning in <TInterpreter::ReadRootmapFile>: class  podio::ObjectID found in libpodioDict.so  is already in libpodio.so
Warning in <TInterpreter::ReadRootmapFile>: class  podio::PythonEventStore found in libpodioDict.so  is already in libpodio.so
root [0]
Attaching file test.root as _file0...
Warning in <TClass::Init>: no dictionary for class podio::ObjectID is available
Warning in <TClass::Init>: no dictionary for class podio::CollectionIDTable is available
(TFile *) 0x2c2c960
root [1] events->Print()
******************************************************************************
*Tree    :events    : Events tree                                            *
*Entries :       10 : Total =           16731 bytes  File  Size =       2494 *
*        :          : Tree compression factor =   8.23                       *
******************************************************************************
*Br    0 :EventHeader : vector<edm4hep::EventHeaderData>                     *
*Entries :       10 : Total  Size=        981 bytes  File Size  =        182 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   2.57     *
*............................................................................*
*Br    1 :MCParticle : vector<edm4hep::MCParticleData>                       *
*Entries :       10 : Total  Size=       9376 bytes  File Size  =        342 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=  25.93     *
*............................................................................*
*Br    2 :MCParticle#0 : Int_t MCParticle#0_                                 *
*Entries :       10 : Total  Size=       2460 bytes  File Size  =        134 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.26     *
*............................................................................*
*Br    3 :MCParticle#0.index : Int_t index[MCParticle#0_]                    *
*Entries :       10 : Total  Size=       1086 bytes  File Size  =        147 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   3.10     *
*............................................................................*
*Br    4 :MCParticle#0.collectionID : Int_t collectionID[MCParticle#0_]      *
*Entries :       10 : Total  Size=       1121 bytes  File Size  =        152 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   3.04     *
*............................................................................*
*Br    5 :MCParticle#1 : Int_t MCParticle#1_                                 *
*Entries :       10 : Total  Size=       2460 bytes  File Size  =        134 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   1.26     *
*............................................................................*
*Br    6 :MCParticle#1.index : Int_t index[MCParticle#1_]                    *
*Entries :       10 : Total  Size=       1086 bytes  File Size  =        155 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   2.94     *
*............................................................................*
*Br    7 :MCParticle#1.collectionID : Int_t collectionID[MCParticle#1_]      *
*Entries :       10 : Total  Size=       1121 bytes  File Size  =        152 *
*Baskets :        1 : Basket Size=      32000 bytes  Compression=   3.04     *
*............................................................................*

As you see here, the contents of the branches are not split and shown here.

My current solution is forcing linking the dictionary library.

gaudi_add_module(Examples ${Examples_srcs}
    INCLUDE_DIRS FWCore GaudiAlgLib GaudiKernel ${podio_INCLUDE_DIRS}
    LINK_LIBRARIES FWCore GaudiAlgLib GaudiKernel
      # Force loading the libraries.
      -Wl,--no-as-needed EDM4HEP::edm4hep EDM4HEP::edm4hepDict ${podio_LIBRARIES} podio::podioRootIO -Wl,--as-needed
)

I am not sure where I am wrong. Thank you!

Tao Lin (IHEP)

Implement Gaudi-based Overlay Processor

To remove the dependency on LCIO files, we need a Gaudi-based implementations of the Overlay Processors that read EDM4hep files directly.

Legacy `GaudiAlg`, `GaudiTool`, etc. will be removed in the future

This has been announced at the Gaudi developers meeting on June 7, 2023. This means we should switch to the "new" family of Gaudi::Algorithm or ideally Gaudi Functional in order to have a smooth transition. This is also relevant for other repositories in key4hep and HEP-FCC. We can use this issue to keep an overview on related issues in the specific repositories.

`k4run` crashes when invoked without arguments

Originally reported here: key4hep/key4hep-spack#498

* ********************************************************************
* Welcome to lxplus924.cern.ch, AlmaLinux release 9.2 (Turquoise Kodkod)
* Archive of news is available in /etc/motd-archive
* Reminder: you have agreed to the CERN
*   computing rules, in particular OC5. CERN implements
*   the measures necessary to ensure compliance.
*   https://cern.ch/ComputingRules
* Puppet environment: production, Roger state: production
* Foreman hostgroup: lxplus/nodes/login
* Availability zone: cern-geneva-b
* LXPLUS Public Login Service - http://lxplusdoc.web.cern.ch/
* An AlmaLinux8 based lxplus8.cern.ch is now available
* Please read LXPLUS Privacy Notice in http://cern.ch/go/TpV7
* ********************************************************************
Last login: Thu Jun 15 14:35:46 2023 from 2001:1458:202:96::101:4142
~$ source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh 
AlmaLinux 9 detected
Setting up the latest Key4HEP software stack from CVMFS ...
 ...  Key4HEP release: 2023-06-15
 ...  Use the following command to reproduce the current environment: 
 ... 
         source /cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-15/x86_64-almalinux9-gcc11.3.1-opt/key4hep-stack/2023-06-15-zkepcb/setup.sh
 ... 
 ...  If you have any issues, comments or requests open an issue at https://github.com/key4hep/key4hep-spack/issues
~$ k4run
Traceback (most recent call last):
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-15/x86_64-almalinux9-gcc11.3.1-opt/k4fwcore/348be068b687353f2886421110190aaa682bd293=develop-yyw2xj/bin/k4run", line 113, in <module>
    opts = parser.parse_known_args()
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-10/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-mbuvpt/lib/python3.10/argparse.py", line 1859, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-10/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-mbuvpt/lib/python3.10/argparse.py", line 2075, in _parse_known_args
    stop_index = consume_positionals(start_index)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-10/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-mbuvpt/lib/python3.10/argparse.py", line 2031, in consume_positionals
    take_action(action, args)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-10/x86_64-almalinux9-gcc11.3.1-opt/python/3.10.10-mbuvpt/lib/python3.10/argparse.py", line 1936, in take_action
    action(self, namespace, argument_values, option_string)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-06-15/x86_64-almalinux9-gcc11.3.1-opt/k4fwcore/348be068b687353f2886421110190aaa682bd293=develop-yyw2xj/bin/k4run", line 37, in __call__
    for conf in dict.fromkeys(ApplicationMgr.allConfigurables.values()):
NameError: name 'ApplicationMgr' is not defined
~$

This looks like some import is missing, although I am also not entirely sure whether argparse could be made to intervene earlier if launched without arguments.

Make `DataSvc` thread safe

As already discussed in #94 eventually we want to move to Gaudi Functional. This requires that our DataHandle / DataSvc works in a thread safe way. #100 introduces Frame based I/O but probably not yet in a thread safe enough way to make it work inside Gaudi Functional.

PodioOutput saves metadata of droped collections into the ROOT file

When collection is dropped in PodioOutput its metadata are saved regardless.

OS version: e.g. centos7
Package version: key4hep-nightlies
Reproduced by: podio-dump on generated file crashes when dropped collection is encountered
Example file generator: https://github.com/HEP-FCC/k4RecCalorimeter/blob/main/RecFCCeeCalorimeter/tests/options/runCaloSim.py
Expected behaviour: Metadata of dropped collections are not saved

Fix getting EventStore (getProvider) in PodioDataSvc

When trying to get the podio::EventStore in PodioDataSvc:

https://github.com/key4hep/k4FWCore/blob/master/k4FWCore/include/k4FWCore/PodioDataSvc.h#L46

it will fail due to deleted function in podio:

https://github.com/AIDASoft/podio/blob/master/include/podio/EventStore.h#L39

Example:

  ServiceHandle<IDataProviderSvc> m_eds ("EventDataSvc", "EDM4hep2LcioTool");
  PodioDataSvc* pds;
  pds = dynamic_cast<PodioDataSvc*>( m_eds.get());

  const podio::EventStore provider = pds->getProvider();

Add test for CellID retrieval in Datahandle

See #53

Segmentation faults in tests when rebuilding k4fwcore using nightly builds

Using

source /cvmfs/sw-nightlies.hsf.org/spackages4/key4hep-stack/master-2022-03-10/x86_64-centos7-gcc8.3.0-opt/tyhv7ehskcxeqrjxcvmqeyjpulgp3lo4/setup.sh

and locally rebuilding, running make test gives a segfault in several tests. (see below).
Running the tests without rebuilding works ok, strangely enough, so the spack install is probably fine, but the environment for the local installation could be at fault.
k4fwcore_segfault.txt

Errors in steering files are not easy to find

Currently if there is any error in the steering file the traceback looks like this:

Traceback (most recent call last):
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-29/x86_64-centos7-gcc12.2.0-opt/k4fwcore/198f3383c8a170ab964d2ee2819c3edb6773ed02=develop-dyison/bin/k4run", line 129, in <module>
    load_file(file)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-29/x86_64-centos7-gcc12.2.0-opt/k4fwcore/198f3383c8a170ab964d2ee2819c3edb6773ed02=develop-dyison/bin/k4run", line 26, in load_file
    exec(file.read(), {})
  File "<string>", line 20, in <module>
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-03/x86_64-centos7-gcc12.2.0-opt/gaudi/36.14-tq6q3o/python/GaudiKernel/Configurable.py", line 517, in __setattr__
    super(Configurable, self).__setattr__(name, value)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-03/x86_64-centos7-gcc12.2.0-opt/gaudi/36.14-tq6q3o/python/GaudiKernel/PropertyProxy.py", line 145, in __set__
    value = _isCompatible(proptype, value)
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2023-09-03/x86_64-centos7-gcc12.2.0-opt/gaudi/36.14-tq6q3o/python/GaudiKernel/PropertyProxy.py", line 66, in _isCompatible
    raise ValueError(errmsg)
ValueError: received an instance of <class 'int'>, but <class 'list'> expected

and while the line in the steering file that caused the error can be extracted (in this case line 20), it's not evident. Ideally the trackeback messages should look more like normal python traceback messages that point you to the line that caused the issue.

`podio::Frame` based I/O

Since the currently used EventStore based I/O functionality is de-facto deprecated and will be removed from podio, we should switch to the Frame based I/O model. Ideally we only use podio components here and do not roll a separate implementation of Readers/Writers. Another goal is to support parallelism of any kind (inter-event and intra-event) out of the box, so that we can also easily switch all algorithm developments to the Gaudi Functional approach. To make this work we have to fulfill the correct Gaudi interface. We (@vvolkl and me) had a brief meeting with Marco Clemencic from the Gaudi team trying to identify the correct one. I will try to put the summary of that meeting here, although it will potentially not bring us all the way to the end, and we might have to have another meeting. (Obviously, the possibility exists that I misunderstood a few things)

Gaudi Functional uses the underlying EvtSvc to populate its inputs/outputs (it is after all only an Algorithm)
For multithreading Gaudi will use several transient stores in parallel (these are managed by the HiveWhiteBoard(?)), which effectively hides the switching between the different stores via an EventContext
The AlgTask (GaudiHive) is managing this context switching and also calls sysExecute on the managed algorithms to actually run them
The IDataProviderSvc seems to be the interface that we need to fulfill. The basic idea would be to make each instance of this service own exactly one podio::Frame and leave the rest of the management to Gaudi

Add external config files (like pythiacards) to metadata

The JobOptionsSvc automatically adds the Gaudi job options, but those often include paths to other configuration files like pythia cards. In the interest of reproducibility it would be good to have a simple interface to save the contents of these files as well as a string in the metadatatree.

Remove legacy data services and functionality

#100 Introduced several Legacy components and services that should be absorbed cleanly into the new Frame based I/O. Once all of that has been done, the legacy functionality can be removed.

Discussion: Testing / CTest / Gaudi Environment

This concerns all framework packages (and key4hep-spack): There is an issue with testing gaudi framework components that I keep stumbling over: Due to the component/plugin mechanism, Gaudi mixes the (spack) concepts of build and run environments. In particular, while the compiler can find libraries via RPaths, ld-config ... the Gaudi plugin system needs the library to be in LD_LIBRARY_PATH to run the contained components.

When using CTest to run tests, they are run in the build environment. Gaudi provides a run script that extends the environment with the necessary variables to run the current package, but they don't cover other gaudi packages that are used in the tests.
There are several approaches to fix this:

All environment information could be set in CTest using the ENVIRONMENT of add_test
Spack could set up the build environment of gaudi packages the same as their run environment
The appoach I tried out here: https://github.com/HEP-FCC/k4Gen/blob/main/.github/workflows/test.yaml#L27 The package is installed first, and then ctest is run in the environment of the installed package.
Not use ctest at all for these kind of integration test, but use a different testing mechanism. Spack can run tests on installed packages, for example: https://spack.readthedocs.io/en/latest/command_index.html#spack-test

When a view is used, all of this is of couse less of an issue.

Put together a brief introduction and how-to page for the main key4hep webpage

The documentation that currently exists in k4MarlinWrapper documentation about how to read EDM4hep input should really be here.

Add Documentation for UniqueIDGenSvc

Somewhere, probably here

k4FWCore/k4FWCore/components/UniqueIDGenSvc.h

Line 3 in d6555ad

There should be an explanation what this service should be used for. And contain all the relevant keywords: reproducible, random, seed

Do not attach gdb by default

Currently k4run automatically attaches and runs gdb on a crash to get a backtrace. To arrive at an actual backtrace a significant amount of time passes. I think this should not be the default.

Read problem for edm4hep::ClusterCollection and edm4hep::VertexCollection

When I use PodioInput to read edm4hep::ClusterCollection or edm4hep::VertexCollection from root file, it raises error, see below:

AlgExecStateSvc     DEBUG adding alg outputalg to 1 slots
PodioReader         DEBUG Registering collection to read TestPFO with id 1
PodioReader         DEBUG Registering collection to read TestCalo with id 4
PodioReader         DEBUG Registering collection to read TestMC with id 6
PodioReader         DEBUG Registering collection to read TestMCRec with id 5
PodioReader         DEBUG Registering collection to read TestCluster with id 2

 *** Break *** segmentation violation



===========================================================
There was a crash.
This is the entire stack trace of all threads:
===========================================================
#0  0x00007f17c261289e in waitpid () from /lib64/libc.so.6
#1  0x00007f17c25a44e9 in do_system () from /lib64/libc.so.6
#2  0x00007f17b53150ba in TUnixSystem::StackTrace() () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-d9e99/x86_64-slc6-gcc8-opt/lib/libCore.so
#3  0x00007f17b5317914 in TUnixSystem::DispatchSignals(ESignals) () from /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.20.02-d9e99/x86_64-slc6-gcc8-opt/lib/libCore.so
#4  <signal handler called>
#5  0x00007f17b35b5e49 in podio::ROOTReader::readCollection (this=0x18f5018, name=...) at /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-slc6/include/c++/8.3.0/bits/stl_vector.h:930
#6  0x00007f17b3d99819 in podio::EventStore::doGet (this=0x18f5080, name=..., collection=
0x7ffc2960a368: 0x0, setReferences=<optimized out>) at /cvmfs/cepcsw.ihep.ac.cn/prototype/releases/externals/97.0.2/podio-src/src/EventStore.cc:73
#7  0x00007f17b3d99ab2 in podio::EventStore::get (this=0x18f5080, id=<optimized out>, collection=

[Reminder] Define metadata interface

Define how we want to make (collection) metadata available in the new Frame based system.

IMHO we should give access to it in the init or starting phases only. As it is supposed to be constant over runtime. Discussion can happen on top of a coming PR of mine.

Further issues with k4run

See discussion in: #134

Two further issues popped up:

list comprehensions in option files stopped working, fixed by: #140
Some FCC option file causes another error that looks like a propName exists more than once, to be investigated further in this issue

Properly handle metadata access with Frame

#119 made the (mutable) metadata frame accessible publicly. This should become private again as it is really only a workaround for getting things unstuck.

Rewrite Overlay components in generic way

The overlay components were originally written for pile-up at FCC-hh, but can be used for other types of beam background, so should be renamed to something more generic.

Failed to find GaudiAlgLib and GaudiKernel when building k4FWCore with my own Gaudi v35

Hi all,

When I build k4FWCore using the key4hep nightlies, there is no problem.
But when I switch to use Gaudi v35 built based on LCG98python3, the problem happens (please see below).

Following is the configure stage, there is no error:

[lint@lxslc709]$ mkdir k4FWCore-build
[lint@lxslc709]$ cd k4FWCore-build/
[lint@lxslc709]$ cmake ../k4FWCore
-- The C compiler identification is GNU 8.3.0
-- The CXX compiler identification is GNU 8.3.0
-- Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/gcc
-- Check for working C compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/gcc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++
-- Check for working CXX compiler: /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - yes
-- Found Threads: TRUE
-- Found Python: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/bin/python3.7 (found suitable version "3.7.6", minimum required is "3.7") found components: Interpreter Development
-- Found ROOT: /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/cmake (found version 6.22.00)
-- Found TBB: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include (found version "2020.2.11102.2") found components: tbb malloc malloc_proxy
-- Found UUID: /usr/include
--   Import target: UUID::uuid
-- Found ZLIB: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/libz.so (found suitable version "1.2.11", minimum required is "1.2.11")
-- Found Rangev3: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include
--   Import target: Rangev3::rangev3
-- Found cppgsl: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include
--   Import target: cppgsl::cppgsl
-- Found nlohmann_json: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/cmake/nlohmann_json/nlohmann_jsonConfig.cmake (found version "3.6.1")
-- Found AIDA: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/src/cpp
--   Import target: AIDA::aida
-- Found XercesC: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/libxerces-c.so (found version "3.2.3")
-- Found HepPDT: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include
--   Import target: HepPDT::heppdt
-- Found CppUnit: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include
--   Import target: CppUnit::cppunit
-- Found unwind: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include
--   Import target: unwind::unwind
-- Found gperftools: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/include (found suitable version "2.7.0", minimum required is "2.7.0")
--   found components: gperftools tcmalloc tcmalloc_debug tcmalloc_and_profiler tcmalloc_minimal profiler
--   Import executable target: gperftools::pprof
-- Found Doxygen: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/bin/doxygen (found suitable version "1.8.15", minimum required is "1.8.15") found components: doxygen dot
-- Found jemalloc: /cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/lib/libjemalloc.so
--   Import target: jemalloc::jemalloc
-- Found Gaudi: /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/Gaudi/lib/cmake/Gaudi (found version 35.0)
-- Found podio: /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/lib64/cmake/podio/podioConfig.cmake
-- Found EDM4HEP: /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/EDM4hep/lib64/cmake/EDM4HEP/EDM4HEPConfig.cmake
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build

Following is the make stage with VERBOSE=1:

[lint@lxslc709]$ make VERBOSE=1
/cvmfs/sft.cern.ch/lcg/releases/CMake/3.17.3-75516/x86_64-centos7-gcc8-opt/bin/cmake -S/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore -B/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build --check-build-system CMakeFiles/Makefile.cmake 0
/cvmfs/sft.cern.ch/lcg/releases/CMake/3.17.3-75516/x86_64-centos7-gcc8-opt/bin/cmake -E cmake_progress_start /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/CMakeFiles /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/CMakeFiles/progress.marks
make  -f CMakeFiles/Makefile2 all
make[1]: Entering directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
make  -f k4FWCore/CMakeFiles/k4FWCore.dir/build.make k4FWCore/CMakeFiles/k4FWCore.dir/depend
make[2]: Entering directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
cd /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build && /cvmfs/sft.cern.ch/lcg/releases/CMake/3.17.3-75516/x86_64-centos7-gcc8-opt/bin/cmake -E cmake_depends "Unix Makefiles" /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore/k4FWCore /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore/CMakeFiles/k4FWCore.dir/DependInfo.cmake --color=
Dependee "/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore/CMakeFiles/k4FWCore.dir/DependInfo.cmake" is newer than depender "/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore/CMakeFiles/k4FWCore.dir/depend.internal".
Dependee "/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore/CMakeFiles/CMakeDirectoryInformation.cmake" is newer than depender "/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore/CMakeFiles/k4FWCore.dir/depend.internal".
Scanning dependencies of target k4FWCore
make[2]: Leaving directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
make  -f k4FWCore/CMakeFiles/k4FWCore.dir/build.make k4FWCore/CMakeFiles/k4FWCore.dir/build
make[2]: Entering directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
[  7%] Building CXX object k4FWCore/CMakeFiles/k4FWCore.dir/src/PodioDataSvc.cpp.o
cd /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore && /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++  -Dk4FWCore_EXPORTS -I/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore/k4FWCore/include -isystem /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/include -isystem /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/include  -fPIC   -std=gnu++17 -o CMakeFiles/k4FWCore.dir/src/PodioDataSvc.cpp.o -c /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore/k4FWCore/src/PodioDataSvc.cpp
[ 14%] Building CXX object k4FWCore/CMakeFiles/k4FWCore.dir/src/KeepDropSwitch.cpp.o
cd /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore && /cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++  -Dk4FWCore_EXPORTS -I/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore/k4FWCore/include -isystem /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/include -isystem /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/include  -fPIC   -std=gnu++17 -o CMakeFiles/k4FWCore.dir/src/KeepDropSwitch.cpp.o -c /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore/k4FWCore/src/KeepDropSwitch.cpp
[ 21%] Linking CXX shared library libk4FWCore.so
cd /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/k4FWCore && /cvmfs/sft.cern.ch/lcg/releases/CMake/3.17.3-75516/x86_64-centos7-gcc8-opt/bin/cmake -E cmake_link_script CMakeFiles/k4FWCore.dir/link.txt --verbose=1
/cvmfs/sft.cern.ch/lcg/releases/gcc/8.3.0-cebb0/x86_64-centos7/bin/g++ -fPIC   -shared -Wl,-soname,libk4FWCore.so -o libk4FWCore.so CMakeFiles/k4FWCore.dir/src/PodioDataSvc.cpp.o CMakeFiles/k4FWCore.dir/src/KeepDropSwitch.cpp.o   -L/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/.plugins  -Wl,-rpath,/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build/.plugins:/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/lib64:/cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib: -lGaudiAlgLib -lGaudiKernel /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/lib64/libpodioRootIO.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libTree.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libNet.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libRIO.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libMathCore.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libImt.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libThread.so /cvmfs/sft.cern.ch/lcg/releases/ROOT/v6.22.00-be0a0/x86_64-centos7-gcc8-opt/lib/libCore.so -lpthread /tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/podio/lib64/libpodio.so
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.30-e5b21/x86_64-centos7/bin/ld: cannot find -lGaudiAlgLib
/cvmfs/sft.cern.ch/lcg/releases/binutils/2.30-e5b21/x86_64-centos7/bin/ld: cannot find -lGaudiKernel
collect2: error: ld returned 1 exit status
make[2]: *** [k4FWCore/libk4FWCore.so] Error 1
make[2]: Leaving directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
make[1]: *** [k4FWCore/CMakeFiles/k4FWCore.dir/all] Error 2
make[1]: Leaving directory `/tmp/ihep.ac.cn/cepcsw/a/prototype/releases/externals/98.0.0/testing/k4FWCore-build'
make: *** [all] Error 2
[lint@lxslc709]$ which cmake
/cvmfs/sft.cern.ch/lcg/views/LCG_98python3/x86_64-centos7-gcc8-opt/bin/cmake
[lint@lxslc709]$ cmake --version
cmake version 3.17.3

My current solution is adding the Gaudi:: prefix to the targets in the CMakeLists.txt files:

[lint@lxslc709]$ git diff
diff --git a/k4FWCore/CMakeLists.txt b/k4FWCore/CMakeLists.txt
index 55049ff..84bf189 100644
--- a/k4FWCore/CMakeLists.txt
+++ b/k4FWCore/CMakeLists.txt
@@ -10,7 +10,7 @@ gaudi_install(SCRIPTS)
 gaudi_add_library(k4FWCore
                  SOURCES src/PodioDataSvc.cpp
               src/KeepDropSwitch.cpp
-                  LINK GaudiAlgLib GaudiKernel podio::podioRootIO ROOT::Core ROOT::RIO ROOT::Tree
+                  LINK Gaudi::GaudiAlgLib Gaudi::GaudiKernel podio::podioRootIO ROOT::Core ROOT::RIO ROOT::Tree
                   )
 target_include_directories(k4FWCore PUBLIC
   $<BUILD_INTERFACE:${CMAKE_CURRENT_LIST_DIR}/include>
@@ -22,7 +22,7 @@ target_include_directories(k4FWCore PUBLIC
 file(GLOB k4fwcore_plugin_sources components/*.cpp)
 gaudi_add_module(k4FWCorePlugins
                  SOURCES ${k4fwcore_plugin_sources}
-                 LINK GaudiAlgLib GaudiKernel k4FWCore ROOT::Core ROOT::RIO ROOT::Tree)
+                 LINK Gaudi::GaudiAlgLib Gaudi::GaudiKernel k4FWCore ROOT::Core ROOT::RIO ROOT::Tree)
 target_include_directories(k4FWCorePlugins PUBLIC
   $<BUILD_INTERFACE:${CMAKE_CURRENT_LIST_DIR}/include>
   $<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
diff --git a/test/k4TestFWCore/CMakeLists.txt b/test/k4TestFWCore/CMakeLists.txt
index 0101584..9f212b0 100644
--- a/test/k4TestFWCore/CMakeLists.txt
+++ b/test/k4TestFWCore/CMakeLists.txt
@@ -10,7 +10,7 @@ find_package(EDM4HEP)
 file(GLOB k4testfwcore_plugin_sources src/components/*.cpp)
 gaudi_add_module(k4TestFWCorePlugins
                  SOURCES ${k4testfwcore_plugin_sources}
-                 LINK GaudiAlgLib GaudiKernel k4FWCore ROOT::Core ROOT::RIO ROOT::Tree EDM4HEP::edm4hepDict EDM4HEP::edm4hep)
+                 LINK Gaudi::GaudiAlgLib Gaudi::GaudiKernel k4FWCore ROOT::Core ROOT::RIO ROOT::Tree EDM4HEP::edm4hepDict EDM4HEP::edm4hep)


 include(CTest)

Do you have any suggetions?
Thank you!
Tao

k4DataSvc seg faults if input file does not exist

When trying to open a non existing file, k4run will seg fault. This should be handled much more gracefully, as typos are otherwise pretty time consuming

Remove the reader from the `PodioDataSvc` and make the `PodioInput` populate the internal Frame

Currently, the PodioDataSvc holds a ROOTFrameReader which is used for configuring the input file. On the other hand the collections that should be read are configured via the PodioInput algorithm. All of the input configuration should live in the PodioInput.

Other points to address:

Currently all collections have to be stated in the options file. This should become optional to restrict the collections with a default of reading all collections if not stated otherwise.

Food for thought: Could we simply take the default DataHandle from Gaudi and populate it in PodioInput (from the Frame we read there) and take things out again in PodioOutput (put it into a Frame and write things out). Could potentially solve the issues we observe with k4Gen.

[CI] re-add downstream test

For simplicity, I removed the downstream test when moving to spack dev-build for the CI. Should be re-added (easiest to add k4-project-template to spack recipes, and do spack install --test=all k4projecttemlate ^k4fwcore@master ?)

PodioOutput writes additional entries to the output file

When the EvtMax option is set greater than the total event number in the input file(s), the PodioOutput will write 2 more entries than the right number to the output file.

We can simply repeat this problem by the default tests. For example, if we change the EvtMax in test/k4TestFWCore/options/readExampleEventData.py to

EvtMax=110,

The test will be failed, and 102 entries will be saved in the output file "output_k4test_exampledata_2.root".

I traced a problem in PodioDataSvc line 71, where

if (m_eventNum++ > m_eventMax) {

should be

if (++m_eventNum >= m_eventMax) {

Then, we can get the right entries. But the test will still be failed when EvtMax is greater than (and even equal to) 100.

k4run: Expose properties from multiple algorithms of the same type

LCG Init Script sets platform wrong

In principle, all dependencies of K4FWCore should be contained in the lcg releases. However, using the init script of the LCG view 96c_LS the LCG_SYSTEM is set incorrectly:


-- Looking for projects in /cvmfs/sft.cern.ch/lcg/releases/LCG_94
-- project -> Gaudi, version hint -> v32r2
--   found Gaudi v32r2 /cvmfs/sft.cern.ch/lcg/releases/LCG_96c_LS/Gaudi/v32r2/x86_64-centos7-gcc8-opt
CMake Error at /cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/cmake/GaudiProjectConfig.cmake:1020 (message):
  Incompatible values of LCG_SYSTEM:

    K4FWCore v0r1 -> 'x86_64-ubuntu1804-gcc7'
    Gaudi v32r2 -> 'x86_64-centos7-gcc8'

Input of non-podio-collection types.

The dataservice allows output of stl types and types with dicts, but they actually cannot be read in. Need to update the code in DataHandle::get.

Document KeepDropSwitch

Spack-installed gaudi~optional is missing components

The spack recipe for gaudi has clhep and aida only as optional dependencies, which leads to gaudi silently dropping some essential services like the HistogramSvc and the RdmGenSvc. I even got runtime errors using Algorithms without these, so I think they should be added as core dependencies.

Remove --ncpus flag in k4run until it is actually supported

IJobOptionsSvc removed in Gaudi 36.0

The IJobOptionsSvc that is used by the PodioOutput has been deprecated and removed from Gaudi v36 onwards: See CHANGELOG

We are using it here:

k4FWCore/k4FWCore/components/PodioOutput.cpp

Line 2 in 67e5d94

#include "GaudiKernel/IJobOptionsSvc.h"

It is not entirely clear to me what should be used instead.

User defined TTree can't be written to a second file

When I fill a TTree and write it to a second file by myself in an algorithm:

StatusCode AnAlgorithm::initialize()
{
    _file = TFile::Open("tmp.root", "RECREATE");
    _tree = new TTree("mytree", "testing");
    // other things .......
}

// filling _tree in execute() ......

StatusCode AnAlgorithm::finalize()
{
    // _file->cd();  // sometimes this line leads to a crash, or else the "events" TTree will be wrote to "tmp.root"
    _tree->Write();
    // other things .......
}

Then, tmp.root will be an empty file. And the mytree TTree is saved to the same file of EDM4hep by PodioOutput.

Drop Legacy PODIO components

This is a reminder to drop the legacy PODIO components introduced in #100 . Their removal has in particular an impact on DataHandle and DataWrapper.
Once the legacy components are gone, the latter should get a flag whether it owns the data or not. This would replace the currently needed 'reset' method workaround

`PodioOutput` requires DataSvc to be called `EventDataSvc`

This should at least be documented.

PodioOutput will fail when writing collections only in specific events

See PR: #67
If a collection is written only in one event, but not another, it will break
It breaks on filling the TTree* m_datatree

Use python logger instead of print() in k4Run

Running over more events than present or all events (using `-1`) should exit with 0

Currently anything that makes Gaudi use a IEventProcessor::stopRun makes the whole process exit with exit code 4 (see here).

This requires us to do things like this in tests:

k4FWCore/test/k4FWCoreTest/CMakeLists.txt

Lines 60 to 62 in 33ba587

 set_tests_properties( CheckExampleEventData 

 PROPERTIES PASS_REGULAR_EXPRESSION "Application Manager Terminated successfully with a user requested ScheduledStop" 

 DEPENDS CreateExampleEventData)

It also makes it impossible to defensively script (using -e). This should be fixed somehow, although I do not know how.

Make the I/O functionality use the podio provided functionality

For historical reasons the I/O implementations of standalone podio and the one that is present here have diverged a bit and so now they over in principle the same but still slightly different functionality. I think the framework should use the podio facilities as far as possible, and I originally thought that this would be a somewhat mechanical but in the end straight forward thing to do. However, I have realized that there is a bit more work involved, so that I am recording my observations here. In the end, I think that changes to podio are also necessary and that it might be best to first stabilize the interfaces podio offers before we actually start to work on this here.

High level functionality differences

The following table gives an overview of the things were podio and k4FWCore differ in high level (i.e. user perceivable) functionality

	podio	k4FWCore
`vector` user data	`UserDataCollection<T>` (since `v00-14`, compile time limited `T`)	`DataHandle<std::vector<T>>` (dictionary limited `T`, may fail silently(?) on I/O). There is #25 that might impact the usefulness of this feature(?)
other user data	N/A	`DataHandle<T>` (dedicated handling of ints and floats, but in principle again ROOT dictionary limited)

I am not entirely sure how widespread the usage of these features is throughout the Key4hep components. Hence, it is also hard to gauge whether some of the functionality could be easily removed from k4FWCore (e.g. the possibility to store single int/float values per event). This is something that probably needs discussing.

Technicalities

In k4FWCore the PodioDataSvc is handling the actual reading of the collections, it holds a podio::ROOTReader and a podio::EventStore as members that do the heavy lifting in this regard. The k4DataSvc is in essence a very thin wrapper around the PodioDataSvc that exposes the filename(s) as property to be configured from the options file. The PodioInput algorithm is responsible for actually triggering the reading of the collections (that are specified as a property) in its execute method. For this it just loops over the list of collections to read and calls PodioDataSvc::readCollection. For writing collections there is the PodioOutput algorithm, that basically re-implements the functionality of the podio::ROOTWriter. It holds a KeepDropSwitch to control which collections to actually write to file. In all of this there are a few subtle differences between podio and k4FWCore that make a "trivial" switch to podio facilities impossible. The following table provides a (probably incomplete) overview of them:

	podio	k4FWCore
output collection handling	collections are created once via `EventStore::create` and then simply cleared in the event loop after writing.	collections are re-created every event and the `EventStore` in `PodioDataSvc` never gets to know about them
event data tree	owned by `ROOTWriter`, no outside access	owned by `PodioDataSvc`. Possible to get access to it
output collection writing	`ROOTWriter::registerForWrite` collects a list of collection names to write. Checks in `EventStore` if collections are actually available before adding them to the list. In event loop simply take this list and write (i.e. set branch addresses) and fill the event data tree.	In every call to `PodioOutput::execute` get the complete list of collections from `PodioDataSvc` and check via the `KeepDropSwitch` which collections to write, before setting branch addresses and filling the event data tree.
branch creation for user data	`UserDataCollections` are handled the same as other collections	`DataHandle` creates necessary branches as it also has access to the `PodioDataSvc` (and the event data tree therein). The `DataHandle` also makes sure to do the proper branch address re-setting.
file level meta data	N/A	`PodioOutput` writes the options file config into a separate branch of the meta data tree
I/O file formats	ROOT and SIO. (probably incomplete) abstract `IReader` interface for reading. Separate writer implementations (with equal interfaces)	Only ROOT, but at least for reading a switch to the `IReader` interface should enable reading SIO out of the box

In the end to get everything working the same and using the same facilities, some discussion is required to decide which functionality needs to be supported from podio, which functionality can be built on top of podio here, and most importantly how the interfaces have to look like to enable all this functionality.

Memory leak in k4DataSvc for CaloHitContributions if not explicitly loaded

OS version: ubuntu 21.10
Compiler version: gcc 11.2.0
Package version: 061f6b5 (HEAD -> master, tag: v01-00pre14, origin/master, origin/HEAD)
Reproduced by: (see below)
Input:
sim_barrel_sciglass_electron_1.0_10.0.edm4hep.root uploaded to https://drive.google.com/open?id=1-GgZ9EFjEusFOjvTdSoaB2MTNGRndwcN (103 MB, 1000 events with 1-10 GeV electron into EIC SciGlass barrel calorimeter)
Output: (see below)
Goal: analysis of input files with podio/EDM4hep files with the gaudi framework

We are experiencing a memory leak in the EIC framework which we can reproduce in the key4hep framework as well. We take the same approach to the PodioDataSvc, so that's not very surprising, but I'm reporting it here so someone with expertise in podio can take a look too.

Steps to reproduce

spack install [email protected] (or e.g. build an environment with [email protected])
git clone https://github.com/key4hep/k4-project-template, build, install
cp k4ProjectTemplate/options/readExampleEventData.py k4ProjectTemplate/options/readEICEventData.py and apply the following changes to read in the collection 'EcalBarrelHits' for all events in an existing file:

5,7c5
< podioevent.input = "output_k4test_exampledata.root"
< from Configurables import CreateExampleEventData
< producer = CreateExampleEventData()
---
> podioevent.input = "sim_barrel_sciglass_electron_1.0_10.0.edm4hep.root"
11c9
< inp.collections = ["MCParticles", "SimTrackerHit"]
---
> inp.collections = ["MCParticles", "EcalBarrelHits"]
16c14
<                 EvtMax=100,
---
>                 EvtMax=-1,

Download sim_barrel_sciglass_electron_1.0_10.0.edm4hep.root (103 MB), generated by an unmodified DD4hep ddsim script against the EIC geometry.
Back in build directory, run /usr/bin/time -v ./run k4run ../k4ProjectTemplate/options/readEICEventData.py.

Expected behavior

The value under Maximum resident set size (kbytes) should be independent of number of events (and about 500 MB)

Actual behavior

The memory use increases with number of events, and goes up to ~2 GB for this file (with 1000 events). Files with more than 10k events are not able to be run on typical systems due to memory limitations.

Workaround

The memory leak can be avoided by explicitly listing the linked collections, i.e. modifying the diff above to

11c9
< inp.collections = ["MCParticles", "SimTrackerHit"]
---
> inp.collections = ["MCParticles", "EcalBarrelHits", "EcalBarrelHitsContributions"]

in which case the EcalBarrelHitsContributions collection is properly deleted after each event.

This behavior may qualify as a bug since a data model may change without notice, and downstream options file will continue to work but appear to start leaking memory. It would be clearer if it just failed instead (or didn't leak the memory).

Compilation Error due to missing `Gaudi.xenv`

Using Gaudi v32r2, the build fails with the following error:

[ 20%] �[32mBuilding CXX object FWCore/CMakeFiles/FWCore.dir/src/PodioDataSvc.cpp.o�[0m
Traceback (most recent call last):
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/bin/xenv", line 11, in <module>
    load_entry_point('xenv==1.0.0', 'console_scripts', 'xenv')()
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/__init__.py", line 369, in main
    Script().main()
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/__init__.py", line 361, in main
    self._makeEnv()
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/__init__.py", line 272, in _makeEnv
    getattr(control, action)(*args)
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/Control.py", line 307, in loadXML
    self.actions[action](*args)  # pylint: disable=W0142
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/Control.py", line 59, in <lambda>
    'include'] = lambda n, c, h: self.loadXML(self._locate(n, c, h))
  File "/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/lib/python2.7/site-packages/xenv/Control.py", line 128, in _locate
    raise OSError(ENOENT, 'cannot find file in %r' % sp, filename)
OSError: [Errno 2] cannot find file in ['.', '/cvmfs/sft.cern.ch/lcg/views/LCG_96c_LS/x86_64-centos7-gcc8-opt/cmake', '/workspace/build/config/']: 'Gaudi.xenv'

This problem does not appear with Gaudi v29r0.

Empty collection when saved by PodioOutput with "keep *"

When attempting to save Podio collection read by PodioInput using PodioOutput with keep * option, it creates an instance of Podio collection in the output file; however, the collection is empty. Hence this breaks links between PODs such as OneToOneRelations, throwing segfault when trying to read the output file by podio::EventStore->get<T>(std::string) later.

OS version: centos7
Compiler version: GCC 8.3
Package version: v01-00pre09
Reproduced by:

k4run test/k4TestFWCore/options/createExampleEventData.py # creates output_k4test_exampledata.root
k4run test/k4TestFWCore/options/readExampleEventData.py # creates output_k4test_exampledata_2.root

output_k4test_exampledata.root

output_k4test_exampledata_2.root

Use podio meta data for job options

The job options are written directly to the metadata tree at the moment, but they could make use of the new metadata features in podio.

Extend PodioDataSvc to replace GaudiHiveWhiteboard

As stated here: https://codimd.web.cern.ch/E4TGrEhWTIOf2ac9zjvJRw#GaudiHive-in-CEPC-Driftchamber

getCollections() won't get the collections

After trying to get some collections using the PodioDataSvc, read by PodioInput, I found that calling getCollections() won't return the collections.
I opened this PR to fix it: #56 but that breaks the tests.
I figured out that I need to call getReadCollections(), but I still find quite confusing that getCollections() won't get the collections, but the data member, which then doesn't have the collections.

This should be either better documented or refactored to better indicate what the methods actually do.

Remove name demangling code in I/O

No longer necessary with current PODIO version

	set_tests_properties( CheckExampleEventData
	PROPERTIES PASS_REGULAR_EXPRESSION "Application Manager Terminated successfully with a user requested ScheduledStop"
	DEPENDS CreateExampleEventData)