panosc-vinyl / vinyl-project Goto Github PK

• Explain what assumption is included in this module, what modeling method is used, in which occasion this module can be used, in which occasion this module cannot be used.
• Tutorial starting from the simple case 1:
  Generating one diffraction pattern for single-shot experiments
  with the simplest pulse source, the simplest wave propagation parameter, a standard test sample and a diffraction visualization tool.
• Tutorial starting from the simple case 2:
  Generating multiple random diffraction patterns for Coherent Diffractive Imaging experiments:
  with the simplest pulse source, the simplest wave propagation parameter, a standard test sample, a diffraction visualization tool, EMC and then reconstruction.

Add the functionality to publish the simulated data as an abstract method on the AbstractBaseCalculator (if possible). Spezializations must add domain specific metadata

Collect and review the existing source simulation data services. Examples:

• XFEL pulse database XPD, web-interface:
  https://in.xfel.eu/xpd

• Various source simulation tools in Oasys

• Sources in Sirepo:
  https://beta.sirepo.com/light#/light-sources
  https://beta.sirepo.com/light#/calculator
  https://beta.sirepo.com/#/jupyter

Select at least one service for FEL (hard, soft x-ray, polarization control, 3rd harmonic), SR (wigglers, bending magnets, undulators), neutrons to be interfaced from within vinyl

• debug dump()
• API documentation

The following subtasks must be completed at indicated dates in order to make D5.1

• Draft openPMD extension for MD data: Sept. 20 (Juncheng) -> #18
• Draft openPMD extension for Wavefront data: Oct. 21 (Carsten) -> #16
• Draft openPMD extension for Raytracing (neutrons) data: Oct. 25 (Mads) -> #15
• Draft openPMD extension for Raytracing (photons) data: Oct. 21 (Aljosa) -> #15
• Draft report to Jordi: Oct. 30 (Carsten, Juncheng)
• Final report D5.1 submitted: Nov. 31(Jordi, Carsten)

• Rename the repo to libpyvinyl
• Make sure CI and documentation building works
• create a pypi package

After completing the top level pyvinyl API (BaseCalculator/Parameters), we now have to define the intermediate APIs:

• PhotonSource
• NeutronSource
• PhotonPropagation
• NeutronPropagation
• RadiationMatterInteraction
• SignalGeneration
• Detector

Take the corresponding Calculators in SIMEX as templates. Move as many parameters as possible from the specialized parameter classes into the intermediate level. For each level, ensure the API documentation (autogenerated) is ok and add a usage example as a jupyter notebook.

• Inspired by Ross Harder's paper
• I am responsible for generating simulation data to feed into the machine learning algorithm
• The generating speed is a problem. Considering to implement GAPD, after benchmarking it with SingFEL
• How much time Marjan would investigate in this project is also a question

Mads is already on it, just added here for reference

Replace the AbstractBaseClass, AbstractBaseCalculator, AbstractParameters and the intermediate level abstract calculators and parameters (AbstractPhotonPropagator, ...) by their pyvinyl counterparts.

Make pyvinyl a dependency in simex,

• Delta function in space and time
• Gaussian function in space and time

Check out
6a65415 on eucall-software/simex_platform

Prerequisites:

• well defined openPMD output format
• make a generic converter from any openPMD-api format to JSON
• schema validation for openPMD standard
• schema validation for openPMD extension used by the McStas output component

Employ the openpmd-api library to convert data from/to openpmd/native format for a selected calculator in simex. Good candidates are

• propagator (wpg)
• xmdyn pmi calculator
• pysingfel

I would like to know who is in charge of maintaining the openPMD-standard fork in sync with the upstream.

We need the issue tracker also for that repository (currently missing).

Furthermore, where is the particle beam extension documented? Has this been proposed and integrated to the upstream?

• Identify codes to support
• Common input parameters
• Common input datafields
• Common output datafields
• Abstract parameters class
• Abstract data field class
• Calculator

Develop a database for storing, archiving, and retrieving pre-computed intensity distributions (for raytracing) and complex wavefields (for wavefront propagation).

• Consider XFEL XPD (https://in.xfel.eu/xpd) as a example/prototype
  • can we extend it to host more sources and allow deposition from users
• Consider zenodo as a data repository where to store and query versioned datasets.
• Decide on data format (openpmd?) -> converters. re-use solutions from #5, #8
• Develop frontend(s) (web-based, console based, python based)

• Identify codes to support
• Common input parameters
• Common input datafields
• Common output datafields
• Abstract parameters class
• Abstract data field class
• Calculator

• beamline simulations
• target/sample trajectories
• signal generation

Expose coherent mode simulation code "comsyl" (Glass 2017) in simulation frontends.

• Data format harmonization based on openpmd (?)
• API
• Demonstration
• Documentation

• Literature review has been done
• Simulation and experimental support/collaboration required
• Might be a long term case

We defined the following KPIs (Key Performance Indicators) for WP5. This issue is meant to keep track of the current status. Each time one of these numbers increases, add a comment below.

Develop a DFT target (sample) calculator based on ASE and Mousumi's demo notebook

Expose APIs from the libsyned project (L. Rebuffi) in simulation frontends jupyter and oasys

• Demonstration via documented jupyter nb and oasys workflow
• If needed, develop libsyned to achieve above

• beamline simulations
• target/sample trajectories
• signal generation

We need to update the docker images for simex. The goal is to install simex on a cloud instance of jupyter hub (hosted on DESY's cloud servers). We also have to demonstrate that simex / vinyl example notebooks can be run on binder.

• move dependencies in top level cmake file to the Modules that actually cause these dependencies.
• Document better how modules can be selected/deselected.
• Image for master
• Image for develop
• repo2docker script
• at image build time, user should be able to select/deselect the modules he/she/* wants included in the image.

Useful contacts:
Thomas Kluyver (XFEL, WP4), Michael Schuh (DESY, expands)

• Adjust/Finalize the wavefront openpmd extension in collaboration with LUME (SLAC)
• Derive a database schema (json) for wavefront data in openpmd format.
• Setup a prototype wavefront database server (mongodb)
• Web frontend to query for wavefront parameters
• Consider two solutions for data provision:
  • On the fly calculation starting from source wavefronts from xfel's xpd.

Useful contacts: @samoylv, @buzmakov (XFEL)
@ChristopherMayes (SLAC, LUME)

• beamline simulations
• target/sample trajectories
• signal generation

vinyl should expose the following libraries:

• pyvinyl for the APIs
• SIMEX
• McStass Script
• McStass
• jupyter
• oasys

To be bundled in a docker package and runnable on jupyter hub cloud instance, HPC systems, binder.

I suggest spack (spack.io) to manage the dependencies.

• Radiation damage simulation code lacks supporting

• data format I: compatible with experimental data
• data format II: supporting parallel writing, reading, streaming for high-performance computing. It is highly demanded in the photon-sample interaction.

There exists a stub for the API reference manual but the modules have to be loaded with "automodules".

Should also include the demo notebook.

• collaboration with a PhD student in Uppsala
• Support generating diffraction pattern using SimEx

• A new clean repository? 

  • Yes.
  • Focus on data generation calculators.

• A template to start with?

  • SimEx-Data API
    • Decide which converter to implement based on the input and output
    • Data reading
      • openbabel
      • WPG/SRW
    • Data writing
      • openpmd-api

• Standard python package structure

  • Repository
    • SimEx-Lite
      • Calculators inherit from libpyvinyl
        • WavefrontCalculator
          • Extract the wavefront data from the data repository
      • Parameters
    • Test
    • Doc

• Backengines? Where to put?

  • Two options
    • As conda package.
    • As git submodules.
  • WPG
    • Standard beamline setup.
      • Precalculated wavefront.
      • Well-defined parameters
      • XPD database
  • pysingfel
  • crystfel
  • GROMACS
  • XMDYN (Drop)
    • Data format support in

• General usage

  • SimEx-lite users should be able to plug in their own photon-matter interactor and diffraction calculator.
  • Wavefront comes from the wavefront repositories.
  • It also accepts WPG/SRW output.

• Identify codes to support
• Common input parameters
• Common input datafields
• Common output datafields
• Abstract parameters class
• Abstract data field class
• Calculator