ptycho / ptypy Goto Github PK

The default pipeline should be simplified for straightforward cases while keeping more advanced features. For this, one could rehabilitate the DataSource class, which currently is not useful. Consider the possibility of having Ptypy act as a daemon, with data reading provided by completely separate processes.

There should a way that the "init.py" file scans for either engine or scan classes in order to allow users to push their engine to the pool of available Scans and Engines without modification of the init.py file.

Remains to check if we want treedict to be a dependency or if we integrate the code as submodule.

It would be interesting to see how much speed we gain by compiling pieces of the code.

Data preparation sometime fails when running with MPI. We should either debug this or consider falling back to single-process data preparation.

The model construction currently is a nested loop very difficult to update. A more flexible option would be to have a multi-pass approach that starts with a "vanilla pod" and gradually builds the model depending on input parameters. A new feature in the model would be added as an additional pass.

As a first step for acceleration. Anybody identifying obvious bottlenecks please report as a new issue.

Support for multiple focal planes within object array.

If the data is straightforwardly homogeneous, adding methods to unpack / repack numpy arrays would help testing and hacking.

Currently, default parameters are stored at the beginning of multiple files. These parameters are also stored in the file resources/parameter_descriptions.csv.

This information should not be duplicated, and all defaults should be drawn from a unique point, the csv file. We should also consider introducing a more robust validation of the input parameters. Most of the code is ready in utils/validator.py.

All this might have to be done after the transition Param -> treeDict (issue #17)

If num_probe_modes is not 1, diversity should be introduced by default.

Something has been programmed already for this at TUM. To be investigated.

This should be a slight modification compared to DM.

Documentation should evolve with the code, so they should live together.

I vote for permanently changing things in geometry:

energy : stays , switch to electron Volts instead of keV or use actual SI unit. Actually the input here is not so super self explaining
z -> distance
psize_det -> psize or pixelsize
psize_sam -> resolution
N -> shape
lam -> we could decide to permanently discard this one. However a dumb user would mor likely put an SI unit here

Helper function(s) to build parameter structures interactively.

Write heuristic functions that attempt to load data and build parameters from incomplete input, for fast deployment and testing of code on a new setup.

The RL-deconvolution procedure in data.PtyScan consumes a dispropotionate amount of parameters and is non-canonical in its execution. I presume deconvolution has to happen before rebinning? Since the correction is non-linear and also not entirely "common" I very much advise to shift the deconvolution to a subclass. This step may require to outsource the rebinning part of prepocessing in a separate function.

Currently it looks for tree.bmp via a relative path, we should make a function in the test dir that just retrieves the filepath whereever we are.

Many many tests should be added, especially for data preparation recipes and engines.

The renaming just took place. Please watch out next time you clone push or pull.

When working interactively it would be convenient to provide a helper function that guides the user to the next steps in the Ptycho preparation.

The first draft of planned ptypy improvements can be found on this google doc:

https://docs.google.com/document/d/1jp1tqEqPJxw4isc4FTKXCqzsOrgcuiBkq4XjMHWfgMc/edit?usp=sharing

Anyone planning to participate is welcome to take a look and comment or edit. We will then break these general goals in smaller pieces to make a roadmap.

The documentation of ptypy.core.data.PtyScan.load_positions(), currently says that "If parallel reads or log file parsing for each set of frames is not a time critical issue of the subclass, reimplementing this function can be ignored and it is recommended to only reimplement the :py:meth:load method.".

I am not sure if this is a good recommendation. Whether it is expansive or not, providing a way to access all positions in a scan is a useful feature. I would say that this is the recommended way of loading positions, and that alternative per-frame version should be used only if it is not possible to do otherwise...

Now that patent issues are solved we can provide an ePIE implementation without risk.

@alexbjorling and I noticed that tab completion in ipython breaks when you import ptypy.

Has to be done at some point.

Any initial test is most welcome. Options are

• OpenMP
• OpenCL
• Arrayfire
• Cuda
• Maybe others...

The current unit tests are quite verbose and overlap a lot. Whilst some overlap is good, we should minimise the time for the tests to run by refactoring.

Not clear yet what are the steps to update the gh-pages branch.

New development guidelines should be implemented as of 1st of September.

Adding test should be easy. Ptypy should have a framework of guidelines to do this. Then, we would encourage all new features to come with their own set of tests and example code.

Introduce ptypy.interactive or ptypy.consoletools with functions to load data, setup the pods, run the engines, etc, à la matlab and pylab. This would help for debugging and developing new ideas.

Seems that the flexible containers are finding an interest elsewhere. We could factor it out, but would then have to maintain a compatible API.

Any suggestion for this is welcome. The problem is nearly identical, what would be needed mostly is mapping of parameter names.

Various approaches exist, and most are engine dependent. Any implementation is welcome.

A convolution kernel representing the detector PSF or partial coherence can be refined iteratively during the reconstruction. This is active research - how can this be tied nicely to mixed states features?

The ML code in ptypy has never been tested as thoroughly as in its ancestor ptycho.

Implement the method in http://dx.doi.org/10.1364/OE.24.008360

Should be added in documentation.

See Marchesini paper on SHARP. Could also be adaptive gain.

To make POD's behaviour uniform, a "fake" exit view self._exit is created in POD.__init__. This is a waste of memory. Possible solution: create the array on-the-fly when POD.exit is accessed, or store the array as a class variable (which would have to depend on the geometry).

Eliminate in the parameters all instances of None that mean something else than “not initialised”.

New doc section.

Check whether pyFFTW would do a better job. Large diffraction patterns (for NF-ptychography) could benefit from parallelised FFTs, but a proper implementation of this could be very tricky.

Get started registering on Travis CI automatic testing.

Currently the data preparation allows for statistical weights to be defined, but further on these weights are interpreted as boolean masks. This is inconsistent and should be handled much better, especially when working with ML.

Finalise PEP8 standards throughout.

One nice idea would be to have a html5 client for real-time data visualisation and additional interactions such as pausing, changing parameters on-the-fly, etc.

Propagator is probably the most obvious first step for optimisation. A user switch has to be provided in the input parameters to decide to use compiled or native python code.