ccp4 / dui Goto Github PK

Problem:

1. User logs on for a tutorial
2. User types module load dui
3. User processes data
4. Oh no! The user has filled their home directory with junk!

Solution:

• Instead of putting everything in the current working directory, just put everything in e.g. a dui subdirectory.

If I run a stage and choose some custom advanced settings, then go back and run it again, there is no way to tell what settings I chose (and thus the difference) between the two stages.

e.g. you could get to the end of your analysis, integration, and come up with two separate results where one is much better, and you want to go back and see what settings you used (in order to use those settings in other analyses).

Related to #23, in that to re-run a stage you have to select the parent of the stage - in a tree situation with potentially many branches, selecting the parent gives no context for which child you want to change.

The text, like "|<" in buttons is not visually appealing.

Please replace with suitable icons instead.

Because template= used; Eiger files do not use templates as such

The information about what space groups correspond to each lattice can be a useful indicator on the terminal. It'd be nice to have it displayed with the table e.g. something like:

It would be much better if the image viewer could behave more like dials.image_viewer/albula/google maps when zooming with the mouse wheel, i.e. the pixel under the mouse cursor should remain under the mouse cursor when the zoom level is changed.

A user can select a number of images however the software is too clever and auto selects too many files. If the user only wants 15 images the software should only use the 15 images.

Seems the DUI records everything in dials.state - removing this allows me to process a fresh data set.

Each step of the tree, like 'spot finding' should have some key result, like number of spots found.

cannot do real space grid search

Trying to import this dataset:

/dls/mx-scratch/rjgildea/Eiger/hdrmx_medsbio/cbf/P3p09_0.5d_0.05s_12.65keV_bslz4_1_#####.cbf.bz2

Output from terminal - note the "*" is inserted in the incorrect place, not in place of the ##### as it should be:

dat.tmpl_str = /dls/mx-scratch/rjgildea/Eiger/hdrmx_medsbio/cbf/P3p09_0.5d_0.05s_12.65keV_bslz4_1_#####.cbf.bz2
1
failed to << check_gray_outs() >>
update_pbar_msg = click << find >> to go ahead, or click << Retry >>
it_index = <PyQt4.QtCore.QModelIndex object at 0x7f521a326aa0>
TreeNavWidget(node_clicked)
command_lst[1] = ['import', '/dls/mx-scratch/rjgildea/Eiger/hdrmx_medsbio/cbf/P3p09_*.5d_0.05s_12.65keV_bslz4_1_00001.cbf.bz2']


full_cmd_lst[1] = ['dials.import', '/dls/mx-scratch/rjgildea/Eiger/hdrmx_medsbio/cbf/P3p09_*.5d_0.05s_12.65keV_bslz4_1_00001.cbf.bz2', 'output.datablock=dials_files/1_datablock.json', 'output.log=dials_files/1_import.log', 'output.debug_log=dials_files/1_import.debug.log']

If you're on the export node and click retry, it created a new node (and names it export, but it's actually integrate), however if you run it then it tries an integrate run but fails and creates a red integrate node.
A user may do this to try and export the output of the already run integrate with a new output name.
Solution - you shouldn't be able to click retry on export.

As in title. The way a user selects the images required in iMosflm is really useful. Also the way of ordering the files i.e. most recent first, name ascending etc.

would like to add a mask before indexing

Requires:

• drawing tools for circle, rectangle
• drawing tools for polygon
• passing details of the shape drawn to dials.generate_mask
• automatic run of dials.apply_mask to include the mask in the datablock.json or experiments.json (see also dials/dials#603)
• display of masked areas by e.g. filling masked shapes with block colour

In the simple and advanced tabs of each process to run there are a number of parameters that can be edited. It may be useful to include more information about what each of these parameters is doing to help the user decide what would be good to change. This could be a pop up that appears when you hover over or a '?' button that when clicked gives a pop up window with more information.

am testing it now from my desktop (ws072) on a dataset with 1500 images.

it took 1min+ to load the image names for the chosen dataset (that’s bad).
it’s taking minutes to find spots – is that normal? Took 2 minutes in the end.
I like that the next step is the only button available
the bravais table didn’t pop up first time round. The shell was prompting me to choose the lattice but I had to click on the history tree to eventually have the table popping up.
I like the history tree.
in the bravais lattice table, I got confused with cell parameters because some are in decimal, some (if number >100) are in scientific notation.
The diffraction image does not show what image I’m looking at. Is it a raw image or is it an average? Or something else?
The report view tab is great
The report (and GUI in general) does not show there’s a problem. I only noticed because the log file turned red in the end. There’s an error that does not cause any visual feedback elsewhere.
I used dataset /dls/labxchem/data/2017/lb16993-1/processing/beamline/lb16993-2/DM15/DM15-x0009/DM15-x0009_1_####.cbf
Resolution rings in the image would be great.

Luis, I like it, it’s neat and compact. There are some snags, but looks very good already.

Cheers,
Jose’

Traceback (most recent call last):
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/integrate.py", line 568, in
halraiser(e)
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/integrate.py", line 566, in
script.run()
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/integrate.py", line 325, in run
reflections = integrator.integrate()
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/integration/integrator.py", line 1150, in integrate
self.reflections, _, time_info = processor.process()
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/integration/processor.py", line 271, in process
preserve_exception_message = True)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/util/mp.py", line 171, in multi_node_parallel_map
preserve_exception_message = preserve_exception_message)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/util/mp.py", line 53, in parallel_map
preserve_exception_message = preserve_exception_message)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/easy_mp.py", line 623, in parallel_map
result = res()
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in call
self.traceback( exception = self.exception() )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 115, in call
self.raise_handler( exception = exception )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/mainthread.py", line 100, in poll
value = target( *args, **kwargs )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/util/mp.py", line 91, in call
preserve_exception_message = self.preserve_exception_message)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/easy_mp.py", line 623, in parallel_map
result = res()
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/result.py", line 119, in call
self.traceback( exception = self.exception() )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 115, in call
self.raise_handler( exception = exception )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 145, in raise_with_stacktrace
self.stacktrace( exception = exception )
File "/dls_sw/apps/dui/dials_w_miniconda/modules/cctbx_project/libtbx/scheduling/stacktrace.py", line 86, in call
raise exception
RuntimeError: Please report this error to [email protected]:
There was a problem allocating memory for shoeboxes. Possible solutions
include increasing the percentage of memory allowed for shoeboxes or
decreasing the block size. This could also be caused by a highly mosaic
crystal model - is your crystal really this mosaic?
Total system memory: 6.26399 GB
Limit shoebox memory: 4.69799 GB
Required shoebox memory: 12.7337 GB

Maybe call it 'expert user', to me it means 'expert mode' otherwise, i.e. the interface will do everything

It would be good to have more control on the output file name/location. Also, pressing export on history tree should provide the file name and path of what was written.

In the simple and advanced tabs of each process to run there are a number of parameters that can be edited. Often the naming of these isn't sensible, not making it obvious what these parameters are actually doing. These could be re-named to some more sensible/obvious.

E.g. hovering over any point should show you the resolution at that point - in the status bar (with x, y, I) would be great.

Additionally, showing a live 'circle' showing the resolution, where you've hovered over sounded very useful to people in the room - making it easier to see anisotropic reflections was mentioned.

• Show resolution in the status bar

I go to a directory and want to select some images... I wait

and wait

several minutes looking at

On the reindex tab, it would be nice to see the predicted spots without having to close the window. Single clicking the indexing choice should update the viewer. Currently it only does this if you doubleclick, which closes the reindex window.

If we want to index a sample you may wish to only select orthogonal images, or other, rather than using all of the images in a dataset.

User should be given the choice of output directory when setting up a processing job, rather than simply dumping files in situ.

When in an advanced tab, clicking 'reset to defaults' jumps back to the 'Simple' tab. Is this desirable behaviour?

Dials and idials display a cascading structure but the output directory format is flat. This makes it a little tricky to navigate to the output file from a particular job. It might make things easier if the folder structure matched the dials interface

e.g.
dials_home/find_spots/index/refine_bravais_1
dials_home/find_spots/index/refine_bravais_2
dials_home/find_spots/index/refine_bravais_3/refine/integrate

It would also be useful to display the SG somewhere in the output names to keep track of files, maybe refine_P4_1?

I don't like the status bar telling the user what to try next

First, I think "<< index >>" etc. is ugly. Secondly, it is implicitly clear what can be done next by the graying-out of unavailable actions.

The indexing tab has a simple and advanced settings page, but the parameters on simple seem an odd choice. I'd say resolution, maybe spot I/sigI or known cell/SG would be better choices than indexing algorithm.

After running the index step and the automatic refine, the table it displays shows even three-digit numbers (e.g. column C and B2) in scientific form, when clearly the number fits. This ends up obfuscating the number somewhat.

Having a single button that starts/stops video would work better than two separate buttons.

On the image viewer there is no display of the name of the current image. It would be nice to know which frame you are looking at.

The default long step ">>" is 10 images, and step is 1 image. I can change the step size in Viewing Tools by setting "Image Jump Step", but the long step is not changed. As a result I could set ">" to, say, 15 images but ">>" will remain 10.

This is confusing. I propose that the long step defaults to the larger of 10 or 2 times the step size.

The layout on the reindexing choice tab isn't very intuitive. Would be better in my view to have lattice, cell, judgement then penalty. Rmsd, min cc and max cc don't really add a lot.

Having penalty after the OK score coloured tab would separate it and make it easier to see clear boundaries.

As a member of beamline staff I want to be able to edit the input parameters. For example, wavelength, beam centre, detector distance. I would also like to have the option of fixing these parameters. This helps to diagnose issues with the beamline. This is also useful if the header information is incorrect.

Scrolling from full image to view 2-3 spots is slow due to small step size, could we increase the default step or have it as a customisable feature?

Error after clicking "Open Reciprocal Lattice Viewer" in the reindex step but before selecting a row in the table

running Popen>>> dials.reciprocal_lattice_viewer None dials_files/lin_4_bravais_summary.json ,  True <<< 

Hi from QThread(run)  ___________________ Before Loop

############################################################################
#                                                                          #
# The DIALS reference paper has now been published:                        #
#                                                                          #
# DIALS: implementation and evaluation of a new integration package (2018) #
# https://doi.org/10.1107/S2059798317017235        Acta Cryst. D74, 85-97. #
#                                                                          #
############################################################################


Sorry: Unable to handle the following arguments:
 "None" failed repeatedly during processing:
   File None does not exist

 dials_files/lin_4_bravais_summary.json did not appear to conform to any  expected format:
   - DataBlock:      Expected __id__ 'DataBlock', but found None
   - ExperimentList: Expected __id__ 'ExperimentList', but found None
   - Reflections:    Appears to be an invalid pickle file
_________________________________________ Loop ended
after ...close()
self.kill_my_proc
time to kill 10185
attempting to kill pid #: 10185

When clicking between two finished jobs on the same history tree it takes too long to update results (due to 'Updating Image/Reflections Data' at each time).

If you are on an integrate node and click on reindex, the mtz output name box does not grey out. Edit: actually this button never greys out, for example if you click on a run integrate node.

The integrate step is immediately followed by export. That behaviour is fine, but it means that the report from the integrate step disappears and the user has to navigate backwards to see it

At the bottom of the gui where unit cell space group etc. are displayed, technically this is not a space group but infact the point group/Bravais lattice type. This should be updated accordingly.

Currently if you want to redo the indexing step, you click on find_spots in the tree. It's a little unintuitive. The buttons on the top behave as a user might expect.

Opening DUI from nothing gives the 'Next stage' button as 'Run' when it should probably be import. This means you have to click the toolbar import button explicitly.

Clicking 'import' and then cancel changes the button to say 'Import' as it probably should anyway, but clicking doesn't actually do anything (perhaps the first click should open the file dialog? It's slightly non-obvious that you have to click the import icon then the text button)

Would be nice to be able to detach a window to be able to view more than one tab (e.g. view image and log at the same time).

Point raised by Frank von Delft at the CCP4 Annual Review. The tree is great, but we need ways to mark which path is most useful

.. as in the actual images selected aren't necessarily reflected in the import. e.g. if you only select:

X4_0001.cbf
X4_0002.cbf

It gives the filter:

X4_000#.cbf

Which resolves to nine images. Probably a less directly useful example (given less common split experiment data), but selecting the two files:

X4_0001.cbf
X4_0090.cbf

Gives the filter:

X4_00##.cbf

Importing 90 files, when only two were selected!

As well as the whole red/green issue we discussed the red is very intense and makes it hard to read. When we discussed this, we talked about possibly making it a more muted/desaturated red And this is what I meant:

Which is higher contrast and easier to see the N, even without the colours. Perhaps do the same thing for the green.

Other options include patterns but without the low-contrast conflict between text and background the letter might be enough.

Detecting centroid outliers using the MCD algorithm
4096 reflections have been flagged as outliers

Summary statistics for 26268 observations matched to predictions:

| | Min | Q1 | Med | Q3 | Max |

| Xc - Xo (mm) | -0.2613 | -0.04677 | 0.001643 | 0.04439 | 0.2422 |
| Yc - Yo (mm) | -0.3084 | -0.05679 | 0.003756 | 0.06494 | 0.326 |
| Phic - Phio (deg) | -0.7391 | -0.1501 | 0.01417 | 0.1722 | 0.9084 |
| X weights | 283.5 | 392.4 | 400.7 | 404.4 | 405.6 |
| Y weights | 287.7 | 393 | 400.2 | 404.1 | 405.6 |
| Phi weights | 10.91 | 11.99 | 12 | 12 | 12 |

Traceback (most recent call last):
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/refine.py", line 406, in
halraiser(e)
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/refine.py", line 404, in
script.run()
File "/dls_sw/apps/dui/dials_w_miniconda/build/../modules/dials/command_line/refine.py", line 255, in run
reflections, experiments)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/refinement/refiner.py", line 466, in from_parameters_data_experiments
copy_experiments=copy_experiments)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/refinement/refiner.py", line 538, in _build_components
cls.config_parameterisation(params, experiments, refman, do_stills, verbosity)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/refinement/refiner.py", line 1312, in config_parameterisation
xl_ori_params, xl_uc_params)
File "/dls_sw/apps/dui/dials_w_miniconda/modules/dials/algorithms/refinement/refiner.py", line 1419, in config_restraints
if len(tie.sigmas) != 6:
TypeError: Please report this error to [email protected]: object of type 'NoneType' has no len()