DIALS core meeting 2024-02-08

Previous Actions

• ND Investigate getting psana tests running on the DIALS xfel-regression testing
• ND: Investigate using stable-nightly conda builds of cctbx (e.g. cctbx tests) longer term
• ND or DW: Cannot view reciprocal lattice using dials.reciprocal_lattice_viewer #2443
  • Probably needs someone to dig into windows gltbx behaviour, or at least work out where issue lies
  • Reports from CCP4 workshop that people are having issue on latest ubuntu
  • David reproduced the problem which is progress…
  • ASB to point Billy to this; issue affects cctbx.multiplicity_viewer too
  • ASB to bump Billy about this - Billy aware, ASB to bug him about making an issue
• ND: Investigate Graphical Fails on Windows

Agenda

• Action DGW: Arrange electron meeting with AB/MP when here
• ND/GW: MAke sure dls visitor stuff sent/chase up Mark
• Kevin Dalton presents Pink Indexer PR (informal)

ASB / DP visit to DLS

• integration of potato into dials.stills_process
• stills small cell indexer into dials.index
• GW will need time for setting things up
• ASB wants to run XFEL GUI -> some set up needed, schedule this for Monday or Tuesday

Pink Indexer

https://github.com/dials/dials/pull/2604

Pink indexer as per Acta Cryst A 2020 76 121-131 - Rick has been trying to make DIALS work for Laue, brute force indexing algorithm, like the real space grid search. Works with mono and poly chromatic data. Implementation of low resolution spot match #884 and added a new lattice search strategy and a new entry point.

Indexer class infers lattices, strategy class used upstream to do the calculations. Written almost purely in numpy and scipy: has some “hyperparameters”. It is memory intensive so limits the number of reflections e.g. to 50 - for sparse patterns. Bandwidth percentage parameter is one of the most important, also encodes an uncertainty in the cell constants - some balance between memory use and accuracy of result. Defaults found to be reasonable. PR includes a small example: this should go on the dials-data repo. Example case seems OK but is not curated reference - please try and run this and provide feedback.

Note for record: PR build failed because of lack of news fragment => we should add a contributors guide. We have this which for future reference at:

https://github.com/dials/dials/blob/main/CONTRIBUTING.md

Please can we link this more obviously on the front page documentation.

Description of algorithm from https://scripts.iucr.org/cgi-bin/paper?ae5078

Kevin notes that the current version is not well supported, does not work with recent crystFEL - so re-implementation for the greater good. Current implementation makes a mild assumption about you having a monochromatic-ish energy distribution. Laue DIALS will need a refactor.

H5 on-disk reflection table format

• JBE starting work in this area, discussed at Diamond DIALS meeting 06/02/24. First implementation just to provide analagous functionality to existing msgpack format and load to standard flex reflection table (e.g. no special memory handling). Flumpy helps with converting to standard h5 data arrays via numpy. WIP branch at https://github.com/jbeilstenedmands/dials/tree/h5_table_format
• Key idea is to be ‘dials-agnostic’ i.e. data could be written by non-dials code but understood by dials. Column names staying same as this is useful.
• Read/write times similar to msgpack (except reading large files due to absence of efficient creation of flex.int6 from N x 6 array - fixable).

• Questions arose regarding shoeboxes - don’t fit nicely into the table data model, most of memory of a pre-integration reflection table is shoebox data. GW: do we need to generate/keep these from spotfinding (we can still keep the bounding boxes which are useful)? Only need is in integration and if wanting to view maskcodes etc in image_viewer. Could generate as necessary at the point of integration, image_viewer if shoebox data not present in table.

• Aaron notes that there is to and from NeXus tools which will read and write NXreflections from DIALS reflections and experiment files. This is believed to round-trip. This is not a container switch, this is a rewrite of the file. This does not need to be reinvented (Aaron asserts.)
• Aaron agrees that the dials column names are OK, by using links so we can have the “true” name as intensity.value or whatever then make the NeXus correct version as a link to this. NXreflection is maybe being used for stress / strain measurements?
• Action: review practicalities of using this & welcome people looking at James’s work above
• Action: round trip example files could be useful - exists in dials.export
• General enthusiasm for this around the room

https://github.com/cctbx/dxtbx/issues/675

• Same as pilatus issues happening to eiger with new DECTRIS
• Dimensions, fast, slow all inverted so all this needs to be sorted out
• Graeme waiting for example data to sort this out
• There is an open question from Takanori https://github.com/cctbx/dxtbx/issues/675#issuecomment-1884404008
• GW solved by https://github.com/cctbx/dxtbx/pull/676
• 🎉 this has finally been closed

Removing setting batch offset when creating imagesets

• https://github.com/cctbx/dxtbx/pull/662
• We no longer need the batch offset => we can just remove it after changes by DGW
• Jan 2024 testing continues between Dan and James
• Remaining issues in xia2, being addressed by JBE
• Looks like we still need this functionality but don’t know why, further investigation needed

Adding xia2 to conda-forge

• This needs to happen
• Also, graphical fails on windows => ND Action: Fix these
• Diamond has some RSE time on DIALS; This on list to address
• Now work in progress
• Rick asks for having a conda package for the xfel packages: ASB acknowledges the need

JBE Demos

• Demoing structural changes to indexing/retaining a crystal-less experiment object, to explicitly link the unindexed reflections for each imageset. JBE has implemented expeditor class to use at start of command-line programs to maintain compatibility of existing experiment list structures.
• Draft PR https://github.com/dials/dials/pull/2567
• Aaron suggest running xfel.merge to test
• Extended discussion about how far through the processing pipeline the crystal-less experiment object should be kept. If possible to output after integration, then all data reduction programs must be able to handle. Conclusion was that integrated data is fundamentally different to pre-integrated data, therefore have no need to have unindexed experiments in output of integration (this can always be added in future if use case arises).
• JBE to finish PR - updating all command line programs and updates following discussion (before Christmas). Then give people chance to test in new year.
  • DGW has comments, progress being made
• Also need to check xia2, and follow up with other users to assess any impacts - Neutron work? (David McDonagh), Laue work? (K Dalton, R Hewitt).
• GW suggested making a special release in order to test this changeset at DLS (in new year).
  • Wait until PR notes addressed, then build this as a test release for wider consumption
• Aaron would like dials.stills_process to have this feature for indexed output - GW suggested using ImageSequences is a prior requirement for clean implementation. So won’t be included as part of this PR.

DIALS West visit to East

• 12-13 Feb. Aaron and Dan available
• Agenda?
• Pub trip!
• Experiment starts Wed 12
• ASB around for 2 weeks, Dan for 1

11th ASB & co. staying at the Bear, later than that staying at Ridgeway house. Could do a pub visit Sunday night? Coordinate via dials-support Slack.

AOB?

• xfel-regression tests currently have a boost warning that is causing CI failures.
• ASB: yup. C++ type wrapped twice in flex => boost error / warning - hits if you import both definitions. Fix is work in progress.
• Kevin raises https://github.com/dials/dials/issues/2564 as a “please could we” - feels reasonable to flag. DGW self-assigns.

Next meeting

Thursday, February 22nd, 4pm (GMT), 8am (PST)