DIALS core meeting 2024-04-04

Previous Actions

• ND Investigate getting psana tests running on the DIALS xfel-regression testing
• ND: Investigate using stable-nightly conda builds of cctbx (e.g. cctbx tests) longer term
• ND or DW: Cannot view reciprocal lattice using dials.reciprocal_lattice_viewer #2443
  • Probably needs someone to dig into windows gltbx behaviour, or at least work out where issue lies
  • Reports from CCP4 workshop that people are having issue on latest ubuntu
  • David reproduced the problem which is progress…
  • ASB to point Billy to this; issue affects cctbx.multiplicity_viewer too
  • ASB to bump Billy about this - Billy aware, ASB to bug him about making an issue
  • Billy sorted out the gltbx stuff 🎉
• ND: Investigate Graphical Fails on Windows
• JBE: H5 on-disk experiment: review practicalities of using this & welcome people looking at James’s work above. round trip example files could be useful - exists in dials.export. No progress since last meeting.
• ND/Naman: Adding xia2 to conda-forge - is on conda-forge, not extensively tested

Agenda

Removing setting batch offset when creating imagesets

• https://github.com/cctbx/dxtbx/pull/662
• We no longer need the batch offset => we can just remove it after changes by DGW
• Jan 2024 testing continues between Dan and James
• Remaining issues in xia2, being addressed by JBE
• Looks like we still need this functionality but don’t know why, further investigation needed
• Scheduled batch-offset summit next week; further news after that
• Started looking at at DIALS-East last week, ain’t gonna get done any time soon since it’s quite a quagmire
• Fixing counting from 0, 1 problems will involve breaking changes, not backwards compatible probably. Because this is not affecting stills since this affects Scan etc. - image indices may be affected. Need to do an analysis of this
• Does this mean we also need to version data files? So at least can fail noisily if we try to read old format data with new code
• We have a separate tool to explicitly bring data files up to date and make new format files fail to deserialise with old code?
• Then we make this DIALS 4.0: this is a long way in the future

Adding xia2 xfel packages to conda-forge

• Rick asks for having a conda package for the xfel packages: ASB acknowledges the need
• AB has made progress; has a builder; just needs to get back to
• No progress, probably none until summer, so move off agenda

JBE Demos

• Previously:
  • Demoing structural changes to indexing/retaining a crystal-less experiment object, to explicitly link the unindexed reflections for each imageset. JBE has implemented expeditor class to use at start of command-line programs to maintain compatibility of existing experiment list structures.
  • Draft PR https://github.com/dials/dials/pull/2567
  • Aaron suggest running xfel.merge to test
  • Extended discussion about how far through the processing pipeline the crystal-less experiment object should be kept. If possible to output after integration, then all data reduction programs must be able to handle. Conclusion was that integrated data is fundamentally different to pre-integrated data, therefore have no need to have unindexed experiments in output of integration (this can always be added in future if use case arises).
  • JBE to finish PR - updating all command line programs and updates following discussion (before Christmas). Then give people chance to test in new year.
    • DGW has comments, progress being made
  • Also need to check xia2, and follow up with other users to assess any impacts - Neutron work? (David McDonagh), Laue work? (K Dalton, R Hewitt).
  • GW suggested making a special release in order to test this changeset at DLS (in new year).
    • Wait until PR notes addressed, then build this as a test release for wider consumption
  • Aaron would like dials.stills_process to have this feature for indexed output - GW suggested using ImageSequences is a prior requirement for clean implementation. So won’t be included as part of this PR.
• Not been touched much since NY
• Keeps coming up as something that would be useful
• JBE: Rebase this PR
• AB: Run this against Xfel-regression tests: AB pls can you run this James?
• Merge in after that (or in two weeks at next core)
• ND: Make test release using these branches
• Need to consider interaction/removing the need for imageset IDs
• James has rebased this, updated tests, Rick and Kevin feel OK with this => need to pick up some PR comments
• Hope to merge in near future (next month)
• Aaron points to notes

XFEL-Regression

• xfel-regression tests currently have a boost warning that is causing CI failures.
  • ASB: yup. C++ type wrapped twice in flex => boost error / warning - hits if you import both definitions. Fix is work in progress.
  • This is fixed squashed!
• Kevin raises https://github.com/dials/dials/issues/2564 as a “please could we” - feels reasonable to flag. DGW self-assigns.
  • Done! In https://github.com/dials/dials/pull/2605!

AOB?

• Hit indices https://github.com/cctbx/dxtbx/pull/709
  • All happy to merge, discussion about making this a wider fix

“XTC Streams, Heat Death, and You” Discussion

• Chat transcript from Slack (problems with FormatXTC in a non-MPI application):
  • dials.stills_process using FormatXTC uses lazy load so that creating an experiment list happens very quickly, then it splits the list up between mpi ranks, who individually call load_models. Thus a full imported expt file is never created.
  • Note that cctbx.xfel.xtc_process is retired code that interacts directly with psana to create cbf handles in memory, but FormatXTC is the future so the old xtc_process code ain’t relevant anymore
  • Note to the note: psana has multiple modes: idx mode is what FormatXTC uses, but it’s not as fast as smd/rax which is what xtc_process uses. Problem is if FormatXTC was to use smd/rax it would also be required by the psana api to import MPI, which is a bad idea for format classes. Aaron’s tried it, it’s not a good idea. MPI should be in the application, not the Format class, unless something drastic changes to dxtbx

Next meeting

Thursday, April 18th, 4pm (BST), 8am (PDT)