DIALS core meeting 2024-04-18

Previous Actions

• ND Investigate getting psana tests running on the DIALS xfel-regression testing
• ND: Investigate using stable-nightly conda builds of cctbx (e.g. cctbx tests) longer term
• ND or DW: Cannot view reciprocal lattice using dials.reciprocal_lattice_viewer #2443
  • Billy sorted out the gltbx stuff 🎉
  • DIALS 3.19 now out using v2024.3
• JBE: H5 on-disk experiment: review practicalities of using this & welcome people looking at James’s work above. round trip example files could be useful - exists in dials.export.

Agenda

Removing setting batch offset when creating imagesets

• https://github.com/cctbx/dxtbx/pull/662
• We no longer need the batch offset => we can just remove it after changes by DGW
• Jan 2024 testing continues between Dan and James
• Remaining issues in xia2, being addressed by JBE
• Looks like we still need this functionality but don’t know why, further investigation needed
• Scheduled batch-offset summit next week; further news after that
• Started looking at at DIALS-East last week, ain’t gonna get done any time soon since it’s quite a quagmire
• Fixing counting from 0, 1 problems will involve breaking changes, not backwards compatible probably. Because this is not affecting stills since this affects Scan etc. - image indices may be affected. Need to do an analysis of this
• Does this mean we also need to version data files? So at least can fail noisily if we try to read old format data with new code
• We have a separate tool to explicitly bring data files up to date and make new format files fail to deserialise with old code?
• Then we make this DIALS 4.0: this is a long way in the future
  • Found common thread of OptionParser; this is planned to look at

JBE Demos

• Previously [prev]:
  • Draft PR https://github.com/dials/dials/pull/2567
  • GW suggested making a special release in order to test this changeset at DLS (in new year).
    • Wait until PR notes addressed, then build this as a test release for wider consumption
  • Not been touched much since NY
  • Keeps coming up as something that would be useful
• James has rebased this, updated tests, Rick and Kevin feel OK with this => need to pick up some PR comments
• Hope to merge in near future (next month)
• Aaron points to notes
• Previous Actions: Still relevant?
  • AB: Run this against Xfel-regression tests: AB pls can you run this James?
  • ND: Make test release using these branches
• Few more issues outstanding
• 3.19 out now so good time to merge testing - providing xfel-regression works

XFEL-Regression

• Previous boost problem causing test failures - squashed.
• Now back to:

  PackagesNotFoundError: The following packages are not available from current channels:
  
  - hdf5[version='<1.13.*']
  

---

• Hit indices https://github.com/cctbx/dxtbx/pull/709
  • All happy to merge, discussion about making this a wider fix
  • ND Merged 2024-04-18

“XTC Streams, Heat Death, and You” Discussion

• (ND unsure if this was resolved)
• Chat transcript from Slack (problems with FormatXTC in a non-MPI application):
  • dials.stills_process using FormatXTC uses lazy load so that creating an experiment list happens very quickly, then it splits the list up between mpi ranks, who individually call load_models. Thus a full imported expt file is never created.
  • Note that cctbx.xfel.xtc_process is retired code that interacts directly with psana to create cbf handles in memory, but FormatXTC is the future so the old xtc_process code ain’t relevant anymore
  • Note to the note: psana has multiple modes: idx mode is what FormatXTC uses, but it’s not as fast as smd/rax which is what xtc_process uses. Problem is if FormatXTC was to use smd/rax it would also be required by the psana api to import MPI, which is a bad idea for format classes. Aaron’s tried it, it’s not a good idea. MPI should be in the application, not the Format class, unless something drastic changes to dxtbx
• Sorted for now, James needs to investigate more

AOB?

• Discussion about find_spots_server and whether removing the tests was the right thing to do. We all agreed that nobody wanted to work on it but that we don’t exactly like the situation. There are definitely other people out there who are using it.
• Discussion on David McD’s https://github.com/cctbx/dxtbx/pull/702. Discussion came to the general conclusion that this is sensible, with a couple of minor tweaks.

Next meeting

Thursday, May 2nd, 4pm (BST), 8am (PDT)