DIALS core meeting 2022-04-14

Previous Actions

• (*) DXTBX/pycbf [prev]
  • ND: conda-forge pycbf: Make new release to use dials-data directly for tests
  • Nick: Add lots more attribution
• (*) Nick: dxtbx and read-only base
• Nick: Make sure that HKLviewer (prev) works at Diamond on module load
  • Attempted, installing pyside2 is easy, but fails after that (log)
  • Action ND: ask RO about this
  • Sent enquiries, under discussion
• Aaron: Reduce zenodo dataset for contiguous nexus
  • Put on Dan Paley backburner also - time is short
  • On the Todo list now!
• ND to review: TOFBeam - A time-of-flight beam in dxtbx cctbx/dxtbx#492 (prev)
• Continue to discuss mac dispatcher issues (e.g. no GUI launch for image_viewer etc) in issues (cctbx/cctbx_project#739)
• std::string support for messagepack. This is used in XFEL module, should try to fix. dials/dials#1858.
  • Actually try to fix this
  • This is not going to happen before April at the earliest
• GW: Notice and merge pathlib changes dials/dials#2038

Agenda

PyCBF

• Version 0.9.6.5 now out
• Added more attribution in dials/pycbf@b8d4f6 and conda-forge/pycbf-feedstock#10
• Now uses dials-data for testing data
• Added windows build: dials/pycbf#21 - somewhat scattered list of changes applied to CBFlib, at pycbf-build-time in 002_windows.patch
• Wheels live for multiple manylinux flavors, MacOS x86/arm, windows 64bit/32bit, for Pythons 3.6/3.7/3.8/3.9 live on pypi and, conda-forge mac/mac-arm/linux/windows 3.6/7/8/9/10.
• Can we close this item out for now?
  • Yes

Contiguous Nexus

• “Contiguous Nexus” cctbx/dxtbx#356
  • Non-draft state is pending checks against issues @dwpaley raised, and checking for non-contiguous cases [prev]
• AB wants this to be applied against new nexus writing
• RG: Probably useful to mock up an example in the nxmxtests
• Action: Aaron Reduce down zenodo dataset used in the issue (cctbx/dxtbx#93) so that we can include it in dials-data.
• Afterward: Coordinate with RG to make sure the ideas in this are integrated into new nexus reading
• WIP - Dan has reduced to Six images - 300mb, wondering if this is enough. Discussion about what is required for the files and how small to absorb into the dials-data repository.

DXTBX read-only base

• Issue cctbx/dxtbx#444
• Read only base by default would be the ideal
• Previously discussed downstream use cases and if DIALS needs to be included
  • PHENIX - depending on the readonly dxtbx stuff - until that is fixed DIALS will be pinned at 3.2 or similar. They also have a hard stop at Python 3.7, due to a slowly changing dependency.
• Defer for two meetings - pinning on DIALS 3.2/virtualenv script working around for now
• Originally merged in, but turned out to be more involved
• Top Priority
• Reopened under cctbx/dxtbx#511 - main change is regenerating dispatcher immediately during configuration, this mostly seems to work
• AB: going to run in next few days

DIALS src/ layout

• https://github.com/dials/dials/pull/2077
• Moves to the same src/ based layout as dxtbx, xia2. This enables us to write a standard setup.py to actually install DIALS and it’s entry points, like a normal python package. It also provides isolation against accidental namespace imports, namespace pollution.
• Once merged, will go through other open PRs (including draft) to update them the the new layout, though this isn’t expected to be much of a problem.
• Uses the same mechanism as above dxtbx PR, e.g. regenerates the DIALS dispatchers once the environment is updated to properly recognise the package layout
• Relies on dials/dials@cafaf4 to force dials to configure before xfel, so that xfel picks up the correct locations
• Planning to merge this in over the next couple of weeks

  Circular XFEL dependency

• dials/dials#2076 - xfel has crept up from optional to a hard dependency - combine_experiments, cosym and a few others use as unqualified imports
• Not sure how difficult it would be to resolve this and keep all functionality
• stills_process//indexing/nave_parameters.py look reasonably dependent - might have to accept that these won’t work outside a libtbx bootstrap.

Dropping Data Blocks

https://github.com/cctbx/dxtbx/pull/504

• GW PR
• After PR/PRs, ASB to contact e.g. Mike Wall about this, too.
• GW thinks reasonable to merge when these concerns addressed
• Waiting on AB now, will get sorted as soon as can

Still process

• Going to be a PR for “Diffbragg stage 1 refinement”
  • Uses pixel models of image to refine crystal using polychromotic spectra as per Mendez 2020 paper.
  • Implements an override in stills_process

ASB wants to put diffbrag stage 1 into dials.stills_process as a refinement engine. Depends on simtbx. This is very much dependent on GPU. Can use OMP etc.

• Derek Mendez going to send changes to Aaron
• more of a long term thing, drop until PR comes up

Changes from XFEL sessions

• Dan: Image viewer: now support ellipses instead of just rings
• And can write out detector hierarchy file
  • Mentioned an issue with detector Z, DGW remembered a related-sounding issue https://github.com/cctbx/dxtbx/issues/472
• PRs incoming.
• Known orientations parts of stills_process needed to be rewritten to support this

Ransac indexing

• Processing data using stills_process and crystfel, were getting more lattices than DIALS.
• Crystfel uses alternative indexing programs (e.g. ND remembers compiling them specifically for users at DLS)
• “RANdom Subsampling And Consensus”
  • Index image 50 times, randomly subsetting the data
  • Cluster solutions
• In mid-feb implemented similar approach
  • Reduce data in small portions until failing/getting same answer
  • Still unresolved how to choose threshold for ending this
  • Much higher indexing rates! 25% more lattices, and of higher quality

Next meeting

Thursday, April 28th, 4pm UK (BST), 8am PDT.