DIALS core meeting 2021-05-20

Previous Actions

(Note from ND: Marking items with further discussion on the agenda below with *)

• MG: organise a typing intro lecture, likely to happen in June.
• dx2 proposal
  • MG to put in a PR to create a proposal in the proposal-space
  • MG+ND: Come up with a proposal to move away from all code being in header files and consolidate into a single library
• Move to hdf5plugin.
  • GW/MG: Make pull request for moving to hdf5plugin package
  • Wait for the 3.5 release before moving over
• *cbflib conda-forge package
  • ND: Try using old cbflib 0.9.6 (allowing regenerated pycbf) to see if it would be suitable for DIALS/dxtbx to release just the old version
• ND to set up a cbflib_adaptbx pull request for MAR detector fixes. Verify with labelit_regression
• ND: Remove black autoupdate from DLS “Clutter” job
• *XFEL-regression tests running on DIALS azure (prev)
  • MG: Merge dxtbx#351 to use prebuilt cctbx packages
  • MG: Evaluate XFEL-regression pull request dials/dials#1671
• *MG: Put in proposal (“DC3”) to follow NEP-29 directly for Python deprecations

Agenda

Image viewer with two-theta offsets

• dials/dials#1715 “dials.image_viewer fundamentally broken if 2θ != 0”
• Means image viewer broken for I19-1 data
• Fallout from dials/dials#1182 (“Single panel special cases”) is somewhat larger than we had understood (saving, angle bikeshedding)
• Rather urgent to fix for next run (starts June 15th)
• Proposed fix: dials/dials#1716 “add option to display in the best fit frame”
• Discuss
  • Problems working out how to make this default/change defaults
  • Consensus seems to be to merge
  • Problems all gone away.
• Outcome: Merge for 3.5

cbflib conda-forge package

• From past discussions it is clear that getting a newer release of cbflib is currently an insurmountable problem
  • Making changes somewhat problematic as test suite failed for a long time
  • Debian support for raw package very important. Must not break anything underneath their 18 patches, even for new versions
  • pycbf tests never been run on python 3, longstanding problems and pain-points e.g. SWIG_PYTHON_STRICT_BYTE_CHAR
  • HJB estimates 3 months of graduate student effort to fix
  • (In ND personal view) doesn’t seem keen to have us take over maintenance burden
• Need to investigate alternative methods
• Have split off just the pycbf parts of CBFlib into dials/pycbf, based on the 0.9.6 sources
  • Currently doesn’t touch CBFlib sources, generates off of 0.9.6 tag (windows compatibility will need patches)
  • pip install pycbf works today - gets 0.9.6.2
    • 0.9.6.0 gets plain pycbf, regenerated from literate sources via nuweb, CBFlib.html, SWIG 4.0, with SWIG_PYTHON_STRICT_BYTE_CHAR
    • conda-forge staged recipe ndevenish/stages-recipe working but unsubmitted
  • 0.9.6.2 has:
    • Uses dials-data (branch right now) for regression data
    • Expose HAS_SWIG_PYTHON_STRICT_BYTE_CHAR as a python variable to help aid transition away
    • Add missing binding to cbf_read_buffered_file - which we manually bind in dxtbx
    • CBFlib functions that accept basic char *strings will accept both bytes and str objects
    • Basic libimg bindings to allow python-only FormatMarIP support
  • Doesn’t currently support: Windows, HDF5, custom libtiff fork, cbf-REGEX, use as linking/#include target. Extra dependencies are hard, building manylinux wheels.
• Usage in dxtbx depends on removing binary CBFlib dependencies:
  • Binary dependency on cbflib in FormatMarIP via iotbx.detectors.marIP (pointed out here)
    • Somewhat simple to resolve - new libimg bindings built for this, have WIP branch ndevenish/dxtbx@marrpy to make pure python
  • Binary dependency on cbflib for FormatCBFFull DetectorBase interfaces
    • Less simple to resolve without probing and rewriting large parts of CBFAdaptor
    • This is initialised by everything that is based on FormatFullCBF - but only used by one file that only matches FormatFullCBF in the regression tests - SPring8_ADSC_SN916/Xtal17-2phi_3_015.cbf
    • This image is root cause of only two tests that fail (when detectorbase regression tests are skipped)
  • WIP branch at cctbx/dxtbx#368 to do most of this
  • What is Detectorbase otherwise supporting still? Could we only initialise it if requested directly?
• What is left to consider?
  • What’s the usage status of DESY_BW7B/mar345_01_001.mar2300 - is this okay to use in regression tests?
    • need an ‘OK’ to publish the file, should probably be added to dials-data/image-examples. (The file is also used in dxtbx regression tests)
  • Have attempted to run labelit and labelit_regression as requested
    • labelit tests fail on 3.9 - seemingly because they have hardcoded pickle sizes. But passes on 3.6.
    • labelit_regression test suite fails to run with AssertionError: Duplicate tests found
    • Also, $ grep -r dxtbx labelit suggests that it does not use dxtbx?
    • In any case, since tests don’t seem to have worked for a while cannot verify that they are still working
    • Aaron: Look into labelit-regression tests
      • Going to think about, speak to HJB

dxtbx#288 “Construct Experiments directly” post-merge outstanding issues

• dxtbx#336 - DataBlockFactory broken for single images
• XFAIL test for this now put in: dxtbx#337
• Discussion: DataBlock has been deprecated for a long time in DIALS, if someone is using DataBlock they probably don’t want to/can’t use latest DIALS Anyway
• Outcome: No further action
  • Leave ticket for a while and see if any problems caused, revisit later in the year

Python 3.6 Support

• Releases have been built with 3.8 for several versions
• We had a previous agreement in place in dials#1327 to follow conda-forge. At the time, we understood that conda-forge was following NEP-29, which they have now diverged from.
• “DC3” now proposed in dials/kb#5
• Do we want to consider dropping support for 3.6
• LBL: To ask Billy what the phenix plans are for 3.series
• Merge DC3 draft – done: https://dials.github.io/kb/proposals/dc3

Stop cctbx testing at Diamond?

We currently run Jenkins builds for cctbx on Linux and MacOS. At some point we started seeing transient network issues, a fix for this has now been merged (cctbx#614). Build times are between 5 and 35 minutes, and recently we ran on average 3 times a day on Linux and once a day on MacOS. Does running these tests ourselves still add anything? cctbx now has good coverage on Azure, and we rarely (if ever?) commit directly to master, and I suspect most of us only pull from the stable branch.

Outcome: Remove

bootstrap: make --mamba the default

So far we haven’t seen any issues using micromamba on Azure. Should we make this the default?

Outcome: Make it so.

XFEL regression tests on Azure

This has been merged, dials/dials#1707.

• What’s required now?

Only superficial changes remain: currently when a pull request is built, you get the green ticks from the commit build. After this, you do not have any indication that the pull request build is still running until, about 10 minutes later, those results come in. MG will pick this up at some point.

• fix incoming in dials#1718

AOB: Dials regression

• Aaron worked out how to split dials_regression
• Three folders for non-regression-data stuff
  • dials2-scaling
  • doc
  • duelling_profiles
• Everything else data and python scripts
• Nick: Thought main problem was provenance e.g. are we allowed to publish all the data sources
  • If this was solvable we could add them to dials-data - even add the ability to pull from LFS
• Aaron: Intends to go through and work this out
• Markus: There is already a dials-data image_examples definition, which includes those files from dials-regression where we know the provenance and have permission to publish them. Would rather we add to that than go back to the model where one needs to check out test dependencies as a separate step. The exact list of files in dials-regression that we still use is also known and was made explicit in dxtbx#346.

Deferred to next meeting

Next meeting

Thursday, June 3rd, 4pm UK (BST), 8am PDT.