DIALS core meeting 2020-09-03

Previous Actions

• ND: rewrite contribution guidelines (as PR) to include towncrier/newsfragments
• MG: make Python 3.8 the new default version (→ #1373)
• MG: remove DIALS 1.14 tutorials from website (→ #1359)

Agenda

Contribution guidelines

Action: DWP will have a look over them and email comments to ND

Overall architecture discussion

full conda or alternatives?

• move from libtbx dispatchers to normal python dispatchers
  • ND: I don’t see this possible while we are using libtbx/scons
• allow regular python commands access to the libtbx-modules namespaces
• bigger issue, needs more discussion and figuring out how to do this and how to migrate
• Should we remove the DIALS installer from cctbx bootstrap?
  • We don’t advertise or test or use cctbx bootstrap any more. Anyone using it may end up with an unsupported environment.
• When cctbx becomes a conda-forge installable package then there is strong interest from the DIALS side to use that
• GW: Can we have a proposal that includes what the impact, benefits, and costs are.
  • Action: MG & ND to do that

New installer

Following from the overall architecture discussion: How should a new installer look like?

cf. dials#1198 and particularly dials#1198 (comment)

stable master branch

Issue: dials#1353

• Report result from 1+ month evaluation on xia2
  • Actual value was observed in xia2
• How would we distinguish with pull requests that should be reviewed and pull requests that can be merged as soon as the tests have passed?
  • MG: labels may be an option? GitHub plans to introduce an auto-merge feature for pull requests later this year. We could probably do this right now using github actions.
  • GW: pull requests can be moved to draft status to express “hang fire, I would like to look at”
  • MG: A label saying “merge as soon as tests pass” may alleviate concerns from ND and myself about pull requests with a very short shelf-life
  • ASB: Sounds like something that should be mentioned in CONTRIBUTING
  • ASB: cctbx is running CI jobs in branches in batch mode. May want to consider that for DIALS/dxtbx
• Fundamentally it’s possible that tests fail because of an upstream repository change (eg. cctbx/scitbx/constants breaking xia2)
  • May have to monitor this. This could be resolved by using semantically versioned upstream releases (for cctbx_project) and stable build mechanism (for dxtbx) but comes with other drawbacks.
• Agreement: update CONTRIBUTING to include a soft-imposition in a (separate) pull request.
  • Action for ND

dxtbx json/msgpack performance

(not discussed beyond adding the action below)

ASB reports json being problematic when importing e.g. 50,000 experiments, as everything is stored in a single file, so everything needs to be read at once.

• GW: cf. Load before heat death (dxtbx#118). At the core of it are bad design choices in dxtbx.
• Future discussion topics:
  1. treatment of large serial datasets
  2. using an HDF5 backend for reflection files
• ASB: I want check_format to go away.
• GW: discuss in a separate meeting early US, late UK time
• Action GW: set up this meeting

Typing

(not discussed)

Options:

• mypy pre-commit, mypy action (#1357)
• pytype action (#1364)

What do we want to do?

DIALS documentation build

(not discussed)

Should the DIALS documentation build outside a cctbx environment?

• Currently you can only build the DIALS documentation inside a cctbx environment. This means a remote documentation build needs to build cctbx first, so we can’t easily run the documentation build on every commit. When things break they are difficult to fix. We can’t use readthedocs.
• MG: I believe the main obstacles are three cctbx Sphinx plugins that may need to be extracted or removed from DIALS documentation, and some phil parsing logic.

renaming master branch → main

(not discussed)

• Nothing much will happen on this any time soon - either way we’ll wait for GitHub support on this one
• General assent that this is useful
• from 1st of October new GitHub repositories will have main as the default
• MG: Suggest turning this into an issue and removing from agenda until GitHub provides a migration path for existing repositories.

Any other business

(not discussed)

• cbflib_adaptbx dependency
  • We are only referencing compress and uncompress functions, and cbflib_adaptbx has a MSVC-ancient-dependency. Could copy the functions into dxtbx_ext and remove the dependency.

Next meeting

September 17th, 4pm UK time, 8am PDT.