DIALS core meeting 2020-08-11

Previous Actions

• ND: rewrite contribution guidelines (as PR) to include towncrier/newsfragments
• GW: write comment on dials#1327
• MG: fix xia2 issues re. Python 3.8 (xia2#510)

Relevant conda-forge migrations

package	status
Boost 1.72	91% complete
Python 3.8	94% complete
HDF5 1.10.6	79% complete

SCons 4.0 went into conda-forge over the weekend.

Agenda

Python 3.6 deprecation

Python 3.6 deprecation proposal / NEP-29 / dials#1327

Discussion outcome:

• We will follow #1327
• GW wrote comment on issue
• Item can be removed from agenda
• Worth documenting outcome in the /kb repository?
  • GW: How about collecting in a GitHub project in columns ‘proposed’ & ‘implemented’

Next default Python version

Suggestion: moving to Python 3.8 as the next default, skipping 3.7 (cf. #1328-comment)

• Development platform is independent of supported versions. Those will include conda-forge-supported versions at the minimum.
• xia2 currently has issues with 3.8
  • these have now been resolved (xia2#510)
• XFEL regression tests run on 3.8
• Action MG: Make it so.

Overall architecture discussion

left for next meeting as ND absent

full conda or alternatives?

• move from libtbx dispatchers to normal python dispatchers
• allow regular python commands access to the libtbx-modules namespaces
• bigger issue, needs more discussion and figuring out how to do this and how to migrate
• Should we remove the DIALS installer from cctbx bootstrap?
  • We don’t advertise or test or use cctbx bootstrap any more. Anyone using it may end up with an unsupported environment.

New installer

left for next meeting as ND absent

Following from the overall architecture discussion: How should a new installer look like?

Open up DIALS core meetings

Open up these DIALS core meetings to DIALS-W and the world in an effort to move DIALS towards a stage 3 project

• Invite to meetings in dials-support and gitter
• Change time to later in the day
  • try 4pm UK time, 8am US time
• Make meeting minutes public
• Item can be removed from agenda?
  • Yes

DIALS master stable

DIALS master stable / dials#1353

• Currently evaluating on xia2 (1 month = 24th of August)
• How would we distinguish with pull requests that should be reviewed and pull requests that can be merged as soon as the tests have passed?
  • MG: labels may be an option? GitHub plans to introduce an auto-merge feature for pull requests later this year. We could probably do this right now using github actions.
  • GW: pull requests can be moved to draft status to express “hang fire, I would like to look at”
  • MG: A label saying “merge as soon as tests pass” may alleviate concerns from ND and myself about pull requests with a very short shelf-life
  • ASB: Sounds like something that should be mentioned in CONTRIBUTING
  • ASB: cctbx is running CI jobs in branches in batch mode. May want to consider that for DIALS/dxtbx
• Fundamentally it’s possible that tests fail because of an upstream repository change (eg. cctbx/scitbx/constants breaking xia2)
  • May have to monitor this. This could be resolved by using semantically versioned upstream releases (for cctbx_project) and stable build mechanism (for dxtbx) but comes with other drawbacks.

dxtbx json/msgpack performance

ASB reports json being problematic when importing 91208312 experiments, as everything is stored in a single file, so everything needs to be read at once.

• GW: cf. Load before heat death (dxtbx#118). At the core of it are bad design choices in dxtbx.
• Future discussion topics:
  1. treatment of large serial datasets
  2. using an HDF5 backend for reflection files
• ASB: I want check_format to go away.
• GW: discuss in a separate meeting early UK, late US time

Typing

left for next meeting due to time constraints

Options:

• mypy pre-commit, mypy action (#1357)
• pytype action (#1364)

What do we want to do?

DIALS 1.14 tutorials

Can we remove the DIALS 1.14 tutorials from the website? CCP4 is now distributing 2.2 and 1.14 is no longer supported.

• Action MG: remove

DIALS documentation build

left for next meeting due to time constraints

Should the DIALS documentation build outside a cctbx environment?

• Currently you can only build the DIALS documentation inside a cctbx environment. This means a remote documentation build needs to build cctbx first, so we can’t easily run the documentation build on every commit. When things break they are difficult to fix. We can’t use readthedocs.
• MG: I believe the main obstacles are three cctbx Sphinx plugins that may need to be extracted or removed from DIALS documentation, and some phil parsing logic.

Any other business

• renaming master branch :arrow_right: main
  • Nothing much will happen on this any time soon - either way we’ll wait for GitHub support on this one
  • General assent that this is useful

Next meeting

GW suggests first week of September rather than today + 2 weeks