View on GitHub

DIALS // knowledgebase

community resources, howtos, meeting minutes

DIALS core meeting 2021-05-06


Previous Actions


hdf5plugin performance evaluation

HDF5 performance evaluation – comparing hdf5plugin with hdf5-external-plugins package. simply read 3,600 images from an Eiger 16M data set:

import h5py
import sys
import time

with h5py.File(sys.argv[1], "r") as f:
    shape = f["/entry/data/data"].shape

    t0 = time.time()
    for j in range(shape[0]):
        image = f["/entry/data/data"][j]
    t1 = time.time()

    print(f"Reading {shape[0]} images took {(t1-t0):.2f}s")

Using hdf5-external-plugins:

$ time dials.python ../data/*nxs
Reading 3600 images took 270.23s

real    4m30.382s
user    2m46.693s
sys     1m43.610s

With hdf5plugin:


import hdf5plugin

to imports then run:

$ time dials.python ../data/*nxs
Reading 3600 images took278.19s

real    4m38.331s
user    2m39.540s
sys     1m58.604s

8s of 280 or so more time (2.5%) → this may or may not be a thing we care about, but the performance is comparable. EOF.

Outcome: We will move to hdf5plugin, make a pull request and then review.

Adding XFEL-regression to dials Azure tests (dials#1671)

Outcome: Merge dxtbx#351, then MG to look at the actual XFEL-regression pull request

cbflib conda-forge package

dxtbx#288 “Construct Experiments directly” post-merge outstanding issues

Python 3.6 Support

We currently have an agreement in place in dials#1327 to follow conda-forge which aim to follow NEP-29. This is less well defined than following NEP-29.

Outcome: Put in a proposal (“DC3”) to follow NEP-29 directly, with a bit of additional allowance around releases, as in dials#1327.

⇒ DC3 proposal written up in dials/kb#5

Updating pre-commits

We currently update the black pre-commit whenever a new version appears. While this is currently a rare occurrence, it is annoying for one commit (due to the installation time), and possibly requires updating other packages in the environment (eg. pre-commit itself). Q: Should we stop automatically updating pre-commits and turn those updates into conscious moves?

Outcome: Yes. ND to remove.

Next meeting

Thursday, May 20th, 4pm UK (BST), 8am PDT.