DIALS core meeting 2022-04-14
- (*) DXTBX/pycbf [prev]
- ND: conda-forge pycbf: Make new release to use dials-data directly for tests
- Nick: Add lots more attribution
- (*) Nick: dxtbx and read-only base
- Nick: Make sure that HKLviewer (prev) works at Diamond on
- Attempted, installing pyside2 is easy, but fails after that (log)
- Action ND: ask RO about this
- Sent enquiries, under discussion
- Aaron: Reduce zenodo dataset for contiguous nexus
- Put on Dan Paley backburner also - time is short
- On the Todo list now!
- ND to review: TOFBeam - A time-of-flight beam in dxtbx
- Continue to discuss mac dispatcher issues (e.g. no GUI launch for image_viewer etc) in issues (
std::stringsupport for messagepack. This is used in XFEL module, should try to fix.
- Actually try to fix this
- This is not going to happen before April at the earliest
- GW: Notice and merge pathlib changes
- Version 0.9.6.5 now out
- Added more attribution in
- Now uses dials-data for testing data
- Added windows build:
dials/pycbf#21- somewhat scattered list of changes applied to CBFlib, at pycbf-build-time in
- Wheels live for multiple manylinux flavors, MacOS x86/arm, windows 64bit/32bit, for Pythons 3.6/3.7/3.8/3.9 live on pypi and, conda-forge mac/mac-arm/linux/windows 3.6/7/8/9/10.
- Can we close this item out for now?
- “Contiguous Nexus”
- Non-draft state is pending checks against issues @dwpaley raised, and checking for non-contiguous cases [prev]
- AB wants this to be applied against new nexus writing
- RG: Probably useful to mock up an example in the nxmxtests
- Action: Aaron Reduce down zenodo dataset used in the issue (
cctbx/dxtbx#93) so that we can include it in dials-data.
- Afterward: Coordinate with RG to make sure the ideas in this are integrated into new nexus reading
- WIP - Dan has reduced to Six images - 300mb, wondering if this is enough. Discussion about what is required for the files and how small to absorb into the dials-data repository.
DXTBX read-only base
- Read only base by default would be the ideal
- Previously discussed downstream use cases and if DIALS needs to be included
- PHENIX - depending on the readonly dxtbx stuff - until that is fixed DIALS will be pinned at 3.2 or similar. They also have a hard stop at Python 3.7, due to a slowly changing dependency.
- Defer for two meetings - pinning on DIALS 3.2/virtualenv script working around for now
- Originally merged in, but turned out to be more involved
- Top Priority
- Reopened under
cctbx/dxtbx#511- main change is regenerating dispatcher immediately during configuration, this mostly seems to work
- AB: going to run in next few days
- Moves to the same
src/based layout as dxtbx, xia2. This enables us to write a standard
setup.pyto actually install DIALS and it’s entry points, like a normal python package. It also provides isolation against accidental namespace imports, namespace pollution.
- Once merged, will go through other open PRs (including draft) to update them the the new layout, though this isn’t expected to be much of a problem.
- Uses the same mechanism as above dxtbx PR, e.g. regenerates the DIALS dispatchers once the environment is updated to properly recognise the package layout
- Relies on
dials/dials@cafaf4to force dials to configure before xfel, so that xfel picks up the correct locations
- Planning to merge this in over the next couple of weeks
Circular XFEL dependency
dials/dials#2076- xfel has crept up from optional to a hard dependency -
cosymand a few others use as unqualified imports
- Not sure how difficult it would be to resolve this and keep all functionality
/indexing/nave_parameters.pylook reasonably dependent - might have to accept that these won’t work outside a libtbx bootstrap.
Dropping Data Blocks
- GW PR
- After PR/PRs, ASB to contact e.g. Mike Wall about this, too.
- GW thinks reasonable to merge when these concerns addressed
- Waiting on AB now, will get sorted as soon as can
- Going to be a PR for “Diffbragg stage 1 refinement”
- Uses pixel models of image to refine crystal using polychromotic spectra as per Mendez 2020 paper.
- Implements an override in stills_process
ASB wants to put diffbrag stage 1 into dials.stills_process as a refinement engine. Depends on simtbx. This is very much dependent on GPU. Can use OMP etc.
- Derek Mendez going to send changes to Aaron
- more of a long term thing, drop until PR comes up
Changes from XFEL sessions
- Dan: Image viewer: now support ellipses instead of just rings
- And can write out detector hierarchy file
- Mentioned an issue with detector Z, DGW remembered a related-sounding issue https://github.com/cctbx/dxtbx/issues/472
- PRs incoming.
- Known orientations parts of stills_process needed to be rewritten to support this
- Processing data using stills_process and crystfel, were getting more lattices than DIALS.
- Crystfel uses alternative indexing programs (e.g. ND remembers compiling them specifically for users at DLS)
- “RANdom Subsampling And Consensus”
- Index image 50 times, randomly subsetting the data
- Cluster solutions
- In mid-feb implemented similar approach
- Reduce data in small portions until failing/getting same answer
- Still unresolved how to choose threshold for ending this
- Much higher indexing rates! 25% more lattices, and of higher quality
Thursday, April 28th, 4pm UK (BST), 8am PDT.