dials.algorithms.refinement¶
Contains classes used to manage the reflections used during refinement, principally ReflectionManager.
- class dials.algorithms.refinement.reflection_manager.BlockCalculator(experiments, reflections)[source]¶
Bases:
object
Utility class to calculate and set columns in the provided reflection table, which will be used during scan-varying refinement. The columns are a ‘block’ number and an associated ‘block_centre’, giving the image number in the centre of the block
- class dials.algorithms.refinement.reflection_manager.LaueReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None, wavelength_weight=10000000.0)[source]¶
Bases:
ReflectionManager
- __init__(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None, wavelength_weight=10000000.0)[source]¶
- experiment_type = 'laue'¶
- class dials.algorithms.refinement.reflection_manager.ReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)[source]¶
Bases:
object
A class to maintain information about observed and predicted reflections for refinement.
This new version keeps the reflections as a reflection table. Initialisation is not complete until the ReflectionManager is paired with a target function. Then, prediction can be done, followed by outlier rejection and any random sampling to form the working subset.
- __init__(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)[source]¶
- experiment_type = 'scans'¶
- filter_obs(sel)[source]¶
Perform a flex array selection on the managed observations, so that external classes can filter according to criteria not available here
- finalise(analysis=None)[source]¶
Complete initialisation by performing outlier rejection and any requested subsetting. If a list of results from a CentroidAnalysis object is provided, these may be used to determine outlier rejection block widths
- get_accepted_refs_size()[source]¶
Return the number of observations that pass inclusion criteria and can potentially be used for refinement
- get_centroid_analyser(debug=False)[source]¶
Create a CentroidAnalysis object for the current reflections
- get_free_reflections()[source]¶
Return all reflections that were accepted for refinement but not chosen in the working set
- get_sample_size()[source]¶
Return the number of observations in the working set to be used for refinement
- class dials.algorithms.refinement.reflection_manager.ReflectionManagerFactory[source]¶
Bases:
object
- static from_parameters_reflections_experiments(params, reflections, experiments, do_stills=False)[source]¶
Given a set of parameters and models, build a reflection manager
- Params:
params The input parameters
- Returns:
The reflection manager instance
- static get_weighting_strategy_override(params: scope_extract) StatisticalWeightingStrategy | ConstantWeightingStrategy [source]¶
- static laue_manager(experiments: ExperimentList, reflections: reflection_table, params: scope_extract) LaueReflectionManager [source]¶
- static rotation_scan_manager(experiments: ExperimentList, reflections: reflection_table, params: scope_extract) ReflectionManager [source]¶
- static stills_manager(experiments: ExperimentList, reflections: reflection_table, params: scope_extract) StillsReflectionManager [source]¶
- class dials.algorithms.refinement.reflection_manager.StillsReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None)[source]¶
Bases:
ReflectionManager
Overloads for a Reflection Manager that does not exclude reflections too close to the spindle, and reports only information about X, Y, DelPsi residuals
- experiment_type = 'stills'¶
- class dials.algorithms.refinement.reflection_manager.TOFReflectionManager(reflections, experiments, nref_per_degree=None, max_sample_size=None, min_sample_size=0, close_to_spindle_cutoff=0.02, scan_margin=0.0, outlier_detector=None, weighting_strategy_override=None, wavelength_weight=10000000.0)[source]¶
Bases:
LaueReflectionManager
Contains classes used to provide weighting schemes as strategies for ReflectionManagers.
- class dials.algorithms.refinement.weighting_strategies.ConstantWeightingStrategy(wx, wy, wz, stills=False)[source]¶
Bases:
object
- class dials.algorithms.refinement.weighting_strategies.ExternalDelPsiWeightingStrategy[source]¶
Bases:
StatisticalWeightingStrategy
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for stills refinement. This version uses statistical weights for X and Y and assume that the Delta Psi part is already provided in the reflection table
- class dials.algorithms.refinement.weighting_strategies.LaueMixedWeightingStrategy(wavelength_weight: float = 10000000.0)[source]¶
Bases:
StatisticalWeightingStrategy
Use statistical weighting for x and y, and constant weighting for z
- class dials.algorithms.refinement.weighting_strategies.LaueStatisticalWeightingStrategy(wavelength_weight: float = 10000000.0)[source]¶
Bases:
StatisticalWeightingStrategy
Variance in z estimated from sqrt(x^2+y^2)
- class dials.algorithms.refinement.weighting_strategies.StatisticalWeightingStrategy[source]¶
Bases:
object
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement
- class dials.algorithms.refinement.weighting_strategies.StillsWeightingStrategy(delpsi_constant=1000000)[source]¶
Bases:
StatisticalWeightingStrategy
Defines a single method that provides a ReflectionManager with a strategy for calculating weights for refinement. This version uses statistical weights for X and Y and a fixed constant for the delta Psi part, defaulting to 1000000
Refiner is the refinement module public interface. RefinerFactory is what should usually be used to construct a Refiner.
- class dials.algorithms.refinement.refiner.Refiner(experiments, pred_param, param_reporter, refman, target, refinery)[source]¶
Bases:
object
Public interface for performing DIALS refinement.
- Public methods:
run rmsds get_experiments get_matches get_param_reporter parameter_correlation_plot selection_used_for_refinement predict_for_reflection_table predict_for_indexed
Notes
The return value of run is a recorded history of the refinement
The experiments accessor provides a copy of the experiments used by refinement
get_matches exposes the function of the same name from the privately stored reflection manager
The return value of selection_used_for_refinement is a flex.bool
- __init__(experiments, pred_param, param_reporter, refman, target, refinery)[source]¶
- Mandatory arguments:
experiments - a dxtbx ExperimentList object pred_param - An object derived from the PredictionParameterisation class param_reporter -A ParameterReporter object refman - A ReflectionManager object target - An object derived from the Target class refinery - An object derived from the Refinery class
- get_parameter_correlation_matrix(step, col_select=None)[source]¶
Return the correlation matrix between columns of the Jacobian at the specified refinement step. The parameter col_select can be used to select subsets of the full number of columns. The column labels are also returned as a list of strings
- property history¶
Get the refinement engine’s step history
- predict_for_indexed()[source]¶
perform prediction for all the indexed reflections passed into refinement and additionally set the used_in_refinement flag. Do not compose the derivatives of states of the model as this is expensive and they are not needed outside of a refinement run
- predict_for_reflection_table(reflections, skip_derivatives=False)[source]¶
perform prediction for all reflections in the supplied table
- print_exp_rmsd_table()[source]¶
print useful output about refinement steps in the form of a simple table
- print_panel_rmsd_table()[source]¶
print useful output about refinement steps in the form of a simple table
- print_step_table()[source]¶
print useful output about refinement steps in the form of a simple table
- class dials.algorithms.refinement.refiner.RefinerFactory[source]¶
Bases:
object
Factory class to create refiners
- static config_refinery(params, target, pred_param, constraints_manager)[source]¶
Given a set of parameters, a target class, a prediction parameterisation class and a constraints_manager (which could be None), build a refinery
- Params:
params The input parameters
- Returns:
The refinery instance
- static config_restraints(params, pred_param)[source]¶
Given a set of user parameters plus a model parameterisation, create restraints plus a parameterisation of these restraints
- Params:
params: The input PHIL parameters pred_param: A PredictionParameters object
- Returns:
A restraints parameterisation or None
- class dials.algorithms.refinement.refiner.ScanVaryingRefiner(experiments, pred_param, param_reporter, refman, target, refinery)[source]¶
Bases:
Refiner
Includes functionality to update the models with their states at scan-points after scan-varying refinement
Contains classes for refinement engines. Refinery is the shared interface, LevenbergMarquardtIterations, GaussNewtonIterations, SimpleLBFGS and LBFGScurvs are the current concrete implementations
- class dials.algorithms.refinement.engine.AdaptLbfgs(*args, **kwargs)[source]¶
Bases:
Refinery
Adapt Refinery for L-BFGS minimiser
- class dials.algorithms.refinement.engine.AdaptLstbx(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)[source]¶
Bases:
Refinery
,non_linear_ls
,non_linear_ls_mixin
Adapt Refinery for lstbx
- __init__((object)arg1, (object)n_parameters) None : [source]¶
- C++ signature :
void __init__(_object*,int)
__init__( (object)arg1, (object)n_equations, (object)objective, (double)opposite_of_grad_objective, (object)normal_matrix) -> None :
- C++ signature :
void __init__(_object*,unsigned long,double,scitbx::af::shared<double>,scitbx::af::versa<double, scitbx::af::packed_u_accessor>)
- class dials.algorithms.refinement.engine.DisableMPmixin[source]¶
Bases:
object
A mixin class that disables setting of nproc for multiprocessing
- class dials.algorithms.refinement.engine.GaussNewtonIterations(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=20, **kwds)[source]¶
Bases:
AdaptLstbx
,iterations
Refinery implementation, using lstbx Gauss Newton iterations
- __init__((object)arg1, (object)n_parameters) None : [source]¶
- C++ signature :
void __init__(_object*,int)
__init__( (object)arg1, (object)n_equations, (object)objective, (double)opposite_of_grad_objective, (object)normal_matrix) -> None :
- C++ signature :
void __init__(_object*,unsigned long,double,scitbx::af::shared<double>,scitbx::af::versa<double, scitbx::af::packed_u_accessor>)
- convergence_as_shift_over_esd = 1e-05¶
- damping_value = 0.0007¶
- gradient_threshold = 1e-10¶
- max_shift_over_esd = 15¶
- run()[source]¶
To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).
- step_threshold = None¶
- class dials.algorithms.refinement.engine.Journal[source]¶
Bases:
dict
Container in which to store information about refinement history.
This is simply a dict but provides some extra methods for access that maintain values as columns in a table. Refinery classes will use these methods while entering data to ensure the table remains consistent. Methods inherited from dict are not hidden for ease of use of this object when returned to the user.
- add_row()[source]¶
Add an element to the end of each of the columns. Fail if any columns are the wrong length
- del_last_row()[source]¶
Delete the last element from the each of the columns. Fail if any columns are the wrong length
- reason_for_termination = None¶
- class dials.algorithms.refinement.engine.LBFGScurvs(*args, **kwargs)[source]¶
Bases:
AdaptLbfgs
Refinery implementation using cctbx LBFGS with curvatures
- class dials.algorithms.refinement.engine.LevenbergMarquardtIterations(target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=20, **kwds)[source]¶
Bases:
GaussNewtonIterations
Refinery implementation, employing lstbx Levenberg Marquadt iterations
- add_constant_to_diagonal(mu)[source]¶
Add the constant value mu to the diagonal of the normal matrix
- property mu¶
- report_progress(objective)[source]¶
Callback within the refinement main loop that can be overridden to report the value of the objective function (and possibly) other details for long-running methods
- run()[source]¶
To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).
- tau = 0.001¶
- class dials.algorithms.refinement.engine.Refinery(target: Target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)[source]¶
Bases:
object
Interface for Refinery objects. This should be subclassed and the run method implemented.
- __init__(target: Target, prediction_parameterisation, constraints_manager=None, log=None, tracking=None, max_iterations=None)[source]¶
- get_correlation_matrix_for_step(step)[source]¶
For each type of residual (e.g. X, Y, Phi), decompress and return the full 2D correlation matrix between columns of the Jacobian that was stored in the journal at the given step number. If not available, return None
- jacobian_condition_number()[source]¶
Calculate the condition number of the Jacobian, for tracking in the refinement journal, if requested. The condition number of a matrix A is defined as cond(A) = ||A|| ||inv(A)||. For a rectangular matrix the inverse operation refers to the Moore-Penrose pseudoinverse. Various matrix norms can be used, resulting in numerically different condition numbers, however the 2-norm is commonly used. In that case, the definition is equivalent to the ratio of the largest to smallest singular values of the matrix: cond(A) = sig_(A) / sig_min(A). That is the calculation that is performed here.
The condition number is a measure of how accurate the solution x to the equation Ax = b will be. Essentially it measures how errors are amplified through the linear equation. The condition number is large in the case that the columns of A are nearly linearly-dependent (and infinite for a singular matrix). We use it here then to detect situations where the correlation between effects of different parameter shifts becomes large and therefore refinement is problematic.
Note, the Jacobian used here does not include any additional rows due to restraints terms that might be applied, or any parameter reduction due to constraints. Therefore this condition number relates to the pure linearised (Gauss-Newton) step, which might not actually be what the refinement engine uses. It can be indicative of issues in the fundamental set up of the least squares problem, even if these issues are avoided in practice (e.g. by use of an algorithm like Levenberg-Marquardt, inclusion of restraints or parameter reduction).
- run()[source]¶
To be implemented by derived class. It is expected that each step of refinement be preceded by a call to prepare_for_step and followed by calls to update_journal and test_for_termination (in that order).
- set_nproc(nproc)[source]¶
Set number of processors for multiprocessing. Override in derived classes if a policy dictates that this must not be user-controlled
- split_jacobian_into_blocks() List[double] [source]¶
Split the Jacobian into blocks each corresponding to a separate residual, converting sparse to flex.double if appropriate
- class dials.algorithms.refinement.engine.SimpleLBFGS(*args, **kwargs)[source]¶
Bases:
AdaptLbfgs
Refinery implementation, using cctbx LBFGS with basic settings
Contains classes used to construct a target function for refinement, principally Target and ReflectionManager.
- class dials.algorithms.refinement.target.LaueLeastSquaresResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff: float = 0.33333, absolute_cutoffs: list | None = None, gradient_calculation_blocksize=None)[source]¶
Bases:
Target
A Laue implementation of the target class providing a least squares residual in terms of detector impact position X, Y, and observed wavelength
- __init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff: float = 0.33333, absolute_cutoffs: list | None = None, gradient_calculation_blocksize=None)[source]¶
- rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_wavelength']¶
- rmsd_units = ['mm', 'mm', 'A']¶
- class dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶
Bases:
Target
An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and phi, terminating on achieved rmsd (or on intrisic convergence of the chosen minimiser)
- class dials.algorithms.refinement.target.LeastSquaresPositionalResidualWithRmsdCutoffSparse(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶
Bases:
SparseGradientsMixin
,LeastSquaresPositionalResidualWithRmsdCutoff
A version of the LeastSquaresPositionalResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments
- class dials.algorithms.refinement.target.SparseGradientsMixin[source]¶
Bases:
object
Mixin class to build a sparse Jacobian from gradients of the prediction formula stored as sparse vectors, and allow concatenation of gradient vectors that employed sparse storage.
- class dials.algorithms.refinement.target.TOFLeastSquaresResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff: float = 0.33333, absolute_cutoffs: list | None = None, gradient_calculation_blocksize=None)[source]¶
Bases:
LaueLeastSquaresResidualWithRmsdCutoff
- rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_wavelength', 'RMSD_wavelength']¶
- rmsd_units = ['mm', 'mm', 'A', 'frame']¶
- class dials.algorithms.refinement.target.Target(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation=None, gradient_calculation_blocksize=None)[source]¶
Bases:
object
Abstract interface for a target function class
A Target object will be used by a Refinery. It will refer to a Reflection Manager to get a list of observations. It will perform reflection prediction on those observations and update the reflection manager with those predictions. It will then query the reflection manager to get a list of accepted reflections with observed and calculated positions. These are the reflections for use in refinement. It obtains the gradients of reflection positions from a relevant prediction parameterisation object. With all of this information in place, it calculates the value of the target function, the gradients of the target function and auxiliary information (e.g. RMSDs).
Concrete instances of this class implement the actual target function calculation. The base class should not determine what type of target function is used (e.g. least squares target), or limit whether the object is used for a detector space & phi residual, or a reciprocal space residual. This should all be set by a derived class.
- __init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation=None, gradient_calculation_blocksize=None)[source]¶
- calculate_gradients(reflections=None, callback=None)[source]¶
delegate to the prediction_parameterisation object to calculate gradients for all the matched reflections, or just for those specified
- compute_functional_gradients_and_curvatures(block=None)[source]¶
calculate the value of the target function and its gradients. Set approximate curvatures as a side-effect
- compute_residuals_and_gradients(block=None) Tuple[Any, double | matrix, Any] [source]¶
return the vector of residuals plus their gradients and weights for non-linear least squares methods
- compute_restraints_functional_gradients_and_curvatures()[source]¶
use the restraints_parameterisation object, if present, to calculate the least squares restraints objective plus gradients and approximate curvatures
- compute_restraints_residuals_and_gradients()[source]¶
delegate to the restraints_parameterisation object, if present, to calculate the vector of restraints residuals plus their gradients and weights for non-linear least squares methods
- property dim¶
Get the number of dimensions of the target function
- get_num_matches_for_experiment(iexp=0)[source]¶
return the number of reflections currently used in the calculation
- get_num_matches_for_panel(ipanel=0)[source]¶
return the number of reflections currently used in the calculation
- predict()[source]¶
perform reflection prediction for the working reflections and update the reflection manager
- predict_for_free_reflections()[source]¶
perform prediction for the reflections not used for refinement
- predict_for_reflection_table(reflections, skip_derivatives=False)[source]¶
perform prediction for all reflections in the supplied table
- rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_Phi']¶
- rmsd_units = ['mm', 'mm', 'rad']¶
- rmsds_for_reflection_table(reflections)[source]¶
calculate unweighted RMSDs for the specified reflections. Caution: this assumes that the table reflections has the keys expected by _rmsds_core
- split_matches_into_blocks(nproc: int = 1)[source]¶
Return a list of the matches, split into blocks according to the gradient_calculation_blocksize parameter and the number of processes (if relevant). The number of blocks will be set such that the total number of reflections being processed by concurrent processes does not exceed gradient_calculation_blocksize
- class dials.algorithms.refinement.target.TargetFactory[source]¶
Bases:
object
- static from_parameters_and_experiments(params, experiments, reflection_manager, predictor, pred_param, restraints_param, do_stills=False, do_sparse=False)[source]¶
Given a set of parameters, configure a factory to build a target function
- Params:
params The input parameters
- Returns:
The target factory instance
- class dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoff(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶
Bases:
Target
An implementation of the target class providing a least squares residual in terms of detector impact position X, Y and minimum rotation to the Ewald sphere, DeltaPsi. Terminates refinement on achieved rmsd (or on intrisic convergence of the chosen minimiser)
- __init__(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶
- predict_for_reflection_table(reflections, skip_derivatives=False)[source]¶
perform prediction for all reflections in the supplied table
- rmsd_names = ['RMSD_X', 'RMSD_Y', 'RMSD_DeltaPsi']¶
- rmsd_units = ['mm', 'mm', 'rad']¶
- class dials.algorithms.refinement.target_stills.LeastSquaresStillsResidualWithRmsdCutoffSparse(experiments, predictor, reflection_manager, prediction_parameterisation, restraints_parameterisation, frac_binsize_cutoff=0.33333, absolute_cutoffs=None, gradient_calculation_blocksize=None)[source]¶
Bases:
SparseGradientsMixin
,LeastSquaresStillsResidualWithRmsdCutoff
A version of the LeastSquaresStillsResidualWithRmsdCutoff Target that uses a sparse matrix data structure for memory efficiency when there are a large number of Experiments