characterize_gauss_fit – PSF Fitting Characterization Tool
Overview
characterize_gauss_fit is a standalone command-line program that
systematically measures the accuracy of the Gaussian PSF fitter across a
large, configurable parameter space. It generates synthetic PSF images with
known ground-truth parameters, fits them, and reports how closely the fitter
recovers position, sigma, angle, and scale.
Eight focused studies each vary a small set of parameters while holding others fixed. Every study produces:
PNG plots for visual inspection (heatmaps, line plots with error bands, grouped bar charts).
trials.csv– one row per trial with all input parameters and all result metrics, loadable by pandas or any data-analysis tool.summary.json– aggregate statistics per parameter group plus the exact configuration used, formatted for AI-assisted analysis.
Running from the Repository
characterize_gauss_fit is not an installed command-line entry point. It must
be run directly from the repository. First clone the repository and install the
extra dependencies:
git clone https://github.com/SETI/rms-psfmodel.git
cd rms-psfmodel
pip install -e ".[characterize]"
This adds matplotlib and pyyaml to your environment. All commands below
use python -m characterize_gauss_fit and must be run from the repository
root (or any directory where the package is importable).
Quick Start
Run all studies with default settings:
python -m characterize_gauss_fit
Run a quick smoke test across all studies using the bundled reduced-grid configuration (completes in roughly 30–120 seconds):
python -m characterize_gauss_fit --copy-test-config-to test_config.yaml
python -m characterize_gauss_fit --config test_config.yaml
Run a single study:
python -m characterize_gauss_fit --study box_vs_sigma
Run with a custom override file and parallel workers:
python -m characterize_gauss_fit --config my_config.yaml --num-workers 8
List all available study names:
python -m characterize_gauss_fit --list-studies
Copy the built-in default configuration to a local file for editing:
python -m characterize_gauss_fit --copy-default-config-to my_config.yaml
Copy the built-in reduced-grid test configuration to a local file:
python -m characterize_gauss_fit --copy-test-config-to test_config.yaml
Copy the built-in high-resolution configuration to a local file:
python -m characterize_gauss_fit --copy-hires-config-to hires_config.yaml
CLI Reference
usage: python -m characterize_gauss_fit [--config FILE] [--study NAME] [--output-dir DIR]
[--num-workers N] [--list-studies]
[--copy-default-config-to FILE]
[--copy-test-config-to FILE]
[--copy-hires-config-to FILE] [--verbose]
Options:
--config FILE Path to a YAML override file merged onto
built-in defaults.
--study NAME Run only this study (repeatable). Default: all
enabled studies. Use --list-studies to see names.
--output-dir DIR Override the output directory from the config
file.
--num-workers N Number of parallel worker processes. Default:
resolved from config file (built-in default: 1).
1 = sequential in the main process; >1 uses
concurrent.futures.ProcessPoolExecutor.
--list-studies Print available study names and exit.
--copy-default-config-to FILE
Write the built-in default configuration to FILE
and exit. No studies are run.
--copy-test-config-to FILE Write the built-in reduced-grid test
configuration to FILE and exit. No studies are
run.
--copy-hires-config-to FILE Write the built-in high-resolution configuration
to FILE and exit. No studies are run.
--verbose, -v Enable DEBUG-level logging.
Exit code is 0 on success. Exit code 1 if any study raises an unhandled exception; individual trial failures (fitter non-convergence) are not fatal and are recorded as data.
Configuration Reference
All parameters have built-in defaults. You can override any subset by
providing a YAML file with --config. The file is deep-merged onto the
defaults: scalar values and lists replace the default; nested dicts are
merged recursively.
Top-level keys
Key |
Type |
Default |
Description |
|---|---|---|---|
|
string (path) |
|
Root directory for all output files. |
|
int |
|
Worker processes for parallel execution. |
|
int |
|
Default noise realisations for stochastic studies. |
fitting section
Global defaults passed to PSF.find_position. Any study section may include
a fitting subsection to override these for that study only.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
int or null |
|
Polynomial degree for background fitting. |
|
[int, int] |
|
Half-size of central region excluded from background fit (rows, cols). The excluded region is |
|
float or null |
|
Sigma-clipping threshold for background residuals. |
|
float or null |
|
Sigma-clipping threshold for PSF residuals (bad-pixel rejection). |
|
float |
|
Maximum fraction of pixels that can be masked before the fit is abandoned. |
|
bool |
|
Fit a constant base level in addition to the polynomial background. |
|
bool |
|
Reparametrise fit variables as angles for bounded optimization. |
|
float |
|
Powell optimizer convergence tolerance. |
|
[float, float] |
|
Maximum allowed position offset from the starting point [y, x] in pixels. |
|
float |
|
Maximum allowed PSF amplitude scale factor. |
generation section
Controls the synthetic PSF image generator.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
float |
|
PSF amplitude scale factor applied to the normalised Gaussian. |
|
float |
|
Additive base level on the clean PSF before background injection. |
studies section
Each study has an enabled flag and its own parameter set. Per-study
fitting subsections override the global defaults for that study only.
box_vs_sigma
Study 1: How box size relative to PSF sigma affects fitting accuracy.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[int] |
|
Odd box sizes to test. Each must be >= 5. |
|
list[float] |
|
Symmetric PSF sigma values (pixels). |
|
list[[float, float]] |
|
List of sub-pixel [y, x] offsets. Each entry produces a separate set of heatmap plots. |
|
float |
|
PSF rotation angle (radians). |
|
float |
|
PSF amplitude scale factor (overrides |
|
dict |
|
Per-study fitting overrides. |
subpixel_offset
Study 2: How fractional pixel position introduces systematic bias.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
int |
|
Number of evenly-spaced steps in each axis. |
|
[float, float] |
|
Range of offset values. Values outside [0, 0.5] are redundant by symmetry. |
|
list[float] |
|
Sigma values used as a panel variable. |
|
int |
|
Fixed box size. |
|
float |
|
Fixed PSF angle. |
|
dict |
|
Per-study fitting overrides. |
min_detectable_offset
Study 3: Minimum offset delta reliably recoverable vs. PSF sigma and noise.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[float] |
|
Offset deltas to test (pixels). |
|
list[float] |
|
Symmetric sigma values. |
|
int |
|
Fixed box size. |
|
int |
|
Noise realisations per stochastic condition. |
|
list[float] |
|
SNR values (peak / noise RMS) to test. |
|
bool |
|
Also run a noiseless trial (numerical precision floor). |
|
dict |
|
Per-study fitting overrides. |
sigma_asymmetry_angle
Study 4: Sigma asymmetry and angle recovery for elongated, rotated PSFs.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[float] |
|
Ratios sigma_y / sigma_x. |
|
int |
|
Evenly-spaced angles from 0 to pi (inclusive). Must be >= 2. |
|
list[float] |
|
sigma_x values (panel variable). |
|
int |
|
Fixed box size. |
|
[float, float] |
|
Fixed sub-pixel offset. |
|
dict |
|
Per-study fitting overrides. |
constraint_modes
Study 5: Effect of fixing vs. floating sigma/angle on all output metrics.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[float] |
|
Fractional errors applied to sigma when fixed. 0.0 = correct value; 0.2 = fixed at 1.2x true. |
|
float |
|
Absolute angle error (radians) when angle is fixed incorrectly. |
|
list[{sigma: [y,x], angle: rad}] |
3 shapes |
PSF shapes to test. Each entry must have |
|
int |
|
Fixed box size. |
|
[float, float] |
|
Fixed sub-pixel offset. |
|
float |
|
PSF amplitude scale factor. |
background
Study 6: How injected background and fitting model choice interact.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[float] |
|
Background amplitude as fraction of PSF peak. |
|
list[int] |
|
Polynomial degrees to use when fitting the background. |
|
bool |
|
Also test |
|
list[[int,int]] |
|
|
|
list[str] |
|
Background types to inject. |
|
int |
|
Fixed box size. |
|
[float, float] |
|
Fixed PSF sigma [y, x]. |
|
list[[float, float]] |
|
List of sub-pixel [y, x] offsets. Each entry produces a separate set of heatmap plots. |
Valid background_types values:
Value |
Description |
|---|---|
|
No background injected. |
|
Flat pedestal at |
|
Tilted plane (linear gradient). |
|
Bowl-shaped quadratic surface. |
|
Flat pedestal plus Gaussian noise at 0.5x the main noise level. |
noise_sensitivity
Study 7: Position, sigma, and scale accuracy as a function of SNR.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
[float, float] |
|
Log10 range for SNR (min, max). |
|
int |
|
Number of log-spaced SNR points. |
|
list[float] |
|
Sigma values (panel variable). |
|
int |
|
Noise realisations per (SNR, sigma) point. |
|
int |
|
Fixed box size. |
hot_pixel_rejection
Study 8: Effectiveness of num_sigma bad-pixel rejection.
Key |
Type |
Default |
Description |
|---|---|---|---|
|
bool |
|
Enable/disable this study. |
|
list[int] |
|
Number of hot pixels to inject. |
|
list[float] |
|
|
|
bool |
|
Also test |
|
list[float] |
|
Hot pixel amplitude as multiple of PSF peak. |
|
int |
|
Noise realisations per combination (hot pixel positions randomised). |
|
float |
|
Background Gaussian noise SNR (peak / noise RMS). |
|
int |
|
Fixed box size. |
|
[float, float] |
|
Fixed PSF sigma [y, x]. |
|
[float, float] |
|
Fixed sub-pixel offset. |
Output Format Reference
Each study writes its results to {output_dir}/{study_name}/.
trials.csv
One row per trial. Columns:
Column |
Description |
|---|---|
|
Study name string. |
|
Subimage side length (pixels). |
|
True PSF sigma values. |
|
True rotation angle (radians). |
|
True sub-pixel offset. |
|
True amplitude scale factor. |
|
Value sigma was fixed to (empty = floated). |
|
Value angle was fixed to (empty = floated). |
|
Injected background type string. |
|
Injected background amplitude (fraction of peak). |
|
Additive Gaussian noise standard deviation. |
|
Number of hot pixels injected. |
|
Hot pixel amplitude (multiple of PSF peak). |
|
Background fitting polynomial degree (empty = null). |
|
Bad-pixel rejection threshold (empty = null). |
|
Ignore-center half-sizes. |
|
|
|
Signed position errors (fitted - true). |
|
Euclidean position error. |
|
Fitted sigma values (empty if fixed). |
|
Fitted angle (empty if fixed). |
|
Fitted scale factor. |
|
Relative sigma errors (fit - true) / true. |
|
Absolute angle error in radians. |
|
Relative scale error (fit - true) / true. |
Conventions:
Non-applicable fields are empty strings (e.g.
sigma_y_fitwhen sigma was fixed).Failed / non-converged trials have
NaNfor all error fields.Load with
pandas.read_csv(..., na_values=['NaN', '']).
Naming asymmetry warning. Two pairs of columns have similar names but opposite roles:
fit_sigma_y/fit_sigma_xare inputs (the value sigma was constrained to before fitting; empty string when sigma was left to float).
sigma_y_fit/sigma_x_fitare outputs (the sigma value returned by the fitter; empty string when sigma was fixed and not fitted).In short:
fit_*columns describe what you told the fitter;*_fitcolumns describe what the fitter found.Example (Python / pandas):
# Trials where sigma_y was constrained (input column non-empty): constrained = df[df['fit_sigma_y'].notna()] # Trials where sigma_y was floated and a fitted value was returned: fitted = df[df['sigma_y_fit'].notna()]
summary.json
{
"study": "box_vs_sigma",
"total_trials": 72,
"converged_trials": 68,
"convergence_rate": 0.944,
"overall": { ... aggregate stats ... },
"groups": [
{
"box_size": 5,
"sigma": 0.3,
"n_trials": 1,
"n_converged": 1,
"convergence_rate": 1.0,
"pos_err_mean": 0.00123,
"pos_err_std": null,
"sigma_y_err_mean": 0.005,
"scale_err_mean": 0.002,
"angle_err_mean": null
}
],
"config_used": { ... full config dict ... }
}
null in JSON corresponds to None in Python (not enough data to compute,
or metric not applicable). The config_used field contains the exact
configuration that produced these results for full reproducibility.
AI / automated analysis
To load all studies for analysis:
import json
import pathlib
import pandas as pd
results_dir = pathlib.Path('./gauss_fit_results')
dfs = []
for csv_file in results_dir.glob('*/trials.csv'):
dfs.append(pd.read_csv(csv_file, na_values=['NaN', '']))
all_results = pd.concat(dfs, ignore_index=True)
Study Descriptions
Study 1: Box Size vs. Sigma (box_vs_sigma)
Question: How large must the subimage be relative to the PSF width for accurate fitting?
Sweeps box size and PSF sigma on a 2-D grid. The PSF always fills the entire
image (eval_rect size == box_size). Background is disabled so only the
intrinsic truncation effect is measured. Sigma is left to float.
Key plots: Heatmaps of log10(position error), log10(sigma error), and log10(scale error) as functions of box size (rows) and sigma (columns). Cells where the fitter did not converge are shown in grey.
Interpretation: Expect a sharp accuracy cliff when
box_size < 4 * sigma + 1. Small sigmas are well-fitted even in tiny boxes;
large sigmas in small boxes truncate most of the PSF flux.
Study 2: Subpixel Offset (subpixel_offset)
Question: Does the fractional pixel position of the PSF centre introduce systematic bias?
Sweeps offset_y and offset_x from 0 to 0.5 pixels in a grid. The range [0, 0.5] is sufficient by symmetry. Three sigma panel values are tested.
Key plots: 2-D heatmap of position error vs. (offset_y, offset_x) for each sigma. Line plot of error vs. offset_x at fixed offset_y.
Interpretation: The fitter may exhibit a small systematic oscillation at the pixel-period scale due to aliasing. Very small sigmas show larger absolute error because the PSF peak is narrower than a pixel.
Study 3: Minimum Detectable Offset (min_detectable_offset)
Question: How small an offset delta can the fitter reliably recover as a function of PSF size and noise?
Tests a log-spaced grid of offset deltas, applied purely in the X direction,
for each (sigma, SNR) combination. Recovery fraction is defined as the fraction
of trials where |pos_err| < delta/2 (within 50% of the true offset).
Key plots: Log-log line plot of mean position error vs. delta for each sigma (one panel per SNR). Recovery fraction heatmap (sigma vs. delta, one plot per SNR level).
Interpretation: The precision floor in the noiseless case reveals
numerical resolution limits. Noise raises the floor to approximately
sigma / SNR. Recovery fraction drops below 0.5 near the precision floor.
Study 4: Sigma Asymmetry and Angle (sigma_asymmetry_angle)
Question: How well are elongated, rotated PSFs recovered?
Sweeps sigma_ratio (sigma_y / sigma_x) and rotation angle for several sigma_x values. All parameters float. For circular PSFs (ratio ~= 1.0), angle is degenerate and angle error is not meaningful – those cells show NaN.
Key plots: Heatmaps of position error, angle error, and sigma_y error as functions of (ratio, angle), one panel per sigma_x.
Interpretation: Near-circular PSFs (ratio near 1) have degenerate angle; the fitter can converge to any angle without affecting position accuracy. Highly elongated PSFs at edge-case angles (0, pi/2, pi) may have increased error due to the optimizer landscape.
Study 5: Constraint Modes (constraint_modes)
Question: How does fixing vs. floating sigma and angle affect position, scale, sigma, and angle accuracy?
Tests eight constraint configurations on three PSF shapes. Reports all four accuracy metrics.
Key plots: 4-panel grouped bar chart: position error, relative scale error, relative sigma_y error, and absolute angle error, grouped by PSF shape.
Interpretation: Correctly fixing sigma reduces fitting degrees of freedom and generally improves position accuracy at the cost of sigma recovery information. Incorrectly fixed sigma can bias all metrics. Floating angle on circular PSFs wastes degrees of freedom but rarely hurts position accuracy.
Study 6: Background Conditions (background)
Question: How do injected background and fitting model choice interact?
Combines five background types with four fitting-degree options and three ignore-center sizes.
Key plots: Heatmap matrix – rows = injected background type, columns = fitting degree. One heatmap per (amplitude, ignore-center) combination.
Interpretation: Fitting a constant background (degree 0) is generally
sufficient for constant injected backgrounds. Higher-degree backgrounds require
matching or higher fitting degrees. Using bkgnd_degree=null with any
non-zero background will show degraded accuracy.
Study 7: Noise Sensitivity (noise_sensitivity)
Question: At what SNR does fitting accuracy degrade?
Sweeps a log-spaced SNR range with multiple noise realisations per point. Random offsets are used so position error reflects typical (not best-case) accuracy.
Key plots: Line plots with shaded ±1 std bands: position error, sigma_y error, sigma_x error, and scale error vs. SNR (log scale). One line per sigma panel value.
Interpretation: All metrics improve roughly as 1/SNR in the noise-limited regime. The saturation at high SNR reveals the floor set by optimizer precision and pixel-integration discretisation.
Study 8: Hot Pixel Rejection (hot_pixel_rejection)
Question: How effectively does num_sigma rejection handle hot pixels?
Varies the number of hot pixels (0 to 10), their amplitude (5x to 100x peak),
and the num_sigma rejection threshold. Multiple noise realisations randomise
hot pixel positions.
Key plots: Line plot per amplitude panel: X = number of hot pixels,
Y = mean position error, lines = num_sigma settings.
Interpretation: num_sigma=null shows the baseline degradation from
unrejected hot pixels. Aggressive rejection (low num_sigma) can mask real
PSF pixels near the core. num_sigma=3 is typically a good balance.
Bundled Configuration Files
Three YAML configurations ship alongside the package source. Each can be
copied to a local file for editing with the corresponding --copy-*-to
option.
Default configuration (defaults.yaml)
The primary reference configuration. All parameters are set to sensible
defaults that provide a thorough survey of each study’s parameter space.
Output is written to ./gauss_fit_results/.
python -m characterize_gauss_fit --copy-default-config-to my_config.yaml
Reduced-grid test configuration (test_config.yaml)
Runs all eight studies with the smallest viable parameter grids so the entire suite completes in roughly 30–120 seconds on a single core. Use it to verify that all code paths execute after code changes.
python -m characterize_gauss_fit --copy-test-config-to test_config.yaml
python -m characterize_gauss_fit --config test_config.yaml
Output is written to ./gauss_fit_test/ by default.
Study |
Grid size |
Approx. trials |
|---|---|---|
|
2 box sizes x 3 sigmas x 2 offsets |
12 |
|
3 x 3 offset grid, 1 sigma |
9 |
|
3 deltas x 2 sigmas x (1 noiseless + 1 SNR) x 3 samples |
~24 |
|
3 ratios x 3 angles x 1 sigma_x |
9 |
|
5 modes x 2 PSF shapes x 2 sigma-error fractions |
~20 |
|
2 background types x 2 fitting degrees x 2 offsets |
8 |
|
4 SNR points x 1 sigma x 3 samples |
12 |
|
2 num_hot x 2 num_sigma x 1 amplitude x 3 samples |
12 |
High-resolution configuration (hires_config.yaml)
Runs all eight studies with denser parameter grids and larger
noise_samples counts compared with the defaults, while staying within the
same parameter ranges. The goal is smoother heatmaps, less noisy line plots,
and more nuanced detail at intermediate parameter values. Estimated runtime
is 10–30x longer than the default configuration; use --num-workers to
parallelise across CPU cores.
python -m characterize_gauss_fit --copy-hires-config-to hires_config.yaml
python -m characterize_gauss_fit --config hires_config.yaml --num-workers 8
Output is written to ./gauss_fit_hires/ by default.
Study |
Denser axes |
Key changes vs. defaults |
|---|---|---|
|
12 box sizes, 13 sigmas |
adds 15, 19, 41 px boxes; intermediate sigma values |
|
21 x 21 offset grid, 4 sigmas |
0.025 px step; adds sigma=1.5 |
|
11 deltas, 4 SNR conditions, 200 samples |
adds SNR=20 condition |
|
10 ratios, 25 angle steps |
7.5 deg angular resolution |
|
6 sigma-error fractions, 4 PSF shapes |
adds 0.1, 0.3, 0.75 fractions |
|
7 amplitudes, degree=3, 4 ignore_center sizes |
finer amplitude sweep |
|
25 SNR points, 4 sigmas, 200 samples |
adds sigma=1.5; 4x more samples |
|
8 hot-pixel counts, 6 amplitudes, 4 thresholds, 50 samples |
fills gaps in all axes |
Obtaining any bundled file
All three copy commands write the file and exit immediately — no studies are run. The destination path must not already exist.
python -m characterize_gauss_fit --copy-default-config-to my_config.yaml
python -m characterize_gauss_fit --copy-test-config-to test_config.yaml
python -m characterize_gauss_fit --copy-hires-config-to hires_config.yaml
All parameters in any bundled config can be further overridden by combining it with a second user config file or with CLI flags. For example, to run only study 1 with the test grid:
python -m characterize_gauss_fit --config test_config.yaml --study box_vs_sigma
Example User Override File
To override a subset of parameters on top of the defaults, create a YAML file containing only the keys you want to change. The following example runs two studies with custom grids:
output_dir: ./my_results
studies:
box_vs_sigma:
box_sizes: [7, 11, 21]
sigmas: [0.5, 1.0, 2.0, 3.0]
noise_sensitivity:
snr_steps: 10
noise_samples: 20
sigmas: [0.5, 1.0, 2.0]
subpixel_offset:
enabled: false
min_detectable_offset:
enabled: false
sigma_asymmetry_angle:
enabled: false
constraint_modes:
enabled: false
background:
enabled: false
hot_pixel_rejection:
enabled: false
Run with:
python -m characterize_gauss_fit --config my_overrides.yaml