queso-cluster Tutorial#
This is a work in progress!!!
A brief introduction#
This package is still under active development!! I am working on it.
For package installation:
## For Juypter
%pip install queso-cluster
## For pip
# pip3 install /disk/data/sriley/queso-cluster
WARNING: Ignoring invalid distribution ~ueso-cluster (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ueso-cluster (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ueso-cluster (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
Requirement already satisfied: queso-cluster in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (0.3.0a0)
Requirement already satisfied: numpy>=1.25 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (1.26.0)
Requirement already satisfied: dkist in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (1.16.0)
Requirement already satisfied: scipy in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (1.15.2)
Requirement already satisfied: numba in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (0.65.1)
Requirement already satisfied: astropy in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (7.0.1)
Requirement already satisfied: numba_progress in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (1.1.3)
Requirement already satisfied: dask in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (2025.3.0)
Requirement already satisfied: pyyaml in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (6.0.2)
Requirement already satisfied: tol-colors in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (2.2.0)
Requirement already satisfied: pint in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from queso-cluster) (0.25.3)
Requirement already satisfied: pyerfa>=2.0.1.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from astropy->queso-cluster) (2.0.1.5)
Requirement already satisfied: astropy-iers-data>=0.2025.1.31.12.41.4 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from astropy->queso-cluster) (0.2025.4.7.0.35.30)
Requirement already satisfied: packaging>=22.0.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from astropy->queso-cluster) (24.2)
Requirement already satisfied: click>=8.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (8.1.8)
Requirement already satisfied: cloudpickle>=3.0.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (3.1.1)
Requirement already satisfied: fsspec>=2021.09.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (2025.3.2)
Requirement already satisfied: partd>=1.4.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (1.4.2)
Requirement already satisfied: toolz>=0.10.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (1.0.0)
Requirement already satisfied: importlib_metadata>=4.13.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dask->queso-cluster) (8.6.1)
Requirement already satisfied: zipp>=3.20 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from importlib_metadata>=4.13.0->dask->queso-cluster) (3.21.0)
Requirement already satisfied: locket in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from partd>=1.4.0->dask->queso-cluster) (1.0.0)
Requirement already satisfied: aiohttp>=3.9 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (3.11.16)
Requirement already satisfied: asdf>=3.3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (4.1.0)
Requirement already satisfied: asdf-astropy>=0.5.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (0.7.1)
Requirement already satisfied: asdf-coordinates-schemas>=0.3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (0.3.0)
Requirement already satisfied: asdf-standard>=1.1.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (1.1.1)
Requirement already satisfied: asdf-transform-schemas>=0.6.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (0.6.0)
Requirement already satisfied: asdf-wcs-schemas>=0.4.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (0.4.0)
Requirement already satisfied: globus-sdk>=3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (3.54.0)
Requirement already satisfied: gwcs>=0.24.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (0.24.0)
Requirement already satisfied: matplotlib>=3.7 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (3.10.1)
Requirement already satisfied: ndcube>=2.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (2.3.1)
Requirement already satisfied: parfive>=2.2.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from parfive[ftp]>=2.2.0->dkist->queso-cluster) (2.2.0)
Requirement already satisfied: platformdirs>=3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (4.3.7)
Requirement already satisfied: sunpy>=5.0.7 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (6.1.1)
Requirement already satisfied: tqdm>=4.65 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from dkist->queso-cluster) (4.67.1)
Requirement already satisfied: aiohappyeyeballs>=2.3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (2.6.1)
Requirement already satisfied: aiosignal>=1.1.2 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (1.3.2)
Requirement already satisfied: attrs>=17.3.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (25.3.0)
Requirement already satisfied: frozenlist>=1.1.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (1.5.0)
Requirement already satisfied: multidict<7.0,>=4.5 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (6.3.2)
Requirement already satisfied: propcache>=0.2.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (0.3.1)
Requirement already satisfied: yarl<2.0,>=1.17.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from aiohttp>=3.9->dkist->queso-cluster) (1.19.0)
Requirement already satisfied: idna>=2.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from yarl<2.0,>=1.17.0->aiohttp>=3.9->dkist->queso-cluster) (3.10)
Requirement already satisfied: jmespath>=0.6.2 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from asdf>=3.3.0->dkist->queso-cluster) (1.0.1)
Requirement already satisfied: semantic_version>=2.8 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from asdf>=3.3.0->dkist->queso-cluster) (2.10.0)
Requirement already satisfied: requests<3.0.0,>=2.19.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from globus-sdk>=3.0->dkist->queso-cluster) (2.32.3)
Requirement already satisfied: pyjwt<3.0.0,>=2.0.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from pyjwt[crypto]<3.0.0,>=2.0.0->globus-sdk>=3.0->dkist->queso-cluster) (2.10.1)
Requirement already satisfied: cryptography!=3.4.0,>=3.3.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from globus-sdk>=3.0->dkist->queso-cluster) (44.0.2)
Requirement already satisfied: charset-normalizer<4,>=2 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from requests<3.0.0,>=2.19.1->globus-sdk>=3.0->dkist->queso-cluster) (3.4.1)
Requirement already satisfied: urllib3<3,>=1.21.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from requests<3.0.0,>=2.19.1->globus-sdk>=3.0->dkist->queso-cluster) (2.3.0)
Requirement already satisfied: certifi>=2017.4.17 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from requests<3.0.0,>=2.19.1->globus-sdk>=3.0->dkist->queso-cluster) (2025.1.31)
Requirement already satisfied: cffi>=1.12 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from cryptography!=3.4.0,>=3.3.1->globus-sdk>=3.0->dkist->queso-cluster) (1.17.1)
Requirement already satisfied: pycparser in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from cffi>=1.12->cryptography!=3.4.0,>=3.3.1->globus-sdk>=3.0->dkist->queso-cluster) (2.22)
Requirement already satisfied: contourpy>=1.0.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (4.57.0)
Requirement already satisfied: kiwisolver>=1.3.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (1.4.8)
Requirement already satisfied: pillow>=8 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (11.1.0)
Requirement already satisfied: pyparsing>=2.3.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (3.2.3)
Requirement already satisfied: python-dateutil>=2.7 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from matplotlib>=3.7->dkist->queso-cluster) (2.9.0.post0)
Requirement already satisfied: mpl_animators>=1.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (1.2.1)
Requirement already satisfied: reproject>=0.7.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (0.14.1)
Requirement already satisfied: aioftp>=0.17.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from parfive[ftp]>=2.2.0->dkist->queso-cluster) (0.24.1)
Requirement already satisfied: six>=1.5 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib>=3.7->dkist->queso-cluster) (1.17.0)
Requirement already satisfied: astropy-healpix>=1.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from reproject>=0.7.1->ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (1.1.2)
Requirement already satisfied: zarr in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from reproject>=0.7.1->ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (3.0.6)
Requirement already satisfied: beautifulsoup4>=4.11.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (4.13.3)
Requirement already satisfied: drms>=0.7.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (0.9.0)
Requirement already satisfied: zeep>=4.1.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (4.3.1)
Requirement already satisfied: soupsieve>1.2 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from beautifulsoup4>=4.11.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (2.6)
Requirement already satisfied: typing-extensions>=4.0.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from beautifulsoup4>=4.11.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (4.13.1)
Requirement already satisfied: pandas>=1.5.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from drms>=0.7.1->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (2.2.3)
Requirement already satisfied: pytz>=2020.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from pandas>=1.5.1->drms>=0.7.1->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (2025.2)
Requirement already satisfied: tzdata>=2022.7 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from pandas>=1.5.1->drms>=0.7.1->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (2025.2)
Requirement already satisfied: isodate>=0.5.4 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from zeep>=4.1.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (0.7.2)
Requirement already satisfied: lxml>=4.6.0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from zeep>=4.1.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (5.3.2)
Requirement already satisfied: requests-toolbelt>=0.7.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from zeep>=4.1.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (1.0.0)
Requirement already satisfied: requests-file>=1.5.1 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from zeep>=4.1.0->sunpy[asdf,net]>=5.0.7->dkist->queso-cluster) (2.1.0)
Requirement already satisfied: llvmlite<0.48,>=0.47.0dev0 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from numba->queso-cluster) (0.47.0)
Requirement already satisfied: flexcache>=0.3 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from pint->queso-cluster) (0.3)
Requirement already satisfied: flexparser>=0.4 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from pint->queso-cluster) (0.4)
Requirement already satisfied: donfig>=0.8 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from zarr->reproject>=0.7.1->ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (0.8.1.post1)
Requirement already satisfied: numcodecs>=0.14 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from numcodecs[crc32c]>=0.14->zarr->reproject>=0.7.1->ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (0.16.0)
Requirement already satisfied: crc32c>=2.7 in /nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages (from numcodecs[crc32c]>=0.14->zarr->reproject>=0.7.1->ndcube[plotting,reproject]>=2.1->dkist->queso-cluster) (2.7.1)
WARNING: Ignoring invalid distribution ~ueso-cluster (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ueso-cluster (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
WARNING: Ignoring invalid distribution ~ (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages)
Note: you may need to restart the kernel to use updated packages.
There is one additional picece that is required for this version of QuESO: the eventManager.yml file. This file will contain the user defined configuration for not just the clustering but also extra bits of data calibration. An entry looks like
- event:
date: '2023-05-03'
label: 'Strong flare signatures'
run:
- label: 'timeDependent'
approach: 'evo'
overwrite: True
alignment_dir: "aligned_x5bin"
config: {frames: [-1, 3, 1],
keepI0: [3],
bbox: [0, 125, 1188, 1629],
primary: {src: 'AEVEG-SPD', S1: [1, 5, 2, 4, 1], lines: {'AEVEG-SPD': ['main']}},
}
src:
- id: {'instrument': 'ViSP', 'data': 'AEVEG', mod: "SPD"}
theme: '#527A00'
residual: False
clustering:
S0:
- {label: 'window', layerConfig: {'bins': [0, 0.52, 0.60, 2]}}
S1:
- {layerGroups: 5, layerConfig: {'converge': 1e-6, ss_thresh: [0, 0, 0, 0, 0, 0]}}
axis_fit: {coeff: [-2.81754321e-8 1.91621915e-3 8.53133874e+2], unit: "angstrom"}
lines:
- {label: 'main', window: [462, 568], continuum: 26, core: 515}
- id: {'instrument': 'ViSP', 'data': 'BZNNG', mod: "SPD"}
theme: {'D1': '#0000FF', 'D2': '#FF0000', 'Ni': '#FF0000'}
residual: True
axis_fit: {coeff: [-1.60283877e-8 1.42050858e-3 5.88875195e+2], unit: "angstrom"}
lines:
- {label: 'D2', continuum: 375, core: 86, window: [51, 121]}
- {label: 'D1', continuum: 375, core: 509, window: [473, 545]}
- {label: 'Ni', continuum: 375, core: 291, window: [256, 327]}
Getting started#
To get started, we are going to import the package (note the difference of the dash and underscore in the package name). And load in some file paths.
import queso_cluster as qc
import os
#os.environ["QUESO-DAT"] = '/disk/data/DKIST/20230503'
One last note, the figure functions output a matplotlib figure object.
Instrument Loader Objects#
To handle the specific physical attributes of the data, we have developed a framework to import only the essentials. Currently, we only have a built-in object for ViSP. The instrument object will hold the dataset compressed into a dask array for lazy loading to minimize memory usage during some of the calculations.
Time-independent clustering#
This is just fancy words for the standard approach to cluster the evolution of spectra. As it suggests, this clustering approach doesn’t use any temporal information during the clustering. See Udeas et al., 2026 for an example.
The basic principle of using this approach is to cluster the entire dataset then afterwards reintroduce the temporal information to get the sequence of spectral profiles associated with the feature. This method does work, but there are some considerations that need to be taken.
Feature density or the number of resolved clustered within the coordinate space. In clustering, a high feature density is actually bad. The more clusters you need the more the boundaries of the clusters is blurred. While there is not a specific threshold for feature density, a smaller “global” feature density is preferred.
**
This short tutorial will go through the steps to use the ti sub-module with a time series spectral dataset.
import numpy as np
from queso_cluster import ti
from queso_cluster.runners import base as runBase
from queso_cluster.atoms import norm as normAtom
from queso_cluster.loaders.visp import visp
def main(config):
#> Note: This creates the instrument object for the ViSP observations
ViSPobj = visp("/disk/data/DKIST/20250503")
ViSPobj.load()
#> Note: Here we are going to take only the first four frames and flatten them. The clustering functions only accept 2D arrays.
ViSPobj.dataSquare = ViSPobj[:4, ...].reshape((vispObj.shape[1]*4, ViSPobj.shape[-1]))
#> Note: This function imports everything from the instrument object and has the specific clustering functions that we will use for the time independent mode.
tiObj = ti.timeIndependent(config, config.runners.label, ViSPobj)
#> Note: This preps the data. Specifically we want to normalize by the continuum intensity
prepSquare = runBase.runPrep(tiObj.dataSquare,
norm=normAtom.normContinuum,
continuumIndx=tiObj.continuum)
#> Note: We are most interested in the flaring pixels. These are very bright compared with the surrounding area,
maskLine = np.ones(prepSquare.shape[0]).astype(bool)
keepI0 = None
if "keepI0" in list(config.runners.config.keys()):
keepI0 = config.runners.config['keepI0']
#> Note: This is where the magic happens. This function will handle calls to the base clustering functions which handle the calculations.
#> Note: The output in a 1D flattened array containing the labels for each pixel and a data object which stores the quality metrics of the cluster.
labelLine, scoreTuple, = tiObj.cluster(prepSquare, maskLine,
keepI0=keepI0, kLst=config.srcLst.clusterConfig['optimized'])
return(labelLine, scoreTuple)
/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
from .autonotebook import tqdm as notebook_tqdm
from queso_cluster.loaders.event import eventInput
eventManager = eventInput("./eventManager.yml", 1, 0)
epochDev, labelLine = main(eventManager.event)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Cell In[4], line 1
----> 1 from queso_cluster.loaders.event import eventInput
2 eventManager = eventInput("./eventManager.yml", 1, 0)
3
4
ImportError: cannot import name 'eventInput' from 'queso_cluster.loaders.event' (/nfs/hl0/data/sriley/src/sriley/lib/python3.11/site-packages/queso_cluster/loaders/event.py)
Time-dependent clustering#
Here is where we enter more experimental waters. The main objective is to try and use the time information inside the algorithm where each cluster is a sequence of spectral profiles. One example of this was presented at the Solar Physics Division conference in 2025, where we clustered each frame independently then glued the results together at the end. The benefit is a smaller number of clusters is needed per time step due to a smaller diversity of spectral features relative to the entire set of frames.
This section of the tutorial will discuss the td sub-module with the same time series spectral dataset as before.
import numpy as np
from queso_cluster import td
# from queso_cluster.runners import base as runBase
# from queso_cluster.atoms import norm as normAtom
from queso_cluster.loaders.visp import visp
def main(config):
#> detail:
#> param type config:
#> return (type):
#> test-method:
# srcUse = config.runners.config['primary']['src']
# srcConfigPrimary = config.srcLst[config.srcLabelLst.index(srcUse)]
# config.srcLst = srcConfigPrimary
ViSPobj = visp('/disk/data/DKIST/20230503/AEVEG/')
ViSPobj.load()
evoDev = td.timeDependent(config, config.runners.label, ViSPobj)
#> Note: Creates a mask for data within a specific coordinate range
bboxMask = np.zeros((evoDev.rasterSize, evoDev.alongSlitSize))
bEx = config.runners.config['bbox']
bboxMask[bEx[0]:bEx[1], bEx[2]:bEx[3]] = 1
bboxMask = bboxMask.reshape(bboxMask.shape[0]*bboxMask.shape[1])
#> Note: Here I know that there is a bright feature at time step 2. Then, I want four frames centered on that peak -2, -1, +1, +2
timeFrames, tLst = evoDev.timeFrames(peakTime=2, nframes=4)
#> Note: As before, we are generally more interested in the brighter features so we will opt to mask out the lower intrinsic bins
#> Note: In this case, we will use the locations of bright pixels in the peak frame to focus the clustering.
keepI0 = None
if "keepI0" in list(config.runners.config.keys()):
keepI0 = config.runners.config['keepI0']
intrinsicLine = base._mainIntrinsic(config.srcLst,
np.floor(timeFrames[1, ...]*100)/100., 0, intrinsicSkip=False)
intrinsicLine = auxAtom.pick_jth_label(intrinsicLine, 0).astype(int)
#> Note: This combines both the coordinate mask and the intrinsic bin mask
maskLine = bboxMask.astype(bool)#np.ones(prepSquare.shape[0]).astype(bool)
if not (keepI0 is None):
i0Mask = np.zeros(timeFrames.shape[1], dtype=bool)
for i in keepI0:
i0Mask[(intrinsicLine == i)] = 1
maskLine *= i0Mask
#> Note: This is our list of cluster groups for each time step
klst = config.runners.config['primary']['S1']
#> Note: This is a specific fucntion which prepares the whole time sequence
prepCube = evoDev.prepSequence(timeFrames, norm=normAtom.normContinuum, continuumIndx=self.continuum)
#> Note: Now the magic happens. Right now this will simply cluster each frame seperately then combine the labels to form a sequence.
labelSquare = evoDev.clusterSequence(prepCube, maskLine, klst, intrinsicLine=intrinsicLine)
return(evoDev, labelSquare)
from queso_cluster.loaders.event import eventInput
eventManager = eventInput("./eventManager.yml", 1, 0)
epochDev, labelLine = main(eventManager.event)