Configuration

geoembed.core.config

Configuration objects for the geoembed pipeline.

StorageConfig(backend='parquet', path='./embeddings.parquet', mode='append', optimize_write=True, cluster_columns=None, store_type='local', store_options=dict()) dataclass

Backend-agnostic storage configuration.

Supports Delta Lake (via Spark), local Parquet (via PyArrow), and cloud object stores (via obstore).

Attributes:

Name	Type	Description
backend	Literal['delta', 'parquet']	Storage format — "delta" for Unity Catalog/Delta Lake, "parquet" for files.
path	str	Table name (delta) or file path (parquet).
mode	Literal['append', 'overwrite']	Write mode — "append" or "overwrite".
optimize_write	bool	Enable write optimisation (Delta auto-compaction).
cluster_columns	list[str] | None	Columns for Delta liquid clustering (ALTER TABLE CLUSTER BY). Liquid clustering replaces Z-ordering — it's incremental, automatic, and doesn't require manual OPTIMIZE runs.
store_type	Literal['local', 'az', 's3', 'gcs']	Object store backend for obstore (local, az, s3, gcs).
store_options	dict[str, str]	Backend-specific options (account keys, endpoints, etc.).

PipelineConfig(model='dofa', backend='spark', chip_size=224, batch_size=128, mixed_precision=True, device=None, num_workers=4, prefetch_factor=2, persistent_workers=True, tile_aligned=True, input_storage=StorageConfig(), output_storage=StorageConfig()) dataclass

Top-level pipeline configuration.

DataLoader tuning follows findings from Microsoft's "Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data" (Zaytar et al., 2025): - Block-aligned reads reduce redundant tile decompression - persistent_workers=True eliminates worker spawn overhead - prefetch_factor hides I/O latency (2 local, 8 remote) - num_workers scales with storage type (4 local, 64 remote)

Attributes:

Name	Type	Description
model	str	Model name (from registry) or Embedder instance.
backend	Literal['spark', 'local']	Compute backend — "spark" for distributed GPU, "local" for single-machine.
chip_size	int	Spatial dimension of input chips (e.g., 224 for 224x224).
batch_size	int	Number of chips per inference batch.
mixed_precision	bool	Use FP16/BF16 autocasting during inference.
device	str | None	Explicit device string. If None, auto-detects.
num_workers	int	DataLoader worker processes. Higher for remote (64), lower for local (4-8).
prefetch_factor	int	Batches to prefetch per worker. Higher hides network latency (8 remote).
persistent_workers	bool	Keep workers alive between batches (always True for performance).
tile_aligned	bool	Sort chip reads to align with COG tile boundaries, reducing decompression.
input_storage	StorageConfig	Where to read chip metadata from.
output_storage	StorageConfig	Where to write embeddings to.

get_default_device()

Automatically selects the best available hardware accelerator.

Priority: CUDA > MPS > CPU. Requires torch to be installed.

Source code in src/geoembed/core/config.py

def get_default_device() -> str:
    """
    Automatically selects the best available hardware accelerator.

    Priority: CUDA > MPS > CPU. Requires torch to be installed.
    """
    import torch

    if torch.cuda.is_available():
        return "cuda"
    if torch.backends.mps.is_available():
        return "mps"
    return "cpu"