Funktion

A Categorical Dataset Naming System for the FCC datasets

Juraj Smieško (CERN)

IT-FCC Computing Resources Meeting

CERN

3 September 2026

Core Idea: Naming as Algebra

  • A dataset is the deterministic output of a composed chain of pure functions
  • So a serialized expression of that chain is itself a globally unique, self-describing identifier
  • The name is not a pointer into a database — it is the mathematical blueprint of its own creation
  • Built on Symmetric Monoidal Categories: processing steps as morphisms, join() as the monoidal product, names as string diagrams

Functional Taxonomy

Processing Steps

vertical physics pipeline

  • $gen_{N,P,E,G,T}$ — events, process, energy, generator, tune
  • $overlay_{B}$ — mix in a background pool $B$
  • $sim_{D,S}$ — detector version, sim type
  • $rec_{D,R}$ — detector version, reco type

Structural Operations

horizontal data algebra, file-level only

  • range(a,b) — chunk boundaries
  • join(A,B,...) — monoidal product ($\otimes$)
  • drop([c],D) — prune corrupted chunks
  • label(x) — opaque registry-resolved ref

Three Independent Axes of Versioning

  • Schema — a bare name (gen, rec, …) maps to an append-only set of registered signatures; the parser tells them apart structurally (argument count, literal shape), never by resolving a value first
  • Global parameters (software stack, campaign, …) — a comma-separated list in a mandatory environment prefix — never in a step's name or signature
  • Parameter type (DetectorVersion, Tune, …) — every parameter is typed at registration; a type can evolve on its own without touching the signature

The Grammar Registry

  • Version-controlled single source of truth for every Processing Step name and its registered signatures — Structural Operations are fixed grammar primitives, not registry entries
  • A registered signature is strictly immutable once used to register physical data; new signatures may be added under the same name, never altered
  • Every parameter is typed as resolved (opaque, matched against a sub-registry) or literal (parsed directly, e.g. event counts)
  • Two same-typed parameters in one signature (e.g. overlay's two background pools) are told apart by fixed position
  • Every name must carry an environment prefix — a comma-separated list of one or more global parameters

Examples

Overlaying background events

mc_production:key4hep_v2026>sim(IDEA_o2_v03, full,
    overlay(label(bkg_pool_A), gen(10k, ZH, 240GeV, whizard, tune_A)))

Scaling statistics via monoidal joins

mc_production:key4hep_v2026>sim(IDEA_o2_v03, full, join(
    gen(10k, ZH, 240GeV, whizard, tune_A, range(1, 100)),
    gen(10k, ZH, 240GeV, whizard, tune_A, range(101, 200))))

Recovering from a corrupted chunk

mc_production:key4hep_v2026>drop([42], rec(IDEA_o2_v03, standard,
    sim(IDEA_o2_v03, full, gen(1M, ZH, 240GeV, whizard, tune_B))))

Summary

  • Absolute provenance — the name confesses its own derivation
  • Immutability — a changed parameter is a new name, never a mutation
  • Searchability — dataset discovery becomes string parsing
  • Pragmatic scale — designed for HL–LHC-scale, multi-decade operation

Backup

Mapping to Infrastructure

  • Rucio — chunks become Rucio Files; every processing/structural step becomes a Rucio Container nesting its parent, mirroring the syntax tree exactly
  • DiracX — a cache miss unrolls the name into a DAG of jobs, each bound to the software container named by the environment prefix
  • Aliases — human-readable labels live entirely outside the grammar, as an external symlink service