Volume 09 — Authoring Formats: YAML Grammar, Packages, Compilation

Programmes for the Nest VMNest runtime / Nest virtual machine (Nest VM)The whole machine this set specifies: wave log + processors (topology and observers) + membrane + settle loop. In the reference project the bare word Nest historically named the workbench shell; this set reserves the machine sense and calls the workbench Nest Studio. (Vol. 00 §5)Volume 15 — Terminology are authored outside the machine as YAML documents and compiled by a loaderLoaderThe component that splits an authoring pipeline into individual records and commits them as seeds. aka splitter. (Vol. 09 §6)Volume 15 — Terminology into individual recordsIndividual recordA single activation-knot or bind-descriptor definition, split out of the authoring pipeline at load and committed to the wave log. (Vol. 04 §2)Volume 15 — Terminology committed to the log. This volume is the current snapshot of that grammar — the exact closed key sets, the validation rules, the package layout, the JSON Schema subset, and the compilation algorithm — plus the format-level compatibility rules that make pipelines portable between implementations. Constructs the format declares but the runtime does not yet execute are marked DECLARED and specified in Vol. 11 §4–5.

§1. Two artefact classes

1. Loadable authoring pipelinesAuthoring pipelineThe external, human-authored document describing a whole scenario. It lives outside the machine. (Vol. 09)Volume 15 — Terminology — documents this volume's grammar accepts; the compiler turns them into seeds. The reference implementation bundles its pipelines in one catalogue tree (refimpl 00 §4, refimpl 08 §2.2).
2. Reference artefactsReference artefactA format document fixing target vocabulary that the current compiler deliberately rejects beyond its stage. (Vol. 09 §1)Volume 15 — Terminology — YAML records documenting target shapes: knotActivation knotAn accumulating unit that matches wave tuples, winds them, and tests readiness; it accumulates understanding. When ready, it activates bind descriptors. Its minimal form is a single condition matcher — a degenerate configuration of the same accumulator, not a different kind. aka knot, clew (for the per-key instance see clew). (Vol. 05)Volume 15 — Terminology, bind, and controllerOutput controllerA receiver of committed tuples facing outward: it claims tuples by kind/fact type and discharges them to the external world exactly once per runtime. It neither judges nor integrates — a third station beside the activation knot and the bind descriptor, standing outside the propagation cycle on its own cursor. aka effector, egress handler. (Vol. 07 §2–3)Volume 15 — Terminology records, and one full-shape pipeline declaring the reserved future constructs (illustrated in Vol. 11 §5). They fix vocabulary and direction; the compiler deliberately rejects their beyond-stage constructs with explicit messages. A conforming implementation MUST NOT silently accept or drop them.

§2. Pipeline document: top level

pipeline: <id>            # required, non-empty
title: <string>           # optional
runtime:                  # optional; validated, constrains nothing further yet
  mode: single-thread     # only value at this stage
  isolation: declared-only
inputs:    [ … ]          # §3
branches:  [ … ]          # §4 (DECLARED beyond 'main')
knots:     [ … ]          # §5.1
descriptors: [ … ]        # §5.2

Closed key set: pipeline, title, runtime, inputs, branches, knots, descriptors. Unknown keys are errors. Throughout the grammar, every mapping has a closed key set and every unknown key is reported with its path — the uniform diagnostic discipline of §6.

§3. Inputs

inputs:
  - id: message                     # unique
    writes: chat.message.received   # fact type the shell commits
    field: message                  # payload field for the value; default 'value'

Inputs declare the machine's ingressIngressThe inward half of the membrane: the edge through which external events — the world's answers among them — are committed to the wave log as fact tuples. (Vol. 07 §6)Volume 15 — Terminology surface. Shells generate their run forms from compiled.inputs — the UI draws what the compiler declares, nothing more. At run start each supplied value becomes one domain.fact {factType: writes, data: {[field]: value}} on the run key; repeated values are permitted (arrays produce one fact each, in order).

§4. Branches (DECLARED)

branches:
  - { id: main, role: root }

main is implicit and the only executable lane at this stage. Additional branch declarations are validated (unique non-empty ids); any listen_on/on_branch other than main is rejected with "only the 'main' branch is executable at this stage". The construct reserves the plane/level machinery of Vol. 11 §4.3.

§5. Records

§5.1 Knots

Common shape and keys (id, strategy, listen_on, wind, condition):

knots:
  - id: intake.ready
    strategy: deterministic            # or semantic_evaluator
    listen_on: main                    # optional; must be 'main'
    wind:
      collect:                         # ≥ 1 rule; keys: as, match|match_type, reduce, field, where
        - { as: topic, match: topic.received, reduce: latest }
        - { as: query, match: query.received, reduce: latest }
      # semantic-only: integrate: local|through_world · lane: <string> · budget: <int ≥ 1>
      reset: on_ready                  # or never
    condition: "topic != null && query != null"      # deterministic: expression string

  - id: intent.understanding
    strategy: semantic_evaluator
    wind:
      collect:
        - { as: user_messages, match: chat.message.received, reduce: append, field: message }
      integrate: through_world
    condition:                         # semantic: structured mapping
      questions: ["Is the user's end goal clear?", "…"]
      threshold_grade: 0.8
      # evaluate_understanding is implied true by compilation

Validation highlights (each with a path-anchored error):

• wind.collect non-empty; rule keys closed (as, match/match_type, reduce ∈ append|latest|overwrite, field, where with field+equals/not_equals);
• deterministic: condition must parse in the null-guard subset over declared slot names; integrate forbidden; duplicate as names forbidden;
• semantic: condition requires ≥ 1 question string and a numeric threshold_grade; lane non-empty string; budget positive integer;
• condition keys evaluate and polarity are recognised as future and rejected as "declared but not yet supported at this stage" (guard polarity, Vol. 11 §5).

§5.2 Descriptors

Keys: id, on, on_as, action, writes, on_branch, demands, gates, service. Two actions compile today:

descriptors:
  - id: notify.analyst          # emit descriptor
    on: intent.understanding    # must name a declared knot
    action: emit
    writes: intent.understanding.ready

  - id: frame.service           # operator bind
    on: frame.truth             # optional; must name a declared knot
    on_as: frame                # scope name of the activation entry
    action: operator
    demands:                    # named bindings; 'as' unique across scope
      - { as: journal, knot: reasoning.journal }
    gates:
      - { scope: frame, min_grade: 0.75 }     # scope name must be bound
    service:
      instruction: >
        Form the answer required by the frame.
      schema: schemas/frame.schema.json       # file ref, resolved and embedded
      emit: { writes: problem.frame.ready }   # or unfold: (§5.3)

Validation highlights: emit descriptorsEmit descriptorThe minimal bind form: one activation, immediate truthful publication. (Vol. 06 §2)Volume 15 — Terminology require on + writes and forbid operator-only keys; operator bindsOperator bindThe full bind form: activation + demands + gates + a service crossing the membrane. (Vol. 06 §3)Volume 15 — Terminology forbid top-level writes (the emit lives in service), check every demandDemandA named binding (as:) to an activation knot whose integrated understanding the bind descriptor gathers back. The gathering face of affinity, and the second method through which understanding enters the bind. aka tendril. (Vol. 06 §3–4)Volume 15 — Terminology knotActivation knotAn accumulating unit that matches wave tuples, winds them, and tests readiness; it accumulates understanding. When ready, it activates bind descriptors. Its minimal form is a single condition matcher — a degenerate configuration of the same accumulator, not a different kind. aka knot, clew (for the per-key instance see clew). (Vol. 05)Volume 15 — Terminology exists, forbid duplicate scope names, check gateGateA predicate over the bound scope. Gates decide between emit and reject; the current form is min_grade over a named entry. (Vol. 06 §5)Volume 15 — Terminology scopes against the statically known scope names; future actions (unfold as an action, and the keys affinity, synthesize, fanout, generate, terminal) are rejected as future constructs (Vol. 11 §5).

§5.3 The unfold emit template

    service:
      instruction: "Unfold the problem into 3–5 orthogonal questions."
      schema: schemas/questions.schema.json    # REQUIRED for unfold
      emit:
        unfold:
          for_each: questions                  # schema property: array with items
          knot:                                # sown per item; id must contain {index}
            id: "cell.q{index}"
            strategy: semantic_evaluator       # only strategy at this stage
            wind:
              collect:
                - { as: seed, match: question.seeded, reduce: append, field: text,
                    where: [ { field: cell, equals: "q{index}" } ] }
              integrate: through_world
              lane: "q{index}"
              budget: 4
            condition: { questions: ["{item.question}"], threshold_grade: 0.7 }
          head:                                # fact seeding each cell
            writes: question.seeded
            data: { cell: "q{index}", text: "{item.question}", emphasis: "{scope.frame}" }
          close:                               # the closing bind, once
            id: harvest.frame
            demands_from_items: { as: "q{index}", knot: "cell.q{index}" }   # {index} required
            service:
              instruction: "Integrate the ripened answers."
              schema: schemas/harvest.schema.json
              emit: { writes: problem.frame.ready }    # simple writes only

Static coherence enforced at compile time (Vol. 06 §7.3): for_each names an array property with items in the declared schema; every {item.*} path exists in the items schema; every {scope.*} names a bound scopeBound scopeThe record of names — from the activation tuple and the demands — over which gates and the service are evaluated. (Vol. 06 §4)Volume 15 — Terminology entry; knotActivation knotAn accumulating unit that matches wave tuples, winds them, and tests readiness; it accumulates understanding. When ready, it activates bind descriptors. Its minimal form is a single condition matcher — a degenerate configuration of the same accumulator, not a different kind. aka knot, clew (for the per-key instance see clew). (Vol. 05)Volume 15 — Terminology id and demands_from_items carry {index}; the closing serviceServiceThe function evaluated over the bound scope to produce the integration. Its kind (model instruction, deterministic function, other) is an assembly property. aka service function. (Vol. 06 §3, Vol. 08 §5)Volume 15 — Terminology may not unfoldUnfoldThe constrained in-wave authoring form: a service result instantiates declared knot/close/head templates. (Vol. 06 §7)Volume 15 — Terminology.

§6. Compilation algorithm and diagnostics

compile(document, resolveSchemaRef?) → { pipelineId, title, errors[], seeds[], inputs[] }
 1  validate the top level (closed keys; runtime values)
 2  collect branch ids ('main' implicit); validate inputs (unique ids)
 3  for each knot: validate; build the strategy config; parse deterministic
    conditions; seeds += sys.knot.defined
 4  for each descriptor: validate against declared knot ids; resolve and embed
    schemas (assertSupportedSchema); compile emit declarations and templates;
    seeds += sys.descriptor.defined
 5  if errors ≠ ∅: seeds ← []                        // all-or-nothing

Normative rules:

1. Explicit rejection. Unsupported syntax is rejected with a path-anchored message (knots[1].condition: …); nothing is silently dropped. Future-stage keys get the distinct message "declared but not yet supported at this stage".
2. All-or-nothing. A document with any error compiles to zero seeds. There is no partial registration from authoring (the same atomicity the Cell admission extends to runtime-authored packages, Vol. 12 §6, check 6).
3. Self-contained records. Schema file references are resolved at compile time and embedded into the records, so a run never depends on mutable files. The resolver is a callback: file-based for packages, in-memory for drafts and (in the extension) for blueprint-embedded schema maps — one compiler, many sources.
4. One compiler. Every shell and every extension MUST reuse the same compilation functions; two grammars for one machine are non-conforming (restated for blueprints in Vol. 12 §6, check 6).
5. Diagnostics are data (stable paths, one message per violation), suitable for attachment to authoring UI elements.

§7. The JSON Schema subset

Result contracts embed a deliberately closed JSON Schema subset:

• keywords: type, properties, required, items, enum, description, minItems, maxItems, minimum, maximum;
• types: object, array, string, number, integer, boolean.

Rules:

1. assertSupportedSchema rejects any other keyword explicitly at compile time.
2. Validation semantics (controller-side, Vol. 07 §4.2): enum by strict equality; objects check required presence and recurse into declared properties (extra properties are currently permitted); arrays check minItems/maxItems and recurse into items; numbers check minimum/maximum and integrality for integer.
3. Expansion of the subset (unions, $ref, patterns, additionalProperties) is a deliberate extension decision (Vol. 11 §4.6): admission-time validation remains the final authority even where an outer schema is permissive.

§8. Packages

A pipeline that declares schemas is a package: a directory with pipeline.yaml beside schemas/*.schema.json. Loading resolves service.schema references relative to the package (never outside it, refimpl 07 §1). Drafts and templates are ordinary packages in dedicated catalogue subtrees — a template additionally carries a template.json provenance sidecar: source draft, proving run, promotion time (refimpl 08 §2.2, §2.4). Promotion of a draft requires a settledSettledQuiescent with no discharge in flight. (Vol. 03 §7)Volume 15 — Terminology proving run of that same pipeline; it refuses non-drafts, unknown or unsettled runs, and runs of a different pipeline.

The seed direction for richer packages (case/evidence/rubric/manifest layers) is an extension concern (Vol. 13 §4–5; the proposal's library layout in Vol. 12 §10) — the executable centre remains this volume's pipeline packagePipeline packageA directory of pipeline.yaml beside schemas/* — the portable programme unit; drafts and templates are ordinary packages. (Vol. 09 §8)Volume 15 — Terminology.

ReachabilityAttribution, reachability, terminationThe three load-bearing obligations of the decoupled model. (Vol. 01 §7)Volume 15 — Terminology note. The current compiler checks reference integrity (knotsActivation knotAn accumulating unit that matches wave tuples, winds them, and tests readiness; it accumulates understanding. When ready, it activates bind descriptors. Its minimal form is a single condition matcher — a degenerate configuration of the same accumulator, not a different kind. aka knot, clew (for the per-key instance see clew). (Vol. 05)Volume 15 — Terminology named by descriptors exist; placeholders resolve) but not end-to-end fact-type reachability; the static input→knot→readinessReadinessThe state an activation knot reaches when its accumulated understanding satisfies its test; readiness is what activates bind descriptors. Readiness is reified: the topology commits it as a tuple carrying the understanding snapshot. (Vol. 02 §5, Vol. 04 §2.3)Volume 15 — Terminology→bind→emit path check is recorded as a validate-stage candidate (Vol. 01 §7) and becomes mandatory at admission in the Cell extension (Vol. 12 §6, check 11).

§9. Format compatibility rules

1. Pipelines are the primary portable artefact between Nest implementations and products. A conforming implementation MUST accept this volume's grammar or reject with diagnostics — never reinterpret.
2. Compiled record shapes (Vol. 04 §2) are the compatibility surface between the loaderLoaderThe component that splits an authoring pipeline into individual records and commits them as seeds. aka splitter. (Vol. 09 §6)Volume 15 — Terminology and the machine; implementations exchanging pipelines MUST produce equivalent records for equivalent documents (the reference proves this with record-for-record fixture tests against the golden demo).
3. Grammar growth is additive behind explicit stage gatesGateA predicate over the bound scope. Gates decide between emit and reject; the current form is min_grade over a named entry. (Vol. 06 §5)Volume 15 — Terminology: a construct moves from rejected-future to compiled only together with its runtime, tests, and an update to this volume (the same-work-item rule of Vol. 00 §7).
4. match (authoring alias) normalises to match_type (canonical record field); latest normalises to the overwrite semantics. Aliases exist only at the authoring surface.