Logging architecture

ZeroClaw has exactly one logging surface: the zeroclaw_log::record! macro. Every emission in the workspace, agent loop activity, channel I/O, cron runs, tool calls, memory ops, session lifecycle, errors, flows through it. The macro fires a tracing event that the installed subscriber feeds to two sibling layers: the stderr fmt layer (terminal output) and the LogCaptureLayer. The fmt layer prints colored, alias-prefixed lines on stderr (muted unless --verbose). The LogCaptureLayer materializes a structured LogEvent and fans it out, via writer::record_event, to:

1. The optional Observer bridge (observer_bridge::forward) for the subset of actions that map to typed Prometheus / OTel events, but only when a caller has installed a binding with set_observer_bridge. The current production bootstrap does not install one.
2. The optional native OTLP log exporter (export_bridge plus runtime otel_logs) when an observability-otel build runs with backend = "otel".
3. The process-wide broadcast channel for live subscribers such as the dashboard’s SSE stream.
4. The asynchronous JSONL writer for <workspace>/state/runtime-trace.jsonl (when [observability] log_persistence is "rolling", "full", or "rotating").

When configured, the Observer projection, OTLP queue, and broadcast send happen before the persistence enqueue attempt. Those four destinations have different completeness and durability guarantees; sharing one LogEvent does not make them interchangeable.

Read this first: attribution is not attrs

Every log event carries two completely separate channels of structured data. Confusing them is the single most common mistake at a call site, so internalize the split before anything else:

The rule that falls out of this: if a value identifies who or what scope an event belongs to, it comes from a span and must never appear at the call site. Attribution flows in automatically from attribution_span! / scope! wrappers opened higher up the stack; the layer walks the span scope leaf→root when an event fires and merges every contribution into the event’s zeroclaw.* block. The call site that fires record! names none of it.

Because attribution is the load-bearing half of this split and the half that trips people up, it comes first.

Attribution: it all comes from spans

Attribution is never a call-site argument. Read that again. Channel composite, agent_alias, model_provider, tool, session_key, cron_job_id: none of these are ever typed into a record! call. They flow in through tracing spans opened at entry points and walked by the layer when an event fires. If you find yourself wanting to pass agent_alias or tool to record!, stop: the value is already in scope through a span, or it should be, and the fix is to open or fix the span, not to thread the value into the call site.

The mechanism, end to end:

1. A “thing” (channel, provider, agent, tool, cron job, memory backend, …) implements Attributable once, next to its struct.
2. Its entry point wraps work in attribution_span!(self), which opens a tracing span carrying that thing’s role and alias.
3. Every record! fired anywhere inside that span, directly or nested arbitrarily deep, inherits the attribution automatically.
4. When the event fires, the layer walks the span scope leaf→root, merges every Attributable’s contribution, and writes the merged zeroclaw.* block. The call site named none of it.

This is the whole point of the design: per-thing logging code is zero. You impl the trait once and wrap the entry point once; every emission underneath is attributed for free.

When inheriting attribution, a composite group belongs to the nearest span whose stored attribution contains any of its keys: <prefix>, <prefix>_type, or <prefix>_alias. Missing members of that group do not inherit from an outer span unless the inner group contains only <prefix>_type and the outer group has the same type. For example, an inner scope!(channel: "telegram", ...) retains an outer channel = "telegram.outer" and channel_alias = "outer", while an inner scope!(channel: "webhook", ...) stays type-only under that outer Telegram span. An explicit inner composite such as telegram.inner keeps its own instance even when the outer type matches. Plain attribution fields and unrelated composite groups still inherit normally. Late field recording within one span continues to fill that span’s missing fields.

The Attributable trait

Lives in crates/zeroclaw-api/src/attribution.rs so every crate can implement it without depending on zeroclaw-log:

#![allow(unused)]
fn main() {
pub trait Attributable {
    fn role(&self) -> Role;
    fn alias(&self) -> &str;
}
}

Each “thing” in the workspace (a TelegramChannel, an AnthropicModelProvider, an Agent, a cron job, a tool, a memory backend, a peer group, a skill bundle, an MCP bundle, a session) impls Attributable once next to its struct.

The Role taxonomy

Closed nested enum:

#![allow(unused)]
fn main() {
pub enum Role {
    Swarm,
    Agent,
    Channel(ChannelKind),       // Telegram, Discord, Slack, Matrix, Lark, ...
    Tool(ToolKind),             // Shell, HttpRequest, FetchUrl, ...
    Cron(CronKind),             // Interval, At, Cron, Once
    Provider(ProviderKind),     // Model, Tts, Transcription, Tunnel
    Memory(MemoryKind),         // Sqlite, Json, InMemory, Markdown, Qdrant, ...
    PeerGroup,
    Skill,
    Mcp,
    Sop,
    Session,
    System,
}
}

ChannelKind, ToolKind, CronKind, MemoryKind, and the four ProviderKind sub-enums (ModelProviderKind, TtsProviderKind, TranscriptionProviderKind, TunnelProviderKind) are all closed. The variant’s snake_case form via strum::IntoStaticStr is the canonical <type> portion of the <type>.<alias> composite. Adding a new implementation: extend the relevant Kind enum, that’s it.

Opening a span, do this at every entry point

Wrap an entry-point’s work with attribution_span!(thing). The macro returns a Span carrying the thing’s role and alias as structured fields. .instrument(span) the future (or let _g = span.entered() in sync code). A spawned task that does not re-establish the span loses attribution: every tokio::spawn body that emits must carry the same attribution_span! / scope! the parent used, or its emissions land un-attributed.

#![allow(unused)]
fn main() {
use zeroclaw_log::Instrument;

let span = zeroclaw_log::attribution_span!(self);  // self impls Attributable
async move {
    // every record! inside automatically carries the alias-bound fields
    record!(INFO, Event::new(module_path!(), Action::Start), "channel online");
    self.poll_loop().await
}.instrument(span).await
}

The layer walks the span scope leaf→root when an event fires, merges every Attributable’s contribution into the event’s zeroclaw.* attribution block, and emits the composite (channel = "telegram.clamps", channel_type = "telegram", channel_alias = "clamps") without the call site naming any of those keys.

The scope! macro, non-role context

attribution_span! is for role-bearing Attributable things. For per-scope identifiers that aren’t tied to one (sender id, message id, turn id, request id), use scope!:

#![allow(unused)]
fn main() {
zeroclaw_log::scope!(
    sender: msg.sender.as_str(),
    message_id: msg.id.as_str(),
    => async move { process_message(msg).await }
).await
}

SOP step execution uses the same mechanism with sop_run_id. Unlike a turn’s trace_id, this attribution is stable across every nested step turn in the run, so the gateway, CLI, and TUI can all query one durable correlation key.

scope! straddles the attribution/attrs line deliberately: field keys that match the alias-bound ATTRIBUTION_FIELDS / COMPOSITE_PREFIXES (in crates/zeroclaw-log/src/event.rs) land in the typed zeroclaw.* attribution slot; everything else lands in the event attributes map for every descendant emission. Either way the value rides on every nested record! without being a call-site argument.

The record! macro and its call-site contract

The tracing crate is zeroclaw-log’s implementation detail: the record! / scope! / attribution_span! macros expand to zeroclaw_log::__private::tracing so a call site never names a tracing type. The log-event macros themselves (tracing::{trace,debug,info,warn,error}, log::*, std::dbg, plus bare anyhow::anyhow!) are hard-banned workspace-wide as disallowed-macros in clippy.toml. With -D warnings in CI, any direct tracing::info! etc. fails the build, with a clippy message naming ::zeroclaw_log::record! as the replacement. This is not a convention; it is enforced.

The only exemptions are the few files inside crates/zeroclaw-log/ that bootstrap the pipeline and carry a local #![allow(clippy::disallowed_macros)]. A handful of crates (zeroclaw-api, zeroclaw-spawn, zeroclaw-providers, zeroclaw-hardware, zeroclaw-log) still list tracing / tracing-subscriber in Cargo.toml, but only for span and subscriber plumbing, not for emitting log macros. The dependency being present does not license calling the banned macros. (tokio::spawn is banned the same way via disallowed-methods; use ::zeroclaw_spawn::spawn! so spawned tasks inherit the caller’s attribution span.)

The macro is locked-shape: it takes a level, a single Event expression, and a message literal.

#![allow(unused)]
fn main() {
use zeroclaw_log::{record, Event, Action, EventCategory, EventOutcome};

record!(INFO, Event::new(module_path!(), Action::Start), "starting step");
record!(WARN, Event::new(module_path!(), Action::Fail).with_outcome(EventOutcome::Failure).with_attrs(serde_json::json!({"exit_code": 137})), "tool failed");
}

module_path!() is the canonical source of the event name: it’s the Rust module path of the call site (e.g. zeroclaw_channels::telegram), so events are searchable, jump-to-source-able, and impossible to typo. The same convention is used at every record! site in the workspace.

The macro injects file!() and line!() automatically. The LogCaptureLayer attaches them to the event’s attributes map as _file and _line so operators jump to source from a log viewer.

Call-site contract

Every record! call is a single line of code that says what happened, not who did it or under what context.

• The single positional argument after the level is an Event expression.
• The next argument is a string literal for the human-readable message.
• That is everything. Channel, agent_alias, provider, tool, session_key, cron_job_id, model: none of those are call-site arguments. They flow in from spans (see Attribution: it all comes from spans).

The shape is enforced by the Event struct: unknown fields are a compile error.

When attrs are warranted

Event::with_attrs(serde_json::json!({...})) is for per-event measurements and ad-hoc data that exist nowhere in the surrounding scope. Concretely:

• Per-event measurements: bytes_received, tokens_used, retry_count, status_code, queue_depth.
• Error payloads when the error is the event itself: anyhow chain text, HTTP error body, parse-error details.
• External-system identifiers: a remote API’s request_id, an upstream trace header.
• Derived state captured at this instant: in-flight count, retry-after seconds.

Attrs are NOT for anything that comes from the surrounding scope: channel composite, agent_alias, model_provider, tool, session_key, cron_job_id, sender, message_id, etc. Those belong in a wrapping attribution_span! or scope!.

The serde rule: pass the raw value, never format!("{}", v) or format!("{:?}", v). serde_json::json! serializes strings as strings, numbers as numbers, Vec<T> as arrays, Option<T> as null-or-value. Wrap with .to_string() only when the type doesn’t impl Serialize (e.g. anyhow::Error, reqwest::Error, std::io::Error, Path::Display, StatusCode).

Placeholder rule

Rust string-literal placeholders like "raw error body: {body}" are forbidden inside record! messages. Rust 2021’s implicit format-string capture does not flow through record!: every {var} becomes a literal substring with no substitution. The conversion rule:

#![allow(unused)]
fn main() {
// BAD — {body} is a literal, never interpolated
record!(WARN, Event::new(module_path!(), Action::Fail), "raw error body: {body}");

// GOOD — body in attrs, message is plain prose
record!(WARN, Event::new(module_path!(), Action::Fail).with_attrs(serde_json::json!({"body": body})), "raw error body");
}

Event, Action, EventOutcome, EventCategory

All four are closed enums defined in crates/zeroclaw-log/src/event.rs. Adding a value is the only point of change: call sites do not invent strings.

• Action: closed verb set, snake-cased on disk via strum::IntoStaticStr: Start, Complete, Fail, Cancel, Skip, Timeout, Retry, Inbound, Outbound, Send, Receive, Connect, Disconnect, Reconnect, Spawn, Kill, Tick, Trigger, Schedule, Approve, Reject, Defer, Read, Write, Delete, List, Query, Invoke, Dispatch, Resolve, Register, Unregister, Load, Save, Migrate, Validate, Note.
• EventOutcome: Success, Failure, Unknown. Unknown is the default and is skipped on serialization (omitted from the on-disk event.outcome), so a row with no outcome key is implicitly Unknown.
• EventCategory: Agent, Channel, Cron, Memory, Tool, Provider, Session, System, Internal. Derived from the innermost role span unless overridden via Event::with_category(...).

Tool input/output propagation

The central tool executor (crates/zeroclaw-runtime/src/agent/tool_execution.rs::execute_one_tool) wraps every Tool::execute(args) call with invoke/complete/fail events. Each event’s name is module_path!() (the executor’s own module), not a hardcoded string; the Action and severity distinguish them:

1. Before running: record!(DEBUG, Event::new(module_path!(), Action::Invoke).with_category(EventCategory::Tool).with_attrs(...)) with tool, tool_call_id, and the full input in attrs.
2. Runs execute(args).await.
3. On success (r.success): record!(DEBUG, ... Action::Complete) with Outcome::Success, the duration, and tool / tool_call_id / input / output in attrs.
4. On tool-reported failure (!r.success): record!(WARN, ... Action::Fail) with Outcome::Failure, the duration, and tool / tool_call_id / input / error / output in attrs.
5. On Err from execute: record!(ERROR, ... Action::Fail) with Outcome::Failure, the duration, and the debug-formatted error in attrs.

These events are emitted inside a scope!-style span (target = "zeroclaw_log_internal_scope", field tool = <name>) opened around the call, so the tool field rides on every descendant emission too. Per-tool Tool::execute impls add zero logging code.

LogCaptureLayer and the on-disk schema

The layer in crates/zeroclaw-log/src/layer.rs is a tracing-subscriber Layer that:

1. On span creation/record with target "zeroclaw_log_internal_attribution" (the target the attribution_span! macro opens with): parses the role + alias fields into a ZeroclawAttribution snapshot stored on the span’s extensions.
2. On span creation/record with target "zeroclaw_log_internal_scope" (scope!-opened): parses ad-hoc kvps and stashes them similarly.
3. On event emission with target "zeroclaw_log_event" (the target the record! macro fires through): builds a LogEvent from the zc_* field set, walks the span scope leaf→root merging every attribution snapshot it finds, parses the zc_attrs JSON blob into the event attributes, attaches _file/_line from auto-captured source location, and hands the final event to writer::record_event, which fans out in this order:
   • Observer bridge (observer_bridge.rs) for mapped Prometheus / OTel typed events when an Observer is bound.
   • Native structured-log bridge (export_bridge.rs) for every canonical event when an exporter is installed; the observability-otel runtime maps it to an SDK LogRecord and queues OTLP/HTTP protobuf export.
   • Broadcast hook (broadcast.rs) for current SSE/dashboard subscribers when a sender is installed.
   • JSONL persistence (writer.rs), offered last to the asynchronous writer queue only when log_persistence is enabled.

The on-disk JSON shape (LogEvent in event.rs):

{
  "id": "<uuid>",
  "@timestamp": "2026-05-16T10:08:59.002Z",
  "severity_number": 9,
  "severity_text": "INFO",
  "event": { "category": "channel", "action": "inbound", "outcome": "success" },
  "service": { "name": "zeroclaw", "version": "0.8.6" },
  "trace_id": "<turn id>",
  "span_id": "<sub-span id>",
  "zeroclaw": {
    "channel": "telegram.clamps",
    "channel_type": "telegram",
    "channel_alias": "clamps",
    "agent_alias": "clamps",
    "model_provider": "anthropic.clamps",
    "model_provider_type": "anthropic",
    "model_provider_alias": "clamps",
    "model": "claude-sonnet-4-6"
  },
  "message": "inbound message",
  "attributes": { "sender": "...", "_file": "...", "_line": 42 },
  "schema_version": 2
}

@timestamp is chrono::DateTime<Utc> serialized as RFC 3339 with Z. The schema version is 2; older version: 1 rows are migrated in place at daemon startup by migrate::migrate_legacy_jsonl_in_place.

Delivery surfaces have different guarantees

writer::record_event builds the persisted value once, then derives the other deliveries from the same LogEvent. Each destination has a separate contract:

Do not use Observer, OTLP, or SSE delivery to prove that every canonical event was retained. Conversely, do not assume a row absent from JSONL was never emitted: it may have reached OTLP or live broadcast, and the Observer bridge when bound, before the persistence queue dropped or failed it.

Observer events share the same bus

Typed observer events (agent_start, agent_end, llm_request, tool_call, tool_call_start, history_trimmed, error) reach the same broadcast channel through a separate path: zeroclaw_runtime::observability::broadcast. Every observer built by create_observer tees into one process-wide broadcast hook, and an EventBus (the live sender plus a 500-frame history buffer) supplies that hook.

• Under the daemon, daemon::run creates the bus and installs its hook once, whether or not the gateway runs. RPC logs/subscribe therefore carries these frames with the gateway off. A supervised gateway reuses the daemon’s bus and installs nothing, so each frame is delivered once and buffered once.
• A standalone gateway (zeroclaw gateway start) builds and installs its own bus.
• History: GET /api/events/history and RPC events/history (grant Logs:Read) replay the buffered observer frames, oldest first. Like live delivery, they never carry pairing credentials: frames with the ephemeral marker are withheld from history, and logs/subscribe withholds them from live delivery.

Log-layer frames sent directly on the channel are live-only; the history buffer holds observer frames only.

RPC subscriptions are replayable and never end silently

RPC streams do not read the broadcast channel directly. A SubscriptionHub (zeroclaw_runtime::rpc::subscription) copies the bus into bounded rings, one per source, and every subscriber is just a cursor into a ring. Two sources exist today:

• Bounds: each ring is capped at 2,048 frames and 4 MiB, and all rings share a 16 MiB process-wide budget. Over budget, the oldest frame across all rings is evicted.
• Producers never wait for a subscriber; a slow subscriber only falls behind.
• Sequence numbers: start at 1 per source. Every notification carries subscription_id and its seq. The subscribe result returns the newest seq and the hub’s epoch.
• Epochs: every hub (every daemon start or reload) gets a new random epoch, and numbering restarts at 1, so a sequence number means nothing outside its epoch.
• Resume: X/subscribe{since_seq, epoch} replays the frames after since_seq that are still buffered, then continues live, but only when epoch matches.
  • With a different epoch, or none, the client gets subscription/lagged with epoch_changed: true, then every frame the new hub still buffers.
  • A since_seq ahead of the newest frame in the same epoch is refused with INVALID_PARAMS.
• Gaps: a gap is reported, never skipped. A cursor that points at frames that are gone (evicted by the caps or the budget, or overrun on the bus) receives subscription/lagged{subscription_id, from_seq, resume_seq} and continues at resume_seq. from_seq is never greater than resume_seq. A replay batch stops at a gap, so the cursor never jumps over a recorded loss.
• Cancel: subscription/cancel{subscription_id} (grant Logs:Read, the same as subscribing) ends one subscription on the calling connection. Closing the connection ends them all.
• Authority: these streams and events/history are unscoped-only. Their frames come from every principal’s work (session messages, cron results, log lines), and many name no owner, so they cannot be filtered per principal. The caller needs Logs:Read, and it must be an administrator or the unauthenticated shared operator, the same principals that bypass session ownership. An authenticated non-admin principal is refused, even with the * agent selector: addressing every agent is not owning every user’s sessions.
• Recheck: every delivery, whether a frame or a lagged notice, is rechecked against the caller’s live credential. When the policy changes, the principal is resolved again, and demoting it or removing Logs:Read ends the stream.
• Replay reach: an administrator or the shared operator can replay frames from before it connected, up to the ring caps.
• Pairing credentials are dropped before they reach a ring, so neither stream can deliver or replay them.

Reader cursors span the active file and retained archives

GET /api/logs and logs/query call reader::query_log_page, which owns segment enumeration. What it merges depends on the installed storage policy: only rotating owns archives, so only under rotating does the reader merge the active file with retained archives into one logical event stream. Under rolling, full, and none it reads the active file alone, even when timestamped files from an earlier rotating configuration are still present beside it; those are unmanaged and not part of the policy’s logical stream. Segments are scanned oldest-archive-first and the merged result is returned newest-first, identical to the prior single-file semantics. The merge is sequential and in-process; no event is re-serialized.

Segment order comes from a sequence number written into each archive name at rotation time, not from file mtimes. Ordering by mtime would be an observation made at enumeration time, and several rotations landing during one read can leave it describing an order that no longer holds. Archives written before sequence numbering existed carry no number; they are ordered by mtime and sort before every numbered archive.

The primary pagination cursor is next_segment_cursor, an opaque string. Pass it back verbatim as until_segment_cursor to resume; do not construct or parse it. Its internal shape depends on which segment the page ended in, because the three segment kinds have different stability properties:

• A numbered archive is addressed by the sequence number written into its name at rotation. A best-effort sidecar keeps numbers monotonic across normal restarts. If its write fails, retention removes all numbered archives, and the daemon restarts, a number can be reused and an outstanding cursor can bind to newer history.
• A legacy archive, written before sequence numbering existed, is addressed by filename. An archive name is never reassigned to different content, so this is stable in the same way.
• The active file is addressed by offset plus an anchor event id. Its path is stable but its content is replaced on each rotation, so the anchor is what lets the reader detect that a rotation happened and resume from the archive the anchored event now lives in.

The legacy byte-offset cursor next_cursor_line_offset / until_line_offset remains valid and resolves against the active file only. It is None when the oldest event on the current page is in an archive file. For multi-segment deployments, use next_segment_cursor instead.

The offset is not a durable event identity or cross-file checkpoint. A segment cursor becomes unresolvable when the segment it names has been pruned by retention; the reader then reports the history as finished rather than resuming at an unrelated position, so a paging client stops where it was instead of silently jumping back to the newest events. A byte-offset cursor becomes stale whenever the active file’s bytes are replaced or its path changes:

• rotating renames the active file to an archive; the next append creates a new active file.
• schema migration rewrites the active file through a temporary file and atomic rename.
• a daemon config reload can install a new persistence path.

After one of those boundaries, restart pagination from the newest page. The legacy timestamp/ID cursor remains for compatibility but is deprecated under #8012 because lexicographic ID ordering can skip tied events.

Enumeration and reading are not atomic, so a rotation can land between them: the listing names the pre-rotation active file, the rename turns that file into an archive, and the scan reads the fresh replacement instead. The rotated-away events would then sit unlisted between two segments the page did read, unreachable forever because pagination only walks toward older history. The reader re-checks the archive set after the read and redoes the page when it moved. Checking afterwards rather than before is what closes the window, because a check before opening leaves room for a rotation between the check and the open. The same race exists inside enumeration itself: the archives are listed first and the active path is stat’d second, and a rotation landing between those steps hides the data from both halves, because the vacuum between the rename and the next append leaves the active path absent. So the re-check is skipped only for a page that stops short of a listed active file; archives are immutable, so such a scan cannot have raced anything, but a listing with no active file at all is verified like any other.

A segment that cannot be read is logged, left out of the merged view, and marked in the response with incomplete: true rather than failing the whole query, since one bad file should not cost the caller every other segment. at_end is then scoped to what could be read, and the flag is what tells the caller so; the daemon’s own log reaches the operator, not the client that has to decide whether to keep paging. Single-event lookup carries the same distinction: a hit is authoritative however many segments were skipped, but a miss over a skipped segment is reported as unreadable-history rather than not found.

Archive discovery rejects symlinks. Matching the writer’s filename shape is not path confinement, and following a link would let anyone who can write into the log directory point an archive-shaped name at any file the daemon can read. Validation and open are separate operations, so an actor who can replace entries in the log directory can still race a validated archive into a symlink before it is opened; deployments must keep the log directory trusted until the reader uses no-follow or descriptor-stable opens.

Persistence policy owns rewrites and retention

StoragePolicy in config.rs controls only the JSONL destination. Observer, OTLP, and broadcast delivery remain independent of it.

Age and count retention run only after rotation. They do not sweep continuously, do not apply to full or rolling, and do not delete arbitrary neighboring files: archive discovery accepts only names generated from the active path’s archive shape. Under rotating, the live /api/logs reader merges the active file with all retained archives; under the other policies it reads the active file alone. See Reader cursors.

Schema migration is an active-file rewrite

When persistence is enabled and the active path exists, writer::init_from_config runs migrate::migrate_legacy_jsonl_in_place before starting the disk worker. The migrator streams non-empty rows through a temporary file, converts legacy rows with timestamp but no @timestamp, preserves already-current rows, skips malformed JSON with a warning, syncs the temporary file, and atomically renames it over the active path.

Migration is best-effort. Its cheap schema check stops at the first non-empty row. Migration runs when that row is malformed or has timestamp without @timestamp; any other parseable JSON is treated as current, even when it is an unknown or invalid schema, so later legacy rows can remain unmigrated. If migration returns an error, initialization warns and continues, so later v2 appends can coexist with old rows that the v2 reader cannot deserialize. Rotated archives are not migrated.

LogEvent is the schema source of truth. For each schema change, assess migration compatibility, active-file deserialization, HTTP and RPC serialization/consumers, and the architecture and operator documentation; update only the boundaries whose behavior or compatibility changes. The RPC log surfaces live in crates/zeroclaw-runtime/src/rpc/types.rs and dispatch.rs. A migration that replaces the active file invalidates byte-offset cursors, while an additive compatible change that does not rewrite existing bytes does not.

LogConfig vs ObservabilityConfig

zeroclaw-log defines its own minimal LogConfig (in crates/zeroclaw-log/src/config.rs): log_persistence, log_persistence_path, log_persistence_max_entries, log_persistence_max_bytes, log_persistence_rotate_daily, log_persistence_retention_max_files, log_persistence_retention_max_age_days, log_persistence_max_entries_per_segment, log_tool_io, log_tool_io_truncate_bytes, log_tool_io_denylist. This breaks what would otherwise be a dep cycle: zeroclaw-config::ObservabilityConfig carries the full schema (with TOML deserialization and validation), and the runtime converts to LogConfig at startup and after daemon config reload via crates/zeroclaw-runtime/src/observability/runtime_trace.rs::to_log_config. The result: zeroclaw-config can record! without inverting the dep tree, while log persistence and rotation policy changes still take effect on the next daemon reload.

Subscriber installation

The daemon installs the global subscriber via:

#![allow(unused)]
fn main() {
zeroclaw_log::install_global_subscriber(
    recording_filter.as_deref(),   // Option<&str> — the --log-level flag, if set
    &default_filter,               // &str — fallback filter when no flag and no RUST_LOG
    cli.verbose,                   // bool — gates the stderr fmt (terminal) layer
);
}

Two independent axes: the recording floor (what reaches LogCaptureLayer, resolved as flag → RUST_LOG → default) and terminal display (the stderr fmt layer, muted entirely unless verbose is true). That single call sets up the agent-alias-prefixed terminal formatter + the LogCaptureLayer over a tracing-subscriber::Registry. src/main.rs is the only place that calls it. Tests use zeroclaw_log::try_install_capture_subscriber() + zeroclaw_log::subscribe_or_install() to drain emitted events through the broadcast hook without any tracing types named in the test crate.

When to extend the closed enums

• New channel impl: add a variant to ChannelKind. The snake_case form is the on-disk channel_type string. Add #[strum(serialize = "...")] only when the variant name doesn’t snake-case to the desired value (e.g. OpenAi → "openai").
• New tool impl (workspace built-in): add to ToolKind.
• New cron schedule shape: add to CronKind.
• New model / TTS / transcription / tunnel provider: add to the relevant *ProviderKind sub-enum under ProviderKind.
• New memory backend: add to MemoryKind.
• New Role family altogether (PeerGroup / Skill / Mcp gain sub-types): nest with its own Kind on the fly: the pattern is uniform.

Then add impl Attributable for X next to the new struct (fn role() -> Role::Family(Kind::Variant), fn alias() -> &str { &self.alias }) and wrap its entry point with attribution_span!(self). The layer picks up everything else automatically.

Operator concerns

For configuration knobs (log_persistence, log_tool_io, OTel export) and query syntax, see Logs & observability.