feat(memory): operator alert when embedding/reranker model provisioning fails

feeds/skills/.system/files/netclaw-memory/SKILL.md

@@ -3,7 +3,7 @@ name: netclaw-memory
 description: "REQUIRED when the user asks what you remember, recall, or know from past conversations, previous sessions, cross-session memory, memory classes, or memory types. Also before using memory tools: find_memories, get_memories, store_memory, update_memory."
 metadata:
   author: netclaw
-  version: "1.13.0"
+  version: "1.14.0"
 ---
 
 # Netclaw Memory
@@ -288,9 +288,16 @@ Useful log events:
 
 Embeddings are provisioned at daemon start when `Memory.Embeddings.Enabled` is
 `true` (default `false` for now). When unavailable:
-- Log: `memory_embedding_unavailable`
+- Log: `memory_embedding_unavailable` (embedder) or `memory_relevance_gate_unavailable`
+  (relevance/cross-encoder model)
 - Daemon status shows: `embeddings: degraded`
 - Lexical recall continues to work normally
+- An operator alert (`memory.embedding_model.unavailable` /
+  `memory.relevance_model.unavailable`, pushed via the same notification sink as
+  `provider.unreachable`/`reminder.execution.failed`) fires once per model per
+  daemon run, naming the model, the failure reason, and the consequence (lexical-only
+  recall/dedup, or an unfiltered relevance gate) — this is the push-based signal;
+  `netclaw doctor`/`netclaw status` remain the pull-based ones
 
 `netclaw doctor`'s Memory Embeddings check reports whether the active model
 has a query prefix (`queryPrefix=True/False`) and the effective retrieval

feeds/skills/.system/files/netclaw-operations/SKILL.md

@@ -3,7 +3,7 @@ name: netclaw-operations
 description: "REQUIRED when the user asks about scheduling, reminders, cron jobs, timers, background jobs, diagnostics, troubleshooting, MCP tools, daemon health, identity updates, or Netclaw capabilities and self-maintenance."
 metadata:
   author: netclaw
-  version: "2.25.0"
+  version: "2.26.0"
 ---
 
 # Netclaw Operations

feeds/skills/.system/files/netclaw-operations/references/diagnostics.md

@@ -28,8 +28,10 @@ Log split — one stream, partitioned locally by session:
   id — nothing is duplicated locally.
 - `daemon.log` holds only sessionless, daemon-wide lines: startup/config, session
   start/stop, and operational **alerts** (e.g. the `provider.unreachable` /
-  `provider.failover` alert raised when an inference provider goes down — surfaced
-  here, and to webhooks, by the notification sink). Note the *per-call* failover/retry
+  `provider.failover` alert raised when an inference provider goes down, or
+  `memory.embedding_model.unavailable` / `memory.relevance_model.unavailable` when a
+  memory ONNX model fails to provision or load — surfaced here, and to webhooks, by
+  the notification sink). Note the *per-call* failover/retry
   log lines emitted while serving a specific session carry that session's id, so they
   partition into its `session.log`; the daemon-wide outage signal is the alert in
   `daemon.log`. Rolled daily, capped at 10 MB per file.
@@ -75,6 +77,7 @@ debugging a daemon-wide problem → read `daemon.log`.
 | No LLM responses | `netclaw doctor`; verify provider credentials |
 | Missing tools | `netclaw mcp list`; check MCP connection state |
 | Memory recall degraded | `netclaw status` memory section |
+| Memory embedding/relevance model unavailable | Fires a `memory.embedding_model.unavailable` / `memory.relevance_model.unavailable` operational alert (once per model per daemon run) naming the model, failure reason, and consequence when `Memory.Embeddings.Enabled=true` and either ONNX model fails to provision or load; see `netclaw-memory`'s Embeddings section and `netclaw doctor`'s Memory Embeddings / Memory Relevance Gate checks |
 | Daemon won't start | crash logs at `~/.netclaw/logs/crash-*.log` |
 | Docker daemon cannot create `/home/netclaw/.netclaw/*` | Official image entrypoint repairs writable bind mounts to UID/GID `1654:1654`; if bypassed or read-only, run `sudo chown -R 1654:1654 <host-data-dir>` or use a Docker named volume |
 | Discord/Slack channel offline | `netclaw status` shows the channel `disconnected` with a reason. Discord may also report `degraded` when Discord.Net says the socket is connected but the gateway is not ready, such as after a resumed session that Netclaw is replacing with a clean reconnect. A misconfigured channel (bad token, missing Discord Message Content intent) degrades only that channel — the daemon keeps running and other channels are unaffected. A transient network failure retries automatically; a config/permission failure stays offline until the operator fixes the config and restarts the daemon. |

src/Netclaw.Configuration/OperationalAlert.cs

@@ -37,6 +37,8 @@ public enum AlertType
     DaemonStopping,
     DaemonCrashed,
     UpdateAvailable,
+    MemoryEmbeddingModelUnavailable,
+    MemoryRelevanceModelUnavailable,
 }
 
 /// <summary>

src/Netclaw.Daemon.Tests/Services/EmbeddingWarmupHostedServiceTests.cs

@@ -286,6 +286,137 @@ public async Task Relevance_gate_disabled_config_leaves_the_relevance_holder_at_
         Assert.Same(initialRelevance, relevanceHolder.Current);
     }
 
+    // ── Operator alerting (memory embedding/reranker provisioning-failure alert) ──
+
+    [Fact]
+    public async Task Embedder_provisioning_failure_emits_exactly_one_operator_alert_naming_the_model_and_reason()
+    {
+        // No PrePlaceValidModelFiles() call -- the embedder fails. The relevance model succeeds so
+        // only the embedder's alert is under test here.
+        PrePlaceValidRelevanceModelFiles();
+
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "warmup not yet run"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = true, ModelId = ModelId, AutoDownload = false } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(relevanceHolder.Current.IsAvailable);
+        var alert = Assert.Single(sink.Alerts);
+        Assert.Equal(AlertType.MemoryEmbeddingModelUnavailable, alert.Category);
+        Assert.Equal(ModelId, alert.Source);
+        Assert.Contains(ModelId, alert.Summary);
+        Assert.Equal(ModelId, alert.Context?["modelId"]);
+        Assert.False(string.IsNullOrWhiteSpace(alert.Context?["reason"]));
+        Assert.Contains("lexical-only", alert.Context?["consequence"]);
+        Assert.Contains("netclaw doctor", alert.Context?["remediation"]);
+    }
+
+    [Fact]
+    public async Task Relevance_model_provisioning_failure_emits_exactly_one_operator_alert_naming_the_model_and_reason()
+    {
+        // Embedder succeeds; the relevance model fails (no PrePlaceValidRelevanceModelFiles call).
+        PrePlaceValidModelFiles();
+
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "warmup not yet run"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = true, ModelId = ModelId, AutoDownload = false } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(holder.Current.IsAvailable);
+        var alert = Assert.Single(sink.Alerts);
+        Assert.Equal(AlertType.MemoryRelevanceModelUnavailable, alert.Category);
+        Assert.Equal(RelevanceModelId, alert.Source);
+        Assert.Contains(RelevanceModelId, alert.Summary);
+        Assert.Equal(RelevanceModelId, alert.Context?["modelId"]);
+        Assert.False(string.IsNullOrWhiteSpace(alert.Context?["reason"]));
+        Assert.Contains("relevance gate is disabled", alert.Context?["consequence"]);
+        // The relevance model has no backfill-embeddings analogue -- its remediation must not
+        // suggest that command (mirrors MemoryRelevanceGateDoctorCheck's own wording).
+        Assert.DoesNotContain("backfill-embeddings", alert.Context?["remediation"]);
+    }
+
+    [Fact]
+    public async Task Both_models_failing_emits_two_distinct_operator_alerts()
+    {
+        // Neither PrePlaceValidModelFiles() nor PrePlaceValidRelevanceModelFiles() is called --
+        // both models fail to provision independently.
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "warmup not yet run"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = true, ModelId = ModelId, AutoDownload = false } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        Assert.False(holder.Current.IsAvailable);
+        Assert.False(relevanceHolder.Current.IsAvailable);
+        Assert.Equal(2, sink.Alerts.Count);
+        Assert.Contains(sink.Alerts, a => a.Category == AlertType.MemoryEmbeddingModelUnavailable);
+        Assert.Contains(sink.Alerts, a => a.Category == AlertType.MemoryRelevanceModelUnavailable);
+        // Distinct alert ids -- these are two independent events, not one duplicated.
+        Assert.NotEqual(sink.Alerts[0].AlertId, sink.Alerts[1].AlertId);
+    }
+
+    [Fact]
+    public async Task Success_path_emits_no_operator_alerts()
+    {
+        PrePlaceValidModelFiles();
+        PrePlaceValidRelevanceModelFiles();
+
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "warmup not yet run"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = true, ModelId = ModelId, AutoDownload = true } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        Assert.True(holder.Current.IsAvailable);
+        Assert.True(relevanceHolder.Current.IsAvailable);
+        Assert.Empty(sink.Alerts);
+    }
+
+    [Fact]
+    public async Task Disabled_config_emits_no_operator_alerts()
+    {
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "embeddings disabled"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = false, ModelId = ModelId } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        // Embeddings disabled is an intentional, not degraded, state -- no alert should fire.
+        Assert.Empty(sink.Alerts);
+    }
+
+    [Fact]
+    public async Task Provisioning_failure_alert_is_latched_and_does_not_refire_across_repeated_warmup_runs()
+    {
+        // Neither model's fixture files are placed -- both fail every time WarmUpAsync runs.
+        var holder = new MemoryEmbedderHolder(new UnavailableMemoryEmbedder(ModelId, "warmup not yet run"), initialQueryPrefix: "", initialCalibratedMinCosineSimilarity: null);
+        var relevanceHolder = CreateRelevanceScorerHolder();
+        var memoryConfig = new MemoryConfig { Embeddings = { Enabled = true, ModelId = ModelId, AutoDownload = false } };
+        var sink = new FakeNotificationSink();
+        var service = CreateService(holder, memoryConfig, relevanceHolder, RelevanceFixtureAllowlist(), notificationSink: sink);
+
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+        await service.WarmUpAsync(TestContext.Current.CancellationToken);
+
+        // Exactly one alert per model in total across both runs -- the latch, not the retry count,
+        // governs how many alerts an operator sees.
+        Assert.Equal(2, sink.Alerts.Count);
+        Assert.Single(sink.Alerts, a => a.Category == AlertType.MemoryEmbeddingModelUnavailable);
+        Assert.Single(sink.Alerts, a => a.Category == AlertType.MemoryRelevanceModelUnavailable);
+    }
+
     // ── Keep-warm ticks (memory-relevance-gate 2026-07 canary fix) ──
     //
     // These tests exercise KeepWarmTickAsync/KeepWarmLoopAsync directly against simple signaling
@@ -415,9 +546,11 @@ private EmbeddingWarmupHostedService CreateService(
         MemoryConfig memoryConfig,
         RelevanceScorerHolder relevanceScorerHolder,
         IReadOnlyDictionary<string, RelevanceModelManifestEntry> relevanceAllowlist,
-        TimeProvider? timeProvider = null)
+        TimeProvider? timeProvider = null,
+        IOperationalNotificationSink? notificationSink = null)
         => new(_provisioner, _store, holder, relevanceScorerHolder, _allowlist, relevanceAllowlist, memoryConfig, _paths,
-            timeProvider ?? TimeProvider.System, NullLogger<EmbeddingWarmupHostedService>.Instance);
+            timeProvider ?? TimeProvider.System, notificationSink ?? NullNotificationSink.Instance,
+            NullLogger<EmbeddingWarmupHostedService>.Instance);
 
     private static RelevanceScorerHolder CreateRelevanceScorerHolder()
         => new(new UnavailableRelevanceScorer(RelevanceModelId, "warmup not yet run"), initialCalibratedThreshold: 0.0);
@@ -550,4 +683,17 @@ public ValueTask<IReadOnlyList<double>> ScoreAsync(string query, IReadOnlyList<s
             return ValueTask.FromResult<IReadOnlyList<double>>(candidates.Select(_ => 1.0).ToArray());
         }
     }
+
+    /// <summary>
+    /// Captures every <see cref="OperationalAlert"/> emitted during a test — mirrors
+    /// <c>McpReconnectionServiceTests.FakeNotificationSink</c>'s shape. Tests below only ever
+    /// await <c>WarmUpAsync</c> to completion before inspecting <see cref="Alerts"/>, so no
+    /// additional synchronization is needed.
+    /// </summary>
+    private sealed class FakeNotificationSink : IOperationalNotificationSink
+    {
+        public List<OperationalAlert> Alerts { get; } = [];
+
+        public void Emit(OperationalAlert alert) => Alerts.Add(alert);
+    }
 }