SIGN IN SIGN UP
koala73 / worldmonitor UNCLAIMED

Real-time global intelligence dashboard. AI-powered news aggregation, geopolitical monitoring, and infrastructure tracking in a unified situational awareness interface

0 0 157 TypeScript

fix(cyber-seed): populate firstSeenAt for all sources (#4008) (#4011)

* fix(cyber-seed): populate firstSeenAt for all sources (#4008)

firstSeenAt was 0 for ~100% of cyber:threats:v2 records (verified live
2026-06-01: 0/1006 non-zero). Two causes:

1. Parse bug: URLhaus /v1/urls/recent/ returns `date_added` (snake_case) but
   the seeder read `dateadded`, so URLhaus (180) + the same-field lastSeen both
   fell through to 0. Fixed the field name (old key kept as fallback).
2. Structural gap: AbuseIPDB blacklist (500) and C2Intel plaintext (300) carry
   no upstream first-seen at all.

Fix: persist a WorldMonitor-observed first-seen per indicator across runs
(`cache:cyber:first-seen:v1`). Upstream discovery date wins when present
(min, so first-seen never moves later); otherwise the first run that observes
an indicator stamps it and subsequent runs carry it forward. The map is rebuilt
from the current feed each run, so it self-prunes to ~feed size.

This is the prerequisite for genuine burst-vs-sustained cyberDigital
discrimination (#3971 discovery-day grouping / #4009 EWMA). No effect on current
scoring: the merged #4006 per-snapshot cap reads neither firstSeenAt nor
lastSeenAt, so no cache bump is needed.

- mergeObservedFirstSeen: pure, exported, unit-tested merge logic
- applyObservedFirstSeen: non-fatal Redis read/write wrapper
- isMain guard so the test can import the helper without running the seed
- tests/seed-cyber-first-seen.test.mjs: 7 cases (stamp/carry-forward/upstream-
  wins/min/self-prune/cross-source/invalid-prior)

Fixes #4008

* fix(cyber-seed): make cross-source first-seen order-independent (two-pass)

Addresses Greptile P1 on #4011: when an indicator appeared in multiple sources
in one run, the single-pass merge finalized each occurrence as it was visited,
so on the first run (empty prior) the dated and undated rows could end up with
different firstSeenAt in the same snapshot — and the suggested next[key] fallback
only fixed it when the dated row came first.

Two-pass merge: pass 1 folds the earliest real date per indicator across all
occurrences + prior; pass 2 stamps nowMs for keys with no real date anywhere and
assigns the resolved value to every occurrence. Now every row for the same IOC
shares one first-seen on the first run, regardless of source row order.

Added an order-reversed regression test (dated row second).

Refs #4008
E
Elie Habib committed
67554c3ce1effc37229ac16ac4f1db0b64b2ba1b
Parent: 7d5c77d
Committed by GitHub <noreply@github.com> on 6/1/2026, 8:43:13 AM