Building Magatama: The Architecture Decisions That Hurt

Magatama is a unified security platform: six pillars covering AppSec, Infrastructure, AI/LLM, Offensive, Runtime, and Compliance. The current architecture is a pnpm monorepo with a shared core, six pillar packages, a dashboard, and a CLI. It looks clean now. Getting here involved three decisions that each cost significant time — one of which I'd reverse.

ShieldX and ShieldY were built as completely independent projects — different repos, different CI, different package namespaces. Migrating two mature codebases into a monorepo took 3 days: reconciling TypeScript configurations, resolving naming conflicts in shared types, aligning test setups.

That cost was worth it. Building them separately first meant each component was independently useful, independently tested, and had real production usage before platform incorporation. Building the platform architecture first would have added overhead to every early decision. Separate-first, unify-later is correct for security components that each need to stand alone.

Magatama pillars communicate through a shared event bus — a TypeScript EventEmitter implementation with typed events. I built this instead of Redis Streams or NATS because I wanted no external dependencies and a simple local development setup.

This was wrong. I now have: no event persistence (events during pillar restart are lost), no replay capability (can't re-process historical security events), no horizontal scaling (EventEmitter doesn't cross process boundaries), and no dead letter handling. Every operational requirement I've encountered since needs capabilities that message queues provide by default.

The correct answer was Redis Streams from day one. Redis runs in a container, operational overhead is minimal, and the feature set matches security event pipeline needs exactly. Migration means rewriting every pillar's event emission and subscription code — I'll do this in Q3 2026. It's unavoidable.

The Magatama dashboard stores all security findings as Lexical JSON — the same rich text format Ghost CMS uses. Strange choice for a security platform. The reasoning: findings need rich formatting (code blocks, tables, severity markers), need to be searchable, and need to render in multiple contexts (web UI, PDF reports, API responses).

Lexical JSON handles all three. Security findings are structured narratives, not database records. The initial friction of implementing a Lexical renderer paid back immediately when generating PDF compliance reports from the same finding data — zero additional transformation layer needed.

The pattern that works well: a pillar registry that each pillar registers with on startup. The registry maintains pillar health, routes events between pillars, and provides the unified API surface for the dashboard. Individual pillars know nothing about each other — only the registry interface.

Adding a new pillar: implement the interface, register on startup, events flow automatically. Removing a pillar: deregister, remove the package, the registry handles the missing pillar gracefully. The architecture is additive — no existing code requires modification to add new capabilities.

A security engineer installs the Magatama CLI, runs magatama init, and gets a configured instance with all six pillars available locally within 5 minutes. Cloud deployment via magatama deploy targets a single server. Local and cloud instances share configuration format and event schema.

That experience didn't exist at the start. It exists now because the pillar registry makes it composable — you don't install six separate security tools, you install one platform with six dimensions of coverage.