Redis -> Redis + Postgres Migration

This guide shows how to move a live Network Manager network from Redis-only to Redis + Postgres, including existing data.

The goal is a safe migration with clear checks at every phase.

Why Migrate?

While the Network Manager's memory footprint remains small, Redis is a poor way to store large amount of data. If you want to use the NetworkAPI database features, then it'll be a good idea to migrate your network to use Postgres for data storage so you can keep your Redis cache small.

What This Migration Changes

When you switch to REDIS_PLUS_SQL, Network Manager:

1. Keeps Redis active for live network coordination.
2. Mirrors writes into SQL for durability.
3. Lets you backfill old Redis data into SQL.
4. Lets you verify parity between Redis and SQL.

So this is not an instant hard cutover. It is a staged migration.

Important

Built-in system namespaces are still seeded as REDIS_ONLY and system-managed. You still get SQL copies for durability/migration workflows. SQL-authoritative cutover is mainly for custom owner.namespace document namespaces.

---

Before You Start

1. Take a Maintenance Window

Do this during low traffic.

Lower churn means cleaner migration results and easier troubleshooting.

2. Confirm All Servers Run the Same Plugin Build

Do not migrate while mixed plugin versions are online.

3. Confirm Redis Is Healthy

From each server console:

netdebug ping

If Redis is not healthy, stop and fix that first.

4. Backup

At minimum:

1. Backup Redis.
2. Backup network-manager-config.json on all nodes.
3. Backup Postgres if you are reusing an existing database.

---

Phase 0: First-Time Postgres Setup (One Time)

If you already have a working Postgres database/user for Network Manager, skip to Phase A.

0.1 Create Database and User

Run this in Postgres as an admin role:

CREATE DATABASE nm_prod;
CREATE USER nm_user WITH PASSWORD 'replace_with_strong_password';
GRANT ALL PRIVILEGES ON DATABASE nm_prod TO nm_user;

Then:

\c nm_prod
GRANT USAGE, CREATE ON SCHEMA public TO nm_user;

How to decode this:

1. CREATE ... or GRANT means success.
2. already exists usually means you already created it.
3. permission denied means your current SQL role is not privileged enough.

0.2 Remote vs Local Setup Checks

If your Postgres is remote (most production networks)

1. Confirm host, port, database, user, and password.
2. Confirm your server IP is allowlisted/firewall-approved.
3. Confirm SSL mode (managed hosts commonly require sslmode=require).

If your Postgres is local (Docker/same machine)

1. Confirm Postgres is actually running and port is mapped.
2. Use local host and disable SSL unless you explicitly configured SSL.

Example local JDBC URL:

jdbc:postgresql://127.0.0.1:5432/nm_prod?sslmode=disable

0.3 Preflight Connection Test (Before Enabling SQL Mode)

Use DBeaver (or any SQL client) with the same credentials you will enter in Network Manager config.

Run:

SELECT current_database(), current_user;
SELECT 1;
CREATE TABLE IF NOT EXISTS public.nm_preflight_test(id int);
DROP TABLE public.nm_preflight_test;

How to decode this:

1. If all queries succeed, connection and schema permissions are good.
2. If SELECT 1 fails, connection details are wrong (host/port/user/pass/SSL).
3. If create/drop fails, your SQL user is missing schema privileges.

0.4 Table Creation Behavior

You do not manually create Network Manager tables.

When SQL mode is enabled with a valid connection, Network Manager creates required tables/indexes automatically.

0.5 Common First-Time SQL Errors

1. password authentication failed: wrong username/password.
2. connection timed out or could not connect: host/port/firewall issue.
3. SSL is required: set JDBC URL with ?sslmode=require.
4. permission denied for schema public: missing USAGE, CREATE on schema.

---

Phase A: Configure Redis + Postgres Mode

1. Collect Connection Values

Gather these values first:

1. Host
2. Port
3. Database
4. Username
5. Password
6. SSL mode requirement

Build JDBC URL:

jdbc:postgresql://<HOST>:<PORT>/<DATABASE>?sslmode=require

For local environments:

jdbc:postgresql://127.0.0.1:5432/nm_prod?sslmode=disable

2. Set SQL Config from Staff Network Panel

Go to:

Staff Network Panel
|- Configs
   |- network-manager-config.json
      |- Database
         |- Sql

Opening SQL config fields from the Staff Network Panel.

Set these values:

Field	Value
Database.PersistenceMode	REDIS_PLUS_SQL
Database.Sql.Engine	POSTGRES
Database.Sql.JdbcUrl	jdbc:postgresql://<HOST>:<PORT>/<DATABASE>?sslmode=require
Database.Sql.Username	<DB_USERNAME>
Database.Sql.Password	<DB_PASSWORD>
Database.Sql.Schema	public
Database.Sql.TablePrefix	nm_
Database.Sql.MirrorEnabled	true
Database.Sql.Migration.Auto	true
Database.Sql.Migration.RedisWinsUnlessEmpty	true

These only need to be set once because these fields are network-synced.

Important

JdbcUrl, Schema, and TablePrefix must match across all nodes. MirrorEnabled must be true. For password fields, avoid accidental leading/trailing spaces.

Controlled Rollout

If you want manual control, set Database.Sql.Migration.Auto=false, restart, and run migration commands manually.

Restart all nodes after Phase A.

---

Phase B: Validate Persistence Is Active

Run on your operator node:

netdebug persistence status

How to decode this:

1. enabled=true means SQL layer is active.
2. health=OK (or briefly MIGRATING) is expected.
3. health=DEGRADED_SQL_UNAVAILABLE means SQL is unreachable.
4. dlqCount should be 0 or drain back to 0.
5. Startup failure fields should be empty.

If enabled=false, do not continue to backfill yet.

Phase B: Persistence status check output.

---

Phase C: Namespace Discovery

1. List registered namespaces

netdebug persistence namespace list

You should see entries like:

• permissions
• adminpanel
• bans
• mutes
• announcements
• players
• staffchat
• party
• networkmanager.persistence_registry

Phase C.1: Registered namespace list output.

2. Discover unknown namespaces in Redis

netdebug persistence namespace discover unregisteredOnly=true

How to decode this:

1. unknownNamespaces=0 means nothing hidden.
2. unknownNamespaces>0 means Redis has namespaces not in registry.

Note

During this phase, you should confirm unknownNamespaces=0 before continuing to cutover. If you only see presence as unknown, that can come from runtime presence keys. For strict cutover preflight, register presence so this check returns zero.

Phase C.2: Example of the presence unknown namespace case.

Register custom namespaces before cutover workflows:

netdebug persistence namespace register owner=siegenet namespace=siegenet.stats enabled=true mirror=true restore=true parity=true

Note

Custom namespaces must be owner-prefixed, like owner.namespace.

Phase C.2: Unknown namespace discovery output.

---

Phase D: Backfill Existing Data into SQL

When Manual Phase D May Be Optional

If Database.Sql.Migration.Auto=true, startup already runs migration automatically. If your checks below are clean, you can skip manual Phase D and continue to Phase E/F:

1. netdebug persistence status shows enabled=true, healthy state, and no startupMigrationFailure.
2. Migration state rows are completed:

   SELECT domain, state, started_at, completed_at
   FROM nm_migration_state
   ORDER BY domain;
   

3. netdebug persistence verify scope=all mode=deep reports parity verified.

1. Dry run

netdebug persistence migrate scope=all dryrun=true force=false

Check counters like:

1. documents
2. roleDefinitions
3. roleGrants
4. wrongTypeSkips
5. decodeFailures

If dry run fails, do not run real migration yet.

Phase D.1: Migration dry run output.

2. Real migration

netdebug persistence migrate scope=all dryrun=false force=false

If Database.Sql.Migration.RedisWinsUnlessEmpty=true, SQL rows for selected scopes are cleared first, then rebuilt from Redis. This is expected.

Phase D.2: Real migration output.

---

Phase E: Verify Redis vs SQL Parity

Run:

netdebug persistence verify scope=all mode=quick
netdebug persistence verify scope=all mode=deep

How to decode this:

1. Parity verified means counts/signatures line up.
2. Parity mismatch means data differs somewhere in selected scopes.
3. deepDecodeFailures should stay 0.

If quick passes but deep fails, collect the full counter line and inspect the specific *.deep.match=0 domains before proceeding.

Phase E: Parity verification output.

---

Phase F: Operator QA Checklist

1. Runtime health check

netdebug persistence status

Expected:

1. health=OK (or short-lived MIGRATING).
2. dlqCount=0.
3. Startup failure fields blank.
4. Namespace counters stable.

Phase F.1: Runtime health status output.

2. DBeaver table presence check

Go to:

Databases -> <your_db> -> Schemas -> public -> Tables

Confirm:

• nm_documents
• nm_role_definitions
• nm_role_grants
• nm_mirror_applied_events
• nm_migration_state
• nm_cache_projection_outbox

3. DBeaver data spot check

SELECT namespace, COUNT(*) AS rows
FROM public.nm_documents
WHERE deleted = false
GROUP BY namespace
ORDER BY namespace;

SELECT COUNT(*) AS role_definitions
FROM public.nm_role_definitions
WHERE deleted = false;

SELECT COUNT(*) AS role_grants
FROM public.nm_role_grants
WHERE deleted = false;

SELECT COUNT(*) AS outbox_pending
FROM public.nm_cache_projection_outbox
WHERE processed_at = 0;

How to decode these:

1. Active namespaces should show non-zero rows where you expect data.
2. Role counts should be non-zero if you already had role data.
3. outbox_pending should usually be 0 (or low and draining).

---

Optional Phase G: SQL-Authoritative Cutover (Advanced)

Most networks can stop at Redis + SQL mirror mode.

Only use this if you intentionally want SQL as runtime authority for selected custom document namespaces.

Prereqs:

1. Database.Sql.Cutover.PromotionEnabled=true
2. Database.Sql.AuthoritativeWritesEnabled=true
3. Database.Sql.Authoritative.AuthoritativeReadsEnabled=true
4. Database.Sql.Authoritative.RequireReadyMarker=true

Dry run:

netdebug persistence cutover run scope=all dryrun=true force=false

Real run:

netdebug persistence cutover run scope=all dryrun=false force=false

Status:

netdebug persistence cutover status

Warning

System-managed namespaces are not normal SQL-authoritative cutover targets. Use this for custom document namespaces unless you explicitly know otherwise.

---

Rollback

If migration is unstable:

1. Set Database.PersistenceMode back to REDIS_ONLY.
2. Restart all nodes.
3. Keep Redis as source of truth.
4. Keep SQL data for backup/forensics.

---

Troubleshooting

SQL durability mirror is disabled

Check:

1. Database.PersistenceMode=REDIS_PLUS_SQL
2. Database.Sql.MirrorEnabled=true
3. Database.Sql.JdbcUrl is reachable
4. Username/password has no accidental whitespace

Cutover preflight blocked due to unregistered Redis namespaces

Fix:

1. Run namespace discovery.
2. Register missing custom namespaces.
3. Re-run preflight.

health=OUT_OF_SYNC

Check:

1. pending, unreadLag, retryQueue, processingQueue
2. dlqCount
3. SQL availability

decodeFailures in migrate/verify

Meaning: one or more Redis payloads are malformed or unreadable.

Fix:

1. Identify affected namespace from counters/logs.
2. Repair or remove broken records.
3. Re-run migrate/verify.