Data Residency & Retention

The Sankofa Engine uses a multi-tier storage architecture to balance performance, cost, and regulatory compliance. This page documents where data resides, how long it is retained, and how it moves between tiers.

Storage Tiers

Tier Descriptions

Hot Tier (ScyllaDB 6.2) The hot tier stores the active transaction ledger and account state. ScyllaDB provides low-latency reads and writes for real-time transaction processing. Data remains in the hot tier for the configured retention period (default 2 years) before becoming eligible for archival.

Projection Tier (PostgreSQL 16) The projection tier maintains CQRS read models — materialized views of account balances and other derived state. This tier always reflects the current state and is continuously updated as transactions are processed. It does not store historical transactions; it stores the computed result of applying all transactions.

Cold Tier (S3 / Local Filesystem) The cold tier stores archived transactions that have aged out of the hot tier. Cold storage provides cost-effective long-term retention. The storage backend is configurable — S3 for cloud deployments, local filesystem for on-premises deployments.

Event Log (NATS JetStream) The event log retains all transaction events and signed receipts for 7 years by default. It serves as the system of record for event replay, audit reconstruction, and disaster recovery.

Data Classification

The Sankofa Engine classifies data to apply appropriate encryption keys and retention policies:

Data Flow Between Tiers

                    ┌─────────────┐
  Incoming Txn ────▶│  NATS       │  Event published (retained 7 years)
                    │  JetStream  │
                    └──────┬──────┘
                           │
                    ┌──────▼──────┐
                    │  Shard      │  Transaction processed
                    │  Worker     │
                    └──┬─────┬───┘
                       │     │
              ┌────────▼┐  ┌─▼──────────┐
              │ ScyllaDB │  │ PostgreSQL │
              │ (Hot)    │  │(Projection)│
              └────┬─────┘  └────────────┘
                   │
                   │  After retention period
                   │
              ┌────▼─────┐
              │ S3 / FS  │
              │ (Cold)   │
              └──────────┘

Retention Policies

Default Retention Periods

Customizing Retention

Retention periods are configured at deployment time. Customers can adjust retention to meet their specific regulatory requirements:

• Banking regulations may require 5-7 year retention of financial transaction records.
• Anti-money laundering (AML) requirements may mandate specific retention windows for transaction data.
• Tax compliance may require retention of financial records for a defined period after the tax year closes.
• Data minimization regulations (e.g., GDPR) may require shorter retention for certain data categories.

Retention policies are evaluated nightly by the Archival Service.

Archival Process

Archival Service

The Archival Service runs as a Kubernetes CronJob on a nightly schedule:

Cross-Tier Queries

Queries seamlessly bridge hot and cold tiers:

• The query engine first checks the hot tier (ScyllaDB) for matching records.
• If the query time range extends beyond the hot tier retention period, the engine follows archival root references to retrieve data from cold storage.
• Results from both tiers are merged and returned as a unified response.
• The caller does not need to know which tier the data resides in.

Data Deletion

Purge Capabilities

The Sankofa Engine supports data deletion for regulatory compliance:

Cryptographic Erasure

Cryptographic erasure is the fastest and most complete method of data disposal:

1. The KMS destroys the DEK for the target data classification or account.
2. All data encrypted with that DEK becomes immediately and permanently unrecoverable.
3. The ciphertext may remain in storage but is indistinguishable from random data without the key.
4. This approach satisfies data disposal requirements without requiring physical media destruction or individual record deletion.

Retention Policy Enforcement

The Archival Service enforces retention policies automatically:

• Transactions exceeding the hot tier retention period are archived to cold storage.
• Cold storage data exceeding the cold tier retention period (if configured) is deleted.
• Event log messages exceeding the NATS JetStream retention period are automatically purged by NATS.
• All deletion events are logged for audit purposes.

Geographic Considerations

Deployment-Dependent Data Residency

Data residency in the Sankofa Engine is determined by the deployment configuration, not by the application code:

Customer-Configurable Residency

Customers can specify data residency requirements during deployment:

• Region selection: Deploy the entire stack in a specific geographic region (e.g., US-East, EU-West, AP-Southeast).
• Node affinity: Kubernetes node affinity rules ensure pods run only on nodes in the target geography.
• Storage class selection: Kubernetes storage classes can be configured to use region-specific storage backends.
• Cross-region restrictions: Network policies can prevent data from leaving the designated region.

Regulatory Alignment

Sankofa Labs works with customers to configure deployments that meet their specific data residency obligations.