Log Management Architecture and Retention for Compliance

Log management for compliance requires satisfying overlapping retention and integrity requirements across multiple regulatory frameworks. PCI DSS v4.0 Requirement 10.7 mandates retention of audit logs for at least 12 months, with at least 3 months immediately available for analysis. SOX Section 802 (and SEC Rule 17a-4 for broker-dealers) requires retention of audit and financial records for 7 years. HIPAA §164.312(b) requires audit logs to be retained for 6 years. GDPR does not specify log retention periods directly, but the accountability principle (Article 5(2)) and data subject rights (Article 15 access requests) imply logs must be retained long enough to demonstrate compliance processing. DORA Article 12 requires that ICT operations logs be retained for a period enabling detection and investigation of ICT-related incidents, with the EBA expecting minimum 2-year retention for transaction and security logs. NIS2 Article 21(d) requires logging measures as part of basic security hygiene.

The engineering architecture for compliant log management has four layers. First, collection: log sources must cover the full regulated asset population — network devices (syslog RFC 5424), operating systems (Windows Event Log via WEF, Linux via rsyslog/syslog-ng), applications (structured JSON logs via ECS or OCSF schema), cloud services (CloudTrail, Azure Monitor, GCP Cloud Logging), and databases (audit plugins: MySQL Enterprise Audit, Oracle Unified Auditing, PostgreSQL pgaudit). Second, transport: logs must be transmitted via encrypted channels (TLS 1.2+) to prevent interception and must use a reliable protocol (TCP syslog, Beats/Logstash, Fluentd) to minimize log loss. Third, storage: compliance log storage must be immutable — AWS S3 Object Lock (COMPLIANCE mode), Azure Blob immutable storage, or WORM (Write Once Read Many) storage — to prevent alteration or deletion within the retention window. Fourth, integrity: log records should carry cryptographic timestamps and hash chains to detect tampering, as required under eIDAS and some financial regulation audit trail standards.

Log volume management is a persistent engineering challenge: a mid-size financial institution may generate 100–500 GB of raw logs per day, making full retention at native resolution expensive. Tiered retention architectures address this: hot storage (SIEM-indexed, immediately queryable) for 90 days; warm storage (compressed object storage, queryable via Athena/BigQuery) for 12 months; cold storage (Glacier/Archive) for regulatory retention periods up to 7 years. Log reduction through filtering at collection agents must be documented and auditable — PCI DSS prohibits filtering that removes required event types (logins, access control changes, audit log access) before storage. A common compliance gap is the absence of log monitoring for the log management infrastructure itself: changes to logging configurations or log pipeline failures must themselves generate alerts and be logged.

We design log management architectures for regulated environments covering source coverage mapping, encrypted transport pipelines, immutable tiered storage with framework-specific retention configuration, log integrity with cryptographic timestamping, and documentation of log reduction filtering decisions. Our platforms produce automated coverage reports demonstrating that all required log sources are captured.

Compliance-Native Architecture Guide

Design principles and a structured checklist for building software that is compliant by default — not compliant by retrofit. Covers data architecture, access controls, audit trails, and vendor due diligence.