RHACS Enterprise Deployment - Stop Fucking Around With Security At Scale

Here's the thing about RHACS architecture - get it wrong early and you'll spend the next two years fixing it while security violations pile up and executives ask why their fancy security platform can't tell them if they're actually secure.

I've seen teams waste six months trying to retrofit their architecture because they didn't think through multi-cluster networking. Don't be those teams. At my last job, we had to rebuild everything because someone decided to ignore the networking design and just "figure it out later." Spoiler: later sucked.

Hub-and-Spoke vs. Federated Central Models

RHACS uses a distributed architecture where Central services manage multiple secured clusters through Sensor agents. For enterprise deployments, you have two primary architectural approaches that will make or break your deployment:

Single Central Hub (Works Until It Doesn't)

• One Central trying to babysit all your clusters
• Looks simple until Central dies at 3am and nobody can deploy anything
• You need serious hardware: 16+ cores, 32+ GB RAM, 1TB+ storage (budget 2x what Red Hat's sizing guide says because it's always wrong)
• Every cluster needs to phone home on port 443 (good luck with your corporate firewall team)
• Perfect for teams that like single points of failure and 3am pages

Regional Central Federation (The Adult Option)

• Multiple Central instances that don't all die simultaneously
• Each one handles 50-150 clusters before choking (tested this the hard way with 200+ clusters)
• Actually works when your European data center loses internet connectivity
• More shit to manage but you won't get fired when one region implodes
• Mandatory if you have air-gapped clusters or compliance people who actually read frameworks

Central Placement Strategy

Dedicated Security Cluster (Do This)

• Put Central on its own cluster so app deployments can't kill your security monitoring
• When devs break production, your security tools still work
• Easier to explain to auditors why your security platform is actually secure
• Size it properly: 3 nodes minimum, 16 vCPU/32GB RAM each (or watch it die under load like mine did last month)
• Storage will eat your budget: 2TB+ for Central DB, 1TB+ for Scanner (and growing fast - we're at 3TB now)

Shared Management Cluster (The Cheap Option)

• Cram RHACS onto the same cluster as RHACM to save money
• Works fine until both tools fight for resources during a security incident
• Perfect for when your CFO cares more about costs than uptime
• Requires constant babysitting and resource tuning

Network Architecture (AKA Firewall Hell)

Here's where your network team will hate you. RHACS needs specific ports open and your enterprise firewall rules probably block half of them.

Central to Secured Clusters (The Fun Part):

• Port 443: Sensors phone home constantly (prepare for "why is there so much traffic?" questions)
• Port 8443: API access for roxctl and CI/CD (don't forget to document this or your automation will break)

Within Central Cluster (The Easy Part):

• PostgreSQL: Keep internal (obviously - exposing your security database to the internet is a resume-generating event)
• Scanner: Keep internal (unless you want vulnerability data leaking to places it shouldn't)
• Central UI: External access required (good luck with your load balancer configuration and certificate bullshit)

Air-Gapped Deployments (Maximum Pain Mode):

• Scanner needs to sync vulnerability databases offline (50-100GB of fun)
• Internal CA certificates that expire at the worst possible moment (usually during vacation)
• Scanner V4 database mirroring will eat storage like crazy - I've seen it go from 50GB to 200GB in a month
• Plan for certificate hell and prepare to become best friends with your security team

High Availability Design

Central High Availability:

• Central StatefulSet with persistent storage (not yet clustered)
• Database backup strategy: PostgreSQL dumps every 6 hours minimum
• Recovery time objective: Target 1-4 hours with proper backup/restore procedures
• Single point of failure: Central DB cannot be clustered yet

Sensor High Availability:

• Sensors automatically reconnect to Central after network outages
• Policy cache enables limited offline operation (24-48 hours)
• Multiple Sensor replicas for large clusters (1000+ nodes)
• Node affinity to spread Sensors across availability zones

Scanner Architecture at Scale

Scanner V4 vs StackRox Scanner:
Look, Scanner V4 is finally stable enough for production (took them long enough). It's way better than the old scanner, though it'll still peg your CPU when scanning those 2GB images that developers somehow think are reasonable. SBOM generation is required for compliance frameworks - auditors fucking love this feature. Language-specific vulnerability detection covers Go, Java, Node.js, Python, Ruby - basically everything your devs are throwing at production these days.

Delegated Scanning Strategy:

• Enable Scanner on secured clusters for registry-local images
• Central Scanner for shared/external registries
• Reduces network traffic and scanning latency
• Each secured cluster needs 2-4 CPU cores, 4-8GB RAM for Scanner

Registry Integration Patterns:

• Quay integration: Webhook-based scanning
• Harbor, Artifactory: API-based integration
• Air-gapped registries: Manual certificate and credential management

This architectural foundation determines your operational model, scalability limits, and disaster recovery capabilities. With your architecture decided, the next critical question becomes: how much hardware do you actually need to support your deployment scale? The sizing decisions you make now will directly impact your monthly cloud bills and operational complexity.

When you're running RHACS at scale, shit breaks in ways you never imagined. Here's how to keep it running when developers are pushing broken images at 2am and compliance scans decide to eat all your CPU during peak traffic.

Monitoring and Alerting (Or How to Sleep at Night)

Shit You Must Monitor or Get Fired:

• Central health (because when it's down, everything's down)
• Sensor connectivity (offline sensors = blind spots)
• Scanner queue depth (when it hits 500+ images, you're screwed)
• Database performance (PostgreSQL doesn't scale infinitely)
• Policy violation flood warnings (cryptominers cause alert storms)

Prometheus Metrics Integration:
RHACS exposes metrics that integrate with OpenShift monitoring, standalone Prometheus, and Grafana dashboards. Here's what you actually need to track:

## Key RHACS metrics for alerting
- stackrox_central_db_connections
- stackrox_scanner_queue_length  
- stackrox_sensor_last_contact_time
- stackrox_policy_violations_total
- stackrox_compliance_scan_duration

Grafana Dashboard Requirements:

• Central cluster resource utilization (CPU, memory, storage)
• Multi-cluster Sensor status overview
• Scanner performance metrics and image processing rates
• Security posture trends and policy violation patterns
• System health scorecard with SLA metrics

Backup and Disaster Recovery

Central Database Backup Strategy:
Backup your Central database or lose everything. Here's the simple approach that actually works:

## Automated PostgreSQL backup every 6 hours
kubectl exec -n stackrox central-db-0 -- pg_dump -U postgres stackrox > backup-$(date +%Y%m%d-%H%M).sql

Recovery Procedures:

1. Central Cluster Recovery:

   • Restore Central DB from latest backup
   • Verify Sensor reconnection (automatic within 5 minutes)
   • Validate policy synchronization

2. Secured Cluster Recovery:

   • Sensors operate with cached policies for 24-48 hours
   • Reinstall Sensor components if cluster rebuilt
   • Historical data may be lost but new monitoring begins immediately

Cross-Region Backup:

• Database backups stored in multiple availability zones
• Configuration backup includes policies, integrations, RBAC settings
• Recovery time objective: 2-4 hours for full Central restoration

Upgrade and Maintenance Procedures

Rolling Upgrade Strategy (Recommended):

1. Upgrade Central first - maintains backward compatibility with older Sensors
2. Upgrade Sensors by region/environment - development → staging → production
3. Validate functionality at each stage before proceeding

Version Compatibility Matrix:

• Central supports Sensors up to 2 minor versions behind
• Scanner V4 requires Central and Sensor version alignment
• Policy compatibility maintained across minor version upgrades

Maintenance Windows:

• Central upgrades: 30-60 minutes planned downtime
• Sensor upgrades: Zero-downtime rolling updates
• Scanner database updates: 10-30 minutes (during upgrade)

Common Operational Challenges and Solutions

Challenge 1: Scanner Performance Bottlenecks
Symptoms: Long image scan queues, delayed CI/CD pipelines
Solutions:

• Enable delegated scanning on high-volume clusters
• Implement Scanner cache optimization
• Scale Scanner V4 horizontally (multiple replicas)
• Use dedicated high-IOPS storage for Scanner databases

Challenge 2: Policy Alert Fatigue
Symptoms: Thousands of policy violations, ignored alerts
Solutions:

• Start with "inform" mode policies, gradually enable enforcement
• Create environment-specific policy sets
• Use policy exceptions for legitimate business requirements
• Implement alert prioritization based on risk scores

Challenge 3: Central Database Growth (AKA The Storage Bill from Hell)
Symptoms: AWS storage bills that make your CFO cry, query timeouts during compliance scans

Real story: Our DB ballooned to like 500GB overnight, maybe more, and AWS started screaming at us about costs. Took us three fucking days to figure out why compliance scans were timing out - turns out nobody configured data retention and we had like 18 months of violation data just sitting there eating storage like a hungry hippo. The whole time executives are asking "why can't we see our security dashboard" while we're trying to vacuum a half-terabyte postgres database.

Solutions:

• Configure data retention policies (90 days default, not 365 - learned this the expensive way)
• Archive historical violation data before it bankrupts you
• Monitor database vacuum jobs or PostgreSQL will shit the bed
• Plan storage capacity - growth is exponential and will surprise you

Challenge 4: Network Connectivity Issues
Symptoms: Sensors appearing offline, inconsistent policy enforcement
Solutions:

• Implement network monitoring between clusters and Central
• Configure proxy settings for air-gapped environments
• Validate firewall rules for required ports (443, 8443)
• Set up automated Sensor restart procedures for network outages

Performance Optimization

Central Performance Tuning:

• PostgreSQL configuration optimization for RHACS workload
• JVM heap sizing for Central based on cluster count
• Load balancer configuration for high availability

Scanner Optimization:

• Image layer caching strategy to reduce registry load
• Registry mirror placement for geographic distribution
• Scanner database indexing optimization

Network Optimization:

• Sensor communication batching and compression
• Policy update distribution optimization
• Event aggregation to reduce Central load

These operational practices ensure RHACS maintains security effectiveness as your Kubernetes environment scales. But even with solid operations, you'll face specific deployment challenges that every enterprise team encounters. Let's address the most common questions and gotchas that come up during real-world RHACS implementations.

Your security platform has the keys to everything. If someone compromises RHACS Central, they can see every vulnerability, every policy violation, and every cluster configuration. Don't be the team that gets breached through their own security tool - I've seen this happen and the post-mortem is fucking embarrassing.

Here's how to not fuck this up:

Central Cluster Security Hardening

Network Segmentation (Build a Fucking Moat):
Lock down Central cluster networking like it contains nuclear launch codes:

## Ports you MUST have open (and nothing else)
- Port 443: Sensors, UI, API (unavoidable)
- Port 8443: roxctl/CI/CD access (document this well)
- Port 5432: PostgreSQL (internal only - expose this and die)
- Port 22: SSH (bastion host only - no direct access)

Pod Security Standards:
Apply restricted Pod Security Standards to the RHACS namespace following Kubernetes security best practices:

apiVersion: v1
kind: Namespace
metadata:
  name: stackrox
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

Resource Quotas and Limits:
Prevent resource exhaustion attacks with proper limits:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: rhacs-quota
  namespace: stackrox
spec:
  hard:
    limits.cpu: "64"
    limits.memory: "128Gi"
    persistentvolumeclaims: "10"
    requests.storage: "5Ti"

Certificate Management and TLS

Custom CA Integration:
For air-gapped environments, configure RHACS to trust internal certificate authorities:

## Add custom CA to Central
kubectl create configmap custom-ca --from-file=ca.crt=internal-ca.crt -n stackrox
kubectl patch deployment central -n stackrox -p '{"spec":{"template":{"spec":{"volumes":[{"name":"custom-ca","configMap":{"name":"custom-ca"}}]}}}}'

Certificate Rotation:
Set up automated certificate rotation or prepare for 3am certificate expiry emergencies:

• Central TLS certificates: 90-day rotation cycle (or whenever they expire and break everything)
• Sensor client certificates: Central handles this automatically (one less thing to break)
• Scanner TLS certificates: Sync with Central rotation or suffer
• Database TLS certificates: Annual rotation is fine (unless your security team has opinions)

Identity and Access Management

RBAC Integration:
Hook RHACS into your corporate identity nightmare (LDAP, AD, OAuth - pick your poison). Follow standard RBAC practices for secure authentication, assuming your enterprise identity system actually works:

## Example RHACS role for security engineers
apiVersion: platform.stackrox.io/v1alpha1
kind: Role
metadata:
  name: security-engineer
rules:
- resources: ["Alert", "Policy", "Compliance"]
  verbs: ["read", "write"]
- resources: ["Cluster", "Deployment"]
  verbs: ["read"]

Service Account Security:
Follow principle of least privilege for all service accounts:

• Central service account: Minimal cluster-admin permissions
• Sensor service accounts: Namespace-scoped read/write only
• Scanner service accounts: Image pull and storage access only

API Access Controls:
Secure RHACS API access for automation and integrations:

• Use API tokens with limited scope and expiration
• Implement API rate limiting to prevent abuse
• Log all API access for security auditing
• Rotate API tokens every 90 days

Monitoring and Incident Response

Security Event Monitoring:
Configure comprehensive logging and monitoring for RHACS itself using OpenShift logging and enterprise SIEM integration:

## Forward RHACS logs to central logging
apiVersion: logging.coreos.com/v1
kind: ClusterLogForwarder
metadata:
  name: rhacs-logs
spec:
  outputs:
  - name: rhacs-siem
    type: splunk
    url: https://splunk.company.com:8088
  pipelines:
  - name: rhacs-security-logs
    inputRefs:
    - application
    filterRefs:
    - rhacs-filter
    outputRefs:
    - rhacs-siem

Incident Response Integration:
Integrate RHACS with enterprise incident response processes following standard frameworks:

• Configure high-severity policy violations to create ServiceNow or Jira tickets automatically
• Set up on-call escalation using PagerDuty or Opsgenie for Central/Scanner service failures
• Define runbooks for common operational scenarios
• Test disaster recovery procedures quarterly using chaos engineering principles

Compliance and Governance

Audit Trail Requirements:
Maintain comprehensive audit trails for compliance:

• All policy changes and approvals
• Access control modifications
• System configuration changes
• Security violation acknowledgments and exceptions

Policy Governance:
Establish formal policy management processes:

• Security team approval required for policy changes
• Change management process for production policy updates
• Regular policy review and tuning cycles
• Documentation of business justifications for policy exceptions

Data Retention and Privacy:
Configure appropriate data retention for compliance requirements:

## Configure retention in Central
apiVersion: v1
kind: ConfigMap
metadata:
  name: central-config
  namespace: stackrox
data:
  retention.yaml: |
    alertRetentionDays: 30
    imageRetentionDays: 7
    auditLogRetentionDays: 90

Performance and Reliability

High Availability Configuration:
While Central itself is not clustered, implement reliability practices:

• Database replication for read replicas (if needed for reporting)
• Automated backup testing and validation
• Health check endpoints for load balancer integration
• Graceful degradation during maintenance windows

Capacity Planning:
Monitor and plan for growth patterns:

• Track cluster onboarding rates and resource impact
• Monitor policy violation trends and storage growth
• Plan scanner capacity for peak scanning periods
• Establish metrics-based alerting for resource exhaustion

Disaster Recovery Testing:
Regularly test disaster recovery procedures:

• Complete Central cluster rebuild from backups
• Network partition scenarios between Central and Sensors
• Database corruption and recovery procedures
• Cross-region failover for federated deployments

Integration Security

CI/CD Pipeline Security:
Secure RHACS integrations with development toolchains following DevSecOps best practices:

• Use dedicated service accounts with limited permissions per principle of least privilege
• Implement break-glass procedures for emergency deployments
• Monitor scanner API usage with Prometheus alerting rules for anomalous patterns
• Regular security review of pipeline integrations using OWASP DevSecOps guidelines

This comprehensive security foundation ensures RHACS itself remains protected while providing security oversight for your Kubernetes infrastructure. You now have the complete picture: architecture decisions, resource sizing, operational procedures, and security hardening. The final piece is knowing where to go for additional resources and support as you implement these enterprise deployment practices.