Falco - Linux Security Monitoring That Actually Works

Look, I've been burned by security tools before. Spent 6 months with Twistlock that burned through 15% CPU and alerted every time someone ran ps aux. Falco is different - been running it in production for 2+ years and the worst performance hit I've seen is 7% CPU on a database node running like shit anyway. The CNCF graduation status (February 2024) gave our compliance team confidence, plus you can actually audit what it's doing since it's open source.

What Falco Actually Does

Falco sits on your Linux systems watching syscalls through eBPF (or kernel modules if you're stuck on older kernels). When someone tries to escape a container, escalate privileges, or run a reverse shell, Falco catches it in real-time and sends you an alert that actually means something. The modern eBPF driver uses CO-RE which means it works across kernel versions without recompilation.

The key difference from other tools: Falco knows the difference between your application doing legitimate work and someone trying to pwn your system. I've seen it catch everything from cryptominers to privilege escalation attempts that other tools missed completely. Check out the detection capabilities - it's not just another log parser.

Real Production Experience

CPU overhead: Despite the bullshit marketing claims, I've measured 1-3% CPU usage on busy nodes. Version 0.38.x had memory leaks with high syscall volumes - make sure you're on 0.39+ if you're monitoring chatty applications. The current stable version as of September 2025 is 0.41.x, which includes significant performance improvements and better eBPF probe reliability. The performance impact varies significantly based on your syscall volume.

Memory usage: Starts around 50MB but scales with your rule complexity. I've seen it climb to 200MB+ on nodes with aggressive custom rules and verbose logging enabled. Monitor the built-in metrics if you're running tight on resources - Falco exposes Prometheus endpoints starting from 0.38.

Event volume: Can handle thousands of events per second, but your SIEM integration will be the bottleneck. Learned this when our intern deployed a crypto miner and Falco sent 47,000 alerts in 8 minutes, completely destroying our Splunk cluster. That was a fun Monday morning. Check the event dropping documentation if you're seeing ratelimit errors.

The Three Driver Options (And Which to Use)

Modern eBPF (recommended): Uses CO-RE technology so it works across kernel versions without recompilation. Requires kernel 5.8+ with BTF support enabled. This is what you want unless you have a specific reason not to use it. Default since Falco 0.38.0, with significant stability improvements in 0.40+.

Classic eBPF: Still requires kernel headers but more compatible than Modern eBPF. Use this if Modern eBPF doesn't work on your distro. The libs repository has performance comparisons between drivers.

Kernel Module: Maximum compatibility but requires root and kernel headers. Only use this if eBPF completely fails on your system. Check the host installation docs for kernel module setup.

Pro tip: The Modern eBPF driver fails to load on RHEL 7.6 nodes constantly - saw bpf_map_create failed: Operation not permitted errors for 3 days before realizing our kernel was compiled without BTF support. Always have kernel headers installed as a fallback, and check the troubleshooting guide when you see those cryptic eBPF errors.

Integration Hell (And How to Avoid It)

Falco has 50+ output integrations through Falcosidekick but most are community plugins with varying quality. Here's what actually works in production:

Slack/Teams: Works great for initial setup, becomes noise after a week
Elasticsearch: Solid if you already have ELK, pain in the ass to set up from scratch - see the Elastic integration guide
S3: Cheap storage for compliance logging
Webhook: Most flexible - build your own integration using the webhook documentation

Don't try to send every alert to your SIEM initially. Start with critical alerts only or you'll get buried in false positives. The rule adoption guide has good strategies for tuning.

Kubernetes Deployment Reality

Speaking of getting things working properly, let's talk about the actual deployment process.

The Kubernetes operator is still in tech preview (as of 0.41.0) and I've seen it crash on YAML edge cases. Stick with the official Helm charts unless you like debugging operator logs at 3am.

## This actually works:
helm repo add falcosecurity https://falcosecurity.github.io/charts
helm install falco falcosecurity/falco --namespace falco-system --create-namespace

The DaemonSet approach ensures every node gets monitored, but watch out for kernel header issues on mixed node types. Auto-scaling can break if new node AMIs don't have headers pre-installed. Check the EKS deployment guide for AWS-specific gotchas, or the falcoctl documentation for artifact management.

But getting Falco installed is just the beginning. The real challenge starts when you try to deploy it in production and discover all the ways it can break. Let's dive into the painful reality of actually making this thing work reliably.

Now that you understand what Falco is and why it's useful, let's get into the real shit - actually deploying this thing in production without wanting to throw your laptop out the window.

Rule Tuning: Where Most People Give Up

The default Falco rules will flood you with false positives. Took me 3 weeks to tune them for our environment. The rules repository contains the source code if you need to understand what's triggering. Here's what actually works:

Start with these priorities only:

• Terminal shell in container - catches obvious container escapes
• Write below binary dir - detects people modifying system binaries
• Create files below dev - spots privilege escalation attempts

Don't enable everything at once or you'll get 500 alerts about package managers doing normal shit.

Common Gotchas That Will Bite You

Kernel Headers Hell: Modern eBPF sounds great but I've seen it fail on:

• RHEL/CentOS with custom kernels - check the host installation guide
• Ubuntu 18.04 with older headers - see kernel compatibility notes
• EKS nodes without linux-headers packages - AWS has specific instructions

Keep this handy for when eBPF fails:

## Check if headers exist
ls /lib/modules/$(uname -r)/build

## If missing, Falco falls back to kernel module
## which requires: apt-get install linux-headers-$(uname -r)

The startup troubleshooting guide covers these scenarios in detail.

Memory Leaks You'll Hit: Version 0.38.2 has a memory leak that will fuck your day up. Watched our database node's Falco pod climb from 80MB to 1.2GB over 6 hours before OOMKilling itself and taking down alerting for the entire cluster. Happened twice before I realized what was going on. Upgrade to 0.39+ or you'll be debugging this at 3am like I was. Monitor with the built-in Prometheus metrics or set aggressive memory limits.

Container Runtime Issues: Falco integration breaks differently on each runtime:

• Docker: Works fine, just make sure socket is accessible
• containerd: Needs CRI socket configuration (--cri /run/containerd/containerd.sock)
• CRI-O: Pain in the ass - requires proper SELinux contexts on RHEL

Performance Reality Check

Once you get past the initial setup hurdles, you'll want to know what this thing actually costs you in terms of resources.

Don't believe the "2% CPU overhead" marketing bullshit. Here's real numbers from our 200-node cluster - similar to what the performance documentation shows:

CPU usage scales with syscall volume:

• Idle nodes: 0.5-1% CPU
• Application nodes: 1-3% CPU
• Database/high-I/O nodes: 3-7% CPU

Memory grows with rule complexity:

• Default rules: ~50MB
• Custom rules + verbose logging: 150-200MB
• Poorly written rules with regex: 500MB+ (I learned this the hard way)

Use the Grafana dashboard to track actual resource usage and the performance tuning guide for optimization.

Monitor these metrics or you'll get surprised:

kubectl top pod -n falco-system
## Watch for OOMKilled pods - your rules are probably too aggressive

Integration War Stories

Elasticsearch: Works fine until you get owned. During a cryptomining incident, Falco sent 47,853 alerts in 12 minutes and completely destroyed our 3-node ES cluster. Had to restore from backup and explain to management why our logging was down for 4 hours. Rate limiting is not optional.

Slack Integration: Started useful, became the most annoying thing on the planet within 72 hours. Our #alerts channel got 2,400 messages in one day about package managers doing normal shit. Only send CRITICAL and ERROR priority alerts to Slack or your team will mute the channel forever.

S3 Storage: Cheap compliance logging that isn't actually cheap. Hit $347/month in S3 costs before I realized we were storing 600GB of debug logs. Implement log rotation immediately or prepare to explain your AWS bill.

Cloud Deployment Gotchas

AWS EKS: Works fine but Graviton ARM nodes need different container images. Mixed architecture clusters will break your DaemonSet. Check the EKS-specific deployment guide for Bottlerocket compatibility.

GKE: Google's hardened nodes block some eBPF features. Use kernel module driver or switch to standard nodes if you need full functionality. The GKE setup tutorial covers network policy issues.

Azure AKS: SELinux policies can interfere with Falco. Disable SELinux on AKS node pools or you'll get permission errors. The container runtime documentation has AKS-specific notes.

The Plugin Ecosystem Reality

Falco's plugin ecosystem is hit-or-miss. The official plugin repository has varying quality:

Actually work in prod:

• CloudTrail plugin: Solid for AWS API monitoring
• Kubernetes audit logs: Essential if you're monitoring K8s API abuse

Buggy/experimental:

• Okta plugin: Crashes on malformed events
• GitHub plugin: Rate limiting issues with large orgs

Build your own: The plugin SDK documentation is decent if you know Go. We built a custom plugin for our internal API monitoring that's been rock solid for 18 months. Check the plugin development guide for getting started.

Compliance and Enterprise Use

Despite the rough edges, Falco handles compliance well once tuned. We use it for:

• SOC 2: Runtime monitoring and incident response
• PCI-DSS: File integrity monitoring and access controls
• GDPR: Data access auditing (with custom rules)

The CNCF graduation gives our compliance team confidence, plus having open source code means we can actually audit what it's doing. Check out the enterprise case studies and Incepto Medical's production deployment for real-world compliance examples.

After fighting through all these deployment challenges, you're probably wondering if you should just pay for a commercial alternative instead. The grass always looks greener when you're debugging kernel module issues at 3am. Let's look at how Falco stacks up against the competition.