Pulumi Cloud for Platform Engineering - Build Self-Service Infrastructure at Scale

Platform engineering was supposed to solve the DevOps chaos. Instead of developers clicking around AWS consoles at 3 AM or writing one-off Terraform that works exactly once, platform teams would build internal developer platforms that actually let developers self-serve infrastructure.

The theory was beautiful. The practice has been a goddamn nightmare.

Turns out what I've been saying for two years is finally sinking in: most Backstage installations collect dust while engineers still SSH into production to fix shit. I've seen this personally - organizations blow 12-18 months building developer portals that collect dust while engineers still SSH into production to restart services. There's a massive "Backstage backlash" happening as teams realize they've been building expensive tech demos instead of platforms.

The Fundamental Problem: Portal-First Thinking

Most platform teams start by asking "what portal should we build?" This is fucking backwards. Portals are just the frontend. Building a platform is like building an application - you need both a frontend and a robust backend. Starting with the UI is like building a house by putting up the front door first.

I've watched teams spend 8 months building a gorgeous Backstage catalog that lists services nobody can actually deploy. The "Create New Service" button submits a Jira ticket to the ops team. That's not self-service - that's a $300K form.

The portal-first approach fails because:

Infrastructure Anarchy: Without standardized infrastructure building blocks, developers still click around cloud consoles or write ad-hoc scripts that break in production. The portal becomes a pretty wrapper around the same manual chaos.

Backend Logic in Frontend: Teams cram business logic into Backstage plugins or custom portal code, violating basic application architecture principles. I've debugged Backstage plugins at 2 AM - when everything is in the frontend, nothing works reliably. Ever try to figure out why a TypeScript scaffolding template is generating malformed YAML? It's a special kind of hell.

No Real Self-Service: True self-service requires programmable infrastructure primitives that can be composed into higher-level abstractions. Without this foundation, "self-service" actually means "please submit a ticket and wait 3 days."

Operations Nightmare: Platform teams become glorified ticket handlers, manually provisioning infrastructure through the portal. The ops team workload increases instead of decreasing - I've seen platform teams that handle way more tickets after building their "self-service" portal.

The Missing Foundation: Infrastructure-First Architecture

Successful platform engineering starts with the infrastructure layer, not the portal layer. You need standardized, reusable infrastructure components that can be consumed programmatically before you worry about user interfaces.

Most teams get this completely backwards.

The infrastructure-first approach means:

1. Standardized Building Blocks: Create reusable infrastructure components that encapsulate best practices, security policies, and operational requirements
2. Programmable APIs: Enable infrastructure consumption through code, not just UI clicks
3. Golden Paths: Provide opinionated templates that guide developers toward best practices while maintaining flexibility
4. Policy Enforcement: Embed compliance and security rules directly into the infrastructure components

Only after establishing this foundation should you add portal interfaces on top.

Platform Engineering Finally Gets Its Shit Together

The platform engineering space is finally maturing past the "let's install Backstage" phase. Here's what's actually happening:

Portal Backlash is Real: Teams are realizing that Backstage is not your platform. Portals are interfaces to platforms, not platforms themselves. I've watched three different companies spend a year customizing Backstage plugins just to discover they still can't actually deploy anything.

Console Access is Going Away: Developers are losing direct access to infrastructure. The days of unrestricted cloud console access are ending - partly because security finally got tired of explaining why the intern has Admin access to production S3 buckets.

Everyone Has to Win: Platform initiatives that only help developers while making ops teams' lives worse are failing. You can't just move the complexity around - you have to actually eliminate it.

Real Platform Engineering Success Stories

Organizations that get platform engineering right share common patterns:

BMW scaled their platform to support 11,000+ developers handling hundreds of thousands of builds daily using hybrid cloud infrastructure. They abstracted complex infrastructure into standardized, repeatable components instead of letting each team build their own solutions.

Unity reduced deployment time from weeks to hours - an 80% improvement - by implementing standardized infrastructure components that developers could self-serve. The key insight: they built reusable infrastructure libraries once instead of each team rolling their own solutions.

Mercedes-Benz eliminated most of their manual infrastructure operations by building reusable components that developers could self-serve through code. Note: they still have manual operations for the edge cases, but the 80% common use cases became automated.

The common thread: these organizations started with standardized infrastructure building blocks and added user interfaces later, not the other way around.

The Cost of Getting It Wrong

I've watched platform engineering initiatives burn money like a dumpster fire. Here's the real cost breakdown:

• Engineer Time: Senior engineers spending way too much time per week building DIY solutions instead of features. When you're paying senior engineer salaries for infrastructure firefighting, that adds up fast.
• Infrastructure Tickets: Development teams submit 50+ infrastructure tickets per month, each taking 2-4 hours to fulfill. Platform teams hire more ops engineers to handle the "self-service" workload.
• Security Incidents: Manual processes and inconsistent configurations. I've seen a company get breached because someone fat-fingered an S3 bucket policy in the "self-service" portal, making customer data public for 6 hours before anyone noticed.
• Cloud Waste: Unmanaged resources and oversized instances. Developers provision c5.4xlarge instances for development because the portal doesn't have guardrails, then forget they exist until the monthly AWS bill shows up.

Real talk: a typical failed platform engineering project burns through anywhere from half a mil to a couple million just in engineer salaries. That's not counting the business impact when features get delayed by months or that one security incident where someone fat-fingered an S3 bucket policy and exposed customer data. All for a platform that ends up being a glorified bookmark page.

The Infrastructure-First Alternative

Platform engineering done right eliminates these costs by focusing on what actually matters:

Build Once, Use Everywhere: Instead of 47 different ways to deploy a web app, you have one standardized component that works. Security policies and operational knowledge are baked in, not documented in a wiki that nobody reads.

Actual Self-Service: Developers can provision what they need without submitting tickets. The key word is "actual" - not just a fancy form that creates a ticket behind the scenes.

Policy as Code: Instead of hoping people follow the security guidelines, the infrastructure components enforce the rules. You can't create an internet-facing database by accident because the component won't let you.

Treat Infrastructure Like Software: Version control, testing, code review, CI/CD. Apply the same engineering rigor to infrastructure that you apply to applications.

This infrastructure-first approach is exactly what Pulumi IDP was designed to enable. Instead of starting with portals and working backward to infrastructure, you start with solid infrastructure foundations and build user experiences on top.

The result: platform engineering that actually works at scale, with developer adoption that justifies the investment.

Pulumi IDP launched May 6, 2025, and it's the first platform engineering tool that doesn't make me want to throw my laptop out the window. Instead of starting with developer portals and hoping infrastructure problems magically solve themselves, Pulumi IDP forces you to build actual infrastructure foundations first. This matters because every other platform tool I've used feels like putting lipstick on a pig.

The Five-Layer Platform Architecture That Actually Works

Pulumi IDP addresses all five critical layers of internal developer platforms, starting with the foundation that everyone else fucks up:

Layer 1: Resources - The foundation that most platforms get completely wrong

• 160+ cloud providers including AWS, Azure, GCP, Kubernetes. 160+ providers means it works with whatever weird shit your company runs.
• Modern architectures: containers, serverless, AI/ML workloads, data lakes. Translation: it works with the stuff you actually use, not just EC2 instances.
• Multi-cloud and hybrid cloud support. This actually matters when leadership changes their mind about cloud strategy every 6 months.

Layer 2: Security & Identity - Built into the foundation, not duck-taped on later

• Pulumi CrossGuard for policy-as-code with auto-remediation. No more hoping people read the security wiki.
• Pulumi ESC for secrets management with automatic rotation. Finally, secrets that don't live in plaintext YAML files.
• Fine-grained RBAC and audit logging. Your security team will actually like you for once.

Layer 3: Integration & Delivery - Infrastructure CI/CD that doesn't make you cry

• Pulumi Automation API embeds IaC directly in applications. Deploy infrastructure from your app code, not separate Jenkins jobs from 2018.
• Integration with existing CI/CD systems and GitOps workflows. Works with whatever CI/CD disaster you already have.
• Testing frameworks for infrastructure using standard programming languages. Test your infrastructure like you test your application code.

Layer 4: Monitoring & Logging - Operational visibility that actually helps at 3 AM

• Pulumi Insights for advanced search and analytics. Find that one misconfigured security group that's causing the outage.
• Cost optimization recommendations powered by AI. Stop paying $3000/month for that t3.micro instance someone misconfigured.
• Resource lifecycle management and drift detection. Know when someone manually changed something in the console.

Layer 5: Developer Control Plane - Multiple ways to fuck up your infrastructure

• No-code deployments through templates. For the product managers who insist they need to deploy things.
• Low-code with YAML for simple use cases. When you want infrastructure-as-code but not too much code.
• Full-code with TypeScript, Python, Go, C#, Java for complex scenarios. When you need real programming languages for real infrastructure.

The Private Registry: Your Single Source of Truth

The foundation of Pulumi IDP is the Pulumi Private Registry, which actually solves the discoverability and lifecycle management clusterfuck that destroys most platform initiatives.

Discoverability: Teams store infrastructure components in Git repos where they disappear into the void. I've spent hours hunting for "that Terraform module Sarah wrote 6 months ago" buried in some random repo. The Private Registry centralizes all standardized building blocks with automatic documentation generation and searchable metadata.

Lifecycle Management: Track usage across all components and templates. See which teams are using which versions, assess impact of changes, and identify version drift across environments. This matters when you need to update a security policy and don't want to break 47 different services.

Standardization: Publish once, consume everywhere. The same components work across all programming languages and consumption models. No more maintaining the same infrastructure component in TypeScript, Python, and Go.

With a single pulumi publish command, platform engineers make their standardized building blocks discoverable from a central location. Teams can explore README files, browse automatically generated API documentation, and understand installation and usage patterns.

Three Consumption Models: Meeting Developers Where They Are

Most platforms fail because they force developers into a single consumption model. Pulumi IDP recognizes that different teams have different needs:

No-Code Workflows: Non-technical users deploy infrastructure through point-and-click interfaces powered by organization templates. Templates are stored as infrastructure-as-code but consumed through web interfaces.

Low-Code Workflows: Developers compose infrastructure using standardized components in Pulumi YAML programs. Platform teams define the components, developers assemble them without writing complex infrastructure code.

Full-Code Workflows: Advanced teams scaffold infrastructure using templates from the CLI and extend with custom code in their preferred programming language.

The critical insight: the same infrastructure components power all three models. Platform teams don't maintain multiple systems - they build once and support multiple consumption patterns.

Pulumi Services: Organizational Context That Matters

Traditional infrastructure tools organize resources by technical boundaries (VPCs, databases, load balancers). Pulumi Services organize resources by business context - the way your organization actually thinks about applications and systems.

Services enable teams to:

• Logically group related stacks, environments, and resources across projects
• Add business metadata like observability dashboard links, Slack channels, and owner information
• Track dependencies between business services, not just technical resources
• Implement governance at the service level with appropriate access controls

A service represents what your organization actually cares about - the customer-facing API, the analytics pipeline, the mobile app backend - with all its associated infrastructure grouped together logically.

AI-Powered Operations: Pulumi Copilot Integration

Pulumi IDP includes deep integration with Pulumi Copilot, and I'll be honest - it's the first AI tool for infrastructure that doesn't make me want to disable it immediately:

Infrastructure Generation: Generate complete infrastructure programs from natural language descriptions. "Create a microservice with load balancer, auto-scaling group, and RDS backend" actually becomes working code. I tried this on some fucked up Node.js deployment that needed Postgres - finally got ECS working after fighting with the health checks for an hour. Turns out the path was wrong, as usual. Copilot at least pointed me in the right direction when the load balancer kept returning 502s.

Error Diagnosis: When deployments fail, Copilot analyzes errors in the context of YOUR infrastructure and provides actionable solutions. Not generic Stack Overflow answers - actual fixes for your specific clusterfuck.

Resource Discovery: "Show me all publicly accessible resources" or "Which services are running in us-west-2" with intelligent filtering and security analysis. This saved my ass during a security audit when I had to find every internet-facing resource across 12 AWS accounts.

Cost Optimization: Identify oversized resources, unused assets, and optimization opportunities with specific dollar impact calculations. Found this RDS instance burning through like $2000/month - turns out nobody was actually using it because some config was fucked and the app was silently failing to connect.

As of May 2025, Copilot is available directly in the CLI with pulumi ai commands, making it accessible in developer workflows, not just web interfaces.

Enterprise-Grade Platform Engineering

Pulumi IDP scales to enterprise requirements with features that solo-developer platforms lack:

Multi-Team Isolation: Organizations can have multiple platform teams managing different domains (networking, databases, security) with appropriate boundaries and governance.

Compliance Integration: SOC 2 Type II certification and built-in compliance policies for major frameworks (SOX, FedRAMP, GDPR).

Audit Everything: Every action logged with user attribution, from component publications to infrastructure deployments to policy violations.

High Availability: Self-hosted deployment options for organizations that require infrastructure platforms to run entirely within their environments.

Real Implementation: BMW's Platform Engineering Success

BMW's infrastructure modernization demonstrates Pulumi IDP principles at scale:

• 6 months saved by using standardized components instead of building custom solutions
• Multiple consumption models supporting both infrastructure engineers and application developers
• Cross-team collaboration between platform, security, and development teams using shared components
• Compliance automation through policy-as-code integrated into all infrastructure components

BMW's success came from starting with infrastructure standardization and building user experiences on top, not the other way around.

The Bottom-Up Platform Engineering Revolution

Pulumi IDP represents the maturation of platform engineering from "let's build a portal" to "let's build a platform." The bottom-up approach - starting with infrastructure foundations and building up to user experiences - is the key insight that separates successful platform initiatives from expensive failures.

The result: Platform engineering that scales with your organization, supports multiple team workflows, and actually gets adopted because it solves real infrastructure problems rather than just creating better-looking interfaces to the same underlying chaos.

This isn't theoretical anymore. As we head into 2025, the platform engineering landscape is consolidating around infrastructure-first approaches that deliver measurable results. The portal-first era is ending because organizations finally understand the difference between building interfaces and building platforms.

Moving from traditional infrastructure management to a mature platform engineering practice is like refactoring a legacy monolith - it seems impossible until you break it down systematically. I've helped dozens of teams make this transition, and here's the roadmap that actually works (and the gotchas that will bite you if you skip steps).

First, Figure Out What Mess You're Dealing With

Before building anything, understand what clusterfuck you're working with. I guarantee you have more infrastructure sprawl than anyone admits. This usually takes a month if you're lucky, longer if you have the kind of infrastructure archaeology I've seen - like that EC2 instance from 2019 that's still running Windows Server 2012 because "it just works" and nobody remembers what it does.

Infrastructure Audit - The "Oh Shit" Discovery Phase

• Import existing resources into Pulumi Cloud regardless of how they were provisioned. This includes the AWS resources that Dave created manually 3 years ago and never documented, plus that c5.24xlarge instance someone spun up for "testing" that's been running at $3,500/month (probably breaks with the new AWS SDK v3 but nobody wants to touch it).
• Use Pulumi Insights to discover shadow IT and unmanaged resources. You'll find running instances that nobody remembers creating, load balancers pointing to nothing, and S3 buckets with names like "temp-data-backup-delete-me-2022" that are still accumulating charges.
• Identify patterns in your current infrastructure that could become standardized components. Spoiler alert: you probably have 17 different ways to deploy a web application, and none of them have proper health checks.

Team Skills Assessment

• Evaluate current team programming language preferences (TypeScript, Python, Go, C#, Java)
• Assess comfort level with infrastructure-as-code concepts
• Identify champions who can drive adoption across teams

Business Context Mapping

• Define your organization's key services and applications (these become Pulumi Services)
• Map current infrastructure to business services, not technical boundaries
• Understand compliance requirements and security policies that need automation

Success Metrics Definition

• Time to provision development environments
• Number of manual infrastructure tickets per month
• Security policy violations and remediation time
• Developer satisfaction with infrastructure workflows

Next, Standardize the Stuff You Actually Use

Start with your most common infrastructure patterns. Don't try to boil the ocean - focus on the 20% of use cases that represent 80% of your infrastructure requests. I've seen teams try to standardize everything at once and burn out after 6 months with nothing to show for it. This part usually takes 3-4 months if you don't get distracted by every edge case.

Real talk: you'll probably fuck this up the first time. I watched one team spend 4 months building a "universal web app component" that couldn't handle their Django app because the health check endpoint was different. Started over with three simple components instead - Node.js, Python, and Go apps - and had working deployments in two weeks.

Identify Golden Path Patterns

• Web applications with load balancers and auto-scaling
• Microservices with container orchestration
• Data pipelines with storage and processing components
• Database deployments with backup and monitoring

Build Reusable Components

• Create Pulumi Components that encapsulate best practices
• Embed security policies using CrossGuard rules
• Include monitoring and observability by default
• Write good documentation and examples

Publish to Private Registry

• Use pulumi package publish to make components discoverable
• Include rich README files and API documentation
• Tag components with metadata for easy filtering and search
• Version components properly to support lifecycle management

Example: Web Application Component

import * as pulumi from \"@pulumi/pulumi\";
import * as aws from \"@pulumi/aws\";

export interface WebAppArgs {
    imageUrl: pulumi.Input<string>;
    desiredCount?: pulumi.Input<number>; // TODO: add proper validation when I have time (Dave's going to hate this naming convention)
    environment?: pulumi.Input<{[key: string]: pulumi.Input<string>}>;
}

export class WebApp extends pulumi.ComponentResource {
    // This works but Dave will probably complain about the naming
    public readonly url: pulumi.Output<string>;
    
    constructor(name: string, args: WebAppArgs, opts?: pulumi.ComponentResourceOptions) {
        super(\"company:platform:WebApp\", name, {}, opts);
        
        // Load balancer, auto-scaling group, security groups, etc.
        // All with company security policies and monitoring built-in
        // NOTE: health checks still broken in us-west-1, ask Sarah about it
    }
}

Then Build the Self-Service Layer

With standardized components available, implement multiple consumption models to meet different team needs. This is where things get tricky because you're trying to please everyone - developers who want flexibility, ops who want control, and security who want everything locked down.

No-Code Templates

• Create organization templates for common deployment scenarios
• Store configuration in Pulumi ESC for easy updates
• Enable non-technical teams to deploy standardized infrastructure through web interfaces

Low-Code YAML Programs

• Provide YAML examples that compose existing components
• Create starter templates for common use cases
• Document configuration options and customization points

Full-Code Development

• Scaffold projects using pulumi new with organization templates
• Provide component libraries as npm/PyPI packages
• Enable advanced teams to extend and customize as needed

Integration with Developer Portals

• Connect with Backstage for teams already invested in catalog-driven development
• Provide APIs for custom internal portals using Automation API
• Support GitOps workflows with automated deployments

Make It Actually Work in Production

Scale your platform engineering practice with operational tooling and processes that prevent it from becoming a bottleneck. This is where you find out if your fancy platform actually works when shit hits the fan at 3am.

Policy Automation

• Implement CrossGuard policies for security, compliance, and cost controls
• Automate policy violations with remediation workflows
• Create policy test suites to validate compliance before deployment

Secrets Management

• Centralize secrets in Pulumi ESC with automatic rotation
• Implement least-privilege access to sensitive configurations
• Audit secret usage across all environments and teams

Cost Optimization

• Use Pulumi Insights to identify oversized resources and unused assets
• Implement automated cost alerts and budget controls
• Provide cost attribution by team and project for chargeback

Monitoring and Alerting

• Deploy observability stack as standardized components
• Monitor platform adoption metrics and component usage
• Alert on policy violations, cost overruns, and deployment failures

Finally, Add the AI Stuff (If It Actually Helps)

Layer AI capabilities on top of your mature platform to accelerate developer productivity and operational efficiency. Do this last - AI won't save a shitty platform, but it can make a good one better.

Enable Pulumi Copilot

• Activate Pulumi Copilot for your organization
• Train teams on AI-assisted infrastructure debugging and generation
• Integrate CLI AI features into developer workflows

Intelligent Resource Management

• Use AI insights for proactive infrastructure optimization
• Automate capacity planning based on usage patterns
• Predict infrastructure failures before they impact applications

Enhanced Developer Experience

• Provide natural language interfaces to infrastructure operations
• Generate infrastructure code from requirements descriptions
• Automate common troubleshooting and remediation tasks

Common Implementation Pitfalls to Avoid

Starting with Portals: Don't begin with user interfaces. Build infrastructure foundations first, then add UI layers. I've seen teams spend a year building Backstage scaffolding templates that generate broken code because nobody validated the underlying infrastructure patterns.

Perfectionism: Don't try to standardize everything immediately. Focus on high-impact, frequently-used patterns first. The team that tries to create a "universal deployment component" that handles every possible edge case will still be arguing about YAML schemas 18 months later.

Single Team Ownership: Platform engineering requires buy-in from multiple teams. Include security, operations, and development teams in design decisions. The platform team that builds in isolation creates technically excellent solutions that violate every security policy and nobody can actually use.

Ignoring Existing Workflows: Don't force dramatic workflow changes. Meet teams where they are and gradually introduce platform capabilities. If your developers are used to kubectl apply -f, don't make them learn a completely new deployment system on day one.

Underestimating Change Management: Technical implementation is often easier than organizational adoption. Plan for training, documentation, and gradual migration. I've watched perfect technical platforms fail because nobody bothered to teach developers how to use them, and the old manual processes were "just easier."

Success Indicators: What Good Looks Like

After a few months, if you're lucky, teams start using the standardized components. Infrastructure ticket volume might decrease by 40%, but some teams still submit tickets because they're afraid to break shit.

Around 6 months, you might see multiple consumption models being used. Security policy violations should decrease by 60% due to automated enforcement, though someone will inevitably find a creative way to fuck up IAM roles.

Later in the year, AI-enhanced operations might provide useful optimization recommendations. Infrastructure costs get optimized and attributed by business service - after you've identified that one team running 50 idle instances "just in case."

Eventually, the platform engineering practice becomes self-sustaining with standardized components maintained collaboratively. New infrastructure patterns get evaluated for componentization, and you finally stop getting paged at 3am for stuff that should have been automated from day one.

The key insight: successful platform engineering implementations prioritize infrastructure standardization and developer adoption over tool proliferation. Focus on solving real problems with solid foundations, and the user experience improvements will follow naturally.