How to Build Your Own Internal Developer Platform (IDP) Using Crossplane

In the modern cloud-native landscape, the friction between operation stability and developer velocity remains a critical bottleneck. As organizations scale, the manual ticketing systems traditionally used to provision infrastructure become unsustainable. The solution lies in platform engineering: building an Internal Developer Platform (IDP) that enables self-service capabilities without sacrificing governance or security.

While tools like Terraform have dominated Infrastructure as Code (IaC), they often rely on client-side state and imperative CI/CD pipelines. This article explores a control-plane-centric approach using Crossplane, an open-source framework that extends Kubernetes to manage infrastructure. By leveraging Crossplane, we can build an IDP that treats infrastructure as data, enforcing continuous reconciliation and drift detection.

For organizations looking to accelerate this transition, partnering with experts in cloud engineering services remote teams can significantly reduce the architectural overhead required to implement these control planes effectively.

Product Engineering Services

Work with our in-house Project Managers, Software Engineers and QA Testers to build your new custom software product or to support your current workflow, following Agile, DevOps and Lean methodologies.

Build with 4Geeks

Why Crossplane for your IDP?

Standard IaC tools are often "fire and forget." Once the pipeline runs, the state of the live infrastructure can drift from the configuration code without immediate detection. Crossplane fundamentally shifts this paradigm by using the Kubernetes Operator pattern.

1. Continuous Reconciliation: Just as Kubernetes ensures a Pod is running, Crossplane actively monitors external resources (like AWS RDS or Google Cloud SQL) and ensures they match the desired state defined in your cluster.
2. API-Driven Abstraction: Crossplane allows Platform Engineers to define Composite Resource Definitions (XRDs). These expose high-level, simplified APIs to developers (e.g., PostgresDB) while hiding the complex infrastructure details (e.g., VPC peering, subnet groups, parameter groups) behind the scenes in Compositions.
3. Unified Control Plane: You manage application workloads and infrastructure using the same Kubernetes API and tooling (kubectl, Helm, ArgoCD).

Architecture: The Composition Model

To build an IDP, we must distinguish between two roles:

• Platform Engineers: Author the XRDs and Compositions (the "Classes").
• Application Developers: Create Claims (the "Instances").

We will implement a scenario where a developer needs a PostgreSQL database. Instead of writing 500 lines of Terraform, they will apply a simple Kubernetes manifest.

Step 1: Installing Crossplane

First, install Crossplane into your management Kubernetes cluster using Helm.

helm repo add crossplane-stable https://charts.crossplane.io/stable
helm repo update

helm install crossplane \
  crossplane-stable/crossplane \
  --namespace crossplane-system \
  --create-namespace

Step 2: Configuring the Provider

We need a Provider to communicate with the cloud vendor. We will use AWS for this example.

Note: Ensure you create a Kubernetes Secret containing your AWS credentials (aws-creds) before applying the ProviderConfig.

apiVersion: pkg.crossplane.io/v1
kind: Provider
metadata:
  name: provider-aws-s3
spec:
  package: xpkg.upbound.io/crossplane-contrib/provider-aws:v0.47.0
---
apiVersion: aws.crossplane.io/v1beta1
kind: ProviderConfig
metadata:
  name: default
spec:
  credentials:
    source: Secret
    secretRef:
      namespace: crossplane-system
      name: aws-creds
      key: creds

Step 3: Defining the API (XRD)

The CompositeResourceDefinition (XRD) defines the schema of the API you are offering to developers. Here, we define a resource type XPostgresInstance.

apiVersion: apiextensions.crossplane.io/v1
kind: CompositeResourceDefinition
metadata:
  name: xpostgresinstances.database.example.org
spec:
  group: database.example.org
  names:
    kind: XPostgresInstance
    plural: xpostgresinstances
  claimNames:
    kind: PostgresInstance
    plural: postgresinstances
  versions:
  - name: v1alpha1
    served: true
    referenceable: true
    schema:
      openAPIV3Schema:
        type: object
        properties:
          spec:
            type: object
            properties:
              parameters:
                type: object
                properties:
                  storageGB:
                    type: integer
                  nodeSize:
                    type: string
                    enum: [ "small", "medium", "large" ]
            required:
            - parameters

Step 4: Creating the Composition

The Composition maps the high-level inputs from the XRD (like small or large) to specific cloud resources. This is where you enforce compliance (e.g., ensuring encryption is always on).

apiVersion: apiextensions.crossplane.io/v1
kind: Composition
metadata:
  name: postgres-aws-composition
  labels:
    provider: aws
    type: backend-db
spec:
  compositeTypeRef:
    apiVersion: database.example.org/v1alpha1
    kind: XPostgresInstance
  resources:
  - name: rds-instance
    base:
      apiVersion: database.aws.crossplane.io/v1beta1
      kind: RDSInstance
      spec:
        forProvider:
          region: us-east-1
          dbInstanceClass: db.t3.micro
          masterUsername: masteruser
          engine: postgres
          engineVersion: "13.4"
          skipFinalSnapshot: true
          publiclyAccessible: false
          storageEncrypted: true # Enforcing security policy
        writeConnectionSecretToRef:
          namespace: crossplane-system
    patches:
    - fromFieldPath: "spec.parameters.storageGB"
      toFieldPath: "spec.forProvider.allocatedStorage"
    - fromFieldPath: "spec.parameters.nodeSize"
      toFieldPath: "spec.forProvider.dbInstanceClass"
      transforms:
      - type: map
        map:
          small: "db.t3.micro"
          medium: "db.t3.medium"
          large: "db.m5.large"

Step 5: The Developer Experience (The Claim)

Once the Platform Engineering team applies the XRD and Composition, developers can provision a database via a Claim. This manifest is what lives in the application repository.

apiVersion: database.example.org/v1alpha1
kind: PostgresInstance
metadata:
  name: my-service-db
  namespace: app-team-a
spec:
  parameters:
    storageGB: 20
    nodeSize: small
  compositionSelector:
    matchLabels:
      provider: aws
      type: backend-db

When this YAML is applied, Crossplane instantly creates the underlying AWS RDS instance, wires up the networking (if defined in the Composition), and outputs the connection string to a Kubernetes Secret in the developer's namespace.

Product Engineering Services

Work with our in-house Project Managers, Software Engineers and QA Testers to build your new custom software product or to support your current workflow, following Agile, DevOps and Lean methodologies.

Build with 4Geeks

Handling Complexity and Governance

Building a robust IDP goes beyond simple provisioning.

• GitOps Integration: Because Crossplane resources are standard Kubernetes objects, you can manage your entire platform using ArgoCD or Flux. Infrastructure changes become Pull Requests.
• Policy as Code: Integrate Open Policy Agent (OPA) or Kyverno to validate Claims before they are applied. For example, you can prevent developers from requesting large instances in the dev environment to control costs.
• Secret Management: Crossplane can integrate with HashiCorp Vault or AWS Secrets Manager to ensure database credentials are never stored in plain text within the cluster.

Conclusion

Transitioning to a control-plane-based IDP with Crossplane empowers developers with true self-service while allowing leadership to maintain rigorous standards regarding security and cost. It transforms infrastructure from a bottleneck into a product.

However, the shift involves a steep learning curve regarding Kubernetes Operators and API design. For organizations aiming to implement cloud engineering services remote teams to scale this architecture, 4Geeks provides the specialized expertise required to build, secure, and maintain high-performance Internal Developer Platforms.

FAQs