Portfolio · Notes · Dotfiles

Search everything

Search case studies, engineering notes, and Dotfiles documentation.

    all case studies

    Case study 07

    Kafka topics as code: adopting 550 live topics

    About 550 live Kafka topics gained versioned definitions and named owners without recreation. Admission policies then constrained changes that could bypass the reviewed workflow.

    4 min read

    My role
    Built the migration generator, designed the environment layout, ran the adoption, and introduced the admission policies.
    Evidence
    About 550 topics adopted without recreation; replica changes and partition reductions rejected at admission; production topic ownership synchronized to the service catalog.
    reliabilitydeliverysecurity

    Adopt the running system without recreating it

    Kafka topics were managed manually across clusters and environments. A Markdown runbook recorded creation commands, but it could not reliably describe the current inventory, ownership, or differences between environments.

    I moved about 550 live topics into Git as Strimzi KafkaTopic resources. The main constraint was preserving the existing topics and their data while bringing their configuration under review.

    Reconcile documentation against live state

    I wrote a generator that parsed the runbook’s creation commands and queried the live brokers. It compared the two, used production as the baseline, and produced three kinds of configuration:

    Shared topic. Common configuration across environments

    Environment-only topic. A topic present only in that environment

    Override patch. Only the fields that differ from the shared definition

    The generator removed settings that merely repeated broker defaults and resolved owning teams from the service catalog. It also produced a discrepancy report, exposing topics and settings that the documentation and clusters disagreed about.

    EXHIBIT — TWO SOURCES OF TRUTH, ONE GENERATED TREE
    Markdown runbookthe --create commands, parsedLive clustersevery broker queried, in parallelReconcileproduction is the baselineShared topicstopics that existin every environmentEnvironment extrasone folder perenvironmentOverride patchesonly the fieldsthat differDiscrepancy reportwhat the runbook andreality disagreed on

    Each resulting topic is a small reviewable file. This example uses a fictional topic and owner:

    topics/shared/payment-events.yaml
    apiVersion: kafka.strimzi.io/v1
    kind: KafkaTopic
    metadata:
      name: payment-events
      namespace: messaging
      labels:
        strimzi.io/cluster: messaging
        owner: team_payments
    spec:
      topicName: payment.events
      partitions: 12
      replicas: 3
      config:
        cleanup.policy: compact
        retention.ms: -1
        min.compaction.lag.ms: 86400000
        segment.bytes: 1000000000

    Keeping metadata.name separate from spec.topicName preserves Kafka names that contain characters Kubernetes object names cannot use. The config block carries workload-specific retention, compaction, and segment settings alongside partitions and replicas, making those decisions visible in the same review.

    Control both adoption and later changes

    Because the generated definitions matched live state, the operator adopted the topics instead of recreating them. During the migration, automatic deployment applied changes with deletion disabled.

    I introduced Kyverno policies that reject changes to replica counts and reductions in partition counts. A separate policy blocks the developers group from opening shells in broker pods, closing the manual path that had previously bypassed review. Direct topic deletion is also restricted through RBAC.

    Production topic resources are synchronized into the service catalog, connecting the deployed inventory to the teams that own it.

    What changed

    About 550 topics came under management without recreation. Topic changes became pull requests with owners, review, and history. The generated definitions replaced the runbook as the inventory, and the admission policies helped keep subsequent changes inside that workflow.