The Art of Knowing When to Write: Lessons from Building Certificate Infrastructure

Picture this: You open your laptop, and within seconds it authenticates to your corporate WiFi network using a certificate that didn't exist sixty minutes ago. Not a certificate your IT department provisioned or one you requested through a portal—but one your code generated, signed by your own CA, and deployed entirely through automation you'd just finished writing.

That's exactly what happened to one developer who built their own identity management infrastructure at home using FreeIPA. But the real story isn't the certificates themselves—it's the surprisingly thorny question they had to answer: when, exactly, should you write something down?

The Declarative Infrastructure Dream

If you've spent any time with infrastructure-as-code tools, you know the pitch: describe what you want, let the system make it happen, and trust that the system understands the current state. The dream is declarative configuration where you say "this server should exist with these settings" and the tool figures out how to get there.

The problem? That dream quietly assumes your operations are atomic. Create a file, done. Delete a user, done. But real infrastructure is messier. Real operations have intermediate steps. Real steps have consequences that outlast the operation itself.

The developer in question built a tool called swamp to manage FreeIPA resources declaratively. FreeIPA is an identity platform that handles Kerberos, LDAP, certificate authorities, and more—the kind of stack that companies use for single sign-on, except this particular deployment was operated entirely by hand.

The goal was simple: describe an identity object, have something reconcile it, and keep a record of what happened. But here's where it gets interesting.

The Three-Layer Write Problem

When a mutating operation runs, there's a deceptively simple question: when do you record what happened?

Most tools default to "throw before write." If the operation fails, persist nothing. A stored record claiming success when the world never delivered it is a lie—and that lie will poison every downstream system that reads it.

This is correct advice. It remains correct. But it quietly assumes every kind of "what happened" wants to be written at the same instant. And sometimes that assumption breaks.

Layer One: Irreplaceable Material

When you issue a certificate, your code typically generates a keypair first. The public key goes into a CSR, the CA signs it, and you get back a certificate. The private key? That only exists once. It was generated in memory, and if something crashes after the CA signs but before you've saved the key, you've lost it forever.

But the certificate already exists. It's real. It's out in the world, signed and valid. And it will never be usable without the private key you just lost.

So here's the insight: write the irreplaceable thing the moment it's real, before anything else gets a chance to fail. Flip the default. Save the private key immediately, even if the operation isn't complete yet, because you might never get another chance.

Layer Two: Partial Failure

Real operations often do multiple things. Create a user and add them to three groups. Provision a service and register its DNS records. Issue a certificate and configure the service to use it.

What happens when the second step fails? You know the operation "failed," but you don't know what's broken. The user might exist but be in only one of three groups. The certificate might be signed but the service never restarted.

A failed mutation that persists nothing tells you nothing. But a partial record—one that honestly says "step one succeeded, step two failed"—gives you something actionable. It tells you where to look.

Layer Three: Audit Records

Here's the third case where the simple rule breaks: audit records.

An audit record isn't state. It's telemetry. When it says "success: false," it's telling the truth, not lying. Unlike the final state of a resource, an audit record of failure is accurate. It describes reality accurately: this operation was attempted and did not complete.

So write audit records on both paths—success and failure. The failure audit record is honest, which means it's safe to persist.

The Three Rules, Summarized

This leads to a small but robust framework for mutation persistence:

1. State gets written only on success. A failed mutation should never publish state claiming otherwise. The original rule holds.

2. Irreplaceable material gets written the moment it's real. Private keys, generated secrets, anything that exists exactly once. Write it before the operation completes, because it might not get a second chance.

3. Audit records get written on both paths. Success, failure—doesn't matter. An audit record describes what happened, and a failure record that accurately describes failure isn't lying about anything.

This isn't as clean as a single rule. It's three rules that occasionally pull in different directions. But that's the point: real infrastructure has real complexity, and a single slogan about when to persist data doesn't cover all the cases.

Why This Matters for Your Infrastructure

You might not be running FreeIPA at home or building certificate authorities from scratch. But you're probably running databases, deploying services, or managing cloud resources. And at some point, you'll write automation that creates something irreplaceable.

Maybe it's a database encryption key. Maybe it's a signing key for your artifacts. Maybe it's a bootstrap token for a new service.

When that moment comes, the question isn't just "how do I create this thing?" It's "when do I make sure this thing survives a crash?" And the answer isn't always "at the end, when everything succeeds."

Sometimes you need to write the fragile thing first. Sometimes you need to accept partial success as better than total failure. And sometimes you need audit records that honestly say "it didn't work" instead of pretending the operation never happened.

The boring part—knowing when to persist data—is the part worth getting right.

If you're exploring infrastructure automation, certificate management, or declarative configuration, these patterns are worth carrying in your toolkit. The principles scale: whether you're managing a homelab or a fleet of cloud services, the question of when your automation protects its own work is never just an implementation detail.