Chamber 🏰 of Tech Secrets #50: Living on the Edge Part I

Establishing Trust via a Custom Attestation Server

The Chamber 🏰 of Tech Secrets is open for the 50th time! A quick note of thanks to all the readers and commenters. I am grateful you take the time to read and engage. 🙏

Designing a Custom Certificate Authority and Attestation Server

Highly distributed and rapidly developed applications are wonderful… but they also open lots of new attack vectors for the proverbial “bad guys”. 🦹🏻

In the world of edge computing and the emerging AI-dominated world, the basics of security will matter more than ever. One of the most essential components of securing the modern internet is certificates, issued by certificate authorities (aka “CA”).

In this post, we’ll take a look at what certificates are, how they come to be and how they work in practice. Then we’ll explore why they are essential to edge environments, and how I designed a custom CA and attestation server for my personal edge lab. Let’s go!

🔐 Certificates and Certificate Authorities

First, we’ll ensure we have a baseline understanding of the certificate concepts and how they apply to security and identity in the era of the internet.

What is a certificate?

Certificates are built on Public Key Infrastructure (PKI) which uses asymmetric cryptography, a technique in which a public key is shared widely and an associated private key is kept secret.

A certificate is a digitally signed document that binds a public key to the identity of a system, device, or person. It is signed by an authority's private key, allowing others to verify the identity using the authority’s public key as a foundation of trust.

A typical certificate contains:

• Public Key – The key used to establish secure connections.

• Identity Information – Details like a domain name, device name, or organization.

• Issuer Signature – The CA’s digital signature verifying authenticity.

• Validity Period – Defines the window during which the certificate is valid.

• Extensions – Additional fields like usage constraints or Subject Alternative Names (SANs).

What is a certificate authority?

A Certificate Authority (CA) is a trusted entity that issues and signs certificates. The CA vouches for the authenticity of the identity information included in the certificates it signs. This trust model applies to both public internet CAs (like DigiCert or Let's Encrypt) and private, internal CAs (self-signed) within organizations.

When a client—such as a web server or edge device—wants a certificate, it generates a key pair and sends a Certificate Signing Request (CSR) to the CA. The CA verifies the attributes in the request (through automated checks or manual validation), and if approved, returns a certificate signed with its own private key.

To protect its root private key, which acts as the foundation of trust, most Certificate Authorities (CAs) do not use it directly to sign end-entity certificates. Instead, they create one or more intermediate CAs, each with its own private key and certificate. The root CA signs the certificate of the intermediate, delegating trust. This layered model reduces risk while maintaining a verifiable chain of trust back to the root.

Now that we know how certificates get created, let’s look at how they work in practice.

🌎 How Certificates Work on the Web

When you visit a secure website like https://brianchambers.substack.com, the site presents a TLS certificate as part of the initial connection handshake. This certificate includes the domain name and a digital signature from a trusted public Certificate Authority (CA), such as DigiCert or Let’s Encrypt.

Web browsers come preloaded with a list of trusted root certificates, each of which contains the public key of a known and trusted CA. When the browser receives a website’s certificate, it performs a cryptographic signature validation using the public key from the issuing CA (which may be an intermediate certificate signed by a root CA) to validate the signature and build a chain of trust. If the signature is valid, the certificate has not expired, and the domain name matches, the browser proceeds with establishing the secure connection.

Source: https://www.exoprise.com/2019/07/29/monitor-ssl-expiration-spoofing-changes/

In short, the certificate proves the identity of the website and enables the browser to trust that it is safe to proceed with communications.

🖥️ How Certificates Work for Devices

Certificates work similarly for devices, such as edge compute nodes. The goal is to ensure that when a device connects to a network or service, you can trust who it claims to be.

Here’s how that typically works:

• The device generates a key pair (public/private).

• It sends its public key, along with identity metadata (e.g., serial number, device type), in a Certificate Signing Request (CSR) to a CA.

• The CA issues a certificate for the device, signing its public key and identity data.

• The device can now authenticate to systems, access APIs, or join secure networks—presenting this certificate as proof of identity.

In this model, the certificate is endorsing the device's identity (like its serial number, hardware ID, or assigned hostname). In some cases, this also attests to to the device’s measured boot state, which proves it booted into a known and trusted configuration. (more on that in a future post).

🤝 The Unifying Concept: Certificates Prove Identity

Whether it’s a website or a device, a certificate is doing the same fundamental thing: It cryptographically proves the identity of a digital entity and enables others to trust it.

If you just wanted to learn about the basics, you can sign off now. If you’d like to read about how I designed my own certificate authority and attestation server for onboarding edge devices, read on!

Building a Custom Certificate Authority

Certificates are an important construct for establishing trust at the edge. In order to do attestation—the process of ensuring a device is in a known, trusted, and unmodified state—we need an attestation server to and a CA to issue certificates.

In support of my Edge Lab project, I decided to write my own certificate authority, which will have authority over my local edge domain. This affords me the opportunity to shape a REST API around my edge device attestation requirements as well. This is a logical approach for an “enterprise” of my size (1 employee, $0 revenue) where the one employee is just trying to learn by doing. 😎

High Level Architecture

We’ll unpack this target architecture as we go.

System Design & Requirements

Here are the key system design attributes for the attestation server:

• Goals

  • Establish a hardware root of trust using the device’s onboard TPM. We won’t get into these details until the next post, but our server plays a critical role.

  • Facilitate secure edge node onboarding and identity management using Public Key Infrastructure (PKI) patterns and certificates.

  • Enable measured boot at attestation time to ensure zero tampering has taken place and the device is in a trusted state.

  • Facilitate zero touch provisioning, meaning no human has to be present to complete any steps on the edge device.

  • Handle ongoing attestations, key rotations, and other necessary features.

  • Stop the device from participating in the edge cluster if it is tampered with, if certificates expire, or if it boots in an unexpected state.

  • Enable certificate revocation from edge devices or edge attestation servers when needed.

• Requirements

  • Handle device attestation bundle inputs, which are subsequently validated, resulting in certificate outputs.

  • Serve root and intermediate certificates for dynamic signature validation. In other words, come to the attestation server if you need the certificates to validate something signed by the CA.

  • Sign intermediate certificates for edge site local CAs. Since we contemplate having 1..n clusters (I’ll have 2+ at my house over time), we need to be able to issue intermediates so that we can run local attestation servers (the ideal model, though those must bootstrap themselves by talking to our central attestation server).

  • Support deployment of cloud or edge environments.

  • Support certificate renewal using previously issued certificate and PCR + nonce combo (see Chamber of Tech Secrets #51).

  • The root private key for the CA must be air-gapped and stored completely offline. Intermediate private keys for cloud deployments will be stored in Google Cloud KMS (you could use AWS or another service as well). Edge sub-intermediate private keys will be generated in the TPM on the edge attestation server and send to the cloud attestation server for signature.

  • Allow configuration to determine where the attestation server looks for its signing key since it could reside in the TPM or in a cloud service (KMS).

• Implementation Details

  • REST API implementation following Google API design patterns w/ OpenAPI 3.1 specification.

  • Implemented in Golang w/ bias for idiomatic practices.

  • PostgreSQL backend stores attestation records, certificate metadata, and device registration state.

  • Containerized deployment targeting Google Cloud Run (Cloud) and K3s (Edge).

  • Possible reverse proxy or API Gateway to offload concerns. Architecture TBD.

API Design 0.0.1

This RESTful API will cover our initial requirements but will likely undergo some iteration over the course of the project. I am sure I forgot something…hence the version 0.0.1 indicating an unstable API (< version 1.0).

*** revocation endpoint is not currently defined and will be added later.

Certificate Hierarchy

Finally, let’s recap how this certificate hierarchy works between cloud and edge.

🏠 Root Certificate Authority (CA) — Offline

• Self-signed

• Only used to sign intermediate CA certs

• Private key stored in offline, air-gapped Hardware Security Module (HSM)

☁️ Cloud Intermediate CA — GCP KMS

• Private Key stored in GCP KMS.

• Signs intermediate certs for local attestation servers (delegated intermediates)

• Signs edge device certificates if they participate in direct-to-cloud attestation

🖥️ Local Attestation Server (Edge "Sub-Intermediate")

• Has a private key generated & stored in a TPM on the local server

• CSR is sent to cloud intermediate CA, which issues an intermediate cert

• That intermediate cert:

  • Identifies this as an authorized signing authority

  • Includes name constraints or other policy constraints to limit scope. “These constraints may include name constraints, Subject DNs, or certificate policy OIDs that identify the delegated scope of each attestation server. For example:

    • OID = 1.2.3.4.5.100 for cloud-issued device certs

    • OID = 1.2.3.4.5.XXXXX for site-issued device certs, where XXXXX indicates our site unique identifier in integer form (.1, .4365, .50042, etc.)

    • In this case we’ll use the sample OID “1.2.3.4.5.XX” since we are not expecting to share our certs externally and do not want to bother registering for an enterprise OID arc with IANA.

• The server:

  • Acts as the CA of record and signs CSRs from edge devices in its environment

  • Validates measured boot values

📦 Edge Devices

• Each device generates its own attestation key in TPM

• Sends a CSR to the local attestation server

• Receives a cert signed by that local server’s TPM-held key

• Uses that cert for secure identity and attestation

Conclusion

In this post, we learned how certificates get created and how they work as vouchers for a device or domain identity. We also learned how a Certificate Authority and Attestation Server are foundational building blocks for secure, scalable edge computing deployments, and reviewed the design specification for the one I’ll use in my edge lab.

So, what’s next?

• I’ll finish implementation of the Attestation Server + CA and I’ll publish the code to GitHub for those that are interested. I have a 16 month old, so I am not as fast as I once was.

• The next blog post in the series is about how to establish a hardware root of trust for a new edge node and implement measured boot.

• I’ll implement an initial device config and the edge node code (hey, a rhyme!) that talks to our attestation server(s) on boot. This will get delivered via a minimal PXE boot when the device boots the first time.

• I’ll write another blog post about how credential sealing works, which will provide another layer of security for our device, once again depending on our trusty TPM.

• I have yet to mention networking, which is pretty important. I’ll write a separate post about that soon enough!

• Then we’re on to the easy stuff… PXE booting a “permanent” image, K3s install, GitOps, observability, WASM, GPUs, radar IoT devices, and more.

Thanks for reading — this one gets into a lot of sticky technical design details and probably is tough reading, but I hope you stuck with it, learned something new, and tool the time to understand the concepts.