Transport Layer Security (TLS) lapses caused outages; Venafi with cert?manager automated F5/NGINX Ingress renewals and audit trail

Overview

An online publisher experienced recurring outages when TLS certificates expired on load balancers, Kubernetes ingress, and internal services. Certificates were issued by multiple certificate authorities (CAs), tracked in spreadsheets, and renewed manually. Ownership was unclear, renewal reminders were inconsistent, and inventories were incomplete for audit. Intelligex integrated Venafi as the certificate lifecycle system, connected it to load balancers and Kubernetes ingress via automation, and tied internal Public Key Infrastructure (PKI) into the same workflow. Ownership metadata, renewal policies, and alerts were enforced end?to?end. Outages from lapses subsided, renewals followed predictable paths, and audit evidence drew from a complete inventory—while existing infrastructure and change processes stayed in place.

Client Profile

• Industry: Digital media and online publishing
• Company size (range): Multi?brand portfolio with global sites and APIs
• Stage: Mix of public and internal CAs; F5 and NGINX?based load balancers; Kubernetes ingress for services; fragmented certificate tracking; manual renewals
• Department owner: IT & Infrastructure (Platform/SRE and Network)
• Other stakeholders: Security, Web Engineering, API Platform, DevOps, Compliance, Legal, Customer Support, Internal Audit

The Challenge

Certificates lived everywhere. Web properties terminated TLS on F5 and NGINX appliances; microservices used Kubernetes ingress; internal apps fronted by proxies or service meshes relied on local keystores. Public and private CAs issued certificates, and some keys were generated on endpoints. Spreadsheet trackers fell out of date, owners changed, and auto?renew jobs worked only for a subset of endpoints. When an expiry approached, finding the right certificate, CA, endpoint, and owner took time the team didn’t always have.

There was no consistent lifecycle. Domains moved between brands, wildcard certificates hid usage, and SAN scopes grew over time without review. Certificates were requested via email, CSR generation varied by platform, and key storage practices were inconsistent. Renewal efforts collided with change freezes or marketing events. Audit asked for a complete inventory, ownership, issuance path, and renewal evidence; assembling that required log dives and manual reconciliations.

Why It Was Happening

Root causes were fragmented ownership and a missing system of record for certificates. Issuance flowed through multiple paths: public CAs for external domains, internal PKI for services, and ad hoc processes for one?off needs. Load balancers and clusters managed their own keystores. Request, approve, issue, install, and renew steps weren’t connected, and ownership metadata wasn’t attached to the certificate object. Reminders were tied to inboxes, not to a governed workflow. Without a discovery and policy layer, certificates expired unnoticed or were renewed late with manual hotfixes.

Timing and governance were misaligned. Networking owned edge termination, Platform owned Kubernetes ingress, Security owned PKI, and Web Engineering owned domains. No single orchestration layer tied issuance and renewal to owners, endpoints, and change approvals with reliable alerts and evidence.

The Solution

Intelligex deployed Venafi as the certificate lifecycle backbone and integrated it with load balancers, Kubernetes ingress, and internal PKI. Certificates were discovered and inventoried with ownership, service mappings, and renewal policies. Automation handled CSR generation, approval, issuance, and installation. Kubernetes integrations used cert?manager with Venafi issuers; load balancers synchronized certificates through APIs; internal PKI issued private certificates through controlled workflows. Renewal alerts and change approvals were enforced, and evidence was logged. The design leveraged Venafi documentation (Venafi Docs), Kubernetes ingress concepts (Kubernetes Ingress), and cert?manager’s Venafi issuer integration (cert?manager Venafi).

• Integrations: Venafi policy and inventory; F5 BIG?IP and NGINX?based load balancers via API; Kubernetes ingress via cert?manager Venafi issuers; internal PKI (for example, Microsoft AD CS) for private issuance; DNS validation and domain management; ITSM for approvals and notifications.
• Ownership and metadata: Certificate objects tagged with owner, service, environment, renewal policy, and endpoint mappings; linkbacks to CMDB application services where available.
• Policy and issuance: Approved CAs and templates per domain category; key size and algorithm standards; SAN and wildcard controls; automated CSR generation; maker?checker approval for sensitive scopes.
• Discovery and import: Network and endpoint scans to find unmanaged certificates; ingestion of existing keystores; normalization under one inventory with lifecycle state.
• Automation to endpoints: API?driven install/rotate on F5 and NGINX; Kubernetes certificates managed by cert?manager with Venafi issuers; blue/green certificate swaps to minimize impact.
• Renewals and alerts: Renewal windows by risk tier; alerting to owners and on?call; ITSM change creation for high?impact endpoints; escalation paths for missed responses.
• Security and key handling: Private key protections and device trust; HSM-backed CAs where applicable; no keys printed in logs; role?based access for requests and approvals.
• Dashboards and evidence: Expiry posture, ownership gaps, policy violations, renewal pipeline status, and endpoint sync health; exportable evidence for audits.

Implementation

• Discovery: Cataloged domains, endpoints, load balancers, ingress controllers, and PKI paths; scanned for unmanaged certificates; inventoried current CA templates and policies; reviewed outage timelines tied to expirations; gathered audit and change requirements.
• Design: Defined certificate metadata and ownership tags; authored CA and template policies by domain category; mapped cert?manager issuers for clusters; designed install and rotation flows for load balancers and proxies; specified renewal windows, alerts, and escalation; planned dashboards and evidence exports.
• Build: Implemented Venafi as the inventory and policy engine; integrated F5/NGINX APIs for install and rotate; configured cert?manager with Venafi issuers per cluster/namespace; linked internal PKI templates; enabled discovery and import; connected ITSM for approvals and renewal tickets; assembled dashboards and logging.
• Testing/QA: Ran discovery in read?only mode; imported known certificates and reconciled owners; piloted automated renewals on non?critical endpoints; rehearsed blue/green swaps; validated cert?manager issuance and renewals; tested alerting and escalation paths; verified audit logs and evidence packs.
• Rollout: Enabled automation for edge properties first, then internal services; kept manual renewals as a controlled fallback during initial cycles; expanded cluster integrations as stability grew; required ownership tags and policy compliance for new requests.
• Training/hand?off: Delivered sessions for Network, Platform, Security, and Web Engineering on requests, ownership tagging, and endpoint sync; updated SOPs for domain onboarding, wildcard and SAN requests, and renewals; transferred policy and dashboard ownership to Security and Platform under change control.
• Human?in?the?loop review: Established recurring reviews for expiring high?impact certs, ownership gaps, wildcard/SAN governance, and failed installs; decisions recorded with rationale and effective dates; improvements fed back into policies and automation.

Results

Certificate lifecycle moved from ad hoc reminders to a predictable flow. Owners saw upcoming renewals with clear windows, approvals and installations followed the same steps every time, and endpoints updated through automation. Expiry?related incidents declined because coverage and alerts spanned edge properties, clusters, and internal apps, and installations used blue/green swaps where possible.

Visibility and audit posture improved. The inventory reflected reality: every certificate carried owners, endpoints, policy template, and lifecycle state. Dashboards highlighted gaps early, and evidence packs showed issuance and renewal history with approvals and install logs. Venafi, load balancers, Kubernetes, and PKI remained; the addition was a lifecycle and governance layer that tied them together.

What Changed for the Team

• Before: Certificates tracked in spreadsheets and inboxes. After: A governed inventory with owners, endpoints, and renewal policies.
• Before: Manual CSRs and installs varied by platform. After: Automated CSR, approve, issue, and install flows via APIs and cert?manager.
• Before: Wildcards and SANs drifted. After: Policies and approvals governed scope and template selection.
• Before: Renewals collided with business events. After: Renewal windows aligned with change calendars and alerts routed to owners.
• Before: Outages traced to lapses. After: Expiry posture and endpoint sync health flagged issues well before impact.
• Before: Audit evidence was reconstructed. After: Issuance, approvals, and installs linked to tickets with exportable logs.

Key Takeaways

• Make certificate lifecycle a system, not a spreadsheet; centralize inventory, policy, and ownership.
• Automate end?to?end; generate CSRs, approve, issue, and install through APIs and cluster?native controllers.
• Control scope; govern wildcard and SAN usage with templates and maker?checker approvals.
• Integrate renewal with change; align windows, alerts, and installs to avoid business conflicts.
• Discover and import first; reconcile existing keystores before enforcing new paths.
• Integrate, don’t replace; keep load balancers, clusters, and PKI, and add a lifecycle and governance layer.

FAQ

What tools did this integrate with? Venafi served as the lifecycle and policy engine (Venafi Docs). Kubernetes integrations used cert?manager with Venafi issuers (cert?manager Venafi) and standard ingress patterns (Kubernetes Ingress). Load balancers such as F5 and NGINX synchronized certificates via APIs. Internal PKI (for example, Microsoft AD CS) issued private certificates through approved templates. ITSM handled approvals and renewal notifications.

How did you handle quality control and governance? CA and template policies enforced key algorithms, SAN limits, and issuance paths. Requests required ownership tags and, for sensitive scopes, maker?checker approvals. Discovery identified unmanaged certificates and reconciled them under policy. All issuance, renewal, install, and approval events were immutably logged, and dashboards highlighted policy violations and ownership gaps.

How did you roll this out without disruption? Discovery and import ran first in read?only mode to build an accurate inventory. Automated renewals were piloted on low?risk endpoints and non?production clusters. Manual renewals remained a controlled fallback during initial cycles. Edge properties moved first, then internal services, with change windows and rollback steps defined.

How were Kubernetes certificates managed? cert?manager used Venafi issuers scoped by namespace or cluster to request and renew certificates. Ingress resources referenced Certificate objects, and renewals propagated automatically. Policies governed which issuer and template applied, and events were logged for evidence.

How did you protect private keys? Keys were generated and stored according to platform best practices, with HSM?backed CAs where applicable. No keys were exposed in logs or exports. Access to requests, templates, and installs followed role?based permissions, and endpoint APIs used scoped credentials.

How were wildcards and SAN sprawl controlled? Template policies limited wildcard issuance and SAN counts by domain category. Requests with broad scope required additional approvals and documented justification. Periodic reviews flagged certificates with unused or risky entries and proposed replacements with narrower scope.

What about internal vs. public certificates? Domain categories mapped to approved CAs and templates. External domains used public CAs under Venafi control; internal services used internal PKI. The same lifecycle steps and evidence applied, with different validation and approval tiers as policy required.