Hardcoded secrets sprawled across repos and CI; HashiCorp Vault and GitHub scanning moved access to short?lived tokens

Overview

A fintech company struggled with secrets sprawl: API keys and database credentials were hardcoded in repos, copied across CI jobs, and shared in long?lived cloud users. Rotations lagged, leaks surfaced in pull requests, and access was hard to trace to an owner or approval. Intelligex deployed HashiCorp Vault as the broker for application?to?application authentication with short?lived tokens, integrated GitHub secret scanning and push protection, and enforced policy?as?code for requests and rotations. Hardcoded secrets declined, pipelines became cleaner, and access requests carried approvals and lineage—while GitHub, CI/CD, Kubernetes, and cloud platforms remained in use.

Client Profile

• Industry: Financial technology (payments and lending)
• Company size (range): Multi?team engineering organization with shared platform and security functions
• Stage: GitHub for source control and Actions; Jenkins for legacy CI; Kubernetes for services; AWS and GCP for infrastructure; secrets scattered across repo files, environment variables, and cloud IAM users
• Department owner: IT & Infrastructure (Security Engineering and Platform/SRE)
• Other stakeholders: Application Engineering, Data Platform, DevOps/Build, Risk and Compliance, Legal, Internal Audit, Customer Support (for incident response)

The Challenge

Secrets lived in too many places. Teams kept API keys in GitHub Actions secrets, Jenkins creds, Kubernetes ConfigMaps, and sometimes in code or Dockerfiles. Cloud access relied on long?lived keys shared across services. When a leak was detected, rotations rippled through code, CI, and runtime with unclear ownership. Operations could not answer quickly who requested access, who approved it, and which systems used a given credential.

CI/CD pipelines were inconsistent. Some jobs pulled from a central vaulting pattern; others exported secrets as environment variables with broad scope and long lifetimes. Developers used personal tokens for local testing that became embedded in sample configs. Database connections used static users with wide privileges and rare rotations. Incident response for leaked keys consumed time because there was no single place to revoke or rotate with confidence.

Governance lagged the pace of delivery. Approval for sensitive secrets happened in chats or tickets without binding to issuance. Policies for who could request which path varied by team. Secret scanning existed but did not block pushes, and findings required manual triage. Audit asked for a clean inventory, owners, expiration, and access evidence; assembling that required combing through CI logs, pull requests, and cloud consoles.

Why It Was Happening

Root causes were the absence of a central broker and the reliance on static credentials. Teams defaulted to per?tool secret stores and hardcoded values because application identity was not standardized. Secret policies were documented rather than encoded, and issuance was decoupled from approvals. Git history retained credentials from old experiments, and push protection was not uniform across repos. Cloud access patterns were built on long?lived users because pipelines lacked a first?class mechanism to assume roles or mint short?lived tokens.

Ownership and timing were split. Platform owned Kubernetes, Security owned policies, developers owned pipeline code, and IAM teams owned cloud users. Without a shared lifecycle—request, approve, issue, rotate, revoke—leaks and drift accumulated, and remediation depended on who was on call.

The Solution

Intelligex implemented a secrets lifecycle anchored in HashiCorp Vault for short?lived access and GitHub scanning for leak prevention. Vault became the broker for database, cloud, and internal service credentials; applications authenticated via scoped methods (Kubernetes, AppRole, OIDC); and all secrets carried policy?as?code and ownership metadata. GitHub secret scanning and push protection stopped new leaks at commit time, and a remediation workflow cleaned up historical exposures. Approvals were bound to issuance, and rotations and revocations were automated. The design leveraged HashiCorp Vault, GitHub secret scanning and push protection (Secret scanning), and hardening practices such as GitHub Actions OIDC for cloud federation (GitHub Actions OIDC).

• Integrations: Vault auth via Kubernetes, AppRole, and OIDC; Vault secret engines for databases and cloud (for example, AWS/GCP dynamic creds); GitHub secret scanning with push protection; CI/CD (GitHub Actions, Jenkins) for token brokering; Kubernetes for sidecar/agentless fetch; SIEM for Vault audit logs; ITSM for approvals.
• Application identity: Kubernetes auth for pods by service account; AppRole or OIDC for jobs; workload identity mapped to Vault policies; no static keys embedded in code.
• Dynamic secrets: Short?lived database users and cloud credentials minted on demand; automatic expiration and rotation; scoped privileges per role.
• Policy?as?code: Vault policies and Sentinel rules encoded who can request which path, for how long, with what approval; reason codes and ownership required.
• CI/CD guardrails: Pipelines retrieved short?lived tokens just?in?time; environment variables minimized and scrubbed; GitHub Actions used OIDC to assume cloud roles rather than storing keys.
• Scanning and remediation: Repository?wide secret scanning and push protection; triage workflow to revoke, rotate, and remove historical exposures; false positive handling documented.
• Access request workflow: Catalog of secrets with owners and approvers; maker?checker path for sensitive scopes; time?bound exceptions; audit?ready evidence.
• Dashboards and lineage: Visibility into active leases, policy changes, requests and approvals, and secret usage by service; linkbacks to repos and pipelines.
• Security and key handling: HSM?backed backends where applicable; no secrets printed to logs; least?privilege defaults; role?based access to Vault paths and scanning alerts.

Implementation

• Discovery: Cataloged where secrets lived (repos, CI, Kubernetes, cloud IAM); inventoried high?risk services and data flows; reviewed incident timelines tied to leaked keys; mapped existing GitHub scanning and CI patterns; gathered audit requirements for approvals and traceability.
• Design: Defined workload identity patterns (Kubernetes, AppRole, OIDC); authored Vault policies and roles with ownership and reason codes; selected secret engines (databases, cloud, generic KV) and lease durations; designed CI templates for token fetching and scrubbing; planned scanning rules, push protection rollout, and remediation workflows; mapped ITSM approvals to Vault issuance.
• Build: Deployed Vault with audit devices; configured Kubernetes auth, AppRole, and OIDC; enabled database and cloud secret engines; created policies and Sentinel rules; built CI templates and reusable actions to fetch and revoke tokens; enabled GitHub secret scanning and push protection across orgs; integrated ITSM for requests and approvals; wired SIEM dashboards.
• Testing/QA: Ran in shadow mode: pipelines fetched short?lived tokens alongside existing secrets; validated lease expirations and forced revocations; exercised database and cloud dynamic creds in non?production; seeded scanning with safe test secrets to verify detection and push protection; piloted remediation on a subset of repos.
• Rollout: Switched CI jobs and services to Vault?backed tokens by service area; disabled long?lived users in phases; turned on push protection org?wide; retained legacy secrets as a controlled fallback until stability; enforced maker?checker for sensitive paths after training.
• Training/hand?off: Delivered sessions for developers, build engineers, and SREs on vaulting patterns, OIDC federation, and scanning triage; provided code snippets and CI templates; updated SOPs for requesting, rotating, and revoking secrets; transferred ownership of policies, engines, and dashboards to Security Engineering under change control.
• Human?in?the?loop review: Established a governance forum for exception requests, policy updates, and recurring scanning findings; decisions recorded with rationale and effective dates; improvements fed back into templates and rules.

Results

Hardcoded secrets declined as services and pipelines adopted short?lived tokens. Applications authenticated using workload identity, Vault minted time?bound credentials on demand, and revocation stopped access without code changes. CI templates removed secrets from long?lived environment variables, and cloud pipelines used OIDC federation instead of static keys.

Governance and audit posture strengthened. Each access request carried owner, rationale, and approval; policies defined scope and duration; and Vault audit logs tied usage to identities and pipelines. GitHub scanning and push protection caught exposures early, with a defined path to revoke and rotate. The result was cleaner pipelines, fewer surprise leaks, and a traceable lifecycle from request to expiration.

What Changed for the Team

• Before: Keys lived in repos, CI vars, and config files. After: Applications and jobs fetched short?lived tokens from Vault at runtime.
• Before: Cloud access used shared long?lived users. After: Pipelines assumed roles via OIDC or minted scoped, expiring creds.
• Before: Approvals happened in chats. After: Requests flowed through a catalog with maker?checker and ownership metadata.
• Before: Secret scanning created noisy follow?ups. After: Push protection blocked risky commits and triage linked to revoke/rotate steps.
• Before: Rotations required broad code changes. After: Revocations and rotations happened centrally with minimal code impact.
• Before: Audit asked “who used this and why?”. After: Dashboards showed leases, policies, and evidence tied to identities and tickets.

Key Takeaways

• Broker access, don’t copy secrets; use Vault to mint short?lived credentials on demand.
• Adopt workload identity; prefer Kubernetes, AppRole, or OIDC over embedded keys.
• Make policy executable; encode who can request what, for how long, with approvals and rationale.
• Shift left on leaks; enable secret scanning and push protection, and bind findings to revoke/rotate workflows.
• Harden CI templates; fetch at job start, scrub env vars, and use OIDC for cloud access.
• Integrate, don’t replace; keep GitHub, CI/CD, Kubernetes, and clouds, and add a governed secrets lifecycle.

FAQ

What tools did this integrate with? Vault served as the broker and policy engine (HashiCorp Vault). GitHub secret scanning and push protection prevented new leaks (GitHub secret scanning). Pipelines used GitHub Actions and Jenkins; cloud access leveraged GitHub Actions OIDC to assume roles without static keys (Actions OIDC). Kubernetes authenticated to Vault via the native auth method; databases and cloud platforms consumed dynamic credentials.

How did you handle quality control and governance? Policies lived in code with owners and rationale, and sensitive paths required maker?checker approvals. Every issuance, lease renewal, revocation, and policy change was logged. Requests captured reason codes and ownership, and time?bound exceptions were reviewed on a schedule. Scanning alerts flowed into a triage workflow that revoked and rotated secrets with evidence attached.

How did you roll this out without disruption? Pipelines fetched short?lived tokens alongside existing secrets in shadow mode. After validating expirations and revocations, jobs switched to Vault?backed tokens by service area. Push protection began with warnings, then moved to blocking. Long?lived users were disabled in phases, and a documented fallback remained available during early cycles.

How did applications authenticate without hardcoded keys? Kubernetes workloads used service account tokens with Vault’s Kubernetes auth to obtain scoped tokens. CI jobs used AppRole or OIDC to authenticate and fetch secrets at runtime. Where feasible, GitHub Actions assumed cloud roles via OIDC instead of storing cloud keys.

How were leaked secrets in history handled? GitHub secret scanning identified exposures; the remediation playbook revoked the credential at the source (cloud, database, API), rotated it, and opened a pull request to remove and invalidate references. For high?risk findings, push protection blocked commits until remediation steps were captured.

What changed for developer workflows and local testing? Templates and SDK snippets showed how to request tokens from Vault for local runs with least privilege and limited lifetime. Sample configs no longer contained real secrets. Developers could request temporary access through the catalog with approvals and clear expirations.

How did you handle databases and cloud credentials specifically? Vault’s database engines created short?lived users per service with scoped roles; leases expired automatically, and revocation removed users. Cloud access used dynamic creds from Vault or OIDC?based role assumption in CI. Policies defined which services could request which roles, and rotations were automated.