Secrets Management: Securely Managing Passwords, API Keys, and Certificates

TL;DR

Secrets are all confidential access information: passwords, API keys, database credentials, SSH keys, TLS certificates, encryption keys, and service account tokens.
The most common mistakes: secrets in source code, in Git repositories (including the history), in environment variables (visible in process listings and logs), or in unencrypted configuration files.
Centralized secrets management systems like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault store secrets encrypted, control access, and log every usage.
Secrets rotation is just as important as secure storage. Secrets that are never rotated give an attacker unlimited access once they have been obtained.
For smaller teams and projects, tools like SOPS (encrypted files in Git) or Doppler (cloud-based secrets management) are pragmatic alternatives to Vault.

What Secrets Are and Why They Need Special Protection

Secrets are confidential information that enables access to systems, services, and data. These include database passwords and connection strings, API keys for external services (payment providers, cloud APIs, email providers), SSH keys for server access, TLS/SSL certificates and their private keys, encryption keys for data encryption, service account credentials for automated processes, OAuth client secrets and token signing keys, and backup encryption passwords.

Secrets are particularly worth protecting because they provide direct access to your infrastructure. A compromised database password gives the attacker full access to all data. A compromised API key enables actions on behalf of your organization. This topic directly relates to your authorization concept. A compromised SSH key opens the door to your servers.

Unlike a phishing attack that compromises a single user account, a compromised service secret often affects the entire application or infrastructure. A single exposed database connection string can grant access to all customer data.

Where Secrets Are Typically Stored Incorrectly

Most secrets compromises happen not through sophisticated attacks but through basic storage mistakes.

In Source Code

The classic mistake: a developer writes the API key directly into the code. const API_KEY = "sk_live_abc123...". Even if the key is removed later, it remains in the Git history. Anyone with access to the repository (and that can be many people, especially with open-source projects) can extract the key from the history.

GitHub has been automatically scanning public repositories for exposed secrets since 2022 (GitHub Secret Scanning) and notifies the token issuer. But for private repositories, self-hosted Git servers, or other platforms, this protection is often missing.

In Environment Variables

Environment variables are better than secrets in code, but not ideal. Environment variables appear in /proc/[pid]/environ on Linux systems and can be read by any process of the same user. They show up in process listings. They are written into crash dumps and debug logs. Docker exposes them via docker inspect. Kubernetes pods show them via kubectl describe pod.

Environment variables are an acceptable intermediate step on the way to a real secrets management system, but not a permanent solution for sensitive production secrets.

In Configuration Files

Unencrypted configuration files (config.yaml, .env, application.properties) with database passwords and API keys are a common pattern. The problem: these files are often accidentally committed to Git (despite .gitignore, which is easily forgotten), placed on servers with overly broad file permissions, or backed up in archive files where they are less protected.

In Deployment Tools

CI/CD pipelines need access to secrets to deploy applications. Jenkins credentials, GitLab CI variables, GitHub Actions secrets store this information. If the pipeline configuration is not carefully protected, developers can exfiltrate secrets through manipulated pipeline steps (e.g., echo $SECRET in a build log).

Principles of Secrets Management

Centralization

Instead of scattering secrets across different locations (an .env file here, a Jenkins credential there, an encrypted Ansible Vault file elsewhere), you collect all secrets in a central location. This central location is your single source of truth for secrets. Applications and systems obtain their secrets exclusively from there.

Encryption

Secrets are always stored encrypted, both at rest (on disk) and in transit (during transmission). Even if an attacker gains access to the storage of the secrets management system, they cannot read the secrets.

Access Control

Not every application and not every developer needs access to all secrets. The principle of least privilege applies here too: the production application has access to the production database passwords but not to development secrets. The developer has access to development secrets but not to production secrets. The CI/CD pipeline has access to deploy credentials but not to backup encryption passwords.

Auditing

Every access to a secret is logged: who read which secret when? This enables traceability and detection of suspicious access patterns (e.g., a developer account reading production secrets at 3 AM).

Rotation

Secrets have a limited lifespan. They are regularly rotated (replaced with new ones) to limit the time window during which a compromised secret is usable. Ideally, rotation is automated and requires no manual intervention.

Tools Compared

HashiCorp Vault

Vault is the de facto standard for secrets management in medium and large environments. It is a centralized service that stores secrets, controls access, and logs usage.

Feature set: Encrypted storage of key-value secrets, dynamic secrets (Vault generates short-lived database credentials on demand and automatically revokes them), encryption as a service (applications can encrypt and decrypt data via the Vault API without knowing the key), PKI (Vault can issue TLS certificates and automatically rotate them), transit engine (encrypts data without storing it), and identity-based access control.

Dynamic secrets are a particularly powerful feature. Instead of storing a static database password in Vault, you configure Vault to create a new, short-lived database user with each request. This user has a lifespan of, say, 1 hour. After expiration, it is automatically deleted. If a secret is compromised, it is only valid for a short time.

Deployment options: Self-hosted (open source, free), HCP Vault (managed cloud service from HashiCorp, paid). Self-hosted requires operational overhead (high availability, backup, updates) but offers full control.

Suitable for: Organizations with multiple applications and environments that need a centralized, powerful secrets management solution. Environments with Kubernetes, where Vault is integrated via the Vault Agent Injector or Vault CSI Provider.

AWS Secrets Manager

AWS Secrets Manager is a managed AWS service for storing and rotating secrets.

Feature set: Encrypted storage of secrets (AES-256 via AWS KMS), automatic rotation of secrets (with Lambda functions), native integration with AWS services (RDS, Redshift, DocumentDB), versioning of secrets, and fine-grained access control via IAM policies.

Cost: $0.40 per secret per month + $0.05 per 10,000 API calls. For 50 secrets: approximately $20 per month.

Suitable for: Organizations that primarily use AWS and prefer a managed solution without operational overhead.

Azure Key Vault

Azure Key Vault is Azure's counterpart to AWS Secrets Manager.

Feature set: Storage of secrets, keys, and certificates, HSM-backed key storage (Hardware Security Module), integration with Azure services (App Service, Functions, AKS), access logging via Azure Monitor.

Suitable for: Organizations that primarily use Azure and Microsoft 365.

SOPS (Secrets OPerationS)

SOPS is an open-source tool by Mozilla that selectively encrypts files (YAML, JSON, ENV, INI). It encrypts only the values in a file, not the key names. This keeps the files versionable and diffable in Git.

Feature set: Selective encryption of file values, support for various encryption backends (AWS KMS, GCP KMS, Azure Key Vault, age, PGP), Git-friendly (encrypted files can be normally committed and merged), no central service required.

Workflow: You create a configuration file with secrets, encrypt it with sops --encrypt, and commit the encrypted file to Git. During deployment, the CI/CD pipeline decrypts the file with sops --decrypt and provides the secrets to the application.

Suitable for: Smaller teams and projects that do not want to operate a central Vault infrastructure but still want to securely version secrets in Git.

Doppler

Doppler is a cloud-based secrets management service designed for developer-friendliness.

Feature set: Web dashboard and CLI for secrets management, environment separation (development, staging, production), integrations with CI/CD systems, cloud providers, and frameworks, secrets injection into applications (via CLI wrapper or SDKs), audit logging, and versioning.

Cost: Free for up to 5 users and unlimited secrets. Team plan starting at $4 per user per month.

Suitable for: Small to medium teams looking for a straightforward, cloud-native solution.

Secrets Management in the CI/CD Pipeline

As part of the Secure Development Lifecycle, the CI/CD pipeline is a particularly sensitive area because it needs access to secrets (deploy credentials, container registry access, database migration passwords), pipeline logs are frequently visible to many developers, and pipeline configurations are often reviewed less strictly than application code.

Best Practices for Pipeline Secrets

Never output secrets in pipeline logs. Most CI/CD systems mask known secrets in logs, but masking is not always reliable. Regularly check your logs for exposed secrets.

Use short-lived credentials. Instead of storing static deploy credentials in the pipeline, use OIDC-based authentication (GitHub Actions can authenticate directly with AWS, Azure, or GCP without static credentials) or short-lived tokens that Vault issues on demand.

Provide secrets only to the steps that need them. In GitHub Actions, you can limit secrets per job or step. Not every step needs database admin access.

Review pipeline configurations as code. Changes to pipeline configuration (.github/workflows, .gitlab-ci.yml, Jenkinsfile) should go through the same review process as application code.

Secrets Rotation: Why and How

Static secrets that are never rotated are a persistent problem. If an API key is compromised and you never rotate it, the attacker has unlimited access. If you rotate the key every 90 days, the attacker has at most 90 days of access (assuming you detect the compromise in time and rotation occurs on schedule).

Automatic Rotation

The most effective rotation is automatic. AWS Secrets Manager can automatically rotate RDS database passwords: a Lambda function generates a new password, updates it in the database and in Secrets Manager. HashiCorp Vault can dynamically create short-lived database users and revoke them after expiration. Let's Encrypt automatically renews TLS certificates every 90 days.

Automatic rotation eliminates manual effort and the risk that rotation is forgotten. However, it requires that all applications using the secret automatically pick up the change. Applications that read the database password from a configuration file at startup and never update it again will not work with automatic rotation.

Rotation Intervals

Different secret types have different recommendations.

Database passwords: 90 days, preferably 30 days, best of all dynamic secrets with hourly or minute-level lifespans.

API keys: 90 to 180 days. Immediate rotation if compromise is suspected.

SSH keys: 180 to 365 days. Use SSH certificates with short lifespans instead of long-lived keys.

TLS certificates: Let's Encrypt renews every 60 days (with 90-day validity). For internal certificates: 90 to 365 days.

Encryption keys: 365 days or longer, with key versioning (old data remains decryptable with the old key, new data is encrypted with the new key).

Secrets in Emergency Situations

Secrets management must also work in emergencies. If your Vault server fails or the cloud secrets manager is unreachable, applications still need to be able to start.

Break-Glass Procedures

Have an emergency procedure ready for critical secrets. Encrypted copies of the most critical secrets (database admin password, cloud admin credentials) in a physical safe. A sealed envelope with the Vault unseal key or master key for the secrets manager. Documentation of the break-glass procedure: who may open the envelope, under what circumstances, and what notifications must occur.

High Availability

If secrets management is critical for operations (and it usually is), the system itself must be highly available. Vault supports HA deployment with various backends (Consul, Raft). AWS Secrets Manager and Azure Key Vault are inherently highly available as managed services.

Secrets Management and Compliance

ISO 27001

Annex A, Control A.5.17 (Authentication information), requires that authentication information is controlled, managed, and protected during assignment. A secrets management system with access control, encryption, and audit logging fulfills this requirement.

NIS2

NIS2 Article 21 generally requires appropriate security measures. The secure management of access credentials falls under the requirements for access control and cryptography.

DSGVO (GDPR)

If secrets enable access to personal data (database passwords, API keys for services that process personal data), they are part of the technical and organizational measures (TOMs) under Article 32 GDPR. The secure management of these secrets is an obligation.

Getting Started Pragmatically

If you have no secrets management today, do not start with a Vault Enterprise installation. Start pragmatically.

Step 1: Create an inventory. Find out where all your secrets are. Search Git repositories for hardcoded secrets (tools: TruffleHog, git-secrets, gitleaks). Check .env files, configuration files, and CI/CD variables.

Step 2: Fix the worst cases. Remove secrets from Git repositories and rotate them (the old secrets are compromised even if you delete the commit). Move hardcoded secrets into configuration files or environment variables.

Step 3: Introduce a simple tool. For small teams, SOPS or Doppler is sufficient. For AWS environments: AWS Secrets Manager. For larger or heterogeneous environments: HashiCorp Vault. In ISMS Lite, you document your secrets management policy and rotation intervals so that you can demonstrate for audits at any time how your credentials are managed.

Step 4: Introduce rotation. Start with the most critical secrets (database admin passwords, cloud admin credentials) and expand gradually.

Step 5: Install pre-commit hooks. Install git-secrets or gitleaks as a pre-commit hook in all repositories. This prevents secrets from being accidentally committed.

Common Mistakes in Secrets Management

Not rotating secrets after removing them from Git. If a secret has ever been in a Git commit, it remains in the history, even if you overwrite the commit. The only safe measure is to consider the secret compromised and rotate immediately. Many teams remove the commit but do not rotate because they shy away from the effort.

Using the same password for all environments. The database password in the development environment is identical to the one in production because it is "easier." If a developer's laptop is compromised, the attacker has direct access to the production database. Always use separate secrets per environment.

No monitoring of secret access. Vault and other tools log every access, but nobody looks at the logs. A compromised service account that suddenly queries secrets it normally does not need goes unnoticed. Set up alerts for unusual access patterns.

Forgetting backup secrets. The backup encryption passwords and the Vault unseal keys are not securely documented anywhere. If the Vault server fails and nobody knows the unseal key, all secrets are lost. Keep these meta-secrets in a physical safe, separate from the digital system.

Overly broad access rights to the secrets store. All developers have read access to all secrets because fine-grained configuration seemed too time-consuming. This violates the principle of least privilege and significantly increases the attack surface. Invest the time in a clean access structure — it pays off.