Managing Secrets in Production

Production applications depend on many secrets, including database passwords, API keys, private tokens, encryption keys and service credentials. These values are different from ordinary configuration because exposing them can give attackers direct access to systems, data or infrastructure.

Managing secrets securely is therefore an important part of production engineering. Secrets should be stored separately from application source code, accessed only by the services and people that need them, and rotated regularly when appropriate.

What Is a Production Secret?

A production secret is confidential information required by an application or infrastructure component to authenticate, authorize access or perform a protected operation. Common examples include database credentials, API keys, cloud provider credentials, private certificates and signing keys.

Why Production Secrets Matter

A leaked production secret can have consequences far beyond the application itself. Depending on the credential, an attacker may access databases, modify cloud resources, read private data, send requests through paid services or impersonate trusted systems.

• Protect customer and application data.
• Prevent unauthorized infrastructure access.
• Reduce the impact of credential leaks.
• Limit access between services.
• Support secure deployment workflows.
• Make credential rotation easier.

Never Hardcode Secrets

Production credentials should not be embedded directly in application source code. Hardcoded values can accidentally become part of Git history, source archives, logs, build artifacts or publicly accessible repositories.

const databasePassword = "production-password";
const apiKey = "sk-production-example";

Use Environment Variables Carefully

Environment variables are commonly used to provide secrets to applications during deployment. They keep sensitive values outside the source code and allow the same application build to run with different configuration in different environments.

DATABASE_URL=postgresql://user:password@database:5432/app
API_KEY=production-api-key
JWT_SECRET=production-signing-secret

Environment variables are useful, but they are not automatically a complete secrets management solution. Depending on the platform, environment values may be visible to administrators, deployment systems, debugging tools or processes with sufficient operating system privileges.

Development vs Production Secrets

Development and production environments should use different credentials. Reusing production passwords or API keys during local development increases the chance that a production credential will appear in a developer machine, local configuration file or debugging output.

Keep .env Files Out of Git

Local .env files often contain credentials and should normally be excluded from version control. A project can commit an example configuration containing placeholder values while keeping real credentials on the developer's machine or deployment platform.

.env
.env.local
.env.production
.env.*.local

Use an Example Environment File

An example environment file documents which configuration values an application expects without exposing the real credentials. It can contain variable names and safe placeholder values that developers can replace with local configuration.

DATABASE_URL=
API_KEY=
JWT_SECRET=
REDIS_URL=

Dedicated Secret Management Systems

Larger production environments often use dedicated secret management systems instead of storing sensitive values directly in deployment configuration. These systems can provide encrypted storage, access policies, auditing, versioning and controlled secret retrieval.

Principle of Least Privilege

Applications and team members should receive only the secrets and permissions required to perform their jobs. A service that only needs read access to one database should not receive administrator credentials for an entire cloud account.

• Give services only the credentials they require.
• Avoid shared administrator credentials.
• Separate credentials by environment.
• Restrict access to production secrets.
• Review permissions regularly.

Separate Secrets by Service

Different production services should preferably use different credentials. If one service is compromised, isolated credentials can prevent an attacker from automatically gaining access to unrelated systems.

Secret Rotation

Secret rotation means replacing an existing credential with a new one. Regular rotation limits how long a leaked or compromised secret remains useful and can also satisfy organizational security requirements.

Old credential
      ↓
Create new credential
      ↓
Deploy application with new credential
      ↓
Verify production traffic
      ↓
Revoke old credential

Handling a Leaked Secret

If a production secret is exposed, assume that unauthorized parties may have obtained it. The correct response is to revoke or rotate the credential as quickly as possible, investigate where it was exposed and review related access logs.

• Identify the exposed credential.
• Revoke or rotate it immediately.
• Deploy the replacement credential.
• Inspect relevant access logs.
• Determine how the secret was exposed.
• Remove the secret from future accessible locations.
• Review and improve the secret management process.

Secrets in CI/CD Pipelines

Continuous integration and deployment systems often need credentials to build, test and deploy applications. These values should be stored using the CI/CD platform's protected secret mechanism rather than being written directly into workflow files.

steps:
  - name: Deploy application
    env:
      API_KEY: ${secrets.API_KEY}
    run: npm run deploy

CI/CD systems should also prevent secrets from appearing in command output. Commands that print environment variables or authentication headers can accidentally expose credentials in build logs.

Avoid Secrets in Logs

Application logs, monitoring systems and error tracking platforms often have long retention periods and may be accessible to many people. Secrets should therefore never be logged intentionally, even when debugging production problems.

console.log({
  userId,
  apiKey,
  authorizationHeader,
});

Masking Sensitive Values

When sensitive values must appear in diagnostic output, applications can replace most of the value with masking characters. Masking is useful for debugging, but it should not replace proper secret storage and access controls.

Original:
sk_live_1234567890abcdef

Masked:
sk_live_************cdef

Secrets in Docker Applications

Containerized applications require special care because secrets can accidentally become part of images, build layers, environment dumps or container configuration. A secret required only at runtime should generally not be baked into a reusable container image.

• Do not hardcode production credentials in Dockerfiles.
• Avoid copying .env files into images.
• Provide runtime configuration during deployment.
• Restrict access to container environments.
• Review image layers for accidentally included secrets.

Secrets in Kubernetes

Kubernetes provides Secret resources for storing values such as passwords, tokens and keys. However, Kubernetes Secrets should not automatically be considered equivalent to a complete enterprise secret management system. Access permissions, encryption at rest and cluster security still matter.

apiVersion: v1
kind: Secret
metadata:
  name: database-credentials
type: Opaque
stringData:
  username: app
  password: change-me

Encrypt Secrets at Rest

Secret management systems and infrastructure platforms should protect stored credentials using encryption at rest. Encryption reduces the risk of exposing plaintext values if the underlying storage is accessed without authorization.

Encryption at rest should be combined with access control, authentication, auditing and secure key management. Encryption alone does not prevent an authorized but compromised application from retrieving a secret.

Access Auditing

Production environments should provide visibility into who or which services access sensitive credentials. Audit records can help identify suspicious access, investigate incidents and verify that permissions match the intended architecture.

Avoid Sharing Production Credentials

Production credentials should not be distributed through chat messages, email, screenshots or shared documents. Every additional copy increases the number of places where a secret can be exposed or forgotten.

Use Short-Lived Credentials When Possible

Short-lived credentials reduce the useful lifetime of stolen authentication material. Temporary tokens, workload identities and automatically issued credentials can be preferable to permanent passwords when the infrastructure supports them.

Common Mistakes

Most secret management failures are caused by operational mistakes rather than a lack of sophisticated security technology. Small configuration shortcuts can create long-lasting exposure when production credentials are involved.

• Committing secrets to Git repositories.
• Using the same credentials in development and production.
• Sharing production credentials through chat or email.
• Logging API keys or authorization headers.
• Building secrets directly into container images.
• Using administrator credentials for application services.
• Never rotating long-lived credentials.
• Leaving unnecessary users with production access.

Best Practices

• Keep production secrets outside application source code.
• Use separate credentials for each environment.
• Apply least-privilege access policies.
• Use dedicated secret management when appropriate.
• Rotate credentials regularly or after exposure.
• Prevent secrets from appearing in logs.
• Protect CI/CD credentials with platform secret storage.
• Audit access to sensitive credentials.
• Use short-lived credentials when practical.
• Remove unnecessary production access.

Production Secret Management Checklist

Frequently Asked Questions

Helpful Security Tools

A Secret Generator creates strong random values for passwords, tokens and other credentials, a Kubernetes Secret Generator creates Kubernetes Secret manifests, an Environment Variable Generator helps build environment configuration, a dotenv Encryptor protects sensitive dotenv files, and a Secret Mask Generator creates masked representations of sensitive values for safer logs and debugging output.

Conclusion

Managing secrets in production is a combination of secure storage, controlled access, careful deployment practices and reliable credential rotation. Production secrets should remain outside source code, be separated by environment and service, and be accessible only to the systems and people that actually need them. By preventing secrets from entering repositories and logs, using appropriate secret management systems and responding quickly to exposed credentials, teams can significantly reduce the impact of credential leaks and build more secure production environments.