Managing Secrets and API Keys in Python Projects (.env Guide)

Image by Author

Introduction to Keeping Secrets

Storing sensitive information such as API keys, database passwords, or tokens directly within Python code presents significant security risks. A leak of these secrets could lead to system breaches, loss of trust, and severe financial or legal repercussions for an organization. To prevent this, secrets should be externalized, meaning they should never be hardcoded or committed to version control. A widely accepted practice involves storing secrets in environment variables, keeping them entirely separate from the codebase. While manual environment variable setup is possible, using a single .env file for local development offers a convenient solution.

This article outlines seven practical methods for managing secrets in Python projects, complete with code examples and discussions on common pitfalls.

Technique 1: Using a .env File Locally (And Loading it Safely)

A .env file is a local text file containing KEY=value pairs, which should not be committed to version control. It enables the definition of environment-specific settings and secrets for development purposes. A typical project structure might look like this:

my_project/
  app/
    main.py
    settings.py
  .env              # NOT committed – contains real secrets
  .env.example      # committed – lists keys without real values
  .gitignore
  pyproject.toml

Your actual secrets are stored in the local .env file, for example:

# .env (local only, never commit)
OPENAI_API_KEY=your_real_key_here
DATABASE_URL=postgresql://user:pass@localhost:5432/mydb
DEBUG=true

Conversely, .env.example serves as a template that is committed to the repository, informing other developers about the necessary keys:

# .env.example (commit this)
OPENAI_API_KEY=
DATABASE_URL=
DEBUG=false

To prevent accidental commits of sensitive files, add patterns to your Git ignore file:

.env
.env.*

This ensures that your secret .env file is never inadvertently checked into version control. In Python, the standard approach involves using the python-dotenv library to load the .env file at runtime. For instance, in app/main.py, one might include:

# app/main.py
import os
from dotenv import load_dotenv

load_dotenv()  # reads variables from .env into os.environ

api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
    raise RuntimeError("Missing OPENAI_API_KEY. Set it in your environment or .env file.")

print("App started (key loaded).")

The load_dotenv() function automatically locates the .env file in the working directory and populates os.environ with its key=value pairs, unless a variable is already defined. This method helps avoid common errors like committing .env files or sharing them insecurely, while providing a clean and reproducible development environment. It allows for seamless transitions between different machines or development setups without altering code, ensuring local secrets remain protected.

Technique 2: Read Secrets from the Environment

Some developers might use placeholders like API_KEY=”test” directly in their code or assume environment variables are always set during development. While this might function on a developer’s machine, it can lead to failures in production. If a secret is missing, a placeholder could be used, posing a security risk. Instead, secrets should always be retrieved from environment variables at runtime. Python offers os.environ or os.getenv for securely accessing these values. For example:

def require_env(name: str) -> str:
    value = os.getenv(name)
    if not value:
        raise RuntimeError(f"Missing required environment variable: {name}")
    return value

OPENAI_API_KEY = require_env("OPENAI_API_KEY")

This practice causes an application to fail quickly upon startup if a required secret is absent, which is significantly safer than proceeding with a missing or default value.

Technique 3: Validate Configuration with a Settings Module

As projects expand, numerous scattered calls to os.getenv can become disorganized and prone to errors. Utilizing a settings class, such as Pydantic’s BaseSettings, centralizes configuration, performs type validation, and loads values from both .env files and the environment. Consider this example:

# app/settings.py
from pydantic_settings import BaseSettings, SettingsConfigDict
from pydantic import Field

class Settings(BaseSettings):
    model_config = SettingsConfigDict(env_file=".env", extra="ignore")

    openai_api_key: str = Field(min_length=1)
    database_url: str = Field(min_length=1)
    debug: bool = False

settings = Settings()

Then, within the application:

# app/main.py
from app.settings import settings

if settings.debug:
    print("Debug mode on")
api_key = settings.openai_api_key

This approach helps prevent errors like mistyped keys, incorrect type parsing (e.g., “false” versus False), or redundant environment lookups. Employing a settings class guarantees that the application will fail promptly if secrets are missing, thereby preventing “works on my machine” issues.

Technique 4: Using Platform/CI secrets for Deployments

During deployment to a production environment, copying your local .env file is not recommended. Instead, leverage the secret management features provided by your hosting or CI platform. For instance, when using GitHub Actions for CI, secrets can be stored encrypted within the repository settings and then injected into workflows. This method ensures that the CI or cloud platform supplies the actual secret values at runtime, keeping them out of code and logs.

Technique 5: Docker

When working with Docker, it is crucial to avoid embedding secrets directly into images or using plain ENV instructions. Docker and Kubernetes offer dedicated secret mechanisms that are more secure than standard environment variables, which could potentially be exposed through process listings or logs. For local development, a .env file combined with python-dotenv is effective, but for production containers, secrets should be mounted or managed using Docker’s secret features. Refrain from using ENV API_KEY=… in Dockerfiles or committing Compose files that contain secrets. This practice reduces the risk of secrets being permanently exposed in container images and simplifies key rotation.

Technique 6: Adding Guardrails

Since human error is inevitable, automating secret protection is essential. Tools like GitHub push protection can block commits that contain secrets, while CI/CD secret-scanning tools such as TruffleHog or Gitleaks can identify leaked credentials before they are merged. Developers often overlook security best practices in favor of speed, leading to accidental commits. Guardrails are designed to prevent leaks from entering the repository, significantly enhancing the safety of using .env files and environment variables across development and deployment stages.

Technique 7: Using a Real Secrets Manager

For larger applications, implementing a dedicated secrets manager like HashiCorp Vault, AWS Secrets Manager, or Azure Key Vault is advisable. These systems provide granular control over who can access secrets, log all access attempts, and automate key rotation. Without such a manager, teams might inadvertently reuse passwords or neglect to rotate them, which introduces considerable risk. A secrets manager centralizes control, simplifies rotation, and safeguards production systems even if a developer’s computer or local .env file is compromised.

Wrapping Up

Ensuring the security of secrets goes beyond merely following rules; it involves establishing a workflow that makes projects secure, maintainable, and adaptable across various environments. To assist with this, a checklist for Python projects is provided:

1. .env is included in .gitignore (never commit actual credentials).
2. .env.example exists and is committed with empty values.
3. Code retrieves secrets only via environment variables (e.g., os.getenv, a settings class).
4. The application fails fast with a clear error if a required secret is missing.
5. Different secrets are used for development, staging, and production environments (never reuse the same key).
6. CI and deployment processes utilize encrypted secrets (e.g., GitHub Actions secrets, AWS Parameter Store).
7. Push protection or secret scanning is enabled on repositories.
8. A rotation policy is in place (keys are rotated immediately if leaked and regularly otherwise).