Survival Guide For Python Template

Building Python Templates That Don't Fall Apart

Most Python project templates you find online are either too minimal to be useful or so bloated with opinions that they slow you down. The Survival Guide For Python Template isn't about finding one perfect starting point. It's about understanding what actually breaks when you ship code, then architecting your project structure around those failure points before they happen. I spent years writing Python scripts that worked fine locally and then failed in production because I didn't think about configuration drift, logging, or error handling across environments. Eventually I stopped reinventing the wheel for every project and built a consistent template structure. What follows is that structure and the reasons each piece exists.

Survival Guide For Python Template

Start with a directory layout that separates concerns early. A flat project with everything in the root directory works for a weekend hack. It does not work for anything that needs to be maintained past two weeks. Here is the structure I use and why: A top-level pyproject.toml handles dependency management. Pipenv or poetry.lock files are optional but pinning versions explicitly prevents the dependency resolution hell that eats weekends. I've seen projects derail for three days because a transitive dependency updated and silently changed behavior. Pinning costs nothing and prevents that.

The src/ directory contains your actual code. This is the source package layout, and it matters more than people admit. When your code lives inside src/, imports become explicit and you avoid the common mistake of accidentally importing from the wrong path during testing. I learned this the hard way when a test suite kept passing locally but failing in CI because the project root was on sys.path and shadowed an installed package. Moving to src/ layout eliminated that entire class of bug. A tests/ directory at the root level, separate from src/. Keep them apart. Mixing test code with production code creates import ambiguity and makes it too easy to ship test-only helpers into production environments. A config/ directory for environment-specific settings. Never hardcode API keys, database URLs, or feature flags. Use a .env file loaded through python-dotenv in development, and configure secrets through your deployment platform in production. I once deployed a scraper that logged database credentials to stdout because I forgot to move the config extraction out of the main module. That one mistake meant every log aggregator in our stack had the password. It took me four hours to rotate all credentials and another six to audit access logs.

Logging configuration goes in a dedicated module, not scattered across files. Set up logging at the project entry point with a consistent format that includes timestamps, log levels, and the module name. The default Python logging setup prints to stderr with minimal context. That is fine for quick scripts. It is inadequate for anything that needs debugging after hours. Error handling deserves its own module. Define custom exception classes that inherit from a base application exception. This lets you distinguish between expected failures (like a rate limit being hit) and unexpected crashes (like a null pointer dereference in your own code). A blanket except clause that catches Exception and continues silently is the fastest way to hide bugs until they surface in the worst possible moment.

Dependency Management

Use requirements.in files for direct dependencies and requirements.txt for resolved versions. This distinction is standard in the Python ecosystem but routinely ignored. Keeping them separate makes it clear which packages you directly need versus which ones were pulled in transitively. Pin your major dependencies. Python package updates are not always backward compatible. The jump from requests 2.25 to 2.26 broke my HTTP retry logic because the connection pool behavior changed subtly. If I had pinned to ~=2.25, that update would never have reached my system automatically. Regularly run pip-audit or safety check against your dependency tree. Vulnerabilities in transitive dependencies are real and often overlooked. I found a known CVE in a dependency that only my dependency depended on, and it would have gone unnoticed without the audit step.

Testing Strategy

pytest is the default choice and for good reason. It requires less boilerplate than unittest and the plugin ecosystem covers most needs. Structure your tests with a conftest.py file for shared fixtures. This keeps test code DRY without pushing fixture logic into individual test files where it gets lost. Parameterized tests save significant time when you have repetitive test cases. Instead of writing five nearly identical test functions for five input variations, use pytest.mark.parametrize to declare them in one place. This is especially valuable for validation logic where edge cases multiply quickly. Mock external dependencies. Database calls, HTTP requests, and file I/O should never happen during unit tests. Use pytest-mock or the standard unittest.mock library to intercept these calls. A test that hits a real API is a flaky test, and flaky tests destroy trust in your test suite faster than anything else.

I once spent two days debugging a failing test that turned out to be caused by a race condition in an integration test hitting a live service. The test passed 95 percent of the time and failed randomly. Mocking the external dependency made it pass deterministically and reduced test execution from eight minutes to forty seconds.

Configuration Management

Use pydantic-settings for configuration validation. It validates types and provides defaults at runtime. A configuration value that should be an integer but arrives as a string will raise a clear error at startup instead of causing a cryptic failure deep in your logic. This shifts the failure point from production to initialization, which is infinitely easier to debug. Environment-specific configuration files (config/dev.yaml, config/prod.yaml) let you override defaults per environment without code changes. Merge them in your settings module so that prod settings take precedence over dev settings when running in production. I've seen teams hardcode environment checks throughout their codebase, which makes refactoring painful and introduces inconsistencies.

Logging Best Practices

Configure logging before your application does anything meaningful. If you set up logging inside individual modules, you may miss events that occur during module import. Initialize logging in your main entry point, before any application logic runs. Use structured logging where possible. JSON-formatted logs are easier to parse in log aggregation systems than plain text. Tools like structlog or even manual JSON serialization in your log handler make this straightforward. Unstructured log output is fine for local development. It becomes a maintenance burden as soon as you need to search through thousands of log lines to find a specific error pattern. Log at appropriate levels. DEBUG for detailed diagnostic information that may be needed during troubleshooting. INFO for significant operational events. WARNING for recoverable issues. ERROR for failures that need attention. CRITICAL for system-level failures. The most common mistake is logging everything at INFO level, which makes it impossible to distinguish normal operations from problems later.

Common Pitfalls

Global mutable state is the first thing to remove. Modules that maintain internal state through global variables create hidden dependencies between components. Those dependencies make tests harder to write and bugs harder to trace. Pass state explicitly through function arguments or class instances instead. Thread safety is another area where Python templates frequently fail. The GIL prevents true parallelism in CPython, but it does not make your code thread-safe. Shared resources accessed by multiple threads without synchronization will produce intermittent, nondeterministic bugs. If your template involves concurrency, include proper locking mechanisms from the start rather than adding them after symptoms appear. File paths that assume a specific working directory are fragile. Use pathlib.Path for all file operations and resolve paths relative to the project root or the executing module, not to os.getcwd(). The working directory changes depending on how and where you run the script, and code that depends on it will fail inconsistently.

I discovered this when a background worker that ran fine from the command line failed silently when deployed as a systemd service, because the service started in / while the script expected to run from the project directory. Switching to absolute paths based on the module location fixed it immediately.

What This Template Does Not Solve

This structure does not handle deployment, containerization, or continuous integration. Those are separate concerns that depend entirely on your infrastructure. The template assumes you will add Dockerfiles, CI pipelines, and monitoring on top of this foundation as your project grows. It also does not address framework-specific patterns. Django projects, FastAPI applications, and CLI tools each have conventions that this generic structure does not enforce. Use this as a starting point and adapt it to your framework's recommendations rather than treating it as a replacement for framework-specific guidance. For small scripts or one-off automation tasks, this template is overkill. The overhead of maintaining a structured project with logging, testing, and configuration modules is not justified for code that will be used once. Start simple and add structure when the project proves it needs it.