Why I Started Cleaning My Logs Every Single Night
I used to ignore my logs for months at a time. They accumulated—thousands of entries, duplicate events, stale artifacts that hadn't been relevant since the last deployment. Then one Tuesday morning I had to do an incident review and couldn't find anything because the log volume was just too high to search through. That was the turning point. It's not a fancy product. It's a routine. You make it a habit to go through your system logs—whether that's syslog, journald, Docker container output, application logs, whatever—and clean them before they become unmanageable. There is no tool that does this fully automatically for everything, so part of why this matters is just discipline. The actual process breaks down into a few steps. First, identify which logs matter for your operations. Second, figure out retention requirements—compliance, debugging, audit trail. Third, set up rotation or archival. Fourth, verify it's actually working.
I run this on a Ubuntu server with journald and a few application logs. Here's what I actually do each evening. I start by checking journal sizes with journalctl --disk-usage. If it's over 500MB, something is wrong. I check JournaliumMaxSize= in /etc/systemd/journald.conf to set limits. For application logs, I use logrotate with a simple config. Here's a logrotate example I actually use for a Node.js app: /var/log/myapp/*.log { weekly rotate 4 compress delaycompress missingok notifempty }
That keeps four weeks of compressed logs and stops the directory from becoming a nightmare. The compress step is important—uncompressed rotated logs take up way more space than you'd expect.
Edge Case That Almost Broke Me
Last year I had a situation where my log cleanup was silently failing. Docker was writing container logs to /var/lib/docker/containers, and I had configured log rotation for my application but not for Docker itself. The logs filled the disk and my entire service stack became unreliable. I found out because alerts started firing about disk pressure, not because the logs were the obvious problem. The fix was adding a Docker logging configuration. In docker-compose I added a block to each service that sets the max-size and max-file options. Here's what it looks like: services: myapp: logging: driver: json-file options: max-size: "10m" max-file: "3"
This means each container log file maxes out at 10 megabytes, with three versions kept. After applying that, disk usage stabilized immediately. The reason it took me so long to diagnose is that Docker doesn't show log-related errors in the usual places. You have to look at the disk utilization and then trace it back to the container storage path.
Get the Full Details
One thing nobody warns you about is that compressing old logs changes their modification time. If you rely on mtime for any automation—like triggering a backup of old logs to cold storage—you need to account for the fact that compression rewrites the timestamp. I had a script that copied logs based on mtime and it was pulling the wrong files for about six months. The workaround was switching to atime or using a custom naming scheme with dates baked in.
When This Approach Fails
Daily decluttering works well for small to medium scale setups. If you're dealing with hundreds of gigabytes of log data per day from a distributed system, this manual approach won't scale. You'll need something like a centralized logging solution—ELK stack, Loki, Datadog, Splunk—where the ingestion pipeline handles retention and rotation for you. Setting that up takes more effort upfront, but maintaining daily log cleanup across fifty microservices by hand is not sustainable. Another limitation is compliance. If you're in a regulated industry that requires specific retention periods, you can't just delete logs after a week because it's convenient. You need to archive them properly. I store old logs on S3 with lifecycle rules that move them to Glacier after thirty days. The daily cleanup then becomes: compress locally, upload to S3, delete the local copy. This usually takes about ten to fifteen minutes for a mid-sized server. If you want to see what I'm running, the configuration files are all available on my public git repo. The logrotate configs, the journald settings, and the Docker logging templates are all there. I don't charge for anything. It's just plain configs.
