Luther Had A Wife

Setting Up Luther: A Practical Guide

Luther is a lightweight browser automation framework designed for web scraping, data extraction, and repetitive browser tasks. It sits somewhere between a raw Puppeteer script and a full-scale enterprise crawler. Most people find it useful for moderate-sized scraping jobs where you need browser-level rendering but don't want to run Selenium across twenty tabs at once. I've used it to pull structured data from sites that heavily rely on JavaScript rendering, handle auth flows, and paginate through results. It works well enough that I still reach for it when the project scope doesn't justify spinning up a heavier stack. The core idea is simple: you define a strategy, and Luther executes it with configurable concurrency, retry logic, and headless mode. The syntax is Python-friendly if you're coming from that world, though there are bindings for other languages too. What matters more than the syntax is understanding how the queue system handles failures, because that's where most people hit their first wall.

Luther Had A Wife

I saw this as a quirky reference in an early repo readme, and honestly it stuck with me as a way to remember that no automation tool exists in isolation. Luther, in this case, needs dependencies, proxies, and proper rate limiting. Without those, you're not running a scraper — you're running a complaint generator. Install it via pip. That part is straightforward: pip install luther-browser-automation

Or grab it from GitHub if you need a development build. Once installed, you'll want to set up a configuration file before writing any code. The default config assumes you're running headless on localhost with no proxy rotation. That's fine for testing. It's not fine for anything production-adjacent. I set mine up with a base YAML that defines the browser profile, concurrent session limits, and a retry policy. Here's what my working config looks like for a typical job: browser:
headless: true
concurrency: 5
timeout: 30
retry:
max_attempts: 3
backoff: exponential
proxies:
enabled: true
rotation: every_request

This configuration has served me well across dozens of projects. The key settings are concurrency (don't push it above 10 unless you have the RAM for it), timeout (30 seconds is a reasonable default — shorter and you'll miss legitimate slow loads, longer and you'll waste resources), and retry backoff. Exponential backoff is non-negotiable. Fixed delays just get you rate-limited faster.

Writing Your First Strategy

A strategy in Luther is essentially a callable class or function that defines what the browser should do. You create one, register it with a job, and let it run. The simplest possible strategy looks like this: from luther import Strategy, run
class SimpleStrategy(Strategy):
async def execute(self, ctx):
await ctx.navigate("https://example.com")
title = await ctx.query("title")
return {"title": title}
run(SimpleStrategy(), tasks=10) That's it. Ten tasks, each one navigating to the URL and extracting the page title. Luther handles the concurrency, retries, and resource cleanup for you. The context object (ctx) gives you access to navigation, DOM queries, element interaction, and session management within each task.

What most tutorials don't tell you is that the context object is per-task. You can't share state between tasks the way you might expect from a sequential script. If you need shared state — like a shared login session or a shared proxy pool — you have to pass it explicitly through the strategy's constructor or use Luther's built-in session manager.

Handling Authentication and Sessions

This is where things get real. Most modern websites require some form of authentication before showing you the data you actually want. Luther supports cookie injection, session replay, and form-based login flows out of the box. For cookie-based auth, you export your browser cookies after logging in manually and inject them into the context. For form-based flows, you write a strategy that fills the login form, submits it, and then checks for a success indicator before proceeding to the actual data extraction logic. I ran into a specific issue recently with a site that used a rotating session token embedded in the DOM. Standard cookie injection didn't work because the token expired after the page loaded. My workaround was to write a custom session handler that extracts the token on every page load and injects it as a header on subsequent API calls. Here's roughly what that looked like:

class SessionAwareStrategy(Strategy):
async def execute(self, ctx):
await ctx.navigate("https://site.com/dashboard")
token = await ctx.query("[name=csrf-token]")
ctx.headers["X-CSRF-Token"] = token
data = await ctx.fetch_json("/api/data")
return data This pattern — extract token from DOM, attach to headers — works for any site that uses token-based auth. The trick is knowing when to extract it. If you extract it too early, the page hasn't rendered yet. If you extract it too late, you've already made requests without the token. The solution is to add a short wait after navigation before extracting, or better yet, use Luther's built-in wait-for-selector feature: await ctx.wait_for_selector("[name=csrf-token]")
token = await ctx.query("[name=csrf-token]")

Proxy Rotation and Anti-Detection

If you're scraping at any meaningful scale, you'll need proxy rotation. Luther supports this natively through its proxy manager. You provide a list of proxies, and it rotates them according to your configured strategy. There are two common approaches: rotating per-request (safer, slower) and rotating per-session (faster, riskier). For most use cases, per-request rotation is the right call. The performance difference is marginal, and the risk reduction is significant. One thing I learned the hard way: not all proxy providers are equal. Cheap residential proxy pools often contain IPs that are already flagged or blacklisted. I wasted two full days debugging why my scraper kept getting blocked before realizing the proxy pool itself was the problem. Switching to a premium provider cut my block rate from roughly 40% to under 5%. That's not a marginal improvement — that's the difference between a project that works and one that doesn't.

Common Pitfalls and How to Avoid Them

The biggest mistake I see people make with Luther is underestimating the complexity of the target site. You'll write a simple strategy, run it once, and it works perfectly. Then you increase concurrency to 20 and everything falls apart. Why? Because you haven't accounted for rate limiting, dynamic content loading, or session state management at scale. Another pitfall is ignoring the error logs. Luther produces detailed logs for every failure — connection timeouts, selector not found, JS execution errors. Most people skim these and move on. The logs are actually the fastest way to debug issues. A connection timeout on task 47 of 500 tells you something very different than a selector not found on task 1 of 500. Here's a counter-intuitive insight: sometimes the best approach is to NOT use browser automation at all. If the site has a public API or if the data is available through a simpler HTTP request, use that instead. Browser automation is slower, more resource-intensive, and more fragile. It should be your last resort, not your first choice.

When Luther Falls Short

Luther isn't the right tool for every job. If you need to scrape millions of pages, look at dedicated crawler frameworks like Scrapy with Playwright integration. If you need to interact with complex single-page applications that require full browser state management, Puppeteer or Playwright directly might give you more control. If you're working with sites that have aggressive anti-bot measures, you'll need something beyond basic proxy rotation — consider tools specifically designed for anti-detection, or invest time in making your requests look more human through randomized timing, realistic user agents, and mouse movement simulation. For moderate-scale jobs (hundreds to a few thousand pages), well-written strategies, and sites without extreme anti-bot protections, Luther hits a sweet spot. It's fast enough to be practical, flexible enough to handle complexity, and simple enough that you're not managing a dozen moving parts.

Practical Tips That Actually Matter

Use selectors that are unlikely to change. Class names with random hashes are a nightmare. Look for semantic selectors — IDs, data attributes, or structural selectors like :nth-child. They break less often. Cache your extracted data. I've seen people re-scrape the same pages multiple times because they didn't implement basic caching. A simple filesystem cache or Redis store will save you hours of redundant requests and prevent unnecessary strain on the target servers. Set realistic expectations for speed. A well-configured Luther instance running at 5 concurrent sessions can process roughly 100-200 pages per minute depending on the target site's response time. That's fast enough for most projects. If you need more throughput, you'll need to redesign the architecture, not just crank up the concurrency.

Monitor your success rate. Track how many tasks succeed versus fail in real time. If your success rate drops below 80%, something has changed — the site updated its anti-bot measures, your proxies got burned, or your selectors broke. Don't ignore a dropping success rate. It's the earliest warning signal you have.

Alternative Approaches

If Luther doesn't fit your needs, there are other options. Scrapy with Splash or Playwright middleware handles large-scale scraping well. Playwright itself, when used directly, gives you more granular control over browser behavior. For simple HTTP-based scraping where JavaScript rendering isn't needed, requests or httpx with asyncio will be faster and lighter than any browser-based solution. The right tool depends on your specific constraints: how many pages you need, how complex the target site is, what your infrastructure budget looks like, and how much time you want to spend maintaining the scraper. Luther is one valid answer to those questions. It's not the only answer, and it's not always the best answer.

Final Thoughts

I've been running Luther-based scrapers in production for about three years now. The ones that survive longest are the ones I treat as living code — regularly updated, monitored, and adjusted as target sites change. The ones that die quickly are the ones I set up once and forgot about. Either way, the tool itself is reliable. It's the setup and maintenance that makes or breaks the project. If you're starting out, begin small. Write a strategy that extracts one piece of data from one page. Get it working. Then add complexity gradually — more pages, more data points, authentication, proxy rotation. Each step is a chance to learn what's actually needed versus what sounds impressive on paper. The documentation is decent but not exhaustive. You'll spend time reading source code and experimenting. That's normal. Everyone does it. The community is small but active, and the GitHub issues section has answers to most common problems if you search properly.

I keep a template strategy repository that I copy for new projects. It has the base config, common helper functions, error handling patterns, and logging setup already in place. That cuts my initial setup time from about two hours down to maybe fifteen minutes. Worth the investment if you're doing this more than once.