8square

Getting 8square to behave when the documentation says it should work

I spent three weeks last year trying to make 8square actually run in a staging environment that looked exactly like production. The quick-start guide on the readme claims you can have it up and running in under ten minutes. That's true if you don't care about memory leaks, connection pooling, or the fact that the default configuration will silently drop events during burst traffic. Here's what actually happened and what I ended up doing. 8square is a bidirectional event relay layer. It sits between your service mesh and your downstream consumers, buffering messages in a ring buffer and applying backpressure when anyone falls behind. It looks like a queueing system from the outside, but it's fundamentally different because it doesn't assign ownership to a single consumer. Multiple subscribers can pull the same stream independently, each at their own pace. That design decision is what makes it useful and also what makes debugging it painful. I first ran into it when our team was replacing a Kafka-based pipeline with something lighter. We wanted sub-10ms tail latency for internal telemetry and didn't want to maintain a broker cluster for that. 8square fit that niche well, provided you understand its limits. It does not persist messages across restarts unless you configure an external journal backend. If your process crashes mid-flight, you lose whatever was in the in-memory ring buffer. I learned that the hard way during a deploy that took longer than expected on a Tuesday.

Installation and the first gotcha

The binary ships as a single executable with no dependencies. Download it from the releases page, drop it somewhere in your PATH, and run it with the default config. That default config listens on 0.0.0.0:8443, uses TLS self-signed certs by default, and enables every subscriber type. Your first mistake will be assuming the self-signed cert is fine for internal use without noticing that most service mesh sidecars refuse to connect to unverified endpoints. I fixed this by generating proper mTLS certs and putting them in /etc/8square/certs/. The config file lives at /etc/8square/config.yaml. Here's what I ended up with after iterations:

listen: "0.0.0.0:8443"
tls:
  cert: /etc/8square/certs/server.crt
  key: /etc/8square/certs/server.key
  ca: /etc/8square/certs/ca.crt
ring_buffer:
  capacity: 65536
  segments: 8
backpressure:
  mode: adaptive
  threshold_pct: 80
streams:
  - name: telemetry
    retention: 2m
    max_consumers: 4
  - name: events
    retention: 30s
    max_consumers: 2

The ring buffer capacity matters more than people realize. A 65536-slot buffer with 8 segments gives you roughly 8KB per segment on a 64-bit build. At 100k events per second with 64-byte payloads, you'd drain and refill the buffer roughly every 4 milliseconds. If your slowest consumer can't keep up, the adaptive backpressure kicks in within one sampling window. That's usually fast enough for internal traffic but not for anything cross-region. Each service that produces or consumes from 8square needs the client library. The Go SDK is the most mature, but there are Python and Rust bindings that work adequately. The trick is getting the connection string right. 8square uses a schemaless topic path, so your producer connects to something like tls://host:8443/telemetry and the consumer uses the same path. The path is not validated server-side until the first write, which means a typo silently creates a new stream instead of erroring. I caught this during load testing when our error rate jumped unexpectedly. The client library had a hardcoded path typo, and we were producing to /teliometry instead of /telemetry. The server created the stream on demand, accepted the data, and nobody noticed because the consumer was still listening on the correct path. Fix was a config validation flag I added to our deploy pipeline that runs 8square-admin streams list before promotion.

Backpressure and the edge case nobody documents

Adaptive backpressure in 8square works by sampling ring buffer occupancy every 50ms and throttling producers whose messages are being rejected. The problem is that when multiple producers share a stream, the throttle applies uniformly. A healthy producer gets dragged down by a slow one. I ran into this when one of our log exporters started failing GC cycles and became the bottleneck for an entire telemetry pipeline. The workaround is per-producer rate limiting at the application level, not relying on 8square to sort it out. Add a token bucket around each exporter with a cap of roughly 80% of the stream's theoretical throughput. That leaves headroom for the healthy producers while the slow one crawls through its backlog. It adds code complexity but it's the only reliable method I've found. The 8square maintainers have acknowledged the issue on GitHub but the fix, when it lands, will probably be opt-in rather than default.

Monitoring and observability

8square exposes Prometheus metrics on /metrics by default. The important ones are 8square_ring_usage_pct, 8square_backpressure_active, and 8square_consumer_lag_seconds. The consumer lag metric is somewhat misleading because it measures time since the consumer last read, not the actual delay in message delivery. A consumer that reads once per minute will show 60 seconds of lag even if it processes messages in real time while idle. I built a Grafana dashboard that tracks the derivative of ring usage instead of the raw value. That catches the early signs of a consumer falling behind before the backpressure kicks in. The threshold I settled on is a ring usage increase of more than 15% over any 10-second window. When that happens, you have roughly 30 seconds before backpressure engages. That gives your SRE team time to investigate without panicking.

When 8square is the wrong tool

Don't use 8square if you need message persistence across node failures. It is not a replacement for Kafka, NATS JetStream, or RabbitMQ when durability is a requirement. It's designed for ephemeral event relay within a single datacenter or availability zone. If your architecture spans regions or requires exactly-once semantics across restarts, look elsewhere. Also don't use it for high-cardinality streaming analytics. The ring buffer design favors low-latency delivery over complex filtering. If you need SQL-like queries on live data, you're better off shipping events to a proper data platform and letting 8square do what it does well: move bytes quickly between services that already trust each other.

My final takeaway

8square works well when you respect its scope. It's a relay, not a storage engine, not a broker, and not a magic fix for bad architecture. I've run it in production for six months now across three microservice clusters handling roughly 2 million events per minute. It's been stable, the operator burden is low, and the team only touches it when we adjust stream configurations. The places where it surprised me were the places I didn't read the source code carefully enough before deploying it. Do that, and it's genuinely useful.