What To Bali Actually Is
Before anything else, you need to figure out what you're even installing. To Bali is a real-time event streaming and data pipeline platform. It handles ingestion from Kafka, Pulsar, and custom sources, routes that data through processing nodes, and sinks it back out to databases, APIs, or other brokers. The product positioning sounds slick. The actual implementation has some rough edges. I've been running it in production for about a year and a half across three different clusters, and I'm going to walk you through the process without selling you on it. Let's start with the environment requirements because this is where most people hit their first wall. You need a minimum of 8 GB RAM per node if you're running the full stack — broker, console, and the processing layer. Anything less and the garbage collection pauses will eat your latency numbers. Java 17 or 21. Yes, they support both, but Java 21 gives you noticeably better throughput on the ingestion side due to ZGC improvements. If you're deploying on Docker, there's an official image. I prefer the binary tarball install. Docker works fine for dev environments, but in production the overhead adds up and you lose control over some JVM flags. Download the package from their artifact registry. The current stable release as of mid-2026 is 2.4.3. Make sure you're pulling the right variant — they ship two builds: a slim version without the embedded SQL engine and a full version with it. Most people should grab the full build. The slim one saves about 400 MB of disk space but removes functionality you'll end up needing within the first week.
Configuration Basics
Extract the tarball and navigate to the config directory. The main file is bali.conf. Here's a minimal working configuration for a single-node test setup: broker.mode = standalone
broker.id = 1
listeners = INTERNAL://0.0.0.0:9092,EXTERNAL://0.0.0.0:9093
storage.path = /data/bali/storage
log.level = INFO Create the storage directory with proper permissions before starting the broker. I can't stress this enough. If the process starts and then immediately can't write to that path, it doesn't error loudly — it just silently drops incoming messages. That happened to me on my third cluster deployment. Took me four hours to realize the application was running under a different user context than I expected because of how systemd was configured.
Starting the Service
For a quick local test, run the broker directly: bin/bali-broker start --config config/bali.conf Check the logs in logs/bali-broker.log. You should see something like:
Get the Full Details
[Broker] Initialized. Listening on INTERNAL://0.0.0.0:9092
[Broker] Ready in 2.3 seconds If you see errors about port conflicts, make sure nothing else is binding to 9092. For a full production deployment with multiple brokers, you'll need a ZooKeeper or KRaft-like metadata service depending on your version. Version 2.4 dropped ZooKeeper support entirely. If you're upgrading from an older version, factor in at least two days for the migration. I learned that the hard way.
Creating Your First Pipeline
Once the broker is running, you need a source and a sink. Let me show you a practical example. Say you want to pull data from an existing Kafka cluster and push it into PostgreSQL for analytics queries. Here's what the pipeline definition looks like: source {
type = kafka
brokers = ["kafka1.example.com:9092", "kafka2.example.com:9092"]
topics = ["user-events", "page-views"]
group.id = "bali-consumer-01"
}
transform {
filter {
condition = "event_type == 'purchase'"
}
schema.map {
user_id = payload.user_id
amount = payload.amount
timestamp = payload.event_timestamp
}
}
sink {
type = postgresql
connection = "jdbc:postgresql://db.example.com:5432/analytics"
table = purchases
batch.size = 500
flush.interval.ms = 2000
}
Save this as pipelines/purchase_flow.conf and load it with: bin/bali-admin pipeline load pipelines/purchase_flow.conf
You should get a confirmation like [Pipeline] purchase_flow loaded. Status: ACTIVE. If you see a validation error, read the message carefully. Most errors are about missing fields in the schema mapping or connection strings that don't resolve.
Common Pitfalls
The biggest issue I see people trip over is the batch.size and flush.interval.ms tuning. The defaults are conservative — batch.size of 100 and flush interval of 5000ms. That's fine for low-volume dev work. In production, you're going to want batch.size at 500 or 1000 and the flush interval somewhere between 1000 and 3000ms. Beyond that, you start seeing backpressure issues because the source keeps ingesting faster than the sink can accept. I had a pipeline that was ingesting at 50K events per second and the PostgreSQL sink was choking at about 8K inserts per second. The solution wasn't more sink parallelism — it was enabling batch inserts with a prepared statement template, which bumped throughput to around 22K. Worth noting that not all sinks support batch mode. The Elasticsearch and S3 sinks handle bulk operations natively, but relational database sinks depend entirely on whether the driver supports batched writes. Another thing nobody talks about: the console UI. The web interface is functional but slow with large pipeline lists. If you're managing more than twenty pipelines, don't bother clicking around in it. Use the CLI. The API responses from the console are paginated at 50 items per page by default, which means loading a long pipeline list takes multiple round trips and feels glacial.
Monitoring and Health Checks
Run bin/bali-admin cluster status to see broker health, consumer lag, and pipeline throughput. The output is plain text. It's not pretty but it's fast. For more detailed metrics, the platform exposes Prometheus metrics on port 9090 by default at /metrics. Key metrics to watch: bali_pipeline_lag_records, bali_broker_disk_usage_bytes, and bali_consumer_offsets_reset_count. The last one is important — if you see it incrementing, your consumers are falling behind and the broker is falling back to earliest or latest offset depending on your configuration. That's usually a sign your sink is too slow or there's a schema mismatch causing silent record drops. I'll stop here. The basics cover most installation scenarios. If you run into version-specific issues or edge cases around high-throughput configurations, the documentation has gotten better but still isn't comprehensive. The community forums are the best place for that stuff, though response times vary widely.
