Zeebe is a horizontally scalable, language agnostic, highly available, process orchestration engine that can execute a BPMN 2.0 workflow.
Key aspects of Scalability
- No central database
- Event sources (append only logs). Logs are written to disks
- Log replication (Atomix) and Compaction (delete event after all exporters have processed an event).
- RocksDb for snapshots / Projections.
- Support for many exporters like Elasticsearch, Kafka and Hazelcast.
- One thread per log means no locking contention.
![Scale1](https://cdn.blot.im/blog_10daedb3834b4720bfc0ea67fba4b7ba/_image_cache/c652a42e-c4e9-47dc-a78a-314b14027055.png)
- Peer to peer clusters.
- Requires 2 nodes initially (main and contact point), then GOSSIP (that’s how EPIDEMICS spread).
![Scale2](https://cdn.blot.im/blog_10daedb3834b4720bfc0ea67fba4b7ba/_image_cache/11b27467-3128-4b7c-ae1e-2d3aab835b7f.png)
Replication using RAFT consensus algorithm with ATOMIX as the implementation. http://thesecretlivesofdata.com/raft/
Leaders can only write, followers are for replicas.
Zeebe is a CP system in the CAP theorem.
To achieve consistency, QUORUM must be obtained.
[ quorum ≥(replication group size / 2) + 1 ]
QUOROM avoids split-brain phenomena.
![Scale3](https://cdn.blot.im/blog_10daedb3834b4720bfc0ea67fba4b7ba/_image_cache/9892ddf2-1b83-4206-838a-339a7b444957.png)
- Partitions for throughout and multi-threading:
- Each partition is a separate physical append only log.
- Each partition has its own leader that gets write access.
- All workflow events of a single flow go to the same partition for single write access
- Use of gRPC for remote communication.
![Scale4](https://cdn.blot.im/blog_10daedb3834b4720bfc0ea67fba4b7ba/_image_cache/aa1e1e5c-750b-46e7-a39e-e3d7870c0739.png)