|||

Some notes on Zeebe (A scalable process orchestrator)

Zeebe is a horizontally scalable, language agnostic, highly available, process orchestration engine that can execute a BPMN 2.0 workflow.

Key aspects of Scalability

  • No central database
  • Event sources (append only logs). Logs are written to disks
  • Log replication (Atomix) and Compaction (delete event after all exporters have processed an event).
  • RocksDb for snapshots / Projections.
  • Support for many exporters like Elasticsearch, Kafka and Hazelcast.
  • One thread per log means no locking contention.

Scale1

  • Peer to peer clusters.
  • Requires 2 nodes initially (main and contact point), then GOSSIP (that’s how EPIDEMICS spread).

Scale2

  • Replication using RAFT consensus algorithm with ATOMIX as the implementation. http://thesecretlivesofdata.com/raft/

  • Leaders can only write, followers are for replicas.

  • Zeebe is a CP system in the CAP theorem.

  • To achieve consistency, QUORUM must be obtained.

        [ quorum ≥(replication group size / 2) + 1 ]
  • QUOROM avoids split-brain phenomena.

Scale3

  • Partitions for throughout and multi-threading:
  • Each partition is a separate physical append only log.
  • Each partition has its own leader that gets write access.
  • All workflow events of a single flow go to the same partition for single write access
  • Use of gRPC for remote communication.

Scale4

Up next Docker-Compose in AWS ECS with EFS volume mounts A hashicorp packer project to provision an AWS AMI with node, pm2 & mongodb
Latest posts Refactor react code to use state store instead of multiple useState hooks Notes on Python Threat Modelling - Using Microsoft STRIDE Model WCAG - Notes Flutter CI/CD with Azure Devops & Firebase - iOS - Part 1 Flutter CI/CD with Azure Devops & Firebase - Android - Part 2 How to samples with AWS CDK A hashicorp packer project to provision an AWS AMI with node, pm2 & mongodb Some notes on Zeebe (A scalable process orchestrator) Docker-Compose in AWS ECS with EFS volume mounts Domain Driven Design Core Principles Apple Push Notifications With Amazon SNS AWS VPC Notes Building and Deploying apps using VSTS and HockeyApp - Part 3 : Windows Phone Building and Deploying apps using VSTS and HockeyApp - Part 2 : Android Building and Deploying apps using VSTS and HockeyApp - Part 1 : iOS How I diagnosed High CPU usage using Windbg WCF service NETBIOS name resolution woes The troublesome Git-Svn Marriage GTD (Getting things done) — A simplified view Javascript Refresher Sharing common connection strings between projects A simple image carousel prototype using Asp.net webforms and SignalR Simple logging with NLog Application logger SVN Externals — Share common assembly code between solutions Simple async in .net 2.0 & Winforms Clean sources Plus Console 2 — A tabbed console window