DistributedCompute-OpenSource-List

Distributed System OpenSource List

Apache Beam : Apache Beam is a unified model for defining both batch and streaming data-parallel processing pipelines, as well as a set of language-specific SDKs for constructing pipelines and Runners for executing them on distributed processing backends, including Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.

2017-Sandglass : Sandglass is a distributed, horizontally scalable, persistent, time sorted message queue.
2018-PhxQueue : PhxQueue 是微信开源的一款基于 Paxos 协议实现的高可用、高吞吐和高可靠的分布式队列，保证 At-Least-Once Delivery，在微信内部广泛支持微信支付、公众平台等多个重要业务。
2018-Jocko : Kafka implemented in Golang with built-in coordination (No ZK dep, single binary install, Cloud Native)
2018-QMQ : QMQ 是去哪儿网内部广泛使用的消息中间件，自 2012 年诞生以来在去哪儿网所有业务场景中广泛的应用，包括跟交易息息相关的订单场景；也包括报价搜索等高吞吐量场景。
2018-NATS : High-Performance server for NATS, the cloud native messaging system.
2018-Waltz : Waltz is a quorum-based distributed write-ahead log for replicating transactions.
2018-TubeMQ : TubeMQ focuses on high-performance storage and transmission of massive data in big data scenarios
Disque : Disque is a distributed message broker.

Kowl : Kowl (previously known as Kafka Owl) is a web application that helps you to explore messages in your Apache Kafka cluster and get better insights on what is actually happening in your Kafka cluster in the most comfortable way.

hazelcast-jet : A general purpose distributed data processing engine, built on top of Hazelcast.
Flink : Apache Flink is an open source stream processing framework with powerful stream- and batch-processing capabilities.
Wallaroo : Wallaroo is a fast, elastic data processing engine that rapidly takes you from prototype to production by eliminating infrastructure complexity.
Bigslice : Bigslice is a system for fast, large-scale, serverless data processing using Go.
StreamPark : StreamPark, Make stream processing easier! easy-to-use streaming application development framework and operation platform
Memphis : Memphis{dev} is an open-source real-time data processing platform that provides end-to-end support for in-app streaming use cases using Memphis distributed message broker. Memphis’ platform requires zero ops, enables rapid development, extreme cost reduction, eliminates coding barriers, and saves a great amount of dev time for data-oriented developers and data engineers.

OpenEdge : Extend cloud computing, data and service seamlessly to edge devices.
KubeEdge : KubeEdge is built upon Kubernetes and extends native containerized application orchestration and device management to hosts at the Edge.

2017-Elastic Job : Elastic-Job is a distributed scheduled job solution. Elastic-Job is composited from 2 independent sub projects: Elastic-Job-Lite and Elastic-Job-Cloud.
TaskBotJS : The best JavaScript/TypeScript job processing framework on the planet.
SIA : SIA 是我们公司基础开发平台 Simple is Awesome 的简称，SIA-TASK（微服务任务调度平台）是其中的一项重要产品，SIA-TASK 契合当前微服务架构模式，具有跨平台，可编排，高可用，无侵入，一致性，异步并行，动态扩展，实时监控等特点。
2018-XXL-JOB : A distributed task scheduling framework.（分布式任务调度平台 XXL-JOB）。
2019-Mantis : A platform that makes it easy for developers to build realtime, cost-effective, operations-focused applications.
2019-Ofelia : Ofelia is a modern and low footprint job scheduler for docker environments, built on Go. Ofelia aims to be a replacement for the old fashioned cron.
2020-ballista : Distributed compute platform implemented in Rust, and powered by Apache Arrow.
2022-PowerJob : Enterprise job scheduling middleware with distributed computing ability.

2018-Conductor : Conductor is a Workflow Orchestration engine that runs in the cloud.
2019-Dolphin Scheduler : Dolphin Scheduler is a distributed and easy-to-expand visual DAG workflow scheduling system, dedicated to solving the complex dependencies in data processing, making the scheduling system out of the box for data processing.
2020-PowerJob : 新一代分布式任务调度与计算框架，支持 CRON、API、固定频率、固定延迟等调度策略，提供工作流来编排任务解决依赖关系，使用简单，功能强大，文档齐全，欢迎各位接入使用！

最近更新于 2023-04-16