mq-bridge library

      ┌────── mq-bridge-lib ──────┐
──────┴───────────────────────────┴──────
            crossing streams

mq-bridge is an asynchronous message library for Rust. It connects message brokers, databases, files, HTTP/WebSocket endpoints, and in-memory channels behind one small set of traits.

It is not only a forwarder. A route can transform, filter, fan out, retry, rate-limit, deduplicate, or turn a request into a response before the message reaches the next system. The core is built on Tokio and keeps the transport details at the edge, so application code can mostly work with CanonicalMessages and handlers.

Move data from A to B — inside your own service

If you need to move data or events reliably between systems and you write code (Rust, Python, or Node), mq-bridge is a strong default. It is a library you embed, not a daemon or control plane you operate.

Prefer not to write code? mq-bridge-app runs the exact same engine as a standalone, zero-code ETL service configured entirely by YAML or environment variables — move data from A to B without writing a line. It ships a Postman-style UI to build, send, and inspect messages against a route, and can import Postman collections and AsyncAPI documents to scaffold routes and endpoints for you.

• 16+ transports, one API: Kafka, NATS, AMQP (RabbitMQ), MQTT, MongoDB, Postgres CDC (logical replication), PostgreSQL / MySQL / SQLite (SQLx), ClickHouse, HTTP, WebSocket, gRPC, ZeroMQ, Redis Streams, AWS SQS/SNS, cloud object storage (S3 / GCS / Azure), IBM MQ, files, dir_spool (a crash-safe directory FIFO queue), and in-memory channels — all behind the same receive_batch / send_batch shape.
• Change Data Capture: stream row-level changes from Postgres (logical replication / pgoutput) and MongoDB (change streams) as flat rows with an operation marker.
• Restart-safe delivery: batch-aware ack/nack with commit sequencing for cumulative-ack brokers; the integration suite shows no data loss during in-flight broker restarts, including a Postgres CDC restart-safety test.
• Reliability middleware, not a framework: retries, dead-letter queues, deduplication, rate limiting, and cookie/session persistence wrap any endpoint.
• Polyglot on one engine: the same Rust core ships as native Python and Node.js bindings; routing, batching, and broker I/O stay in Rust.
• TLS everywhere, one config shape: a single TlsConfig block (CA bundle, client cert/key for mTLS, insecure-skip) is reused across transports.
• Self-hosted, no daemon: generate config in the optional UI, paste it into your code, run it in-process. No hosted control plane, no separate scheduler.

Throughput & footprint. In our own benchmarks, the same engine — driven the zero-code way through mq-bridge-app — on a CSV→JSONL file conversion (1,000,000 mixed-type rows, ~116 MiB) sustained 2,824,858 rows/s at ~28 MiB (mq-bridge 0.4.12), about ~145x faster and ~16x leaner in memory than Meltano (tap-csv → target-jsonl, ~19,500 rows/s / ~444 MiB, measured in an earlier session on the same machine). On the same file in the same session it also ran ~1.4x faster than DuckDB using all of this machine's cores (2,036,659 rows/s) while holding ~17x less memory — DuckDB being a throughput ceiling for the conversion itself, not an ETL tool. Methodology and reporting rules are in benches/ETL_BENCHMARKS.md; the measured numbers and their baselines are in the ETL benchmark harness.

Kafka → file. In a 1,000,000-row relay with the default file format and no transform, the engine was ~65% faster than Sea Streamer comparing both on the mimalloc allocator (~80% against its default-allocator build). The mq-bridge-app benchmark contains the reproducible helper and native file-format caveats.

Language Bindings

mq-bridge is a Rust library, but the same engine ships as native bindings for Python and Node.js. The Tokio runtime, broker I/O, routing, and batching all stay in Rust; the binding is a thin layer for handlers and configuration.

The constructor names are kept aligned across languages, so a config loader reads the same in either binding (Python uses snake_case, Node uses camelCase):

• Route.from_file / Route.fromFile — load a route from a YAML/JSON file
• Route.from_str / Route.fromStr — load from an in-memory YAML/JSON string
• Route.from_config / Route.fromConfig — load from a dict / JS object
• the matching Publisher.* constructors build a publisher endpoint

The name argument is optional in both: pass it to select one entry from a routes:/publishers: document, or omit it to treat the config as a single bare route/endpoint body.

The Python binding also holds up well under load on the third-party http-arena.com requests-per-second HTTP benchmark (live leaderboard — rankings shift over time). See the Python analysis notes for the local HTTP comparison harness.

Architecture

See ARCHITECTURE.md for a detailed overview of the internal design, extensibility, and usage patterns.

Usage Types:

1. Event Handler (TypedHandler): Communicate between applications using strongly-typed message handlers, optionally with response support.
2. Compute Handler: Generally receive and process messages with a custom handler
3. Direct Endpoint Usage: Use publish / publish_batch and receive / receive_batch directly on endpoints. This mode requires manual commit, batch sequencing, and concurrency handling.

For implementation details and quick start examples for each usage type, see the Architecture Guide.

Features