Introduction to Redpanda Connect
Redpanda Connect is a declarative data streaming service that solves a wide range of data engineering problems with simple, chained, stateless processing steps. It implements transaction based resiliency with back pressure, so when connecting to at-least-once sources and sinks it’s able to guarantee at-least-once delivery without needing to persist messages during transit.
It’s simple to deploy, comes with a wide range of connectors, and is totally data agnostic, making it easy to drop into your existing infrastructure. Redpanda Connect has functionality that overlaps with integration frameworks, log aggregators and ETL workflow engines, and can therefore be used to complement these traditional data engineering tools or act as a simpler alternative.
Components
Inputs
-
Services
-
AWS
-
Azure
-
Utility
-
Integration
-
Local
-
Social
-
GCP
-
Network
Inputs that consume from storage or message streaming services.
Inputs that consume from Amazon Web Services products.
Inputs that consume from Microsoft Azure services.
Inputs that provide utility by generating data or combining/wrapping other inputs.
Inputs that consume from Google Cloud Platform services.
Inputs that consume directly from low level network protocols.
Processors
-
Parsing
-
Utility
-
Mapping
-
Integration
-
Azure
-
Composition
-
Services
Processors that specialize in translating messages from one format to another.
Processors that provide general utility or do not fit in another category.
Processors that specialize in restructuring messages.
Processors that interact with external services.
Higher level processors that compose other processors and modify their behavior.
Outputs
-
Services
-
AWS
-
Azure
-
Utility
-
Social
-
Local
-
GCP
-
Network
Outputs that write to storage or message streaming services.
Outputs that write to Amazon Web Services products.
Outputs that write to Microsoft Azure services.
Outputs that provide utility by combining/wrapping other outputs.
Outputs that write to social applications and services.
Outputs that write to Google Cloud Platform services.
Outputs that write directly to low level network protocols.