Kemux stream Processor classes

Base class for Processor classes.

Processor dataclass

Processor base Class

Provides a common interface for Processor classes to read and write data to or from Kafka topics.

Attributes:
  • topic (str) –

    The Kafka topic to read and write data to or from.

  • schema (InputSchema) –

    The schema to use for reading and writing data, used by Faust to serialize and deserialize data.

  • logger (Logger) –

    The logger to use for logging messages.

  • topic_handler (TopicT) –

    The Faust topic handler for the Kafka topic.

initialize_handler(app) classmethod

Initialize the Faust topic handler for the Kafka topic.

Parameters:
  • app (App) –

    The Faust application to initialize the topic handler for.
Returns:
  • None –

    None

Input Processor

Base class for stream input classes.

InputProcessor dataclass

Bases: Processor

InputProcessor Class

Provides a common interface for stream input classes to read data from Kafka topic.

ingest(message) staticmethod

Ingest a message from a Kafka topic. Must be implemented by user when defining a stream input. This is done instead of enforcing implementation via abstract classes, due to the fact that this class is never instantiated directly, but rather used as an interface.

Parameters:
  • message (dict) –

    Faust record payload deserialized to dict for ingestation, with structure defined by input schema.
Returns:
  • dict( dict ) –

    Ingested message.
Raises:
  • NotImplementedError –

    If not implemented by user.

Output Processor

Base class for stream output classes.

OutputProcessor dataclass

Bases: Processor

OutputProcessor Class

Provides a common interface for stream output classes to write data to Kafka topic.

declare() async classmethod

Declare the output topic and send an init message.

The init message is used to ensure that the topic is created before any messages are sent. It is always sent as the first message to the topic with the following structure:

{
    '__kemux_init__': <current datetime in ISO format>
}
Raises:
  • ValueError –

    If invalid topic handler.

filter(message) staticmethod

Check if a message qualifies for the output topic. Must be implemented by user when defining a stream output. This is done instead of enforcing implementation via abstract classes, due to the fact that this class is never instantiated directly, but rather used as an interface.

Parameters:
  • message (dict) –

    Message to check, with structure defined by schema for input records.
Returns:
  • bool( bool ) –

    True if message qualifies for output topic, False otherwise.
Raises:
  • NotImplementedError –

    If not implemented by user.

send(message) async classmethod

Send a message to the output topic.

Parameters:
  • message (dict) –

    Message to send, with structure defined by schema for output records.