Kafka
We will integrate Kafka in the next few weeks to break our monolith. If you want to learn about it you can log in to Aiven account using:
email: engineering@hungryhub.com
password: s3cretl1gr3@tl1
once you log in, you can modify the Kafka instance's Topics, Integrations, and Connectors and learn more about it on O'Reilly: Production-Ready Microservices Building Event-Driven Microservices or Event Streaming in Rails with Kafka Apache Kafka We will build service for: Inventory
- Review
- Restaurant GMT20230214-090026_Recording_1920x1080.mp4 You can try to create event-driven architecture using the new two rails app and one node js app using this library : Ruby-kafka: A Ruby Client Library for Apache Kafka https://github.com/zendesk/ruby-kafka Kafka Stream concepts
Kafka Stream is a Java client library to connect to Kafka server
A critical aspect in stream processing is the notion of
time
windowing
- Event time - The point in time when an event or data record occurred, i.e. was originally created "at the source". Example: If the event is a geo-location change reported by a GPS sensor in a car, then the associated event-time would be the time when the GPS sensor captured the location change.
- Processing time - The point in time when the event or data record happens to be processed by the stream processing application, i.e. when the record is being consumed. The processing time may be milliseconds, hours, or days etc. later than the original event time. Example: Imagine an analytics application that reads and processes the geo-location data reported from car sensors to present it to a fleet management dashboard. Here, processing-time in the analytics application might be milliseconds or seconds (e.g. for real-time pipelines based on Apache Kafka and Kafka Streams) or hours (e.g. for batch pipelines based on Apache Hadoop or Apache Spark) after event-time.
- Ingestion time - The point in time when an event or data record is stored in a topic partition by a Kafka broker. The difference to event time is that this ingestion timestamp is generated when the record is appended to the target topic by the Kafka broker, not when the record is created "at the source". The difference to processing time is that processing time is when the stream processing application processes the record. For example, if a record is never processed, there is no notion of processing time for it, but it still has an ingestion time.