Recommendation Service Architecture
https://drive.google.com/file/d/1zEtYgWcmyyrdK7FzvXj0GOdieSTPF29U/view?usp=drivesdk
Right now, we used to write our data transactions from clevertap directly to DB, and it caused very high metrics and so many pods that we decided to redesign our service to:
We create a DB queue using Bull and proceed with our data based on how many workers are present and how much data the workers can write. So, if the worker is capable of writing 50 data per minute, only 50 data will be processed. So the CPU on DB is not high because the data was waiting to proceed on DB queue.
This service requires redis, which is also used by DBqueue and cache.We will create another Redis DB for this service.
So, if the user needs a clevertap recommendation, it will first get the data from cache; if the cache is empty, it will search the database.