Celery is an efficient and scalable way to execute tasks asynchronously and acts as a distributor of high-level tasks where tasks are queued and run concurrently using various paradigms such as multiprocessing or gevent.
An efficient and scalable way to perform tasks asynchronously is to use a queue library as Celery. With this library, you define workers that are processes for executing the heavy tasks. An interesting aspect of this solution is that there can be many workers (even in different servers) executing the tasks.
The architecture of the solution is as follows:
- Consumer: This is the application that users use. If it is a web application, it can be a Django or Flask application.
- Producer: This is the worker who does the heavy work.
- Broker: This is the mechanism that the consumer uses to store the pending work.
- Backend: This is the mechanism that's used by the producer to store the results of the task.
At this point, we can see how the elements interact with each other in the Celery architecture:
- The consumer application is responsible for generating tasks that will receive a message broker like RabbitMQ or Redis. In this chapter, we will work with the Redis message broker.
- The message broker allows you to send and receive messages, and it generates tasks to be executed in Celery workers.
- When the tasks are finished, Celery stores this information in the form of events.