Run Background Tasks with Celery and Django

GITHUB LINK

You’ve created a subscription-based app where users pay a monthly fee to access premium features. Everything works well, but as your user base grows, you encounter a challenge: deducting monthly subscription fees from user wallets. Doing this synchronously would slow down your application, causing delays and a poor user experience.

This is where Celery comes in handy. By running tasks in the background, Celery keeps your application responsive while handling time-consuming tasks like subscription payments. In this post, I’ll guide you through automating monthly subscription payments using Celery, Docker, Docker Compose, Celery Beat, Redis, and Flower. Plus, I’ll provide step-by-step instructions to set it all up.

Blocking the Main Thread

In a subscription-based application, deducting monthly fees involves:

Fetching the user’s wallet balance.
Deducting the subscription amount.
Updating the wallet balance in the database.
Sending a payment confirmation email.

If these tasks run synchronously, they block the main thread, leading to:

Slow response times.
Poor user experience.
Potential server timeouts.

Solution

Offload these tasks to a background worker using Celery.

Celery

Celery is a distributed task queue that allows you to run tasks asynchronously. Here’s how it solves our problem:

Background Task Execution: Deduct subscription fees in the background without blocking the main application.
Scalability: Handle thousands of users by adding more Celery workers.
Reliability: Retry failed tasks to ensure no payment is missed.

Docker and Docker Compose

Now, let’s talk about Docker and Docker Compose. Why are they important?

Consistency: Docker ensures your application runs the same way in development, testing, and production. No more "it works on my machine" issues.
Isolation: Each service (Django, Celery, Redis, etc.) runs in its own container, avoiding conflicts.
Ease of Setup: With Docker Compose, you can define all your services (Django, Celery workers, Redis, Flower) in a single file and start them with one command.

Here’s how Docker Compose simplifies your life:

Redis: Celery needs a message broker like Redis to manage tasks.
Celery Worker: Handles background tasks.
Celery Beat: Schedules periodic tasks (e.g., deducting monthly fees).
Flower: Monitors Celery workers and tasks in real-time.

Django-Celery Setup

Let’s dive into the technical details. Here’s how to set up Celery in your Django project to automate monthly subscription payments.

Install Required Packages

First, install the necessary packages in your virtual environment, create django project and account app

pip install django celery redis djangorestframework gunicorn psycopg2-binary flower python-dotenv
pip freeze > requirements.txt

# create django project and account app
django-admin startproject considerthis .
python manage.py startapp account

Create a User and Wallet Model

Create a Wallet model to store user balances in models.py:

from django.db import models
from django.contrib.auth.models import AbstractUser

class User(AbstractUser):
    email = models.EmailField(unique=True)

class Wallet(models.Model):
    user = models.OneToOneField(User, on_delete=models.CASCADE, related_name='wallet')
    balance = models.DecimalField(max_digits=10, decimal_places=2, default=0.00)

    def __str__(self):
        return f"{self.user.username}'s Wallet"

# to hold details of background tasks that failed after certain number of retries
class FailedTask(models.Model):
    task_id = models.CharField(max_length=255)
    exc = models.TextField()
    args = models.JSONField()
    kwargs = models.JSONField()
    einfo = models.TextField()

Update your settings.py to use the custom user model

import os
from dotenv import load_dotenv

load_dotenv()


AUTH_USER_MODEL = 'account.User' # assuming account is your app name: I like using account for user management
MONTHLY_SUBSCRIPTION_FEE = os.getenv("MONTHLY_SUBSCRIPTION_FEE")

INSTALLED_APPS = [
    # other apps...
    "account",
]

CELERY_BROKER_URL = "redis://considerthis-redis:6379/0"

# point django to the containerized postgres db
DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": os.getenv('DB_NAME'),
        "USER": os.getenv('DB_USER'),
        "PASSWORD": os.getenv('DB_PASSWORD'),
        "HOST": 'db',
        "PORT": "5432",
    }
}

Create a Celery Task

create tasks.py in account app ( or your app that will run the background task) app, define the task to deduct the subscription fee:

from celery import shared_task, Task
from django.core.mail import send_mail
from .models import Wallet, FailedTask, User
from django.conf import settings
from django.db import transaction
from decimal import Decimal


class CallbackTask(Task):
    def on_failure(self, exc, task_id, args, kwargs, einfo):
        # log the exception or update DB to indicate email send failure
        FailedTask.objects.create(
            task_id=task_id, exc=str(exc), args=args, kwargs=kwargs, einfo=str(einfo)
        )

@shared_task(bind=True, base=CallbackTask, max_retries=4)
@transaction.atomic
def deduct_subscription_fee(self, user_id):
    try:
        user = User.objects.get(id=user_id)
        wallet = user.wallet
        subscription_fee = Decimal(settings.MONTHLY_SUBSCRIPTION_FEE)  # Monthly subscription fee

        if wallet.balance >= subscription_fee:
            wallet.balance -= subscription_fee
            wallet.save()
            # Send payment confirmation email
            send_mail(
                subject='Subscription Payment Successful',
                message=f'Your monthly subscription fee of ${subscription_fee} has been deducted. Your new balance is ${wallet.balance}.',
                from_email='noreply@considerthis.com',
                recipient_list=[wallet.user.email],
            )
        else:
            # Notify user of insufficient balance
            send_mail(
                subject='Subscription Payment Failed',
                message=f'Your monthly subscription fee of ${subscription_fee} could not be deducted due to insufficient balance.',
                from_email='noreply@yourstore.com',
                recipient_list=[wallet.user.email],
            )
    except Wallet.DoesNotExist:
        print(f"Wallet for user {user_id} not found.")
    except Exception as exc:
        # Retry the task
        raise self.retry(exc=exc, countdown=min(5 * (self.request.retries + 1), 300))

Schedule the Task with Celery Beat

In celery.py, configure Celery Beat to run the task monthly:

from celery.schedules import crontab
import os
from celery import Celery


# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'considerthis.settings') # considerthis is django project name

app = Celery('considerthis')

app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django app configs.
app.autodiscover_tasks()

app.conf.beat_schedule = {
        'deduct-monthly-subscription': {
            'task': 'account.tasks.deduct_subscription_fee', # account is the app name
            'schedule': crontab(day_of_month=1, hour=0, minute=0),  # Run on the 1st of every month at midnight
        },
}

Update __init__.py in the django project folder to

from __future__ import absolute_import, unicode_literals

# This will make sure the app is always imported when Django starts
from .celery import app as celery_app

__all__ = ('celery_app',)

Dockerize Your Django Project

Create a Dockerfile for your Django application:

 # Use an official Python runtime based on Debian 10 "buster" as a parent image
 FROM python:3.12.0a3-slim-buster

 # Set environment variables
 ENV PYTHONUNBUFFERED 1
 ENV PYTHONDONTWRITEBYTECODE 1

 # Create and set working directory
 WORKDIR /app

 # Install system dependencies
 RUN apt-get update && apt-get install -y \
     gcc \
     python3-dev \
     musl-dev \
     libpq-dev

 # Install Python dependencies
 COPY requirements.txt .
 RUN pip install --no-cache-dir -r requirements.txt

 # Copy project files into the container currently just three files
 COPY . /app/

 # Make port 8000 available to the world outside this container
 EXPOSE 8000

Add Celery Worker, Celery Beat, and Flower services to docker-compose.yml:

 services:
   web:
     container_name: considerthis-django
     build: .
     command: gunicorn considerthis.wsgi:application --bind 0.0.0.0:8000
     volumes:
       - .:/app
     ports:
       - "8000:8000"
     depends_on:
       - redis
       - db
     networks:
       - considerthis

   db:
     container_name: considerthis-postgres
     image: postgres:${PG_VERSION:-16}
     volumes:
       - postgres-data:/var/lib/postgresql/data
     environment:
       - POSTGRES_DB=${DB_NAME}
       - POSTGRES_USER=${DB_USER}
       - POSTGRES_PASSWORD=${DB_PASSWORD}
     networks:
       - considerthis

   worker:
     build: .
     container_name: considerthis-celery
     command: celery -A considerthis worker --loglevel=info
     volumes:
       - .:/app
     depends_on:
       - redis
     networks:
       - considerthis

   beat:
     container_name: considerthis-celery-beat
     build: .
     command: celery -A considerthis beat --loglevel=info
     volumes:
       - .:/app
     depends_on:
       - redis
     networks:
       - considerthis

   redis:
     container_name: considerthis-redis
     image: redis:latest
     ports:
       - "6379:6379"
     networks:
       - considerthis
     volumes:
       - considerthis-redis-data:/data
     command: ["redis-server", "--appendonly", "yes"]

   flower:
     container_name: considerthis-flower
     build: .
     command: celery -A considerthis flower
     ports:
       - "5555:5555"
     depends_on:
       - redis
     networks:
       - considerthis

 volumes:
   postgres-data:
   considerthis-redis-data:

 networks:
   considerthis:

Run your services in detached mode
```
 docker-compose up --build -d
```

Monitor Tasks with Flower

Flower is a web-based tool for monitoring Celery workers and tasks. Access it at http://localhost:5555 to:

Monitor task progress in real-time.
Identify failed tasks and retry them.
Track worker performance and resource usage.

Testing the Setup

Create superuser to access the admin dashboard
```
 python manage.py createsuperuser
```

Add a user and wallet to your database:

 from account.models import User, Wallet

 user = User.objects.create_user(username='testuser', email='test@example.com', password='testpass123')
 Wallet.objects.create(user=user, balance=50.00)

Wait for the scheduled task to run (or trigger it manually for testing via the shell):

 python manage.py shell

 >>> from account.tasks import deduct_subscription_fee
 >>> from account.models import User

 >>> user = User.objects.filter(username = "testuser").first()

 >>> deduct_subscription_fee.delay(user.id)

Check the Celery worker logs and Flower dashboard to ensure the task is executed.

Conclusion

By using Celery to handle background tasks, you can automate monthly subscription payments and keep your Django application responsive. Docker and Docker Compose make it easy to manage all the moving parts, while Celery Beat and Flower add scheduling and monitoring capabilities.

In our subscription-based application example, this setup ensures that users are billed reliably every month, and you can scale your application as your user base grows.

Have you faced similar challenges in your projects? How did you solve them? Share your thoughts in the comments below! If you found this post helpful, don’t forget to share it with your network.

Stay tuned for the next post in the Consider This series, where we continue to tackle real-world business challenges using Django!

Consider This: Automating Monthly Subscription Payments with Celery and Django