django使用celery及踩坑

树下的老石头

 关注

阅读 90

2022-03-20

项目结构

在这里插入图片描述

项目依赖

celery==5.1.2   
Django==3.2.12  
django-celery-beat==2.2.1  
django-celery-results==2.2.0  
django-redis==5.2.0  
eventlet==0.33.0  
greenlet==1.1.2   
redis==4.1.4

安装package

  • 安装celery
pip install celery
  • 安装django-celery-results
    用于保存celery任务结果
pip install celery
  • 安装django-celery-beat
    用于执行周期任务
pip install celery
  • 安装eventlet
    windows环境下需要
pip install eventlet

在django项目中使用

  • 在项目设置文件settings.py中配置celery
    settings.py
"""  
Django settings for demo project.  
  
Generated by 'django-admin startproject' using Django 3.2.12.  
  
For more information on this file, see  
https://docs.djangoproject.com/en/3.2/topics/settings/  
  
For the full list of settings and their values, see  
https://docs.djangoproject.com/en/3.2/ref/settings/  
"""  
  
from pathlib import Path  
  
# Build paths inside the project like this: BASE_DIR / 'subdir'.  
BASE_DIR = Path(__file__).resolve().parent.parent  
  
# Quick-start development settings - unsuitable for production  
# See https://docs.djangoproject.com/en/3.2/howto/deployment/checklist/  
  
# SECURITY WARNING: keep the secret key used in production secret!  
SECRET_KEY = 'django-insecure-%8q+wpy=)*@237)o#!pgkn&^$qr9-h!w12jociuv4i^f#&z70d'  
  
# SECURITY WARNING: don't run with debug turned on in production!  
DEBUG = True  
  
ALLOWED_HOSTS = []  
  
# Application definition  
  
INSTALLED_APPS = [  
    'django.contrib.admin',  
 'django.contrib.auth',  
 'django.contrib.contenttypes',  
 'django.contrib.sessions',  
 'django.contrib.messages',  
 'django.contrib.staticfiles',  
 'app1',  #注册应用
 'django_celery_results',  #用于保存celery任务结果
 'django_celery_beat',  #用于执行celery周期任务
]  
  
MIDDLEWARE = [  
    'django.middleware.security.SecurityMiddleware',  
 'django.contrib.sessions.middleware.SessionMiddleware',  
 'django.middleware.common.CommonMiddleware',  
 'django.middleware.csrf.CsrfViewMiddleware',  
 'django.contrib.auth.middleware.AuthenticationMiddleware',  
 'django.contrib.messages.middleware.MessageMiddleware',  
 'django.middleware.clickjacking.XFrameOptionsMiddleware',  
]  
  
ROOT_URLCONF = 'demo.urls'  
  
TEMPLATES = [  
    {  
        'BACKEND': 'django.template.backends.django.DjangoTemplates',  
 'DIRS': [BASE_DIR / 'templates']  
        ,  
 'APP_DIRS': True,  
 'OPTIONS': {  
            'context_processors': [  
                'django.template.context_processors.debug',  
 'django.template.context_processors.request',  
 'django.contrib.auth.context_processors.auth',  
 'django.contrib.messages.context_processors.messages',  
 ],  
 },  
 },  
]  
  
WSGI_APPLICATION = 'demo.wsgi.application'  
  
# Database  
# https://docs.djangoproject.com/en/3.2/ref/settings/#databases  
  
DATABASES = {  
    'default': {  
        'ENGINE': 'django.db.backends.sqlite3',  
 'NAME': BASE_DIR / 'db.sqlite3',  
 }  
}  
  
# Password validation  
# https://docs.djangoproject.com/en/3.2/ref/settings/#auth-password-validators  
  
AUTH_PASSWORD_VALIDATORS = [  
    {  
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',  
 },  
 {  
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',  
 },  
 {  
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',  
 },  
 {  
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',  
 },  
]  
  
# Internationalization  
# https://docs.djangoproject.com/en/3.2/topics/i18n/  
  
LANGUAGE_CODE = 'zh-hans'  #语言修改为中文
  
TIME_ZONE = 'Asia/Shanghai'  #时区修改为上海
  
USE_I18N = True  
  
USE_L10N = True  
  
USE_TZ = False  
  
# Static files (CSS, JavaScript, Images)  
# https://docs.djangoproject.com/en/3.2/howto/static-files/  
  
STATIC_URL = '/static/'  
  
# Default primary key field type  
# https://docs.djangoproject.com/en/3.2/ref/settings/#default-auto-field  
  
DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'  
  
# celery配置  
# 任务结果使用数据库来储存  
CELERY_RESULT_BACKEND = 'django-db'  
# celery 缓存
CELERY_CACHE_BACKEND = 'django-cache'

# 用了存放消息队列的位置，此处使用redis，默认使用rabbitmq
CELERY_BROKER_URL = 'redis://127.0.0.1:6379/2'  
  
#: Only add pickle to this list if your broker is secured  
#: from unwanted access (see userguide/security.html)  
CELERY_ACCEPT_CONTENT = ['json']  
CELERY_TASK_SERIALIZER = 'json'  
  
# 缓存配置  
CACHES = {  
    'default': {  
        'BACKEND': 'django_redis.cache.RedisCache',  
 'LOCATION': 'redis://127.0.0.1:6379/0',  
 },  
 'celery': {  
        'BACKEND': 'django_redis.cache.RedisCache',  
 'LOCATION': 'redis://127.0.0.1:6379/1',  
 },  
}
  • 在项目同名目录下创建celery.py文件
    celery.py
from __future__ import absolute_import, unicode_literals  
import os  
from celery import Celery  
from celery.schedules import timedelta  
# set the default Django settings module for the 'celery' program.  
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'demo.settings')  
  
app = Celery('demo')  
app.conf.timezone = 'Asia/Shanghai'  
app.conf.enable_utc = False  
  
# Using a string here means the worker doesn't have to serialize  
# the configuration object to child processes.  
# - namespace='CELERY' means all celery-related configuration keys  
#   should have a `CELERY_` prefix.  
app.config_from_object('django.conf:settings', namespace='CELERY')  
# 周期任务配置
app.conf.beat_schedule = {  
    'add-every-10-seconds': {  
        'task': 'app1.tasks.func2',  
 'schedule': timedelta(seconds=10),  
 'args': ()  
    },  
}  
# Load task modules from all registered Django app configs.  
app.autodiscover_tasks()
  • 在项目同名目录下的__init__.py中作如下配置,这样可以确保在Django启动时加载应用程序，以便@shared_task装饰器将使用该应用程序
    __init__py
from .celery import app as celery_app  
  
__all__ = ('celery_app',)
  • 在app目录下新建tasks.py文件
    task.py
from celery import shared_task  
  
  
@shared_task  
def func1():  
    return "异步任务"  
  
  
@shared_task  
def func2():  
    return "周期任务"
  • 任务调用：异步任务为主动触发，定时和周期任务需有beat分发任务，worker去执行
    views.py
from django.http import JsonResponse  
from .tasks import func1  
  
  
# Create your views here.  
  
def async_task(request):  
    func1.delay()  
    return JsonResponse({"message": "这是celery异步任务！"},json_dumps_params={"ensure_ascii":False})

项目运行

  • 启动django项目
python manage.py runserver 127.0.0.1:8000
  • 启动celery beat（当需要执行定时任务时需要启动beat）
celery -A demo beat -l INFO

在这里插入图片描述

  • 正确启动 celery worker
celery -A demo worker -l INFO --pool=solo

在这里插入图片描述

踩坑

  • 不建议使用django-celery，容易出现很多包版本配合的问题
  • 使用 celery -A demo worker -l INFO 启动worker会出现
ValueError: not enough values to unpack (expected 3, got 0)

在这里插入图片描述

  • 使用 celery -A demo worker -l INFO -P eventlet 启动worker会出现
django.db.utils.DatabaseError: DatabaseWrapper objects created in a thread can only be used in that same thread

在这里插入图片描述

相关推荐

python3.7 Django2.x celery 踩坑

Separes 66 0 0

django之使用celery

尤克乔乔 71 0 0

Django Celery 的配置及使用---最详细教程

金穗_ec4b 13 0 0

celery笔记四之在Django中使用celery

禾木瞎写 103 0 0

django中配置使用celery

westfallon 260 0 0

Django Celery

最不爱吃鱼 66 0 0

MobSF的安装使用及踩坑指南

小亦同学321 91 0 0

使用Djongo把mongoDB连接到Django的踩坑记录

落花时节又逢君to 87 0 0

Django框架-使用celery（一）：django使用celery的通用配置，不受版本影响

新鲜小饼干 70 0 0

Django 2.1.7 Celery 4.3.0 在项目中使用Celery

c一段旅程c 232 0 0

精彩评论（0）

0 0 举报