#StackBounty: #curl #flask #docker-compose #celery #flower Trigger Celery task in a RESTful Route

Bounty: 50

I would like to set a route for my Celery tasks and monitor them.


This is the code I have in my flask app running at localhost:5000

background.py

Task:

@celery.task(queue='cache')
def cache_user_tracks_with_features():
    return {'status': 'Task completed!'}

Route:

@task_bp.route('/filter', methods=['GET', 'POST'])
def cache_user_with_features():
    # task
    task = cache_user_tracks_with_features.apply_async()
    while not task.ready():
      sleep(2)

    response_object = {
        'status': 'fail',
        'message': 'User does not exist'
    }
    try:
        user = User.query.filter_by(id=1)).first()
        if not user:
            return jsonify(response_object), 404
        else:
            response_object = {
                'status': 'success',
                'data': {
                    'task_id': task.id,
                    'username': user.username,
                    'email': user.email,
                    'active': user.active
                }
            }
            return jsonify(response_object), 200
    except ValueError:
        return jsonify(response_object), 404

Trigger attempt

I am trying to test it using CURL at terminal, like so:

$ curl -X POST http://localhost:5001/filter -H "Content-Type: application/json" 

But either I get curl: (52) Empty reply from server or else it just hangs. If I remove task from function and curl POST, I get:

{
  "data": {
    "active": true, 
    "email": "me@mac.com", 
    "username": "me"
  }, 
  "status": "success"
}

Docker logs give me:

nginx_1    | 172.21.0.1 - - [03/Apr/2019:22:26:41 +0000] "GET /manifest.json HTTP/1.1" 304 0 "http://localhost/filter" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/73.0.3683.86 Safari/537.36" "-"

web-db_1   | 2019-04-01 19:52:52.415 UTC [1] LOG:  background worker "logical replication launcher" (PID 25) exited with exit code 1

This is how I configured Celery and Flower (Celery monitoring) in my docker-compose file:

docker-compose-dev.yml

version: '3.6'

services:

  celery:
    image: dev3_web
    restart: always
    volumes:
      - ./services/web:/usr/src/app
      - ./services/web/logs:/usr/src/app/logs

    command: celery worker -A celery_worker.celery --loglevel=INFO --logfile=logs/celery.log -Q cache
    environment:
      - CELERY_BROKER=redis://redis:6379/0
      - CELERY_RESULT_BACKEND=redis://redis:6379/0
    depends_on:
      - web
      - redis
    links:
      - redis:redis
      - web-db

  redis:
    image: redis:5.0.3-alpine
    restart: always
    expose:
      - '6379'
    ports:
      - '6379:6379'

  monitor:
    image: dev3_web
    ports:
      - 5555:5555
    command:  flower -A celery_worker.celery --port=5555 --broker=redis://redis:6379/0
    depends_on:
      - web
      - redis

web/logs/celery_log

[2019-04-02 02:51:07,338: INFO/MainProcess] Connected to redis://redis:6379/0
[2019-04-02 02:51:07,375: INFO/MainProcess] mingle: searching for neighbors
[2019-04-02 02:51:08,491: INFO/MainProcess] mingle: all alone
[2019-04-02 02:51:08,582: INFO/MainProcess] celery@59ed7459ac14 ready.
[2019-04-02 02:51:08,661: INFO/MainProcess] Events of group {task} enabled by remote.

Flower shows worker with an active status at dashboard:

enter image description here

QUESTION

What am I missing? How do I trigger this Celery task and monitor it?


Get this bounty!!!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.