Repository navigation
Research and tune download failures across API and Celery #6704
Description
Activity
except Exception:
# sets download-failed in Redis
logger.exception("Download failed...")
task_utils.delete_redis_value('download-queued:{}'.format(task_id))
task_utils.set_redis_value('download-failed:{}'.format(task_id), True, age=7200)I believe this succeed behavior is because there's no raise, it returns NONE which then celery views as suceeded
What we’re after
Research and implement better handling for download failures across the API and Celery apps.
Logs show jobs can hit DB statement timeouts. The task writes
download-failed:<task_id>for the API flow, but Celery reports the task assucceeded.Action item(s)
Download failedshould re-raise so Celery marks the task failed.Completion criteria
References/resources/technical considerations
webservices/tasks/download.py: genericExceptionlogsDownload failed, writes Redis failure state, and does not re-raise.webservices/resources/download.py: API polling readsdownload-failed:<task_id>and returns HTTP 500.