-
-
Notifications
You must be signed in to change notification settings - Fork 4.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Django 2.2 + Celery 5.2.7 unit testing does not Retrying the task #8056
Comments
Hey @Emilianissimo 👋, We also offer priority support for our sponsors. |
I'm running into, what I believe to be, a similar issue. I have a Django v3.2.19 application currently running Celery v4.3.0 and am attempting to upgrade to v5.2.7. After upgrading, a few test cases around task retry logic started failing which I believe to be related to this PR. Below is an example showcasing the issue class SomeException(Exception):
""" Custom Exception Class """
pass
def some_function(number):
if number < 0:
raise SomeException
return True
@app.task(bind=True)
def some_task(self):
try:
random_number = random.randint(0, 100)
some_function(random_number)
except SomeException as e:
retry_countdown = int(random.uniform(2, 4) ** self.request.retries)
raise self.retry(countdown=retry_countdown, max_retries=5)
class CeleryRetryTestCase(TestCase):
@mock.patch('some_function')
def test_celery_retry(self, mocked_some_function):
retryable_exceptions = [
SomeException(),
]
mocked_some_function.side_effect = retryable_exceptions * 10
with self.assertRaises(MaxRetriesExceededError):
some_task.delay()
# we should have called some_function() 6 times
self.assertEqual(mocked_some_function.call_count, 6) Running the ======================================================================
ERROR: test_celery_retry (CeleryRetryTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "redacted/test.py", line 174, in some_task
some_function(random_number)
File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1062, in __call__
return _mock_self._mock_call(*args, **kwargs)
File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1123, in _mock_call
raise result
SomeException
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/redacted/lib/python3.9/site-packages/mock/mock.py", line 1305, in patched
return func(*args, **keywargs)
File "/redacted/test.py", line 2816, in test_celery_retry
some_task.delay()
File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 425, in delay
return self.apply_async(args, kwargs)
File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 572, in apply_async
return self.apply(args, kwargs, task_id=task_id or uuid(),
File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 794, in apply
ret = tracer(task_id, args, kwargs, request)
File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 464, in trace_task
I, R, state, retval = on_error(
File "/redacted/lib/python3.9/site-packages/celery/app/trace.py", line 451, in trace_task
R = retval = fun(*args, **kwargs)
File "/redacted/test.py", line 177, in some_task
raise self.retry(countdown=retry_countdown, max_retries=5)
File "/redacted/lib/python3.9/site-packages/celery/app/task.py", line 691, in retry
request = self.request
celery.exceptions.Retry: Retry in 1s
----------------------------------------------------------------------
Ran 1 test in 1.477s
FAILED (errors=1) After several Google searches and failed attempts (e.g. calling I was able to get my tests passing successfully after altering Celerys ret = Retry(exc=exc, when=eta or countdown, is_eager=is_eager, sig=S)
if is_eager:
# if task was executed eagerly using apply(),
# then the retry must also be executed eagerly in apply method
S.apply()
if throw:
raise ret
return ret @mchataigner could you help me understand why the removal of |
thanks for tracing down to it. I'm not fully sure now if it was a regression! I would be happy to review a PR with that part re added and possibly more unit and integration tests |
@auvipy that would be great! I can get a PR opened early next week for your review |
just ping me after opening |
Checklist
main
branch of Celery.contribution guide
on reporting bugs.
for similar or identical bug reports.
for existing proposed fixes.
to find out if the bug was already fixed in the main branch.
in this issue (If there are none, check this box anyway).
Mandatory Debugging Information
celery -A proj report
in the issue.(if you are not able to do this, then at least specify the Celery
version affected).
main
branch of Celery.pip freeze
in the issue.to reproduce this bug.
Optional Debugging Information
and/or implementation.
result backend.
broker and/or result backend.
ETA/Countdown & rate limits disabled.
and/or upgrading Celery and its dependencies.
Related Issues and Possible Duplicates
Related Issues
Possible Duplicates
Environment & Settings
Celery version: 5.2.7
celery report
Output:Steps to Reproduce
Required Dependencies
Python Packages
pip freeze
Output:Other Dependencies
N/A
Minimally Reproducible Test Case
Expected Behavior
I'm expecting, that after raising Retry inside Django tests it will retry task and set patched_counter to 2
Actual Behavior
So it is just running trough first side_effect, raises Retry, so assertRaises works, and going through test function without retry task.
My suggest is this is in case of:
a) Retry just cannot be faster, then test case (it is not blocking, so it is async maybe?)
b) Retry just do not running (I'm not sure why?)
It worked before update to 5.2.7 from 4.3.1, also updated billiard and pytz to required.
Also here is the ENVs:
And especially for testing:
Tried to run with eager_mode to False, but it says that Retry does not raising.
Here also the celery_app file:
The text was updated successfully, but these errors were encountered: