Add more details about retrying tasks

src/docs/services/queue.mdx

@@ -11,14 +11,15 @@ Sentry relies on the [Celery](https://docs.celeryproject.org/) library for manag
 Sentry configures tasks with a special decorator that allows us more explicit control over the callable.
 
 ```python
-from sentry.tasks.base import instrumented_task
+from sentry.tasks.base import instrumented_task, retry
 
 @instrumented_task(
     name="sentry.tasks.do_work",
     queue="important_queue",
+    max_retries=3,
     default_retry_delay=60 * 5,
-    max_retries=None,
 )
+@retry(on=(<RetryableError>), exclude=(<ReraiseError>), ignore=(<IgnorableError>))
 def do_work(kind_of_work, **kwargs):
     # ...
 ```
@@ -33,7 +34,7 @@ There are a few important points:
   names which makes the name tied to the location of the code and more
   brittle for future code maintenance.
 
-- Tasks _must_ accept `\*\*kwargs`` to handle rolling compatibility.
+- Tasks _must_ accept `kwargs` to handle rolling compatibility.
 
   This ensures tasks will accept any message which happens to be in
   the queue rather than fail for unknown arguments.  It helps with
@@ -47,7 +48,25 @@ There are a few important points:
   a migration a little and gives some more operator flexibility, but
   message loss because of unknown arguments is still not acceptable.
 
-- Tasks _should_ automatically retry on failure.
+- Tasks _should_ retry on exception.
+
+  Without retries, any exceptions raised while executing the task will stop 
+  the task from moving forward. Generally, there are three categories of 
+  exceptions that could be raised:
+
+  1. Retryable exceptions: transient errors (e.g. connection timeouts, 
+  DB unavailable, query read timeouts) 
+  2. Ignorable exceptions: this is task specific but an example of an 
+    ignorable exception would be if your task is attempting to delete a row 
+    but the row doesn't exist and raises `DoesNotExist` exception. 
+  3. Non-ignorable programming errors that should be re-raised and fixed 
+    (AttributeError, TypeError, ValueError, etc) 
+
+  Use the `@retry` decorator to specify which exceptions should be retried, 
+  re-raised, and ignored. It might not be possible to know ahead of time 
+  all the possible exceptions that could be raised in your task, so starting 
+  with just the `@retry(on=(<RetryableError>))` might make sense and 
+  updating the types of exceptions to handle after encountering them.
 
 - Tasks arguments _should_ be primitive types and small.
 
@@ -70,6 +89,9 @@ There are a few important points:
   containing a task must be added to the `CELERY_IMPORTS` setting in
   `src/sentry/conf/server.py`.
 
+- Tasks specifying a named `queue` must be added to `CELERY_QUEUES`.
+
+
 ## Running a Worker
 
 Workers can be run by using the [Sentry CLI](https://docs.sentry.io/product/cli/).