ref(derived_code_mappings): Rename to Automatic Source Code Configuration #83313

armenzg · 2025-01-13T18:04:57Z

The original name of the feature was Derived Code Mappings, however, we have plans to expand it to also automatically do the following configurations beyond code mappings:

In-app stack trace rules
Import CODEOWNERs
Sync team mappings

There is also an upcoming effort from the integrations to bring feature parity across integrations, thus, this is a good time to decouple the code.

This first change focuses on renaming and moving files around while the following pull request will focus on refactoring and decoupling the code.

codecov · 2025-01-13T18:25:54Z

Codecov Report

Attention: Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.

✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
...try/issues/auto_source_code_config/code_mapping.py	75.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #83313      +/-   ##
==========================================
+ Coverage   78.15%   87.54%   +9.38%     
==========================================
  Files        9403     9405       +2     
  Lines      537402   537615     +213     
  Branches    21175    21175              
==========================================
+ Hits       420019   470660   +50641     
+ Misses     117035    66607   -50428     
  Partials      348      348

armenzg · 2025-01-14T14:16:05Z

src/sentry/issues/auto_source_code_config/code_mapping.py

@@ -13,6 +13,8 @@
 logger = logging.getLogger(__name__)
 logger.setLevel(logging.INFO)

+SUPPORTED_LANGUAGES = ["javascript", "python", "node", "ruby", "php", "go", "csharp"]


This constant is related to code mappings, thus, placing it here.

armenzg · 2025-01-14T14:16:22Z

src/sentry/issues/auto_source_code_config/code_mapping.py

@@ -109,7 +111,9 @@ def __init__(self, frame_file_path: str) -> None:
    def __repr__(self) -> str:
        return f"FrameFilename: {self.full_path}"

-    def __eq__(self, other) -> bool:
+    def __eq__(self, other: object) -> bool:


Improving the typing.

armenzg · 2025-01-14T14:43:46Z

.github/CODEOWNERS

@@ -530,6 +530,7 @@ tests/sentry/api/endpoints/test_organization_dashboard_widget_details.py     @ge
 /src/sentry/event_manager.py                                @getsentry/issues
 /src/sentry/eventstore/models.py                            @getsentry/issues
 /src/sentry/grouping/                                       @getsentry/issues
+/src/sentry/issues/                                         @getsentry/issues


We had never added this. Tagging along in this PR.

armenzg · 2025-01-14T14:45:39Z

src/sentry/conf/server.py

    Queue("derive_code_mappings", routing_key="derive_code_mappings"),
+    Queue("auto_source_code_config", routing_key="auto_source_code_config"),


Once it gets created, we can switch over to it in a follow-up PR.

armenzg · 2025-01-14T14:46:33Z

src/sentry/tasks/auto_source_code_config.py

-    name="sentry.tasks.derive_code_mappings.derive_code_mappings",
-    queue="derive_code_mappings",
+    name="sentry.tasks.auto_source_code_config.derive_code_mappings",
+    queue="derive_code_mappings",  # XXX: To be renamed to auto_source_code_config


We will switch over to the new queue on a follow-up PR.

armenzg · 2025-01-14T14:47:12Z

src/sentry/utils/sdk.py

@@ -71,8 +72,6 @@
    "sentry.tasks.summaries.weekly_reports.prepare_organization_report": 0.1
    * settings.SENTRY_BACKEND_APM_SAMPLING,
    "sentry.profiles.task.process_profile": 0.1 * settings.SENTRY_BACKEND_APM_SAMPLING,
-    "sentry.tasks.derive_code_mappings.process_organizations": settings.SAMPLED_DEFAULT_RATE,


process_organizations was removed a long time ago.

tests/sentry/integrations/github/tasks/test_pr_comment.py

armenzg · 2025-01-14T14:47:51Z

tests/sentry/integrations/github/tasks/test_pr_comment.py

@@ -400,7 +400,13 @@ def test_format_comment(self):
        ]

        formatted_comment = format_comment(issues)
-        expected_comment = "## Suspect Issues\nThis pull request was deployed and Sentry observed the following issues:\n\n- ‼️ **TypeError** `sentry.tasks.derive_code_mappings.derive_code_m...` [View Issue](https://sentry.sentry.io/issues/?referrer=github-pr-bot)\n- ‼️ **KafkaException** `query_subscription_consumer_process_message` [View Issue](https://sentry.sentry.io/stats/?referrer=github-pr-bot)\n\n<sub>Did you find this useful? React with a 👍 or 👎</sub>"
+        expected_comment = """## Suspect Issues


Splitting it over since it is much easier to read.

asottile-sentry · 2025-01-14T15:15:12Z

src/sentry/tasks/auto_source_code_config.py

-    name="sentry.tasks.derive_code_mappings.derive_code_mappings",
-    queue="derive_code_mappings",
+    name="sentry.tasks.auto_source_code_config.derive_code_mappings",


I believe this name needs to remain constant as well

You can't remove/change the name of a task if there are any spawn calls left in the code that is currently in production, as old web/workers will be spawning the old task, and that new task cannot be handled by new pods.

Instead, you should leave add the new task definition. You'll also need to retain the existing instrumented_task decorator and function signature. The 'old' function can call the new function to not duplicate logic.

If you want to avoid any task drops, you'd also need to not call the new task in these changes, as a new worker could spawn tasks that cannot be handled by an 'old' worker. Instead you could deploy the new task, and then separately deploy changes that start using the new task name.

Thank you both! I will look into it.

cathteng

the rename lgtm other than the task already mentioned

armenzg · 2025-01-14T19:15:19Z

src/sentry/tasks/auto_source_code_config.py

@@ -89,6 +92,20 @@ def process_error(error: ApiError, extra: dict[str, str]) -> None:
 def derive_code_mappings(
    project_id: int,
    data: NodeData,
+) -> None:
+    derive_code_mappings_new(project_id, data)


@markstory @asottile-sentry
Let me know if I understood your comments correctly.

armenzg · 2025-01-14T19:16:16Z

src/sentry/tasks/auto_source_code_config.py

+    default_retry_delay=60 * 10,
+    max_retries=3,
+)
+def derive_code_mappings_new(


In a follow-up PR I will update the queue and name for this function.

markstory · 2025-01-15T18:39:38Z

src/sentry/tasks/post_process.py

@@ -1016,7 +1017,7 @@ def process_code_mappings(job: PostProcessJob) -> None:
        else:
            return

-        derive_code_mappings.delay(project.id, event.data)
+        auto_source_code_config.delay(project.id, event.data)


You could drop task activations here. During a deploy workers are replaced incrementally, and a 'new' worker could spawn this task, which is processed by an 'old' worker. The old worker won't have the new task definition and the activation will be lost.

What I've done in the past for this kind of situation is use an option to choose which task is scheduled. Once all workers are restart, the option can be flipped and the new task will be scheduled. This helps clean up to be done in one pull request and deploy instead of needing to use two or three.

armenzg · 2025-01-15T21:06:10Z

src/sentry/features/temporary.py

@@ -34,6 +34,8 @@ def register_temporary_features(manager: FeatureManager):

    # Enables user registration.
    manager.add("auth:register", SystemFeature, FeatureHandlerStrategy.INTERNAL, default=True)
+    # Switch to new queue for auto source code config
+    manager.add("new-auto-source-code-config-queue", SystemFeature, FeatureHandlerStrategy.FLAGPOLE)


This is the option to control the queue.

armenzg · 2025-01-15T21:06:57Z

src/sentry/tasks/post_process.py

+        if features.has("new-auto-source-code-config-queue"):
+            auto_source_code_config.delay(project.id, data=event.data, event_id=event.event_id)
+        else:
+            derive_code_mappings.delay(project.id, data=event.data, event_id=event.event_id)


@markstory please let me know if I got this right.

armenzg · 2025-01-15T21:07:41Z

tests/sentry/tasks/test_post_process.py

@@ -375,6 +375,28 @@ def test_skipping_an_issue_doesnt_mark_it_processed(self, mock_derive_code_mappi
            self._call_post_process_group(charlie_event2)
            assert mock_derive_code_mappings.delay.call_count == 2

+    # XXX: Delete this test once we've migrated


@markstory these two tests will be deleted in the future but it tests that the right queue is used.

armenzg added 5 commits January 13, 2025 12:10

Move task to new location

9431901

Refactor get_organization_installation

ceeee90

Move code_mapping file

d104ce6

Fix a bunch of renaming and moving files

2d8d091

Typing and minor fixing

af5248d

armenzg self-assigned this Jan 13, 2025

github-actions bot added the Scope: Backend Automatically applied to PRs that change backend components label Jan 13, 2025

vercel bot deployed to Preview January 13, 2025 18:07 View deployment

armenzg added 5 commits January 13, 2025 14:18

Minor typing

803be9d

Fix patching

4d8f431

Fix test

6c9471a

Move test plus fix patching

fee1c2b

Include tests.sentry.issues as owned by our team

1e1829d

vercel bot deployed to Preview January 14, 2025 13:57 View deployment

armenzg added 3 commits January 14, 2025 09:08

Make it all singular

a56e3a8

Reduce changes

f2ce573

Fix test

39bcaa3

vercel bot deployed to Preview January 14, 2025 14:46 View deployment

armenzg commented Jan 14, 2025

View reviewed changes

armenzg changed the title ~~WIP~~ ref(derived_code_mappings): Introduce new naming: Automatic Source Code Configuration Jan 14, 2025

armenzg changed the title ~~ref(derived_code_mappings): Introduce new naming: Automatic Source Code Configuration~~ ref(derived_code_mappings): Rename to Automatic Source Code Configuration Jan 14, 2025

armenzg added 2 commits January 14, 2025 10:02

Update codeowners

8ccca3d

Fix tests

551588c

armenzg marked this pull request as ready for review January 14, 2025 15:13

armenzg requested review from a team as code owners January 14, 2025 15:13

asottile-sentry reviewed Jan 14, 2025

View reviewed changes

vercel bot deployed to Preview January 14, 2025 15:16 View deployment

Rename back

ebf3d8a

cathteng reviewed Jan 14, 2025

View reviewed changes

armenzg added 2 commits January 14, 2025 14:12

Old and new task

8d0fc95

Merge branch 'master' into ref/auto_source_config/armenzg

790af64

armenzg commented Jan 14, 2025

View reviewed changes

Remove SUPPORTED_LANGUAGES import

e2d3b6e

vercel bot deployed to Preview January 14, 2025 19:22 View deployment

Fix name

3afe2d7

vercel bot deployed to Preview January 15, 2025 01:08 View deployment

armenzg requested review from asottile-sentry and markstory January 15, 2025 02:00

armenzg added 2 commits January 15, 2025 13:08

Merge branch 'master' into ref/auto_source_config/armenzg

09c0e1e

Use new queue

338a105

vercel bot deployed to Preview January 15, 2025 18:18 View deployment

markstory reviewed Jan 15, 2025

View reviewed changes

Use flag to switch over

713332d

vercel bot deployed to Preview January 15, 2025 20:19 View deployment

Add test

a723dc5

vercel bot deployed to Preview January 15, 2025 20:56 View deployment

Merge branch 'master' into ref/auto_source_config/armenzg

5a68731

armenzg commented Jan 15, 2025

View reviewed changes

armenzg requested a review from markstory January 15, 2025 21:07

vercel bot deployed to Preview January 15, 2025 21:08 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ref(derived_code_mappings): Rename to Automatic Source Code Configuration #83313

ref(derived_code_mappings): Rename to Automatic Source Code Configuration #83313

armenzg commented Jan 13, 2025 •

edited

Loading

codecov bot commented Jan 13, 2025 •

edited

Loading

armenzg Jan 14, 2025

armenzg Jan 14, 2025

armenzg Jan 14, 2025

armenzg Jan 14, 2025

armenzg Jan 14, 2025

armenzg Jan 14, 2025

armenzg Jan 14, 2025

asottile-sentry Jan 14, 2025

markstory Jan 14, 2025

armenzg Jan 14, 2025

cathteng left a comment

armenzg Jan 14, 2025

armenzg Jan 14, 2025

markstory Jan 15, 2025

armenzg Jan 15, 2025

armenzg Jan 15, 2025

armenzg Jan 15, 2025

		Queue("derive_code_mappings", routing_key="derive_code_mappings"),
		Queue("auto_source_code_config", routing_key="auto_source_code_config"),

ref(derived_code_mappings): Rename to Automatic Source Code Configuration #83313

Are you sure you want to change the base?

ref(derived_code_mappings): Rename to Automatic Source Code Configuration #83313

Conversation

armenzg commented Jan 13, 2025 • edited Loading

codecov bot commented Jan 13, 2025 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cathteng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

armenzg commented Jan 13, 2025 •

edited

Loading

codecov bot commented Jan 13, 2025 •

edited

Loading