ModuleNotFoundError: No module named 'pkg_resources'

[jkotas]

Build

https://dev.azure.com/dnceng-public/cbb18261-c48f-4abb-8651-8cdcb5474649/_build/results?buildId=910866

Build leg reported

ComInterfaceGenerator.Tests.WorkItemExecution

Pull Request

dotnet/runtime#110558

Known issue core information

Fill out the known issue JSON section by following the step by step documentation on how to create a known issue

 {
    "ErrorMessage" : "ModuleNotFoundError: No module named 'pkg_resources'",
    "BuildRetry": false,
    "ErrorPattern": "",
    "ExcludeConsoleLog": false
 }

@dotnet/dnceng

Release Note Category

• Feature changes/additions
• Bug fixes
• Internal Infrastructure Improvements

Release Note Description

Additional information about the issue reported

No response

Known issue validation

Build: 🔎 https://dev.azure.com/dnceng-public/public/_build/results?buildId=910866
Error message validated: [ModuleNotFoundError: No module named 'pkg_resources']
Result validation: ✅ Known issue matched with the provided build.
Validation performed at: 1/8/2025 10:20:43 PM UTC

Report

Build	Definition	Test	Pull Request
912698	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	dotnet/runtime#111259
912093	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#110736
912683	dotnet/runtime	System.Diagnostics.TraceSource.Config.Tests.WorkItemExecution	dotnet/runtime#111136
912677	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#111257
912500	dotnet/runtime	System.Xml.Schema.Extensions.Tests.WorkItemExecution	dotnet/runtime#108542
912659	dotnet/runtime	System.Diagnostics.Debug.Tests.WorkItemExecution	dotnet/runtime#110676
912635	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#109378
912630	dotnet/runtime	System.Formats.Asn1.Tests.WorkItemExecution	dotnet/runtime#105004
2616941	dotnet-dotnet-monitor	Microsoft.Diagnostics.Monitoring.WebApi.UnitTests-net8.0-arm64.WorkItemExecution
912604	dotnet/runtime	System.Data.DataSetExtensions.Tests.WorkItemExecution
912586	dotnet/runtime	System.Dynamic.Runtime.Tests.WorkItemExecution	dotnet/runtime#107115
912573	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution
912570	dotnet/runtime	System.Globalization.Calendars.Tests.WorkItemExecution
912534	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#111255
912516	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#111254
912509	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution	dotnet/runtime#111253
912442	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#111108
911419	dotnet/runtime	System.Diagnostics.DiagnosticSource.Switches.Tests.WorkItemExecution	dotnet/runtime#110688
912421	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution
912412	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution	dotnet/runtime#108799
912382	dotnet/runtime	System.Diagnostics.Contracts.Tests.WorkItemExecution	dotnet/runtime#111170
912373	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution	dotnet/runtime#111214
912360	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution	dotnet/runtime#111245
912354	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	dotnet/runtime#104906
912090	dotnet/runtime	jit64_2.WorkItemExecution	dotnet/runtime#109493
912311	dotnet/runtime	System.Diagnostics.TraceSource.Config.Tests.WorkItemExecution	dotnet/runtime#111136
912307	dotnet/runtime	System.Diagnostics.FileVersionInfo.Tests.WorkItemExecution	dotnet/runtime#110676
912238	dotnet/runtime	System.Diagnostics.Contracts.Tests.WorkItemExecution	dotnet/runtime#111247
910866	dotnet/runtime	System.Formats.Nrbf.Tests.WorkItemExecution	dotnet/runtime#110558
912285	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#111218
912281	dotnet/runtime	System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution	dotnet/runtime#111093
912271	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#110818
912245	dotnet/runtime	System.Diagnostics.TextWriterTraceListener.Tests.WorkItemExecution	dotnet/runtime#111215
912224	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	dotnet/runtime#106309
912248	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#109530
912330	dotnet/runtime	Microsoft.XmlSerializer.Generator.Tests.WorkItemExecution	dotnet/runtime#111179
912206	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	dotnet/runtime#111244
912191	dotnet/runtime	System.Diagnostics.StackTrace.Tests.WorkItemExecution	dotnet/runtime#101264
912186	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution	dotnet/runtime#101264
912163	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#109091
912141	dotnet/runtime	Regressions.WorkItemExecution	dotnet/runtime#110268
912147	dotnet/runtime	Methodical_d1.WorkItemExecution	dotnet/runtime#110269
912144	dotnet/runtime	Regression_2.WorkItemExecution	dotnet/runtime#110271
912107	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution	dotnet/runtime#105004
912096	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	dotnet/runtime#111108
912074	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#110281
911863	dotnet/runtime	System.Xml.Linq.Streaming.Tests.WorkItemExecution	dotnet/runtime#106309
912039	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution	dotnet/runtime#111237
912035	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution	dotnet/runtime#111236
911999	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#111235
911922	dotnet/runtime	System.Globalization.CalendarsWithConfigSwitch.Tests.WorkItemExecution	dotnet/runtime#111209
911878	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	dotnet/runtime#110856
911835	dotnet/runtime	System.Text.Json.SourceGeneration.Roslyn4.4.Unit.Tests.WorkItemExecution
911855	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#107766
911813	dotnet/runtime	System.Linq.Parallel.Tests.WorkItemExecution
911799	dotnet/runtime	System.Net.Sockets.Tests.WorkItemExecution	dotnet/runtime#110692
911851	dotnet/runtime	System.Runtime.ReflectionInvokeEmit.Tests.WorkItemExecution	dotnet/runtime#111213
911830	dotnet/runtime	System.Formats.Asn1.Tests.WorkItemExecution
911816	dotnet/runtime	System.Formats.Cbor.Tests.WorkItemExecution
911754	dotnet/runtime	System.Formats.Asn1.Tests.WorkItemExecution	dotnet/runtime#111229
911757	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#111230
911742	dotnet/runtime	System.Diagnostics.StackTrace.Tests.WorkItemExecution	dotnet/runtime#107283
911722	dotnet/runtime	System.IO.FileSystem.DriveInfo.Tests.WorkItemExecution	dotnet/runtime#107118
911717	dotnet/runtime	System.Configuration.ConfigurationManager.Tests.WorkItemExecution	dotnet/runtime#111178
911709	dotnet/runtime	System.IO.UnmanagedMemoryStream.Tests.WorkItemExecution	dotnet/runtime#111227
911613	dotnet/runtime	System.IO.FileSystem.DriveInfo.Tests.WorkItemExecution	dotnet/runtime#111149
911700	dotnet/runtime	System.Net.Primitives.Functional.Tests.WorkItemExecution	dotnet/runtime#111222
910999	dotnet/runtime	System.Net.NameResolution.Functional.Tests.WorkItemExecution	dotnet/runtime#111213
911665	dotnet/runtime	System.Net.Primitives.Functional.Tests.WorkItemExecution	dotnet/runtime#111220
911636	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#111226
911627	dotnet/runtime	System.IO.Hashing.Tests.WorkItemExecution
911619	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution	dotnet/runtime#111225
911601	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution	dotnet/runtime#110780
911600	dotnet/runtime	System.Diagnostics.DiagnosticSource.Tests.WorkItemExecution
911590	dotnet/runtime	System.IO.FileSystem.DriveInfo.Tests.WorkItemExecution	dotnet/runtime#106309
911561	dotnet/runtime	System.Globalization.Calendars.Tests.WorkItemExecution	dotnet/runtime#111223
911564	dotnet/runtime	GC-scenarios1.WorkItemExecution	dotnet/runtime#111145
911557	dotnet/runtime	System.IO.UnmanagedMemoryStream.Tests.WorkItemExecution	dotnet/runtime#111222
911501	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution	dotnet/runtime#105004
911496	dotnet/runtime	System.DirectoryServices.Protocols.Tests.WorkItemExecution	dotnet/runtime#110935
911284	dotnet/runtime	System.ComponentModel.TypeConverter.Tests.WorkItemExecution	dotnet/runtime#111179
911179	dotnet/runtime	JIT_ro.WorkItemExecution	dotnet/runtime#109493
2616242	dotnet-dotnet-monitor	Microsoft.Diagnostics.Monitoring.WebApi.UnitTests-net8.0-x64.WorkItemExecution
2616241	dotnet-dotnet-monitor	Microsoft.Diagnostics.Monitoring.WebApi.UnitTests-net9.0-x64.WorkItemExecution
911297	dotnet/runtime	System.Diagnostics.TraceSource.Config.Tests.WorkItemExecution	dotnet/runtime#105004
911248	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#111218
911210	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution	dotnet/runtime#111220
911182	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#110736
911186	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#111108
911131	dotnet/runtime	System.Runtime.Extensions.Tests.WorkItemExecution
911128	dotnet/runtime	System.Formats.Tar.Manual.Tests.WorkItemExecution	dotnet/runtime#109207
2616084	dotnet-dotnet-monitor	Microsoft.Diagnostics.Monitoring.WebApi.UnitTests-net9.0-x64.WorkItemExecution
911117	dotnet/runtime	System.Drawing.Primitives.Tests.WorkItemExecution	dotnet/runtime#111219
911114	dotnet/runtime	System.Formats.Tar.Tests.WorkItemExecution	dotnet/runtime#109378
911029	dotnet/runtime	JIT.performance.WorkItemExecution
911052	dotnet/runtime	System.Threading.Timer.Tests.WorkItemExecution	dotnet/runtime#109364
911033	dotnet/runtime	iOS.Simulator.Aot-Llvm.Test.WorkItemExecution
911068	dotnet/runtime	System.Diagnostics.Tools.Tests.WorkItemExecution	dotnet/runtime#111204
911046	dotnet/runtime	System.Diagnostics.TraceSource.Tests.WorkItemExecution	dotnet/runtime#111209
911020	dotnet/runtime	System.Diagnostics.Tracing.Tests.WorkItemExecution	dotnet/runtime#111215
Displaying 100 of 106 results

Summary

24-Hour Hit Count	7-Day Hit Count	1-Month Count
51	106	106

[dougbu]

we're tracking work on this in #4751. we found the problem another way and noticed it seems to be specific to ArmArch Linux Docker containers just yesterday. hoping to move away from the deprecated and, now, apparently sometimes unavailable package in time to include the fix in our next rollout

am I correct the containers failing in your build are quite infrequently used❓

[jkotas]

This is failing on many PRs in dotnet/runtime. The affected containers are used by default dotnet/runtime CI configuration.

[jkotas]

This is failing on many PRs in dotnet/runtime

You can see it in the stats in the top post.

[jkotas]

it seems to be specific to ArmArch Linux

This affects number of Linux and macOS variants.

For example, here is a log from macOS x64: https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/Microsoft.Extensions.Configuration.Tests/1/console.47b8555d.log?helixlogtype=result

[akoeplinger]

I think the common factor is having Python 3.12, e.g. if you look at https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/ComInterfaceGenerator.Tests/1/console.aa96da09.log?helixlogtype=result from the same job as Jan posted above it worked, because the dci-mac-build-133 macOS machine is not using Python 3.12 (I guess because it wasn't updated yet)

[dougbu]

I think the common factor is having Python 3.12, e.g. if you look at https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111218-merge-42cc8e62a78b4da997/ComInterfaceGenerator.Tests/1/console.aa96da09.log?helixlogtype=result from the same job as Jan posted above it worked, because the dci-mac-build-133 macOS machine is not using Python 3.12 (I guess because it wasn't updated yet)

I agree. it also seems like the problem is only partially under our direct control. we use pkg_resources in a couple of places but so does the azure package version we rely on. other dependencies seem to have the right try / catch setup to fallback when pkg_resources isn't available.

---

unless we missed some older builds that failed, problems started just after our rollout began updating queues yesterday. the earliest error I saw had timestamp 2025-01-08T21:20:05.2091700Z and the queue rollout started real work at 2025-01-08T20:45:17.1221544Z (slightly late due to retries of earlier jobs in our pipeline). the rollout picked up new OS packages for Python3 on Linux machines as well as slightly changed Python requirements for our helix-scripts/ code. the second part might have impacted OSX machines

[dougbu]

Python 3.12.8 released on 2024-12-03. it contains a bunch of fixes and dependency updates though nothing obviously linked to pkg_resources. it did upgrade its bundled pip to 24.3.1 but the pip changelog also doesn't include an obvious smoking gun. it also upgraded its libexpat dependency to 2.6.3 but that's written in C

I checked the setuptools changelog as well b/c we let that float as much as the Python and pip versions allow. nothing obvious there either

[dougbu]

we're reverting yesterday's rollout due to the problems discussed in this issue. queues are getting updated as I type this note

[dougbu]

revert is now complete but I don't see a clear signal that this particular problem has been resolved. please let us know

[akoeplinger]

Seems to be working again. Though I noticed that e.g. a build on dci-mac-build-108 which failed before is still printing No module named 'pkg_resources', but it seems to not be an error anymore.

https://helixr1107v0xdeko0k025g8.blob.core.windows.net/dotnet-runtime-refs-pull-111257-merge-ab67e12681af4b7b92/System.Collections.NonGeneric.Tests/1/console.05a00ae9.log?helixlogtype=result