Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix stack dump on shutdown #1023

Merged
merged 2 commits into from
Nov 1, 2024
Merged

Conversation

martin-purplefish
Copy link
Contributor

When shutting down a job, I'd see:

2024-11-01 10:08:47,004 ERROR [livekit.agents] [log.py:21] [trace_id=1def6cfeeb92063a45eb46e9e9500303 span_id=d183818fae3b8fa7 resource.service.name= trace_sampled=True] - Error in _read_ipc_task                                                                                                                                                                                                                                                                                                    
Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 35, in recv_bytes
    len_bytes = await self._reader.readexactly(4)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/streams.py", line 750, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/job_main.py", line 211, in _read_ipc_task
    msg = await channel.arecv_message(cch, proto.IPC_MESSAGES)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/channel.py", line 47, in arecv_message
    return _read_message(await dplx.recv_bytes(), messages)
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 43, in recv_bytes
    raise DuplexClosed()
livekit.agents.utils.aio.duplex_unix.DuplexClosed
2024-11-01 10:08:47,004 ERROR [livekit.agents] [log.py:21] [trace_id=2d6cb01bbbe36fb6a5eeb5d8dda86246 span_id=1fcf9fb9b3a0edb4 resource.service.name= trace_sampled=True] - Error in _read_ipc_task
Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 35, in recv_bytes
    len_bytes = await self._reader.readexactly(4)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/streams.py", line 750, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/job_main.py", line 211, in _read_ipc_task
    msg = await channel.arecv_message(cch, proto.IPC_MESSAGES)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/channel.py", line 47, in arecv_message
    return _read_message(await dplx.recv_bytes(), messages)
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 43, in recv_bytes
    raise DuplexClosed()
livekit.agents.utils.aio.duplex_unix.DuplexClosed
2024-11-01 10:08:47,004 ERROR [livekit.agents] [log.py:21] [trace_id=52453bc25471c64df1daa34327a4df39 span_id=4e4334920ba359f6 resource.service.name= trace_sampled=True] - Error in _read_ipc_task
Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 35, in recv_bytes
    len_bytes = await self._reader.readexactly(4)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/homebrew/Cellar/[email protected]/3.12.7_1/Frameworks/Python.framework/Versions/3.12/lib/python3.12/asyncio/streams.py", line 750, in readexactly
    raise exceptions.IncompleteReadError(incomplete, n)
asyncio.exceptions.IncompleteReadError: 0 bytes read on a total of 4 expected bytes

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/log.py", line 16, in async_fn_logs
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/job_main.py", line 211, in _read_ipc_task
    msg = await channel.arecv_message(cch, proto.IPC_MESSAGES)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/ipc/channel.py", line 47, in arecv_message
    return _read_message(await dplx.recv_bytes(), messages)
                         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/martin/Library/Caches/pypoetry/virtualenvs/agent-x_OPPxuI-py3.12/lib/python3.12/site-packages/livekit/agents/utils/aio/duplex_unix.py", line 43, in recv_bytes
    raise DuplexClosed()
livekit.agents.utils.aio.duplex_unix.DuplexClosed

Copy link

changeset-bot bot commented Nov 1, 2024

🦋 Changeset detected

Latest commit: 4cfa8f2

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
livekit-agents Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@martin-purplefish
Copy link
Contributor Author

PTAL @theomonnom @davidzhao

@martin-purplefish
Copy link
Contributor Author

Closing this for now - this might be our highlight.io integration breaking the contextlib.suppress

@martin-purplefish
Copy link
Contributor Author

Not highlight - I think this is right?

@theomonnom
Copy link
Member

Thanks!

@theomonnom theomonnom merged commit 2ff2493 into livekit:main Nov 1, 2024
4 checks passed
This was referenced Oct 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants