We've identified a memory leak when multiple publishers share a single room. We can reproduce this against livekit 0.7.49 and livekit-server v1.9.0:
- Run
livekit-server and have two publishers A and B join the room as separate processes, both publishing an audio track.
- Have publisher
A close their connection first with B still publishing and have B close a few seconds later.
- Repeat this for many iterations.
You'll observe proc-A leaking memory on the live heap (check stats.allocated on jemalloc), roughly 70kb per join. This behavior is flipped if we have B close the connection first (proc-B will leak in that case).
However, we have noticed that if each side explicitly calls unpublish on its own tracks before closing the connection, the leak disappears entirely. This feels like a workaround however, instead of the correct way to address the core issue (i.e. the remote publisher does not get cleaned up on close() for some reason).
I assume there are some deeper intricacies in the connection closing process we are not privy to, and I'm happy to provide a simple gist or self-contained repro if useful. Notably, this does not reproduce if A and B are in a single process.
Note: The linked PR does resolve the leak when trying to locally reproduce, but not sure if this is the best approach.
We've identified a memory leak when multiple publishers share a single room. We can reproduce this against
livekit 0.7.49andlivekit-server v1.9.0:livekit-serverand have two publishersAandBjoin the room as separate processes, both publishing an audio track.Aclose their connection first withBstill publishing and haveBclose a few seconds later.You'll observe
proc-Aleaking memory on the live heap (checkstats.allocatedon jemalloc), roughly 70kb per join. This behavior is flipped if we haveBclose the connection first (proc-Bwill leak in that case).However, we have noticed that if each side explicitly calls
unpublishon its own tracks before closing the connection, the leak disappears entirely. This feels like a workaround however, instead of the correct way to address the core issue (i.e. the remote publisher does not get cleaned up onclose()for some reason).I assume there are some deeper intricacies in the connection closing process we are not privy to, and I'm happy to provide a simple gist or self-contained repro if useful. Notably, this does not reproduce if
AandBare in a single process.Note: The linked PR does resolve the leak when trying to locally reproduce, but not sure if this is the best approach.