Repository navigation
Conversation
AudioFrame.protoInfo() and VideoFrame.dataPtr built the pointer from `new Uint8Array(this.data.buffer)`, which drops the view's byteOffset. Any frame whose data is a subarray() of a larger buffer, or a view into a pooled Node Buffer, handed native code the start of the whole ArrayBuffer. AudioResampler, AudioSource/VideoSource.captureFrame and VideoFrame.convert/getPlane then read the wrong bytes: the sample count was right but the content was the head of the buffer, so 10ms frames sliced from a clip resampled to near-silence. Pass a Uint8Array over exactly the view's bytes instead. The native binding already returns the view's own data pointer, offset included. Fixes livekit#679
🦋 Changeset detectedLatest commit: b94a173 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #679
AudioFrame.protoInfo()andVideoFrame.dataPtrpassretrievePtr(new Uint8Array(this.data.buffer))to native code. That view starts at byte 0 of the backingArrayBuffer, sodata.byteOffsetis dropped. When a frame is built from asubarray()(which theAudioFrameconstructor comment suggests) or from a pooled NodeBuffer, native code reads from the start of the backing buffer instead of the frame's own samples.AudioResampler.push()still passes the right size, so the output length looks correct, but every push resamples the head of the buffer. That explains the issue: 10 ms frames sliced from a clip come out near-silent (presumably the clip starts quiet), while 1 s pushes look roughly fine. The same pointer is used byAudioSource.captureFrame,VideoSource.captureFrame,VideoFrame.convert()andVideoFrame.getPlane().Fix
Create the view over exactly the frame's bytes:
The native side returns the typed array's
as_ptr(), which already accounts for the view's offset (on Node 22,buf.subarray(n)gives base + n), so no extra pointer arithmetic is needed. Frames that start at offset 0 get the same pointer as before.Tests
audio_frame.test.tssubarray()-backed frame back through native code (copyBuffer(protoInfo().dataPtr, byteLength)) and expects the frame's own samplessubarray()frames vs.slice()copies throughAudioResampler(16000, 24000), asserting the residual is below 1% of the signal RMS. It's a tolerance rather than exact equality because soxr dithers 16-bit output, so two runs over the same input can differ by 1–2 LSB.video_frame.test.ts: an I420 frame that starts halfway into its backing buffer;getPlane(0..2)must return its own Y/U/V bytes.All three fail on
mainand pass with this change. With the issue's repro on a synthetic 16 kHz clip that starts with 300 ms of near-silence (input RMS ≈ 1531):mainpnpm --filter @livekit/rtc-node testpasses (66 passed, 30 e2e tests skipped without credentials), and lint andformat:checkare clean. A patch changeset for@livekit/rtc-nodeis included.