gr.WebRTC design notes

How WebRTC streams work in Gradio after PR #13965: the component, the queue changes it relies on, the process executor, and gr.Server with the JS client. This page explains the design; the PR description lists the changes to shared code and what to review.

Describes the PR as of commit a2df195. Part of #13907.

1. The idea in one paragraph

Until now, WebRTC in a Gradio app meant the external FastRTC library, whose component connected each browser directly and kept its own limits: past concurrency_limit a visitor was simply rejected. This PR moves the component into Gradio core as gr.WebRTC and makes each stream one queue event, from the moment the user presses Record until the stream ends. Media still goes over a peer connection and never enters the queue, but the queue decides when a stream may start and when it must end. So concurrency_limit and time_limit mean what they mean for any other event: a visitor beyond the limit waits in line with a place and an ETA, and starts by themselves when a slot frees up.

import gradio as gr
from gradio.webrtc import AdditionalOutputs

def process(frame, flip):
    out = frame[::-1] if flip else frame
    return out, AdditionalOutputs(frame.shape)

with gr.Blocks() as demo:
    webrtc = gr.WebRTC(mode="send-receive", modality="video")
    flip = gr.Checkbox(label="Flip")
    shape = gr.Textbox()
    webrtc.stream(process, inputs=[webrtc, flip], outputs=[webrtc, shape],
                  time_limit=60, concurrency_limit=2)

2. Four channels between the page and the server

A stream uses four separate channels. Only the first one is the queue; the media never passes through it.

3. The life of a stream

The diagram follows one stream from Record to the end. The four lanes are the page, the queue, the stream event's run (a generator the queue steps), and the Stream object in gradio/webrtc/connection.py that owns the peer connection.

• Waiting. Record sets the component value to a fresh 128-bit stream id and dispatches the private stream event. The page shows its place in line and a countdown; Cancel removes the event from the line.
• The turn. The event's first step admits the stream id. Later steps cannot admit another id, so one place in the concurrency limit holds one stream. The server answers signalling only for the browser session whose event admitted the stream, so a stream id seen by someone else is useless to them.
• Streaming. Each AdditionalOutputs the handler returns wakes the event, and the next step delivers it. Nothing has to wake the event just to keep it alive.
• The end. Whatever ends the stream, the peer connection closes at once, so the media stops on time even when every worker slot is busy. The event then delivers what is still pending and finishes, and the next stream in line gets the slot.

4. Stream states and how a stream ends

A Stream exists from its admission until its queue event is over, and it has a single way to end: close(). Every reason to stop goes through it, so the handler shuts down, the media stops and the pending outputs are delivered the same way each time.

Each stream has a watcher that checks its time limit, the connect timeout and whether its event is still live every 0.25 s. An error from the handler ends the stream and fails its event, so the page shows it once and .failure() runs, as for any event whose function raises.

5. Two kinds of slots

A running event normally holds two things from start to finish: a place in its function's concurrency_limit (the GPU, say) and one of the queue's worker slots (max_threads, 40 by default). A stream spends almost all of its life waiting for the next chunk or output, so holding a worker slot all that time let 40 open streams block every other event in the app. Streams now keep the first and give back the second while they wait.

The lease

A gr.WebRTC stream event is a lease (BlockFunction.lease): an async generator that the queue steps each time the event is woken. The queue does not time out its wait and does not cut it at its time limit; the stream keeps its own time limit, because the queue's cut would end the event before it delivers what is pending. A step that calls EventHandle.finish() is the last one, and its outputs go out with the completion. Only gr.WebRTC and server.webrtc() events are leases.

Taking a slot back

A waiting stream takes a worker slot back with take_worker_back when it has something to do, ahead of new events, so a busy queue does not stall running streams. gr.Audio and gr.Image streams get the same treatment between chunks, which is why the PR also changes their behavior.

RunStep

The route that runs a call used to infer from the event id whether the call continues the generator left by the event's previous call. Stream calls now keep their event id (so gr.Info and progress work after the first chunk), and the queue, which alone knows, says it explicitly with a RunStep: the event's first call, a call that continues the run, and whether to keep the run for later.

Place in line

For the ETA, a running stream counts as holding its place for its time_limit, so each stream ahead in line adds time_limit / concurrency_limit. Without a time limit there is no ETA. The line also moves up as soon as someone leaves it.

6. One code path for gr.WebRTC and gr.Server

The component and gr.Server register the same kind of event, and everything after that is shared. Signalling and inputs find a stream by its id in a registry, never through the component instance: a gr.update() gives a session a new copy of the component, and every session's copy has the same component id.

Inside @gr.render, .stream() raises: a render that runs again replaces the events it made, including a running stream's, and with the process executor the next call of the replaced event would fail.

7. With executor="process"

With the process executor (the default on 7.0-dev), the queue runs in a gateway process and the app's functions in app processes. A stream's function, its ConnectionManager, its handler and its media all live in the app process that admitted it, while the queue that steps it lives in the gateway.

• EventHandle. gr.WebRTC never touches the queue's Event directly. Queue.event_handle() gives the running function a small handle (wake, finish, stream_admitted, live), and in an app process ForwardingQueue.event_handle() gives a stand-in. What the function asks during a call comes back with the call's result; a wake between steps goes as an event_action message.
• Staying with its process. A stream's later calls, its signalling and its close go to the process that admitted it, and the session follows the stream until it ends, also through a drain. That process counts as active meanwhile, so the autoscaler does not retire a process with live media.
• A process that exits. Its streams fail at once instead of being admitted again by another process: the gateway wakes them when it loses the process.
• Liveness. Whether a stream event is live comes from the event itself, as with threads. A page whose heartbeat dropped, such as a phone tab in the background, is taken as closed by the server but may still be streaming and sending events: its running stream keeps going, and its next stream can start. A stream ends only when its queue event does.
• Load. A stream counts as one finished event for its process (/processes, replicas="auto"), however long it runs.

8. The page side

Following the run it started

Gradio.dispatch(event, data, observer) takes an optional observer that hears when the run of the event's function starts (with a handle that ends just that run), its statuses (place in line, ETA, stream state) and how it ended, exactly once on every path. gr.WebRTC uses it to know its turn and to end only its own run. A call without an observer behaves as before.

Ending on the page still finishes on the server

When the page ends a stream (Stop, CloseStream, a server-side end it notices), it asks the server to end it and reads the event to its end (close_stream({ read_to_end: true }) in the JS client), so the last outputs show and .success() or .failure() runs. Before, the page closed its iterator at once and the last outputs were lost; this applies to gr.Audio and gr.Image streams as well.

Reloads

A stream cannot continue in a new page. With session resume, a reloaded page now asks the server to end its resumed streams and reads what the server sends up to their end, instead of resuming them and leaving them to hold a slot until they time out.

gr.Server and the JS client

server.webrtc() registers a handler under an endpoint name, and the page streams to it with client.webrtc(), which joins the queue, takes the first message as the turn, connects, sends the inputs and delivers the outputs. The connection code of gr.WebRTC's frontend lives in @gradio/client and is exported for custom pages. When a handler takes gr.OAuthToken, the app gets the same sign-in routes as a Blocks app with a LoginButton, which is how the guide issues TURN credentials on Spaces.

const stream = app.webrtc("/flip", {
  media: await navigator.mediaDevices.getUserMedia({ video: true }),
  inputs: [1.0],
  on_track: (event) => (video.srcObject = event.streams[0]),
  on_output: ([mean]) => console.log(mean)
});
stream.set_inputs([1.5]);
await stream.stop();

9. Measurements

Headless Chromium on one machine, fake camera at 20 fps, a handler that returns each frame with AdditionalOutputs(server time), concurrency_limit=None. Output lag is the browser's clock minus the server time in the output.

With max_threads=2, two open streams used to exhaust both slots, and the plain button never returned. The extra lag with five streams comes from five browsers encoding video on one machine, not from waiting for a slot. With concurrency_limit=1, the second user starts 109 to 130 ms after the first stops.

10. For FastRTC users

What changes in code written for FastRTC's component:

• AdditionalOutputs(a, b) updates the components in outputs, in order; on_additional_outputs() is gone.
• Handlers get exactly the extra inputs: fn(frame, *extra) for video functions, fn(*extra) for receive-mode generators.
• .stream() takes a handler class (one instance per stream) or an instance with copy(). async def functions are refused; async code goes in an Async*StreamHandler.
• VideoStreamHandler(fn, fps, skip_frames) is renamed VideoOptions. VideoStreamHandler and AsyncVideoStreamHandler are now handler classes for video-only streams, with start_up() and shutdown(). skip_frames defaults to True.
• Video frames are RGB, like gr.Image's numpy values. Audio uses gr.Audio's numpy layout both ways.
• WebRTCError is gone: raise gr.Error. Errors, gr.Warning and gr.Info behave as in any event.
• Handlers and rtc_configuration functions can take gr.Request, gr.OAuthProfile and gr.OAuthToken.
• The look of the controls moves into stream_options=gr.StreamOptions(...); media constraints into webcam_options and microphone_options.
• update_connection() is gone: .stream() returns its event, so cancels=[stream_event] ends a stream.
• Not ported: variant="textbox", full_screen, send_input_on, the websocket and telephone transports, the TURN credential helpers, ReplyOnPause and the STT/TTS models.

Questions and comments go on the PR. The demo Spaces linked from the PR description show these streams running, including on ZeroGPU.