CLUE Switching Mixer Example

This document presents an example multipoint use case scenario for CLUE. This example uses the media switching variety of the Topo-Mixer RTP topology. This example is intended to promote discussion about how to implement it using the CLUE Framework, and whether or not the framework as currently defined is sufficient to enable this use case. From the requirements document: “REQMT-13: The solution MUST support both transcoding and switching approaches to providing multipoint conferences.” This example uses the switching approach. says media-switching mixer is one of the RTP topologies relevant for CLUE. The media switching variety of Topo-Mixer is described in section 3.6.2 of . In this topology, the mixer provides one or more conceptual sources selecting one source at a time from the original sources. The mixer creates a conference-wide RTP session by sharing remote SSRC values as CSRCs to all conference participants. The basic scenario for this example is a multipoint conference consisting of some traditional single-camera single-screen endpoints and some 3-camera multi-screen endpoints. Each endpoint receives multiple Capture Encodings that originated from several other endpoints. The multi-screen endpoints show the currently speaking endpoint’s video using a large area of the display screens, and also show other recent speakers in smaller size using less screen space. Since the middlebox (the mixer) is of the switching variety it is not doing any video composition. The endpoints are responsible for composing video streams to be rendered on the endpoint’s display screens. The mixer sends several Capture Encodings to each endpoint, with those Capture Encodings originally coming from several other endpoints. So each endpoint receives many capture encodings, representing Media Captures that originate at other endpoints. The multi-camera endpoints send multiple Media Captures, while the single-camera endpoints send just one Media Capture. Each Media Capture could have multiple Capture Encodings, however. The mixer selects which original sources it sends to the endpoints based on speech activity, using a policy defined by the mixer. When completed, this example should be added to the examples in the Framework.

From the human user’s point of view, this example is a more specific case of the general multipoint scenario in [ref use cases]. Consider a conference with these endpoints: Endpoint A – 4 screens, 3 cameras Endpoint B – 3 screens, 3 cameras Endpoint C – 3 screens, 3 cameras Endpoint D – 3 screens, 3 cameras Endpoint E – 1 screen, 1 camera Endpoint F – 2 screens, 1 cameras Endpoint G – 1 screen, 1 camera This example focuses on what the user in one of the 3-camera multi-screen endpoints sees. Call this person User A, at Endpoint A. There are 4 large display screens at Endpoint A. Whenever somebody at another site is speaking, all the video captures from that endpoint are shown on the large screens. If the talker is at a 3-camera site, then the video from those 3 cameras fills 3 of the screens. If the talker is at a single-camera site, then video from that camera fills one of the screens, while the other screens show video from other single-camera endpoints. User A can also see video from other endpoints, in addition to the current talker, although much smaller in size. Endpoint A has 4 screens, so one of those screens shows up to 9 other Media Captures in a tiled fashion. User B at Endpoint B sees a similar arrangement, except there are only 3 screens, so the 9 other Media Captures are spread out across the bottom of the 3 displays, in a picture-in-picture (PIP) format. When somebody at a different endpoint becomes the current talker, then User A and User B both see the video from the new talker appear on their large screen area, while the previous talker takes one of the smaller tiled or PIP areas. The person who is the current talker doesn’t see themselves, they see the previous talker in their large screen area. TBD - Diagrams go here

The Media Producer in the mixer sends a CLUE Advertisement to each endpoint in the conference. There are different possibilities for how the mixer might construct advertisements.

The Producer in the mixer can advertise one Capture Scene, with many Capture Scene Entries (CSE), each with a different number of Media Captures. Say the Producer wants to send up to 12 Media Captures, it could advertise one CSE with 12 switched captures, one with 11, one with 10, etc. These switched Media Captures are distinct from the Media Captures sent from the endpoints. But these switched media captures get their media from those endpoint Media Captures (really their encodings). A Consumer could then pick the CSE that had the number of Media Captures the Consumer wants to receive. Questions: What does it mean about spatial relationships, when there is a CSE with 12 switched captures? Does the Producer include any spatial information for this type of CSE?

The Producer could advertise multiple scenes, each one representing a different level in the recent talker list. Each Scene could have a CSE with 3 Media Captures. Scene 1: current and most recent talkers Scene 2: next most recent talkers Scene 3: next most recent talkers Scene 4: next most recent talkers The grouping of Media Captures, 3 at a time, into the CSEs in each Scene indicates the mixer is responsible for maintaining a useful spatial relationship between the original source Media Captures it switches into these conceptual Media Captures. The mixer provides spatial information, probably using “no scale” coordinates. The mixer should use the priority attribute to indicate the Media Captures in Scene 1 are highest priority, Scene 2 is next highest, and so on. Questions: How does the Provider know to use CSEs with 3 captures? Why not some other number? Or could it also work if it included multiple CSEs, say with 1 to 4 captures?

What other ways should be considered?

This section describes how the Endpoint Consumer selects Media Captures from the advertisement.

The multi-screen Consumer knows it wants to receive 12 captures, so it picks the Media Captures in the CSE that has 12 captures. A single screen endpoint might choose to receive only 1 capture, or maybe 3 or 4, depending on how it wants to render video for showing to the user.

The multi-screen Consumer knows it wants to receive 12 captures, so it picks the Media Captures with the highest priority first (one CSE of 3 captures), then the next highest (another CSE in another Scene with 3 captures), and so on until it has picked 12 captures.

Questions: How does the consumer know the actual spatial relationships between all the captures it receives? All 12 captures don’t really have a combined relationship with each other. Does the Consumer need to have access to all the spatial relationships advertised by the Media Producers in all the other endpoints? If so, how is this information distributed throughout the conference? In CLUE messages? Extension to SIP Event Package and XCON Data Model?

For the multiple scene approach, the spatial relationships can be handled in a straightforward manner by the spatial attributes the mixer puts in its advertisement. The mixer can ensure that when it switches media captures from a multi-camera source into its outgoing captures, it puts them together in the correct order that it described in the advertisement. And when it switches captures from single-camera sources, it could also pick multiple single camera sources and assign them to a consistent conceptual spatial relation, even though they don’t have a real physical relationship. The Consumer renderer assigns the scene with highest priority captures to the largest areas on its display screens, and it assigns each other scene to the smaller areas on its screens. These assigments can remain static, they don’t need to change when the mixer switches between sources for these media captures.

Add audio considerations – how to switch and render audio consistent with video. Add audio to the example Consider how the scene-switch-policy attribute can be used with this scenario

Thanks to Stephan Wenger, Rob Hansen, and Andy Pepperell for contributing to the ideas in this example.