Add support for Simulcast/SVC.

[cloudwebrtc]

Summary

Add multiple resolution/rate support for the video track.

Motivation

In a video conference or live broadcast scenario, multiple resolutions need to be generated at the same time on the same PC, and different devices can subscribe to the resolution they want.

SVC can adapt to network transmission by dynamically adjusting resolution and bit rate during the SFU Relay process.

Describe alternatives you've considered

mediasoup already implements similar functionality, but it uses C ++

Additional context

[Sean-Der]

@cloudwebrtc that would be an amazing feature! Also would be really nice to just have it be on the Track

I will need to think about a little bit though. I need to do some research/API thinking and will come back on this!

thanks for filing, this could really open up some cool things.

[sgotti]

@Sean-Der I started studying how adding simulcast will work and doing an initial implementation. Since it'll break various pion packages api (primarily webrtc and srtp) (like also discussed in #1025 for track api) I'd like to propose a design doc to cover and explain the required changes and discuss the possible options. Is the wiki the right place (a page like https://github.com/pion/webrtc/wiki/PlanningV3) to add and discuss such design doc or do you have something better in mind?

[Sean-Der]

@sgotti Yes the wiki is the best place, and please edit away!

I don't think it will be a substantial amount of work, but I am just timid to change the API and not do it right. The /v2 API was dramatically different then the original one, and introduced a whole new set of different issues :(

[sgotti]

@Sean-Der

I don't think it will be a substantial amount of work, but I am just timid to change the API and not do it right. The /v2 API was dramatically different then the original one, and introduced a whole new set of different issues :(

I'll start writing something here the keep you updated, I'll try to summarize my work until now and explain why we need a different pion Track API.

I have an initial working implementation of simulcast handling and, to be honest, is a quite big amount of work. I'll clean it up and split in multiple patches and then publish it in the next days so you can take a look at it.

It's a big amount of work because it has to change the way tracks are handled. We have to handle mid, rid rtp/rtcp extensions, sdp extmap, rid and simulcast attributes.

The biggest issue, is related to the current pion implementation of Track. I don't know why it's called Track instead of MediaTrack but it has some common fields with webrtc official MediaTrack (it's provided by an OnTrack event, it has kind, id, label fields) but it's also quite different since it has to expose low levels api to directly access the related rtp streams (really required for implementing and SFU or, practically, all the pion examples).
In webrtc a media track is a source or sink of "raw" data: i.e. data that has already been decoded by the transceiver receiver or that will be encoded by the transceiver sender. In pion the receiver and sender don't do any sort of encoding/coding and their rtp/rtcp stream could be read/written using additional pion Track methods (obviously not provided by the webrtc api). Track is also used, on the sending side, for generating the offer and so adds additional fields (like ssrc), while in the webrtc spec this behavior is done using AddTransceiver options (like using sendEncoding to enable simulcast).

Additionally currently track is tied to a single SSRC (in webrtc spec API the rtp stream ssrc is completely hidden). Also without simulcast this isn't quite right since in a mediatrack we could receive multiple streams also without simulcast (i.e the repaired stream) that are currently skipped and it's not possible to send them.
With simulcast this is much more evident since a receiver/sender will receive/send multiple rtp streams. Continuing on the current pion Track implementation it needs to be refactored to handle multiple streams. So a track will have a kind, id and label but it'll have multiple rtp stream everyone with its own rid (streamId), ssrc (that could also change!), payload/codec.

Additionally on the sdp side, a simulcast media track won't provide ssrc information anymore (since ssrc could change due to different condition explained on the various rfc) and instead you'll receive information on the rtp stream ids (rid attributes) and how they'll compose the media track (simulcast attribute). So you won't know the rtp streams ssrc of a media track at negotiation time but you'll have to assign the incoming rtpreadstream for an incoming ssrc to a specific receiver when receiving the first packet, inspecting the rtp header mid and streamId extensions and map them to the right transceiver and the right stream.

[cretz]

(edit: my issues, and I suspect issues that are going to come up w/ the simulcast impl, are bitrate/bandwidth limitations enforced by Chrome based on their bandwidth estimation which I believe is handled via RTCP...I am still researching and will get back)

[cretz]

Ok, I hacked together a simulcast example in a branch on my fork. Quick notes:

• I need this functionality for my SFU. I don't really expect this to be a PR or get merged because it is a bit of a hack.
• I needed to accept undeclared SSRCs, yet the regular code only did it if there was a single media description which wasn't acceptable for my example because I also have a data channel. So I added a little config value.
• I was wondering why Chrome wasn't sending all three streams (see edits for my previous comment). Well, turns out they have a default max bitrate (300kbps) and if you don't work w/ their prober over RTCP, it will just skip higher-bitrate items. To alleviate, instead of doing a bunch of measurements, I just have a hardcoded 5mbps via a ReceiverEstimatedMaximumBitrate RTCP message. In a more serious setting, I'd do some measurements based on the reports.
• I only tested this in Chrome. FF surely doesn't work yet, but if I change it for them to use sender set parameters (or use webrtc-adapter), and do a few different things on the SDP, it'll work I'm sure.

[sgotti]

Just as an update, simulcast works correctly on both chrome and firefox. Before opening a PR I'm trying to split all my changes in many small patches so they can be easily reviewed just because there're many changes required to handle tracks with multiple sources and all the features required for simulcast (extmaps negotiation, handling rid based streams etc...).

I also have two track proposals to make them multistream. One is compatible with v2 API (but it's not the cleanest one), the other one is a proposal for a future v3 api.