FrameCryptorStateMissingKey and Smoothness

[garfieldbear333]

I tried to use encryption and decryption functions in my live streaming project and found two problems. Can you help me with that?

1. On iOS platform, when viewers watch the anchor, they must first encrypt_enableEncryption before decrypting_enableDecryption in order to decrypt successfully. Otherwise, an error FrameCryptorStateMissingKey will occur.
2. When viewers encrypt and then successfully decrypt the video stream, the picture will freeze and the smoothness will decrease from 30 frames to 3 frames. If not encrypted, the picture will be smooth.

[cloudwebrtc]

hey @garfieldbear333, thanks for finding these bugs, do you have a minimal repro code? I may need to use debug compilation to track what is going on inside libwebrtc.

I tested it on flutter and it seems to work correctly, there is no frame rate drop

[cloudwebrtc]

1. On iOS platform, when viewers watch the anchor, they must first encrypt_enableEncryption before decrypting_enableDecryption in order to decrypt successfully. Otherwise, an error FrameCryptorStateMissingKey will occur.

The decrypted code here returns the logic of missingkey,

https://github.com/webrtc-sdk/webrtc/blob/e2ee/aes-key-derive-using-pbkdf2/api/crypto/frame_crypto_transformer.cc#L491-L505

I guess because the unencrypted frame does not contain the correct keyindex in frameTrailer.

   rtc::Buffer frameTrailer(2);
   frameTrailer[0] = date_in[date_in. size() - 2];
   frameTrailer[1] = date_in[date_in. size() - 1];
   uint8_t ivLength = frameTrailer[0];
   uint8_t key_index = frameTrailer[1];

[garfieldbear333]

The difference between demo and MediaSever in WebRTC is that demo uses local peerconnection to create offer and set answer, remote peerconnection creates answer and sets offer. However, in live streaming projects, MediaSever is used where both the host and the audience create an offer to the server which returns an answer to establish a connection. The code settings are almost the same as demo, only that peerconnection has a different responsibility

[garfieldbear333]

When decrypting, frameCyrptor.setKeyIndex(0) has already been set. If the decryption party does not encrypt first, the decryption will still fail. This problem only occurs on the iOS platform and not on Android.

  void _enableDecryption({bool video = false, bool enabled = true}) async {
    var receivers = await _localPeerConnection?.receivers;
    var kind = video ? 'video' : 'audio';
    receivers?.forEach((element) async {
      if (kind != element.track?.kind) return;
      var trackId = element.track?.id;
      var id = kind + '_' + trackId! + '_receiver';

      if (!_frameCyrptors.containsKey(id)) {
        var frameCyrptor =
            await _frameCyrptorFactory.createFrameCryptorForRtpReceiver(
                participantId: id,
                receiver: element,
                algorithm: Algorithm.kAesGcm,
                keyManager: _keyManager!);
        frameCyrptor.onFrameCryptorStateChanged = (participantId, state) =>
            print('DE onFrameCryptorStateChanged $participantId $state');
        _frameCyrptors[id] = frameCyrptor;
        await frameCyrptor.setKeyIndex(0);
      }

      var _frameCyrptor = _frameCyrptors[id];
      if (enabled) {
        await _frameCyrptor?.setEnabled(true);
        await _keyManager?.setKey(participantId: id, index: 0, key: aesKey);
      } else {
        await _frameCyrptor?.setEnabled(false);
      }
      await _frameCyrptor?.updateCodec(
          kind == 'video' ? "VP8" : "OPUS");
    });
  }