Communication Between Client and Server

mediasoup does not provide any signaling protocol to communicate clients and server. It's up to the application communicate them by using WebSocket, HTTP or whichever communication means, and exchange mediasoup related parameters, requests/responses and notifications between clients and server. In most scenarios this communication must be bidirectional so a full-duplex channel is usually required. However the application can reuse the same channel for non mediasoup related message exchange (such as authentication procedures, chat messages, file transfer and whatever the application wishes to implement).

• Guidelines for mediasoup-client and libmediasoupclient
  • Signaling and Peers
  • Device Loading
  • Creating Transports
  • Producing Media
  • Consuming Media
  • Producing Data (DataChannels)
  • Consuming Data (DataChannels)
  • Communicating Actions and Events
• Guidelines for FFmpeg and GStreamer
  • Producing Media from an External Endpoint (RTP In)
  • Consuming Media in an External Endpoint (RTP Out)
  • Example: Inject Audio and Video using FFmpeg
• Guidelines for SRTP Capable Endpoints
  • SRTP Endpoint as SDP Offerer
  • SRTP Endpoint as SDP Answerer
• Guidelines for DataChannel termination in Node.js
• Guidelines for node-sctp (SCTP/DataChannel in Node.js)

Guidelines for mediasoup-client and libmediasoupclient

Let's assume our JavaScript or C++ client side application instantiates a mediasoup-client Device or a libmediasoupclient Device object to connect to a mediasoup Router (already created in the server) and send and receive media over WebRTC.

Signaling and Peers

The application may use WebSocket and associate each authenticated WebSocket connection with a “peer”.

Device Loading

The client side application loads its mediasoup device by providing it with the RTP capabilities of the server side mediasoup router. See device.load().

Creating Transports

Both mediasoup-client and libmediasoupclient need separate WebRTC transports for sending and receiving. Typically, the client application creates those transports in advance, before even wishing to send or receive media.

For sending media:

• A WebRTC transport must be first created in the mediasoup router: router.createWebRtcTransport().
• And then replicated in the client side application: device.createSendTransport().
• The client application must subscribe to the “connect” and “produce” events in the local transport.

For receiving media:

• A WebRTC transport must be first created in the mediasoup router: router.createWebRtcTransport().
• And then replicated in the client side application: device.createRecvTransport().
• The client application must subscribe to the “connect” event in the local transport.

If SCTP (AKA DataChannel in WebRTC) are desired on those transports, enableSctp must be enabled in them (with proper numSctpStreams) and other SCTP related settings.

Producing Media

Once the send transport is created, the client side application can produce multiple audio and video tracks on it.

• The application obtains a track (e.g. by using the navigator.mediaDevices.getUserMedia() API).
• It calls transport.produce() in the local send transport.
  • The transport will emit “connect” if this is the first call to transport.produce().
  • The transport will emit “produce” so the application will transmit the event parameters to the server and will create a Producer instance in server side.
• Finally transport.produce() will resolve with a Producer instance in client side.

Consuming Media

Once the receive transport is created, the client side application can consume multiple audio and video tracks on it. However the order is the opposite (here the consumer must be created in the server first).

• The client application signals its device.rtpCapabilities to the server (it may have done it in advance).
• The server application should check whether the remote device can consume a specific producer (this is, whether it supports the producer media codecs). It can do it by using the router.canConsume() method.
• Then the server application calls transport.consume() in the WebRTC transport the client created for receiving media, thus generating a server side Consumer.
  • As explained in the transport.consume() documentation, it's strongly recommended to create the server side consumer with paused: true and resume it once created in the remote endpoint.
• The server application transmits the consumer information and parameters to the remote client application, which calls transport.consume() in the local receive transport.
  • The transport will emit “connect” if this is the first call to transport.consume().
• Finally transport.consume() will resolve with a Consumer instance in client side.

Producing Data (DataChannels)

Once the send transport is created, the client side application can produce multiple DataChannels on it.

• The application calls transport.produceData() in the local send transport.
  • The transport will emit “connect” if this is the first call to transport.produceData().
  • The transport will emit “producedata” so the application will transmit the event parameters to the server and will create a DataProducer instance in server side.
• Finally transport.produceData() will resolve with a DataProducer instance in client side.

Consuming Data (DataChannels)

Once the receive transport is created, the client side application can consume multiple DataChannels on it. However the order is the opposite (here the consumer must be created in the server first).

• The server application calls transport.consumeData() in the WebRTC transport the client created for receiving, thus generating a server side DataConsumer.
• The server application transmits the consumer information and parameters to the client application, which calls transport.consumeData() in the local receive transport.
  • The transport will emit “connect” if this is the first call to transport.consumeData().
• Finally transport.consumeData() will resolve with a DataConsumer instance in client side.

Communicating Actions and Events

When a transport, producer, consumer, data producer or data consumer is closed in client or server side (e.g. by calling close() on it), the application should signal its closure to the other side which should also call close() on the corresponding entity. In addition, the server side application should listen for the following closure events and notify the client about them:

• Transport “routerclose”. The client should call close() on the corresponding local transport.
• Producer “transportclose”. The client should call close() on the corresponding local producer.
• Consumer “transportclose”. The client should call close() on the corresponding local consumer.
• Consumer “producerclose”. The client should call close() on the corresponding local consumer.
• DataProducer “transportclose”. The client should call close() on the corresponding local data producer.
• DataConsumer “transportclose”. The client should call close() on the corresponding local data consumer.
• DataConsumer “dataproducerclose”. The client should call close() on the corresponding local data consumer.

The same happens when pausing a RTP producer or consumer in client or server side. The action must be signaled to the other side. In addition, the server side application should listen for the following events and notify the client about them:

• Consumer “producerpause”. The client should call pause() on the corresponding local consumer.
• Consumer “producerresume”. The client should call resume() on the corresponding local consumer (unless the consumer itself was also paused on purpose).

When simulcast or SVC is in use, the application may be interested in signaling preferred layers and effective layers between client and server side consumers.

• The server side application sets the consumer preferred layers via consumer.setPreferredLayers().
• The server side consumer subscribes to the “layerschange” event and notifies the client application about the effective layers being transmitted.

Guidelines for FFmpeg and GStreamer

Both, FFmpeg and GStreamer (and any other similar software), can be used to inject media into a mediasoup router or to consume media from a mediasoup router (for recording purposes, transcoding, streaming using HLS, etc).

This can be done by creating a server side plain transport (via router.createPlainTransport()) and then calling produce() or consume() on it with the appropriate parameters.

Some useful resources:

• FFmpeg documentation
• GStreamer documentation
• libopus encoding options in FFmpeg
• node-fluent-ffmpeg
• node-gstreamer-superficial

Producing Media from an External Endpoint (RTP In)

If you wish to produce media in a mediasoup router by using an external tool (such as FFmpeg or GStreamer) or make mediasoup receive media produced by other RTP source:

• Check your mediasoup router.rtpCapabilities as those determine the kind of media (codecs configuration, RTCP capabilities, RTP header extensions, etc) that mediasoup can receive.
• Create (if not already created) a plain transport in mediasoup and get its local IP and port for RTP (and optionally for RTCP if your external endpoint does not support RTCP-mux).
• Decide the RTP settings of your external endpoint (SSRC values, codec payload type, RTCP capabilities, etc.) and create a RtpSendParameters object. Those RTP parameters must match what your endpoint will send to mediasoup.
• Create a producer (via transport.produce()) on top of the plain transport with those RTP parameters.
• Instruct your external endpoint to send media to the local IP and port(s) of the mediasoup plain transport.

Consuming Media in an External Endpoint (RTP Out)

If you wish to route the media of a producer to an external RTP device or endpoint (such as FFmpeg or GStreamer):

• Create (if not already created) a plain transport in mediasoup and get its local IP and port for RTP (and optionally for RTCP if your media endpoint does not support RTCP-mux).
• Check your mediasoup router.rtpCapabilities and create a subset of them with the RTP capabilities supported by your external endpoint. It's critical that you keep the same codec preferredPayloadType values and RTP header extension preferredId values.
• Create a consumer (via transport.consume()) on top of the plain transport with the corresponding producerId and the generated rtpCapabilities of your external endpoint.
• Get the consumer.rtpParameters and the transport local RTP IP and port(s) and instruct your external endpoint to consume RTP based on those parameters.
  • You may need to build a “remote” SDP offer based on those transport and RTP parameters if your endpoint requires a SDP.
  • Or you may need to tell your external endpoint about the media source parameters (via FFmpeg or GStreamer command line arguments).

Example: Inject Audio and Video using FFmpeg

Let's assume we have a /home/foo/party.mp4 file with a stereo audio track and a video track that we want to inject into a mediasoup router. We run FFmpeg in the server host so media transmission takes places in the localhost network.

• Create a plain transport in the mediasoup router to send the audio track:

const audioTransport = await router.createPlainTransport(
  { 
    listenIp : '127.0.0.1',
    rtcpMux  : false,
    comedia  : true
  });

// Read the transport local RTP port.
const audioRtpPort = audioTransport.tuple.localPort;
// => 3301

// Read the transport local RTCP port.
const audioRtcpPort = audioTransport.rtcpTuple.localPort;
// => 4502

• Create a plain transport in the mediasoup router to send the video track:

const videoTransport = await router.createPlainTransport(
  { 
    listenIp : '127.0.0.1',
    rtcpMux  : false,
    comedia  : true
  });

// Read the transport local RTP port.
const videoRtpPort = videoTransport.tuple.localPort;
// => 3501

// Read the transport local RTCP port.
const videoRtcpPort = videoTransport.rtcpTuple.localPort;
// => 2989

• Create an audio producer on the first transport:

const audioProducer = await audioTransport.produce(
  {
    kind          : 'audio',
    rtpParameters :
    {
      codecs :
      [
        {
          mimeType     : 'audio/opus',
          clockRate    : 48000,
          payloadType  : 101,
          channels     : 2,
          rtcpFeedback : [ ],
          parameters   : { sprop-stereo: 1 }
        }
      ],
      encodings : [ { ssrc: 11111111 } ]
    }
  });

• Create a video producer on the second transport:

const videoProducer = await videoTransport.produce(
  {
    kind          : 'video',
    rtpParameters :
    {
      codecs :
      [
        {
          mimeType     : 'video/vp8',
          clockRate    : 90000,
          payloadType  : 102,
          rtcpFeedback : [ ], // FFmpeg does not support NACK nor PLI/FIR.
        }
      ],
      encodings : [ { ssrc: 22222222 } ]
    }
  });

• Instruct FFmpeg to encode the mp4 file into two RTP streams with the above selected codec payload types and SSRC values, and transmit them to the IPs and ports of the audio and video transports:

ffmpeg \
  -re \
  -v info \
  -stream_loop -1 \
  -i /home/foo/party.mp4 \
  -map 0:a:0 \
  -acodec libopus -ab 128k -ac 2 -ar 48000 \
  -map 0:v:0 \
  -pix_fmt yuv420p -c:v libvpx -b:v 1000k -deadline realtime -cpu-used 4 \
  -f tee \
  "[select=a:f=rtp:ssrc=11111111:payload_type=101]rtp://127.0.0.1:3301?rtcpport=4502|[select=v:f=rtp:ssrc=22222222:payload_type=102]rtp://127.0.0.1:3501?rtcpport=2989"

• Once done, other endpoints (WebRTC endpoints or any others) can receive both, the FFmpeg audio and video track, by using the transport.consume() API as usual.

Guidelines for SRTP Capable Endpoints

Since mediasoup 3.5.0, both pipe and plain transports support SRTP. There are endpoints that do not support WebRTC but do support SRTP. Those endpoints can be connected to mediasoup via a plain transport by enabling SRTP for encrypted RTP transmission.

SRTP Endpoint as SDP Offerer

Let's assume RTCP-mux support in the SRTP Endpoint and also comedia mode (read the PlainTransportOptions API documentation in mediasoup for more information about it).

• In mediasoup, create a plain transport via router.createPlainTransport(options) with comedia: true, rtcpMux: true and enable: true.
• Get the SDP offer from the SRTP endpoint. Such a SDP should have a media attribute similar to “a=crypto:1 AES_CM_128_HMAC_SHA1_80 inline:PS1uQCVeeCFCanVmcjkpPywjNWhcYD0mXXtxaVBR|2^20|1:32”:
  • Get the crypto suite: “AES_CM_128_HMAC_SHA1_80”.
  • Get the key material in Base64: “PS1uQCVeeCFCanVmcjkpPywjNWhcYD0mXXtxaVBR” (must be 40 bytes long).
• In mediasoup, call plainTransport.connect({ srtpParameters }):

await plainTransport.connect(
  {
    cryptoSuite: "AES_CM_128_HMAC_SHA1_80",
    keyBase64: "PS1uQCVeeCFCanVmcjkpPywjNWhcYD0mXXtxaVBR"
  });

• The SRTP device needs now a SDP anwer with SRTP enabled. Among others, read the new plainTransport.srtpParameters and build the SDP answer with a “a=crypto” attribute containing the cryptoSuite and keyBase64 fields of plainTransport.srtpParameters.

SRTP Endpoint as SDP Answerer

In this case (if the SRPT endpoint won't send RTP but just receive it), comedia: true is not valid.

• In mediasoup, create a plain transport via router.createPlainTransport(options) with comedia: true, rtcpMux: true and enable: true (and optionally with srtpCryptoSuite: "AES_CM_128_HMAC_SHA1_32" if the SRTP endpoint does not support “AES_CM_128_HMAC_SHA1_80” crypto suite).
• Generate the remote SDP offer to be sent to the SRTP endpoint by reading plainTransport.srtpParameters and adding the corresponding “a=crypto” line with them.
• Get the SDP answer from the SRTP endpoint and parse the crypto suite and the key material in Base64 (40 bytes long) from its “a=crypto” attribute.
• In mediasoup, call plainTransport.connect({ ip, port, srtpParameters }).

Guidelines for DataChannel termination in Node.js

• Just use router.createDirectTransport() to create a DirectTransport and then create use directTransport.consumeData() on it to create a DataConsumer that will receive the data messages sent by WebRTC endpoints.
• The new dataConsumer.on('message') event will trigger with those received messages so the Node.js application can handle them.
• In the other direction, use directTransport.produceData() to create a DataProducer in Node.js and make the WebRTC peers consume it as usual.
• Then use dataProducer.send() method to send text or binary messages to them.

Related:

• mediasoup-demo: Refactor the message Bot with new DataChannel termination in mediasoup

Guidelines for node-sctp (SCTP/DataChannel in Node.js)

The node-sctp library can be used to send and receive SCTP messages into/from a mediasoup router and, hence, interact via DataChannel with WebRTC endpoints.

mediasoup (also) supports SCTP over plain UDP, which is also supported by node-sctp. Therefore, in order to create a DataProducer in Node.js:

• Create a plain transport with SCTP enabled.
• Create a Node.js UDP socket and connect the mediasoup transport to its local IP:port.
• Create a SCTP socket using the node-sctp API and make it use the UDP socket as transport.
  • The SCTP source and destination ports must be set to 5000. This makes it possible for mediasoup to demultiplex SCTP and RTP/RTCP packets on top of the same UDP 5-tuple.
• Create a SCTP stream with the desired streamId.
• Create a DataProducer on the mediasoup transport via transport.produceData() with the same streamId.
• Write data into the SCTP stream with the proper PPID value (it must be 51 for WebRTC String and 53 for WebRTC Binary).

See a complete usage example with both, Node.js DataProducers and DataConsumers, in the server/lib/Bot.js file of the mediasoup-demo project.