I have a ffmpeg-based worker that handles video-generation jobs at very high throughput.
Long videos need to be streamed while being generated.
For that purpose, I have introduced a WebRTC server named Janus-Gateway with its streaming plugin, and set the application's output to an rtp:// endpoint at that server (ffmpeg can stream a single stream using the RTP protocol).
In order to avoid buffering problems on the other hand, the streaming is done through the ffmpeg's -re option, which means that the stream will be streamed in real-time, i.e. it slows it down to simulate live streaming.
[ffmpeg-based app] (#1)--> [rtp://janus:port # webrtc server] (#2)--> [webrtc subscribers]
How can I continue processing video jobs at high throughput while streaming the results at real-time speed? I need somehow to decouple ffmpeg output (stage #1) so that consumers at stage #2 get streams at natural playback speed.
Related
Is it possible to store and load decoder contexts to resume a decoding session across process launches?
Context/Goal: use video compression to send image data over an expensive link; use intermediate frames almost exclusively (hours in between key frames). Allow consumer side to resume decoding in the event of interruption without requiring new key-frames from producer side.
According to RFC7540:
An HTTP request/response exchange fully consumes a single stream. A request starts with the HEADERS frame that puts the stream into an "open" state. The request ends with a frame bearing END_STREAM, which causes the stream to become "half-closed (local)" for the client and "half-closed (remote)" for the server. A response starts with a HEADERS frame and ends with a frame bearing END_STREAM, which places the stream in the "closed" state.
Knowing that a stream cannot be reopened once it's closed, this means that if I want to implement a long-lived connection where the client sends a stream of requests to the server, I will have to use a new stream for each request. But there is a finite number of streams available, so in theory, I could run out of streams and have to restart the connection.
Why did the writers of the specification design a request/response exchange to completely consume a stream? Wouldn't it have been easy to make a stream like a single thread of exchanges, where you can have multiple exchanges done in serial in one stream?
The point of having many streams multiplexed in a single connection is to interleave them, so that if one cannot proceed, others can.
Reusing a stream for more than one request means just reusing its stream id. I don't see much benefit reusing 4-byte integers -- on the contrary the implementation would become quite more complicated.
For example, the server can inform the client of the last stream that it processed when it's about to close a connection. If stream ids are reused, it would not be possible to report this reliably.
Also, imagine the case where the client sends requestA on stream5; this arrives on the server where its processing takes time; the client times out, sends a RST_STREAM for stream5 (to cancel requestA) and then requestB on stream5. While these are in-flight, the server finishes the processing of requestA and sends the response for requestA on stream5. Now the client reads a response, but it does not know if it is that of requestA or that of requestB.
But there is a finite number of streams available, so in theory, I could run out of streams and have to restart the connection.
That is correct. At 1 ms per exchange, it will take about 12 days to consume the stream ids for a single connection ((2^31-1)/1000/3600/24/2=12.4 days) -- remember that stream ids are incremented by 2 (clients only send odd stream ids).
While this is possible, I have never encountered this case in all the HTTP/2 deployments that I have seen -- typically the connection goes idle and gets closed well before consuming all stream ids.
The specification preferred simplicity and stable features over the ability to reuse stream ids.
Also, bear in mind that HTTP/2 was designed mostly with the web in mind, where browsers make a number of requests to download a web page and its resources, but then stay idle for a while.
The case where an HTTP/2 connection is bombed with non-stop requests is definitely possible, but much rarer and as such it has not probably been deemed important enough in the design -- using 8 bytes for stream ids seems overkill and a cost that is paid for each request even if the 4 bytes limit is never, practically, reached.
I want to do this:
ffmpeg listens to 2 incoming rtp streams, and continuously send those mixed together to a single outgoing rtp stream
However, the 2 incoming streams are going to be starting and stopping randomly and independently of each other, as they have audio. They don't send audio silence, they stop sending when there is silence and then they start sending again when there is audio. ffmpeg seems to error on this situation.
I have this:
ffmpeg -i rtp://0.0.0.0:11000 -i rtp://0.0.0.0:12000 -filter_complex amix -f rtp rtp://10.10.10.10:13000
This is what happens with ffmpeg:
It wait for the audio to start, and then it sends to the output. But, when the input stop sending, I get this error:
rtp://0.0.0.0:11000 Unknown error
rtp://0.0.0.0:12000 Unknown error
and it crashes.
How can I keep it active even when one or the other input isn't sending, and how can I prevent it from crashing?
If ffmpeg outputs silence all the time when it doesn't receive anything, that would be acceptable too.
in RFC 7540 section 5.1.1. (https://www.rfc-editor.org/rfc/rfc7540#section-5.1.1), it specifies as following:
The identifier of a newly established stream MUST be numerically greater than all streams that the initiating endpoint has opened or reserved.
I searched a lot on Google, but still no one explained why the stream ID must be in an ascending order. I don't see any benefit from making this rule to the protocol. From my point of view, out of order stream IDs should also work well if the server just consider the "stream ID" as an ID and use it to distinguish HTTP2 request.
So could anyone can help out explaining the exact reason for this specification?
Thanks a lot!
Strictly ascending stream IDs are an easy way to make them unique (per connection), and it's super-easy to implement.
Choosing - like you say - "out of order" stream IDs is potentially more complicated, as it requires to avoid clashes, and potentially consumes more resources, as you have to remember all the stream IDs that are in use.
I don't think there is any particular reason to specify that stream IDs must be ascending apart simplicity.
6.8. GOAWAY
The GOAWAY frame (type=0x7) is used to initiate shutdown of a
connection or to signal serious error conditions. GOAWAY allows an
endpoint to gracefully stop accepting new streams while still
finishing processing of previously established streams. This enables
administrative actions, like server maintenance.
There is an inherent race condition between an endpoint starting new
streams and the remote sending a GOAWAY frame. To deal with this
case, the GOAWAY contains the stream identifier of the last peer-
initiated stream that was or might be processed on the sending
endpoint in this connection. For instance, if the server sends a
GOAWAY frame, the identified stream is the highest-numbered stream
initiated by the client.
Once sent, the sender will ignore frames sent on streams initiated by
the receiver if the stream has an identifier higher than the included
last stream identifier. Receivers of a GOAWAY frame MUST NOT open
additional streams on the connection, although a new connection can
be established for new streams.
If the receiver of the GOAWAY has sent data on streams with a higher
stream identifier than what is indicated in the GOAWAY frame, those
streams are not or will not be processed. The receiver of the GOAWAY
frame can treat the streams as though they had never been created at
all, thereby allowing those streams to be retried later on a new
connection.
I have been looking at RTSP of Live555 and seems they are following RTSP as per definition form IETF. So far they seem to have reporting for transmission (data sent) on the server end, and reception (data received ) client end.
I am wondering is it possible to implement send/receive statistics (QoS) reports for both the client and the server? I am in a requirement to gather statistics of data sent and received at both the server and the client.
I am new to Live555 and the documentation is pretty obscure in this aspect so any direction is appreciated!
Thanks
For the client side, a example could be found from openRTSP test programs.
openRTSP could display QOS client information :
Outputting QOS statistics
Use the "-Q" option to output QOS ("quality of service") statistics
about the data stream (when the program exits). These statistics
include the (minimum, average, maximum) bitrate, packet loss rate, and
inter-packet gap. The "-Q" option takes an optional
parameter, which specifies the length of the
time intervals - in multiples of 100ms - over which the "minimum,
average, maximum" statistics are computed. The default value of this
parameter is "10", meaning that these statistics are measured every 1
second (i.e., 10x100ms).
For the server side, you can get the QOS informations from the RTPSink::transmissionStatsDB().