Can I have a rawvideo stream in a MPEG TS container - ffmpeg

Question

I receive an MPEG TS container over network (UDP). It contains two streams: an mpeg2video vidoe stream with yuv420p pixel format and a data stream encoded using a propitiatory KLV format.
My receiver program must be in Python. So, I can't use FFMPEG library (like AVFormat, AVCodec) directly.
Now my problem is as follows:
I need to receive video frames and save them as RGB image as raw numpy array. I also need for each frame to parse the corresponding KLV data. There is a one to one relationship between video frames and KLV data units.
I thought I use ffprobe to output the packets including their payload data from incoming container and then parse the output of ffprobe to get the images and metadata:
$ ffprobe -show_packets -show_data -print_format json udp://127.0.0.1:12345 > test_video.packets.data.json
This gives me an output (in test_video.packets.data.json file) like:
{
"codec_type": "video",
"stream_index": 0,
"pts": 140400,
"pts_time": "1.560000",
"dts": 136800,
"dts_time": "1.520000",
"duration": 3600,
"duration_time": "0.040000",
"size": "21301",
"pos": "3788012",
"flags": "K_",
"side_data_list": [
{
"side_data_type": "MPEGTS Stream ID"
}
],
"data": "... "
},
{
"codec_type": "data",
"stream_index": 1,
"pts": 140400,
"pts_time": "1.560000",
"dts": 140400,
"dts_time": "1.560000",
"size": "850",
"pos": "3817904",
"flags": "K_",
"side_data_list": [
{
"side_data_type": "MPEGTS Stream ID"
}
],
"data": ".... "
}
I can extract the KLV data from the data packets and parse it. However the data from the video packets in encoded as mpeg2video video with yuv420p pixel format.
My Questions:
How can I get the raw pixel values from that mpeg2 encoded payload?
Is it possible to use ffmpeg to receive the original container and copy it (with both streams) into a new container, but with raw video instead of mpeg2 video? if yes, how? what should be the command? I tried for example: ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -codec rawvideo -pix_fmt rgb24 -map 0:1 -codec copy -f mpegts udp://127.0.0.1:11112, but it gives me again mpeg2 encoded video data in payload of video packets

Answer 1

MPEG-TS supports a limited number of video codecs. However, ffmpeg's muxer will silently mux even unsupported streams as private data streams.
To mux a raw RGB stream, convert to rgb24 pixel format and code using rawvideo codec.
ffmpeg -i udp://127.0.0.1:12345 -map 0:0 -map 0:1 -c copy -c:v rawvideo -pix_fmt rgb24 -f mpegts udp://127.0.0.1:11112