What is the best setup for livestreaming from dji drone to local server? - dji-sdk

Question

I need to stream livevideo data from DJI drones to a local server using Android Mobile SDK. The video data from the server will be processed by other software (human recognition). What is the recommended approach to prove best streaming quality? We use Mavic 2 for development and Matrice 210 v2 RTK for production purposes. So the answer should primarily focus on possibilites of Matrice 210 v2 RTK.
At the moment we use DJIs Livestreammanager like this
DJISDKManager.getInstance().getLiveStreamManager().setLiveUrl( rtmpURL );
DJISDKManager.getInstance().getLiveStreamManager().startStream();
We use nginx Webserver with rtmp module as the streaming endpoint which pushes stream to the network and records the stream to disk at same time. This works but we see quite bad quality with this streams. The stream lags quite often and it pixels on faster drone movement.
Since i am not familiar with video processing i need recommendation how to solve this requirement. My Questions are like:
-Do i need to use ffmpeg library to preprocess the stream in some way?
-Is there a better solution than nginx+rtmp module for the backend server?
-Would is be possible to utilize the outputs of matrice remote controller to deliver higher quality?

Answer 1

-Do i need to use ffmpeg library to preprocess the stream in some way?
If you want to handle the stream yourself you need to use ffmpeg to decode the raw livestream into actual picture frames. See my answer to question 3.
-Is there a better solution than nginx+rtmp module for the backend server?
This highly depends on the use case. If you are not satisfied with the video quality, it'd be more reliable to decode the stream yourself with ffmpeg. I actually don't know how much nginx+rtmp do automatically, since I've never worked with it.
-Would is be possible to utilize the outputs of matrice remote controller to deliver higher quality?
The raw livestream which can be accessed with the SDK is a raw h264 or h265 stream (NAL Units), depending on the drone model used.
In my application I register the VideoDataCallback in the onComponentChange method with the camera componentKey and send it away via LocalBroadcastManager.
The livestream can then be accessed anywhere in the app by simply subscribing to the broadcast.
private VideoFeeder.VideoDataListener videoDataCallback = (videoBuffer, size) -> {
Intent intent = new Intent("videoStream");
intent.putExtra("content", videoBuffer);
intent.putExtra("size", size);
LocalBroadcastManager.getInstance(this).sendBroadcast(intent);
};
A simple way to send the livestream over the network is using a TCP-Socket: start a thread and open a Socket using an IP and Port:
this.s = new Socket(this.ipAddress, this.port);
this.s.setKeepAlive(true);
this.s.setTcpNoDelay(true);
this.dos = new DataOutputStream(this.s.getOutputStream());
Put a loop around to reconnect it if the connection/sending fails.
In the BroadcastReceiver, accumulate the bytes by analyzing the size (bos is a simple ByteArrayOutputStream):
private BroadcastReceiver frameReceiverNAL = new BroadcastReceiver() {
#Override
public void onReceive(Context context, Intent intent) {
try {
int frameSize = intent.getIntExtra("size", 0);
if(frameSize == 2032) {
bos.write(intent.getByteArrayExtra("content"), 0, frameSize);
} else {
bos.write(intent.getByteArrayExtra("content"), 0, frameSize);
final byte[] sendUnit = bos.toByteArray();
bos.reset();
if(numAsyncTasks < MAX_NUMBER_ASYNCTASKS) {
numAsyncTasks++;
AsyncTask.execute(() -> sendNAL(sendUnit));
}
}
} catch (Exception e) {
e.printStackTrace();
}
}
};
private void sendNAL(byte[] sendUnit){
try {
if(dos != null) {
dos.write(sendUnit);
dos.flush();
}
} catch (IOException e) {
e.printStackTrace();
}
numAsyncTasks--;
}
The MAX_NUMBER_ASYNCTASKS is used as an upper cap to prevent the app from crashing if it fails to send the NALs fast enough (resulted in 500+ unfinished AsyncTasks before crashing). In my case I set the number to 128.
On the Server side you need to create a ServerSocket to receive the byteStream. Depending on the programming language you can either use ffmpeg directly or use a native interface, e.g. bytedeco to create the frames from the stream.
Another way would be to decode the stream on the smartphone into YUV images (2.8MB each) and optionally convert them to jpeg to reduce the size of a single frame (~100KB each). The YUV or jpeg image can then be sent to the server using a TCP Socket. The SDK provides a method for creating YUV images in the class DJICodecManager and the tutorials contain an example for converting them to JPEG.