Align two audios with similar parts - audio-processing

Question

I have two audios tracks, extracted from two videos.
They sound almost equally except for a few differences.
Different duration. E.g. first track is 10min length, and second track is 10.5min length because it's stretched.
First audio has only English voice. Second audio contains English + foreign-language voice, and you can hear both because they're mixed as voiceover. In other words: audio 1 has Music, Noises, English speech; audio 2 has Music, Noises, English speech, Foreign language speech.
First and second tracks can differ by its windows, or gaps.
E.g. first track could be
scene 1, gap 1sec, scene 2, gap 1sec, scene 3,
and second track could be
scene 1, gap 2sec, scene 2, gap 2sec, scene 3.
I wonder if there are any solutions that could align these two tracks.
That's what I've tried so far:
Cubase 10.5. https://www.youtube.com/watch?v=BGXkHdzjzMg It doesn't work if tracks have different voices.
Revoice Pro. Same - doesn't recognize tracks with different voices. Probably doesn't support long audios.

Answer 1

Dynamic Time Warping (DTW) is the canonical algorithm for aligning sequences of data that might have slight differences in length/speed. The Python library librosa has a brief tutorial on using it for music syncronization.
There might also be DTW implementations in some graphical audio editors, but I am not familiar with any.

Answer 2

I would try to eliminate the silence gaps between every scene from both audios, so that you are left with only a pair of lists of clean audio clips for each scene.
Then I would recreate both audio signals. The stretched signal would have a constant-length gap between every scene. The original (non-stretched) signal would have variable gaps between scenes, equal to [length of constant gap] + [length of stretched scene - length of normal scene]. This would make every scene start at precisely the same time.
If the gaps between the scenes drop the audio signal to a perfect zero level, detecting and removing the gaps should be trivial.
Otherwise, this could be a little bit tricky (there is usually some DC shift and/or some background noise signal making it a bit difficult to detect "silence" from a time-domain wave representation).
I have successfully used an acoustic energy computation before, to precisely detect where an audio signal begins/ends. This implies sliding a Fourier Transform along the audio (be sure to use a tapered transform, with a Hann or Hamming window). Once you obtain the transform results, you can compute the energy by performing the following computation:
E = Sum(r[x]*r[x] + i[x]*i[x])
Where x goes from 0 to the [length of your Fourier transform] / 2 - 1, r represents the real part of each result bin, and i the imaginary part of each result bin.
This computation is performed repeatedly while sliding the Fourier transform along the audio, while you record the energy along the way. With some proper thresholding, you could probably successfully isolate the audio portions for each scene.
The length of the Fourier Transform can be small (probably something between the range of 64-256 would suffice, as you don't want a fine frequency resolution, just an estimate of the total energy present at a certain point in time)
Here is an example of a tapered Fourier Transform call (using the fftw3 library) computing the energy in a range of frequency bands:
double EnergyAnalyzer::GetEnergy(array<double>^ audioFrame, Int32 startIndex) {
if( startIndex + FrameSize > audioFrame->Length ) {
throw gcnew ArgumentException("The value of startIndex would overflow the array's boundary", "startIndex");
}
// Prepare input to the fourier transform. The signal is tapered using a Hann window
for( int i = 0; i < FrameSize; i++ ) {
_pIn[i] = audioFrame[startIndex + i] * _hann[i];
}
fftw_execute(_fftPlan);
double energy = 0.0;
for( int i = _binStart; i <= _binStop; i++ ) {
energy += _pOut[i][0] * _pOut[i][0] + _pOut[i][1] * _pOut[i][1];
}
return energy;
}