I want to intercept a frame of my video file, so I used the av_seek_frame function, the timestamp value I tried using timestamp*AV_TIME_BASE and timestamp/(av_q2d(videostream->time_base)*av_TIME_BASE), but no matter what I use Which one, the obtained frame is always not what I want, and it is observed that the frame time that I want is very different (so the accuracy problem of AVSEEK_FLAG_BACKWARD is excluded here).
My code is as follows:
double tmp0 = timeStamp/(av_q2d(inputVideoStream->time_base)*AV_TIME_BASE);
// double tmp0 = timeStamp * AV_TIME_BASE;
code = av_seek_frame(inputAVFormatContext, videoStreamIndex, tmp0, AVSEEK_FLAG_BACKWARD);
Related
I'm having a little trouble working with the linearRampToValueAtTime on a BiQuadFilter applied to a WebAudio.
The audio works ok, and the initial lowpass filter is applied.
Problem is, as soon as I use the linearRamp method to bring up the frequency, it seems to ignore the endTime parameter (or better, it's not time correctly).
Some code to explain it better.
Here's the instancing:
this.audioLoader.load( 'public/media/soundtrack-es_cobwebs_in_the_sky.mp3', buffer => {
this.sounds.soundtrack = new THREE.Audio(this.listener);
const audioContext = this.sounds.soundtrack.context;
this.biquadFilter = audioContext.createBiquadFilter();
this.biquadFilter.type = "lowpass"; // Low pass filter
this.biquadFilter.frequency.setValueAtTime(200, audioContext.currentTime);
this.sounds.soundtrack.setBuffer(buffer);
this.sounds.soundtrack.setFilter(this.biquadFilter);
this.sounds.soundtrack.setVolume(0.5);
this.sounds.soundtrack.play();
})
Until here, everything looks ok. The sound plays muffled as needed.
Then, after a certain event, there's a camera transition, where I want the sound to gradually open up.
As a endTime parameter, I'm passing 2 seconds + the internal context delta.
this.sounds.soundtrack.filters[0].frequency.linearRampToValueAtTime(2400, 2 + this.sounds.soundtrack.context.currentTime);
Expecting to hear the ramp in two seconds, but the sound opens up immediately.
What am I missing?
The linear ramp will be applied using the previous event as the startTime. In your case that will be audioContext.currentTime at the point in time when you created the filter. If that is sufficiently long ago it will sound as if the ramp jumps right to the end value. You can fix that by inserting a new event right before the ramp.
const currentTime = this.sounds.soundtrack.context.currentTime;
const filter = this.sounds.soundtrack.filters[0];
filter.frequency.setValueAtTime(200, currentTime);
filter.frequency.linearRampToValueAtTime(2400, currentTime + 2);
It is very critical to learn the start time of each frame of a video.
I need to determine the starting point manually ( for example 848 here) by using below matlab code:
v = VideoReader('video1.avi','CurrentTime',848);
while hasFrame(v)
video_frame = readFrame(v);
counter=counter+1;
if counter==1
imshow(video_frame)
imhist(video_frame(:,:,1))
end
end
What I want is to distinguish some video frame from the others by using histogram. At the end my aim is to reach the exact showing time of the distinguished frames.
After editting:
This is frame histogram outputs:
Histogram size of the some frames are different from the previous one, do you know the reason?
difference=[difference sum(abs(histcounts(video_frame)-histcounts(lastframe)))];
Because of the taking the difference of the I had remove the different histogram sized frames but it causes missing some frames.
i havent found an video example that looks like what you discribe. please condsider always to have an example.
This example code calculates the differences in the histcounts. please notice that waitforbuttonpressis in the loop so you have to click for each frame while testing or remove it when the video is too long. Does this works on your file?
v = VideoReader('sample.avi','CurrentTime',1);
figure1=figure('unit','normalized','Position',[0.2 0.2 0.4 0.6]);
axes1=subplot(3,1,1);
axes2=subplot(3,1,2);
axes3 = subplot(3,1,3);
counter=0;
difference=[];
video_frame=readFrame(v);
while hasFrame(v)
lastframe=video_frame;
video_frame = readFrame(v);
counter=counter+1;
imshow(video_frame,'Parent',axes1);
[a,b]=histcounts(video_frame(:,:,1));
plot(b(1:end-1),a,'Parent',axes2);
difference=[difference sum(abs(histcounts(video_frame,0:255)-histcounts(lastframe,0:255)))];
bar(1:counter,difference,'Parent',axes3);
waitforbuttonpress
end
[~,onedistinguished]=max(difference);
%defining a threshold like every value that is bigger 4000
multidistinguished=find(difference>4000);
disp(['majorly changed at: ' num2str(distinguished)]);
I am using matlab 2013a software for my project.
I face a problem while splitting video into individual frames.
I want to know how to get frames at a specific intervals from video.. i.e., i want to grab frames at the rate of one frame per second(frame/sec) .My input video has 50 frames/sec. In the code I have used step() to slice the video into frames.
The following is my code , basically a face detection code(detects multiple faces in a video) . This code captures every frame in the video(i.e 50fp approx) and processes it. I want to process frames at the rate of 1 fps. Please help me.
clear classes;
videoFileReader = vision.VideoFileReader('C:\Users\Desktop\project\05.mp4');
**videoFrame = step(videoFileReader);**
faceDetector = vision.CascadeObjectDetector(); % Finds faces by default
tracker = MultiObjectTrackerKLT;
videoPlayer = vision.VideoPlayer('Position',[200 100 fliplr(frameSize(1:2)+30)]);
bboxes = [];
while isempty(bboxes)
**framergb = step(videoFileReader);**
frame = rgb2gray(framergb);
bboxes = faceDetector.step(frame);
end
tracker.addDetections(frame, bboxes);
frameNumber = 0;
keepRunning = true;
while keepRunning
**framergb = step(videoFileReader);**
frame = rgb2gray(framergb);
if mod(frameNumber, 10) == 0
bboxes = 2 * faceDetector.step(imresize(frame, 0.5));
if ~isempty(bboxes)
tracker.addDetections(frame, bboxes);
end
else
% Track faces
tracker.track(frame);
end
end
%% Clean up
release(videoPlayer);
But this actually considers every frame. I want to grab 1fps.
It cannot be done directly in Matlab 2013a, because the video access library does not provide the feature you want. Writing the necessary code to implement an efficient frame skipping routine is not really possible using just Matlab code (you would need to look inside the video libraries)
Working around it, you have two basic options:
Do as little work as possible on frames that you do not want to process.
Where you currently have
framergb = step(videoFileReader);
Instead do something like
for i=1:49,
step(videoFileReader);
end
framergb = step(videoFileReader);
(NB this does not allow for going beyond end of input)
Pre-process your file with a tool like ffmpeg, and reduce the frame-rate before you use Matlab.
The ffmpeg command might look something like this:
ffmpeg -i 05.mp4 -r 1 05_at_1fps.mp4
I use packet duration to translate from frame index to pts and back, and I'd like to be sure that this is a reliable method of doing so.
Alternatively, is there a better way to translate pts to a frame index and vice versa?
A snippet showing my usage:
bool seekFrame(int64_t frame)
{
if(frame > container.frameCount)
frame = container.frameCount;
// Seek to a frame behind the desired frame because nextFrame() will also increment the frame index
int64_t seek = pts_cache[frame-1]; // pts_cache is an array of all frame pts values
// get the nearest prior keyframe
int preceedingKeyframe = av_index_search_timestamp(container.video_st, seek, AVSEEK_FLAG_BACKWARD);
// here's where I'm worried that packetDuration isn't a reliable method of translating frame index to
// pts value
int64_t nearestKeyframePts = preceedingKeyframe * container.packetDuration;
avcodec_flush_buffers(container.pCodecCtx);
int ret = av_seek_frame(container.pFormatCtx, container.videoStreamIndex, nearestKeyframePts, AVSEEK_FLAG_ANY);
if(ret < 0) return false;
container.lastPts = nearestKeyframePts;
AVFrame *pFrame = NULL;
while(nextFrame(pFrame, NULL) && container.lastPts < seek)
{
;
}
container.currentFrame = frame-1;
av_free(pFrame);
return true;
}
No, not guaranteed. It may work with some codec/container combination where frame-rate is static. avi, h264 raw (annex-b) and yuv4mpeg come to mind. But other containers like flv, mp4, ts, have a PTS/DTS (or CTS) for EVERY frame. The source could be variable frame rate, or frames could have be dropped at some point during processing due to bandwidth. Also some codecs will remove duplicate frames.
So unless you created the file yourself. Do not trust it. There is no guaranteed way to look at a frame and know its 'index' except start at the beginning and count.
Your method, MAY be good enough for most files however.
I'm using ffms2 (aka FFmpegSource) for decoding video frames and display on UI based on wxWidgets.
My player works fine for low resolution video (320*240, 640*480) but for higher resolution (1080) it is very slow. I'm not able to meed the desired frame for high resolution video.
After time analysis I found that FFMS_GetFrame() frame function takes much longer time for high resolution frame.
Here are the results.
1. 320*240 FFMS_GetFrame takes 4-6ms
2. 640*480 FFMS_GetFrame takes >20ms
3. 1080*720 FFMS_GetFrame takes >40
Which means that I'll never meets 30 fps requirement for 1080p frame with FFMS2. But I'm not sure if this is the case.
Please suggest what could be going wrong.
void SetPosition(int64 pos)
{
uint8_t* data_ptr = NULL;
/*check if position is valid*/
if (!m_track || pos < 0 && pos > m_videoProp->NumFrames - 1)
return; // ERR_POS;
wxMilliClock_t start_wx_t = wxGetLocalTimeMillis();
long long start_t = start_wx_t.GetValue();
m_frameId = pos;
if(m_video)
{
m_frameProp = FFMS_GetFrame(m_video, m_frameId, &m_errInfo);
if(!m_frameProp) return;
if(m_frameProp)
{
m_width_ffms2 = m_frameProp->EncodedWidth;
m_height_ffms2 = m_frameProp->EncodedHeight;
}
wxMilliClock_t end_wx_t = wxGetLocalTimeMillis();
long long end_t = end_wx_t.GetValue();
long long diff_t = end_t - start_t;
wxLogDebug(wxString(wxT("Frame Grabe Millisec") + ToString(diff_t)));
//m_frameInfo = FFMS_GetFrameInfo(m_track, FFMS_TYPE_VIDEO);
/* If you want to change the output colorspace or resize the output frame size, now is the time to do it.
IMPORTANT: This step is also required to prevent resolution and colorspace changes midstream. You can
always tell a frame's original properties by examining the Encoded properties in FFMS_Frame. */
/* A -1 terminated list of the acceptable output formats (see pixfmt.h for the list of pixel formats/colorspaces).
To get the name of a given pixel format, strip the leading PIX_FMT_ and convert to lowercase. For example,
PIX_FMT_YUV420P becomes "yuv420p". */
#if 0
int pixfmt[2];
pixfmt[0] = FFMS_GetPixFmt("bgr24");
pixfmt[1] = -1;
#endif
// FFMS_SetOutputFormatV2 returns 0 on success. It Returns non-0 and sets ErrorMsg on failure.
int failure = FFMS_SetOutputFormatV2(m_video, pixfmt, m_width_ffms2, m_height_ffms2, FFMS_RESIZER_BICUBIC, &m_errInfo);
if (failure)
{
//FFMS_DestroyVideoSource(m_video);
//m_video = NULL;
return; //return ERR_POS;
}
data_ptr = m_frameProp->Data[0];
}
else
{
m_width_ffms2 = 320;
m_height_ffms2 = 240;
}
if(data_ptr)
{
memcpy(m_buf, data_ptr, 3*m_height_ffms2 * m_width_ffms2);
}
else
{
memset(m_buf, 0, 3*m_height_ffms2 * m_width_ffms2);
}
}
Slower video decoding with larger frames is totally normal. 1080x720 has about ten times as many pixels as 320x240, so having GetFrame take about ten times as long is not surprising (it's not a strictly linear relationship as there's a lot of other factors that play into decoding speed, but pixel count and time to decode are fairly correlated).
Setting the output format for every frame is unnecessary and is going to be making things a lot slower. Unless you specifically want the output format to change you should call it just once after opening the video, and it'll apply to all frames requested after that.