I want to split the video, do some logical processing, and finally merge it
import ffmpeg
info = ffmpeg.probe("test.mp4")
vs = next(c for c in info['streams'] if c['codec_type'] == 'video')
num_frames = vs['nb_frames']
arr = []
in_file = ffmpeg.input('test.mp4')
for i in range(int(int(num_frames) / 30) + 1):
startTime = i * 30 + 1
endTime = (1 + i) * 30
if endTime >= int(num_frames):
endTime = int(num_frames)
# more more
arr.append(in_file.trim(start_frame=startTime, end_frame=endTime))
(
ffmpeg
.concat(arr)
.output('out.mp4')
.run()
)
I don't understand why this is happening
TypeError: Expected incoming stream(s) to be of one of the following types: ffmpeg.nodes.FilterableStream; got <class 'list'>
Perhaps this is a little too late but you could try
.concat(*arr)
This worked for me with a list of defined start- and endframes
The intention of the program is to run through a directory, if a file is an excel spreadsheet it should open it, extract and manipulate some data then move onto the next file. As this is a laborious process I have tried to split the task across multiple threads. Even after this is only using 20% of the total CPU capacity, and it didn't particular speed up.
def extract_data(unique_file_names):
global rootdir
global newarray
global counter
global t0
string = rootdir + "\\" + str(unique_file_names[0])
wb = load_workbook(string, read_only = True, data_only=True)
ws = wb["Sheet1"]
df = pd.DataFrame(ws.values)
newarray = df.loc[4:43,:13].values
counter = 0
print("Starting pool")
pool = ThreadPool(processes=20)
pool.map(process, unique_file_names)
pool.close()
def process(filename):
global newarray
global unique_file_names
global counter
global t0
counter+=1
try:
file_name = rootdir + "/" + str(filename)
wb = load_workbook(file_name, read_only = True, data_only=True)
ws = wb["Sheet1"]
df = pd.DataFrame(ws.values)
newarray = np.hstack((newarray, df.loc[4:43,4:13].values))
except:
print("Failure")
pass
print("Time %.2f, Completed %.2f %% " %((time.clock()-t0),counter*100/len(unique_file_names)))
So it roughly takes around a second and a half to process one spreadsheet, but like I said pool.map() made next to no difference. Any suggestions?
Thanks in advance
After setting system time via a call to a real-time-clock (RTC), repeated calls to time() always return the same value. Does the system time actually progress on Arduinos or do I need to continue querying the RTC every time I need the time?
To illustrate:
void InitRTC(void) {
DateTime rtcDT; // type defined by the RTC library
time_t bin_time;
rtcDT = rtc.now();
bin_time = rtcDT.secondstime(); // returns unixlike time
set_system_time(bin_time); //AVR call to set the sys time
}
void Dump(time_t t) {
char debug_log_message[MAX_DEBUG_LENGTH];
sprintf(debug_log_message, " time_t:\t %lu", t);
DebugLog(debug_log_message); //....routine to print to the serial port
}
void setup() {
InitRTC();
time_t now;
while (1) {
now = time(0);
Dump(now);
}
}
(safety checks omitted, serial code omitted).
This simply prints the same time for ever - it never progresses!
The standard library has few platform dependencies, so is very portable. However one of those dependencies is the source for real-time, which is entirely platform dependent, and as such it is common for libraries to leave the function as a stub to be re-defined by the user to suit the specific platform or to implement hook functions or call-backs to run the standard library clock.
With the avr-libc used by AVR based Arduino platformsfFor time() to advance it is necessary to call the hook function system_tick() at 1 second intervals (typically from a timer or RTC interrupt) (refer to the avr-libc time.h documentation. It is also necessary to set the time at initialisation using the set_system_time() function.
Invoking system_tick() from a 1Hz timer will then maintain time(), but it is also possible to use an RTC alarm interrupt, by advancing the alarm match target on each interrupt. So you might for example have:
void rtc_interrupt()
{
// Set next interrupt for 1 seconds time.
RTC rtc ;
int next = rtc.getSeconds() + 1 ;
if( next == 60 )
{
next = 0 ;
}
rtc.setAlarmSeconds( next ) ;
// update std time
system_tick() ;
}
void init_system_time()
{
tm component_time ;
RTC rtc ;
// Get *consistent* time components - i.e ensure the
// components are not read either side of a minute boundary
do
{
component_time.tm_sec = rtc.getSeconds() ;
component_time.tm_min = rtc.getMInutes(),
component_time.tm_hour = rtc.getHours(),
component_time.tm_mday = rtc.getDay(),
component_time.tm_mon = rtc.getMonth() - 1, // January = 0 in struct tm
component_time.tm_year = rtc.getYear() + 100 // Years since 1900
} while( component_time.tm_min != rtc.getMinutes() ) ;
set_system_time( mktime( &component_time ) - UNIX_OFFSET ) ;
// Set alarm for now + one second
rtc.attachInterrupt( rtc_interrupt ) ;
rtc.setAlarmSeconds( rtc.getSeconds() + 1 ) ;
rtc.enableAlarm( rtc.MATCH_SS ) ;
}
An alternative is to override time() completely and read the RTC directly on each call - this has the advantage of not requiring any interrupt handlers for example:
#include <time.h>
extern "C" time_t time( time_t* time )
{
tm component_time ;
RTC rtc ;
// Get *consistent* time components - i.e ensure the
// components are not read either side of a minute boundary
do
{
component_time.tm_sec = rtc.getSeconds() ;
component_time.tm_min = rtc.getMInutes(),
component_time.tm_hour = rtc.getHours(),
component_time.tm_mday = rtc.getDay(),
component_time.tm_mon = rtc.getMonth() - 1, // January = 0 in struct tm
component_time.tm_year = rtc.getYear() + 100 // Years since 1900
} while( component_time.tm_min != rtc.getMinutes() ) ;
return mktime( &component_time ) ;
}
I have assumed that the Arduino library getYear() returns years since 2000, and that getMonth() returns 1-12, but neither is documented, so modify as necessary.
Linking the above function before linking libc will cause the library version to be overridden.
I am having problem that the sendInterval I am setting is exceeding the limit I set for it.
Below I just added two lines that I am commenting
void EtherTrafGen::scheduleNextPacket(simtime_t previous)
{
simtime_t next;
if (previous == -1) {
next = simTime() <= startTime ? startTime : simTime();
timerMsg->setKind(START);
}
else {
next = previous + sendInterval->doubleValue();
EV << "THE PREVIOUS TIME IS " << previous << " THE SEND INTERVAL IS " << sendInterval->doubleValue() << endl; // Testing
timerMsg->setKind(NEXT);
}
if (stopTime < SIMTIME_ZERO || next < stopTime)
{
scheduleAt(next, timerMsg);
emit(sigSendInterval,sendInterval->doubleValue()); // Vector collect Send Interval Time
}
}
I tested first with fixed sendInterval value
**.Host5.app.sendInterval = 1ms
THE PREVIOUS TIME IS 0.001 THE SEND INTERVAL IS 0.001
THE PREVIOUS TIME IS 0.002 THE SEND INTERVAL IS 0.001
THE PREVIOUS TIME IS 0.003 THE SEND INTERVAL IS 0.001
From that I assumed Previous Time 2 = Previous Time 1 + Send Interval
Second I assumed The interval time is the same as the one in table under Value (The line shown in code)
Then with random sendInterval value
**.Host5.app.sendInterval = uniform(0.99ms,1.01ms)
THE PREVIOUS TIME IS 0.001001856892 THE SEND INTERVAL IS 0.000998752
THE PREVIOUS TIME IS 0.001999544526 THE SEND INTERVAL IS 0.00100136
THE PREVIOUS TIME IS 0.002999144069 THE SEND INTERVAL IS 0.000997365
Previous Time 2 not equal Previous Time 1 + Send Interval
0.001001856892 + 0.000998752 = 0.002000608892
0.002000608892 - 0.001999544526 = 1.064366 usec
The interval time is not the same as the one under Value as shown in the below table
This is causing that the Host is sending out of the range 0.99ms,1.01ms
The parameter sendInterval is declared in EtherTrafGen.ned as:
volatile double sendInterval #unit(s);
According to the OMNeT++ Simulation Manual:
The volatile modifier indicates that a parameter is re-read every time
a value is needed during simulation.
The variable sendInterval is declared as a pointer to the object which has access to the sendInterval parameter. As a consequence, each time one reads the value of sendInterval, a new random value is returned.
In order to control the current value of sendInterval I suggest reading it only once and storing the result in temporary variable, for example:
void EtherTrafGen::scheduleNextPacket(simtime_t previous)
{
simtime_t next;
double sendIntervalTemp = sendInterval->doubleValue(); // one time read
if (previous == -1) {
next = simTime() <= startTime ? startTime : simTime();
timerMsg->setKind(START);
}
else {
next = previous + sendIntervalTemp ;
EV << "THE PREVIOUS TIME IS " << previous <<
" THE SEND INTERVAL IS " << sendIntervalTemp << endl; // Testing
timerMsg->setKind(NEXT);
}
if (stopTime < SIMTIME_ZERO || next < stopTime)
{
scheduleAt(next, timerMsg);
emit(sigSendInterval, sendIntervalTemp); // Vector collect Send Interval Time
}
}
There are many reports of slow performance of Octave's dlmread. I was hoping that this was fixed in 3.2.4, but when I tried to load a csv file that has a size of ca. 8 * 4 mil (32 mil in total), it also took very, very long time. I searched the web but could not find a workaround for this. Does anybody know a good workaround?
I experienced the same problem and had R handy, so my solution was to use "read.csv" in R, and then use the R package "R.matlab" to write a ".mat" file, and then load that in Octave.
"read.csv" can be pretty slow too, but this worked very well in my case.
The reason is that Octave has a bug that adding data to a very large matrix takes more time then adding the same amount of data to a small matrix.
Below is my try. I choose to save data each 50000 lines, so meanwhile I could already take a look instead of being forced to wait. It is slower for small files, but much faster for larger files.
function alldata = load_data(filename)
fid = fopen(filename,'r');
s=0;
data=[];
alldata=[];
save "temp.mat" alldata;
if fid == -1
disp("Couldn't find file mydata");
else
while (~feof(fid))
line = fgetl(fid);
[t1,t2,t3,t4,d] = sscanf(line,'%i:%i:%i:%i %f', "C"); #reading time as hh:mm:ss:ms and data as float
s++;
t = (t1 * 3600000 + t2 * 60000 + t3 * 1000 + t4);
data = [data; t, d];
if (mod(s,10000) == 0)
#disp(s), disp(" "), disp(t), disp(" "), disp(d), disp("\n");
disp(s);
fflush(stdout);
end
if (mod(s,50000) == 0)
load "temp.mat";
alldata=[alldata; data];
data=[];
save "temp.mat" alldata;
disp("data saved");
fflush(stdout);
end
end
disp(s);
load "temp.mat";
alldata=[alldata; data];
save "temp.mat" alldata;
disp("data saved");
fflush(stdout);
end
fclose(fid);
Here is a workaround that I am using.
I did not find that sscanf will parse input lines as indicated above. Also, I didn't use the temp file.
My .csv file has a large number of rows. They begin with a header of 18 lines and are followed by a data block, each of which has 135 columns. The following code has been tested. My file also begins each row with a dd/mm/yyyy hh:mm field. This will also catch poor lines and indicate where they are by using try/catch.
My .csv file came from a customer who dumped his PARCView load in an Excel file.
function [tags,descr,alldata] = fbcsvread(filename)
fid = fopen(filename,'r');
s = 0;
data=[];
alldata=zeros(1,135);
if fid==-1
disp("Couldn't find file %s\n",filename);
else
linecount = 1;
while (~feof(fid))
line = fgetl(fid);
data2 = zeros(1,135);
if linecount == 1
tags = strsplit(line,",");
elseif linecount == 2
descr = strsplit(line,",");
elseif linecount >= 19
data = strsplit(line,",");
datetime = strsplit(char(data(1))," ");
modyyr = strsplit(char(datetime(1)),"/");
hrmin = strsplit(char(datetime(2)),":");
year1 = sscanf(char(modyyr(3)),"%d","C");
day1 = sscanf(char(modyyr(2)),"%d","C");
month1 = sscanf(char(modyyr(1)),"%d","C");
hour1 = sscanf(char(hrmin(1)),"%d","C");
minute1 = sscanf(char(hrmin(2)),"%d","C");
realtime = datenum(year1,month1,day1,hour1,minute1);
data2(1) = realtime;
for location = 2:134
try
data2(location) = sscanf(char(data(location)),"%f","C");
catch
printf("Error at %s %s\n",char(datetime(1)),char(datetime(2)) );
fflush(stdout);
end_try_catch
endfor
alldata(linecount-18,:) = data2;
if mod(linecount,50) == 0
printf(".");
fflush(stdout);
endif
endif
linecount = linecount + 1;
endwhile
fclose(fid);
endif
endfunction