There are many reports of slow performance of Octave's dlmread. I was hoping that this was fixed in 3.2.4, but when I tried to load a csv file that has a size of ca. 8 * 4 mil (32 mil in total), it also took very, very long time. I searched the web but could not find a workaround for this. Does anybody know a good workaround?
I experienced the same problem and had R handy, so my solution was to use "read.csv" in R, and then use the R package "R.matlab" to write a ".mat" file, and then load that in Octave.
"read.csv" can be pretty slow too, but this worked very well in my case.
The reason is that Octave has a bug that adding data to a very large matrix takes more time then adding the same amount of data to a small matrix.
Below is my try. I choose to save data each 50000 lines, so meanwhile I could already take a look instead of being forced to wait. It is slower for small files, but much faster for larger files.
function alldata = load_data(filename)
fid = fopen(filename,'r');
s=0;
data=[];
alldata=[];
save "temp.mat" alldata;
if fid == -1
disp("Couldn't find file mydata");
else
while (~feof(fid))
line = fgetl(fid);
[t1,t2,t3,t4,d] = sscanf(line,'%i:%i:%i:%i %f', "C"); #reading time as hh:mm:ss:ms and data as float
s++;
t = (t1 * 3600000 + t2 * 60000 + t3 * 1000 + t4);
data = [data; t, d];
if (mod(s,10000) == 0)
#disp(s), disp(" "), disp(t), disp(" "), disp(d), disp("\n");
disp(s);
fflush(stdout);
end
if (mod(s,50000) == 0)
load "temp.mat";
alldata=[alldata; data];
data=[];
save "temp.mat" alldata;
disp("data saved");
fflush(stdout);
end
end
disp(s);
load "temp.mat";
alldata=[alldata; data];
save "temp.mat" alldata;
disp("data saved");
fflush(stdout);
end
fclose(fid);
Here is a workaround that I am using.
I did not find that sscanf will parse input lines as indicated above. Also, I didn't use the temp file.
My .csv file has a large number of rows. They begin with a header of 18 lines and are followed by a data block, each of which has 135 columns. The following code has been tested. My file also begins each row with a dd/mm/yyyy hh:mm field. This will also catch poor lines and indicate where they are by using try/catch.
My .csv file came from a customer who dumped his PARCView load in an Excel file.
function [tags,descr,alldata] = fbcsvread(filename)
fid = fopen(filename,'r');
s = 0;
data=[];
alldata=zeros(1,135);
if fid==-1
disp("Couldn't find file %s\n",filename);
else
linecount = 1;
while (~feof(fid))
line = fgetl(fid);
data2 = zeros(1,135);
if linecount == 1
tags = strsplit(line,",");
elseif linecount == 2
descr = strsplit(line,",");
elseif linecount >= 19
data = strsplit(line,",");
datetime = strsplit(char(data(1))," ");
modyyr = strsplit(char(datetime(1)),"/");
hrmin = strsplit(char(datetime(2)),":");
year1 = sscanf(char(modyyr(3)),"%d","C");
day1 = sscanf(char(modyyr(2)),"%d","C");
month1 = sscanf(char(modyyr(1)),"%d","C");
hour1 = sscanf(char(hrmin(1)),"%d","C");
minute1 = sscanf(char(hrmin(2)),"%d","C");
realtime = datenum(year1,month1,day1,hour1,minute1);
data2(1) = realtime;
for location = 2:134
try
data2(location) = sscanf(char(data(location)),"%f","C");
catch
printf("Error at %s %s\n",char(datetime(1)),char(datetime(2)) );
fflush(stdout);
end_try_catch
endfor
alldata(linecount-18,:) = data2;
if mod(linecount,50) == 0
printf(".");
fflush(stdout);
endif
endif
linecount = linecount + 1;
endwhile
fclose(fid);
endif
endfunction
Related
Long story short, I have this bit of Google Script that clears content automatically in a GSheet. It is set on a trigger and it works...the code does what it's supposed to do. The issue is that it runs slow. It takes 2 to 3 minutes for the iterator to run. To help you scope the size of the task: there is 150 rows on each of the 8 sheets.
The objective of the code is to clear a number of rows on each sheet based on the value of the cell in the first column of a row.
So I would like to know if anyone has any insight or suggestion to improve the running time. I understand my method of using a for loop checks rows one by one, and that's a time-consuming task. I couldn't think of an alternate method with arrays or something?
Thanks all!
Here's the code:
function Reset_Button() {
var sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
for (var i = 1; i < sheets.length ; i++ ) {
var sheet = sheets[i];
sheet.getRange("C2").setValue(new Date());
var rangeData = sheet.getDataRange();
var lastRow = rangeData.getLastRow();
var searchRange = sheet.getRange(1,1, lastRow, 1);
for ( j = 1 ; j < lastRow ; j++){
var value = sheet.getRange(j,1).getValue()
if(value === 0){
sheet.getRange(j,2,1,5).clearContent()
}}}}
Typically you want to do as few writes to the spreadsheet as possible. Currently your code goes through each line and edits it if necessary. Instead get the entire data range you will be working with into one variable (let's say dRange and use .getValues() to get a 2d array of all the values into a second variable (let's say dValues). Then simply iterate over dValues, setting a blank "" in each you want to clear. Once you are done going over all values, just do a dRange.setValues(dValues) (that's why I said to keep the range in a separate variable). So as an example, the following will clear columns B through F if column A has a 0
function test(){
var sheets = SpreadsheetApp.getActiveSpreadsheet().getSheets();
for (var i = 1; i <sheets.length; i++) {
sheets[i].getRange("C2").setValue(new Date());
var dRange = sheets[i].getDataRange();
var dValues = dRange.getValues();
for (var j = 1; j < dRange.getLastRow(); j++){
if (dValues[j][0] == 0) {
for (var c = 1; c < 6; c++) {
dValues[j][c] = ""
}
}
}
dRange.setValues(dValues);
}
}
For a single sheet of ~170 rows this takes a few seconds. One thing to note is that I wrote it based on your script, you set a date value in C2 however in your sript (and thus in the one I wrote based on yours) that falls within the range you are checking to be cleared, so double check your ranges
In EMEditor, is there a way to get the number of occurrences of a "find in files" search per file? In other words, it finds 10,000 "hits" across 25 files, I'd like to know that 1200 where in file1 etc.
Notepad++ does a great job of this by allowing you to collapse the results by file and showing a summary for each, but I haven't seen a way to get the information in EMEditor.
After Find in Files, you can run this macro while the results document is active. Save this code as, for instance, statistics.jsee, and then select this file from Select... in the Macros menu. Finally, do Find in Files, and select Run in the Macros menu while the results document is active.
// Creates statistics from Find in Files Results.
// 2020-06-27
Redraw = false;
sOutput = "";
y = 1;
yMax = document.GetLines();
for( ;; ) {
document.selection.SetActivePoint( eePosLogical, 1, y++ );
document.selection.Mode = eeModeStream | eeModeKeyboard;
bFound = document.selection.Find("\\(\\d+?\\)\\:",eeFindNext | eeFindReplaceCase | eeFindReplaceRegExp,0);
document.selection.Mode = eeModeStream;
if( !bFound ) {
break;
}
sFile = document.selection.Text;
n = sFile.lastIndexOf("(");
sFile = sFile.substr( 0, n );
nCount = 1;
for( ;; ) {
document.selection.SetActivePoint( eePosLogical, 1, y );
sLine = document.GetLine( y );
if( sLine.length > sFile.length && sLine.substr( 0, sFile.length ) == sFile ) {
++nCount;
++y;
}
else {
sOutput += sFile + "\t" + nCount + "\n";
break;
}
}
}
document.selection.Mode = eeModeStream;
Redraw = true;
editor.NewFile();
document.write( sOutput );
editor.ExecuteCommandByID(4471); // switch to TSV mode
I'm trying to make flv file from images via the following Matlab code.The problem is converting bmp images fo flv.
I'm almost new in this section of Matlab. do you have any Idea?
clear
clc
vidobj = videoinput('winvideo',2);
preview(vidobj);
No_snapshot = 5;
interval = 1;
Format = 'bmp';
PathName = uigetdir;
tic;
count = 0;
date_temp = datevec(now);
date_string_vid = [num2str(date_temp(1)),'-',num2str(date_temp(2)),'-',num2str(date_temp(3)),'-',...
num2str(date_temp(4)),'-',num2str(date_temp(5)),'-',num2str(date_temp(6))];
while 1
if fix(toc/interval) > count
count = fix(toc/interval);
date_string = num2str(count);
imwrite(getsnapshot(vidobj),[PathName,'\',date_string,'.' Format], Format);
end
if count >= No_snapshot
break;
end
end
closepreview;
delete(vidobj);
% =========================================================================
PathName = 'G:\capture_video\movie1\';
obj=VideoWriter(date_string_vid,'Grayscale AVI');
open(obj)
for m=1:No_snapshot
m1=imread([PathName,num2str(m),'.bmp']);
% m1=double(m1(:,:,1));
F = im2frame(m1);
aviObject = addframe(obj,F); % Add the frame to the AVI file
end
close(obj)
Anyone on ideas on a more efficient solution than the if else function below?? This takes the bulk of the time for the code so I need to reduce it.
The full function is
function result = vre(t,r,e,n,d)
if (e==4 && r>0)
result = 0;
elseif (e==4 && r==0)
result = 1;
elseif (e<4 && r==1)
result = t;
elseif (e<4 && r==2)
result = d;
else
result=n;
end
end
If this function is taking most of your processing time, it is almost certainly because you're calling it too many times. In turn, this is likely because you are calling it on each element of a vector or matrix individually. I suggest changing the function to accept matrix inputs for e and r, so you can perform all the checks at once - matlab is built for matrix operations, so taking advantage of those is always a good idea.
function result = vre(t,r,e,n,d)
#% add error checking for size of input args if desired
result = ones(size(e))*n; #% default result; next assign special cases
result(e==4 & r>0) = 0; #% note the single & for element-wise 'and'
result(e==4 & r==0) = 1;
result(e<4 & r==1) = t;
result(e<4 & r==2) = d;
end
The function now returns a matrix that is the same size as the input matrices - for single elements it will work exactly the same as your current version, but for higher dimensional inputs it will work too, and probably give you a substantial speed boost.
function result = vre(t,r,e,n,d)
if (e==4) {
if(r>0)
result = 0;
elseif (r==0)
result = 1;
}
elseif (e<4) {
if(r==1)
result = t;
elseif (r==2)
result = d;
}
else
result=n;
end
end
By doing it this way you'll only verify (e==4) and (e<4) once, avoiding unnecessary verifications.
Hope it saves some processing time.
PS: Not tested since I don't have MatLab installed.
Try this:
function result = vre(t,r,e,n,d)
if (e==4)
result = (r==0);
elseif (e<4)
result = (r==1)*t+(r==2)*d;
else
result=n;
end
end
I can't guarantee that it's more efficient (I use octave rather than matlab, so speed testing isn't going to help). But I think it will be.
I have the following method as part of a password generating program to generate a random password which is then validated.
My problem is that the srand function never meets the validation requirements and keeps looping back to create a new password.
Im posting the code below to ask if anyone has a more efficient way to create the random password so that it will meet validation requirements instead of looping back continously.Thanks.
static bool verifyThat(bool condition, const char* error) {
if(!condition) printf("%s", error);
return !condition;
}
//method to generate a random password for user following password guidelines.
void generatePass()
{
FILE *fptr;//file pointer
int iChar,iUpper,iLower,iSymbol,iNumber,iTotal;
printf("\n\n\t\tGenerate Password selected ");
get_user_password:
printf("\n\n\t\tPassword creation in progress... ");
int i,iResult,iCount;
char password[10 + 1];
char strLower[59+1] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRTUVWXYZ!£$%^&*";
srand(time (0));
for(i = 0; i < 10;i++)
{
password[i] = strLower[(rand() % 52)];
}
password[i] = '\0';
iChar = countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
//folowing statements used to validate password
iChar = countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
iUpper = countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
iLower =countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
iSymbol =countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
iNumber = countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
iTotal = countLetters(password,&iUpper,&iLower,&iSymbol,&iNumber,&iTotal);
if(verifyThat(iUpper >= 2, "Not enough uppercase letters!!!\n")
|| verifyThat(iLower >= 2, "Not enough lowercase letters!!!\n")
|| verifyThat(iSymbol >= 1, "Not enough symbols!!!\n")
|| verifyThat(iNumber >= 2, "Not enough numbers!!!\n")
|| verifyThat(iTotal >= 9, "Not enough characters!!!\n")
|| verifyThat(iTotal <= 15, "Too many characters!!!\n"))
goto get_user_password;
iResult = checkWordInFile("dictionary.txt", password);
if(verifyThat(iResult != gC_FOUND, "Password contains small common 3 letter word/s."))
goto get_user_password;
iResult = checkWordInFile("passHistory.txt",password);
if(verifyThat(iResult != gC_FOUND, "Password contains previously used password."))
goto get_user_password;
printf("\n\n\n Your new password is verified ");
printf(password);
//writing password to passHistroy file.
fptr = fopen("passHistory.txt", "w"); // create or open the file
for( iCount = 0; iCount < 8; iCount++)
{
fprintf(fptr, "%s\n", password[iCount]);
}
fclose(fptr);
printf("\n\n\n");
system("pause");
}//end of generatePass method.
I looked at your code at glance and I think I have found the reasons inspite of which validation requirements aren`t meet.
I suggest you to pay attention to the following parts of your code:
1) char strLower[59+1] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRTUVWXYZ!£$%^&*";
here you should add numbers 0..9, this is one of the reasons why requirements could not be met, because how number can be picked if it isn`t in the set of numbers from which you pick?!
replace it for ex. with:
char strLower[] = "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRTUVWXYZ!£$%^&*0123456789";
2) password[i] = strLower[(rand() % 52)]; - and in this part of the code, replace 52 with total number of symbols in string from which you randomly pick numbers.
I recommend you to replace it with the following code:
password[i] = strLower[(rand() % (sizeof(strLower) / sizeof(char) - 1))];
you could alter your algorithm.
choose at random a number of Upper characeters that is above 2.
choose at random a number of Lower character that is above 2.
choose at random a number of Sybmol character that is above 1.
choose at random a number of Number characters that is above 2.
and then recompose your password with the random items, re-ordered at random. Fill with whatever character you want to pas the verifyThat predicates: >=9 and <= 15.
And please: don't use goto. Make function calls instead.