I have about 300 files I would like to access and import in Matlab, all these files are inside 300 folders.
The first file lie in the directory users/matt/Documents/folder_1 with the filename line.csv the 2nd file lie in users/matt/Documents/folder_2 with filename line.csv
So I would like to import the data from the 300 line.csv files in Matlab so I can take the average value. Is this possible? I am using mac osx btw.
I know what do with the .csv file, but I have no clue how to access them efficiently.
This should work: All we are doing is generating the string for every file path using sprintf and the loop index i, and then reading the csv file using csvread and storing the data in a cell array.
for i = 1:300 % Loop 300 times.
% Full path pointing to the csv file.
file_path = sprintf('users/matt/Documents/folder_%d/line.csv', i);
% Read data from csv and store it in a cell array.
data{i} = csvread(file_path);
end
% Do your computations here.
% ...
Remember to replace the 300 by the actual number of folders that you have.
Related
When I create a list from values in a CSV file, I am able to print out the contents of the list using the built in print function, but when I try to enter the elements of the list into an input box using pyautogui,only the first element of the list is printed.
Here is the code which is not working with screenshot of error attached.
Idle Error Message
import csv, pyautogui, time
file = open('serials.csv', 'r')
serial_list = list(csv.reader(file))
file.close
print(serial_list)
time.sleep(5)
i = 0
for serial in serial_list:
pyautogui.typewrite(serial_list[i][i])
i +=1
I was able to get the desired results using the Python code below which I generated using VBA, but would like to learn how to do this properly using Python.
import pyautogui, time
time.sleep(3)
pyautogui.write("2300-xxxx1")
pyautogui.press('tab')
pyautogui.write("2300-xxxx2")
pyautogui.press('tab')
pyautogui.write("2300-xxxx3")
pyautogui.press('tab')
pyautogui.write("2300-xxxx4")
pyautogui.press('tab')
pyautogui.write("2300-xxxx5")
yo, I think issue is in this line of code:
pyautogui.typewrite(serial_list[i][i])
Here, you trying to access a specific element of the 2D list serial_list using serial_list[i][i] where i is the index variable. However, this will only give you the first element of the list.
You should use the variable serial that you defined in the for loop to access the current element of the list:
pyautogui.typewrite(serial)
Also, since the serial_list is a 2D list of strings, you want to convert the elements of list to strings before passing it to the pyautogui.typewrite() function like this:
for serial in serial_list:
pyautogui.typewrite(str(serial))
Also, you should call the file.close method with paranthesis file.close() to close the file properly.
Also, you should use the pyautogui.press('tab') function after each pyautogui.typewrite() function call to move to the next input box.
So the final code will look like:
import csv, pyautogui, time
file = open('serials.csv', 'r')
serial_list = list(csv.reader(file))
file.close()
print(serial_list)
time.sleep(5)
for serial in serial_list:
pyautogui.typewrite(str(serial))
pyautogui.press('tab')
Now it will work.
why? cuz 🅱️
I have a huge amount of datafiles (.csv) saved in one directory. I want now to fit and evaluate several parameters for each file. Since there are over 300.000 files in this directory, mathematica is not able to run my script. The first attempt i tried was to set the directory to this folder and then i tried to emport each file alone through a 'For-loop' (for i=1,i<=imax,i++ where imax is the number of files in there), do the whole fitting evaluation etc and then starting the loop again and importing the second file,.... to save memory. Unfortunately, this approach didn't work at all and mathematica crashed almost immediately.
So, my question is now, can I handle such a huge amount of files in a single directory somehow without running out of memory?
The method below loads all the data from all the CSVs into a function variable called data. So if you have CSVs called file1.csv and file2.csv their data will be loaded into variables called data["file1.csv"] and data["file2.csv"]. Then, for example, the data in the first column of every CSV is read into a variable called xvalues and the data in the second column of every CSV is read into a variable called yvalues.
SetDirectory["C:\\Users\\yourname\\datadirectory"];
files = FileNames["*.csv"];
(data[#] = Import[#]) & /# files;
xvalues = yvalues = {};
(xvalues = Join[xvalues, data[#][[All, 1]]]) & /# files;
xvalues = Flatten[xvalues];
(yvalues = Join[yvalues, data[#][[All, 2]]]) & /# files;
yvalues = Flatten[yvalues];
A fit can then be calculated.
fit = Fit[Transpose[{xvalues, yvalues}], {1, x}, x]
In Hadoop, I can point an app to a path which then the mappers will process the files individually. I have to handle it this way because I need to parse the file name and path to match up with other files that I load directly in the mappers.
In pyspark, passing the path to SparkContext's textFile creates one RDD. Is there any way to replicate the same Hadoop behavior in Spark / pyspark?
I hope this resolve some of your confusions :
sparkContext.wholeTextFiles(path) returns a pairRDD (helpful link: https://www.safaribooksonline.com/library/view/learning-spark/9781449359034/ch04.html)
In short, pairRDD is more like a map (i.e. have key, value)
rdd = sparkContext.wholeTextFiles(path)
def func_work_on_individual_files(x):
# x is a tuple which will receive both (key, value) for the pairRDD Row Elements passed. key -> file path, value -> content of a file with line seperated by '/n' (as you mentioned). To access key use x[0], to access value use x[1].
# your logic to do something useful with file data,
# to get separate lines you can use: x[1].split('\n')
# end function by return the values you want to return out of a file's data.
# I am simply returning the whole content of file
return x[1]
#loop over each of the file in the pairRdd created above
file_contents = rdd.map(func_work_on_individual_files)
#this will create just one partition out of all elements in list (as you mentioned)
consolidated_contents = file_contents.repartition(1)
#Save final output - this will create just one path like Hadoop
consolidated_contents.saveAsTextFile(path)
Pyspark provides a function for this use case: sparkContext.wholeTextFiles(path). It will read a directory of text files and produce a key-value pair, where key is the path of each file and value is the content of each file.
I have a folder of jpg files and I want to make them into a movie. I am using this script:
% Create video out of list of jpgs
clear
clc
% Folder with all the image files you want to create a movie from, choose this folder using:
ImagesFolder = uigetdir;
% Verify that all the images are in the correct time order, this could be useful if you were using any kind of time lapse photography. We can do that by using dir to map our images and create a structure with information on each file.
jpegFiles = dir(strcat(ImagesFolder,'\*.jpg'));
% Sort by date from the datenum information.
S = [jpegFiles(:).datenum];
[S,S] = sort(S);
jpegFilesS = jpegFiles(S);
% The sub-structures within jpegFilesS is now sorted in ascending time order.
% Notice that datenum is a serial date number, for example, if you would like to get the time difference in hours between two images you need to subtract their datenum values and multiply by 1440.
% Create a VideoWriter object, in order to write video data to an .avi file using a jpeg compression.
VideoFile = strcat(ImagesFolder,'\MyVideo');
writerObj = VideoWriter(VideoFile);
% Define the video frames per second speed (fps)
fps = 1;
writerObj.FrameRate = fps;
% Open file for writing video data
open(writerObj);
% Running over all the files, converting them to movie frames using im2frame and writing the video data to file using writeVideo
for t = 1:length(jpegFilesS)
Frame = imread(strcat(ImagesFolder,'\',jpegFilesS(t).name));
writeVideo(writerObj,im2frame(Frame));
end
% Close the file after writing the video data
close(writerObj);
(Courtesy of http://imageprocessingblog.com/how-to-create-a-video-from-image-files/)
But it gives me this error:
Warning: No video frames were written to this file. The file may be invalid.
> In VideoWriter.VideoWriter>VideoWriter.close at 289
In Movie_jpgCompilation at 37
I'm sure my jpg files are fine, and they are in the folder I specify. What is the problem?
(This is my first post ever, so I hope it helps).
If you're on Linux, don't the backslashes need to be forward slashes? When I ran it on my Mac, my jpegFiles was an empty Struct. When I changed them around it worked:
% Create video out of list of jpgs
clear
clc
% Folder with all the image files you want to create a movie from, choose this folder using:
ImagesFolder = uigetdir;
% Verify that all the images are in the correct time order, this could be useful if you were using any kind of time lapse photography. We can do that by using dir to map our images and create a structure with information on each file.
jpegFiles = dir(strcat(ImagesFolder,'/*.jpg'));
% Sort by date from the datenum information.
S = [jpegFiles(:).datenum];
[S,S] = sort(S);
jpegFilesS = jpegFiles(S);
% The sub-structures within jpegFilesS is now sorted in ascending time order.
% Notice that datenum is a serial date number, for example, if you would like to get the time difference in hours between two images you need to subtract their datenum values and multiply by 1440.
% Create a VideoWriter object, in order to write video data to an .avi file using a jpeg compression.
VideoFile = strcat(ImagesFolder,'/MyVideo.avi');
writerObj = VideoWriter(VideoFile);
% Define the video frames per second speed (fps)
fps = 1;
writerObj.FrameRate = fps;
% Open file for writing video data
open(writerObj);
% Running over all the files, converting them to movie frames using im2frame and writing the video data to file using writeVideo
for t = 1:length(jpegFilesS)
Frame = imread(strcat(ImagesFolder,'/',jpegFilesS(t).name));
writeVideo(writerObj,im2frame(Frame));
end
% Close the file after writing the video data
close(writerObj);
Edit: You can also use filesep so that the file separator is OS-specific. http://www.mathworks.com/help/matlab/ref/filesep.html
It would be simpler to use Windows Movie Maker [windows] or iMovie [mac]. For your purposes though you should use PowerPoint.
I need to get the total number of lines that an IO object contains before looping through each line in the IO object. How can I do this in ruby?
You can't really, unless you want to shell out to wc and parse the result of that - otherwise you'll need to do two passes - one to get the line numbers, and another to do your actual work.
(assuming we're talking about a File IO instance - neither of those approaches work for network sockets etc)
in rails (the only difference is how I generate the file object instance)
file = File.open(File.join(Rails.root, 'lib', 'assets', 'file.json'))
linecount = file.readlines.size
io.lines.count would give you the number of lines.
io.lines.each_with_index {|line, index|} would give you each line and which line number it is (starting at 0).
But I don't know if it's possible to count the number of lines without reading a file.
You may want to read a file, and then use io.rewind to read it again.
If your file is not humongous, slurp it into memory(array) and count the the number of items( ie lines).