I've just been really surprised by how slow printf from F# is. I have a number of C# programs that process large data files and write out a number of CSV files. I originally started by using fprintf writer "%s,%d,%f,%f,%f,%s" thinking that that would be simple and reasonably efficient.
However after a while I was getting a bit fed up with waiting for the files to process. (I've got 4gb XML files to go through and write out entries from them.).
When I ran my applications through a profiler, I was amazed to see printf as being one of the really slow methods.
I changed the code to not use printf and now performance is so much better. Printf performance was killing my overall application performance.
To give an example, my original code is:
fprintf sectorWriter "\"%s\",%f,%f,%d,%d,\"%s\",\"%s\",\"%s\",%d,%d,%d,%d,\"%s\",%d,%d,%d,%d,%s,%d"
sector.Label sector.Longitude sector.Latitude sector.RNCId sector.CellId
siteName sector.Switch sector.Technology (int sector.Azimuth) sector.PrimaryScramblingCode
(int sector.FrequencyBand) (int sector.Height) sector.PatternName (int sector.Beamwidth)
(int sector.ElectricalTilt) (int sector.MechanicalTilt) (int (sector.ElectricalTilt + sector.MechanicalTilt))
sector.SectorType (int sector.Radius)
And I've changed it to be the following
seq {
yield sector.Label; yield string sector.Longitude; yield string sector.Latitude; yield string sector.RNCId; yield string sector.CellId;
yield siteName; yield sector.Switch; yield sector.Technology; yield string (int sector.Azimuth); yield string sector.PrimaryScramblingCode;
yield string (int sector.FrequencyBand); yield string (int sector.Height); yield sector.PatternName; yield string (int sector.Beamwidth);
yield string (int sector.ElectricalTilt); yield string (int sector.MechanicalTilt);
yield string (int (sector.ElectricalTilt + sector.MechanicalTilt));
yield sector.SectorType; yield string (int sector.Radius)
}
|> writeCSV sectorWriter
Helper functions
let writeDelimited delimiter (writer:TextWriter) (values:seq<string>) =
values
|> Seq.fold (fun (s:string) v -> if s.Length = 0 then v else s + delimiter + v) ""
|> writer.WriteLine
let writeCSV (writer:TextWriter) (values:seq<string>) = writeDelimited "," writer values
I'm writing out files with about 30,000 rows. Nothing special.
I am not sure how much it matters, but...
Inspecting the code for printf:
https://github.com/fsharp/fsharp/blob/master/src/fsharp/FSharp.Core/printf.fs
I see
// The general technique used this file is to interpret
// a format string and use reflection to construct a function value that matches
// the specification of the format string.
and I think the word 'reflection' probably answers the question.
printf is great for writing simple type-safe output, but if you want good perf in an inner loop, you might want to use a lower-level .NET API to write output. I haven't done my own benchmarking to see.
TextWriter already buffers its output. I recommend using Write to output each value, one at a time, instead of formatting an entire line and passing it to WriteLine. On my laptop, writing 100,000 lines takes nearly a minute using your function, while, using the following function, it runs in half a second.
let writeRow (writer:TextWriter) siteName (sector:Sector) =
let inline write (value:'a) (delim:char) =
writer.Write(value)
writer.Write(delim)
let inline quote s = "\"" + s + "\""
write (quote sector.Label) ','
write sector.Longitude ','
write sector.Latitude ','
write sector.RNCId ','
write sector.CellId ','
write (quote siteName) ','
write (quote sector.Switch) ','
write (quote sector.Technology) ','
write (int sector.Azimuth) ','
write sector.PrimaryScramblingCode ','
write (int sector.FrequencyBand) ','
write (int sector.Height) ','
write (quote sector.PatternName) ','
write (int sector.Beamwidth) ','
write (int sector.ElectricalTilt) ','
write (int sector.MechanicalTilt) ','
write (int (sector.ElectricalTilt + sector.MechanicalTilt)) ','
write sector.SectorType ','
write (int sector.Radius) '\n'
Now that F# 3.1 has been preview released, the performance of printf is claimed to have increased by 40x. You might want to have a look at this:
F# 3.1 Compiler/Library Additions
Printf performance
The F# 3.1 core library sees improved performance of the printf family
of functions for type-safe formatting. For example, printing using
the following format string now runs up to 40x faster (though your
exact mileage may vary):
sprintf "%d: %d, %x %X %d %d %s" No changes in your code are needed to
take advantage of this improved performance, though you do need to be
using the F# 3.1 FSharp.Core.dll runtime component.
EDIT: This answer is only valid for simple format strings, like "%s" or "%d". See comments below.
It is also interesting to note that if you can make a curried function and reuse that, the reflection will only be carried out once. Sample:
let w = new System.IO.StringWriter() :> System.IO.TextWriter
let printer = fprintf w "%d"
let printer2 d = fprintf w "%d" d
let print1() =
for i = 1 to 100000 do
printer 2
let print2() =
for i = 1 to 100000 do
printer2 2
let time f =
let sw = System.Diagnostics.Stopwatch()
sw.Start()
f()
printfn "%s" (sw.ElapsedMilliseconds.ToString())
time print1
time print2
print1 takes 48 ms on my machine while print2 takes 1158 ms.
Related
I have some MWE below. What I want is to have a subsection of a range, interact with the rest of the range, but not itself.
For instance if the range is 1:100, I want to have a for loop that will have each index in 4:6, interact with all values of 1:100 BUT NOT 4:6.
I want to do this using ranges/filters to avoid generating temporary arrays.
In my case the total range is the number of atoms in the system. The sub-range, is the atoms in a specific molecule. I need to do calculations where each atom in a molecule interacts with all other atoms, but not the atoms in the same molecule.
Further
I am trying to avoid using if statements because that messes up parallel codes. Doing this with an if statement would be
for i=4:6
for j = 1:100
if j == 4 || j==5 || j==6
continue
end
println(i, " ", j)
end
end
I have actual indexing in my code, I would never hardcode values like the above... But I want to avoid that if statement.
Trials
The following does what I want, but I now realize that using filter is bad when it comes to memory and the amount used scales linearly with b.
a = 4:6
b = 1:100
for i in a
for j in filter((b) -> !(b in a),b)
print(i, " ", j)
end
end
Is there a way to get the double for loop I want where the outer is a sub-range of the inner, but the inner does not include the outer sub-range and most importantly is fast and does not create alot of memory usage like filter does?
If memory usage is really a concern, consider two for loops using the range components:
systemrange = 1:50
moleculerange = 4:12
for i in systemrange[1]:moleculerange[1]-1
println(i)
end
for i in moleculerange[end]+1:systemrange[end]
println(i)
end
You might be able to do each loop in its own thread.
What about creating a custom iterator?
Note that example below needs some adjustments depending on how you define the exception lists (for example for long list with non continues indices you should use binary search).
struct RangeExcept
start::Int
stop::Int
except::UnitRange{Int}
end
function Base.iterate(it::RangeExcept, (el, stop, except)=(it.except.start > 1 ? it.start : it.except.stop+1, it.stop, it.except))
new_el = el+1
if new_el in except
new_el = except.stop+1
end
el > stop && return nothing
return (el, (new_el, stop,except))
end
Now let us test the code:
julia> for i in RangeExcept(1,10,3:7)
println(i)
end
1
2
8
9
10
I'm trying to write out an image from a camera to disk. To keep it simple for post processing down stream, we chose to use a CSV format (Other systems we have use this format as opposed to binary). The issue is to write a 1MB image it takes ~1 Second.
I'm looking to speed this up by at-least 10x. Ideally we maintain the CSV format, but we can move to a binary file if needed. This all is running on a WIN 10 machine, and compiled with VS 2017.
The code below is inefficient but prints prints out a rectangle image with ',' delineation between pixels. I'm sure 99% of the inefficiency comes from the Million repetitive IO requests for every loop, but I'm not sure how to append ',' without something like this...
int toCSV(uns16 * bufferCopy, uns32 bufsize, uns32 imgWidth, string directory, string fileout)
{
std::ofstream outputFile(directory + fileout);
for (uns32 i = 0; i < bufsize; i++)
{
outputFile << (bufferCopy[i]);
if (i % (imgWidth+1) < imgWidth)
outputFile << ",";
else
outputFile << "\n";
}
outputFile.close();
return 0;
}
An example of the output looks like this (Just with ~400x~600 rows):
32.323,23.456,54.323,45.332
45.343,73.846,35.328,15.842
32.323,23.456,54.323,45.332
45.343,73.846,35.328,15.842
Update:
If I move to a binary format, I'll be using the question below as reference code:
Writing a binary file in C++ very fast
Most Frequent Character
Design a program that prompts the user to enter a string, and displays the character that appears most frequently in the string.
It is a homework question, but my teacher wasn't helpful and its driving me crazy i can't figure this out.
Thank You in advance.
This is what i have so far!
Declare String str
Declare Integer maxChar
Declare Integer index
Set maxChar = 0
Display “Enter anything you want.”
Input str
For index = 0 To length(str) – 1
If str[index] =
And now im stuck. I dont think its right and i dont know where to go with it!
It seems to me that the way you want to do it is:
"Go through every character in the string and remember the character we've seen most times".
However, that won't work. If we only remember the count for a single character, like "the character we've seen most times is 'a' with 5 occurrences", we can't know if perhaps the character in the 2nd place doesn't jump ahead.
So, what you have to do is this:
Go through every character of the string.
For every character, increase the occurrence count for that character. Yes, you have to save this count for every single character you encounter. Simple variables like string or int are not going to be enough here.
When you're done, you're left with a bunch of data looking like "a"=5, "b"=2, "e"=7,... you have to go though that and find the highest number (I'm sure you can find examples for finding the highest number in a sequence), then return the letter which this corresponds to.
Not a complete answer, I know, but that's all I'm going to say.
If you're stuck, I suggest getting a pen and a piece of paper and trying to calculate it manually. Try to think - how would you do it without a computer? If your answer is "look and see", what if the text is 10 pages? I know it can be pretty confusing, but the point of all this is to get you used to a different way of thinking. If you figure this one out, the next time will be easier because the basic principles are always the same.
This is the code I have created to count all occurences in a string.
String abc = "aabcabccc";
char[] x = abc.toCharArray();
String _array = "";
for(int i = 0; i < x.length; i++) //copy distinct data to a new string
{
if(_array.indexOf(x[i]) == -1)
_array = _array+x[i];
}
char[] y = _array.toCharArray();
int[] count1 = new int[_array.length()];
for(int j = 0; j<x.length;j++) //count occurences
{
count1[new String(String.valueOf(y)).indexOf(x[j])]++;
}
for(int i = 0; i<y.length;i++) //display
{
System.out.println(y[i] + " = " + count1[i]);
}
I need to read some data from a file in chuck of 128M, and then for each line, I will do some processing, naive way to do is using split to convert the string into collection of lines and then process each line, but maybe that is not effective as it will create a collection which simply stores the temp result which could be costy. Is there is a way with better performance?
The file is huge, so I kicked off several thread, each thread will pick up 128 chuck, in the following script rawString is a chuck of 128M.
randomAccessFile.seek(start)
randomAccessFile.read(byteBuffer)
val rawString = new String(byteBuffer)
val lines=rawString.split("\n")
for(line <- lines){
...
}
It'd be better to read text line by line:
import scala.io.Source
for(line <- Source.fromFile("file.txt").getLines()) {
...
}
I'm not sure what you're going to do with the trailing bits of lines at the beginning and end of the chunk. I'll leave that to you to figure out--this solution captures everything delimited on both sides by \n.
Anyway, assuming that byteBuffer is actually an array of bytes and not a java.nio.ByteBuffer, and that you're okay with just handling Unix line encodings, you would want to
def lines(bs: Array[Byte]): Array[String] = {
val xs = Array.newBuilder[Int]
var i = 0
while (i<bs.length) {
if (bs(i)=='\n') xs += i
i += 1
}
val ix = xs.result
val ss = new Array[String](0 max (ix.length-1))
i = 1
while (i < ix.length) {
ss(i-1) = new String(bs, ix(i-1)+1, ix(i)-ix(i-1)-1)
i += 1
}
ss
}
Of course this is rather long and messy code, but if you're really worried about performance this sort of thing (heavy use of low-level operations on primitives) is the way to go. (This also takes only ~3x the memory of the chunk on disk instead of ~5x (for mostly/entirely ASCII data) since you don't need the full string representation around.)
the question about a comparison of the upper and lower case..how can i do that in my sort function.any idea?
Ex: Inputfile : " I am Happy! "
Outputfile:
Happy!
I
am
thats what's happen with my program, but i would like so have:
am
I
Happy
My code:
-module(wp)
-compile([export_all]). % Open the File
sortFile(File1,File2) ->
{ok, File_Read} = file:read_file(File1),
% making a list
Liste = string:tokens(binary_to_list(File_Read), "\n "),
% isort List
Sort_List = isort(Liste),
ISort = string:join(Sort_List,"\n"),
%Written in the File.
{ok,Datei_Schreiben} = file:open(File2, write),
file:write(File_Write, Isort),
file:close(File_Write).
isort([]) -> [];
isort([X|XS])-> insert(X, isort(XS)).
insert(Elem, []) -> [Elem];
insert(Elem, [X|XS]) when Elem= [Elem,X|XS];
insert(Elem, [X|XS]) -> [X|insert(Elem,XS)].
how about something like this:
qsort1([]) -> [];
qsort1([H|T]) ->
qsort1([X || X <- T, string:to_lower(X) < string:to_lower(H)])
++ [H]
++ qsort1([X || X <- T, string:to_lower(X) >= string:to_lower(H)]).
7> qsort1(["I", "am","Happy"]).
["am","Happy","I"]
I believe that "happy" sorts less than "i"
8> "happy" < "i".
true
which is why my sorted order is a little differenct than your original post.
When there is at least N*log2(N) comparisons in sorting there is not necessary to make N*log2(N) but only N case transformations. (Almost all perl developers knows this trick.)
{ok, Bin} = file:read_file(?INPUT_FILE),
Toks = string:tokens(binary_to_list(Bin),"\n "),
Result = [[X,$\n] || {_,X} <- lists:sort([{string:to_lower(X), X} || X<-Toks])],
file:write_file(?OUTPUT_FILE, Result).
BTW lists:sort/1 merge sort has granted N*log2(N) and is pretty efficient in contrary to concise but less efficient quick sort implementation. What worse, quick sort has N^2 worst case.
Now, depending on whether you are on Windows or Unix/Linux, the lines in the files will be ended with different characters. Lets go with windows where its normally \r\n. Now assuming the input files are not too big, we can read them at once into a binary. The stream of data we get must be split into lines, then each line split into words (spaces). If the input file is very big and cannot fit in memory, then you will have to read it, line by line, in which case you might need an IN-Memory buffer to hold all the words ready for sorting, this would require ETS Table, or Memcached (an option i wont illustrate here). Lets write the code
-module(sick_sort).
-compile(export_all).
-define(INPUT_FILE,"C:/SICK_SORT/input.txt").
-define(OUTPUT_FILE_PATH,"C:/SICK_SORT/").
-define(OUTPUT_FILENAME,"output.txt").
start()->
case file:read_file(?INPUT_FILE) of
{ok,Binary} ->
%% input file read
AllLines = string:tokens(binary_to_list(Binary),"\r\n"),
SortedText = lists:flatten([XX ++ "\r\n" || XX <- lists:sort(string:tokens(AllLines," "))]),
EndFile = filename:join(?OUTPUT_FILE_PATH,?OUTPUT_FILENAME),
file:write_file(EndFile,SortedText),
ok;
Error -> {error,Error}
end.
That should work. Change the macros in the source file to suit your settings and then, just run sick_sort:start().
you have to compare low cap in your sort function:
(nitrogen#127.0.0.1)25> F= fun(X,Y) -> string:to_lower(X) < string:to_lower(Y) end.
#Fun<erl_eval.12.111823515>
(nitrogen#127.0.0.1)26> lists:sort(F,["I","am","Happy"]).
["am","Happy","I"]
(nitrogen#127.0.0.1)27>
EDIT:
In your code, the function that allows to sort the list are the operators > and < (if you want to see replicated string one of them should include =, otherwise you will do a usort). If you want to use a different comparison you can define it in a normal or anonymous function and then use it in the quicksort:
mycompare(X,Y) ->
string:to_lower(X) < string:to_lower(Y).
quicksort ([])->[];
([X|XS])-> quicksort([Y||Y<-XS,mycompare(X,Y)])++[X]++quicksort([Y||Y<-XS,mycompare(X,Y) == false]).