I have always used the Common Lisp time macro for checking the speed of an implementation. However, I would like to be able to write the timing output to a stream (file) so that I can do some statistical analysis on it or to filter it in some way.
So far I have not found any commands or libraries that make this possible for me. I would greatly appreciate assistance with this.
(with-open-file (*trace-output* "/tmp/time.text"
:direction :output)
(time (sleep 1)))
Related
As I understand, MATLAB cannot use pass by reference when sending arguments to other functions. I am doing audio processing, and I frequently have to pass waveforms as arguments into functions, and because MATLAB uses pass by value for these arguments, it really eats up a lot of RAM when I do this.
I was considering using global variables as a method to pass my waveforms into functions, but everywhere I read there seems to be a general opinion that this is a bad idea, for organization of code, and potentially performance issues... but I haven't really read any detailed answers on how this might impact performance...
My question: What are the negative impacts of using global variables (with sizes > 100MB) to pass arguments to other functions in MATLAB, both in terms of 1) performance and 2) general code organization and good practice.
EDIT: From #Justin's answer below, it turns out MATLAB does on occasion use pass by reference when you do not modify the argument within the function! From this, I have a second related question about global variable performance:
Will using global variables be any slower than using pass by reference arguments to functions?
MATLAB does use pass by reference, but also uses copy-on-write. That is to say, your variable will be passed by reference into the function (and so won't double up on RAM), but if you change the variable within the the function, then MATLAB will create a copy and change the copy (leaving the original unaffected).
This fact doesn't seem to be too well known, but there's a good post on Loren's blog discussing it.
Bottom line: it sounds like you don't need to use global variables at all (which are a bad idea as #Adriaan says).
While relying on copy on write as Justin suggested is typically the best choice, you can easily implement pass by reference. With Matlab oop being nearly as fast as traditional functions in Matlab 2015b or newer, using handle is a reasonable option.
I encountered an interesting use case of a global variable yesterday. I tried to parallellise a piece of code (1200 lines, multiple functions inside the main function, not written by me), using parfor.
Some weird errors came out and it turned out that this piece of code wrote to a log file, but used multiple functions to write to the log file. Rather than opening and closing the relevant log file every time a function wanted to write to it, which is very slow, the file ID was made global, so that all write-functions could access it.
For the serial case this made perfect sense, but when trying to parallellise this, using global apparently breaks the scope of a worker instance as well. So suddenly we had 4 workers all trying to write into the same log file, which resulted in some weird errors.
So all in all, I maintain my position that using global variables is generally a bad idea, although I can see its use in specific cases, provided you know what you're doing.
Using global variables in Matlab may increase performance alot. This is because you can avoid copying of data in some cases.
Before attempting to gain such performance tweaks, think carefully of the cost to your project, in terms of the many drawbacks that global variables come with. There are also pitfalls to using globals with bad consequences to performance, and those may be difficult to avoid(although possible). Any code that is littered with globals tend to be difficult to comprehend.
If you want to see globals in use for performance, you can look at this real-time toolbox for optical flow that I made. This is the only project in native Matlab that is capable of real-time optical flow that I know of. Using globals was one of the reasons this was doable. It is also a reason to why the code is quite difficult to grasp: Globals are evil.
That globals can be used this way is not a way to argue for their use, rather it should be a hint that something should be updated with Matlabs unflexible notions of workspace and inefficient alternatives to globals such as guidata/getappdata/setappdata.
Racket has the nice read-bytes-async! function, which I believe exists in every other programming language in the world. It reads what it can from an input stream, without blocking, into a buffer, returning the number of bytes written.
Said function seems like an absolutely essential function for efficiently implementing, say, the Unix cat tool, yet Chicken Scheme seems to lack any such function. Of course, I can use (read-byte) and (write-byte), but that is slow and eats up all my CPU.
Even (copy-port) seems to not have any such implementation. Instead, before the stream is closed, the data is copied buffer-by-buffer only when the buffers fill. This means that (copy-port (current-input-port) (current-output-port)) does not behave like cat at all.
Am I just suffering from a terrible blind spot in reading the documentation, or does Chicken shockingly actually lack such a function? So cat can't even be written efficiently in Chicken?
I fixed my problem. The posix library has the file-read function that does what I want, albeit on a file descriptor. Fortunately, ports in Chicken are just thin wrappers around file descriptors; there is a port to file descriptor converter in the posix library as well.
Interestingly, these functions work on Windows as well. posix seems to not be limited to POSIX systems.
as you said the posix unit is the key ,but to your question what seems more relevant is set-buffering-mode!
this applies to any port.
So here i was trying to figure out how to log any anomalies i might get inside my code to a log file. First i noticed the trace function, but then i saw that it only outputs to the stdin .
Then i see the logger module , but that runs inside the IO monad so its a bit of a hassle what with compromising purity and all. Then i figured that if i made a function a->b->b with the a parameter being of type IO () in my case all would be ok.
Indeed the compiler didn't see anything wrong with it but alas the append was never actually called so i was still back to basics. What i actually want to know is :
a) Are there any functions that perform IO while still having a pure signature (like unsafePerformIO) that could help me with my logging
b) is there any way to force the compiler to evaluate the first parameter in the function i built even though i's never actually used?
thank you guys in advance
Then i figured that if i made a function a->b->b with the a parameter being of type IO () in my case all would be ok.
Nope, wrong. This will do nothing, even if you "evaluate" the first argument. You cannot implement trace without unsafePerformIO.
IO values are just values, no more. Only when they happen in the course of the execution of main (or due to unsafePerformIO) are they actually executed.
It's not clear, though -- trace outputs to stderr. Is there a reason you can't just do
./MyHaskellExecutable 2>dumpStdErrToThisFile
Logging is side effecting, so has to be in some monad for that effect. Doing otherwise risk the compiler optimizing away your semantically-unnecessary logging calls.
If you are building an app with a plan to support logging, you will need to have it run in some kind of logging environment. IO is overkill, but perhaps a simpler Log monad would be more appropriate (kind of a magic Writer, with ST-like properties of local encapsulation).
Does anyone know how to use Win32's BitBlt() using Ruby? It seems like you need to use a destination DC (device context) and how can Ruby handle that? I use GetPixel() and it is really slow even to get 100 pixels (takes about 10 seconds). thanks.
Second Carl's other post: use win32 screenshot http://github.com/jarmo/win32screenshot
How about writing all of your image processing code in C++ as a Win32 executable and launching it as a separate process or calling into it from Ruby using the Win32API class? It makes sense to do it this way instead of writing a cludge in Ruby which will never perform as well.
I'd just like to note that I'm seeing a similar thing. GetPixel is exceedingly slow on Vista. I've determined that this is the case when you're using that Aero thing. If you set desktop theme to one of the "Classic" settings, it's fast again. And the OP's estimate of ~10 seconds is not wrong, believe it or not. It's really exceedingly slow.
I, too, am looking for a BitBlt in Ruby. I expect I'll need to figure it out/write it, but was hoping someone else has already posted an example.
I wouldn't go quite as far as Ed; chances are you only want to do some relatively simple and commonplace things that a normal GUI would do. For this, such libraries already exist and don't need to be written.
One example would be wxRuby.
I'm afraid I'm not familiar with it, but maybe it already offers some screen-grabbing utility methods.
I'd like to know how debugging is achieved in a lazy functional language.
Can you use breakpoints, print statements and traditional techniques? Is this even a good idea?
It is my understanding that pure functional programming does not allow side-effects, with the exception of monads.
Order of execution is also not guaranteed.
Would you have to program a monad for every section of code you want to test?
I'd like some insight into this question from someone more experienced in this area.
Nothing prevents you from using breakpoints in a lazily evaluated functional program. The difference to eager evaluation is when the program will stop at the breakpoint and what the trace will look like. The program will stop when the expression a breakpoint is set on is actually being reduced (obviously).
Instead of the stack trace you're used to you get the reductions that led up to the reduction of the expression with the breakpoint on it.
Small silly example. You have this Haskell program.
add_two x = 2 + x
times_two x = 2 * x
foo = times_two (add_two 42)
And you put a breakpoint on the first line (add_two), then evaluate foo. When the program stops on the breakpoint, in an eager language you'd expect to have a trace like
add_two
foo
and times_two hasn't even begun to be evaluated, but in the GHCi debugger you get
-1 : foo (debug.hs:5:17-26)
-2 : times_two (debug.hs:3:14-18)
-3 : times_two (debug.hs:3:0-18)
-4 : foo (debug.hs:5:6-27)
<end of history>
which is the list of reductions that led up to the reduction of the expression you put the breakpoint on. Note that it looks like times_two "called" foo even though it does not do so explicitly. You can see from this that the evaluation of 2 * x in times_two (-2) did force the evaluation of (add_two 42) (-1) from the foo line. From there you can perform a step as in an imperative debugger (perform the next reduction).
Another difference to debugging in an eager language is that variables may be not yet evaluated thunks. For example, at step -2 in the above trace and inspect x, you'll find it's still an unevaluated thunk (indicated by brackets in GHCi).
For far more detailed information and examples (how to step through the trace, inspect values, ...), see the GHCi Debugger section in the GHC manual. There's also the Leksah IDE which I haven't used yet as I'm a VIM and terminal user, but it has a graphical frontend to the GHCi debugger according to the manual.
You also asked for print statements. Only with pure functions, this is not so easily possible as a print statement would have to be within the IO monad. So, you have a pure function
foo :: Int -> Int
and wish to add a trace statement, the print would return an action in the IO monad and so you'd have to adjust the signature of the function you wish to put that trace statement in, and the signatures of the functions that call it, ...
This is not a good idea. So, you need some way to break purity to achieve trace statements. In Haskell, this can be done with unsafePerformIO. There's the Debug.Trace module that already has a function
trace :: String -> a -> a
which outputs the string and returns the second parameter. It would be impossible to write as a pure function (well, if you intend to really output the string, that is). It uses unsafePerformIO under the hood. You can put that into a pure function to output a trace print.
Would you have to program a monad for every section of code you want to test?
I'd suggest rather the opposite, make as many functions pure as possible (I'm assuming here you mean the IO monad for printing, monads are not necessarily impure). Lazy evaluation allows you to separate IO code from processing code very cleanly.
Whether imperative debugging techniques are a good idea or not depends on the situation (as usual). I find testing with QuickCheck/SmallCheck much more useful than unit testing in imperative languages, so I'd go that route first to avoid as much debugging as possible. QuickCheck properties actually make nice concise function specifications (a lot of test code in imperative languages looks like just another blob of code to me).
One trick to avoid a lot of debugging is to decompose the function into many smaller subfunctions and test as many of them as possible. This may be a bit unusal when coming from imperative programming, but it's a good habit no matter what language you're using.
Then again, debugging != testing and if something goes wrong somewhere, breakpoints and traces may help you out.
I don't think this topic can be dealt within a short space. Please read the papers available at the following links:
A Theory of Tracing Pure Functional Programs.
The Haskell Tracer publications.
Haskell Debugging Technologies.
I've never delved into anything terribly complicated in Haskell, but the fact that side effects are virtually gone has eliminated most of the need for debugging. Pure functions are extremely simple to test and verify without a debugger.
On the other hand, I did experience a couple times I needed to debug something within a monad, in which case I already was able to print/log/whatever.
At least for smaller programs or systems, debugging kind of goes out the window. Strong typing and static type-checking really further eliminate the traditional bugs you find in procedural programming. Most bugs, if any, are logical bugs (called the wrong functions, mathematical error, etc) -- very easy to test interactively.
From experience with Clojure (which is lazy, functional, and encourages but does not enforce purity):
You can set breakpoints just as with any other language. However, becuase of lazy evaluation, these might not get called immediately, but will be hit as soon as evaluation the lazy structure is forced.
In lazy functional languages that allow side effects (Clojure included) you can insert printlns and other debug logging relatively easily. I personally find these very useful. You have to be careful about when these get called because of laziness, but if you don't see the output at all it can be a hint that your code isn't being evaluated because of laziness.....
Having said all the above, I have never so far needed to resort to the debugger. Often a few simple tests (perhaps on the REPL) are enough to verify that functional code is working correctly, and if these fail then it's usually quite obvious what is going wrong.
Allow me to advertise a tool of my own to debug laziness problems. It helped me resolve in an hour a laziness-related memory leak that I already spent 2 days debugging.
http://www.haskell.org/pipermail/haskell-cafe/2012-January/098847.html
http://hackage.haskell.org/package/htrace