This might seem like a strange question but i'm really curious... I have the habit of making loops like this for example :
do {
start-sleep 1
"anything"
} until ($a)
Instead of plain :
do {
"anything"
} until ($a)
Because I somehow have this feeling that if I don't use a quick sleep at every iteration, it will "spam" the command instantly a hundred times and affect performance.
My question is, am I right in thinking like this or can I use it in it's plain form without worrying about excessive memory usage?
I'm pretty sure i'm not but I developped an habit and would like your input on this ;)
Related
Myself and some other people at work have been trying to figure out exactly why this excerpt of this script runs so much faster in ISE than in the shell.
For context, the entire script (which compares AD hashes to a list of known compromised hashes), will run in ISE in about 30 minutes with the expected results. However, when invoked remotely or run locally from the shell, it takes up to 10 days in some cases.
We've found that this little bit of code in a function is where things go wonky. I'm not 100% certain, but I believe it may be resulting from the use of System.IO.StreamReader. Specifically, calling the ReadLine() method; but really not sure.
$fsHashDictionary = New-Object IO.Filestream $HashDictionary,'Open','Read','Read'
$frHashDictionary = New-Object System.IO.StreamReader($fsHashDictionary)
while (($lineHashDictionary = $frHashDictionary.ReadLine()) -ne $null) {
if($htADNTHashes.ContainsKey($lineHashDictionary.Split(":")[0].ToUpper()))
{
$foFoundObject = [PSCustomObject]#{
User = $htADNTHashes[$lineHashDictionary.Split(":")[0].ToUpper()]
Frequency = $lineHashDictionary.Split(":")[1]
Hash = $linehashDictionary.Split(":")[0].ToUpper()
}
$mrMatchedResults += $foFoundObject
}
Afaik, there isn't anything that can explain a "Script runs hundreds of times faster in ISE than in the shell" therefore I suspect the available memory differences between one and the other session are causing your script to run into performance issues.
Knowing that custom PowerShell objects are pretty heavy. To give you an idea how much memory they consume, try something like this:
$memBefore = (Get-Process -id $pid).WS
$foFoundObject = [PSCustomObject]#{
User = $htADNTHashes[$lineHashDictionary.Split(":")[0].ToUpper()]
Frequency = $lineHashDictionary.Split(":")[1]
Hash = $linehashDictionary.Split(":")[0].ToUpper()
}
$memAfter = (Get-Process -id $pid).WS
$memAfter - $memBefore
Together with the fact that arrays (as $mrMatchedResults) are mutual and therefore causing the array to be rebuild every time you use the increase assignment operator (+=), the PowerShell session might be running out of physically memory causing Windows to constantly swapping memory pages.
.Net methods like [System.IO.StreamReader] are definitely a lot faster then PowerShell cmdlets (as e.g. Get-Content) but that doesn't mean that you have to pot everything into memory. Meaning, instead of assigning the results to $lineHashDictionary (which loads all lines into memory), stream each object to the next cmdlet.
Especially For you main object, try to respect the PowerShell pipeline. As recommended in Why should I avoid using the increase assignment operator (+=) to create a collection?, you better not assign the output at all but pass the pipeline output directly to the next cmdlet (and eventually release to its destination, as e.g. display, AD, disk) to free up memory.
And if you do use .Net classes (along with the StreamReader class) make sure that you dispose the object as shown in the PowerShell scripting performance considerations article, otherwise you function might leak even more memory than required.
the performance of a complete (PowerShell) solution is supposed to be better than the sum of its parts. Meaning, don't focus too much on a single function if it concerns performance issues, instead look at you whole solution. The PowerShell pipeline gives you the opportunity to e.g. load objects from AD and process them almost simultaneously and using just a little more memory than each single object.
It's probably because ISE uses the WPF framework and benefits from hardware acceleration, a PowerShell console does not.
First question is: Is this script as fast as possible as it can get?
I'm talking about the commands in the start are they necessary/unnecessary? do they help at all when its about simple key remapping?
I would like it to run as fast as possible since I do pretty intense & fast stuff with this script.
The 2nd question is: How to prevent this script from spamclicking?
If I keep "E or R" held down, it will spam click which I do not want it to do.
How to fix this?
#NoEnv
#MaxHotkeysPerInterval 99000000
#HotkeyInterval 99000000
#KeyHistory 0
ListLines Off
Process, Priority, , A
SetBatchLines, -1
SetKeyDelay, -1, -1
SetMouseDelay, -1
SetDefaultMouseSpeed, 0
SetWinDelay, -1
SetControlDelay, -1
SendMode Input
*e::Click
return
*r::Click
return
Ins::Suspend
return
Answer to the second question, add the KeyWait command to your hotkeys, KeyWait - Syntax & Usage | AutoHotkey
*e::
Click
KeyWait, e
return
*r::
Click
KeyWait, r
return
Answer to the first question, it looks like you may have gotten the header lines from here: How to optimize the speed of a script as much as possible. The post explains each line (see below) and even includes some benchmarks for other things.
In your case, if you're only doing simple key remapping, it's probably not worth including any of these lines with exception to SendMode , Input, as that is the most reliable send mode. You mention doing "pretty intense & fast stuff with this script"; what this means exactly will determine if the header lines are necessary.
Notes
1. #NoEnv is recommended for all scripts, it disables environment variables.
2. The default #MaxHotkeysPerInterval along with #HotkeyInterval will stop your script by showing message boxes if you have some kind of rapid autofire loop in it. Just put some insane unreachable high number to ignore this limit.
3. ListLines and #KeyHistory are functions used to "log your keys". Disable them as they're only useful for debugging purposes.
4. Setting an higher priority to a Windows program is supposed to improve its performance. Use AboveNormal/A. If you feel like it's making things worse, comment or remove this line.
5. The default SetBatchLines value makes your script sleep 10 milliseconds every line. Make it -1 to not sleep (but remember to include at least one Sleep in your loops, if any!)
6. Even though SendInput ignores SetKeyDelay, SetMouseDelay and SetDefaultMouseSpeed, having these delays at -1 improves SendEvent's speed just in case SendInput is not available and falls back to SendEvent.
7. SetWinDelay and SetControlDelay may affect performance depending on the script.
8. SendInput is the fastest send method. SendEvent (the default one) is 2nd place, SendPlay a far 3rd place (it's the most compatible one though). SendInput does not obey to SetKeyDelay, SetMouseDelay, SetDefaultMouseSpeed; there is no delay between keystrokes in that mode.
9. If you're not using any SetTimer, the high precision sleep function is useful when you need millisecond reliability in your scripts. It may be problematic when used with SetTimer in some situations because this sleep method pauses the entire script. To make use of it, here's an example that waits 16,67 milliseconds:
DllCall("Sleep",UInt,16.67)
10. When using PixelSearch to scan a single pixel of a single color variation, don't use the Fast parameter. According to my benchmarks, regular PixelSearch is faster than PixelSearch Fast in that case.
11. According to the documentation (this text is found in the setup file), the Unicode x64bit version of AHK is faster, use it when available.
Okay, so I have a shell script that uses an sqlite database for storing data. However, it's a long running script that runs a lot of small queries over time.
Currently I'm calling sqlite3 'my.db' 'SELECT * FROM foo' or similar for every query, however I don't expect that this is terribly efficient, as sqlite will have to re-open the database each time (unless it sneakily keeps them open for a while?).
However, this also means that if I need to do any setup, such as specifying the timeout option, then I will need to do this every time I call sqlite3 as well.
What I'm wondering is if there is some alternative way I can structure my script, for example opening the sqlite3 process in the background in interactive mode and somehow feeding it with queries and grabbing responses. Is there a good, robust way to do this?
I've tried something along these lines, but I don't always know how many results a query will return (if any), so I end up having to do queries like SELECT COUNT(*) FROM foo WHERE bar=1; SELECT * FROM foo WHERE bar=1, so that I always get back at least one line with the number of results to expect, but this is no good if I (somehow) get more results than expected, as they will still be waiting when I run the next query.
Here's a shortened form of what I've tried (please forgive any typos, I'm sure to make some):
#!/bin/sh
mkfifo '/tmp/db_in'
mkfifo '/tmp/db_out'
# Create the sqlite process in the background (assume database setup already)
sqlite3 'my.db' </tmp/db_in >/tmp/db_out &
# Set connection parameters
echo '.timeout 1000' >/tmp/db_in
# Perform a query with count in first result
echo 'SELECT COUNT(*) FROM foo WHERE bar=1;SELECT * FROM bar WHERE bar=1;' >/tmp/db_in
read results </tmp/db_out
while [ $results -gt 0 ]; do
IFS='|'; read key value </tmp/db_out
echo "$key=$value"
results=$(($results - 1))
done
Is this along the right lines, or is there a better way to do this? I haven't been able to think of a way to get sqlite to return something at the end of output for each query, which would be a better way to do things I think. I know the echo option can be used to echo a query before its output, which I can at least use to make sure I don't read anything that's leftover from a previous query, but I really want to be able to handle arbitrary length results more correctly (without needing a count first, since this could go out of sync).
The only other alternative seems to be dumping results into a file, but as I may be handling a large number of queries I don't really want to be doing that; at least with FIFOs everything's still in memory.
[edit]
I've just noticed that some of the shells I try the above with close sqlite3 after writing to db_in, in those cases I think the fifos may need to be bound to a file-descriptor to keep them open. I'm not going to add this to the example just now as it makes it more complex, but if file-descriptors are used by a solution then it should probably be added for completeness.
I am making an app with a TON of features. My problem is that applescript seems to have a cut-off point. After a certain number of lines, the script stopps working. It basically only works until the end. Once it gets to that point it stops. I have moved the code around to make sure that it is not an error within the code. Help?
I might be wrong, but I believe a long script is not a good way to put your code.
It's really hard to read, to debug or to maintain as one slight change in a part can have unexpected consequences at the other part of you file.
If your script is very long, I suggest you break your code in multiple parts.
First, you may use functions if some part of the code is reused several times.
Another benefit of the functions is that you can validate them separately from the rest of the execution code.
Besides, it makes your code easier to read.
on doWhatYouHaveTo(anArgument)
say "hello!"
end doWhatYouHaveTo
If the functions are used by different scripts, you may want to have your functions in a seperate library that you will call at need.
set cc to load script alias ((path to library folder as string) & "Scripts:Common:CommonLibrary.app")
cc's doWhatYouHaveTo(oneArgument)
At last, a thing that I sometimes do is calling a different script with some arguments, if a long code fits for slightly different purposes:
run script file {mainFileName} with parameters {oneWay}
This last trick has a great yet curious benefit : it can accelerate the execution time for a reason I never explained (and when I say accelerate, I say reduce execution time by 17 or so for the very same code).
I had an implementation of a min heap in ruby that I wanted to test against more professional code but I cannot get Kanwei's MinHeap to work properly.
This:
mh = Containers::MinHeap.new # Min_Binary_Heap.new
for i in 0..99999
mh.push(rand(9572943))
end
t = Time.now
for i in 0..99999
mh.pop
end
t = Time.now - t
print "#{t}s"
The version I have performs the same popping operations on 100,000 values in ~2.2s, which I thought was extremely slow, but this won't even finish running. Is that expected or am I doing something wrong?
I don't think you are doing something wrong.
Looking at the source (https://github.com/kanwei/algorithms/blob/master/lib/containers/heap.rb), put a puts statement for when you finish setting up the heap. It looks like a very memory intensive operation to put the elements in (potentially resorting each time), so it might help you working through it.
I'm also not sure about him creating a node class for each actual node. Since they won't get cleaned up, there's going to be around 100,000 objects in memory by the time you are done.
Not sure how much help that is, maybe see how the source differs from your attempt?