I have table and it updated every time if the loop didn't find the Match in table key.
for d in io.popen("ls -a /mnt/mediashare/net"):lines() do
OnlineCheck[#OnlineCheck+1] = d
end
First I add the values in to tables then I run another loop to find the match form it if it founds the match the loop breaks but if not then it should reload the table with new values and start matching again. I can't figured it out how to do this all in single loop.
for d1,d2 in pairs(OnlineCheck) do
if d2==NASFolder then
print("NAS Connected")
break
else
print("NAS is Offline")
end
end
In Lua 5.2 you should be able to do:
local connected = os.execute(('[ -d /mnt/mediashare/net/%q ]'):format(NASFolder))
(For Lua 5.1, compare os.execute()'s result with 0.)
You don't need to collect lines in table, perform match on the fly:
function findUntilMatch(NASFolder)
for line in io.popen("ls -a /mnt/mediashare/net"):lines() do
if line == NASFolder then
print("NAS Connected")
return
end
print("NAS is Offline")
return findUntilMatch(NASFolder) -- tail call recursion
end
Keep in mind - it may run forever ...
Update: I have reread the question and now I'm not sure what topic starter means as:
I can't figured it out how to do this all in single loop.
My original answer was the attempt to write all-in-single-loop including reread info from device.
It is also possible to perform the search by shell (inspired by another answer):
local connected = io.popen("ls -a /mnt/mediashare/net | grep '^" .. NASFolder .. "$'"):read()
Device reread logic is missed in this example.
Related
I am building a file-based index for the sorted haveibeenpwned passwords text file and it got me wondering what's the fastest way to do this?
I figured a good way to build a quickly grep-able index would be split the sorted file into 256 files named with the first two hex digits (i.e. FF.txt, FE.txt, etc). I found ripgrep rg to be about 5 times faster than grep on my computer. So I tried something like this:
for i in {255..0}
do
start=$(date +%s)
hex="$(printf '%02x' $i | tr [:lower:] [:upper:])"
rg "^$hex" pwned-passwords-ntlm-ordered-by-hash-v4.txt > ntlm/$hex-ntlm.txt
echo 0x$hex completed in $(($(date +%s) - $start)) seconds
done
This is the fastest solution I could come up with. ripgrep is able to create each file in 25 seconds. So I'm looking at about 100 minutes to create this index. When I split the job in half, and run them in parallel, each pair of files is created in 80 seconds. So it seems best to just let ripgrep work its magic and work in series.
Obviously, I won't be indexing this list too often, but it's just fun to think about. Any thoughts on a faster way (aside from using a database) to index this file?
ripgrep, like any other tool that's able to work with unsorted input files at all, is the wrong tool for this job. When you're trying to grep sorted inputs, you want something that can bisect your input file to find a position in logarithmic time. For big enough inputs, even a slow O(log n) implementation will be faster than a highly optimized O(n) one.
pts-line-bisect is one such tool, though of course you're also welcome to write your own. You'll need to write it in a language with full access to the seek() syscall, which is not exposed in bash.
You are reading through the file 256 times, doing a full file scan every time. Consider an approach that reads the file once, writing each line into an open file descriptor. I'm thinking python would be an easy choice of implementation (if that's your thing). You could optimize by keeping the file open until you hit a new hex code at the beginning of the line. If you want to be even more clever, there is no need to go through the sorted file line by line. Based on Charles Duffy's hint, you could create a heuristic for sampling the file (using seek()) to get to the next hex value. Once the program has found the byte offset of the next hex value, the block of bytes can be written to the new file. However, since this is tagged as 'bash' let's keep the solution set in that domain:
while
read line
do
hex=${line:0:2}
echo $line >> ntlm/$hex-ntlm.txt
done < pwned-passwords-ntlm-ordered-by-hash-v4.txt
I wrote a Python3 script that solves fast binary-search lookups in the hash file without having to create an index. It doesn't directly address your question (indexing) but probably solves the underlying problem that you wanted to solve with an index - to quickly look up individual hashes. This script checks hundreds of passwords in seconds.
import argparse
import hashlib
parser = argparse.ArgumentParser(description='Searches passwords in https://haveibeenpwned.com/Passwords database.')
parser.add_argument('passwords', metavar='TEST', type=str, help='text file with passwords to test, one per line, utf-8')
parser.add_argument('database', metavar='DATABASE', type=str, help='the downloaded text file with sha-1:count')
args = parser.parse_args()
def search(f: object, pattern: str) -> str:
def search(left, right: int) -> str:
if left >= right:
return None
middle = (left + right) // 2
if middle == 0:
f.seek(0, 0)
test = f.readline()
else:
f.seek(middle - 1, 0)
_ = f.readline()
test = f.readline()
if test.upper().startswith(pattern):
return test
elif left == middle:
return None
elif pattern < test:
return search(left, middle)
else:
return search(middle, right)
f.seek(0, 2)
return search(0, f.tell())
fsource = open(args.passwords)
fdatabase = open(args.database)
source_lines = fsource.readlines()
for l in source_lines:
line = l.strip()
hash_object = hashlib.sha1(line.encode("utf-8"))
pattern = hash_object.hexdigest().upper()
print("%s:%s" % (line, str(search(fdatabase, pattern)).strip()))
fsource.close()
fdatabase.close()
I am receiving some data which is parsed in a Ruby script, a sample of the parsed data looks like this;
{"address":"00","data":"FF"}
{"address":"01","data":"00"}
That data relates to the status (on/off) of plant items (Fans, coolers, heaters etc.) the address is a HEX number to tell you which set of bits the data refers to. So in the example above the lookup table would be; Both of these values are received as HEX as in this example.
Bit1 Bit2 Bit3 Bit4 Bit5 Bit6 Bit7 Bit8
Address 00: Fan1 Fan2 Fan3 Fan4 Cool1 Cool2 Cool3 Heat1
Address 01: Hum1 Hum2 Fan5 Fan6 Heat2 Heat3 Cool4 Cool5
16 Addresses per block (This example is 00-0F)
Data: FF tells me that all items in Address 00 are set on (high/1) I then need to output the result of the lookup for each individual bit e.g
{"element":"FAN1","data":{"type":"STAT","state":"1"}}
{"element":"FAN2","data":{"type":"STAT","state":"1"}}
{"element":"FAN3","data":{"type":"STAT","state":"1"}}
{"element":"FAN4","data":{"type":"STAT","state":"1"}}
{"element":"COOL1","data":{"type":"STAT","state":"1"}}
{"element":"COOL2","data":{"type":"STAT","state":"1"}}
{"element":"COOL3","data":{"type":"STAT","state":"1"}}
{"element":"HEAT1","data":{"type":"STAT","state":"1"}}
A lookup table could be anything up to 2048 bits (though I don't have anything that size in use at the moment - this is maximum I'd need to scale to)
The data field is the status of the all 8 bits per address, some may be on some may be off and this updates every time my source pushes new data at me.
I'm looking for a way to do this in code ideally for the lay-person as I'm still very new to doing much with Ruby. There was a code example here, but it was not used in the end and has been removed from the question so as not to confuse.
Based on the example below I've used the following code to make some progress. (note this integrates with an existing script all of which is not shown here. Nor is the lookup table shown as its quite big now.)
data = [feeder]
data.each do |str|
hash = JSON.parse(str)
address = hash["address"]
number = hash["data"].to_i(16)
binary_str = sprintf("%0.8b", number)
binary_str.reverse.each_char.with_index do |char, i|
break if i+1 > max_binary_digits
mouse = {"element"=>+table[address][i], "data"=>{"type"=>'STAT', "state"=>char}}
mousetrap = JSON.generate(mouse)
puts mousetrap
end
end
This gives me an output of {"element":"COOL1","data":{"type":"STAT","state":"0"}} etc... which in turn gives the correct output via my node.js script.
I have a new problem/query having got this to work and captured a whole bunch of data from last night & this morning. It appears that now I've built my lookup table I need some of the results to be modified based on the result of the lookup. I have other sensors which need to generate a different output to feed my SVG for example;
FAN objects need to output {"element":"FAN1","data":{"type":"STAT","state":"1"}}
DOOR objects need to output {"element":"DOOR1","data":{"type":"LAT","state":"1"}}
SWIPE objects need to output {"element":"SWIPE6","data":{"type":"ROUTE","state":"1"}}
ALARM objects need to output {"element":"PIR1","data":{"type":"PIR","state":"0"}}
This is due to the way the SVG deals with updating - I'm not in a position to modify the DOM stuff so would need to fix this in my Ruby script.
So to address this what I ended up doing was making an exact copy of my existing lookup table and rather than listing the devices I listed the type of output like so;
Address 00: STAT STAT STAT ROUTE ROUTE LAT LAT PIR
Address 01: PIR PIR STAT ROUTE ROUTE LAT LAT PIR
This might be very dirty (and it also means I have to duplicate my lookup table, but it actually might be better for my specific needs as devices within the dataset could have any name (I have no control over the received data) Having built a new lookup table I modified the code I had been provided with below and already used for the original lookup but I had to remove these 2 lines. Without removing them I was getting the result of the lookup output 8 times!
binary_str.reverse.each_char.with_index do |char, i|
break if i+1 > max_binary_digits
The final array was built using the following;
mouse = {"element"=>+table[address][i], "data"=>{"type"=>typetable[address][i], "state"=>char}}
mousetrap = JSON.generate(mouse)
puts mousetrap
This gave me exactly the output I require and was able to integrate with both the existing script, node.js websocket & mongodb 'state' database (which is read on initial load)
There is one last thing I'd like to try and do with this code, when certain element states are set to 1 I'd like to be able to look something else up (and then use that result) I'm thinking this may be best done with a find query to my MongoDB and then just use the result. Doing that would hit the db for every query, but there would only ever be a handful or results so most things would return null which is fine. Am I along the right method of thinking?
require 'json'
table = {
"00" => ["Fan1", "Fan2", "Fan3"],
"01" => ["Hum1", "Hum2", "Fan5"],
}
max_binary_digits = table.first[1].size
data = [
%Q[{"address": "00","data":"FF"}],
%Q[{"address": "01","data":"00"}],
%Q[{"address": "01","data":"03"}],
]
data.each do |str|
hash = JSON.parse(str)
address = hash["address"]
number = hash["data"].to_i(16)
binary_str = sprintf("%0.8b", number)
p binary_str
binary_str.reverse.each_char.with_index do |char, i|
break if i+1 > max_binary_digits
puts %Q[{"element":#{table[address][i]},"data":{"type":"STAT","state":"#{char}"}}}]
end
puts "-" * 20
end
--output:--
"11111111"
{"element":Fan1,"data":{"type":"STAT","state":"1"}}}
{"element":Fan2,"data":{"type":"STAT","state":"1"}}}
{"element":Fan3,"data":{"type":"STAT","state":"1"}}}
--------------------
"00000000"
{"element":Hum1,"data":{"type":"STAT","state":"0"}}}
{"element":Hum2,"data":{"type":"STAT","state":"0"}}}
{"element":Fan5,"data":{"type":"STAT","state":"0"}}}
--------------------
"00000011"
{"element":Hum1,"data":{"type":"STAT","state":"1"}}}
{"element":Hum2,"data":{"type":"STAT","state":"1"}}}
{"element":Fan5,"data":{"type":"STAT","state":"0"}}}
--------------------
My answer assumes Bit1 in your table is the least significant bit, if that is not the case remove .reverse in the code.
You can ask me anything you want about the code.
Currently trying to generate a random number in a specific range;
and ensure that it would be unique against others stored records.
Using Mysql. Could be like an id, incremented; but can't be it.
Currently testing other existing records in an 'expensive' manner;
but I'm pretty sure that there would be a clean 1/2 lines of code to use
Currently using :
test = 0
Order.all.each do |ord|
test = (0..899999).to_a.sample.to_s.rjust(6, '0')
if Order.find_by_number(test).nil? then
break
end
end
return test
Thanks for any help
Here your are my one-line solution. It is also the quicker one since calls .pluck to retrieve the numbers from the Order table. .select instantiates an "Order" object for every record (that is very costly and unnecessary) while .pluck does not. It also avoids to iterate again each object with a .map to get the "number" field. We can avoid the second .map as well if we convert, using CAST in this case, to a numeric value from the database.
(Array(0...899999) - Order.pluck("CAST('number' AS UNSIGNED)")).sample.to_s.rjust(6, '0')
I would do something like this:
# gets all existing IDs
existing_ids = Order.all.select(:number).map(&:number).map(&:to_i)
# removes them from the acceptable range
available_numbers = (0..899999).to_a - existing_ids
# choose one (which is not in the DB)
available_numbers.sample.to_s.rjust(6, '0')
I think, you can do something like below :
def uniq_num_add(arr)
loop do
rndm = rand(1..15) # I took this range as an example
# random number will be added to the array, when the number will
# not be present
break arr<< "%02d" % rndm unless arr.include?(rndm)
end
end
array = []
3.times do
uniq_num_add(array)
end
array # => ["02", "15", "04"]
I'm trying to display total calls from a twilio object as well as unique calls.
The total calls is simple enough:
# set up a client to talk to the Twilio REST API
#sub_account_client = Twilio::REST::Client.new(#account_sid, #auth_token)
#subaccount = #sub_account_client.account
#calls = #subaccount.calls
#total_calls = #calls.list.count
However, I'm really struggling to figure out how to display unique calls (people sometimes call back form the same number and I only want to count calls from the same number once). I'm thinking this is a pretty simple method or two but I've burnt quite a few hours trying to figure it out (still a ruby noob).
Currently I've been working it in the console as follows:
#sub_account_client = Twilio::REST::Client.new(#account_sid, #auth_token)
#subaccount = #sub_account_client.account
#subaccount.calls.list({})each do |call|
#"from" returns the phone number that called
print call.from
end
This returns the following strings:
+13304833615+13304833615+13304833615+13304833615+13304567890+13304833615+13304833615+13304833615
There are only two unique numbers there so I'd like to be able to return '2' for this.
Calling class on that output shows strings. I've used "insert" to add a space then have done a split(" ") to turn them into arrays but the output is the following:
[+13304833615][+13304833615][+13304833615][+13304833615][+13304567890][+13304833615][+13304833615][+13304833615]
I can't call 'uniq' on that and I've tried to 'flatten' as well.
Please enlighten me! Thanks!
If what you have is a string that you want to manipulate the below works:
%{+13304833615+13304833615+13304833615+13304833615+13304567890+13304833615+13304833615+13304833615}.split("+").uniq.reject { |x| x.empty? }.count
=> 2
However this is more ideal:
#subaccount.calls.list({}).map(&:from).uniq.count
Can you build an array directly instead of converting it into a string first? Try something like this perhaps?
#calllist = []
#subaccount.calls.list({})each do |call|
#"from" returns the phone number that called
#calllist.push call.from
end
you should then be able to call uniq on #calllist to shorten it to the unique members.
Edit: What type of object is #subaccount.calls.list anyway?
uniq should work for creating a unique list of strings. I think you may be getting confused by other non-related things. You don't want .split, that's for turning a single string into an array of word strings (default splits by spaces). Which has turned each single number string, into an array containing only that number. You may also have been confused by performing your each call in the irb console, which will return the full array iterated on, even if your inner loop did the right thing. Try the following:
unique_numbers = #subaccount.calls.list({}).map {|call| call.from }.uniq
puts unique_numbers.inspect
I'm writing a small parser for Google and I'm not sure what's the best way to design it. The main problem is the way it will remember the position it stopped at.
During parsing it's going to append new searches to the end of a file and go through the file startig with the first line. Now I want to do it so, that if for some reason the execution is interrupted, the script knows the last search it has accomplished successfully.
One way is to delete a line in a file after fetching it, but in this case I have to handle order that threads access file and deleting first line in a file afaik can't be done processor-effectively.
Another way is to write the number of used line to a text file and skip the lines whose numbers are in that file. Or maybe I should use some database instead? TIA
There's nothing wrong with using a state file. The only catch will be that you need to ensure you have fully committed your changes to the state file before your program enters a section where it may be interrupted. Typically this is done with an IO#flush call.
For example, here's a simple state-tracking class that works on a line-by-line basis:
class ProgressTracker
def initialize(filename)
#filename = filename
#file = open(#filename)
#state_filename = File.expand_path(".#{File.basename(#filename)}.position", File.dirname(#filename))
if (File.exist?(#state_filename))
#state_file = open(#state_filename, File::RDWR)
resume!
else
#state_file = open(#state_filename, File::RDWR | File::CREAT)
end
end
def each_line
#file.each_line do |line|
mark_position!
yield(line) if (block_given?)
end
end
protected
def mark_position!
#state_file.rewind
#state_file.puts(#file.pos)
#state_file.flush
end
def resume!
if (position = #state_file.readline)
#file.seek(position.to_i)
end
end
end
You use it with an IO-like block call:
test = ProgressTracker.new(__FILE__)
n = 0
test.each_line do |line|
n += 1
puts "%3d %s" % [ n, line ]
if (n == 10)
raise 'terminate'
end
end
In this case, the program reads itself and will stop after ten lines due to a simulated error. On the second run it should display the next ten lines, if there are that many, or simply exit if there's no additional data to retrieve.
One caveat is that you need to remove the .position file associated with the input data if you want the file to be reprocessed, or if the file has been reset. It's also not possible to edit the file and remove earlier lines or it will throw off the offset tracking. So long as you're simply appending data to the file, or restarting it, everything will be fine.