I'm trying to define methods to parse through an apache log file and pull ip addresses, URLs, requests per hour, and error codes. I've got everything working outside of methods, but when attempting to put that code into the methods I keep getting the error message "Stack level too deep." Here is the code in question.
class CommonLog
def initialize(logfile)
#logfile = logfile
end
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = #readfile.to_s.split(" ")
end
def ip_histogram
#ip_count = 0
#readfile.each_index { |index|
if (#readfile[index] =~ /\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}/ )
puts #readfile[index]
puts #ip_count += 1
end
}
end
def url_histogram
url_count = 0
cleaned_file.each_index { |index|
if (cleaned_file[index] =~ /\/{1}(([a-z]{4,})|(\~{1}))\:{0}\S+/ )
puts cleaned_file[index]
puts url_count += 1
end
}
end
def requests_per_hour
end
def sorted_list
end
end
my_file = CommonLog.new("test_log")
cleaned_file = my_file.readfile
puts cleaned_file.ip_histogram
It looks like the problem lies on you CommonLog#readfile method:
def readfile
#readfile = File.readlines(#logfile).map { |line|
line.split()
}
#readfile = readfile.to_s.split(" ")
end
Notice that inside the implementation of readfile your calling readfile recursively? When it executes it reads the lines from the file, maps them and assign the result the #readfile; then it calls readfile and the method starts to execute again; this goes on forever, until you stack blows up because of too many recursive method calls.
I assume what you actually meant is:
#readfile = #readfile.to_s.split(" ")
Related
I am writing a short ruby script that takes a file as an argument and then parses that file.
I have put together a few conditions in the initialize method to ensure that a file path exists and it is readable and if it nots it prints an error message to the user.
However when I run the file with out a file attached along side the message "please add log file path". I also receive the following error messages.
please add log file path
Traceback (most recent call last):
3: from webserver_log_parser.rb:41:in `<main>'
2: from webserver_log_parser.rb:41:in `new'
1: from webserver_log_parser.rb:6:in `initialize'
webserver_log_parser.rb:6:in `exist?': no implicit conversion of nil into String (TypeError)
I would be really grateful if someone could explain why this happens and a way to fix this issue.
def initialize(log_file_path = nil)
puts 'please add log file path' unless log_file_path
puts 'Could not find the file path' unless File.exist?(log_file_path)
puts '${log_file_path} is unreadable' unless File.readable?(log_file_path)
extract_log_file(log_file_path)
end
def extract_log_file(log_file_path)
webpages = Hash.new { |url, ip_address| url[ip_address] = [] }
File.readlines(log_file_path).each do |line|
url, ip_address = line.split
webpages[url] << ip_address
end
sort_results(webpages)
end
def sort_results(results)
total_views = {}
unique_views = {}
results.each do |url, ip_address|
total_views[url] = ip_address.length
unique_views[url] = ip_address.uniq.length
end
display_results(total_views, unique_views)
end
def display_results(views, unique_views)
puts 'The most viewed pages are as follows:'
total_views_sorted = views.sort_by { |_k, count| -count }
total_views_sorted.each { |key, count| puts "#{key} #{count}" }
puts 'The pages with the most unique views are as follows:'
unique_sort = unique_views.sort_by { |_k, count| -count }
unique_sort.each { |key, count| puts "#{key} #{count}" }
end
end
if $PROGRAM_NAME == __FILE__
LogParser.new(ARGV[0])
end```
If you want to directly terminate the script with a message, you can use abort.
def initialize(log_file_path = nil)
abort("please add log file path") unless log_file_path
abort("Could not find the file path") unless File.exist?(log_file_path)
abort("${log_file_path} is unreadable") unless File.readable?(log_file_path)
extract_log_file(log_file_path)
end
When your guard conditions are triggered, you need to stop further processing (no need to check for readability of a file at file_path if you already established that file_path is nil). It could look like this, for example:
def initialize(log_file_path = nil)
unless log_file_path
puts 'please add log file path'
return
end
unless File.exist?(log_file_path)
puts 'Could not find the file path'
return
end
unless File.readable?(log_file_path)
puts '${log_file_path} is unreadable'
return
end
extract_log_file(log_file_path)
end
I created a script which checks healthcheck and ports status from a .json file populated with microservices.
So for every microservice from the .json file the script will output the HTTP status and healthcheck body and other small details, and I want to add multithreading here in order to return all the output at once.Please see the script below:
#!/usr/bin/env ruby
... get the environment argument part...
file = File.read('./services.json')
data_hash = JSON.parse(file)
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
begin
puts "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
puts "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
puts "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
puts res.code
puts status
puts res.message
end
rescue Net::ReadTimeout
puts "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
Anyway in this way the script return first the puts "Microservice: #{microservice}" and puts "Port: #{port}" and after this it will return the nodes and only after the STATUS.
How can I return all the data for each loop together?
Instead of puts write output to a variable (hash).
If you wand to wait for all threads to finish their job before showing the output, use ThreadsWait class.
require 'thwait'
file = File.read('./services.json')
data_hash = JSON.parse(file)
h = {}
threads = []
service = data_hash.keys
service.each do |microservice|
threads << Thread.new do
thread_id = Thread.current.object_id.to_s(36)
begin
h[thread_id] = "Microservice: #{microservice}"
port = data_hash["#{microservice}"]['port']
h[thread_id] << "Port: #{port}"
nodes = "knife search 'chef_environment:#{env} AND recipe:#{microservice}' -i"
node = %x[ #{nodes} ].split
node.each do |n|
h[thread_id]<< "Node: #{n}"
uri = URI("http://#{n}:#{port}/healthcheck?count=10")
res = Net::HTTP.get_response(uri)
status = Net::HTTP.get(uri)
h[thread_id] << res.code
h[thread_id] << status
h[thread_id] << res.message
end
rescue Net::ReadTimeout
h[thread_id] << "ReadTimeout Error"
next
end
end
end
threads.each do |thread|
thread.join
end
# wait untill all threads finish their job
ThreadsWait.all_waits(*threads)
p h
[edit]
ThreadsWait.all_waits(*threads) is redundant in above code and can be omitted, since line treads.each do |thread| thread.join end does exactely the same thing.
Instead of outputting the data as you get it using puts, you can collect it all in a string and then puts it once at the end. Strings can take the << operator (implemented as a method in Ruby), so you can just initialize the string, add to it, and then output it at the end, like this:
report = ''
report << 'first thing'
report << 'second thing'
puts report
You could even save them all up together and print them all after all were finished if you want.
I want to bulid a multi threaded application. If i do not use threads, everything works fine. When i try to use threads, then nothing is displayed on the browser. when i use the syntax 'puts "%s" %io.read' then it displays on the command prompt and not on the browser. Any help would be appreciated.
require 'sinatra'
require 'thread'
set :environment, :production
get '/price/:upc/:rtype' do
Webupc = "#{params[:upc]}"
Webformat = "#{params[:rtype]}"
MThread = Thread.new do
puts "inside thread"
puts "a = %s" %Webupc
puts "b = %s" %Webformat
#call the price
Maxupclen = 16
padstr = ""
padupc = ""
padlen = (Maxupclen - Webupc.length)
puts "format type: #{params[:rtype]}"
puts "UPC: #{params[:upc]}"
puts "padlen: %s" %padlen
if (Webformat == 'F')
puts "inside format"
if (padlen == 0 ) then
IO.popen("tstprcpd.exe #{Webupc}")
{ |io|
"%s" %io.read
}
elsif (padlen > 0 ) then
for i in 1 .. padlen
padstr = padstr + "0"
end
padupc = padstr + Webupc
puts "padupc %s" %padupc
IO.popen("tstprcpd.exe #{padupc}") { |io|
"%s" %io.read
}
elsif (padlen < 0 ) then
IO.popen("date /T") { |io|
"UPC length must be 16 digits or less." %io.read
}
end
end
end
end
Your code has several problems:
It is not formatted properly
You are using Uppercase names for variables; that makes them constants!
puts will not output to the browser, but to the console. The browser will recieve the return value of the block, i.e. the return value of the last statement in the block. Therefore, you need to build your output differently (see below).
You are never joining the thread
Here's a minimal sinatra app that uses a thread. However, the thread makes no sense in this case because you must wait for its termination anyway before you can output the result to the browser. In order to build the output I have used StringIO, which you can use with puts to build a multiline string conveniently. However, you could also simply initialize res with an empty string with res = "" and then append your lines to this string with res << "new line\n".
require 'sinatra'
require 'thread'
require 'stringio'
get '/' do
res = StringIO.new
th = Thread.new do
res.puts 'Hello, world!'
end
th.join
res.string
end
I have constructed an Event Manager class that performs parsing actions on a CSV file, and produces html letters using erb. It is part of a jumpstart labs tutorial
The program works fine, but I am unable to call multiple methods on an object without the earlier methods interfering with the later methods. As a result, I have opted to create multiple objects to call instance methods on, which seems like a clunky inelegant solution. Is there a better way to do this, where I can create a single new object and call methods on it?
Like so:
eventmg = EventManager.new("event_attendees.csv")
eventmg.print_valid_phone_numbers
eventmg_2 = EventManager.new("event_attendees.csv")
eventmg_2.print_zipcodes
eventmg_3 = EventManager.new("event_attendees.csv")
eventmg_3.time_targeter
eventmg_4 = EventManager.new("event_attendees.csv")
eventmg_4.day_of_week
eventmg_5 = EventManager.new("event_attendees.csv")
eventmg_5.create_thank_you_letters
The complete code is as follows
require 'csv'
require 'sunlight/congress'
require 'erb'
class EventManager
INVALID_PHONE_NUMBER = "0000000000"
Sunlight::Congress.api_key = "e179a6973728c4dd3fb1204283aaccb5"
def initialize(file_name, list_selections = [])
puts "EventManager Initialized."
#file = CSV.open(file_name, {:headers => true,
:header_converters => :symbol} )
#list_selections = list_selections
end
def clean_zipcode(zipcode)
zipcode.to_s.rjust(5,"0")[0..4]
end
def print_zipcodes
puts "Valid Participant Zipcodes"
#file.each do |line|
zipcode = clean_zipcode(line[:zipcode])
puts zipcode
end
end
def clean_phone(phone_number)
converted = phone_number.scan(/\d/).join('').split('')
if converted.count == 10
phone_number
elsif phone_number.to_s.length < 10
INVALID_PHONE_NUMBER
elsif phone_number.to_s.length == 11 && converted[0] == 1
phone_number.shift
phone_number.join('')
elsif phone_number.to_s.length == 11 && converted[0] != 1
INVALID_PHONE_NUMBER
else
phone_number.to_s.length > 11
INVALID_PHONE_NUMBER
end
end
def print_valid_phone_numbers
puts "Valid Participant Phone Numbers"
#file.each do |line|
clean_number = clean_phone(line[:homephone])
puts clean_number
end
end
def time_targeter
busy_times = Array.new(24) {0}
#file.each do |line|
registration = line[:regdate]
prepped_time = DateTime.strptime(registration, "%m/%d/%Y %H:%M")
prepped_time = prepped_time.hour.to_i
# inserts filtered hour into the array 'list_selections'
#list_selections << prepped_time
end
# tallies number of registrations for each hour
i = 0
while i < #list_selections.count
busy_times[#list_selections[i]] += 1
i+=1
end
# delivers a result showing the hour and the number of registrations
puts "Number of Registered Participants by Hour:"
busy_times.each_with_index {|counter, hours| puts "#{hours}\t#{counter}"}
end
def day_of_week
busy_day = Array.new(7) {0}
d_of_w = ["Monday:", "Tuesday:", "Wednesday:", "Thursday:", "Friday:", "Saturday:", "Sunday:"]
#file.each do |line|
registration = line[:regdate]
# you have to reformat date because of parser format
prepped_date = Date.strptime(registration, "%m/%d/%y")
prepped_date = prepped_date.wday
# adds filtered day of week into array 'list selections'
#list_selections << prepped_date
end
i = 0
while i < #list_selections.count
# i is minus one since days of week begin at '1' and arrays begin at '0'
busy_day[#list_selections[i-1]] += 1
i+=1
end
#busy_day.each_with_index {|counter, day| puts "#{day}\t#{counter}"}
prepared = d_of_w.zip(busy_day)
puts "Number of Registered Participants by Day of Week"
prepared.each{|date| puts date.join(" ")}
end
def legislators_by_zipcode(zipcode)
Sunlight::Congress::Legislator.by_zipcode(zipcode)
end
def save_thank_you_letters(id,form_letter)
Dir.mkdir("output") unless Dir.exists?("output")
filename = "output/thanks_#{id}.html"
File.open(filename,'w') do |file|
file.puts form_letter
end
end
def create_thank_you_letters
puts "Thank You Letters Available in Output Folder"
template_letter = File.read "form_letter.erb"
erb_template = ERB.new template_letter
#file.each do |line|
id = line[0]
name = line[:first_name]
zipcode = clean_zipcode(line[:zipcode])
legislators = legislators_by_zipcode(zipcode)
form_letter = erb_template.result(binding)
save_thank_you_letters(id,form_letter)
end
end
end
The reason you're experiencing this problem is because when you apply each to the result of CSV.open you're moving the file pointer each time. When you get to the end of the file with one of your methods, there is nothing for anyone else to read.
An alternative is to read the contents of the file into an instance variable at initialization with readlines. You'll get an array of arrays which you can operate on with each just as easily.
"Is there a better way to do this, where I can create a single new object and call methods on it?"
Probably. If your methods are interfering with one another, it means you're changing state within the manager, instead of working on local variables.
Sometimes, it's the right thing to do (e.g. Array#<<); sometimes not (e.g. Fixnum#+)... Seeing your method names, it probably isn't.
Nail the offenders down and adjust the code accordingly. (I only scanned your code, but those Array#<< calls on an instance variable, in particular, look fishy.)
my environment: ruby 1.9.3p392 (2013-02-22 revision 39386) [x86_64-linux]
The thing is, I can make ruby return the parameters sent over GET. but when i'm trying to use them as arguements to my methods in if/else, ruby wont return anything and I end up with a blank page.
ph and pm return correctly:
http://127.0.0.1/cgi-bin/test.rb?hostname=node00.abit.dk&macadd=23:14:41:51:63
returns:
node00.abit.dk 23:14:41:51:63
Connection to the database (MySQL) works fine
When I test the method newHostName it outputs correctly:
puts newHostName
returns (which is correct)
node25.abit.dk
the code:
#!/usr/bin/ruby
require 'cgi'
require 'sequel'
require 'socket'
require 'timeout'
DB = Sequel.connect(:adapter=>'mysql', :host=>'localhost', :database=>'nodes', :user=>'nodeuser', :password=>'...')
#cgi-part to work
#takes 2 parameters:
#hostname & macadd
cgi = CGI.new
puts cgi.header
p = cgi.params
ph = p['hostname']
pm = p['macadd']
def nodeLookup(hostnameargv)
hostname = DB[:basenode]
h = hostname[:hostname => hostnameargv]
h1 = h[:hostname]
h2 = h[:macadd]
ary = [h1, h2]
return ary
end
def lastHostName()
#TODO: replace with correct sequel-code and NOT raw SQL
DB.fetch("SELECT hostname FROM basenode ORDER BY id DESC LIMIT 1") do |row|
return row[:hostname]
end
end
def newHostName()
org = lastHostName
#Need this 'hack' to make ruby grep for the number
#nodename e.g 'node01.abit.dk'
var1 = org[4]
var2 = org[5]
var3 = var1 + var2
sum = var3.to_i + 1
#puts sum
sum = "node" + sum.to_s + ".abit.dk"
return sum
end
def insertNewNode(newhost, newmac)
newnode = DB[:basenode]
newnode.insert(:hostname => newhost, :macadd => newmac)
return "#{newnode.count}"
end
#puts ph
#puts pm
#puts newHostName
cgi.out() do
cgi.html do
begin
if ph == "node00.abit.dk"
puts newHostName
else
puts nodeLookup(ph)
end
end
end
end
I feel like im missing something here. Any help is very much appreciated!
//M00kaw
What about modify last lines of your code as followed? CGI HTML generation methods take a block and yield the return value of the block as their content. So you should make newHostName or nodeLookup(ph) as the return value of the block passed to cgi.html(), rather than puts sth, which prints the content to your terminal and return nil. That's why cgi.html() got an empty string (nil.to_s).
#puts newHostName
cgi.out() do
cgi.html do
if ph == "node00.abit.dk"
newHostName
else
nodeLookup(ph)
end
end
end
p.s. It's conventional to indent your ruby code with 2 spaces :-)