Im having difficulty using .match to only allow and block selective tweets and display only those from 'does_match?'
def does_match?
allow = "/orange|grape\sfruit|apple/"
block = "/#fruits|coconut/"
allowfruits = "/berry|mango/"
#tweet.match(allow).nil?
#tweet.match(block)
#tweet.match(allowfruits) if #user =~ /\A(twitteruser|anotheraccount)\Z/
#tweet.match(/#[A-Za-z0-9_:-]+/)
return #tweet
end
def show
return #tweet
end
firstly, You are defining Your regexps as a strings
do this instead
allow = /orange|grape\sfruit|apple/
secondly, You're doing some matches bot doing nothing with its return values
do this
if #tweet.match(allow)
# rest of logic
# checking blocked and allowed for user
#tweet # or true
else
nil # or false
end
Related
I have written an assertion that collects new records created while it yields to a block. Here's an example, with a failing assertion inside that block:
product =
assert_latest_record Product do # line 337
post :create,
:product => { ... }
assert false # line 340
end
The source of my assertion is below, but I don't think it's relevant. It does not intercept Minitest exceptions, or even call rescue or ensure.
The problem is when an assertion inside that block fails. The fault diagnostic message reports the line number as 337 the line of the outer assertion, not 340, the line of the inner assertion that failed. This is important if, for example, my colleagues have written a run-on test with way too many lines in it; isolating a failing line becomes more difficult.
Why doesn't Minitest report the correct line number?
The source:
##
# When a test case calls methods that write new ActiveModel records to a database,
# sometimes the test needs to assert those records were created, by fetching them back
# for inspection. +assert_latest_record+ collects every record in the given model or
# models that appear while its block runs, and returns either a single record or a ragged
# array.
#
# ==== Parameters
#
# * +models+ - At least 1 ActiveRecord model or association.
# * +message+ - Optional string or ->{block} to provide more diagnostics at failure time.
# * <code>&block</code> - Required block to call and monitor for new records.
#
# ==== Example
#
# user, email_addresses =
# assert_latest_record User, EmailAddress, ->{ 'Need moar records!' } do
# post :create, ...
# end
# assert_equal 'franklyn', user.login # 1 user, so not an array
# assert_equal 2, email_addresses.size
# assert_equal 'franklyn#gmail.com', email_addresses.first.mail
# assert_equal 'franklyn#hotmail.com', email_addresses.second.mail
#
# ==== Returns
#
# The returned value is a set of one or more created records. The set is normalized,
# so all arrays of one item are replaced with the item itself.
#
# ==== Operations
#
# The last argument to +assert_latest_record+ can be a string or a callable block.
# At failure time the assertion adds this string or this block's return value to
# the diagnostic message.
#
# You may call +assert_latest_record+ with anything that responds to <code>.pluck(:id)</code>
# and <code>.where()</code>, including ActiveRecord associations:
#
# user = User.last
# email_address =
# assert_latest_record user.email_addresses do
# post :add_email_address, user_id: user.id, ...
# end
# assert_equal 'franklyn#philly.com', email_address.mail
# assert_equal email_address.user_id, user.id, 'This assertion is redundant.'
#
def assert_latest_record(*models, &block)
models, message = _get_latest_record_args(models, 'assert')
latests = _get_latest_record(models, block)
latests.include?(nil) and _flunk_latest_record(models, latests, message, true)
pass # Increment the test runner's assertion count
return latests.size > 1 ? latests : latests.first
end
##
# When a test case calls methods that might write new ActiveModel records to a
# database, sometimes the test must check that no records were written.
# +refute_latest_record+ watches for new records in the given class or classes
# that appear while its block runs, and fails if any appear.
#
# ==== Parameters
#
# See +assert_latest_record+.
#
# ==== Operations
#
# refute_latest_record User, EmailAddress, ->{ 'GET should not create records' } do
# get :index
# end
#
# The last argument to +refute_latest_record+ can be a string or a callable block.
# At failure time the assertion adds this string or this block's return value to
# the diagnostic message.
#
# Like +assert_latest_record+, you may call +refute_latest_record+ with anything
# that responds to <code>pluck(:id)</code> and <code>where()</code>, including
# ActiveRecord associations.
#
def refute_latest_record(*models, &block)
models, message = _get_latest_record_args(models, 'refute')
latests = _get_latest_record(models, block)
latests.all?(&:nil?) or _flunk_latest_record(models, latests, message, false)
pass
return
end
##
# Sometimes a test must detect new records without using an assertion that passes
# judgment on whether they should have been written. Call +get_latest_record+ to
# return a ragged array of records created during its block, or +nil+:
#
# user, email_addresses, posts =
# get_latest_record User, EmailAddress, Post do
# post :create, ...
# end
#
# assert_nil posts, "Don't create Post records while creating a User"
#
# Unlike +assert_latest_record+, +get_latest_record+ does not take a +message+ string
# or block, because it has no diagnostic message.
#
# Like +assert_latest_record+, you may call +get_latest_record+ with anything
# that responds to <code>.pluck(:id)</code> and <code>.where()</code>, including
# ActiveRecord associations.
#
def get_latest_record(*models, &block)
assert models.any?, 'Call get_latest_record with one or more ActiveRecord models or associations.'
refute_nil block, 'Call get_latest_record with a block.'
records = _get_latest_record(models, block)
return records.size > 1 ? records : records.first
end # Methods should be easy to use correctly and hard to use incorrectly...
def _get_latest_record_args(models, what) #:nodoc:
message = nil
message = models.pop unless models.last.respond_to?(:pluck)
valid_message = message.nil? || message.kind_of?(String) || message.respond_to?(:call)
models.length > 0 && valid_message and return models, message
raise "call #{what}_latest_record(models..., message) with any number\n" +
'of Model classes or associations, followed by an optional diagnostic message'
end
private :_get_latest_record_args
def _get_latest_record(models, block) #:nodoc:
id_sets = models.map{ |model| model.pluck(*model.primary_key) } # Sorry about your memory!
block.call
record_sets = []
models.each_with_index do |model, index|
pk = model.primary_key
set = id_sets[index]
records =
if set.length == 0
model
elsif pk.is_a?(Array)
pks = pk.map{ |k| "`#{k}` = ?" }.join(' AND ')
pks = [ "(#{pks})" ] * set.length
pks = pks.join(' OR ')
model.where.not(pks, *set.flatten)
else
model.where.not(pk => set)
end
records = records.order(pk).to_a
record_sets.push records.size > 1 ? records : records.first
end
return record_sets
end
private :_get_latest_record
def _flunk_latest_record(models, latests, message, polarity) #:nodoc:
itch_list = []
models.each_with_index do |model, index|
records_found = latests[index] != nil
records_found == polarity or itch_list << model.name
end
itch_list = itch_list.join(', ')
diagnostic = "should#{' not' unless polarity} create new #{itch_list} record(s) in block"
message = nil if message == ''
message = message.call.to_s if message.respond_to?(:call)
message = [ message, diagnostic ].compact.join("\n")
raise Minitest::Assertion, message
end
private :_flunk_latest_record
You could try to configure it to log exceptions in test_helper.rb:
def MiniTest.filter_backtrace(backtrace)
backtrace
end
I'm not sure if this is the default, but depending on your configuration, the backtrace might not be shown.
Im trying to build a sub-domain brute forcer for use with my clients - I work in security/pen testing.
Currently, I am able to get Resolv to look up around 70 hosts in 10 seconds, give or take and wanted to know if there was a way to get it to do more. I have seen alternative scripts out there, mainly Python based that can achieve far greater speeds than this. I don't know how to increase the number of requests Resolv makes in parallel, or if i should split the list up. Please note I have put Google's DNS servers in the sample code, but will be using internal ones for live usage.
My rough code for debugging this issue is:
require 'resolv'
def subdomains
puts "Subdomain enumeration beginning at #{Time.now.strftime("%H:%M:%S")}"
subs = []
domains = File.open("domains.txt", "r") #list of domain names line by line.
Resolv.new(:nameserver => ['8.8.8.8', '8.8.4.4'])
File.open("tiny.txt", "r").each_line do |subdomain|
subdomain.chomp!
domains.each do |d|
puts "Checking #{subdomain}.#{d}"
ip = Resolv.new.getaddress "#{subdomain}.#{d}" rescue ""
if ip != nil
subs << subdomain+"."+d << ip
end
end
end
test = subs.each_slice(4).to_a
test.each do |z|
if !z[1].nil? and !z[3].nil?
puts z[0] + "\t" + z[1] + "\t\t" + z[2] + "\t" + z[3]
end
end
puts "Finished at #{Time.now.strftime("%H:%M:%S")}"
end
subdomains
domains.txt is my list of client domain names, for example google.com, bbc.co.uk, apple.com and 'tiny.txt' is a list of potential subdomain names, for example ftp, www, dev, files, upload. Resolv will then lookup files.bbc.co.uk for example and let me know if it exists.
One thing is you are creating a new Resolv instance with the Google nameservers, but never using it; you create a brand new Resolv instance to do the getaddress call, so that instance is probably using some default nameservers and not the Google ones. You could change the code to something like this:
resolv = Resolv.new(:nameserver => ['8.8.8.8', '8.8.4.4'])
# ...
ip = resolv.getaddress "#{subdomain}.#{d}" rescue ""
In addition, I suggest using the File.readlines method to simplify your code:
domains = File.readlines("domains.txt").map(&:chomp)
subdomains = File.readlines("tiny.txt").map(&:chomp)
Also, you're rescuing the bad ip and setting it to the empty string, but then in the next line you test for not nil, so all results should pass, and I don't think that's what you want.
I've refactored your code, but not tested it. Here is what I came up with, and may be clearer:
def subdomains
puts "Subdomain enumeration beginning at #{Time.now.strftime("%H:%M:%S")}"
domains = File.readlines("domains.txt").map(&:chomp)
subdomains = File.readlines("tiny.txt").map(&:chomp)
resolv = Resolv.new(:nameserver => ['8.8.8.8', '8.8.4.4'])
valid_subdomains = subdomains.each_with_object([]) do |subdomain, valid_subdomains|
domains.each do |domain|
combined_name = "#{subdomain}.#{domain}"
puts "Checking #{combined_name}"
ip = resolv.getaddress(combined_name) rescue nil
valid_subdomains << "#{combined_name}#{ip}" if ip
end
end
valid_subdomains.each_slice(4).each do |z|
if z[1] && z[3]
puts "#{z[0]}\t#{z[1]}\t\t#{z[2]}\t#{z[3]}"
end
end
puts "Finished at #{Time.now.strftime("%H:%M:%S")}"
end
Also, you might want to check out the dnsruby gem (https://github.com/alexdalitz/dnsruby). It might do what you want to do better than Resolv.
[Note: I've rewritten the code so that it fetches the IP addresses in chunks. Please see https://gist.github.com/keithrbennett/3cf0be2a1100a46314f662aea9b368ed. You can modify the RESOLVE_CHUNK_SIZE constant to balance performance with resource load.]
I've rewritten this code using the dnsruby gem (written mainly by Alex Dalitz in the UK, and contributed to by myself and others). This version uses asynchronous message processing so that all requests are being processed pretty much simultaneously. I've posted a gist at https://gist.github.com/keithrbennett/3cf0be2a1100a46314f662aea9b368ed but will also post the code here.
Note that since you are new to Ruby, there are lots of things in the code that might be instructive to you, such as method organization, use of Enumerable methods (e.g. the amazing 'partition' method), the Struct class, rescuing a specific Exception class, %w, and Benchmark.
NOTE: LOOKS LIKE STACK OVERFLOW ENFORCES A MAXIMUM MESSAGE SIZE, SO THIS CODE IS TRUNCATED. GO TO THE GIST IN THE LINK ABOVE FOR THE COMPLETE CODE.
#!/usr/bin/env ruby
# Takes a list of subdomain prefixes (e.g. %w(ftp xyz)) and a list of domains (e.g. %w(nytimes.com afp.com)),
# creates the subdomains combining them, fetches their IP addresses (or nil if not found).
require 'dnsruby'
require 'awesome_print'
RESOLVER = Dnsruby::Resolver.new(:nameserver => %w(8.8.8.8 8.8.4.4))
# Experiment with this to get fast throughput but not overload the dnsruby async mechanism:
RESOLVE_CHUNK_SIZE = 50
IpEntry = Struct.new(:name, :ip) do
def to_s
"#{name}: #{ip ? ip : '(nil)'}"
end
end
def assemble_subdomains(subdomain_prefixes, domains)
domains.each_with_object([]) do |domain, subdomains|
subdomain_prefixes.each do |prefix|
subdomains << "#{prefix}.#{domain}"
end
end
end
def create_query_message(name)
Dnsruby::Message.new(name, 'A')
end
def parse_response_for_address(response)
begin
a_answer = response.answer.detect { |a| a.type == 'A' }
a_answer ? a_answer.rdata.to_s : nil
rescue Dnsruby::NXDomain
return nil
end
end
def get_ip_entries(names)
queue = Queue.new
names.each do |name|
query_message = create_query_message(name)
RESOLVER.send_async(query_message, queue, name)
end
# Note: although map is used here, the record in the output array will not necessarily correspond
# to the record in the input array, since the order of the messages returned is not guaranteed.
# This is indicated by the lack of block variable specified (normally w/map you would use the element).
# That should not matter to us though.
names.map do
_id, result, error = queue.pop
name = _id
case error
when Dnsruby::NXDomain
IpEntry.new(name, nil)
when NilClass
ip = parse_response_for_address(result)
IpEntry.new(name, ip)
else
raise error
end
end
end
def main
# domains = File.readlines("domains.txt").map(&:chomp)
domains = %w(nytimes.com afp.com cnn.com bbc.com)
# subdomain_prefixes = File.readlines("subdomain_prefixes.txt").map(&:chomp)
subdomain_prefixes = %w(www xyz)
subdomains = assemble_subdomains(subdomain_prefixes, domains)
start_time = Time.now
ip_entries = subdomains.each_slice(RESOLVE_CHUNK_SIZE).each_with_object([]) do |ip_entries_chunk, results|
results.concat get_ip_entries(ip_entries_chunk)
end
duration = Time.now - start_time
found, not_found = ip_entries.partition { |entry| entry.ip }
puts "\nFound:\n\n"; puts found.map(&:to_s); puts "\n\n"
puts "Not Found:\n\n"; puts not_found.map(&:to_s); puts "\n\n"
stats = {
duration: duration,
domain_count: ip_entries.size,
found_count: found.size,
not_found_count: not_found.size,
}
ap stats
end
main
I am using RubyDNS.
When I use match block and otherwise, I want to skip some addresses in match so they would be caught by otherwise block.
But it doesn't go to otherwise block.
RubyDNS.run_server(listen: INTERFACES, asynchronous: false) do
upstream = RubyDNS::Resolver.new([[:udp, "8.8.8.8", 53], [:tcp, "8.8.8.8", 53]])
match(/^([\d\.]+)\.in-addr\.arpa$/, IN::PTR) do |transaction, match_data|
domain = nil # just for test
if domain
transaction.respond!(Name.create(domain))
else
# Pass the request to the otherwise handler
# !!! this doesn't work
false
end
end
otherwise do |transaction|
transaction.passthrough!(upstream)
end
end
When I return false from match block - it doesn't go to otherwise block.
How to fix this?
I found how to continue to otherwise block from match block: use 'next!'
match(/^([\d\.]+)\.in-addr\.arpa$/, IN::PTR) do |transaction, match_data|
domain = nil # just for test
if domain
transaction.respond!(Name.create(domain))
else
# Pass the request to the otherwise handler
next!
end
end
otherwise do |transaction|
transaction.passthrough!(upstream)
end
The task is to check if a contact page exists and navigate to it. For the websites not in english, the method looks for an english page and then restarts to check for a contact page.
My conditional works fine, but I figured there must be a better way to do this:
# First, I set the #url variable during Booleans.
# Checks are either to see if a link exists or if a page exists,
# (aka no 404 error).
#
# Here are two examples:
# Boolean, returns true if contact link is present.
def contact_link?
#url = link_with_href('contact')
!#url.nil?
end
# True if contact page '../contact' does NOT get a 404 error.
def contact_page?
#url = page.uri.merge('../contact').to_s
begin
true if Mechanize.new.get(#url)
rescue Mechanize::ResponseCodeError
false
end
end
# #
# Now go to the correct page, based off of checks.
#
def go_to_contact_page
1.times do
case # No redo necessary.
when contact_link? # True if hyperlink exists
get(#url)
when contact_page? # False if 404 error
get(#url)
else # Redo is now necessary.
if english_link? # True if hyperlink exists
get(#url)
redo
elsif en_page? # False if 404 error
get(#url)
redo
elsif english_page? # False if 404 error
redo
end
end
end
end
There are a couple things to draw your attention to:
Is 1.times do the best way to do a single redo? Would begin be better?
Understanding that I set the #url variable in each of these checks, there seems to be redundancy in get(#url) in the conditional branch. Is there a more succinct way?
I am writing redo three times which also seems redundant. Is there a way to call it once and still set the #url variable?
Thanks for the help!
Something like this is more readable and dry
def english_contact_page
..
rescue
nil
end
def contact_page
..
rescue
nil
end
def get_page
#url = link_with_href('contact')
return nil if #url.nil?
contact_page || english_contact_page # left side is evaluated first
rescue
nil
end
I am trying to do a post and run some if statement. What I want to do is:
check all fields are filled
if all fields are filled move on to next step, or else reload page
check if already in data base
add if not already in data base
post "/movies/new" do
title = params[:title]
year = params[:year]
gross = params[:gross]
poster = params[:poster]
trailer = params[:trailer]
if title && year && gross && poster && trailer
movie = Movie.find_by(title: title, year: year, gross: gross)
if movie
redirect "/movies/#{movie.id}"
else
movie = Movie.new(title: title, year: year, gross: gross, poster: poster, trailer: trailer)
if movie.save
redirect "/movies/#{movie.id}"
else
erb :'movies/new'
end
end
else
erb :'movies/new'
end
end
I don't think my if statement is correct. It works even if all my fields are not filled
Your code is doing a lot of work in one single method. I would suggest to restructure it into smaller chunks to make it easier to manage. I mostly code for Rails, so apologies if parts of these do not apply to your framework.
post "/movies/new" do
movie = find_movie || create_movie
if movie
redirect "/movies/#{movie.id}"
else
erb :'movies/new'
end
end
def find_movie
# guard condition to ensure that the required parameters are there
required_params = [:title, :year, :gross]
return nil unless params_present?(required_params)
Movie.find_by(params_from_keys(required_params))
end
def create_movie
required_params = [:title, :year, :gross, :poster, :trailer]
return nil unless params_present?(required_params)
movie = Movie.new(params_from_keys(required_params))
movie.save ? movie : nil # only return the movie if it is successfully saved
end
# utility method to check whether all provided params are present
def params_present?(keys)
keys.each {|key| return false if params[key].blank? }
true
end
# utility method to convert params into the hash format required to create / find a record
def params_from_keys(keys)
paras = {}
keys.each { |key| paras.merge!(key: params[key]) }
paras
end
Even if you type nothing in the HTML fields, they will still be submitted as empty strings.
You can avoid having empty parameters by, for example, filtering them:
post '/movies/new' do
params.reject! { |key, value| value.empty? }
# rest of your code
end
Also I would rather post to /movies rather than to /movies/new, that's more REST-wise.
Try if condition to check fields are blank like below -
unless [title, year, gross, poster, trailer].any?(&:blank?)
This will check any of the field should not be nil or blank("").