I am trying to evaluate user-submitted urls to find out whether they contain valid hostnames (formatting) and if so, extract the hostname. Know of any libraries/methods that could help?
Example:
user_input = "www.google.com"
if user_input.has_valid_host?
hostname = user_input.get_hostname #=> "google.com"
url = "http://" + #hostname #=> "http://google.com"
else
puts "Invalid URL"
end
This example is very simple but I need the url checked against all valid domain extensions and the hostname extracted from any string (assuming that it's present)
I don't know ruby, but I wouldn't think of this as a ruby question.
I would use regex to split out the hostname as you suggest.
Then I would do a system call to the nslookup routine.
On a Windows system from the command prompt it is nslookup.
C:\Users\xyz>nslookup www.google.com
Server: UnKnown
Address: 192.168.237.2
Name: www.l.google.com
Address: 173.194.73.99
Aliases: www.google.com.localdomain
From Ruby you should do an API call instead of using the command line, but both will eventually interface to the DNS service on the local machine.
See: Is there a good DNS server library in ruby?
Related
I have a hierarchy of directories and some files in some of those directories:
/root/development/dir1/file1.txt, file2.txt, ...
/root/development/dir2/file3.txt, file4, ...
/root/development/file6.in, file7.out, ...
...
Some of these files contain IP addresses inside the text. In the form x.x.x.x where each x is a number from 0 to 255 (inclusive).
For example, say we have file1.txt that looks like this:
hello world 127.0.0.1
this is some example 128.99.107.55
file with some correct and incorrect 128.128.4.11 ip 0.11.1115.78 addresses
This file contains only 3 IP addresses, namely 127.0.0.1, 128.99.107.55, and 128.128.4.11, since 0.11.1115.78 is not a valid IP address.
I need to write a program (in java or python) to find all distinct IP addresses from all the files in the /root/development/ directory and print them in lexicographical order.
The input will be a setup shell script, and the code should print the required data to stdout.
An example shell script is as follows:
#!/bin/bash
rm -rf /root/development
mkdir /root/development
mkdir /root/development/dir1
mkdir /root/development/dir2
touch /root/development/dir1/file1.txt
echo -e "hello world 127.0.0.1\nthis is some example 128.99.107.55 \nfile with some correct and incorrect 128.128.4.11 ip 0.11.1115.78 addressesaddresses" >> /root/development/dir1/file1.txt
touch /root/development/dir1/file2.txt
echo -e "hello from 74.0.65.76 and 8.dd.99.88.907 good\nthis is some example 16.1215.76.35 \nfile with some correct and incorrect 15.128.4.65 ip addresses\n0.0.0.0" >> /root/development/dir1/file2.txt
touch /root/development/dir2/file3.txt
echo -e "127.65.64.1 127.0.64.1 127.0.0.1\nexample 128.57.107.76 128.57.907.70 \nfile with some correct and incorrect 67.128.4.11 ip addresses 7.7.7.8" >> /root/development/dir2/file3.txt
touch /root/development/dir2/file4.txt
echo -e "hello world 127.98.0.1\nthis is some example 128.96.107.55 \nfile with some correct and incorrect 128.68.4.11 ip addresses" >> /root/development/dir2/file4.txt
touch /root/development/f.inp
echo -e "hello world 127.0.49.1 \nthis is some example 128.99.58.55 8.88.888.88 77.255.255.254\n7.7.257.25 file with some correct and incorrect 26.56.4.23 ip addresses" >> /root/development/f.inp
Example
For the following /root/development/ directory:
/root/development/dir1/file1.txt
hello world 127.0.0.1
this is some example 128.99.107.55
file with some correct and incorrect 128.128.4.11 ip 0.11.1115.78 addressesaddresses
/root/development/dir1/file2.txt
hello from 74.0.65.76 and 8.dd.99.88.907 good
this is some example 306.5.76.35
file with some correct and incorrect 15.128.4.65 ip addresses
0.0.0.0
/root/development/dir2/file3.txt
127.65.64.1 127.0.64.1 127.0.0.1
exaMple 128.57.107.76 128.57.907.70
file with some correct and incorrect 67.128.4.11 ip addresses 7.7.7.8
/root/devops/dir2/file4.txt
hello world 127.98.0.1
this is some example 128.96.107.55
file with some correct and incorrect 128.68.4.11 ip addresses
/root/development/f.inp
hello world 127.0.49.1
this is some example 128.99.58.55 8.88.888.88 77.255.255.254
7.7.257.25 file with some correct and incorrect 26.56.4.23 ip addresses
The output should be
0.0.0.0
127.0.0.1
127.0.49.1
127.0.64.1
127.65.64.1
127.98.0.1
128.128.4.11
128.57.107.76
128.68.4.11
128.96.107.55
128.99.107.55
128.99.58.55
15.128.4.65
26.56.4.23
67.128.4.11
7.7.7.8
74.0.65.76
77.255.255.254
From your comment, I assume that you have no or very little pre-existing programming knowledge, so I'll try to broadly explain steps of how you could tackle this project.
For each of the following steps, if you don't know how to do it (which is expected if you're new to programming), try an internet search to know how to do that step in your chosen language.
Step 0. First, you should choose between Java and Python. Both can be used for this, it's just a matter of what language you know the best at this time, or have already installed on your computer...
Step 1: write the code to read the content of just one file. You can temporarily write the whole file content to stdout to ensure that this works well.
Step 2. change the code to print only the IP adresses from the file content (maybe using regex to extract the IP adresses from the file content)
Step 3. remove duplicate IP adresses. You'll probably do that by putting IP adresses in a list and apply an existing Java/Python function to that list, and then print the list to stdout
Step 4. sort the list before printing it
Now you want to do the same thing, but on several files:
Step 5. Write code to list all the files of one folder
Step 6. Write code to list all the files of one folder and all it's subfolders, recursively
Step 7. Combine your code from Steps 1-4 with code from Steps 5-6 to achieve the result you want
To approach above problem we will have to be good with regex and file handling. The below code is in python should help you to move forward
Code
from pathlib import Path
import re
# Get lines from the files
myDir = r"C:\Users\myuser\Downloads\find\yourFolder"
result = list(Path(myDir).rglob("*.[tT][xX][tT]"))
ips = []
flatList = []
def getListOfIps():
for i in result:
with open(i) as f:
for text in f.readlines():
text = text.rstrip()
regex = re.findall(
r'(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})', text)
if regex is not None and regex not in ips:
ips.append(regex)
# Get the list of IPs
def check(Ip):
if(re.search(myPattern, Ip)): # re.search returns None (if the pattern doesn’t match)
return True
else:
return False
myPattern = "^((25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])\.){3}(25[0-5]|2[0-4][0-9]|1[0-9][0-9]|[1-9]?[0-9])"
getListOfIps()
flatList = [ item for elem in ips for item in elem]
flatList.sort()
for eachIP in flatList:
if check(eachIP):
print(f"{eachIP}")
# May require for future improvements
# matches for 8.dd.99.88 is (?:[\d]{1,3})\.(?:[\d]|[a-z]{1,3})\.(?:[\d]{1,3})\.(?:[\d]{1,3})
if you are new to programming and need some reference program you may look at my repo https://github.com/Kapil987/Python_prac/tree/master
I'm trying to download a remote file over sftp, and I wrote this:
remote_file 'C:/adnet/mercury_package/Adnet.EASS.zip' do
source 'sftp://myusername:mypassword#packages.domain.com:22/package/Adnet.EASS.zip'
action :create
end
and the output is:
SocketError
-----------
getaddrinfo: No such host is known.
I tried to remove username and password from the URI, and put them in remote_user and remote_password, but then it complains about not having the userinfo:
"no userinfo provided in the sftp URI"
The stacktrace is here
The problem was in single quotes around the source. It works like this:
remote_file 'C:/adnet/mercury_package/Adnet.EASS.zip' do
source "sftp://myusername:mypassword#packages.domain.com:22/package/Adnet.EASS.zip"
action :create
end
I'm using rbox rye gem to run some tests to verify I can ping/connect to a host and then verify that certain directories/files exist on a host and then that a specific file is being uploaded to a host.
Most of the functionality is complete but one thing is missing and that is to verify I can actually connect to the host.
I use the following method to make the host connection.
The return value of the method is a connection string object (username#rye_10.16.11.100)
def connection()
con = Rye::Box.new("#{#host}")
con
end
Later in the code I want to validate that the connection succeeded:
if ((file.connection()) == #{HOST})
puts "Connected to Host"
else
puts "Not able to connect to Host"
end
However, the test fails with the following error:
gems/rye-0.9.12/lib/rye/box.rb:408:in `==': undefined method `host' for 0:Fixnum (NoMethodError)
from test_rye.rb:99:in `<main>'
I assume that the comparrisson operator is not valid to test for the returned rbox host object.
Is there a way that I can perform that comparrisson?
Thanks
Looks like your #host variable is holding a num instead of a hostname. Looking at the source code of Rye::Box, you can see that the comparison method is comparing the hostname that you initialised it with: https://github.com/delano/rye/blob/2d9ffd6d18688cc85ec1a004e35d12d8909b5ed6/lib/rye/box.rb#L408
I would take the following steps:
Add a debugger before con = Rye::Box.new("#{#host}") to check that the #host variable is actually a string.
If it is, add a debugger before if ((file.connection()) == #{HOST}) to see what file.connection returns.
As a side note, you should change #{HOST} to #host in that last statement; that's the correct way to access the instance variable.
Hopefully that'll shed some clarity to the problem!
I am using Ruby 1.9.3 and am running the following command:
open("ftp://user name:password#datafeeds.domain.com/1234/1234.txt.gz")
which returns:
URI::InvalidURIError: bad URI(is not URI?)
Encoding the user name (replacing spaces with %20) does not work either:
Net::FTPPermError: 530 Invalid userid/password
The URI works fine in all browsers and FTP clients tested - just not when using OpenURI. Also, using Net::FTP (which is wrapped by OpenURI) works fine as well:
require 'net/ftp'
ftp = Net::FTP.new
ftp.connect("datafeeds.domain.com", 21)
ftp.login("user name", "password")
ftp.getbinaryfile("/1234/1234.txt.gz")
Any idea why the OpenURI method does not work, while the Net::FTP method does? Thanks.
By definition in the specification, URL user names only allow these characters:
user alphanum2 [ user ]
[...]
alphanum2 alpha | digit | - | _ | . | +
Browsers are notorious for ignoring the specifications so saying they support it isn't a good proof. They shouldn't per the spec.
If cURL supports them, then use the Curb gem and see if that'll let you use them.
According to this StackOverflow answer, you should be able to just escape the special characters in your username and password. You could do something like:
login = URI.escape('user name') + ':' + URI.escape('password')
open("ftp://#{login}#datafeeds.domain.com/1234/1234.txt.gz")
open-uri seems broken in that matter. I had the similar issue with the password which contained # character.
I ended up bypassing set_password, which doesn't allow # char in password, by setting URI's #password instance variable directly.
uri = URI.parse(...)
if uri.password
uri.instance_variable_set "#password", "password_with_#_char"
end
open(uri) # now this works fine
It's hacky, but does the job.
So I got a script (A) that finds a suitable IP address for a new virtual server. At first, it takes a look in the database to see if the first ip he chose isn't already taken by another server. If the IP is not already in use, the script pings it. If there is no response from the ping, then we get to the next step and this is where I'm having a problem.
In the next step, I have to check if the IP address is already registred in the netscaler (router) or not. To do this, I must use another script on the same machine (B). This other script return the list of all the ips defined in the netscaler. When I run it, the output looks like this
x.x.x.x
x.x.x.x
x.x.x.x (and so on..).
I found many ways to execute the script B from whiting the script A, but none of what I found allow me to do what I'd like to.
My goal is to compare the ip my script found with all of those that are listed, without having those last ones printed on the screen.
So, to make it a bit clearer, let's say that the scrip A found the IP : 1.2.3.4
It would then call script B that would return to script A this list
1.2.3.5
1.2.4.5
1.2.5.1
and so on.
and then A would compare 1.2.3.4 with all those returned by script B without actually showing them on screen.
Thank you very much!
I would separate scriptB business logic from scriptB ui (CLI) logic:
scriptA.rb
scriptB.rb
netscaler.rb # extract logic here
Extract your list of all the ips defined in the netscaler logic into separate class/method:
#netscaler.rb
class Netscaler
def self.list_ips
# return array of ips here
end
end
#scriptB.rb
require_relative 'netscaler'
ips = Netscaler.list_ips
puts ips # script B may show these ips on the screen
...
#scriptA.rb
require_relative 'netscaler'
ips = Netscaler.list_ips
# script A will not show them. Instead it will operate on the returned result.
...
You can use backticks to execute script B and return the output:
ip_list = `scriptB`.split("\n")
This can be plugged into Alex's organizational suggestion. I would do this if script B is a non-Ruby script that you don't control.
Note that if there is any leading or trailing whitespace you can add a .map(&:strip) to the end.