I have an object, which pprints into:
#<OpenSSL::ASN1::ASN1Data:0x0000000803ab37a8
#infinite_length=false,
#tag=7,
#tag_class=:CONTEXT_SPECIFIC,
#value="\x7F\x00\x00\x01">]>
It is easy enough to see, that the #value is an IP-address: 127.0.0.1. How do I reliably extract it, however -- so that it works regardless of endianness and other hard-to-test considerations?
I'm doing it the following way for now:
ip = v.value.split('').map {|octet| octet.ord }.join('.')
and it works on my machine, but I wonder, how reliable the method is...
Ok, as I report in a related question, I found an answer in Ruby's own code (openssl/ssl.rb). They deal with both IPv4 and IPv6 addresses thus -- using unpack instead of split-ing the string into characters:
case v.value.size
when 4
ip = v.value.unpack('C*').join('.')
when 16
ip = v.value.unpack('n*').map { |o| sprintf("%X", o) }.join(':')
else
STDERR.print "The encountered IP-address is neither IPv4 nor IPv6\n"
next
end
Related
So I have a big csv file, over 1gb. There's a column with IP addresses in ipv4 and ipv6. I want to convert the ipv6 addresses into numbers, but there are too many rows for libre calc. So I'm wondering if it's possible to use python in the terminal to convert all the ipv6 addresses.
Also, I could split the file up into smaller pieces, then use libre calc, but same problem--I wouldn't know how to script that either.
EDIT:
I don't mind, it might get more complicated though. Also not sure how this should be formatted, but I hope people get the idea...So I have one table with IPv6 addresses like these examples:
2001:db8::cafe:1111
2001:db8:0:a:1:2:3:4
2001:db8:aaaa::c
2001:db8:0:0:1::4
There are a bunch of different rules that govern the formatting--way too hard for me. I've heard that python has a function that will specifically return the conversion, but not sure about the rest (how to get the returned values back into the csv correctly, with formatting unbroken, etc.). Anyway, here's a row from the other table:
"58569107296622255421594597096899477504","58569107375850417935858934690443427839","NG","Nigeria","Abuja Federal Capital Territory","Abuja","9.057350","7.489760"
So the part I need to match is the first two numbers (first two columns), where there are several ranges from
"0","340282366920938463463374607431768211455"
So I wanted to take the IPv6 addresses, convert them to IP numbers, then sort them into their respective ranges.
Yes, this is something you can do in Python. I'll demonstrate with a few short snippets and links to documentation that will fall short of a full solution in favor of empowering you with the resources that you need to put the pieces together yourself.
First off, if you want to load one CSV file line-by-line and write to a second one this is how you would do it:
>>> import csv
>>> with open('eggs.csv', newline='') as in and open('omellette.csv', 'w') as out:
... r = csv.reader(in)
... w = csv.writer(out)
... for row in r:
... print(', '.join(row)) # print unmodified
... row[0] = ipToNum(row[0])
... row[1] = ipToNum(row[1])
... print(', '.join(row)) # print modified
... w.writerow(row)
Spam, Spam, Spam, Spam, Spam, Baked Beans
Spam, Lovely Spam, Wonderful Spam
The original on which this example was based and additional information about python's built-in CSV capabilities can be found here:
https://docs.python.org/3/library/csv.html
You will probably need to make adjustments depending on the exact formatting of your particular CSV file. Now, to convert IP addresses to numbers you can do something like the following:
import socket, struct
def ipToNum(ip):
"convert ipv4/6 string to long integer"
return struct.unpack('>L',socket.inet_pton(ip))[0]
def numToDottedip(n):
"convert long int to ipv4/6"
return socket.inet_ntop(struct.pack('>L',n))
This example is adapted from what I found here:
https://www.oreilly.com/library/view/python-cookbook/0596001673/ch10s06.html
You will have to modify it
Also, if you want to learn more about the socket and struct modules here is the documentation:
https://docs.python.org/3/library/socket.html
https://docs.python.org/3/library/struct.html
You shouldn't need to split the file up since the CSV reader object will only return one line at a time rather than reading in the whole file at once. Of course, you also probably want to actually do something with those numbers once you've read them in but since you didn't specify I'll figuring that out to you.
Also note that I haven't tried any of this code. It's worth repeating here in the form of a metaphor: I'm trying to teach you to fish rather than just giving you fish. It's in your best interest to take this advice and wrestle with getting it to work yourself as that would be your first step toward actually being a programmer.
My problem is the same as the one mentioned in this answer. I've been trying to understand the code and this is what I learned:
It is failing in the file parse_xml.cgi, tries to get messages (return $message{$name}) from a file named messages (located in the html_en directory).
The $messages value comes from the method GetMessageHash in file adminprotocol-lib.pl:
sub GetMessageHash
{
return $ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"}
}
The $ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"} is set in the file streamingadminserver.pl:
$ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"} = $messages{"en"}
I dont know anything about Perl so I have no idea of what the problem can be, for what I saw $messages{"en"} has the correct value (if I do print($messages{"en"}{'SunStr'} I get the value "Sun")).
However, if I try to do print($ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"}{'SunStr'} I get nothing. Seems like $ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"} is not set
I tried this simple example and it worked fine:
$ENV{"HELLO"} = "hello";
print($ENV{"HELLO"});
and it works fine, prints "hello".
Any idea of what the problem can be?
Looks like $messages{"en"} is a HashRef: A pointer to some memory address holding a key-value-store. You could even print the associated memory address:
perl -le 'my $hashref = {}; print $hashref;'
HASH(0x1548e78)
0x1548e78 is the address, but it's only valid within the same running process. Re-run the sample command and you'll get different addresses each time.
HASH(0x1548e78) is also just a human-readable representation of the real stored value. Setting $hashref2="HASH(0x1548e78)"; won't create a real reference, just a copy of the human-readable string.
You could easily proof this theory using print $ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"} in both script.
Data::Dumper is typically used to show the contents of the referenced hash (memory location):
use Data::Dumper;
print Dumper($messages{"en"});
# or
print Dumper($ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"});
This will also show if the pointer/reference could be dereferenced in both scripts.
The solution for your problem is probably passing the value instead of the HashRef:
$ENV{"QTSSADMINSERVER_EN_SUN"} = $messages{"en"}->{SunStr};
Best Practice is using a -> between both keys. The " or ' quotes for the key also optional if the key is a plain word.
But passing everything through environment variables feels wrong. They might not be able to hold references on OSX (I don't know). You might want to extract the string storage to a include file and load it via require.
See http://www.perlmaven.com/ or http://learn.perl.org for more about Perl.
fix code:
$$ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"} = $messages{"en"};
sub GetMessageHash
{
return $$ENV{"QTSSADMINSERVER_EN_MESSAGEHASH"};
}
ref:
https://github.com/guangbin79/dss6.0.3-linux-patch
I just put my project from localhost to my host and using the same (phpass) function I receive different results and the length of the returned string is also different. (and everything works on localhost.)
Here's the result:
Input: 12345
host:$2a$08$t5joFUIbEThylj/IyUVy7.WzMZm8fmKjsy0kZc/YxgqzkbYQWnlre
localhost: $P$9QiKVfJzpRzoToBCJg8FeaDIfcAJJ61
I was wondering what the problem might be.
thanks for all suggest and answer.
I’m updating an old Perl script to Ruby and having a problem with finding a replacement for one Perl library.
In the Perl script we use Net::CIDR::Lite, which takes a start and end ip address range and outputs a CIDR string.
This is a Perl example that shows the functionality:
#!/usr/bin/perl
use Net::CIDR::Lite;
$cidrblocks = Net::CIDR::Lite->new;
$cidrblocks->add_range("109.152.0.0-109.152.7.255");
$coveragezone = "";
#cidrlist = $cidrblocks->list();
$cidrcount=0;
while ( defined $cidrlist[$cidrcount] ) {
$coveragezone .= "$cidrlist[$cidrcount]";
}
continue {
$cidrcount++;
}
print "$coveragezone";
This script returns a string:
=> 109.152.0.0/21
Does anyone know of a Ruby lib or gem I could use to duplicate the functionality of the add_range call?
$cidrblocks = Net::CIDR::Lite->new;
$cidrblocks->add_range("109.152.0.0-109.152.7.255");
You can either use the built-in IPAddr class which also handles CIDR networks, or you use the ipaddress gem which provides some additional helpers.
A simple example would be:
cidrblocks = []
cidrblocks << IPAddr.new("109.152.0.0/21")
included = cidrblocks.find{|net| net.include?("109.152.6.123") }
I prefer the NetAddr gem, in particular its NetAddr::CIDR class.
It's a very rich IPv4/IPv6 gem.
Looking through the three modules I know of, Ruby's built-in IPAddr, NetAddr and IPAddress, mentioned by #holgerjust, none of them give us the ability to supply a start IP and end IP and return the resulting network. They all assume a CIDR form of the network, and then work toward testing the individual IPs to see if they fit into the subnet, or using that subnet definition to generate the IPs themselves.
I found a ruby module which does this translation see
http://wejn.org/stuff/cidr.rb.html
have included it in my rails app and it's perfect
Can't find a suitable gem?
Do what comes naturally - reimpliment the Perl module in Ruby.
While this may not be entirely relevant to your question, I ported part of CIDR::Lite to Ruby:
https://github.com/noahhaon/cidr-lite-ruby
I found its performance to be far better than the available ruby alternatives at the time for merging very large sets of overlapping CIDRs.
HTH
STILL NOT RESOLVED :( [Feb 11th]
I have a large text file full of random data and want to pull out all the email addresses from it.
I would like to do this in Ruby, with pseudo code like this:
monster_data_string = "asfsfsdfsdfsf sfda **joe#example.com** sdfdsf"
monster_data_string.match(EMAIL_REGEX)
Does anyone know what Ruby email regular expression I would use to accomplish this?
Please keep in mind that I'm looking for a Ruby answer to this. I have already tried numerous regex found by googling but most of them cause Ruby runtime errors stating that characters like "+" and "" are invalid/unrecognized.*
What I have already tried is:
monster_data_string.match(/^([^#\s]+)#((?:[-a-z0-9]+\.)+[a-z]{2,})$/i)
but I receive Ruby errors stating that "+" is an invalid character
Thanks in advance
Watch this...
f = File.open("content.txt")
content = f.read
r = Regexp.new(/\b[a-zA-Z0-9._%+-]+#[a-zA-Z0-9.-]+\.[a-zA-Z]{2,4}\b/)
emails = content.scan(r).uniq
puts YAML.dump(emails)
If you're getting an error message about + or * being invalid in regexes, you're doing something very wrong. This is a valid regex in Ruby, although it's not the one you want:
/^([^#\s]+)#((?:[-a-z0-9]+\.)+[a-z]{2,})$/i
For one thing, you don't want to anchor the regex to the start and end of lines (^ and $) if you're trying to pluck the addresses from "random" text. But once you've gotten rid of the anchors, your regex will match **joe#example.com in your test string, which I presume you don't want. This regex from Regular-Expressions.info does a better job, but read that page for tips on tweaking it to meet your particular needs.
/\b[A-Z0-9._%+-]+#[A-Z0-9.-]+\.[A-Z]{2,4}\b/i
Finally (and you may already know this), you won't want to use the match() method because that will only find the first match. Try scan() instead.
Given that it is not possible to parse every valid email address using a regexp you are left with two choices:
Make a regexp that matches as many valid email addresses as possible and live with the the fact that some valid but rarely used forms of email address might get overlooked.
or
Make a regexp that Matches anything that "might be" an email address and then live with the false positives
I use the second approach to weed out obviously wrong email addresses when validating user sign up email addresses on a web page
Gleaned from Ruby Cookbook which has a very good section on email address validation:
valid = '[^ #]+'
/^#{valid}##{valid}\.#{valid}/
Apparently there is a 6343 character Perl regexp written by Paul Warren that does a very good job and also works in Ruby, but even that is not foolproof (I think it might also have some performance implications).
What kind of runtime error messages are you gettting? Is it regarding the regexps as invalid, or is it breaking due to the target string being too large?
To try and help you get there (though not very elegantly, I admit):
I think the start and end anchors (^ and $) aren't helping. You may also want to filter the asterisks?:
irb(main):001:0> mds = "asfsfsdfsdfsf sfda **joe#example.com** sdfdsf"
=> "asfsfsdfsdfsf sfda **joe#example.com** sdfdsf"
irb(main):003:0> mds.match(/^([^#\s]+)#((?:[-a-z0-9]+\.)+[a-z]{2,})$/i)
=> nil
irb(main):004:0> mds.match(/([^#\s]+)#((?:[-a-z0-9]+\.)+[a-z]{2,})/i)
=> #<MatchData "**joe#example.com" 1:"**joe" 2:"example.com">
irb(main):005:0> mds.match(/([^#\s*]+)#((?:[-a-z0-9]+\.)+[a-z]{2,})/i)
=> #<MatchData "joe#example.com" 1:"joe" 2:"example.com">
Even better,
require 'yaml'
content = "asfsfsdfsdfsf sfda **joe#example.com.au** sdfdsf cool_me#example.com.fr"
r = Regexp.new(/\b([a-zA-Z0-9._%+-]+)#([a-zA-Z0-9.-]+?)(\.[a-zA-Z.]*)\b/)
emails = content.scan(r).uniq
puts YAML.dump(emails)
will give you
---
- - joe
- example
- .com.au
- - cool_me
- example
- .com.au