I'm working with mails, and names and subjects sometimes come q-encoded, like this:
=?UTF-8?Q?J=2E_Pablo_Fern=C3=A1ndez?=
Is there a way to decode them in Ruby? It seems TMail should take care of it, but it's not doing it.
I use this to parse email subjects:
You could try the following:
str = "=?UTF-8?Q?J=2E_Pablo_Fern=C3=A1ndez?="
if m = /=\?([A-Za-z0-9\-]+)\?(B|Q)\?([!->#-~]+)\?=/i.match(str)
case m[2]
when "B" # Base64 encoded
decoded = Base64.decode64(m[3])
when "Q" # Q encoded
decoded = m[3].unpack("M").first.gsub('_',' ')
else
p "Could not find keyword!!!"
end
Iconv.conv('utf-8',m[1],decoded) # to convert to utf-8
end
Ruby includes a method of decoding Quoted-Printable strings:
puts "Pablo_Fern=C3=A1ndez".unpack "M"
# => Pablo_Fernández
But this doesn't seem to work on your entire string (including the =?UTF-8?Q? part at the beginning. Maybe you can work it out from there, though.
This is a pretty old question but TMail::Unquoter (or its new incarnation Mail::Encodings) does the job as well.
TMail::Unquoter.unquote_and_convert_to(str, 'utf-8' )
or
Mail::Encodings.unquote_and_convert_to( str, 'utf-8' )
Decoding on a line-per-line basis:
line.unpack("M")
Convert STDIN or file provided input of encoded strings into a decoded output:
if ARGV[0]
lines = File.read(ARGV[0]).lines
else
lines = STDIN.each_line.to_a
end
puts lines.map { |c| c.unpack("M") }.join
This might help anyone wanting to test an email. delivery.html_part is normally encoded, but can be decoded to a straight HTML body using .decoded.
test "email test" do
UserMailer.confirm_email(user).deliver_now
assert_equal 1, ActionMailer::Base.deliveries.size
delivery = ActionMailer::Base.deliveries.last
assert_equal "Please confirm your email", delivery.subject
assert delivery.html_part.decoded =~ /Click the link below to confirm your email/ # DECODING HERE
end
The most efficient and up to date solution it seems to use the value_decode method of the Mail gem.
> Mail::Encodings.value_decode("=?UTF-8?Q?Greg_of_Google?=")
=> "Greg of Google"
https://www.rubydoc.info/github/mikel/mail/Mail/Encodings#value_decode-class_method
Below is Ruby code you can cut-and-paste, if inclined. It will run tests if executed directly with ruby, ruby ./copy-pasted.rb. As done in the code, I use this module as a refinement to the String core class.
A few remarks on the solution:
Other solutions perform .gsub('_', ' ') on the unpacked string. However, I do not believe this is correct, and can result in an incorrect decoding depending on the charsets. RFC2047 Section 4.2 (2) indicates "_ always represents hexidecimal 20", so it seems correct to first substitute =20 for _ then rely on the unpack result. (This also makes the implementation more elegant.) This is also discussed in an answer to a related question.
To be more instructive, I have written the regular expression in free-spacing mode to allow comments (I find this generally helpful for complex regular expressions). If you adjust the regular expression, take note that free-spacing mode changes the matching of white-space, which must then be done escaped or as a character class (as in the code). I've also added the regular expression on regex101, so you can read an explanation of the named capture groups, lazy quantifiers, etc. and experiment yourself.
The regular expression will absorb space ( ; but not TAB or newline) between multiple Q-encoded phrases in a single string, as shown with string test_4. This is because RFC2047 Section 5 (1) indicates that multiple Q encoded phrases must be separated from each other by linear white-space. Depending on your use-case, absorbing the white-space may not be desired.
The regular expression code named capture permits unexpected quoted printable codes (other than [bBqQ] so that a match will occur and the code can raise an error. This helps me to detect unexpected values when processing text. Change the regular expression named capture for code to [bBqQ] if you do not want this behaviour. (There will be no match and the original string will be returned.)
It makes use of the global Regexp.last_match as a convenience in the gsub block. You may need to take care if using this in multi-threaded code, I have not given this any consideration.
Additional references and reading:
https://en.wikipedia.org/wiki/Quoted-printable
https://en.wikipedia.org/wiki/MIME#Encoded-Word
require "minitest/autorun"
module QuotedPrintableDecode
class UnhandledCodeError < StandardError
def initialize(code)
super("Unhandled quoted printable code: '#{code}'.")
end
end
##qp_text_regex = %r{
=\? # Opening literal: `=?`
(?<charset>[^\?]+) # Character set, e.g. "Windows-1252" in `=?Windows-1252?`
\? # Literal: `?`
(?<code>[a-zA-Z]) # Encoding, e.g. "Q" in `?Q?` (`B`ase64); [BbQq] expected, others raise
\? # Literal: `?`
(?<text>[^\?]+?) # Encoded text, lazy (non-greedy) matched, e.g. "Foo_bar" in `?Foo_bar?`
\?= # Closing literal: `?=`
(?:[ ]+(?==\?))? # Optional separating linear whitespace if another Q-encode follows
}x # Free-spacing mode to allow above comments, also changes whitespace match
refine String do
def decode_q_p(to: "UTF-8")
self.gsub(##qp_text_regex) do
code, from, text = Regexp.last_match.values_at(:code, :charset, :text)
q_p_charset_to_charset(code, text, from, to)
end
end
private
def q_p_charset_to_charset(code, text, from, to)
case code
when "q", "Q"
text.gsub("_", "=20").unpack("M")
when "b", "B"
text.unpack("m")
else
raise UnhandledCodeError.new(code)
end.first.encode(to, from)
end
end
end
class TestQPDecode < Minitest::Test
using QuotedPrintableDecode
def test_decode_single_utf_8_phrase
encoded = "=?UTF-8?Q?J=2E_Pablo_Fern=C3=A1ndez?="
assert_equal encoded.decode_q_p, "J. Pablo Fernández"
end
def test_decoding_preserves_space_between_unencoded_phrase
encoded = "=?utf-8?Q?Alfred_Sanford?= <me#example.com>"
assert_equal encoded.decode_q_p, "Alfred Sanford <me#example.com>"
end
def test_decodinge_multiple_adjacent_phrases_absorbs_separating_whitespace
encoded = "=?Windows-1252?Q?Foo_-_D?= =?Windows-1252?Q?ocument_World=9617=96520;_Recor?= =?Windows-1252?Q?d_People_to_C?= =?Windows-1252?Q?anada's_History?="
assert_equal encoded.decode_q_p, "Foo - Document World–17–520; Record People to Canada's History"
end
def test_decoding_string_without_encoded_phrases_preserves_original
encoded = "Contains no QP phrases"
assert_equal encoded.decode_q_p, encoded
end
def test_unhandled_code_raises
klass = QuotedPrintableDecode::UnhandledCodeError
message = "Unhandled quoted printable code: 'Z'."
encoded = "=?utf-8?Z?Unhandled code Z?="
raised_error = assert_raises(klass) { encoded.decode_q_p }
assert_equal message, raised_error.message
end
end
Related
I want to write a regex pattern to look for a delimiter within a string such as this: '//;\n1;2' where the character ; serves as the delimiter for the given string. I'm trying to solve a common coding kata problem, so I don't necessarily want an answer in code, but an idea as to how I can accomplish this with regex. This delimiter is subject to change and could be any character. Any ideas? Here's what I've written so far in my production file.
class StringCalculator
def add(numbers)
return 0 if numbers.empty?
return nil if numbers.end_with?('\n')
numbers.tr!('\n', ',')
numbers.split(',').map(&:to_i).reduce(:+)
end
end
Spec:
context 'when passed different delimiters' do
it 'should support semicolons' do
expect(#calculator.add('//;\n1;2')).to eq(3)
end
it 'should support percents' do
expect(#calculator.add('//%\n1%5')).to eq(6)
end
end
I am writing a 6502 assembler in Ruby. I am looking for a way to validate hexadecimal operands in string form. I understand that the String object provides a "hex" method to return a number, but here's a problem I run into:
"0A".hex #=> 10 - a valid hexadecimal value
"0Z".hex #=> 0 - invalid, produces a zero
"asfd".hex #=> 10 - Why 10? I guess it reads 'a' first and stops at 's'?
You will get some odd results by typing in a bunch of gibberish. What I need is a way to first verify that the value is a legit hex string.
I was playing around with regular expressions, and realized I can do this:
true if "0A" =~ /[A-Fa-f0-9]/
#=> true
true if "0Z" =~ /[A-Fa-f0-9]/
#=> true <-- PROBLEM
I'm not sure how to address this issue. I need to be able to verify that letters are only A-F and that if it is just numbers that is ok too.
I'm hoping to avoid spaghetti code, riddled with "if" statements. I am hoping that someone could provide a "one-liner" or some form of elegent code.
Thanks!
!str[/\H/] will look for invalid hex values.
String#hex does not interpret the whole string as hex, it extracts from the beginning of the string up to as far as it can be interpreted as hex. With "0Z", the "0" is valid hex, so it interpreted that part. With "asfd", the "a" is valid hex, so it interpreted that part.
One method:
str.to_i(16).to_s(16) == str.downcase
Another:
str =~ /\A[a-f0-9]+\Z/i # or simply /\A\h+\Z/ (see hirolau's answer)
About your regex, you have to use anchors (\A for begin of string and \Z for end of string) to say that you want the full string to match. Also, the + repeats the match for one or more characters.
Note that you could use ^ (begin of line) and $ (end of line), but this would allow strings like "something\n0A" to pass.
This is an old question, but I just had the issue myself. I opted for this in my code:
str =~ /^\h+$/
It has the added benefit of returning nil if str is nil.
Since Ruby has literal hex built-in, you can eval the string and rescue the SyntaxError
eval "0xA" => 10
eval "0xZ" => SyntaxError
You can use this on a method like
def is_hex?(str)
begin
eval("0x#{str}")
true
rescue SyntaxError
false
end
end
is_hex?('0A') => true
is_hex?('0Z') => false
Of course since you are using eval, make sure you are sending only safe values to the methods
Given a regular expression:
/say (hullo|goodbye) to my lovely (.*)/
and a string:
"my $2 is happy that you said $1"
What is the best way to obtain a regular expression from the string that contains the capture groups in the regular expression? That is:
/my (.*) is happy that you said (hullo|goodbye)/
Clearly I could use regular expressions on a string representation of the original regular expression, but this would probably present difficulties with nested capture groups.
I'm using Ruby. My simple implementation so far goes along the lines of:
class Regexp
def capture_groups
self.to_s[1..-2].scan(/\(.*?\)/)
end
end
regexp.capture_groups.each_with_index do |capture, idx|
string.gsub!("$#{idx+1}", capture)
end
/^#{string}$/
i guess you need to create your own function that would do this:
create empty dictionaries groups and active_groups and initialize counter = 1
iterate over the characters in the string representation:
if current character = '(' and previous charaster != \:
add counter key to active_groups and increase counter
add current character to all active_groups
if current character = ')' and previous charaster != \:
remove the last item (key, value) from active_groups and add it to groups
convert groups to an array if needed
You might also want to implement:
ignore = True between unescaped '[' and ']'
reset counter if current character = '|' and active_groups is empty (or decrease counter if active_group is not empty)
UPDATES from comments:
ingore non-capturing groups starting with '(?:'
So once I realised that what I actually need is a regular expression parser, things started falling into place. I discovered this project:
https://github.com/dche/randall
which can generate strings that match a regular expression. It defines a regular expression grammar using http://treetop.rubyforge.org/. Unfortunately the grammar it defines is incomplete, though useful for many cases.
I also stumbled past https://github.com/mjijackson/citrus, which does a similar job to Treetop.
I then found this mind blowing gem:
https://github.com/ammar/regexp_parser
which defines a full regexp grammar and parses a regular expression into a walkable tree. I was then able to walk the tree and pick out the parts of the tree I wanted (the capture groups).
Unfortunately there was a minor bug, fixed in my fork: https://github.com/LaunchThing/regexp_parser.
Here's my patch to Regexp, that uses the fixed gem:
class Regexp
def parse
Regexp::Parser.parse(self.to_s, 'ruby/1.9')
end
def walk(e = self.parse, depth = 0, &block)
block.call(e, depth)
unless e.expressions.empty?
e.each do |s|
walk(s, depth+1, &block)
end
end
end
def capture_groups
capture_groups = []
walk do |e, depth|
capture_groups << e.to_s if Regexp::Expression::Group::Capture === e
end
capture_groups
end
end
I can then use this in my application to make replacements in my string - the final goal - along these lines:
from = /^\/search\/(.*)$/
to = '/buy/$1'
to_as_regexp = to.dup
# I should probably make this gsub tighter
from.capture_groups.each_with_index do |capture, idx|
to_as_regexp.gsub!("$#{idx+1}", capture)
end
to_as_regexp = /^#{to_as_regexp}$/
# to_as_regexp = /^\/buy\/(.*)$/
I hope this helps someone else out.
So how can I still be able to write beautiful code such as:
'im a string meing!'.pop
Note: str.chop isn't sufficient answer
It is not what an enumerable string atually enumerates. Is a string a sequence of ...
lines,
characters,
codepoints or
bytes?
The answer is: all of those, any of those, either of those or neither of those, depending on the context. Therefore, you have to tell Ruby which of those you actually want.
There are several methods in the String class which return enumerators for any of the above. If you want the pre-1.9 behavior, your code sample would be
'im a string meing!'.bytes.to_a.pop
This looks kind of ugly, but there is a reason for it: a string is a sequence. You are treating it as a stack. A stack is not a sequence, in fact it pretty much is the opposite of a sequence.
That's not beautiful :)
Also #pop is not part of Enumerable, it's part of Array.
The reason why String is not enumerable is because there are no 'natural' units to enumerate, should it be on a character basis or a line basis? Because of this String does not have an #each
String instead provides the #each_char and #each_byte and #each_line methods for iteration in the way that you choose.
Since you don't like str[str.length], how about
'im a string meing!'[-1] # returns last character as a character value
or
'im a string meing!'[-1,1] # returns last character as a string
or, if you need it modified in place as well, while keeping it an easy one-liner:
class String
def pop
last = self[-1,1]
self.chop!
last
end
end
#!/usr/bin/ruby1.8
s = "I'm a string meing!"
s, last_char = s.rpartition(/./)
p [s, last_char] # => ["I'm a string meing", "!"]
String.rpartition is new for 1.9 but it's been back-ported to 1.8.7. It searches a string for a regular expression, starting at the end and working backwards. It returns the part of the string before the match, the match, and the part of the string after the match (which we discard here).
String#slice! and String#insert is going to get you much closer to what you want without converting your strings to arrays.
For example, to simulate Array#pop you can do:
text = '¡Exclamation!'
mark = text.slice! -1
mark == '!' #=> true
text #=> "¡Exclamation"
Likewise, for Array#shift:
text = "¡Exclamation!"
inverted_mark = text.slice! 0
inverted_mark == '¡' #=> true
text #=> "Exclamation!"
Naturally, to do an Array#push you just use one of the concatenation methods:
text = 'Hello'
text << '!' #=> "Hello!"
text.concat '!' #=> "Hello!!"
To simulate Array#unshift you use String#insert instead, it's a lot like the inverse of slice really:
text = 'World!'
text.insert 0, 'Hello, ' #=> "Hello, World!"
You can also grab chunks from the middle of a string in multiple ways with slice.
First you can pass a start position and length:
text = 'Something!'
thing = text.slice 4, 5
And you can also pass a Range object to grab absolute positions:
text = 'This is only a test.'
only = text.slice (8..11)
In Ruby 1.9 using String#slice like this is identical to String#[], but if you use the bang method String#slice! it will actually remove the substring you specify.
text = 'This is only a test.'
only = text.slice! (8..12)
text == 'This is a test.' #=> true
Here's a slightly more complex example where we reimplement a simple version of String#gsub! to do a search and replace:
text = 'This is only a test.'
search = 'only'
replace = 'not'
index = text =~ /#{search}/
text.slice! index, search.length
text.insert index, replace
text == 'This is not a test.' #=> true
Of course 99.999% of the time, you're going to want to use the aforementioned String.gsub! which will do the exact same thing:
text = 'This is only a test.'
text.gsub! 'only', 'not'
text == 'This is not a test.' #=> true
references:
Ruby String Documentation
I have a string
input = "maybe (this is | that was) some ((nice | ugly) (day |night) | (strange (weather | time)))"
How is the best method in Ruby to parse this string ?
I mean the script should be able to build sententes like this :
maybe this is some ugly night
maybe that was some nice night
maybe this was some strange time
And so on, you got the point...
Should I read the string char by char and bulid a state machine with a stack to store the parenthesis values for later calculation, or is there a better approach ?
Maybe a ready, out of the box library for such purpose ?
Try Treetop. It is a Ruby-like DSL to describe grammars. Parsing the string you've given should be quite easy, and by using a real parser you'll easily be able to extend your grammar later.
An example grammar for the type of string that you want to parse (save as sentences.treetop):
grammar Sentences
rule sentence
# A sentence is a combination of one or more expressions.
expression* <Sentence>
end
rule expression
# An expression is either a literal or a parenthesised expression.
parenthesised / literal
end
rule parenthesised
# A parenthesised expression contains one or more sentences.
"(" (multiple / sentence) ")" <Parenthesised>
end
rule multiple
# Multiple sentences are delimited by a pipe.
sentence "|" (multiple / sentence) <Multiple>
end
rule literal
# A literal string contains of word characters (a-z) and/or spaces.
# Expand the character class to allow other characters too.
[a-zA-Z ]+ <Literal>
end
end
The grammar above needs an accompanying file that defines the classes that allow us to access the node values (save as sentence_nodes.rb).
class Sentence < Treetop::Runtime::SyntaxNode
def combine(a, b)
return b if a.empty?
a.inject([]) do |values, val_a|
values + b.collect { |val_b| val_a + val_b }
end
end
def values
elements.inject([]) do |values, element|
combine(values, element.values)
end
end
end
class Parenthesised < Treetop::Runtime::SyntaxNode
def values
elements[1].values
end
end
class Multiple < Treetop::Runtime::SyntaxNode
def values
elements[0].values + elements[2].values
end
end
class Literal < Treetop::Runtime::SyntaxNode
def values
[text_value]
end
end
The following example program shows that it is quite simple to parse the example sentence that you have given.
require "rubygems"
require "treetop"
require "sentence_nodes"
str = 'maybe (this is|that was) some' +
' ((nice|ugly) (day|night)|(strange (weather|time)))'
Treetop.load "sentences"
if sentence = SentencesParser.new.parse(str)
puts sentence.values
else
puts "Parse error"
end
The output of this program is:
maybe this is some nice day
maybe this is some nice night
maybe this is some ugly day
maybe this is some ugly night
maybe this is some strange weather
maybe this is some strange time
maybe that was some nice day
maybe that was some nice night
maybe that was some ugly day
maybe that was some ugly night
maybe that was some strange weather
maybe that was some strange time
You can also access the syntax tree:
p sentence
The output is here.
There you have it: a scalable parsing solution that should come quite close to what you want to do in about 50 lines of code. Does that help?