(I'm using Mac OS X, and this question might be specific to that variant of Unix)
I'm trying to split a file using csplit with a regular expression. It consists of various articles merged into one single long text file. Each article ends with "All Rights Reserved". This is at the end of the line: grep Reserved$ finds them all. Only, csplit claims there is no match.
csplit filename /Reserved$/
yields
csplit: Reserved$: no match
which is a clear and obvious lie. If I leave out the $, it works; but I want to be sure that I don't get any stray occurrences of 'Reserved' in the middle of the text. I tried a different word with the beginning-of-line character ^, and that seems to work. Other words (which do occur at the end of a line in the data) also do not match when used (eg and$).
Is this a known bug with OS X?
[Update: I made sure it's not a DOS/Unix line end character issue by removing all carriage return characters]
I have downloaded the source code of csplit from http://www.opensource.apple.com/source/text_cmds/text_cmds-84/csplit/csplit.c and tested this in the debugger.
The pattern is compiled with
if (regcomp(&cre, re, REG_BASIC|REG_NOSUB) != 0)
errx(1, "%s: bad regular expression", re);
and the lines are matched with
/* Read and output lines until we get a match. */
first = 1;
while ((p = csplit_getline()) != NULL) {
if (fputs(p, ofp) == EOF)
break;
if (!first && regexec(&cre, p, 0, NULL, 0) == 0)
break;
first = 0;
}
The problem is now that the lines returned by csplit_getline() still have a trailing newline character \n. Therefore "Reserved" are not the last characters in the string and the pattern "Reserved$" does not match.
After a quick-and-dirty insertion of
p[strlen(p)-1] = 0;
to remove the trailing newline from the input string the "Reserved$" pattern worked as expected.
There seem to be more problems with csplit in Mac OS X, see the remarks to the answer of Looking for correct Regular Expression for csplit (the repetition count {*} does also not work).
Remark: You can match "Reserved" at the end of the line with the following trick:
csplit filename /Reserved<Ctrl-V><Ctrl-J>/
where you actually use the Control keys to enter a newline character on the command line.
Related
I would like to search through a file and find all instances where the last non-blank character is a comma and move the line below that up one. Essentially, undoing line continuations like
private static final double SOME_NUMBERS[][] = {
{1.0, -6.032174644509064E-23},
{-0.25, -0.25},
{-0.16624879837036133, -2.6033824355191673E-8}
};
and transforming that to
private static final double SOME_NUMBERS[][] = {
{1.0, -6.032174644509064E-23}, {-0.25, -0.25}, {-0.16624879837036133, -2.6033824355191673E-8}
};
Is there a good way to do this?
As mjswartz suggests in the comments, we need a sed substitution command like s/,\n/ /g. That, however, does not work by itself because, by default, sed reads in only one line at a time. We can fix that by reading in the whole file first and then doing the substitution:
$ sed 'H;1h;$!d;x; s/,[[:blank:]]*\n[[:blank:]]*/, /g;' file
private static final double SOME_NUMBERS[][] = {
{1.0, -6.032174644509064E-23}, {-0.25, -0.25}, {-0.16624879837036133, -2.6033824355191673E-8}
};
Because this reads in the whole file at once, this is not a good approach for huge files.
The above was tested with GNU sed.
How it works
H;1h;$!d;x;
This series of commands reads in the whole file. It is probably simplest to think of this as an idiom. If you really want to know the gory details:
H - Append current line to hold space
1h - If this is the first line, overwrite the hold space with it
$!d - If this is not the last line, delete pattern space and jump to the next line.
x - Exchange hold and pattern space to put whole file in pattern space
s/,[[:blank:]]*\n[[:blank:]]*/, /g
This looks for lines that end with a comma, optionally followed by blanks, followed by a newline and replaces that, and any leading space on the following line, with a comma and a single space.
I think for large files awk would be better:
awk -vRS=", *\n" -vORS=", " '1' file
On lua-shell, just write like this:
function nextlineup()
vim:normal("j^y$k$pjddk")
end
vim:open("code.txt")
vim:normal("G")
while vim:k() do
vim:normal("$")
if(vim.currc == string.byte(',')) nextlineup();
end
If you are not familier with vim ,this script seems a bit scary and not robust. In fact, every operation in it is precise(and much quicker, because tey are built-in functions).
Since you are processing a code file, i suggest you try it.
here is a demo
Here is a perl solution.
cat file | perl -e '{$c = 0; while () { s/^\s+/ / if ($c); s/,\s*$/,/; print($_); $c = (m/,\s*$/) ? 1: 0; }}'
I've gotten lost in an edge case of sorts. I'm working on a conversion of some old plaintext documentation to reST/Sphinx format, with the intent of outputting to a few formats (including HTML and text) from there. Some of the documented functions are for dealing with bitstrings, and a common case within these is a sentence like the following: Starting character is the blank " " which has the value 0.
I tried writing this as an inline literal the following ways: Starting character is the blank `` `` which has the value 0. or Starting character is the blank :literal:` ` which has the value 0. but there are a few problems with how these end up working:
reST syntax objects to a whitespace immediately inside of the literal, and it doesn't get recognized.
The above can be "fixed"--it looks correct in the HTML () and plaintext (" ") output--with a non-breaking space character inside the literal, but technically this is a lie in our case, and if a user copied this character, they wouldn't be copying what they expect.
The space can be wrapped in regular quotes, which allows the literal to be properly recognized, and while the output in HTML is probably fine (" "), in plaintext it ends up double-quoted as "" "".
In both 2/3 above, if the literal falls on the wrap boundary, the plaintext writer (which uses textwrap) will gladly wrap inside the literal and trim the space because it's at the start/end of the line.
I feel like I'm missing something; is there a good way to handle this?
Try using the unicode character codes. If I understand your question, this should work.
Here is a "|space|" and a non-breaking space (|nbspc|)
.. |space| unicode:: U+0020 .. space
.. |nbspc| unicode:: U+00A0 .. non-breaking space
You should see:
Here is a “ ” and a non-breaking space ( )
I was hoping to get out of this without needing custom code to handle it, but, alas, I haven't found a way to do so. I'll wait a few more days before I accept this answer in case someone has a better idea. The code below isn't complete, nor am I sure it's "done" (will sort out exactly what it should look like during our review process) but the basics are intact.
There are two main components to the approach:
introduce a char role which expects the unicode name of a character as its argument, and which produces an inline description of the character while wrapping the character itself in an inline literal node.
modify the text-wrapper Sphinx uses so that it won't break at the space.
Here's the code:
class TextWrapperDeux(TextWrapper):
_wordsep_re = re.compile(
r'((?<!`)\s+(?!`)|' # whitespace not between backticks
r'(?<=\s)(?::[a-z-]+:)`\S+|' # interpreted text start
r'[^\s\w]*\w+[a-zA-Z]-(?=\w+[a-zA-Z])|' # hyphenated words
r'(?<=[\w\!\"\'\&\.\,\?])-{2,}(?=\w))') # em-dash
#property
def wordsep_re(self):
return self._wordsep_re
def char_role(name, rawtext, text, lineno, inliner, options={}, content=[]):
"""Describe a character given by unicode name.
e.g., :char:`SPACE` -> "char:` `(U+00020 SPACE)"
"""
try:
character = nodes.unicodedata.lookup(text)
except KeyError:
msg = inliner.reporter.error(
':char: argument %s must be valid unicode name at line %d' % (text, lineno))
prb = inliner.problematic(rawtext, rawtext, msg)
return [prb], [msg]
app = inliner.document.settings.env.app
describe_char = "(U+%05X %s)" % (ord(character), text)
char = nodes.inline("char:", "char:", nodes.literal(character, character))
char += nodes.inline(describe_char, describe_char)
return [char], []
def setup(app):
app.add_role('char', char_role)
The code above lacks some glue to actually force the use of the new TextWrapper, imports, etc. When a full version settles out I may try to find a meaningful way to republish it; if so I'll link it here.
Markup: Starting character is the :char:`SPACE` which has the value 0.
It'll produce plaintext output like this: Starting character is the char:` `(U+00020 SPACE) which has the value 0.
And HTML output like: Starting character is the <span>char:<code class="docutils literal"> </code><span>(U+00020 SPACE)</span></span> which has the value 0.
The HTML output ends up looking roughly like: Starting character is the char:(U+00020 SPACE) which has the value 0.
Is there a better way to write the following regular expression in Ruby? The first regex matches a string that begins with a (lower case) consonant, the second with a vowel.
I'm trying to figure out if there's a way to write a regular expression that matches the negative of the second expression, versus writing the first expression with several ranges.
string =~ /\A[b-df-hj-np-tv-z]/
string =~ /\A[aeiou]/
The statement
$string =~ /\A[^aeiou]/
will test whether the string starts with a non-vowel character, which includes digits, punctuation, whitespace and control characters. That is fine if you know beforehand that the string begins with a letter, but to check that it starts with a consonant you can use forward look-ahead to test that it starts with both a letter and a non-vowel, like this
$string =~ /\A(?=[^aeiou])(?=[a-z])/i
To match an arbitrary number of consonants, you can use the sub-expression (?i:(?![aeiou])[a-z]) to match a consonant. It is atomic, so you can put a repetition count like {3} right after it. For example, this program finds all the strings in a list that contain three consonants in a row
list = %w/ aab bybt xeix axei AAsE SAEE eAAs xxsa Xxsr /
puts list.select { |word| word =~ /\A(?i:(?![aeiou])[a-z]){3}/ }
output
bybt
xxsa
Xxsr
I modified the answer provided by #Alexander Cherednichenko in order to get rid of the if statements.
/^[^aeiou\W]/i.match(s) != nil
If you want to catch a string that doesn't start with vowels, but only starts with consonants you can use this code below. It returns true if a string starts with any letter other than A, E, I, O, U. s is any string we give to a function
if /^[^aeiou\W]/i.match(s) == nil
return false
else
return true
end
i added at the end to make regular expression case insensitive.
\W is used to catch any non-word character, for example if a string starts with a digit like: "1something"
[^aeiou] means a range of character except a e i o u
And we put ^ at the beginning before [ to indicate that the following range [^aeiou\W] if for the 1st character
Note that ^[^aeiou\W] pattern is not correct because it also matches a line that starts with a digit, or underscore. Borodin's solution is working well, but there is one more possible solution without lookaheads, based on character class subtraction (more here) and using the more contemporary Regexp#match?:
/\A[a-z&&[^aeiou]]/i.match?(word)
See the Rubular demo.
Details
\A - start of a string (^ in Ruby is start of any line)
[a-z&&[^aeiou]] - an a-z character range matching any ASCII letter (/i flag makes it case insensitive) except for the aeiou chars.
See the Ruby demo:
test = %w/ 1word _word ball area programming /
puts test.select { |w| /\A[a-z&&[^aeiou]]/i.match?(w) }
# => ['ball', 'programming']
Edit: I solved this by using strip! to remove leading and trailing whitespaces as I show in this video. Then, I followed up by restoring the white space at the end of each string the array by iterating through and adding whitespace. This problem varies from the "dupe" as my intent is to keep the whitespace at the end. However, strip! will remove both the leading and trailing whitespace if that is your intent. (I would have made this an answer, but as this is incorrectly marked as a dupe, I could only edit my original question to include this.)
I have an array of words where I am trying to remove any whitespace that may exist at the beginning of the word instead of at the end. rstrip! just takes care of the end of a string. I want whitespaces removed from the beginning of a string.
example_array = ['peanut', ' butter', 'sammiches']
desired_output = ['peanut', 'butter', 'sammiches']
As you can see, not all elements in the array have the whitespace problem, so I can't just delete the first character as I would if all elements started with a single whitespace char.
Full code:
words = params[:word].gsub("\n", ",").delete("\r").split(",")
words.delete_if {|x| x == ""}
words.each do |e|
e.lstrip!
end
Sample text that a user may enter on the form:
Corn on the cob,
Fibonacci,
StackOverflow
Chat, Meta, About
Badges
Tags,,
Unanswered
Ask Question
String#lstrip (or String#lstrip!) is what you're after.
desired_output = example_array.map(&:lstrip)
More comments about your code:
delete_if {|x| x == ""} can be replaced with delete_if(&:empty?)
Except you want reject! because delete_if will only return a different array, rather than modify the existing one.
words.each {|e| e.lstrip!} can be replaced with words.each(&:lstrip!)
delete("\r") should be redundant if you're reading a windows-style text document on a Windows machine, or a Unix-style document on a Unix machine
split(",") can be replaced with split(", ") or split(/, */) (or /, ?/ if there should be at most one space)
So now it looks like:
words = params[:word].gsub("\n", ",").split(/, ?/)
words.reject!(&:empty?)
words.each(&:lstrip!)
I'd be able to give more advice if you had the sample text available.
Edit: Ok, here goes:
temp_array = text.split("\n").map do |line|
fields = line.split(/, */)
non_empty_fields = fields.reject(&:empty?)
end
temp_array.flatten(1)
The methods used are String#split, Enumerable#map, Enumerable#reject and Array#flatten.
Ruby also has libraries for parsing comma seperated files, but I think they're a little different between 1.8 and 1.9.
> ' string '.lstrip.chop
=> "string"
Strips both white spaces...
I'm not very good with regex, but here's what I got (the string to parse and the regex are on this page) http://rubular.com/r/iIIYDHkwVF
It just needs to match that exact test string
The regular expression is
^"AddonInfo"$(\n\s*)+^\{\s*
It's looking for
^"AddonInfo"$ — a line containing only "AddonInfo"
(\n\s*)+ — followed by at least one newline and possibly many blank or empty lines
^\{\s* — and finally a line beginning with { followed by optional whitespace
To break down a regular expression into its component pieces, have a look at an answer that explains beginning with the basics.
To match the entire string, use
^"AddonInfo"$(\n\s*)+^\{(\s*".+?"\s+".+?"\s*\n)+^\}
So after the open curly, you're looking for one or more lines such that each contains a pair of quote-delimited simple strings (no escaping).
This one works:
^"AddonInfo"[^{]*{[^}]*}
Explanation:
^"AddonInfo" matches "AddonInfo" in the beginning of a line
[^{]* matches all the following non-{ characters
{ matches the following {
[^}]* matches all the following non-} characters
} matches the following }
^"AddonInfo"(\s*)+^\{\s*(?:"([^"]+)"\s+"([^"]*)"\s+)+\}
You will get $1 to point into first key, $2 first value, $3 second key, $4, second value, and so on.
Notice that key is to be non-empty ("([^"]+"), but value may be empty (uses * instead of +).