My script reads in large text files and grabs the first page with a regex. I need to remove the first two lines of each first page or change the regex to match 1 line after the ==Page 1== string. I include the entire script here because I've been asked to in past questions and because I'm new to ruby and don't always know how integrate snippets as answers:
#!/usr/bin/env ruby -wKU
require 'fileutils'
source = File.open('list.txt')
source.readlines.each do |line|
line.strip!
if File.exists? line
file = File.open(line)
end
text = (File.read(line))
match = text.match(/==Page 1(.*)==Page 2==/m)
puts match
end
Now, when You have updated your question, I had to delete a big part of so good answer :-)
I guess the main point of your problem was that you wanted to use match[1] instead of match. The object returned by Regexp.match method (MatchData) can be treated like an array, which holds the whole matched string as the first element, and each subquery in the following elements. So, in your case the variable match (and match[0]) is the whole matched string (together with '==Page..==' marks), but you wanted just the first subexpression which is hidden in match[1].
Now about other, minor problems I sense in your code. Please, don't be offended in case you already know what I say, but maybe others will profit from the warnings.
The first part of your code (if File.exists? line) was checking whether the file exists, but your code just opened the file (without closing it!) and still was trying to open the file few lines later.
You may use this line instead:
next unless File.exists? line
The second thing is that the program should be prepared to handle the situation when the file has no page marks, so it does not match the pattern. (The variable match would then be nil)
The third suggestion is that a little more complicated pattern might be used. The current one (/==Page 1==(.*)==Page 2==/m) would return the page content with the End-Of-Line mark as the first character. If you use this pattern:
/==Page 1==\s*\n(.*)==Page 2==/m
then the subexpression will not contain the white spaces placed in the same line as the '==Page 1==` text. And if you use this pattern:
/==Page 1==\s*\n(.*\n)==Page 2==/m
then you will be sure that the '==Page 2==' mark starts from the beginning of the line.
And the fourth issue is that very often programmers (sometimes including me, of course) tend to forget about closing the file after they opened it. In your case you have opened the 'source' file, but in the code there was no source.close statement after the loop. The most secure way of handling files is by passing a block to the File.open method, so You might use the following form of the first lines of your program:
File.open('list.txt') do |source|
source.readlines.each do |line|
...but in this case it would be cleaner to write just:
File.readlines('list.txt').each do |line|
Taking it all together, the code might look like this (I changed the variable line to fname for better code readability):
#!/usr/bin/env ruby -wKU
require 'fileutils'
File.readlines('list.txt').each do |fname|
fname.strip!
next unless File.exists? fname
text = File.read(fname)
if match = text.match(/==Page 1==\s*\n(.*\n)==Page 2==/m)
# The whole 'page' (String):
puts match[1].inspect
# The 'page' without the first two lines:
# (in case you really wanted to delete lines):
puts match[1].split("\n")[2..-1].inspect
else
# What to do if the file does not match the pattern?
raise "The file #{fname} does NOT include the page separators."
end
end
Related
I have a file, I want to go over that file and any time it matches this "LINE_MATCH_START" and this is the only text in that line (except comments or preceding whitespace), I want it to print everything following it till it matches "LINE_MATCH_END" (which also has to be the only text in that line, comments are allowed but this has to be first thing in the line except whitespace). I want it to go over the entire file and save it as many times it catches it.
Example,
printf ("this is some text")
// comment LINE_MATCH_START this should be ignored
some_other_code
LINE_MATCH_START // can have spaces before it and comments after it
oh
this
should be
saved
LINE_MATCH_END
some_other_piece_of_code
LINE_MATCH_START
AGAIN
lets save this part as well
LINE_MATCH_END
From the above snippet, there can be space before the "LINE_MATCH_START" and it can have comments on the same line but no other piece of code.
I want my code to save all this part
oh
this
should be
saved
AGAIN
lets save this part as well
How can I do this in ruby?
This looks like it gets your output and maybe helps with an idea, but I would work on something more robust.
f = File.new('output.txt', 'w')
visible = false
IO.foreach('file_name') do |line|
case line
when /\s*LINE_MATCH_START.*/ then visible = true
next
when /\s*LINE_MATCH_END.*/ then visible = false
end
f.write(line) if visible
end
I want to add markdown to key phrases in a (gollum) wiki page that will link to the relevant wiki page in the form:
This is the key phrase.
Becomes
This is the [[key phrase|Glossary#key phrase]].
I have a list of key phrases such as:
keywords = ["golden retriever", "pomeranian", "cat"]
And a document:
Sue has 1 golden retriever. John has two cats.
Jennifer has one pomeranian. Joe has three pomeranians.
I want to iterate over every line and find every match (that isn't already a link) for each keyword. My current attempt looks like this:
File.foreach(target_file) do |line|
glosses.each do |gloss|
len = gloss.length
# Create the regex. Avoid anything that starts with [
# or (, ends with ] or ), and ignore case.
re = /(?<![\[\(])#{gloss}(?![\]\)])/i
# Find every instance of this gloss on this line.
positions = line.enum_for(:scan, re).map {Regexp.last_match.begin(0) }
positions.each do |pos|
line.insert(pos, "[[")
# +2 because we just inserted 2 ahead.
line.insert(pos+len+2, "|#{page}\##{gloss}]]")
end
end
puts line
end
However, this will run into a problem if there are two matches for the same key phrase on the same line. Because I insert things into the line, the position I found for each match isn't accurate after the first one. I know I could adjust for the size of my insertions every time but, because my insertions are a different size for each gloss, it seems like the most brute-force, hacky solution.
Is there a solution that allows me to make multiple insertions on the same line at the same time without several arbitrary adjustments each time?
After looking at #BryceDrew's online python version, I realized ruby probably also has a way to fill in the match. I now have a much more concise and faster solution.
First, I needed to make regexes of my glosses:
glosses.push(/(?<![\[\(])#{gloss}(?![\]\)])/i)
Note: The majority of that regex is look-ahead and look-behind assertions to prevent catching a phrase that's already part of a link.
Then, I needed to make a union of all of them:
re = Regexp.union(glosses)
After that, it's as simple as doing gsub on every line, and filling in my matches:
File.foreach(target_file) do |line|
line = line.gsub(re) {|match| "[[#{match}|Glossary##{match.downcase}]]"}
puts line
end
Suppose, I have an input.txt file with the following text:
First line
Second line
Third line
Fourth line
I want to delete, for example, the second and fourth lines to get this:
First line
Third line
So far, I've managed to delete only one the second line using this code
require 'fileutils'
File.open('output.txt', 'w') do |out_file|
File.foreach('input.txt') do |line|
out_file.puts line unless line =~ /Second/
end
end
FileUtils.mv('output.txt', 'input.txt')
What is the right way to delete multiple lines in text file in Ruby?
Deleting lines cleanly and efficiently from a text file is "difficult" in the general case, but can be simple if you can constrain the problem somewhat.
Here are some questions from SO that have asked a similar question:
How do I remove lines of data in the middle of a text file with Ruby
Deleting a specific line in a text file?
Deleting a line in a text file
Delete a line of information from a text file
There are numerous others, as well.
In your case, if your input file is relatively small, you can easily afford to use the approach that you're using. Really, the only thing that would need to change to meet your criteria is to modify your input file loop and condition to this:
File.open('output.txt', 'w') do |out_file|
File.foreach('input.txt').with_index do |line,line_number|
out_file.puts line if line_number.even? # <== line numbers start at 0
end
end
The changes are to capture the line number, using the with_index method, which can be used due to the fact that File#foreach returns an Enumerator when called without a block; the block now applies to with_index, and gains the line number as a second block argument. Simply using the line number in your comparison gives you the criteria that you specified.
This approach will scale, even for somewhat large files, whereas solutions that read the entire file into memory have a fairly low upper limit on file size. With this solution, you're more constrained by available disk space and speed at which you can read/write the file; for instance, doing this to space-limited online storage may not work as well as you'd like. Writing to local disk or thumb drive, assuming that you have space available, should be no problem at all.
Use File.readlines to get an array of the lines in your input file.
input_lines = File.readlines('input.txt')
Then select only those with an even index.
output_lines = input_lines.select.with_index { |_, i| i.even? }
Finally, write those in your output file.
File.open('output.txt', 'w') do |f|
output_lines.each do |line|
f.write line
end
end
I have a large ruby file that contains product data. I'm trying to split the file into sections based on a regular expression. I have product headers denoted by the word Product followed by a space and then a number. After that, I have a bunch of lines containing product information.
The format looks like this.
Product 1:
...data
Product 2:
...data
...
Product N:
...data
When reading from the file, I would like to ignore the Product Headers and instead only show the product data. For that, I'm trying to split the file based on a regular expression.
file = File.read('products.txt')
products = file.split(/\Aproduct \d+:\z/i)
This regex works and finds all product headers. The problem is, the file isn't being split into the appropriate sections.
When I run puts products[0], the entire file gets printed out to the console.
\A and \z match the beginning and end of a string, respectively. While what you want is to match the beginning and end of a line. Instead, you should use ^ and $anchors:
/^product \d+:$/i
The file will be readed complete, there's no way to avoid it. But you can iterate over each line and ignore the one that matches the expresion:
File.open("file.txt", "r") do |file|
file.each_line do |line|
if # condition
#fancy stuff
else
# not that fancy
end
end
I think I may not have done a good enough job explaining my question the first time.
I want to open a bunch of text, and binary files and scan those files with my regular expression. What I need from the csv is to take the data in the second column, which are the paths to all the files, as the means to point to which file to open.
Once the file is opened and the regexp is scanned thru the file, if it matches anything, it displays to the screen. I am sorry for the confusion and thank you so much for everything! –
Hello,
I am sorry for asking what is probably a simple question. I am new to ruby and will appreciate any guidance.
I am trying to use a csv file as an index to leverage other actions.
In particular, I have a csv file that looks like:
id, file, description, date
1, /dir_a/file1, this is the first file, 02/10/11
2, /dir_b/file2, this is the second file, 02/11/11
I want to open every file defined in the "file" column and search for a regular expression.
I know that you can define the headers in each column with the CSV class
require 'rubygems'
require 'csv'
require 'pp'
index = CSV.read("files.csv", :headers => true)
index.each do |row|
puts row ['file']
end
I know how to create a loop that opens every file and search's for a regexp in each file, and if there is one, displays it:
regex = /[0-9A-Za-z]{8,8}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{12,12}/
Dir.glob('/home/Bob/**/*').each do |file|
next unless File.file?(file)
File.open(file, "rb") do |f|
f.each_line do |line|
f.each_line do |line|
unless (pattern = line.scan(regex)).empty?
puts "#{pattern}"
end
end
end
end
end
Is there a way I can use the contents of the second column in my csv file as my variable to open each of the files, search the regexp and if there is a match in the file, output the the row in the csv that had the match to a new csv?
Thank you in advance!!!!
At a quick glance it looks like you could reduce it to:
index.each do |row|
File.foreach(row['file']) do |line|
puts "#{pattern}" if (line[regex])
end
end
A CSV file shouldn't be binary, so you can drop the 'rb' when opening the file, letting us reduce the file read to foreach, which iterates over the file, returning it line by line.
The depth of the files in your directory hierarchy is in question based on your sample code. It's not real clear what's going on there.
EDIT:
it tells me that "regex" is an undefined variable
In your question you said:
regex = /[0-9A-Za-z]{8,8}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{4,4}-[0-9A-Za-z]{12,12}/
the files I open to do the search on may be a binary.
According to the spec:
Common usage of CSV is US-ASCII, but other character sets defined by IANA for the "text" tree may be used in conjunction with the "charset" parameter.
It goes on to say:
Security considerations:
CSV files contain passive text data that should not pose any
risks. However, it is possible in theory that malicious binary
data may be included in order to exploit potential buffer overruns
in the program processing CSV data. Additionally, private data
may be shared via this format (which of course applies to any text
data).
So, if you're seeing binary data you shouldn't because it's not CSV according to the spec. Unfortunately the spec has been abused over the years, so it's possible you are seeing binary data in the file. If so, continue to use 'rb' as the file mode but do it cautiously.
An important question to ask is whether you can read the file using Ruby's CSV library, which makes a lot of this a moot discussion.