I need to scrape the first three sentences from a paragraph, if they exist, using XPath.
I've already isolated the paragraph I want using:
//h3[contains(., 'Synopsis')]/following-sibling::p[1]
Which returns a plain, unformatted paragraph:
What do we do when the world's walls - its family structures, its value-systems, it political forms - crumble? The central character of this novel, 'Moor' Zogoiby, only son of a wealthy, artistic-bohemian Bombay family, finds himself in such a moment of crisis. His mother, a famous painter and an emotional despot, worships beauty, but Moor is ugly, he has a deformed hand. Moor falls in love, with a married woman; when their secret is revealed, both are expelled; a suicide pact is proposed, but only the woman dies. Moor chooses to accept his fate, plunges into a life of depravity in Bombay, then becomes embroiled in a major financial scandal. The novel ends in Spain, in the studio of a painter who was a lover of Moor's mother: in a violent climax Moor has, one more, to decide whether to save the life of his lover by sacrificing his own.
I only want the first three sentences, and I'm willing to be lenient and ignore that first question mark, I just want whatever comes before the first three periods.
concat(
substring-before(//h3[contains(., 'Synopsis')]/following-sibling::p[1]/text(), '.'),
'.',
substring-before(substring-after(//h3[contains(., 'Synopsis')]/following-sibling::p[1]/text(), '.'), '.'),
'.',
substring-before(substring-after(substring-after(//h3[contains(., 'Synopsis')]/following-sibling::p[1]/text(), '.'), '.'), '.'),
'.'
)
(It's fun to do crazy things with XPath, but in real-life scenarios I wouldn't use it for tasks like this unless forced to it by absolute lack of other possibilities.)
Related
For example, let's say I have text like this:
LOCATION, Text goes on
LOCATION U.S. investigators goes on
LOCATION/LOCATION First Last's goes on
LOCATION (AB) -- The Stack Overflow goes on
WHITE PLAINS, N.Y. (AB) -- Text goes on
PUEBLO, Colo. (AB) -- Text goes on
How would I make an algorithm to determine the boundary between the LOCATION and the article text?
The algorithm should be flexible, since there are many formats of datelines in use.
I know this can be done by algorithm, since the boundary is clearly distinguishable even after randomising the chracters.
SYAEIDUA, Tuqw gzce ox
QOZHANEPAD G.L. qisuxhen aodien
ADFD/QOIEYTYE Qidne Opaidh's wien aidnen
QIUEHN (XC) -- Ehd Towneyd Apenaid goeis he
IQUEN AOIEND, B.I. (OG) -- Qien oane px
OIQHNED, Qien. (PA) -- Nwne oaien pdxdaf
The only confusion I can see is in the second case, where G.L. could either stand for U.S. and part of main text, or part of a city abbreviation and NOT part of main text. e.g.
WASHINGTON D.C. Government officials on Monday...
(Government officials on Monday...)
NEW YORK U.S. Government's latest statement...
(U.S. Government's latest statement...)
NEW YORK A.B. Conglomerate's CEO said on Monday...
(A.B. Conglomerate's CEO said on Monday...)
This is where regex falls short since it can't use a lookup table or the likes to differentiate between the two cases. I can't just hard-code in U.S. as a special case after all (see third case).
Any ideas?
Is it possible to use WordNet to rewrite a sentence so that the semantic meaning of the sentence still ways the same (or mostly the same)?
Let's say I have this sentence:
Obama met with Putin last week.
Is it possible to use WordNet to rephrase the sentence into alternatives like:
Obama and Putin met the previous week.
Obama and Putin met each other a week ago.
If changing the sentence structure is not possible, can WordNet be used to replace only the relevant synonyms?
For example:
Obama met Putin the previous week.
If the question is the possibility to use WordNet to do sentence paraphrases. It is possible with much grammatical/syntax components. You would need system that:
First get the individual semantics of the tokens and parse the sentence for its syntax.
Then understand the overall semantics of the composite sentence (especially if it's metaphorical)
Then rehash the sentence with some grammatical generator.
Up till now I only know of ACE parser/generator that can do something like that but it takes a LOT of hacking the system to make it work as a paraphrase generator. http://sweaglesw.org/linguistics/ace/
So to answer your questions,
Is it possible to use WordNet to rephrase the sentence into alternatives? Sadly, WordNet isn't a silverbullet. You will need more than semantics for a paraphrase task.
If changing the sentence structure is not possible, can WordNet be used to replace only the relevant synonyms? Yes this is possible. BUT to figure out which synonym is replace-able is hard... And you would also need some morphology/syntax component.
First you will run into a problem of multiple senses per word:
from nltk.corpus import wordnet as wn
sent = "Obama met Putin the previous week"
for i in sent.split():
possible_senses = wn.synsets(i)
print i, len(possible_senses), possible_senses
[out]:
Obama 0 []
met 13 [Synset('meet.v.01'), Synset('meet.v.02'), Synset('converge.v.01'), Synset('meet.v.04'), Synset('meet.v.05'), Synset('meet.v.06'), Synset('meet.v.07'), Synset('meet.v.08'), Synset('meet.v.09'), Synset('meet.v.10'), Synset('meet.v.11'), Synset('suffer.v.10'), Synset('touch.v.05')]
Putin 1 [Synset('putin.n.01')]
the 0 []
previous 3 [Synset('previous.s.01'), Synset('former.s.03'), Synset('previous.s.03')]
week 3 [Synset('week.n.01'), Synset('workweek.n.01'), Synset('week.n.03')]
Then even if you know the sense (let's say the first sense), you get multiple words per sense and not every word can be replaced in the sentence. Moreover, they are in the lemma form not a surface form (e.g. verbs are in their base form (simple present tense) and nouns are in singular):
from nltk.corpus import wordnet as wn
sent = "Obama met Putin the previous week"
for i in sent.split():
possible_senses = wn.synsets(i)
if possible_senses:
print i, possible_senses[0].lemma_names
else:
print i
[out]:
Obama
met ['meet', 'run_into', 'encounter', 'run_across', 'come_across', 'see']
Putin ['Putin', 'Vladimir_Putin', 'Vladimir_Vladimirovich_Putin']
the
previous ['previous', 'old']
week ['week', 'hebdomad']
One approach is grammatical analysis with nltk read more here and after analysis convert your sentence in to active voice or passive voice.
I'm trying to extract the text for a document to index it for search. The below mostly works except various words and punctuation run together. When it removes tags, I need to replace them with spaces so I do not get this issue. I have been trying to figure out the most efficient way to do this but I'm coming up empty so far.
doc = Nokogiri::HTML(html)
doc.xpath("//script").remove
doc.xpath("//style").remove
doc.xpath("//a").remove
text = doc.text.gsub(/\s+/,' ')
Here is some sample text I extracted from http://www.washingtontimes.com/blog/redskins-watch/2012/oct/18/redskins-linemen-respond-jason-pierre-paul-rg3-com/
Before the season it was New York Giants defensive end Osi Umenyiora
who made waves by saying he wouldn't call Robert Griffin III by “RG3”
until he did something. Until then, it was “Bob Griffin.”After
Griffin's 76-yard touchdown run in the Washington Redskins' victory
over the Minnesota Vikings, fellow Giants defensive end Jason
Pierre-Paul was the one who had some comments about Griffin.“Don’t
bring it to my side," Pierre-Paul told New York media. “Go the other
way. …“Yes, it'll be a very good matchup. Not on my side, though. Not
on my side. Or the other side.”Griffin, asked jokingly Wednesday about
running for office, said: “I’ve got a lot other guys to be running
away from right now, Pierre-Paul, Osi, all those guys.”But according
to a couple of Redskins linemen, Griffin shouldn't have much to worry
about Sunday if he gets into the open field.“If Robert gets into that
situation, I don't think there's many people that can run him down,”
right guard Chris Chester said. “I'm still going to go out there and
try to block and make sure no one touches Robert at all. But he's a
plenty good athlete to be able to outrun a lot of people in this
league.”Prompted with Pierre-Paul's comments, left tackle Trent
Williams responded: “What do you want me to say about that?”“Robert's
my guy. I don't know Pierre-Paul. I don't know why he would say
something like that,” he said. “Maybe he knows something I don't.”
You could try inserting a space before each p tag:
doc.search('p').each{|el| el.before ' '}
but a better approach probably is something like:
text = doc.search('div.story p').map{|p| p.text}.join(" ")
Other answers are discussing inserting whitespace into the document, but if (as the question asks) your requirement is to replace those nodes with whitespace, Nokogiri has a replace method. So to replace script tags with spaces do:
doc.xpath('//script').each do |node|
node.replace(' ')
end
The question also asks about 'correct' spacing. Most browsers will not insert a space when they render around a <script> tag, so while useful for text extraction, this is not necessarily the 'correct' thing to do.
iPhone has a pretty good telephone number splitting function, for example:
Singapore mobile: +65 9852 4135
Singapore resident line: +65 6325 6524
China mobile: +86 135-6952-3685
China resident line: +86 10-65236528
HongKong: +886 956-238-82
USA: +1 (732) 865-3286
Notice the nice features here:
- the splitting of country code, area code, and the rest is automatic;
- the delimiter is also nicely adopted to different countries, e.g. "()", "-" and space.
Note the parsing logic is doable to me, however, I don't know where to get the knowledge of most countries' telephone number format.
where could i found such knowledge, or an open source code that implemented it?
You can get similar functionality with the libphonenumber code library.
Interestingly enough, you cannot use an NSNumberFormatter for this, but you can write your own custom class for it. Just create a new class, set properties such as countryCode, areaCode and number, and then create a method that formats the number based on the countryCode.
Here's a great example: http://the-lost-beauty.blogspot.com/2010/01/locale-sensitive-phone-number.html
As an aside: a friend told me about a gigantic regular expression he had to maintain that could pick telephone numbers out of intercepted communications from hundreds of countries around the world. It was very non-trivial.
Thankfully your problem is easier, as you can just have a table with the per-country formats:
format[usa] = "+d (ddd) ddd-dddd";
format[hk] = "+ddd ddd-ddd-dd";
format[china_mobile] = "+dd ddd-dddd-dddd";
...
Then when you're printing, you simply output one digit from the phone number string in each d spot as needed. This assumes you know the country, which is a safe enough assumption for telephone devices -- pick "default" formats for the few surrounding countries.
Since some countries have different formats with different lengths you might need to store your table with additional information:
format[germany][10] = "..."
format[germany][11] = "....."
So I'm coding in Ruby and I've got a few sentences:
The sky above the port was the color of television, tuned to a dead channel. "It's not like I'm using," Case heard someone say, as he shouldered his way through the crowd around the door of the Chat. "It's like my body's developed this massive drug deficiency." It was a Sprawl voice and a Sprawl joke. The Chatsubo was a bar for professional expatriates; you could drink there for a week and never hear two words in Japanese.
And I need to modify every word in the paragraph without changing the structure. My original idea was to just split on whitespace and then rejoin it, but the issue with that is you get the punctuation as well. If you split so that you just get the word, it's hard to rejoin because you don't know the proper punctuation.
Are there better ways to do this than the traditional split, map, join combo? Or maybe just a good split regex so it's easy to rejoin?
Use gsub with a block:
str = %q(The sky above the port was the color of television, tuned to a dead channel.
"It's not like I'm using," Case heard someone say, as he shouldered his way through the crowd
around the door of the Chat. "It's like my body's developed this massive drug deficiency."
It was a Sprawl voice and a Sprawl joke. The Chatsubo was a bar for professional expatriates;
you could drink there for a week and never hear two words in Japanese.)
puts str.gsub(/\w+/){|word| word.tr('aeiou','uoaei') }
result:
Tho sky ubevo tho pert wus tho celer ef tolovasaen, tinod te u doud chunnol.
"It's net lako I'm isang," Cuso hourd semoeno suy, us ho sheildorod has wuy threigh tho crewd
ureind tho deer ef tho Chut. "It's lako my bedy's dovolepod thas mussavo drig dofacaoncy."
It wus u Spruwl veaco und u Spruwl jeko. Tho Chutsibe wus u bur fer prefossaenul oxputrautos;
yei ceild drank thoro fer u wook und novor hour twe werds an Jupunoso.
Well, this #tr method would work without the regex, but you get the idea.
I would match words between word boundaries with a regex to avoid affecting punctuation or whitespace, e.g.:
s = "This is a test, ok? Yes, fine!"
s.gsub!(/\b(\w+)\b/) {|x| "_#{x}_"}
s = "_This_ _is_ _a_ _test_, _ok_? _Yes_, _fine_!"