Retrieve online data and generate and xml output of it - windows

Question

The project requires to grep online data and generate an xml file of it. This is how the output should be:
<!DOCTYPE MetaIssue SYSTEM "http://schema.highwire.org/public/toc/MetaIssue.pubids.dtd">
<MetaIssue volume="306" issue="1">
<Provider>Cadmus</Provider>
<IssueDate>January 1, 2014</IssueDate>
<PageRange>C1-C76</PageRange>
<TOC>
<TocSection>
<Heading>Editorial Focus</Heading>
<DOI>10.1152/ajpcell.00342.2013</DOI>
</TocSection>
<TocSection>
<Heading>Review</Heading>
<DOI>10.1152/ajpcell.00281.2013</DOI>
</TocSection>
<TocSection>
<Heading>CALL FOR PAPERS | Stem Cell Physiology and Pathophysiology</Heading>
<DOI>10.1152/ajpcell.00156.2013</DOI>
<DOI>10.1152/ajpcell.00066.2013</DOI>
</TocSection>
<TocSection>
<Heading>Articles</Heading>
<DOI>10.1152/ajpcell.00130.2013</DOI>
<DOI>10.1152/ajpcell.00047.2013</DOI>
<DOI>10.1152/ajpcell.00070.2013</DOI>
<DOI>10.1152/ajpcell.00096.2013</DOI>
</TocSection>
<TocSection>
<Heading>Corrigendum</Heading>
<DOI>10.1152/ajpcell.zh0-7419-corr.2014</DOI>
</TocSection>
</TOC>
</MetaIssue>
The output which I am getting is:
<!DOCTYPE MetaIssue SYSTEM "http://schema.highwire.org/public/toc/MetaIssue.pubids.dtd">
<MetaIssue volume="306" issue="1">
<Provider>Cadmus</Provider>
<IssueDate>January 1, 2014 </IssueDate>
<PageRange>C1-</PageRange>
<TOC>
<TocSection>
<Heading>Review</Heading>
<DOI>10.1152/ajpcell.00281.2013</DOI>
</TocSection>
<TocSection>
<Heading>CALL FOR PAPERS | Stem Cell Physiology and Pathophysiology</Heading>
<DOI>10.1152/ajpcell.00156.2013</DOI>
</TocSection>
<TocSection>
<Heading>Articles</Heading>
<DOI>10.1152/ajpcell.00130.2013</DOI>
</TocSection>
<TocSection>
<Heading>Corrigendum</Heading>
<DOI>10.1152/ajpcell.zh0-7419-corr.2014</DOI>
</TocSection>
</TOC>
</MetaIssue>
The code I tried is:
#!/usr/bin/perl
use strict;
use warnings;
use LWP::Simple;
my $path1 = $ARGV[0];
open(F6, ">meta_issue.xml");
print "Enter the URL:";
my $url = <STDIN>;
chomp $url;
print "Enter the Volume Number:";
my $vol = <STDIN>;
chomp $vol;
print "Enter the Issue Number:";
my $iss = <STDIN>;
chomp $iss;
my $website_content = get($url);
print F6 "\<\!DOCTYPE MetaIssue SYSTEM \"http://schema.highwire.org/public/toc/MetaIssue.pubids.dtd\">\n";
print F6 "<MetaIssue volume=\"$vol\" issue=\"$iss\">\n";
print F6 "<Provider>Cadmus</Provider>\n";
if ($website_content =~ m#<span class="highwire-cite-metadata-date">(.*?)</span>#s) {
#<span class="highwire-cite-metadata-date">January 1, 2014 </span>
print F6 "<IssueDate>$1</IssueDate>\n"; #<IssueDate>January 1, 2014</IssueDate>
}
if ($website_content =~ m#(<span class="label">:</span>\s?(.*?)(-(.*?))?</span>)#gs) {
#.*?(?!<span class="label">:</span>\s?(.*?)(-(.*?))?</span>)$#gs) #<PageRange>C1-C76</PageRange>
my $first = $2;
print F6 "<PageRange>$2-</PageRange>\n";
}
print F6 "<TOC>\n";
while ($website_content =~ m#<h2 id=".*?" class=".*?">(.*?)</h2>#gs) {
my $h = $1;
print F6 "<TocSection>\n";
print F6 "<Heading>$h</Heading>\n";
if ( $website_content =~ m#(.*?<p><span class="label">DOI:</span>\s?(.*?)\n?</p>\s?</span>\s?\n?</div>.*?)#gs ) {
my $doi = $1;
my $doi1 = $2;
print F6 "<DOI>$doi1</DOI>\n";
print F6 "</TocSection>\n";
}
}
print F6 "</TOC>\n</MetaIssue>\n";
Note: Each <Heading> might have one or more <DOI> values, which I am not able to retrieve
I cannot place the particular <DOI> values under that <Heading>.
I cannot retrieve the last occurrence of the digit from
<span class="label">:</span>\s?(.*?)(-(.*?))?</span>
since there are variation such as </span> c14</span> or <span> c12-c14</span>. So from here I need to grep the last digit i.e c14
I execute the code in cmd as follows;
D:\Code>Perl File name (Enter)
Enter the URl: http://ajpcell.physiology.org/content/306/1
Enter the Volume Number: 306
Enter the Issue Number: 1
UPDATE:
In the URL's :
1) http://ajpendo.physiology.org/content/283/5
2) http://ajpendo.physiology.org/content/280/1
The DOI is not there, so in that case, the output in place of
<DOI>$_</DOI> tag
should be
<ResId type=”publisher-id”>$volume/$issue/$first_page</ResId>
where $first_page is specific to that particular section.
I added "else{} loop" in "sub retrieve_doi()" and also in the "for{} loop" below, but not getting the desired output .
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
use HTML::Parser;
use WWW::Mechanize;
my ($date, $first_page, $last_page, #toc);
sub get_date {
my ($self, $tag, $attr) = #_;
if ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-date' eq $attr->{class}
and not defined $date
) {
$self->handler(text => \&next_text_to_date, 'self, text');
} elsif ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-pages' eq $attr->{class}
) {
if (not defined $first_page) {
$self->handler(text => \&parse_first_page, 'self, text');
} else {
$self->handler(text => \&parse_last_page, 'self, text');
}
} elsif ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-doi' eq $attr->{class}
) {
$self->handler(text => \&retrieve_doi, 'self, text');
} elsif ('div' eq $tag
and $attr->{class}
and $attr->{class} =~ /\bissue-toc-section\b/
) {
$self->handler(text => \&next_text_to_toc, 'self, text');
}
}
sub next_text_to_date {
my ($self, $text) = #_;
$text =~ s/^\s+|\s+$//g;
$date = $text;
$self->handler(text => undef);
}
sub parse_first_page {
my ($self, $text) = #_;
if ($text =~ /([A-Z0-9]+)(?:-[0-9A-Z]+)?/) {
$first_page = $1;
$self->handler(text => undef);
}
}
sub parse_last_page {
my ($self, $text) = #_;
if ($text =~ /(?:[A-Z0-9]+-)?([0-9A-Z]+)/) {
$last_page = $1;
$self->handler(text => undef);
}
}
sub next_text_to_toc {
my ($self, $text) = #_;
push #toc, [$text];
$self->handler(text => undef);
}
sub retrieve_doi {
my ($self, $text) = #_;
if ('DOI:' ne $text)
{
$text =~ s/^\s+|\s+$//g;
push #{ $toc[-1] }, $text;
$self->handler(text => undef);
}
else #UPDATE
{
$text =~ s/^\s+|\s+$//g;
push #{ $toc[-1] }, $text;
$self->handler(text => undef);
}
}
print STDERR 'Enter the URL: ';
chomp(my $url = <>);
my ($volume, $issue) = (split m(/), $url)[-2, -1];
my $p = 'HTML::Parser'->new( api_version => 3,
start_h => [ \&get_date, 'self, tagname, attr' ],
);
my $mech = 'WWW::Mechanize'->new(agent => 'Mozilla');
$mech->get($url);
my $contents = $mech->content;
$p->parse($contents);
$p->eof;
my $toc;
for my $section (#toc) {
$toc .= "<TocSection>\n";
$toc .= "<Heading>".shift(#$section)."</Heading>\n";
$toc .= join q(), map "<DOI>$_</DOI>\n", #$section;
$toc .= join q(), map "<ResId type=”publisher-id”>$volume/$issue/$first_page</ResId>\n", #$section; #UPDATE
$toc .= "</TocSection>\n";
}
open (F6, ">meta_issue_$issue.xml");
print F6 <<"__HTML__";
<!DOCTYPE MetaIssue SYSTEM "http://schema.highwire.org/public/toc/MetaIssue.pubids.dtd">
<MetaIssue volume="$volume" issue="$issue">
<Provider>Cadmus</Provider>
<IssueDate>$date</IssueDate>
<PageRange>$first_page-$last_page</PageRange>
<TOC>
$toc</TOC>
</MetaIssue>
__HTML__
Please let me know how to update the code to get the desired output.

Answer 1

Use a proper module to parse the HTML:
#!/usr/bin/perl
use warnings;
use strict;
use feature qw{ say };
use HTML::Parser;
use WWW::Mechanize;
my ($date, $first_page, $last_page, #toc);
sub get_info {
my ($self, $tag, $attr) = #_;
if ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-date' eq $attr->{class}
and not defined $date
) {
$self->handler(text => \&next_text_to_date, 'self, text');
} elsif ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-pages' eq $attr->{class}
) {
if (not defined $first_page) {
$self->handler(text => \&parse_first_page, 'self, text');
} else {
$self->handler(text => \&parse_last_page, 'self, text');
}
} elsif ('span' eq $tag
and $attr->{class}
and 'highwire-cite-metadata-doi' eq $attr->{class}
) {
$self->handler(text => \&retrieve_doi, 'self, text');
} elsif ('div' eq $tag
and $attr->{class}
and $attr->{class} =~ /\bissue-toc-section\b/
) {
$self->handler(text => \&next_text_to_toc, 'self, text');
}
}
sub next_text_to_date {
my ($self, $text) = #_;
$text =~ s/^\s+|\s+$//g;
$date = $text;
$self->handler(text => undef);
}
sub parse_first_page {
my ($self, $text) = #_;
if ($text =~ /([A-Z0-9]+)(?:-[0-9A-Z]+)?/) {
$first_page = $1;
$self->handler(text => undef);
}
}
sub parse_last_page {
my ($self, $text) = #_;
if ($text =~ /(?:[A-Z0-9]+-)?([0-9A-Z]+)/) {
$last_page = $1;
$self->handler(text => undef);
}
}
sub next_text_to_toc {
my ($self, $text) = #_;
push #toc, [$text];
$self->handler(text => undef);
}
sub retrieve_doi {
my ($self, $text) = #_;
if ('DOI:' ne $text) {
$text =~ s/^\s+|\s+$//g;
push #{ $toc[-1] }, $text;
$self->handler(text => undef);
}
}
print STDERR 'Enter the URL: ';
chomp(my $url = <>);
my ($volume, $issue) = (split m(/), $url)[-2, -1];
my $p = 'HTML::Parser'->new( api_version => 3,
start_h => [ \&get_info, 'self, tagname, attr' ],
);
my $mech = 'WWW::Mechanize'->new(agent => 'Mozilla');
$mech->get($url);
my $contents = $mech->content;
$p->parse($contents);
$p->eof;
my $toc;
for my $section (#toc) {
$toc .= " <TocSection>\n";
$toc .= " <Heading>" . shift(#$section) . "</Heading>\n";
$toc .= join q(), map " <DOI>$_</DOI>\n", #$section;
$toc .= " </TocSection>\n";
}
print << "__HTML__";
<!DOCTYPE MetaIssue SYSTEM "http://schema.highwire.org/public/toc/MetaIssue.pubids.dtd">
<MetaIssue volume="$volume" issue="$issue">
<Provider>Cadmus</Provider>
<IssueDate>$date</IssueDate>
<PageRange>$first_page-$last_page</PageRange>
<TOC>
$toc </TOC>
</MetaIssue>
__HTML__
Basic explanation:
HTML::Parser is callback-based, which means you give it subroutines to run when it encounters a given event in the parsed document. I use a general callback get_info, which searches for various indicators of needed information in the HTML. As we are often interested in something like "nearest text after the given span", it just registers the new callback for the text. For example, when the span with the class highwire-cite-metadata-date is found and date is not yet defined, it registers a new text handler, which would run next_text_to_date. The handler just assigns the text to the $date variable and removes the handler. I'm not sure that's the "correct" way to do it, but in this case at least, it works.
I used WWW::Mechanize in order to be able to specify the User Agent. With the default value of the much simpler LWP::Simple, I wasn't getting the whole HTML.
The output smells of a template. Switching to Template might be a good step forward.