Want to get video name and link from https://bubble.io/videos but I can not find href links, any ideas ?
WebAutomation.LaunchChrome.LaunchChrome Url: $'''https://bubble.io/videos''' WindowState: WebAutomation.BrowserWindowState.Normal ClearCache: False ClearCookies: False WaitForPageToLoadTimeout: 60 Timeout: 60 BrowserInstance=> Browser
LOOP FOREACH CurrentItem IN DataFromWebPage
WebAutomation.ExtractData.ExtractListUsingPagingFromNumberOfPages BrowserInstance: Browser Control: $'''html > body > div:eq(1) > div:eq(3) > div > div:eq(1) > div:eq(0) > div''' ExtractionParameters: {[$'''div > div > div > div:eq(1)''', $'''Own Text''', $''''''] } MaxWebPagesToProcess: 50000 PagerCssSelector: $'''html > body > div:eq(1) > div:eq(3) > div > div:eq(1) > div:eq(1) > div:eq(2)''' PostProcessData: True TimeoutInSeconds: 60 ExtractedData=> DataFromWebPage
WebAutomation.GoToWebPage.GoToWebPage BrowserInstance: Browser Url: DataFromWebPage WaitForPageToLoadTimeout: 60
END
File.WriteToCSVFile.WriteCSV VariableToWrite: DataFromWebPage CSVFile: $'''C:\\Users\\myuser\\Documents\\bubble.csv''' CsvFileEncoding: File.CSVEncoding.UTF8 IncludeColumnNames: False IfFileExists: File.IfFileExists.Overwrite ColumnsSeparator: File.CSVColumnsSeparator.SystemDefault
Related
I use following code to label requests by response times.
if (prev.getTime() > 170 && prev.getTime() < 340) {
prev.setSampleLabel(prev.getSampleLabel() + " > 170")
} else if (prev.getTime() > 340 && prev.getTime() < 4000) {
prev.setSampleLabel(prev.getSampleLabel() + " > 340")
} else if (prev.getTime() > 4000 && prev.getTime() < 8000) {
prev.setSampleLabel(prev.getSampleLabel() + " > 4000")
} else if (prev.getTime() > 8000) {
prev.setSampleLabel(prev.getSampleLabel() + " > 8000")
}
Aggregate report and Summary report contain name of requests in a different order than original one in Thread group. Total number of samples per request is not visible in this way.
JMeter's Aggregate Report and Summary Report listeners always store the Sample Results in their execution order and JMeter executes Samplers upside down (or according to the Logic Controllers)
As you can see, Sampler 4 is the first one because it has been executed first, however it is possible to sort requests by label by clicking the column header since JMeter 3.2):
Another option to sort requests by label is generating HTML Reporting Dashboard
Hello this is my first ever post ,
So I am trying to make a Web Spider that will follow the links in invia.cz and copy all the titles from the hotel.
import scrapy
y=0
class invia(scrapy.Spider):
name = 'Kreta'
start_urls = ['https://dovolena.invia.cz/?d_start_from=13.01.2017&sort=nl_sell&page=1']
def parse(self, response):
for x in range (1, 9):
yield {
'titles':response.css("#main > div > div > div > div.col.col-content > div.product-list > div > ul > li:nth-child(%d)>div.head>h2>a>span.name::text"%(x)).extract() ,
}
if (response.css('#main > div > div > div > div.col.col-content >
div.product-list > div > p >
a.next').extract_first()):
y=y+1
go = ["https://dovolena.invia.cz/d_start_from=13.01.2017&sort=nl_sell&page=%d" % y]
print go
yield scrapy.Request(
response.urljoin(go),
callback=self.parse
)
In this website pages are loading with AJAX so I change the value of the URL manually, incremented by one only if the next button appears in the page.
In the scrapy shell when I test if the button appears and the conditions everything is good but when I start the spider it only crawls the first page.
It's my first spider ever so thanks in advance.
Also the errol log Error Log1 Error Log
Your usage of "global" y variable is not only peculiar but won't work either
You're using y to calculate how many times parse was called. Ideally you don't want to access anything outside of the functions scope, so you can achieve the same thing with using request.meta attribute:
def parse(self, response):
y = response.meta.get('index', 1) # default is page 1
y += 1
# ...
#next page
url = 'http://example.com/?p={}'.format(y)
yield Request(url, self.parse, meta={'index':y})
Regarding your pagination issue, your next page url css selector is incorrect since the <a> node you're selecting doesn't have a absolute href attached to it, also this issue makes your y issue obsolete. To solve this try:
def parse(self, response):
next_page = response.css("a.next::attr(data-page)").extract_first()
# replace "page=1" part of the url with next number
url = re.sub('page=\d+', 'page=' + next_page, response.url)
yield Request(url, self.parse, meta={'index':y})
EDIT: Here's the whole working spider:
import scrapy
import re
class InviaSpider(scrapy.Spider):
name = 'invia'
start_urls = ['https://dovolena.invia.cz/?d_start_from=13.01.2017&sort=nl_sell&page=1']
def parse(self, response):
names = response.css('span.name::text').extract()
for name in names:
yield {'name': name}
# next page
next_page = response.css("a.next::attr(data-page)").extract_first()
url = re.sub('page=\d+', 'page=' + next_page, response.url)
yield scrapy.Request(url, self.parse)
im trying to scrape this page:
http://binpar.caicyt.gov.ar/cgi-bin/koha/opac-detail.pl?biblionumber=98723
with this code:
def parse_web9(self, response): #Conicet!!
for publication in response.css('div#wrap > div.main > div.container-fluid > div.row-fluid > div.span9 > div#catalogue_detail_biblio > div.record'):
pubtitle = publication.xpath('./h1[#class="title"]/text()').extract_first()
author = publication.xpath('./span[#class="results_summary publisher"]/span/span/a/text()').extract()
isxn = publication.xpath('./span[#class="results_summary issn"]/span/text()').re(r'\d+-\d+')
yield{
'titulo_publicacion': pubtitle,
'anio_publicacion': None,
'isbn': isxn,
'nombre_autor': author,
'url_link' : None
}
But I 'm getting only the title of the publication, I'm not sure why.
Cheers!
You should get the inner fields by property attributes:
$ scrapy shell http://binpar.caicyt.gov.ar/cgi-bin/koha/opac-detail.pl?biblionumber=98723
>>> for publication in response.css('div#wrap > div.main > div.container-fluid > div.row-fluid > div.span9 > div#catalogue_detail_biblio > div.record'):
... author = publication.css("span[property=contributor] span[property=name]::text").extract_first()
... title = publication.css("h1[property=name]::text").extract_first()
... issn = publication.css("span[property=issn]::text").extract_first()
... print(author, title, issn)
...
(u'Asociaci\xf3n Filat\xe9lica de la Rep\xfablica Argentina', u'AFRA, bolet\xedn informativo. ', u'0001-1193.')
PHP Version 5.3.3-1 Ubuntu 10.10 Apache 2.2
Ckeditor 3.6.1
I can edit and save but the web page I am editing does not update ? The edited text appears in a new window. I want the web page I am editing to be updated.
ckeditor.js, test.html and posteddata.php are all in the same directory /var/www/
test.html
< head>
< title>Test Page < /title >
< meta http-equiv="content-type" content="text/html; charset=utf-8"/ >
< script type="text/javascript" src="ckeditor.js">< /script >
< /head >
< body >
< form action="posteddata.php" method="post" >
< textarea id="editor1" name="editor1" >
<p>Your text goes here</p>
< /textarea>
< script type="text/javascript" >
window.onload = function()
{CKEDITOR.replace( 'editor1' );};
< /script>
< input type="submit" value="Submit"/ >
< /form>
< /body>
< /html>
posteddata.php
< ?php
if ( isset( $_POST ) )
$postArray = &$_POST ; // 4.1.0 or later, use $_POST
else
$postArray = &$HTTP_POST_VARS ; // prior to 4.1.0, use HTTP_POST_VARS
foreach ( $postArray as $sForm => $value )
{
if ( get_magic_quotes_gpc() )
$postedValue = htmlspecialchars( stripslashes( $value ) ) ;
else
$postedValue = htmlspecialchars( $value ) ; ?>
< tr>
< th style="vertical-align: top"><?php echo htmlspecialchars($sForm); ?>
< /th>
< td><pre class="samples"><?php echo $postedValue?></pre></td>
< /tr>
< ?php }
?>
All your code does is print out what you've just typed in. It doesn't save it anywhere.
Probably the simplest way is to store the changes in a database and then load them each time.
Here's a tutorial; worth reading if you want to do anything useful in PHP: http://www.w3schools.com/php/php_mysql_intro.asp
There is a "save" plugin that submits the form. Get the submitted form and save it in your DB or on your file.
I'm trying to include multiple blockquotes using markdown, but I'm stuck on the syntax for closing out the initial blockquote and starting a new one instead of continuing in the current blockquote and adding additional paragraphs...
=== Current syntax ===
> Review1
> -- <cite>Person1</cite>
> Review2
> -- <cite>Person2</cite>
=== Current result ===
<blockquote>
<p>Review1
-- <cite>Person1</cite></p>
<p>Review2
-- <cite>Person2</cite></p>
</blockquote>
=== Wanted result ===
<blockquote>
<p>Review1
-- <cite>Person1</cite></p>
</blockquote>
<blockquote>
<p>Review2
-- <cite>Person2</cite></p>
</blockquote>
Put in a comment
> Review1
> -- <cite>Person1</cite>
<!>
> Review2
> -- <cite>Person2</cite>
or a manual line break
> Review1
> -- <cite>Person1</cite>
<br>
> Review2
> -- <cite>Person2</cite>
Result
Review1
-- Person1
Review2
-- Person2