I want to know about the working of HTML links parser preprocessor that how does it work how to retrieve all links and all other elements that are present in the HTML response. As far as I have checked on each blog it is written that .* will extract all links but what about other elements what if I don't want links and I want to test with other elements like I want to fetch image source or I want to play with drop down or radio button available in response . How can I extract those?
Is there going to be any other regex for that or the same one .*?
As per documentation
This modifier parses HTML response from the server and extracts links and forms
so there are 2 main use cases for the HTML link parser:
site links crawling (spidering)
submitting random data into a form
In both cases you need to provide a Perl-5 Compatible Regular Expression in order to limit crawling to current domain or narrow down options selection.
If you need to fetch image(s) source(s) the best option would be using CSS/JQuery Extractor configured like:
Selector: img
Attribute: src
Related
So I would go to an instagram account, say, https://www.instagram.com/foodie/ to copy its xpath that gives me the number of posts, number of followers, and number of following.
I would then run the command this command on a scrapy shell:
response.xpath('//*[#id="react-root"]/section/main/article/header/section/ul')
to grab the elements on that list but scrapy keeps returning an empty list. Any thoughts on what I'm doing wrong here? Thanks in advance!
This site is a Single Page Application (SPA) so it's javascript that render DOM is not rendered yet at the time your downloader working.
When you use view(response) the javascript that your downloader collected can continue render by your browser, so you can see the page with DOM rendered (but can't interacting with Site API). You can look at your downloaded content via response.text and saw that!
In this case, you can apply selenium + phantomjs to making a rendered page for your spider!
Another trick: You can use regular expression to select the JSON part of Script, parse it to JSON obj and select your correspond attribute value (number of post, following, ...) from script!
Is it possible to get list of URL's / links available in a website by providing base URL as an input. Using nightwatch, if i pass base URL it should give all hyperlinks, child pages links etc. using Nightwatch. If its possible, how we can achieve it.
Not sure Nightwatch is the right choice for what you are trying to achieve. Maybe you can let us know what you want to do with the list of urls you fetch from the page?
You can set the URL via page object configuration and when the page loads (page.navigate()) you can return all link elements. Something like:
browser
.elements("css selector", "a", result => {
// Do something with 'result' that will contain all the elements matching "a"
})
But I recommend you read more about Nightwatch to determine if that is the right tool for your needs.
More about Nightwatch: http://nightwatchjs.org/gettingstarted
More about Page objects: https://martinfowler.com/bliki/PageObject.html
I have a project that require me to do a auto complete search using CI and YUI.
User will enter search keywords in textfield 1 and the result will shown in textfield 2..
YUI is client-side JavaScript. What you should do is read some documentation on autocomplete first, and on CodeIgniter second and then you could start building your solution.
Basically, you have to make a few things.
For one, you should create a client-side page with what is basically an <input> or <textarea> tag and a div for your results and hook a YUI autocomplete handler to that field. To be able to use YUI, you first need to include the YUI script:
<script src="http://yui.yahooapis.com/3.4.1/build/yui/yui-min.js"></script>
And you also need to load the autocomplete module and configure it:
YUI().use('autocomplete', function (Y) {
// enter code here
}
Second, you need to build a server-side script with CI that will listen for queries, do database or whatever searches and return JSON or some other format of results.
You'll then give the URL of this CI query web script to YUI AutoComplete configuration and YUI will do the AJAX and stuff for you.
Those are very general steps, but your question is also pretty generic. If you ask more specific questions, you'll get better answers (or better, just search and read the documentation).
in this address i am trying to scrape a tage (that is Larg price which is bold red one)
i use LIBXML 2.2
when i try to extract the tag through this XPATH
//*[#class='priceLarge']
it works!
but to make queries easier i would like to use FireBug on Firefox.
Using FireBug it gives me this XPath
/html/body/div[2]/form/table[3]/tbody/tr/td/div/table/tbody/tr[2]/td[2]/span/b
using this Xpath it does not work, seems this one does not give a complete query. how can i modify this XPath to scrape the item ?
Firefox and other browsers generate tbody tags in HTML.
In fact, the tbody is probably not there, so you can remove it in your XPath. (/html/body/div[2]/form/table[3]/tr/td/div/table/tr[2]/td[2]/span/b) You can test this by just saving the HTML from your application and viewing it in a text editor.
Since it seems the intent is to pull information from a web page however, your application will probably be more resistant to changes in the web page if you use XPath less dependent on the tree structure (i.e. //b[#class='priceLarge']).
EDIT: It seems that in addition to the tbody problem, Firefox is rendering the div (ID: divsinglecolumnminwidth) element as containing the form element (ID: handleBuy).
Looking at the html with an XML editor shows that the form element is a sibling of that div element, so the expression should start with /html/body/form/table[3].
One tool, among many others, to test your XPath expressions is HAP Testbed.
I have a text area in which I am trying to add youtube embed code and other HTML tags. $this->input->post is converting the <iframe> tags to < and > respectively but not the <h1> and <h2> tags.
Any idea how I can store these values?
If you only have a small number of forms that you need to allow iframes in, I would just write a function to restore the iframe (while validating that it's a valid YouTube embed code).
You can also turn off global_xss_filtering in your config (or not implement it if you're using it), but that's not the ideal solution (turning off all of your security to get one thing to work is generally a horrible idea).
$config['global_xss_filtering'] = FALSE;
To see all of the tags that get filtered out, look in the CI_Input class and search for the '$naughty' variable. You'll see a pipe-delimited list (don't change anything in this class).
Why don't you avoid CIs auto sanitizing and use something like htmlspecialchars($_POST['var']); ? Or make a helper function for sanitizing youtube urls...
Or you could either just ask for the video ID code or parse the code from what you are getting.
This would let you use both the URL or the embed code.
Also storing just the ID takes less space in you database, and you could write a helper function to output the embed code/url.
In this case, use $_POST instead of $this->input->post to get the original text area value, and then use HTML Purifier to clean the contents without losing the <iframe> tag you want.
You will need to check HTML Purifier documentation for details. Please, check this specific documentation page about "Embedding YouTube Videos".