I am new to Scrapy framework. And I am just learning Python language. I need to scrape data from the online shop. But there is pagination with ajax on some pages. I have watched some tutorials about how to add pagination in a scrapy spider, but it's not clear for me. I know that I need to reproduce requests. I found out these requests in Dev tools---Network tab(please see the photo).
Link to the shop
photo
I don't know which (Scrapy framework) methods or requests I can use. Could you help me to solve this issue please?
Related
I'm really confused.
I just desire to implement ajax in my wordpress site: clicking on each link of the menu or on each post, I'd like to show dinamycally the content in the main wrapper of the website (and at the same time I'd like to see the url changing everytime for each page or post).
I have made a research through Google and I found a multitude of tutorials or informations but each one of them is (for me) like incomplete.
Can you suggest me a good tutorial?
Or can you explain me what are the best solutions?
I read somewhere of "Ajaxify", a plugin, can it help me?
Be patient, I have always had some difficulties with ajax and in wordpress things seems to become even more complex.
I guess this tutorial can help you: http://www.designlunatic.com/2011/06/dynamically-load-wordpress-posts-with-jquery/
I am trying to create a one-page WordPress website, something like the ones you sometimes see in ThemeForest's WP section: the whole website is a long page that has everything in one place, from about us, to portfolio, to some blog posts, to contacts.
Placing all things on one page is not difficult. But when I started thinking about how to present individual posts and pages, I realised that I probably need a general way of getting posts' data via AJAX, and create new blocks with JS. How should I go about this? I suppose this was done before, but I struggle to find something this specific on Codex or a tutorial with best practices.
Any advice or link will be greatly appreciated.
You could use a plugin such as jQuery Easytabs, download it here, that has a built-in Ajax component.
I've found that the easiest way is to just get all content to load into the divs ahead of time, vs. trying to load all pages through Ajax. However, appending something like '?ajax/ajax' to the end of your urls through the Easytabs plugin is one option that I have successfully used in the past.
If you decide to use the easytabs functionality, there is ample documentation on the page that I linked to.
I want to fetch linkedin group or company post updates via ajax to be displayed on a webpage. To be specific I'm looking for something like this plugin named jtwitter which fetches twitter posts. Now I just want a simple updates feed so I was wondering if there is any jquery plugin that allows to do so.
I was searching for similar solution.
Here's a jQuery Socialist plugin which can grab latest posts and updates from popular social networks:
https://github.com/iatek/jquery-socialist
I didn't try it yet but it seems that it fit your needs.
Hope this helps :)
My Problem:
My client site which displays more products and it adds more page load/weight. So i decided to use ajax more products loading and it works well. But here it affects the seo - and no products or deals has been indexed(Even i suggest the client to submit product via googlebase but client doesnot like that idea and he wants direct google crawling into site also he wants less time page load).
Question:
Can we identify the googlebot crawling request to the server or mozila like browser user agent request to the site(server).
Suggestion I have
I tried to identify user agent from requests but that doesnot working(or i might missing something?) Please anyone have correct solution for this problem to reduce the page load time using ajax and get googlebot also to crawl the website.
You should just search stackoverflow for "Google AJAX SEO". There are a number of questions around this.
In short, Google has a specification to make AJAX sites crawlable: https://developers.google.com/webmasters/ajax-crawling/docs/getting-started?hl=sv-SE
You can also look into PushState as an SEO option as well.
One tactic that is used to solve this is to harness the pagination function of whatever framework or CMS you are using. You load one page of content and display pagination links in your view then use JavaScript to hide the pagination links and fetch the content of the linked pagination page via Ajax and append it to the current page. Take a look at how infinite-scroll works for inspiration:
http://www.infinite-scroll.com/
Basically you need to be at least loading links to pages that contain the other content so that search engines can crawl the content, but you can hide these links for the users who have JavaScript Enabled.
But to better answer your question, it is possible to redirect robots using htaccess:
redirect all bots using htaccess apache
But it is better SEO, as far as I understand it, to have the content or links to it, actually available on the page.
How to get started with CodeIgniter and Google Maps API.
I've got a CodeIgniter library specifically for this purpose. You can find more information about it here:
http://biostall.com/codeigniter-google-maps-v3-api-library
A few demos of what can be acheived using the library:
http://biostall.com/demos/google-maps-v3-api-codeigniter-library/
Give me a shout if you have any questions or need any help :)
Enjoy!
CodeIgniter is a backend (server-side) framework. Google Maps API is a frontend (client-side) library. There isn't really any specialized knowledge you need to know about CodeIgniter to get Google Maps working, other than being able to render a view. The question is somewhat flawed in that you assume one depends on the other, when in reality, they are very much decoupled.
The question is similar to how do i get jquery working with codeigniter?...
Reading the CodeIgniter documentation would be a start to understanding how to echo HTML and JavaScript to the browser. The rest is almost entirely client-side JavaScript programming.