To bypass referral check - html-content-extraction

Is there any way to bypass the referral check applied by some site in order to avoid there data from being extracted. Like if you follow this link!
You will get Access Denied Error. However , if you just go this link!, it takes you to home page and on filling on any quote say ABAN , it follows exactly the same GET request as the link used above.
Can anyone please suggest me some solution ??

Set your referrer to the correct value. You can spoof the value to anything you want programatically or by visiting the correct url before visiting the target url.

You can use a tool like wget which allows you to set your own referer value. This won't work on sites which expect more than that as their protection value (and anyone interested in protecting the delivery of their content might also look for a cookie or ip address as well).

Related

CakePHP Completely Random 403 AJAX Errors

I using CakePHP with Backbone.JS, I set up a controller just to give me a JSON output for getting my data, e.g. client names etc, to pass into each Backbone model.
This was all working, or appeared to be, however, it seems that it now gives me some random 403 errors when the page / from is saved or reloaded. But I have no idea why? If it can access it to start with, and does, then why would it not have access after a save or reload?
I have tried, $this->Auth->allow and it dose appear to fix the problem but this data is or could be important and need it not to be access my everybody who might guest at my access path.
Now I have read a number of articles on her, most point to read/write access on the files your accessing, but in my case its just a path /XXXX/XXXXX/myjson/clients For example.
Now I can post my code, if needed, but I am not sure what the problem is, is this a CakePHP issue or is Backbone not requesting the data right?
Please be aware that I am dyslexic, please be kind about my question, if I have not explained myself right. Then please be me some time to re-word / edit my post.
Thanks,
For any one else looking at this, I had added autoRegenerate to the Configure Write Session. For some reason it looks like CakePHP was taking to long to regenerate a new cookie and request my information at the same time.

Google Custom Search API- Get results specific user location

As in normal Google search page, Google returns me results specific to my location. As i am considering/assuming for this, it gets my ip address my request, find uses location for it and returns the results.
I am writing the proxy server for search engines. So from user browser i get requests for some search terms and server returns search results.
I am curious to know, if from Custom Search API can i get the such results based on some IP address i provide or anything like that, so that i can mock the user location in my proxy server and pretend the Google Custom Search to be that user and get the location specific results
PS.I am new to this search engine api world, so please understand me.
Yes, you can, but just for the country, not the exact location.
Here is the docs for parameters of a query request:
https://developers.google.com/custom-search/json-api/v1/reference/cse/list
Look at the 'gl' parameter.
I'm not aware of any way to specify the location more precisely then this.
After these many years I'm not sure if you still need it, but I found it hard to find a helpful answear, so here's the solution for whoever needs it.
You can achieve that behavior with the "?gl=" query param as mentioned here, and pass the value as one of the country codes.
Worked for me, hope it helps someone!

How to form complete URLs in Ruby

I'm trying to create a small system where people can fill in info, including their web site address. Only, some people will enter "www.example.com", some will enter "http://www.example.com", and some will enter "example.com".
What is the most clean way to:
Validate al URLs, in the sense that they all are including http:// so when I click on it I get redirected correctly, and also
For the visible text on the link, what's the easiest way to get the version without http://?
I could of course go crazy with some conditional text replacements, but there has to be a nicer way to do this.
I will to this way that I check that address begins with http(s):// if not then add it. Don't do anything with example.com and www.example.com because it can destroy whole address.
So you can write:
url = 'http://' + url unless url =~ /\Ahttps?:\/\//
I'd recommend using a gem for this -- there's no reason to try to write all this yourself.
Here's one that may fit your needs. There may be others, but this one seems useful and has a decent following:
https://github.com/conickal/validates_url_format_of

is it possible to run multiple websites from the same URL?

i'm in the process of adding a US site to my current UK site. I'd like to do this as transaprently as possible so that we don't lose any traffic to existing links. We're currently running this under version 1.4.1.1 of Magento on a shared hosting setup.
The new website (US) will be essentially the same as the current (UK) site, but with US Dollar pricing instead of Pound Sterling.
We currently have a GeoIP setup whereby visitors are redirected to either UK or US site whilst utulising the same URL. This essentially means that we have switch statements in our index.php to indicate what run code to use.
Here's my question:
what's the best way of selecting/overriding the GeoIP selection via the standard store switcher selector dropbox? Both websites are being populated in the dropbox, however, since both are utilising the same URL (www.example.com/boutique) the default one is the only one that's being selected.
I've also tried the &_store= as well as the &_website= arguments with no success.
Any ideas? are URL rewrites in .htaccess the answer? if so, any ideas as what to use?
P.S. this is the method that's pretty much being followed however my aim is to let users override their location-specific website (e.g. US) if necessary:http://www.magentocommerce.com/wiki/4_-_themes_and_template_customization/navigation/multiple-website-setup#multiple_website_setup_for_useuuk_storespricing
Have you tried using a getUrl() method to build the store arguments for you? It can help clear up those little misunderstandings, for example I'm pretty sure the store parameter is supposed to have three underscores but cannot really remember so I use the function instead.
The best way to over-ride is to have a little php program, e.g. 'countries.php' that sets a cookie depending on the country code that you choose or 'auto' to test regular geoip. Then in your index.php have an 'if cookie then use cookie code else use geoip code'. Naturally the cookie can only be set by your test program.
And yes, you only need set 'website' not 'store'. There is no benefit in your US customers being able to see your UK prices (and vice-versa) so don't even bother with setting up a frontend drop-down. Or, if you really want, you can have rest-of-the-world customers choose their currency/website and put your own cookie-setting code in the header for them, with a couple of nice flag icons.

Redirect on certain URLs

Hi I have a problem where I'm setting up an internet Kiosk in a public place and when a user goes to a certain URL I want it to redirect to another particular URL.
For example I want it setup so that if a user goes to www.example.com/step1 I want the browser to automatically go to www.example.com/step2
The only restriction here is that it has to work on Windows due to hardware limitations.
Does anybody know how I could do this?
Thanks
A couple of ways to do it:
Implement a Browser Helper Object, catch the BeforeNavigate event, cancel the navigation and direct it somewhere else.
Use a specialized proxy server that responds to a request for the first URL by returning a redirect to the second, and passes all other requests through.
You could try to modify the Hosts file. In Windows, I think it is found in WINDOWS/system32/drivers/.
It can be used to redirect a request for one IP address to another.
Hope that's useful.

Resources