I'm working on a project that calls an e-commerce website to get some information and does analysis on them.
I wrote the following code to get the information:
use Illuminate\Support\Facades\Http;
class TheController extends Controller {
public function getProductAndCompetitors() {
public function getInformation() {
$url = 'https://www.noon.com/_svc/catalog/api/v3/search?sku=N29905443A';
$response = HTTP::get($url);
dd($response);
}
}
The problem:
the code is working on localhost but returning 403 error when I deployed it to a server.
Additional information:
the external website is based in KSA (no idea where the server is)
when I copy and paste the url in the browser it returns 200
the code is working on a shared hosting server in US
the code is working on localhost in KSA
the code is not working on VPS server is Frankfort
I tried to use GuzzleHttp\Client instead and add some headers but this didn't work as well
I cannot comment due to low reputation, but I need to see the logs. Check your server and let me know what it says.
This could range from a variety of reasons, and I have encountered to be common the issue of the "missing key" (php artisan key:generate)
maybe issue regarding they might not like request being send from server as curl, they allow it on localhost but they sometimes block it on requests from a server.
try{
$url = 'https://www.noon.com/_svc/catalog/api/v3/search?sku=N29905443A';
$response = Http::withHeaders([
'User-Agent' => 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'
])->get($url);
dd($response->body());
} catch(\Illuminate\Http\Client\RequestException $e){
// Log your errors
}
Related
I'm working on an app using Visual Studio 2015 Cordova tools on Windows 8.1. Target is also Windows 8.1.
The app is caching HTTP GET request. So the second GET request to the same resource returns a cached response. I have tested after disabling the network adapter and I still get a response with the cached results.
I am using jsforce libray to connect to salesforce.com. I know I can add a timestamp on the url but I would like to find fix not a work around.
Any ideas?
[UPDATE]
Issue is not related to jsforce as it works well on Android. The error is specific to Windows 8.1 and cordova.
As suggested in the question, timestamping the url helps: I used:
var url = "https://api.myurl.com/" + param1 + "?" + new Date().getTime()
I wrote a simple library to add caching to REST requests for Cordova: https://github.com/glauber-md/mobile-simple-web-call#using-this-library .
The library will use a local database (sqlite) to fetch server data and cache it locally where applicable (e.g. HTTP GET requests with cache-related headers).
Once it receives a 304 Response, it will use the cached data.
To send a GET request, you'd use:
wscall.get(
'http://myserver.org/users/1234',
// (Optional) query strings
null,
function(responseData) {
// Do something when the response is successful
},
function(error) {
// Do something when an error happens
}
);
Then the data would come from remote server or local cache depending of the HTTP server response.
Maybe it will help you.
I'm testing on Windows, trying to simulate POST requests (with different form variables) for load testing. I have tried all kinds of load testing software but failed to get it working.
For GET requests, I know I can just put parameters behind the url
http://www.example.com?id=yyy&t=zzz
But how do I simulate a POST request?
I have a chrome REST Client but I do not know what to put in the headers and data.
Here's what I've tried so far:
class Program
{
static void Main(string[] args)
{
string viewstateid = "/wEPDwUKLTY3NjEyMzE4NWRkK4DxZpjTmZg/RGCS2s13vkEWmwWiEE6v+XrYoWVuxeg=";
string eventid ="/wEdAAoSjOGPZYAAeKGjkZOhQ+aKHfOfr91+YI2XVhP1c/pGR96FYSfo5JULYVvfQ61/Uw4pNGL67qcLo0vAZTfi8zd7jfuWZzOhk6V/gFA/hhJU2fx7PQKw+iST15SoB1LqJ4UpaL7786dp6laCBt9ubQNrfzeO+rrTK8MaO2KNxeFaDhrQ0hxxv9lBZnM1SHtoODXsNUYlOeO/kawcn9fX0BpWN7Brh7U3BIQTZwMNkOzIy+rv+Sj8XkEEA9HaBwlaEjg=";
string username = "user1";
string password = "ttee";
string loginbutton = "Log In";
string URLAuth = "http://localhost/login.aspx";
string postString = string.Format("VIEWSTATE={0}&EVENTVALIDATION={1}&LoginUser_UserName={2}&LoginUser_Password={3}&LoginUser_LoginButton={4}",viewstateid,eventid, username, password,realm,otp,loginbutton);
const string contentType = "application/x-www-form-urlencoded";
System.Net.ServicePointManager.Expect100Continue = false;
CookieContainer cookies = new CookieContainer();
HttpWebRequest webRequest = WebRequest.Create(URLAuth) as HttpWebRequest;
webRequest.Method = "POST";
webRequest.ContentType = contentType;
webRequest.CookieContainer = cookies;
webRequest.ContentLength = postString.Length;
webRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.1) Gecko/2008070208 Firefox/3.0.1";
webRequest.Accept = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
webRequest.Referer = "http://localhost/login.aspx";
StreamWriter requestWriter = new StreamWriter(webRequest.GetRequestStream());
requestWriter.Write(postString);
requestWriter.Close();
StreamReader responseReader = new StreamReader(webRequest.GetResponse().GetResponseStream());
string responseData = responseReader.ReadToEnd();
Console.WriteLine(responseData);
responseReader.Close();
webRequest.GetResponse().Close();
}
}
It would be helpful if you provided more information - e.g. what OS your using, what you want to accomplish, etc. But, generally speaking cURL is a very powerful command-line tool I frequently use (in linux) for imitating HTML requests:
For example:
curl --data "post1=value1&post2=value2&etc=valetc" http://host/resource
OR, for a RESTful API:
curl -X POST -d #file http://host/resource
You can check out more information here-> http://curl.haxx.se/
EDITs:
OK. So basically you're looking to stress test your REST server? Then cURL really isn't helpful unless you want to write your own load-testing program, even then sockets would be the way to go. I would suggest you check out Gatling. The Gatling documentation explains how to set up the tool, and from there your can run all kinds of GET, POST, PUT and DELETE requests.
Unfortunately, short of writing your own program - i.e. spawning a whole bunch of threads and inundating your REST server with different types of requests - you really have to rely on a stress/load-testing toolkit. Just using a REST client to send requests isn't going to put much stress on your server.
More EDITs
So in order to simulate a post request on a socket, you basically have to build the initial socket connection with the server. I am not a C# guy, so I can't tell you exactly how to do that; I'm sure there are 1001 C# socket tutorials on the web. With most RESTful APIs you usually need to provide a URI to tell the server what to do. For example, let's say your API manages a library, and you are using a POST request to tell the server to update information about a book with an id of '34'. Your URI might be
http://localhost/library/book/34
Therefore, you should open a connection to localhost on port 80 (or 8080, or whatever port your server is on), and pass along an HTML request header. Going with the library example above, your request header might look as follows:
POST library/book/34 HTTP/1.0\r\n
X-Requested-With: XMLHttpRequest\r\n
Content-Type: text/html\r\n
Referer: localhost\r\n
Content-length: 36\r\n\r\n
title=Learning+REST&author=Some+Name
From here, the server should shoot back a response header, followed by whatever the API is programed to tell the client - usually something to say the POST succeeded or failed. To stress test your API, you should essentially do this over and over again by creating a threaded process.
Also, if you are posting JSON data, you will have to alter your header and content accordingly. Frankly, if you are looking to do this quick and clean, I would suggest using python (or perl) which has several libraries for creating POST, PUT, GET and DELETE request, as well as POSTing and PUTing JSON data. Otherwise, you might end up doing more programming than stress testing. Hope this helps!
Postman is the best application to test your APIs !
You can import or export your routes and let him remember all your body requests ! :)
EDIT : This comment is 5 yea's old and deprecated :D
Here's the new Postman App :
https://www.postman.com/
Simple way is to use curl from command-line, for example:
DATA="foo=bar&baz=qux"
curl --data "$DATA" --request POST --header "Content-Type:application/x-www-form-urlencoded" http://example.com/api/callback | python -m json.tool
or here is example how to send raw POST request using Bash shell (JSON request):
exec 3<> /dev/tcp/example.com/80
DATA='{"email": "foo#example.com"}'
LEN=$(printf "$DATA" | wc -c)
cat >&3 << EOF
POST /api/retrieveInfo HTTP/1.1
Host: example.com
User-Agent: Bash
Accept: */*
Content-Type:application/json
Content-Length: $LEN
Connection: close
$DATA
EOF
# Read response.
while read line <&3; do
echo $line
done
This should help if you need a publicly exposed website but you're on a dev pc. Also to answer (I can't comment yet): "How do I post to an internal only running development server with this? – stryba "
NGROK creates a secure public URL to a local webserver on your development machine (Permanent URLs available for a fee, temporary for free).
1) Run ngrok.exe to open command line (on desktop)
2) Type ngrok.exe http 80 to start a tunnel,
3) test by browsing to the displayed web address which will forward and display the local default 80 page on your dev pc
Then use some of the tools recommended above to POST to your ngrok site ('https://xxxxxx.ngrok.io') to test your local code.
https://ngrok.com/ ngrok
Dont forget to add user agent since some server will block request if there's no server agent..(you would get Forbidden resource response) example :
curl -X POST -A 'Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0' -d "field=acaca&name=afadxx" https://example.com
I have a url. When I try to access it programmatically, the backend server fails (I don't run the server):
import requests
r = requests.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
r.status_code # 200
print r.content
When I look at the content, it's an error page, though the status code is 200. If you click the link, it'll work in your browser -- you'll get a PDF -- which is what I expect in r.content. So it works in my browser, but fails in Requests.
To diagnose, I'm trying to eliminate differences between my browser and Requests library. So far I've:
Disabled Javascript
Disabled (and deleted) cookies
Set the User-Agent to be the same in each
But I can't get the thing to work properly in Requests or fail in my browser due to disabling something. Can somebody with a better idea of browser-magic help me diagnose and solve this?
Does the request work in Chrome? If so, you can open the web inspector and right-click the request to copy it as a curl command. Then you'll have access to all the headers, params, and request body, which you can play around with to see which are triggering the failure you're seeing with the requests library.
You're probably running into a server that discriminates based on User-Agent. This works:
import requests
S = requests.Session()
S.headers.update({'User-Agent': 'Mozilla/5.0 (compatible; MSIE 9.0; Windows NT 6.1; Trident/5.0)'})
r = S.get('http://www.courts.wa.gov/index.cfm?fa=controller.managefiles&filePath=Opinions&fileName=875146.pdf')
with open('dl.pdf', 'wb') as f:
f.write(r.content)
I have a Firefox quicksearch bookmark that runs a Maxmind query. This worked until recently. I type 'ip 82.176.230.15' (for example) into the URL bar and it queries Maxmind to retrieve the location of the IP:
http://www.maxmind.com/app/locate_demo_ip?ips=82.176.230.15
Within the past week, for reasons unknown, I now get a 403/Forbidden error when I try to access Maxmind.
"You don't have permission to access /app/locate_demo_ip on this server"
Strangely, the same URL is accessible in Chrome and Safari. I can also access the same URL with Firefox, Chrome, or Safari on my Mac.
I've deleted all cookies, disabled all addons, and still can't get it to work. Any idea what could be happening? I know that the 403 has to come from the server, so I don't know why it would work in other browsers. And it's been going on for days, definitely not some glitch on their server.
Get an HTTP debugger like firebug or fiddler (not sure that will work with FireFox, but probably if you set it up right)
Look at the difference between using your quick bookmark and just typing the URL. The server could return 403 whenever it feels like -- see if there's any difference, and what it is.
I recently had the same issue and was able to fix it.
In my case the problem was in headers that Mozilla sent.
Particularly it was because of header:
"User-Agent": "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:100.0) Gecko/20100101 Firefox/100.0"
What makes web-site refuse connection is this part of string "(X11; Ubuntu; Linux x86_64; rv:100.0)" and i have no idea why.
I found a nice solution, you can change Mozilla settings to include other browsers in this header (Chrome and Safari) and it could make sites with this problem works.
Here is how to do it:
Type about:config into the URL bar. Press Enter.
Create a new entry with key=general.useragent.override and add this string there Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.5005.115 Safari/537.36. I found that Google Chrome uses this string as User-Agent header probably to prevent such issues. So you should see something like:
Now save this settings and go reload your page, it should work now
My code at: http://www.mgxvideo.com/mgxcopy-dev/get_browser.php, returns Opera when I run IE8. My source is:
<?php
$browser = get_browser(null, true);
echo $browser['browser'];
?>
It doesn't. The get_browser() function is making educated (but ill-informed) guesses about which browser the user-agent is running. Your browser capabilities file is likely outdated, probably because it was made before IE8 was released. Update it here.
The real IE8 user-agent string looks something like this:
Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.0)
You should use a lower-level tool, like a packet trace or server logging or a header dump to see what is being sent.