CURLE_RECV_ERROR with PayPal API - debugging

I've developed an application that connects to the PayPal API with libcurl, which I use through the OCurl bindings for OCaml from a process running on a Debian server. The code always works when inside the Paypal sandbox (endpoint https://api-3t.sandbox.paypal.com/nvp) but never works when connecting to the actual Paypal servers (endpoint https://api-3t.paypal.com/nvp).
Libcurl always returns CURLE_RECV_ERROR. The general consensus is that this error happens when there is a network problem, so I have investigated that.
I ran the exact same request with the command-line curl tool from the exact same server, using the exact same process uid/gid, and it consistently works. Tracking the transfers with tcpdump does not reveal any significant difference in the structure of transactions made by the working command-line curl and the non-working application, so it all appears as if the HTTP request is successfully performed in both cases. Then again, it's HTTPS, so I cannot be certain.
Here is the OCaml code that performs the request:
let c = new Curl.handle in
let buffer = Buffer.create 1763 in
c # set_url "https://api-3t.paypal.com/nvp" ;
c # set_post true ;
c # set_postfields "SOMEDATA" ;
c # set_postfieldsize (String.length "SOMEDATA") ;
c # set_writefunction (fun x -> Buffer.add_string buffer x ; String.length x) ;
c # perform ;
c # cleanup ;
Buffer.contents buffer
Here is the equivalent curl command line:
curl -X POST https://api-3t.paypal.com/nvp -d SOMEDATA
EDIT: By increasing the libcurl verbosity, I determined that the underlying error is this:
GnuTLS recv error (-9): A TLS packet with unexpected length was received.
What could be the cause of this error? How can I investigate to find out?
EDIT 2: It appears that the difference between command-line and library use is that the command-line version is linked to OpenSSL and the library is linked to GnuTLS.
How do I link both to OpenSSL?

First, the key to further debugging those issues is to use curl's debugging facilities, namely the VERBOSE setting (and also, possibly, the DEBUGFUNCTION setting for printing the data your way).
c # set_verbose true ;
This identified the error as being a problem with GnuTLS that is also discussed here, and which is solved by setting SSLVERSION to 3 to force the use of SSLv3.
c # set_sslversion 3 ;

I always call curl#set_postfieldsize to the length of the data passed to curl#set_postfields. My code would be, then:
let make_get url =
let curl = new Curl.handle in
curl#set_writefunction String.length; (* ignore result *)
curl#set_tcpnodelay true;
curl#set_verbose false;
curl#set_post false;
curl#set_url url;
curl
let make_post url =
let curl = make_get url in
curl#set_post true;
curl#set_httpheader [
"Content-Type: text/xml; charset=\"UTF-8\"";
"SOAPAction: \"\"";
];
curl#set_postfields xml;
curl#set_postfieldsize (String.length xml);
curl
I hope this helps.

Related

Curl taking too long to send https request using command line

I have implemented one shall script which send an https request to proxy with authorization header using GET request.
Here is my command :
curl -s -o /dev/null -w "%{http_code}" -X GET -H "Authorization: 123456:admin05" "https://www.mywebpage/api/request/india/?ID=123456&Number=9456123789&Code=01"
It takes around 12 second to wait and then sending request to proxy and revert back with some code like 200,400,500 etc..
Is it possible to reduce time and make it faster using CURL ?
Please advice me for such a case.
Thanks.
Use option -v or --verbose along with --trace-time
It gives details of actions begin taken along with timings.
Includes DNS resolution, SSL handshake, etc. Line starting with '>' means header/body being sent, '<' means being received.
Based on difference between operation sequence - you can decipher whether server is taking time to respond (time between request and response) or network latency or bandwidth(response taking) time.

How to verify AB responses?

Is there a way to make sure that AB gets proper responses from server? For example:
To force it to output the response of a single request to STDOUT OR
To ask it to check that some text fragment is included into the response body
I want to make sure that authentication worked properly and i am measuring response time of the target page, not the login form.
Currently I just replace ab -n 100 -c 1 -C "$MY_COOKIE" $MY_REQUEST with curl -b "$MY_COOKIE" $MY_REQUEST | lynx -stdin .
If it's not possible, is there an alternative more comprehensive tool that can do that?
You can use the -v option as listed in the man doc:
-v verbosity
Set verbosity level - 4 and above prints information on headers, 3 and above prints response codes (404, 200, etc.), 2 and above prints warnings and info.
https://httpd.apache.org/docs/2.4/programs/ab.html
So it would be:
ab -n 100 -c 1 -C "$MY_COOKIE" -v 4 $MY_REQUEST
This will spit out the response headers and HTML content. The 3 value will be enough to check for a redirect header.
I didn't try piping it to Lynx but grep worked fine.
Apache Benchmark is good for a cursory glance at your system but is not very sophisticated. I am currently attempting to tune a web service and am finding that AB does not measure complete response time when considering the transfer of the body. Also as you mention you can not verify what is returned.
My current recommendation is Apache JMeter. http://jmeter.apache.org/
I am having much better success with it. You may find the Response Assertion useful for your situation. http://jmeter.apache.org/usermanual/component_reference.html#Response_Assertion

POST receiver (server)

I find myself in need of a way to test the post requests made from my application.
I'm sending my request to http://macbook-pro.local/, and I'm trying to figure out what service I can run to read and display the request.
I looked into RestKit without any luck.
Using SignalR could work, but the port to macos isn't working as expected.
What you want is basically a very simple web server, but if all you want is printing out what comes in a HTTP POST request you can get away with using the built-in 'nc' command. You can use Terminal to print out the contents of incoming requests on local port 10000 by running 'nc' in a loop like this:
while true; do nc -l 10000 < /dev/null ; printf '\n\n\n'; done
You can then go to http://localhost:10000 in your browser and see the HTTP request appear in your Terminal window. The web browser will give an error message since 'nc' isn't smart enough to reply.
To test an HTTP POST request you can use 'curl':
curl --data "this-is-POST-data" http://localhost:10000
Again, curl will give an error message because 'nc' simply closes the connection without giving a proper HTTP reply. You can have 'nc' reply with a static HTTP response to all requests like this:
while true; do printf 'HTTP/1.0 200 OK\r\nContent-type: text-plain\r\n\r\nHello, world!' | nc -l 10000 ; printf '\n\n\n'; done
If you need to use port 80 you'll need to run 'nc' as root (e.g. using 'sudo').
If you need to have any kind of real HTTP traffic going on, however, you will need to get a proper web server. OS X comes with the Apache web server which can be started with the command "apachectl start" ("apachectl stop" to stop it). CGI is enabled so you can put executables into /Library/WebServer/CGI-Executables and access them using http://localhost/cgi-bin/filename. For example, if you create the following CGI script:
#!/bin/sh
printf 'Content-type: text/plain\r\n\r\n'
cat > /tmp/post-data
echo OK
call it, say, "test.sh" and place it in the CGI-Executables folder, and run:
chmod +x /Library/WebServer/CGI-Executables/test.sh
Then, whenever you send a POST request to http://localhost/cgi-bin/test.sh it will save the contents of the POST data to the file /tmp/post-data on your computer.
Note: in all examples, "localhost" can be replaced with "macbook-pro.local" for accesses over the network if that is your computer hostname.
Also note that your OS X firewall permissions may block 'nc' and other software from listening to TCP ports. Usually you should get a permission dialog, but if you simply get "permission denied", tweak your firewall settings in System Preferences -> Firewall -> Firewall Options.
Look for SBJson framework
These are sample lines u can write to parse the GET data.
SBJsonParser *parser = [[SBJsonParser alloc] init];
NSDictionary *dict = [parser objectWithData:urlData];
[dictionary setDictionary:dict];
[parser release];
These are sample lines u can write to POST data.
SBJsonWriter *writer = [[SBJsonWriter alloc] init];
jsonStr = [writer stringWithObject:dictionary];
[writer release];
There are many more methods in framework to do some useful stuffs.

Perl Script to Monitor URL Using proxy credentials?

Please help on the following code, this is not working in our environment.
use LWP;
use strict;
my $url = 'http://google.com';
my $username = 'user';
my $password = 'mypassword';
my $browser = LWP::UserAgent->new('Mozilla');
$browser->credentials("172.18.124.11:80","something.co.in",$username=>$password);
$browser->timeout(10);
my $response=$browser->get($url);
print $response->content;
OUTPUT :
Can't connect to google.com:80 (timeout)
LWP::Protocol::http::Socket: connect: timeout at C:/Perl/lib/LWP/Protocol/http.p m line 51.
OS: windows XP
Regards, Gaurav
Do you have a HTTP proxy at 172.18.124.11? I assume LWP is not using the proxy. You might want to use env_proxy => 1 with the new() call.
You also have a mod-perl2 tag in this question. If this code runs inside mod-perl2, it's possible that the http_proxy env variable is not visible to the code. You can check this eg. by printing $browser->proxy('http').
Or just set the proxy with $browser->proxy('http', '172.18.124.11');
Also, I assume you don't have use warnings on, because new() takes a hash, not just a string. It's a good idea to always enable warnings. That will save you lots of trouble.

Sending an email from R using the sendmailR package

I am trying to send an email from R, using the sendmailR package. The code below works fine when I run it on my PC, and I recieve the email. However, when I run it with my macbook pro, it fails with the following error:
library(sendmailR)
from <- sprintf("<sendmailR#%s>", Sys.info()[4])
to <- "<myemail#gmail.com>"
subject <- "TEST"
sendmail(from, to, subject, body,
control=list(smtpServer="ASPMX.L.GOOGLE.COM"))
Error in socketConnection(host = server, port = port, blocking = TRUE) :
cannot open the connection
In addition: Warning message:
In socketConnection(host = server, port = port, blocking = TRUE) :
ASPMX.L.GOOGLE.COM:25 cannot be opened
Any ideas as to why this would work on a PC, but not a mac? I turned the firewall off on both machines.
Are you able to send email via the command-line?
So, first of all, fire up a Terminal and then
$ echo “Test 123” | mail -s “Test” user#domain.com
Look into /var/log/mail.log, or better use
$ tail -f /var/log/mail.log
in a different window while you send your email. If you see something like
... setting up TLS connection to smtp.gmail.com[xxx.xx.xxx.xxx]:587
... Trusted TLS connection established to smtp.gmail.com[xxx.xx.xxx.xxx]:587:\
TLSv1 with cipher RC4-MD5 (128/128 bits)
then you succeeded. Otherwise, it means you have to configure you mailing system. I use postfix with Gmail for two years now, and I never had have problem with it. Basically, you need to grab the Equifax certificates, Equifax_Secure_CA.pem from here: http://www.geotrust.com/resources/root-certificates/. (They were using Thawtee certificates before but they changed last year.) Then, assuming you used Gmail,
Create relay_password in /etc/postfix and put a single line like this (with your correct login and password):
smtp.gmail.com login#gmail.com:password
then in a Terminal,
$ sudo postmap /etc/postfix/relay_password
to update Postfix lookup table.
Add the certificates in /etc/postfix/certs, or any folder you like, then
$ sudo c_rehash /etc/postfix/certs/
(i.e., rehash the certificates with Openssl).
Edit /etc/postfix/main.cf so that it includes the following lines (adjust the paths if needed):
relayhost = smtp.gmail.com:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/relay_password
smtp_sasl_security_options = noanonymous
smtp_tls_security_level = may
smtp_tls_CApath = /etc/postfix/certs
smtp_tls_session_cache_database = btree:/etc/postfix/smtp_scache
smtp_tls_session_cache_timeout = 3600s
smtp_tls_loglevel = 1
tls_random_source = dev:/dev/urandom
Finally, just reload the Postfix process, with e.g.
$ sudo postfix reload
(a combination of start/stop works too).
You can choose a different port for the SMTP, e.g. 465.
It’s still possible to use SASL without TLS (the above steps are basically the same), but in both case the main problem is that your login informations are available in a plan text file... Also, should you want to use your MobileMe account, just replace the Gmail SMTP server with smtp.me.com.

Resources