When using --negotiate with curl on windows, SSL/TLS handshake fails - windows

When using --negotiate (or ntlm) with curl on windows, SSL/TSL handshake fails despite having a valid kerberos ticket cached on my windows 10 (shown below). The same logic and commands works without any issue in Unix/Linux. Any idea/help on how to resolve this issue?
Klist details:
$: Klist
Client: username # XXXX.XXX
Server: cifs/XXXXXXX.XXX # XXXXXXX.XXX
KerbTicket Encryption Type: AES-256-CTS-HMAC-SHA1-96
Ticket Flags XXXXXX -> forwardable renewable pre_authent name_canonicalize
Start Time: 8/27/2020 9:46:36 (local)
End Time: 8/27/2020 19:46:33 (local)
Renew Time: 9/27/2020 9:46:33 (local)
Session Key Type: AES-256-CTS-HMAC-SHA1-96
Cache Flags: 0
Kdc Called: XXXXXXXXX.XXX
curl command using ntlm or negotiate details:
>curl --ntlm -u : https://XXXXX.XXX -v
OR
>curl --negotiate -u : https://XXXXX.XXX -v
* Trying xx.xx.xx.xx...
* TCP_NODELAY set
* Connected to xxxx.xxx (xx.xx.xx.xx) port xxx (#0)
* schannel: SSL/TLS connection with xxxx.xxx port xxx (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 186 bytes...
* schannel: sent initial handshake data: sent 186 bytes
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 1024
* schannel: encrypted data buffer: offset 5120 length 5120
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 817
* schannel: encrypted data buffer: offset 5937 length 6144
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 2/3)
* schannel: encrypted data got 51
* schannel: encrypted data buffer: offset 51 length 6144
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with xxx.xxx port xxx (step 3/3)
* schannel: stored credential handle in session cache
> GET /login HTTP/1.1
> Host: xxx.xxx
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 915
* schannel: encrypted data buffer: offset 915 length 103424
* schannel: decrypted data length: 852
* schannel: decrypted data added: 852
* schannel: decrypted data cached: offset 852 length 102400
* schannel: encrypted data length: 34
* schannel: encrypted data cached: offset 34 length 103424
* schannel: decrypted data length: 5
* schannel: decrypted data added: 5
* schannel: decrypted data cached: offset 857 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 857 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 857
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Application-Context: Apixxxxx:x
< X-RateLimit-Limit-Api_login_anonymous: 1000
< X-RateLimit-Remaining-Api_login_anonymous: 999
< X-RateLimit-Reset-Apixxx_login_anonymous: 0
< X-xxxxx-xxx: xxxxx.xxx
< Date: Thu, 27 Aug 2020 19:50:12 GMT
< Expires: 0
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< WWW-Authenticate: Negotiate
< WWW-Authenticate: Basic realm="Please login with your Windows account"
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=xxx ; includeSubDomains
< TraceId: 5f480e7427f78fd5406fcbef0abf8e6c
< X-Content-Type-Options: nosniff
< Transfer-Encoding: chunked
<
* Ignoring the response-body
* Connection #0 to host xxx.xxx left intact
* Issue another request to this URL: 'https://xxx.xxx/login'
* Found bundle for host xxx.xxx: 0xxxxxxxxx [can pipeline]
* Re-using existing connection! (#0) with host xxx.xxx
* Connected to xxx.xxx (xx.xx.xxx.xx) port xxx (#0)
* Server auth using Negotiate with user ' '
> GET /login HTTP/1.1
> Host: xxx.xxx
> Authorization: Negotiate xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx==
> User-Agent: curl/7.55.1
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 814
* schannel: encrypted data buffer: offset 814 length 103424
* schannel: decrypted data length: 751
* schannel: decrypted data added: 751
* schannel: decrypted data cached: offset 751 length 102400
* schannel: encrypted data length: 34
* schannel: encrypted data cached: offset 34 length 103424
* schannel: decrypted data length: 5
* schannel: decrypted data added: 5
* schannel: decrypted data cached: offset 756 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 756 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 756
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 401
< Cache-Control: private
< Expires: Thu, 01 Jan 1970 00:00:00 GMT
< X-Application-Context: Apixxx:xxxxxxxxxxxxxxxx:x
< X-RateLimit-Limit-Apixxx_login_anonymous: 1000
< X-RateLimit-Remaining-Apixxx_login_anonymous: 999
< X-RateLimit-Reset-Apixxx_login_anonymous: 0
< X-XXXX-xxx: xxxx.xxx
< Date: Thu, 27 Aug 2020 19:50:12 GMT
< Expires: 0
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: DENY
< Cache-Control: no-cache, no-store, max-age=0, must-revalidate
< Pragma: no-cache
< Strict-Transport-Security: max-age=xxx ; includeSubDomains
< TraceId: xxxxxxxxxxxxxxxxxxxxxxxxxxx
< X-Content-Type-Options: nosniff
< Transfer-Encoding: chunked
<
* Connection #0 to host xxx.xxx left intact
>curl --version
curl 7.55.1 (Windows) libcurl/7.55.1 WinSSL
Release-Date: [unreleased]
Protocols: dict file ftp ftps http https imap imaps pop3 pop3s smtp smtps telnet tftp
Features: AsynchDNS IPv6 Largefile SSPI Kerberos SPNEGO NTLM SSL

Related

HTTP request failing – but only when run through Cocoapods

I publish a pod with the following spec:
Pod::Spec.new do |s|
s.name = "AirTurnInterface"
s.version = "4.7.0-b.6"
s.summary = "snip"
s.description = "snip"
s.license = { :file => 'LICENSE', :type => 'AirTurn' }
s.homepage = "snip"
s.author = { "Nick Brook" => "snip#airturn.com" }
s.ios.deployment_target = "11.0"
s.frameworks = 'CoreBluetooth', 'GameController', 'Security', 'UIKit'
s.source = { :http => "https://airturn.com/framework/test/AirTurnInterface.#{s.version}.zip" }
s.vendored_frameworks = "Framework-dynamic/AirTurnInterface.xcframework"
end
This can be tested by putting the following in a file named test.podspec and running pod spec lint test.podspec.
Until recently, this worked fine. Now, running pod spec lint fails when downloading the zip with the error:
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
It also provides the curl command used:
/opt/homebrew/opt/curl/bin/curl -f -L -o /var/folders/9j/jqc5qhp922b7c8jhrvkh7h8m0000gn/T/d20221021-48752-1fqts5p/file.zip https://airturn.com/framework/test/AirTurnInterface.4.7.0-b.6.zip --create-dirs --netrc-optional --retry 2 -A 'CocoaPods/1.11.3 cocoapods-downloader/1.5.1'
The strange thing is, this command works when run directly, it only fails when run from Cocoapods (ruby). I've also tried just performing the command from a simple ruby script, which works fine. Something about Cocoapods is causing the command to fail. I've modified Cocoapods (gems/cocoapods-downloader-1.5.1/lib/cocoapods-downloader/http.rb) to pass -v to curl, which provides the following output:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 35.209.185.188:443...
* Connected to airturn.com (35.209.185.188) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [88 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [187 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [19 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [4039 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [264 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: CN=*.airturn.com
* start date: Oct 5 00:07:55 2022 GMT
* expire date: Jan 3 00:07:54 2023 GMT
* subjectAltName: host "airturn.com" matched cert's "airturn.com"
* issuer: C=US; O=Let's Encrypt; CN=R3
* SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* h2h3 [:method: GET]
* h2h3 [:path: /framework/test/AirTurnInterface.4.7.0-b.6.zip]
* h2h3 [:scheme: https]
* h2h3 [:authority: airturn.com]
* h2h3 [user-agent: 'CocoaPods/1.11.3 cocoapods-downloader/1.5.1']
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x15580f600)
} [5 bytes data]
> GET /framework/test/AirTurnInterface.4.7.0-b.6.zip HTTP/2
> Host: airturn.com
> user-agent: 'CocoaPods/1.11.3 cocoapods-downloader/1.5.1'
> accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
} [5 bytes data]
* HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host airturn.com left intact
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
Forcing http1.1 with --http1.1 doesn't help:
curl: (52) Empty reply from server
I don't know what to try next to debug further!
It looks like the ZIP file is being returned using HTTP/1.1 instead of HTTP/2 which CocoaPods is requiring. Older versions of TLS are no longer supported on some platforms so I'd recommend updating your web server/host to a platform that is using a more modern TLS.

Why do I keep getting BUILD FAILURE when trying to install ruby 3.1.2 in Ubuntu 20.04 WSL2?

I've been trying to install ruby 3.1.2 on my ubunutu 20.04 WSL2 setup.
I get through installing rbenv, but when I try to install ruby I get stuck at the downloading ruby stage, and get a BUILD FAILED message.
I'm not sure what other info to provide, but please just ask and I will get anything I'm missing
bmcbride#G15:~$ rbenv -v
rbenv 1.2.0-16-gc4395e5
bmcbride#G15:~$ rbenv install 3.1.2 --verbose
/tmp/ruby-build.20220812150245.174.oXyOhQ ~
Downloading ruby-3.1.2.tar.gz...
-> https://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.2.tar.gz
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
error: failed to download ruby-3.1.2.tar.gz
BUILD FAILED (Ubuntu 20.04 using ruby-build 20220726-1-ga753b24)
Inspect or clean up the working tree at /tmp/ruby-build.20220812150245.174.oXyOhQ
Results logged to /tmp/ruby-build.20220812150245.174.log
Last 10 log lines:
/tmp/ruby-build.20220812150245.174.oXyOhQ ~
curl: (92) HTTP/2 stream 0 was not closed cleanly: PROTOCOL_ERROR (err 1)
EDIT:
RUBY_BUILD_CURL_OPTS="-0 --verbose" rbenv install 3.1.2 --verbose
/tmp/ruby-build.20220812195807.19239.iSYAyi ~
Downloading ruby-3.1.2.tar.gz...
-> https://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.2.tar.gz
* Trying 151.101.109.178:443...
* TCP_NODELAY set
* Trying 2a04:4e42:1a::434:443...
* TCP_NODELAY set
* Immediate connect fail for 2a04:4e42:1a::434: Network is unreachable
* Connected to cache.ruby-lang.org (151.101.109.178) port 443 (#0)
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [112 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2875 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: CN=*.ruby-lang.org
* start date: Dec 24 19:42:27 2021 GMT
* expire date: Jan 25 19:42:26 2023 GMT
* subjectAltName: host "cache.ruby-lang.org" matched cert's "*.ruby-lang.org"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA H2 2021
* SSL certificate verify ok.
} [5 bytes data]
> GET /pub/ruby/3.1/ruby-3.1.2.tar.gz HTTP/1.0
> Host: cache.ruby-lang.org
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Connection: close
< Content-Length: 20553628
< x-amz-id-2: qUJXy341CDBFg3sNrJh4iAsIj8xlsKtR0W/6qHCp75BYVgRiO21fx6Nh+SEfYbgdaQJWHdyA608=
< x-amz-request-id: 3H67P45YZZZNY8HP
< Last-Modified: Tue, 12 Apr 2022 12:47:13 GMT
< ETag: "3fc61f350eef6c49644eb25f8189874f-3"
< x-amz-version-id: i9hhX3JvclyQw4rFW4_nNjUod9k991xX
< Content-Type: application/x-tar
< Server: AmazonS3
< Via: 1.1 varnish, 1.1 varnish
< Accept-Ranges: bytes
< Age: 1955
< Date: Fri, 12 Aug 2022 11:58:09 GMT
< X-Served-By: cache-iad-kcgs7200133-IAD, cache-tyo11941-TYO
< X-Cache: HIT, HIT
< X-Cache-Hits: 0, 0
< X-Timer: S1660305489.152710,VS0,VE1
<
{ [5 bytes data]
error: failed to download ruby-3.1.2.tar.gz
BUILD FAILED (Ubuntu 20.04 using ruby-build 20220726-1-ga753b24)
EDIT 2
output from forcing rbenv to run ipv4
bmcbride#X13:~$ RUBY_BUILD_CURL_OPTS="-4 --verbose" rbenv install 3.1.2 --verbose
/tmp/ruby-build.20220813083713.327.wrSpG1 ~
Downloading ruby-3.1.2.tar.gz...
-> https://cache.ruby-lang.org/pub/ruby/3.1/ruby-3.1.2.tar.gz
* Trying 146.75.113.178:443...
* TCP_NODELAY set
* Connected to cache.ruby-lang.org (146.75.113.178) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [106 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2875 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [300 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=*.ruby-lang.org
* start date: Dec 24 19:42:27 2021 GMT
* expire date: Jan 25 19:42:26 2023 GMT
* subjectAltName: host "cache.ruby-lang.org" matched cert's "*.ruby-lang.org"
* issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Atlas R3 DV TLS CA H2 2021
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5604c88fa8c0)
} [5 bytes data]
> GET /pub/ruby/3.1/ruby-3.1.2.tar.gz HTTP/2
> Host: cache.ruby-lang.org
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
} [5 bytes data]
< HTTP/2 200
< x-amz-id-2: qUJXy341CDBFg3sNrJh4iAsIj8xlsKtR0W/6qHCp75BYVgRiO21fx6Nh+SEfYbgdaQJWHdyA608=
< x-amz-request-id: 3H67P45YZZZNY8HP
< last-modified: Tue, 12 Apr 2022 12:47:13 GMT
< etag: "3fc61f350eef6c49644eb25f8189874f-3"
< x-amz-version-id: i9hhX3JvclyQw4rFW4_nNjUod9k991xX
< content-type: application/x-tar
< server: AmazonS3
< via: 1.1 varnish, 1.1 varnish
< accept-ranges: bytes
< age: 1122
< date: Sat, 13 Aug 2022 00:37:12 GMT
< x-served-by: cache-iad-kcgs7200103-IAD, cache-nrt-rjtf7700030-NRT
< x-cache: HIT, HIT
< x-cache-hits: 0, 0
< x-timer: S1660351033.733568,VS0,VE1
< content-length: 20553628
<
{ [5 bytes data]
This may help:
https://learn.microsoft.com/en-us/windows/wsl/install-manual#step-4---download-the-linux-kernel-update-package
There's a kernel issue that causes big issues when you try to install the new Ruby on WSL. The above fixes it.

Suppressing schannel output on Windows

I've been using cURL for a while now, though not upgrading the executable very often. I've noticed with v7.60 that when using -v to get the headers as well, I get a smattering of * schannel: messages in the prompt, so that it goes like this:
$ curl -vs "https://example.com"
* Rebuilt URL to: https://example.com/
* Trying 93.184.216.34...
* TCP_NODELAY set
* Connected to example.com (93.184.216.34) port 443 (#0)
* schannel: SSL/TLS connection with example.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 176 bytes...
* schannel: sent initial handshake data: sent 176 bytes
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 4096
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 14
* schannel: encrypted data buffer: offset 14 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 817
* schannel: encrypted data buffer: offset 831 length 4096
* schannel: sending next handshake data: sending 126 bytes...
* schannel: SSL/TLS connection with example.com port 443 (step 2/3)
* schannel: encrypted data got 242
* schannel: encrypted data buffer: offset 242 length 4096
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with example.com port 443 (step 3/3)
* schannel: stored credential handle in session cache
> GET / HTTP/1.1
> Host: example.com
> User-Agent: curl/7.60.0
> Accept: */*
>
* schannel: client wants to read 102400 bytes
* schannel: encdata_buffer resized 103424
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: encrypted data got 1666
* schannel: encrypted data buffer: offset 1666 length 103424
* schannel: decrypted data length: 338
* schannel: decrypted data added: 338
* schannel: decrypted data cached: offset 338 length 102400
* schannel: encrypted data length: 1299
* schannel: encrypted data cached: offset 1299 length 103424
* schannel: decrypted data length: 1270
* schannel: decrypted data added: 1270
* schannel: decrypted data cached: offset 1608 length 102400
* schannel: encrypted data buffer: offset 0 length 103424
* schannel: decrypted data buffer: offset 1608 length 102400
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 1608
* schannel: decrypted data buffer: offset 0 length 102400
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: max-age=604800
< Content-Type: text/html; charset=UTF-8
< Date: Tue, 05 Feb 2019 08:08:57 GMT
< Etag: "1541025663"
< Expires: Tue, 12 Feb 2019 08:08:57 GMT
< Last-Modified: Fri, 09 Aug 2013 23:54:35 GMT
< Server: ECS (dca/24C1)
< Vary: Accept-Encoding
< X-Cache: HIT
< Content-Length: 1270
<
[HTML here]
Is there any way to suppress these messages so that -v still yields me the headers, and I can use > NUL to discard the response body, all without the SChannel noise?
To head off the obvious suggestion, -I won't work for me: I need to be able to send GET and other requests this way because my work involves investigating third-party servers that can (and have been known to) respond to HEAD differently.

MQL4: How to read a CSV from a URL

I'm using this URL to fetch some content from a Quandl website:
https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv?exclude_column_names=true&rows=1&api_key=my_api_key
The Quandl server returns in response to the above request a value of this:
2016-08-01, 144598.0
I need to use the value of 144598.0 within an MQL4 Script, so:
Q1. How do I fetch the content from the URL above to be used within an MQL4 Script?
A very helpful user from SO (https://stackoverflow.com/users/3666197/user3666197) provided the following script (original found at MQL4: Read single value from CSV) (a couple parts added in by myself) to help me achieve this, however, I couldn't get it to work:
// Code created with the help of Stack Overflow question
// https://stackoverflow.com/questions/39279634/mql4-read-single-value-from-csv/39284875#39284875
// Question by p.luck:
// https://stackoverflow.com/users/5551849/p-luck
// Answer by user3666197:
// https://stackoverflow.com/users/3666197/user3666197
void OnStart()
{
string cookie = NULL,
headers;
char post[],
result[];
int res;
/* TODO: *
* Must allow MT4 to access the server URL, *
* you should add URL "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv" *
* in the list of allowed URLs *
* ( MT4 -> Tools -> Options -> [Tab]: "Expert Advisors" ): */
string aDataSOURCE_URL = "https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv";
string aDataSOURCE_API = "?exclude_column_names=true&rows=1&api_key=my_api_key";
//-- Create the body of the POST request for API specifications and API-authorization
ArrayResize( post,
StringToCharArray( aDataSOURCE_API, // string text |--> [in] String to copy.
post, // uchar &array[] <--| [out] Array of uchar type.
0, // int start = 0 |--> [in] Position from which copying starts. Default - 0.
WHOLE_ARRAY, // int count = -1 |--> [in] Number of array elements to copy. Defines length of a resulting string. Default value is -1, which means copying up to the array end, or till terminating '\0'. Terminating zero will also be copied to the recipient array, in this case the size of a dynamic array can be increased if necessary to the size of the string. If the size of the dynamic array exceeds the length of the string, the size of the array will not be reduced.
CP_UTF8 // uint cp = CP_ACP |--> [in] The value of the code page. For the most-used code pages provide appropriate constants.
)
- 1
);
//-- Reset the last error code
ResetLastError();
//-- Loading a html page from Quandl
int timeout = 5000; //-- Timeout below 1000 (1 sec.) is not enough for slow Internet connection
res = WebRequest( "POST", // const string method |--> [in] HTTP method.
aDataSOURCE_URL, // const string URL |--> [in] URL.
cookie, // const string cookie |--> [in] Cookie value.
NULL, // const string referrer |--> [in] Value of the Referer header of the HTTP request.
timeout, // int timeout |--> [in] Timeout in milliseconds.
post, // const char &data |--> [in] Data array of the HTTP message body
ArraySize( post ), // int data_size |--> [in] Size of the data[] array.
result, // char &result <--| [out] An array containing server response data.
headers // string &result_headers <--| [out] Server response headers.
);
//-- Check errors
if ( res == -1 )
{ Print( "WebRequest Error. Error code = ", GetLastError() ); //-- Perhaps the URL is not listed, display a message about the necessity to add the address
MessageBox( "Add the address '" + aDataSOURCE_URL + "' in the list of allowed URLs on tab 'Expert Advisors'", "Error", MB_ICONINFORMATION );
}
else //-- Load was successfull
{
PrintFormat( "The data has been successfully loaded, size = %d bytes.", ArraySize( result ) );
//-- parse the content ---------------------------------------
/*
"2016-08-01, 144598.0"
*/
//-- consume the content -------------------------------------
//...
}
}
I have added the URL of https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csvto the list of allowed URLs in MT4.
If I enable AutoTrading and drag the script, named (tutorial7), on to a chart of USDCAD,M1,I get these messages within the Experts tab:
Script tutorial7 USDCAD,M1: loaded successfuly
tutorial7 USDCAD,M1: initialized
tutorial7 USDCAD,M1: The data has been successfully loaded, size = 0 bytes
tutorial7 USDCAD,M1: uninit reason 0
Surely if the "The data has successfully loaded" it shouldn't say "size = 0 bytes"?
Should this script work correctly if I just copy and paste it straight in to the MetaQuotes Language Editor and compile it?
Apart from adding the URL to the allowed URLs in MT4 and copying and pasting this code in to a script, is there anything else I must do?
If so, how?
I am running the above code as a Script not an Expert Advisor; is this okay?
Q2. Can I set the fetched value of 144598.0 as a variable within my script?
I need to make the value of 144598.0 a variable so that it can be compared to another value.
Would something like this work:
void OnStart()
{
... // above code which fetches the value from the .csv URL
double x = 155876.0 // value manually typed in
y = 144598.0 // value fetched from the .csv URL using the above code
// ignores column 1 consisting of 2016-08-01
if ( x > y ) {
// execute code
}
else {
// execute other code
}
}
A2: YES! This is the easiest part of the journey.A1: BUT!How does it work in practice? How does MetaTrader Terminal actually speak to Quandl, so to get it?
Let me first illustrate the issue of remote, HttpServer-side processing.
There is an easy to prototype program curl ( Linux & DOS versions available ) that will show right inside a terminal window ( or a Windows cmd window ) how the remote HttpServer at Quandl responds to various compositions of the locally assembled requests, communicated over the HTTP-protocol.
Notice, that the just retyped URL produces the whole history to be delivered.
C:\>curl https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
DATE,VALUE
2016-08-01,144598.0
2016-07-01,144447.0
...
..
.
1939-03-01,30280.0
1939-02-01,30101.0
1939-01-01,29923.0
Once we add further parameters to the plain URL, the remote-side ( the HttpServer ) changes the reply to just the one row we are interested in:
C:\>curl -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
2016-08-01,144598.0
Cool, looks great, almost the single value we want!
But here the magic comes.
The real interchange ( dialogue ) between the local process ( curl now, but MetaTrader Terminal later ) looks this way ( using a --verbose option in curl commandline ):
C:\>curl --verbose -G --data rows=1 --data exclude_column_names=true https://www.quandl.com/api/v3/datasets/FRED/PAYEMS.csv
* Trying 54.174.87.84...
* Connected to www.quandl.com (54.174.87.84) port 443 (#0)
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 1/3)
* schannel: checking server certificate revocation
* schannel: sending initial handshake data: sending 70 bytes...
* schannel: sent initial handshake data: sent 70 bytes
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: failed to receive handshake, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4096 length 4096
* schannel: encrypted data length: 4017
* schannel: encrypted data buffer: offset 4017 length 4096
* schannel: received incomplete message, need more data
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 4569 length 5041
* schannel: sending next handshake data: sending 318 bytes...
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 2/3)
* schannel: encrypted data buffer: offset 51 length 5041
* schannel: SSL/TLS handshake complete
* schannel: SSL/TLS connection with www.quandl.com port 443 (step 3/3)
* schannel: incremented credential handle refcount = 1
* schannel: stored credential handle in session cache
> GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true HTTP/1.1
> Host: www.quandl.com
> User-Agent: curl/7.45.0
> Accept: */*
>
* schannel: client wants to read 16384 bytes
* schannel: encdata_buffer resized 17408
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: encrypted data got 653
* schannel: encrypted data buffer: offset 653 length 17408
* schannel: decrypted data length: 623
* schannel: decrypted data added: 623
* schannel: decrypted data cached: offset 623 length 16384
* schannel: encrypted data buffer: offset 0 length 17408
* schannel: decrypted data buffer: offset 623 length 16384
* schannel: schannel_recv cleanup
* schannel: decrypted data returned 623
* schannel: decrypted data buffer: offset 0 length 16384
< HTTP/1.1 200 OK
< Cache-Control: private
< Content-Disposition: attachment; filename=FRED-PAYEMS.csv
< Content-Transfer-Encoding: binary
< Content-Type: text/csv
< Date: Wed, 07 Sep 2016 12:47:20 GMT
< ETag: W/"adfdb97850c493cdd03e2036574bc404"
< Server: openresty
< Vary: Origin
< X-API-Version: 2015-04-09
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Rack-CORS: preflight-hit; no-origin
< X-RateLimit-Limit: 50
< X-RateLimit-Remaining: 42
< X-Request-Id: c609e92d-22d2-40e7-b7d4-cacb07467c76
< X-Runtime: 0.023534
< X-XSS-Protection: 1; mode=block
< Content-Length: 20
< Connection: keep-alive
<
2016-08-01,144598.0
* Connection #0 to host www.quandl.com left intact
Notice the row GET /api/v3/datasets/FRED/PAYEMS.csv?rows=1&exclude_column_names=true
So the magic is to make MetaTrader Terminal to assemble the same, together with allowing the URL in the permitted list in the configuration ( that you have done already in the other post ).
Also might have noticed, that the HTTP GET sends just the <HttpServer_relative_part_of_the.URL>
The magic is in making MQL4 code send the same request as was seen above and get the data back.
WebRequest() has to use HTTP GET as the Quandl HttpServer does not respond to a HTTP POST version of the same request example, returning 0 bytes ( just omit the -G switch from the curl examples above ).
Meeting all the conditions at once should result in receiving 2016-08-01,144598.0 and using:
int anAmountOfBytesRECVd = 0; // how many bytes Quandl sent
string headers_FromHttpSERVER; // stores as a string
char anAnswerFromHttpSERVER[]; // stores as a byte-array ( char[] )
double y_value = NULL;
anAmountOfBytesRECVd = WebRequest( "GET", // MUST use HTTP GET <- Quandl tested
...
anAnswerFromHttpSERVER,
headers_FromHttpSERVER
);
y_value = StrToDouble( CharArrayToString( anAnserFromHttpSERVER, // [2|0|1|6|-|0|8|-|0|1|,|1|4|4|5|98|.|0]
11, //-----------------------^_______________( -1 == TILL EndOfSTRING )
-1
)
);

SSL Client Authenticated Webservice on ruby SSLv3 read server certificate B: certificate verify failed

I'm trying to connect to a webservice that requires SSL Client Authentication using ruby 2.1.2 but
When I use the same client certificate (client_cert.pem) on curl I've got the right response:
curl 'https://mywebservice.xxx.com' --cert client_cert.pem --cacert mycacert.crt
When I verbose curl I can see the handshaking going on like this:
* Hostname was NOT found in DNS cache
* Trying 192.168.0.10...
* Connected to mywebservice.xxx.com (192.168.0.10) port 443 (#0)
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSL connection using DHE-RSA-AES256-SHA
* Server certificate:
* ...
> GET / HTTP/1.1
> User-Agent: curl/7.35.0
> Host: mywebservice.xxx.com
> Accept: */*
>
* SSLv3, TLS handshake, Hello request (0):
* SSLv3, TLS handshake, Client hello (1):
* SSLv3, TLS handshake, Server hello (2):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Server key exchange (12):
* SSLv3, TLS handshake, Request CERT (13):
* SSLv3, TLS handshake, Server finished (14):
* SSLv3, TLS handshake, CERT (11):
* SSLv3, TLS handshake, Client key exchange (16):
* SSLv3, TLS handshake, CERT verify (15):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
* SSLv3, TLS change cipher, Client hello (1):
* SSLv3, TLS handshake, Finished (20):
< HTTP/1.1 200 OK
< Date: Thu, 17 Jul 2014 12:50:26 GMT
* Server Apache/2.2.16 (Debian) is not blacklisted
< Server: Apache/2.2.16 (Debian)
< Pragma: No-cache
< Cache-Control: no-cache
< Expires: Wed, 31 Dec 1969 21:00:00 BRT
< Vary: Accept-Encoding
< Transfer-Encoding: chunked
< Content-Type: text/xml;charset=utf-8
<
<?xml version='1.0' encoding='UTF-8'?> .....the full xml response
That's what I needed, but then I've tried the same on ruby:
#ws_test.rb
require 'net/http'
require 'uri'
require 'openssl'
uri = URI.parse 'https://mywebservice.xxx.com/'
http = Net::HTTP.new(uri.host, 443)
http.use_ssl = true
http.ssl_version = :TLSv1
cert = OpenSSL::X509::Certificate.new(File.read("client_cert.pem"))
http.cert = cert
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
http.cert_store = OpenSSL::X509::Store.new
cacert = OpenSSL::X509::Certificate.new(File.read("mycacert.crt"))
http.cert_store.add_cert(cacert)
http.start do
http.request_get(uri.path) {|res|
print res.body
}
end
But on ruby I've got this problem:
/home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:920:in `connect': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (OpenSSL::SSL::SSLError)
from /home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:920:in `block in connect'
from /home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/timeout.rb:76:in `timeout'
from /home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:920:in `connect'
from /home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:863:in `do_start'
from /home/user/.rbenv/versions/2.1.2/lib/ruby/2.1.0/net/http.rb:852:in `start'
I've tried to change the verify_mode:
http.verify_mode = OpenSSL::SSL::VERIFY_PEER|OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
# same error
and
http.verify_mode = OpenSSL::SSL::VERIFY_PEER
# a different error: " SSL_read: ssl handshake failure (OpenSSL::SSL::SSLError"
I've solved it using the gem rest_client...
This is what I did:
require 'rest_client'
RestClient::Resource.new(
'https://mywebservice.xxx.com/',
:ssl_client_cert => OpenSSL::X509::Certificate.new(File.read("client_cert.crt")),
:ssl_client_key => OpenSSL::PKey::RSA.new(File.read("client_cert.key")),
:ssl_ca_file => "mycacert.crt",
:verify_ssl => OpenSSL::SSL::VERIFY_PEER
).get

Resources