I am trying to download a CSV file from a local webserver (the webserver runs on a signal conditioning instrument) with:
curl -O http://10.0.0.139/crv01.csv
The output from curl is only weird symbols. If I put the same url in Safari the CSV File is correctly depicted. Is there any encoding problem with curl?
I tried the verbose option which gives:
Gunthers-MacBook-Pro:Documents guenther$ curl -v -O http://10.0.0.141/crv01.csv
* About to connect() to 10.0.0.141 port 80 (#0)
* Trying 10.0.0.141...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* connected
* Connected to 10.0.0.141 (10.0.0.141) port 80 (#0)
> GET /crv01.csv HTTP/1.1
> User-Agent: curl/7.24.0 (x86_64-apple-darwin12.0) libcurl/7.24.0 OpenSSL/0.9.8x zlib/1.2.5
> Host: 10.0.0.141
> Accept: */*
>
< HTTP/1.1 200 OK
< Date: Mon Aug 26 21:03:12 2013
< Server: VEGA-HTTP-SERVER
< Cache-Control: public, max-age=3600
< Content-type: text/plain
< Connection: Close
< Content-Encoding: gzip
< Content-Length: 226
<
{ [data not shown]
100 226 100 226 0 0 1883 0 --:--:-- --:--:-- --:--:-- 1965
* Closing connection #0
Gunthers-MacBook-Pro:Documents guenther$ vi crv01.csv
^_<8b>^H^#^#^#^#^#^#^C}<92>Á
<83>0^L<86>ï<82>ï xÙ^N+I;kkO²<95>!l^N6õ-ÆÞÿ¶zФ¬4Ç~$ÿ<9f>?_~^YÞÃs¬P#YÔ<97>ùµø
ÐMýí4~E<85>áñÚOÞMÃû¥¿Ïþð9<96><85>l^D^X!^A<95>CÛ)Õ¡uR´^RBETB<96>b<96>J¢^X^F^G<87>LWª^?ªwWÀt·^F<99>n<82>&tY/Ó]M®^X=g]5D^S½:KÛ,5;Õv^]^]®À\Ù^\EÈRÌRÊ*¡<8b><94>U<8a>RV)JY¥(e¥^M<84><8a>öEÊ*E^Mmd TÜk<89>¶^AÆ÷eÿy<9b>ü^C»üß{E^C^#^#
Source Code of the webpage (Google Chrome) is a plain cdv file. CSV File is create by
http://www.vega.com/en/Signal-conditioning-instrument-VEGAMET391.htm
The --text-ascii option also did not help!
It seems the page is sent back compressed (see the header "Content-Encoding: gzip") even though curl didn't ask for it. If you ask for it, curl will decompress it for you automatically when receiving it. Try:
curl --compressed -O http://10.0.0.139/crv01.csv
That command should work, it works correctly on my system (10.6) when serving a csv file locally.
You could try the command with verbose on to see if there is any issue:
curl -v -O http://10.0.0.139/crv01.csv
How was the CSV created? By Excel or was it always native text? I suspect Safari is rendering while ignoring extraneous binary data.
You could View Source in Safari and make sure it is plain.
Also try curl --trace-ascii to request ASCII context.
Edit:
From your verbose output, it looks like the file is gzipped.
Try saving it as a .gz file instead and then gunzip crv01.gz
curl http://10.0.0.139/crv01.csv -o crv01.gz
gunzip crv01.gz
If there are more crv files, you can also download a range of them at once:
curl "http://10.0.0.139/crv[01-50].csv" -o crv#1.gz
Related
I've been trying to update the AMP cached pages on my website for a couple of days now to no avail.
While the documentation for updating the cache exists, it was probably written by a Google engineer, and as a result, isn't the easiest read.
https://developers.google.com/amp/cache/update-cache
I've followed the directions to the best of my ability.
I've created a private-key and public-key. Created a signature.bin and verified it using the procedure in Google's own documentation.
~$ openssl dgst -sha256 -signature signature.bin -verify
public-key.pem url.txt
Verified OK
The public-key.pem has been renamed to apikey.pub and uploaded to the following directory:
https://irecover.ca/.well-known/amphtml/apikey.pub
To validate that there has been no issue in the copying, I checked the signature using the following:
$ openssl dgst -sha256 -signature signature.bin -verify <(curl https://irecover.ca/.well-known/amphtml/apikey.pub) url.txt
% Total % Received % Xferd Average Speed Time Time Time
Current
Dload Upload Total Spent Left Speed
100 450 100 450 0 0 2653 0 --:--:-- --:--:--
--:--:-- 2662
Verified OK
Now I convert the signature file to base64 and replace the / with _ and the + with -
cat signature.bin | base64 > signature.b64
sed 's///_/g' signature.b64 > signature.b64a
sed 's/+/-/g' signature.b64a > signature.b64b
sed 's/=//g' signature.b64b > signature.b64c
cat signature.b64c | tr -d '\n' > signature.b64
I have made a script that makes the update-cache url for me. It also creates a timestamp right that moment and uses it for the amp_ts variable (So the amp_ts is never out by more than 1 second). I then append that to the end of the query which is about to be cURL'd by the script I have made, so it looks like so:
https://irecover-ca.cdn.ampproject.org/update-cache/c/s/irecover.ca/article?amp_action=flush&_ts=1581446499&_url_signature=KDaKbX0AbVbllwkTpDMFPOsFCRNw2sbk6Vd552bbG3u5QrecEmQ1SoMzmMR7iSXinO7LfM2bRCgJ1aD4y2cCayzrQuICrGz6b_PH7gKpo6tqETz06WmVeiP89xh_pBOu-pyN5rRHf0Pbu8oRkD2lRqgnGrLXDfIrFTTMRmHlO0bsa8GknyXL8RNXxk9ZQaufXAz-UJpoKaZBvT6hJWREAzxoZ-rGnDPVaC3nlBCu3yPorFcTbbr0CBz2svbfGgAYLQl54lLQmUpxI8661AEe1rdOLqAyLIUb4ZiSbO65-PmIkdZWVPFHMdbpSv4GMNdvodleCWBfMAcG2C09v-LR6g
However, this always results in the same error code from google.
Invalid public key due to ingestion error: 404 or 410 error from origin
Does anyone have any idea what I'm doing wrong?
A couple of things to check for apikey.pub accessibility:
The /.well-known/amphtml/apikey.pub file is accessible to both mobile and desktop user agents (e.g no redirect for non-mobile as the AMP cache client may redirect)
The public key is not excluded in robots.txt e.g:
User-agent: *
Allow: /.well-known/amphtml/apikey.pub
The public key response has the expected headers (e.g content-type: text/plain):
curl -I https://amp.example.com/.well-known/amphtml/apikey.pub
HTTP/2 200
date: Sun, 26 Jul 2020 23:48:55 GMT
content-type: text/plain
vary: Accept-Encoding
etag: W/"1c3-173478a8840"
last-modified: Sun, 26 Jul 2020 23:48:55 GMT
With those things in place, I get an "OK" success response from the update/cache endpoint
I'm trying to send binary files using an openSSL server, but the transfer stops just before it should be sending a 0x1A character.
I was able to send 8MB+ text files, but with both binaries I tried sending, it stopped after 20kB, right before the 0x1A character, so the size of the file is not a problem. I'm on Windows 10 and using openSSL to run the server.
winpty openssl s_server -WWW -key key.pem -cert cert.pem -port 8075
Using the following command, I am able to download text files without any issue, but it "fails" before the whole binary file was sent.
winpty curl -v -k https://192.168.1.100:8075/hello-world.bin --output hello-world-recv.bin
The verbose from the curl command is as follows (removed the first part where it's about the certificates and handshake):
GET /hello-world.bin HTTP/1.1
> Host: 192.168.1.100:8075
> User-Agent: curl/7.60.0
> Accept: */*
>
{ [5 bytes data]
* HTTP 1.0, assume close after body
< HTTP/1.0 200 ok
< Content-type: text/plain
<
{ [16339 bytes data]
100 19270 0 19270 0 0 43995 0 --:--:-- --:--:-- --:--:-- 45663
* Closing connection 0
} [5 bytes data]
* TLSv1.2 (OUT), TLS alert, Client hello (1):
} [2 bytes data]
As we can see, only 19kB of data was sent, but the original file is 1.2MB.
Any help would be appreciated, thanks.
Check in OpenSSL-source function www_body in apps/s_server.c (line 3254 in 1.1.1b): it opens the file in default "r" mode, which might be text mode in Windows, that means CRLF will be converted to LF and SUB will be interpreted as end of file.
I have created some curl command to send a POST to my server where I am listening on that port for input to trigger additional action. The command is the following (Just masked the URL):
curl -v -H "Content-Type: application/json" -X POST -d "{\"Location\":\"Some Name\",\"Value\":\"40%\"}" http://example.com:8885/
I get the following output from curl:
About to connect() to example.com port 8885 (#0)
Trying 5.147.XXX.XXX...
Connected to example.com (5.147.XXX.XXX) port 8885 (#0)
POST / HTTP/1.1
User-Agent: curl/7.29.0
Host: example.com:8885
Accept: /
Content-Type: application/json
Content-Length: 40
upload completely sent off: 40 out of 40 bytes
However after that curl does not close the connection. Am I doing something wrong? Also on the server I only receive the POST as soon as I hit ctrl+c.
It sits there waiting for the proper HTTP response, and after that has been received it will exit cleanly.
A minimal HTTP/1.1 response could look something like:
HTTP/1.1 200 OK
Content-Length: 0
... and it needs an extra CRLF after the last header to signal the end of headers.
I'm a bit rusty on this, but according to section 6.1 of RFC7230, you might need to add a Connection: close header as well. Quoting part of the paragraph:
The "close" connection option is defined for a sender to signal
that this connection will be closed after completion of the
response. For example,
Connection: close
in either the request or the response header fields indicates that
the sender is going to close the connection after the current
request/response is complete (Section 6.6).
Let me know if it solves your issue :-)
Is there a question mark in link ?
I found that my link had question mark like http... .com/something/something?properties=1 and i tried header connection: close but it was still active so i tried then removing ?properties etc. and it worked...
I'm using curl 7.54.1 (x86_64-apple-darwin15.6.0) to download a track from the soundcloud API in a bash script
The code looks like this -
# -vsLOJ = --verbose --silent --location --remote-name --remote-header-name
# redirect stderr to stdout to capture the headers
curl_output=$(curl -vsLOJ $track_download_url?client_id=$client_id 2>&1);
This is supposed to:
verbosely print out the request/response (to capture HTTP headers)
silence the download bar
follow the location (the API provides a pretty link that 302's to the actual file)
Create a file using "Content-Disposition" header as the file name (this becomes the output file, not stdout)
redirect stderr (where the verbose output is sent) to stdout
What's happening:
The download is OK, it saves the file to the working directory with the correct name from "Content-Disposition" but $curl_output is filled with garbage data of what looks like a mix of an ls of the working directory and partial verbose data.
Example output cURL-ing https://www.google.com in a test directory with files:
curl_output=$(curl --verbose -o /dev/null "https://www.google.com" 2>&1)
echo $curl_output
fakefile.1 fakefile.2 hello.txt song.mp3 vacation.png Rebuilt URL to:
https://www.google.com/ fakefile.1 fakefile.2 hello.txt song.mp3
vacation.png Trying 172.217.10.100... fakefile.1 fakefile.2 hello.txt
song.mp3 vacation.png TCP_NODELAY set fakefile.1 fakefile.2 hello.txt
song.mp3 vacation.png Connected to www.google.com (172.217.10.100)
port 443 (#0) fakefile.1 fakefile.2 hello.txt song.mp3 vacation.png
TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
fakefile.1 fakefile.2 hello.txt song.mp3 vacation.png Server
certificate: www.google.com fakefile.1 fakefile.2 hello.txt song.mp3
vacation.png Server certificate: Google Internet Authority G2
fakefile.1 fakefile.2 hello.txt song.mp3 vacation.png Server
certificate: GeoTrust Gl < Set-Cookie:
REDACTED=REDACTED######################################################################## 100.0%* Connection #0 to host www.google.com left intact
Completely confusing to me. I've tested this in a bash script and from Terminal. It only seems to be happening when I store the result in a variable, running that cURL (including the stderr redirect) without storing it in $curl_output will correctly write
And, this is happening for any URL I test with
My .curlrc:
user-agent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_19_4) AppleWebKit/603.4.1 (KHTML, like Gecko) Chrome/59.0.3050.56 Safari/603.4.1"
referer = ";auto"
connect-timeout = 10
progress-bar
max-time = 90
remote-time
Put quotes around your $curl_output variable, because it contains * which get interpreted.
% echo "$curl_output"
* Rebuilt URL to: https://www.google.com/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 2a00:1450:4009:806::2004...
* Connected to www.google.com (2a00:1450:4009:806::2004) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
Whereas:
echo $curl_output
would resolve the * into whatever files names are lying in your current directory.
I wrote a bash script that gets output from a website using curl and does a bunch of string manipulation on the html output. The problem is when I run it against a site that is returning its output gzipped. Going to the site in a browser works fine.
When I run curl by hand, I get gzipped output:
$ curl "http://example.com"
Here's the header from that particular site:
HTTP/1.1 200 OK
Server: nginx
Content-Type: text/html; charset=utf-8
X-Powered-By: PHP/5.2.17
Last-Modified: Sat, 03 Dec 2011 00:07:57 GMT
ETag: "6c38e1154f32dbd9ba211db8ad189b27"
Expires: Sun, 19 Nov 1978 05:00:00 GMT
Cache-Control: must-revalidate
Content-Encoding: gzip
Content-Length: 7796
Date: Sat, 03 Dec 2011 00:46:22 GMT
X-Varnish: 1509870407 1509810501
Age: 504
Via: 1.1 varnish
Connection: keep-alive
X-Cache-Svr: p2137050.pubip.peer1.net
X-Cache: HIT
X-Cache-Hits: 425
I know the returned data is gzipped, because this returns html, as expected:
$ curl "http://example.com" | gunzip
I don't want to pipe the output through gunzip, because the script works as-is on other sites, and piping through gzip would break that functionality.
What I've tried
changing the user-agent (I tried the same string my browser sends, "Mozilla/4.0", etc)
man curl
google search
searching stackoverflow
Everything came up empty
Any ideas?
curl will automatically decompress the response if you set the --compressed flag:
curl --compressed "http://example.com"
--compressed
(HTTP) Request a compressed response using one of the algorithms libcurl supports, and save the uncompressed document. If this option is used and the server sends an unsupported encoding, curl will report an error.
gzip is most likely supported, but you can check this by running curl -V and looking for libz somewhere in the "Features" line:
$ curl -V
...
Protocols: ...
Features: GSS-Negotiate IDN IPv6 Largefile NTLM SSL libz
Note that it's really the website in question that is at fault here. If curl did not pass an Accept-Encoding: gzip request header, the server should not have sent a compressed response.
In the relevant bug report Raw compressed output when not using --compressed but server returns gzip data #2836 the developers says:
The server shouldn't send content-encoding: gzip without the client having signaled that it is acceptable.
Besides, when you don't use --compressed with curl, you tell the command line tool you rather store the exact stream (compressed or not). I don't see a curl bug here...
So if the server could be sending gzipped content, use --compressed to let curl decompress it automatically.