why curl cannot display http header by using option -i - shell

I use curl tool to get something within http protocol, and intent to use option -i to display http-header. There's output message without http-header, only http-body from server in the terminal.

Following this question, you can try ---verbose option instead of -i
As the comment by cfeduke in the question mentioned, it depends on the response of the server as well.

Related

How to append a file with the existing one using CURL?

I've searched all over the web but couldn't find anything useful for this. Using CURL, I want to append content to a file which is already existing and named as test.pls. Can anyone please tell me how can I do it using curl. The commands I've tried are
curl http://192.99.8.170:8098/stream --output test.pls
curl -K --output test.pls http://192.99.8.170:8098/stream
curl -a --output test.pls http://192.99.8.170:8098/stream
But all of above starts creating files from scratch.They don't keep the initial content of file Can anyone please help me!
Use the shell's appending output redirection (>>) rather than curl's --output option.
curl http://192.99.8.170:8098/stream >> test.pls
A much easier and cleaner way is as follows:
curl -sS https://address.to.file.txt >> file-name.txt

Different body for same Curl request sent from Windows and Ubuntu

I'm trying to send a curl request from both Windows and Ubuntu system to a Rest API. following is the request
curl -k -X POST http://172.16.76.1:8080/test -d 'sample_param={"user_info":{"name":"abc","age":"20"}}'
When I read this from the server side, I get the following two different content data from each OS
Body for curl request from Ubuntu:
sample_param={"user_info":{"name":"abc","age":"20"}}
Body for curl request from Windows:
sample_param={user_info:{name:abc,age:20}}
(Note that double quotations are missing)
As a result I cannot get the json object from the request.
Can someone point out the mistake and give a solution for this.
Thanks in advance
Changing the curl command to following worked
curl -k -X POST http://172.16.76.1:8080/test -d "sample_param={\"user_info\":{\"name\":\"abc\",\"age\":\"20\"}}"

socat fake http server - use a file as server response

I've been trying to use socat to respond on each connection to a socket it's listening to with a fake HTTP reply. I cannot get it working. It might be because I'm using the cygwin version of socat? I don't know.
Part of the problem is I want the second argument <some_file_response> not to be written to. In other words because it's bidirectional it will read what's in response.txt and then write to that same file, and I don't want that. Even if I do open:response.txt,rdonly it doesn't work repeatedly. system: doesn't seem to do anything. exec seems like it works, for example I can do exec:'cat response.txt' but that never gets sent to the client connecting to port 1234.
socat -vv tcp-listen:1234,reuseaddr,fork <some_file_response>
I want it to read a file to the client that's connected and then close the connection, and do that over and over again (that's why I used fork).
I am putting a bounty on this question. Please only give me solutions that work with the cygwin version from the windows command prompt.
Tested with cygwin:
socat -vv TCP-LISTEN:1234,crlf,reuseaddr,fork SYSTEM:"echo HTTP/1.0 200; echo Content-Type\: text/plain; echo; cat <some_file_response>"
If you do not want a complete HTTP response, leave out the echos:
socat -vv TCP-LISTEN:1234,crlf,reuseaddr,fork SYSTEM:"cat <some_file_response>"
Taken from socat examples
socat -vv TCP-LISTEN:8000,crlf,reuseaddr,fork SYSTEM:"echo HTTP/1.0 200; echo Content-Type\: text/plain; echo; cat"
This one works:
socat -v -v -d -d TCP-LISTEN:8080,reuseaddr,fork exec:"cat http.response",pipes
Two things need to be aware,
should you add crlf, as in other answers. I recommend not.
crlf caused problem sending image
just use \r\n explicitly in http response headers.
without pipes, seems no data sent to client. browser complains:
127.0.0.1 didn’t send any data.
ERR_EMPTY_RESPONSE
tested in cygwin.
== EDIT ==
If you want use inside cmd.exe, make sure PATH is correctly set, so that socat and cat can be found.
Say both socat.exe and cat.exe located under E:\cygwin64\bin
set PATH=%PATH%;E:\cygwin64\bin
Works in cmd.exe, with socat & cat from cygwin.

Bash - download a file, log in if required

I'm using BASH and I need to download a TXT file, which is generated on server-side by request. This means the URL is something like:
http://1.1.1.1:4884/page.aspx?fileID=123456&lang=en&Export=1
Export=1 is caught by the .NET application and I'm provided with a TXT file, based on fileID.
In case I haven't logged in, I'm redirected to a login form with ?ReturnUrl in the URL, redirecting me back to my requested page upon login.
How can I successfully download this file using BASH, cURL/wget/lynx. It has to be non-interactive.
I've tried using the --cookie options for curl and wget and lynx automation (cmd-log). Lynx worked best, but for some reason, the file download prompt could not be automated.
Please help. If any additional info is required, I will provide.
Use curl.
Code following approach:
try to download the file
if failed (redirected to login page), log in and go to begin
You always need to use -c option of curl to store the cookies between curl calls
To log in using curl you need to know the form on the server, that means: names of fields where you usually type your login and password.
To send the data to server use -d option of curl. To send the cookie to server use -b (or --cookie).

command line web browsing

Is there a way to perform http commands GET/PUT/SET whatever via a command line in ubuntu or windows xp? Preferably without installing 3rd party products. Being that http is text based I thought it would be alot easier to run in the cmd line.
I've been able to get what I want out of GET in ubuntu in bash via
$wget google.com
$cat index.html
This is kinda clunky. It would be nice to pipe the output or something, but even that isn't straight forward. C programs are fine too. I'm trying to do something like what we get with Fiddler, but more basic.
telnet google.com 80
GET / HTTP/1.0
Host: google.com
You have to hit return twice after the Host line. It doesn't get any more basic.
If you are familiar with HTTP use telnet.
If you are looking for a browser take a look for Links.
Although it requires a 3rd party tool, these days I use curl. The -X option allows me to specify the HTTP verb. Windows has a few bash clients that allow you to run curl including Cygwin.
Sample Execution
$ curl -H "Content-Type: application/json" -X POST -d '{value: "600"}' http://localhost:8888/my/endpoint

Resources