For example, if I wanna issue a post request to the server. But the website requires the username and password to login first. How should I do these two operations?
If it's requires some programmatic username and password built into the web page, you'd need to submit what it expects for a user logging in, then capture the cookies you get, and then send those cookies back with your post. This can get involved if the login process involves multiple pages which are redirected to. curl can do this, but be prepared to spend some time on it.
To get the cookie being returned by the server, use curl -i to include headers. You can also add -L to automatically follow redirects (which you otherwise would have to do manually by retrieving the URI in the Location: field of an HTTP 301 or 302 response). Example:
curl -i -L stackoverflow.com > /tmp/so.html
grep -i 'Set-Cookie:' /tmp/so.html
Yields:
Set-Cookie: prov=31c24327-c0bf-474d-b504-fc97dc69ab61; domain=.stackoverflow.com; expires=Fri, 01-Jan-2055 00:00:00 GMT; path=/; HttpOnly
(Until you get the predictable logic right and how you need to submit the requests, you'll need to inspect the rest of the headers to be able to accomodate redirects, see if there are multple cookies, etc.)
To submit a cookie, use curl -b:
curl -b "prov=31c24327-c0bf-474d-b504-fc97dc69ab61" [rest of curl command]
Be patient and good luck, and be sure to check the curl man page.
curl -u username:password -X POST --data "name1=value1&name2=value2" http://yourwebpage.com/
Related
I am trying to obtain an authorization code from a Spring OAuth 2.0 authorization server using simple CURL command.
curl -v --header "Authorization: Basic hasfhfashfakhsfakhf712641246" "http://0.0.0.0:0000/oauth-server/oauth2/authorize?response_type=code&client_id=dummyclient&client_secret=dummyclient&redirect_uri=http://oauth2server/oauth2callback/"
However, instead of getting back a redirect URL with the code, I am getting back a 302 redirect response to the login URL. Since I am already sending the username, password in the Authorization Header, is there a way to skip the login page redirect and get the Authorization Code directly?
you might need to tell curl to follow redirect with the -L flag
curl -L -v --header "Authorization: Basic hasfhfashfakhsfakhf712641246" "http://0.0.0.0:0000/oauth-server/oauth2/authorize?response_type=code&client_id=dummyclient&client_secret=dummyclient&redirect_uri=http://oauth2server/oauth2callback/"
My understanding was that curl -i and curl -I would return virtually the same results except that curl -i would return the standard output along with the header and curl -I would only return the header -- the header of both being the same. We've been doing some gzip and un-gzipped testing with Varnish and stumbled upon the oddity that curl -i shows X-Cache: HIT but curl -I returns X-Cache: MISS! How this is possible, I am unsure and that is precisely my question in this post.
Here are some more details that may or may not make a difference:
The URL is usually SSL enforced (https) but both HTTP and HTTPS have been tested to receive same results
The results are consistent
Is Varnish Running site says "Yes! Sort of"
curl sends different HTTP requests to the server (or Varnish in this case) when you use the -I option. Normally, curl will send a GET request, but when you specify -I, it sends HEAD instead (essentially telling the server to just send the header, not the actual content). I'm not particularly familiar with Varnish, but it appears to normally cache both GET and HEAD requests -- but in your case it might be configured to do something different, or the backend server may be triggering a difference... In any case, I'm pretty sure it's GET vs. HEAD that's making the cache respond differently with -i vs. -I.
did you check in different orders?
see: http://anothersysadmin.wordpress.com/2008/04/22/x-cache-and-x-cache-lookup-headers-explained/ for some details on X-Cache
I found this script for authenticating with cas to obtain a protected URL, from the commandline, using curl and bash(See code below). However, I could not get the script to work. I have verified that I am able to extract the LoginTicket, the JSESSION and provided the right username and password. However, it seems the cas-server does not react to it even though I have verified I provide it with all the right info. It just returns the login-page again and again without any error messages.
Is this script still a viable way of doing this? Or do I need to use the cas REST-API, if I want to get a valid cas ticket using the commandline, now adays?
# Taken from https://gist.github.com/dodok1/4134605
# Usage: cas-get.sh {url} {username} {password} # If you have any errors try removing the redirects to get more information
# The service to be called, and a url-encoded version (the url encoding isn't perfect, if you're encoding complex stuff you may wish to replace with a different method)
DEST="$1"
ENCODED_DEST=`echo "$DEST" | perl -p -e 's/([^A-Za-z0-9])/sprintf("%%%02X", ord($1))/seg' | sed 's/%2E/./g' | sed 's/%0A//g'`
#IP Addresses or hostnames are fine here
CAS_HOSTNAME=team.eea.sk
#Authentication details. This script only supports username/password login, but curl can handle certificate login if required
USERNAME=$2
PASSWORD=$3
#Temporary files used by curl to store cookies and http headers
COOKIE_JAR=.cookieJar
HEADER_DUMP_DEST=.headers
rm $COOKIE_JAR
rm $HEADER_DUMP_DEST
#The script itself is below
#Visit CAS and get a login form. This includes a unique ID for the form, which we will store in CAS_ID and attach to our form submission. jsessionid cookie will be set here
CAS_ID=`curl -s -k -c $COOKIE_JAR https://$CAS_HOSTNAME/cas/login?service=$ENCODED_DEST | grep name=.lt | sed 's/.*value..//' | sed 's/\".*//'`
#Submit the login form, using the cookies saved in the cookie jar and the form submission ID just extracted. We keep the headers from this request as the return value should be a 302 including a "ticket" param which we'll need in the next request
curl -s -k --data "username=$USERNAME&password=$PASSWORD<=$CAS_ID&_eventId=submit" -i -b $COOKIE_JAR -c $COOKIE_JAR https://$CAS_HOSTNAME/cas/login?service=$ENCODED_DEST -D $HEADER_DUMP_DEST -o /dev/null
#Linux may not need this line but my response from the previous call has retrieving windows-style linebreaks in OSX
#dos2unix $HEADER_DUMP_DEST > /dev/null
#Visit the URL with the ticket param to finally set the casprivacy and, more importantly, MOD_AUTH_CAS cookie. Now we've got a MOD_AUTH_CAS cookie, anything we do in this session will pass straight through CAS
CURL_DEST=`grep Location $HEADER_DUMP_DEST | sed 's/Location: //'`
curl -s -k -b $COOKIE_JAR -c $COOKIE_JAR $CURL_DEST
#If our destination is not a GET we'll need to do a GET to, say, the user dashboard here
#Visit the place we actually wanted to go to
curl -s -k -b $COOKIE_JAR "$DEST"
You might try extracting the "execution" value like you do the "lt" value and including it in the second curl call.
Just trying to play with google app scripts. In anonymous mode things seem fine. Except that anyone can call my script simply like that snippet shows:
curl "https://script.google.com/macros/s/.../exec?ip=\"$myIp\""
I used this manual for tips on how to authenticate through GoogleLogin. The problem is "401 Unauthorized" I received when sent auth token and "Me(owner)/Only myself" options were set on google side. (The token seems correct itself. If I omit password or mistype it, then I receive "Bad auth") If I set "Anyone, even anonymous" again, it works, but auth stuff seems like ignored. What's the correct way to do the trick?
#!/bin/bash
gmail=$1
password=$2
myIp=$3
GoogleAuthToken=""
GoogleAuthToken=`curl --silent https://www.google.com/accounts/ClientLogin --data-urlencode Email=$gmail \
--data-urlencode Passwd=$password -d accountType=GOOGLE -d source=YouDontSay -d service=lh2`
echo $GoogleAuthToken
GoogleAuthToken=$(echo "$GoogleAuthToken" | grep 'Auth=' | sed s/Auth=//)
echo $GoogleAuthToken
curl -L --silent --header "Authorization: GoogleLogin auth=$GoogleAuthToken" "https://script.google.com/macros/s/.../exec?ip=\"$myIp\""
You use ClientLogin
https://www.google.com/accounts/ClientLogin
This is google error :
Important: ClientLogin has been officially deprecated since April 20,
2012 and is now no longer available. Requests to ClientLogin will fail
with a HTTP 404 response. We encourage you to migrate to OAuth 2.0 as
soon as possible.
Just been tinkering with Sinatra and trying to get a bit of a restful web service going.
The error I'm getting at the moment is very specific though.
Take this example post method
post '/postMan/:someParam' do
#Edited here. This code can be anything. 411 is still the response
puts params[:someParam]
end
Seems simple enough. Take a param, make an object out of it, then go store it in whatever way the objects save method defines.
Heres what I use to post the data using Curl
$curl -I -X POST http://127.0.0.1/postman/123456
The only problem is, I'm getting 411 back and have no idea why.
To the best of my knowledge, 411 is length required. Here is the trace
HTTP/1.1 411 Length Required
Content-Type: text/html; charset=ISO-8859-1
Server: WEBrick/1.3.1 (Ruby/1.9.2/2011-07-09)
Date: Fri, 02 Mar 2012 22:27:09 GMT
Content-Length: 303
Connection: close
I 'Cannot' change the curl message in any way. So might anyone have a way to set the content length to be ignored in sinatra? Or some fix which doesn't involve changing the curl request?
For the record, it doesn't matter whether I use the parameters in the Post method or not. I could have some crazy code inside it, it will still throw the same error
As others said above, WEBrick wrongly requires POST requests to have a Content-Length header. I just pass an empty body, because it's less typing than passing in the header:
curl -X POST -d '' http://webrickwhyyounotakeemptyposts.com/
Are you sure you're on port 80 for your app?
When I run:
ruby -r sinatra -e "post('/postMan/:someParam'){puts params[:someParam]}"
and curl it:
curl -I -X POST http://127.0.0.1:4567/postMan/123456
HTTP/1.1 200 OK
X-Frame-Options: sameorigin
X-XSS-Protection: 1; mode=block
Content-Type: text/html;charset=utf-8
Content-Length: 0
Connection: keep-alive
Server: thin 1.3.1 codename Triple Espresso
it's ok. Had to change the URL to postManthough, your example threw a 404because you had postman.
The output was also as expected:
== Sinatra/1.3.2 has taken the stage on 4567 for development with backup from Thin
>> Thin web server (v1.3.1 codename Triple Espresso)
>> Maximum connections set to 1024
>> Listening on 0.0.0.0:4567, CTRL+C to stop
123456
Ah. Try it without -I. It's probably sending a HEAD request and as such, not sending what you expect. Use -v if you want to show the headers.
curl -v -X POST http://127.0.0.1/postman/123456
curl -v -X POST -d "key=val" http://127.0.0.1/postman/123456
WEBrick erroneously requires POST requests to include the Content-Length header.
curl -H 'Content-Length: 0' -X POST http://example.com
Standardly, however, POST requests don't require a body and therefore don't require the Content-Length header.