I'm writing a small C application to work with Google Keep. I used Python-written gkeepapi which in turn uses gpsoauth as a reference. All HTTPS-related code can be found here.
For HTTPS I use libcurl (and also test it from curl CLI). For master login I perform a query with the following body (stripped):
accountType=HOSTED_OR_GOOGLE&Email=USERNAME%40gmail.com&has_permission=1&add_account=1&EncryptedPassw=AFcb4KRFBAu...6_9jal31XKdZHZkg%3D%3D&service=ac2dm&source=android&androidId=31021392710718&device_country=us&operatorCountry=us&lang=en&sdk_version=17&client_sig=38918a453d07199354f8b19af05ec6562ced5788&callerSig=38918a453d07199354f8b19af05ec6562ced5788&droidguard_results=dummy123
This unfortunately results in 403 with "Error=BadAuthentication".
The same query with same body and seemingly same SSL context options, headers, etc. from gpsoauth works just fine:
>>> import gkeepapi; keep = gkeepapi.Keep(); keep.login('USER#gmail.com', '...')
accountType=HOSTED_OR_GOOGLE&Email=USER%40gmail.com&has_permission=1&add_account=1&EncryptedPasswd=AFcb4KRFBAu...6_9jal31XKdZHZkg%3D%3D&service=ac2dm&source=android&androidId=31021392710718&device_country=us&operatorCountry=us&lang=en&sdk_version=17&client_sig=38918a453d07199354f8b19af05ec6562ced5788&callerSig=38918a453d07199354f8b19af05ec6562ced5788&droidguard_results=dummy123
SID=BAD_COOKIE
LSID=BAD_COOKIE
Auth=TwiJFLqwaaPQrJCtBbiDfK9CUdJN6o4GyHMY4h8B-vTEuL8WAR2M...Gwu5Yufw6gDfxPZqBpw02OIcloqccdDrt5F-9DkRAk9J79Sgf3gTZI1Qes5chgs3b3r5QmIw702ckA=
services=mail,friendview,android,youtube,cl,talk,memento,mymaps,omaha,sierra,googleplay,print,blogger,multilogin,hist,jotspot,ah,dynamite,uif,g1phonebackup,domains
firstName=...
And so on. To double check I even tried hard-coding my generated body into Python code - and it succeeds there as well.
libcurl-related code in my small program:
static CURL* curl_setup(void) {
CURL* curl;
curl = curl_easy_init();
if (!curl) panic("curl_easy_init() failed");
headers = curl_slist_append(NULL, "User-Agent: GoogleAuth/1.4");
headers = curl_slist_append(
headers, "Content-type: application/x-www-form-urlencoded");
curl_easy_setopt(curl, CURLOPT_POST, 1L);
curl_easy_setopt(curl, CURLOPT_URL,
"https://android.clients.google.com/auth");
curl_easy_setopt(curl, CURLOPT_HTTPHEADER, headers);
curl_easy_setopt(curl, CURLOPT_SSL_CIPHER_LIST, "ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20:ECDH+AESGCM:DH+AESGCM:ECDH+AES:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5:!DSS");
curl_easy_setopt(curl, CURLOPT_SSL_ENABLE_ALPN, 0L);
return curl;
}
...
curl_easy_setopt(curl, CURLOPT_POSTFIELDS, body);
resp = curl_easy_perform(curl);
OpenSSL library compiled into Python and curl is of the same version:
bash-5.1$ python3.9 -c "import ssl; print(ssl.OPENSSL_VERSION)"
OpenSSL 1.1.1m 14 Dec 2021
bash-5.1$ curl -V
curl 7.81.0 (x86_64-pc-linux-gnu) libcurl/7.81.0 OpenSSL/1.1.1m zlib/1.2.11 brotli/1.0.9 zstd/1.5.2 c-ares/1.18.1 libidn2/2.3.2 libpsl/0.21.1 (+libidn2/2.3.0) libssh2/1.10.0 nghttp2/1.46.0 OpenLDAP/2.4.59
So far I have no new ideas for now :-) My primary suspect is some TLS-level incompatibility but at glance I have found nothing. Any suggestion on further debugging would be welcome!
Thanks!
Related
How can I get the cookie from a request in playframework?
I have the following test endpoint
def home = Action.async { implicit request =>
println(request)
println(request.session)
println(request.flash)
request.session.get("session") match {
case Some(cookie) => Future(Ok(cookie))
case None =>
Future(BadRequest(Json.obj("message" -> "missing session cookie")))
}
}
When submitting the following request:
curl 'http://local.example.com:9000/home' -H 'User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Accept-Language: en-US,en;q=0.5' --compressed -H 'Connection: keep-alive' -H 'Cookie: session=eyJhbGciOiJSUzI1NiIsImtpZCI...' -H 'Upgrade-Insecure-Requests: 1' -H 'Cache-Control: max-age=0'
I unfortunately get the "missing session cookie" response. and the following printout on the console
GET /home
Session(Map())
Flash(Map())
I don't know what I'm doing wrong. Any help is much appreciated.
Edit: I set the cookie using the following method:
def tokenLogin = Action(parse.json).async { implicit request =>
val loginRequest = request.body.validate[LoginRequest]
loginRequest.fold(
errors =>
{
println(errors)
Future(BadRequest(Json.obj("message" -> JsError.toJson(errors))))
},
request => {
println("in success")
firebaseAdminService
.createSessionCookie(request.idToken)
.map(sessionCookie =>
Ok("success")
.withNewSession
.withCookies(Cookie(name = "session", value = sessionCookie))
)
}
)
}
By default, the session cookie in Play is called "PLAY_SESSION" (configuration play.http.session.cookieName).
So, you would need to use -H "Cookie: PLAY_SESSION=..." with curl.
But note, this won't work with arbitrary data since Play uses JWT and signs the information contained in the session cookie using its crypto secret.
The only thing expected to work is using a session cookie received in a Set-Cookie header from your Play service in another request to the same service (having the same secret).
update after your edit:
When using request.session, you are accessing the session cookie, which is called PLAY_SESSION and the information stored inside it.
But, you are setting a cookie of your own. This is something else.
You can access "normal" cookies with
request.cookies.get("session")
Oh, and in case you really wanted to make use of the session cookie, you can set it like this:
Ok("success").withSession("session" -> sessionCookie)
I have a groovy script that work on Linux Jenkins
import groovy.json.JsonSlurper
try {
List<String> artifacts = new ArrayList<String>()
//jira get summery for list by issue type story and label demo and project 11411
def artifactsUrl = 'https://companyname.atlassian.net/rest/api/2/search?jql=project=11411%20and%20issuetype%20in%20(Story)%20and%20labels%20in%20(demo)+&fields=summary' ;
def artifactsObjectRaw = ["curl", "-u", "someusername#xxxx.com:tokenkey" ,"-X" ,"GET", "-H", "Content-Type: application/json", "-H", "accept: application/json","-K", "--url","${artifactsUrl}"].execute().text;
def parser = new JsonSlurper();
def json = parser.parseText(artifactsObjectRaw );
//insert all result into list
for(item in json.issues){
artifacts.add( item.fields.summary);
}
//return list to extended result
return artifacts ;
}catch (Exception e) {
println "There was a problem fetching the artifacts " + e.message;
}
This script return all the names from Jira jobs by the API ,
But when I tried to run this groovy on Windows Jenkins the script will not work because windows do not have the command curl
def artifactsObjectRaw = ["curl", "-u","someusername#xxxx.com:tokenkey" ,"-X" ,"GET", "-H", "Content-Type: application/json", "-H", "accept: application/json","-K","--url","${artifactsUrl}"].execute().text;
how should I preform this command?
The following code:
import groovy.json.JsonSlurper
try {
def baseUrl = 'https://companyname.atlassian.net'
def artifactsUrl = "${baseUrl}/rest/api/2/search?jql=project=MYPROJECT&fields=summary"
def auth = "someusername#somewhere.com:tokenkey".bytes.encodeBase64()
def headers = ['Content-Type': "application/json",
'Authorization': "Basic ${auth}"]
def response = artifactsUrl.toURL().getText(requestProperties: headers)
def json = new JsonSlurper().parseText(response)
// the below will implicitly return a list of summaries, no
// need to define an 'artifacts' list beforehand
def artifacts = json.issues.collect { issue -> issue.fields.summary }
} catch (Exception e) {
e.printStackTrace()
}
is pure groovy, i.e. no need for curl. It gets the items from the jira instance and returns a List<String> of summaries. Since we don't want any external dependencies like HttpBuidler (as you are doing this from jenkins) we have to manually do the basic auth encoding.
Script tested (the connecting and getting json part, did not test the extraction of summary fields) with:
Groovy Version: 2.4.15 JVM: 1.8.0_201 Vendor: Oracle Corporation OS: Linux
against an atlassian on demand cloud instance.
I removed your jql query as it didn't work for me but you should be able to add it back as needed.
Install curl and set the path in environment variable of windows.
Please follow the link to download curl on windows.
I would consider using HTTP request plugin when making HTTP Requests.
Since you are using a plugin, it does not matter if you are running in Windows or .
Linux as your Jenkins Host
With cURL, we can pass a username with an HTTP web request as follows:
$ curl -u <your_username> https://api.github.com/user
The -u flag accepts a username for authentication, and then cURL will request the password. The cURL example is for Basic authentication with the GitHub Api.
How do we similarly pass a username and password along with Invoke-WebRequest? The ultimate goal is to user PowerShell with Basic authentication in the GitHub API.
I am assuming Basic authentication here.
$cred = Get-Credential
Invoke-WebRequest -Uri 'https://whatever' -Credential $cred
You can get your credential through other means (Import-Clixml, etc.), but it does have to be a [PSCredential] object.
Edit based on comments:
GitHub is breaking RFC as they explain in the link you provided:
The API supports Basic Authentication as defined in RFC2617 with a few
slight differences. The main difference is that the RFC requires
unauthenticated requests to be answered with 401 Unauthorized
responses. In many places, this would disclose the existence of user
data. Instead, the GitHub API responds with 404 Not Found. This may
cause problems for HTTP libraries that assume a 401 Unauthorized
response. The solution is to manually craft the Authorization header.
Powershell's Invoke-WebRequest does to my knowledge wait for a 401 response before sending the credentials, and since GitHub never provides one, your credentials will never be sent.
Manually build the headers
Instead you'll have to create the basic auth headers yourself.
Basic authentication takes a string that consists of the username and password separated by a colon user:pass and then sends the Base64 encoded result of that.
Code like this should work:
$user = 'user'
$pass = 'pass'
$pair = "$($user):$($pass)"
$encodedCreds = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($pair))
$basicAuthValue = "Basic $encodedCreds"
$Headers = #{
Authorization = $basicAuthValue
}
Invoke-WebRequest -Uri 'https://whatever' -Headers $Headers
You could combine some of the string concatenation but I wanted to break it out to make it clearer.
Use this:
$root = 'REST_SERVICE_URL'
$user = "user"
$pass= "password"
$secpasswd = ConvertTo-SecureString $pass -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($user, $secpasswd)
$result = Invoke-RestMethod $root -Credential $credential
If someone would need a one liner:
iwr -Uri 'https://api.github.com/user' -Headers #{ Authorization = "Basic "+ [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes("user:pass")) }
Invoke-WebRequest follows the RFC2617 as #briantist noted, however there are some systems (e.g. JFrog Artifactory) that allow anonymous usage if the Authorization header is absent, but will respond with 401 Forbidden if the header contains invalid credentials.
This can be used to trigger the 401 Forbidden response and get -Credentials to work.
$login = Get-Credential -Message "Enter Credentials for Artifactory"
#Basic foo:bar
$headers = #{ Authorization = "Basic Zm9vOmJhcg==" }
Invoke-WebRequest -Credential $login -Headers $headers -Uri "..."
This will send the invalid header the first time, which will be replaced with the valid credentials in the second request since -Credentials overrides the Authorization header.
Tested with Powershell 5.1
I had to do this to get it to work:
$pair = "$($user):$($pass)"
$encodedCredentials = [System.Convert]::ToBase64String([System.Text.Encoding]::ASCII.GetBytes($Pair))
$headers = #{ Authorization = "Basic $encodedCredentials" }
Invoke-WebRequest -Uri $url -Method Get -Headers $headers -OutFile Config.html
Here is another way using WebRequest, I hope it will work for you
$user = 'whatever'
$pass = 'whatever'
$secpasswd = ConvertTo-SecureString $pass -AsPlainText -Force
$credential = New-Object System.Management.Automation.PSCredential($user, $secpasswd)
$headers = #{ Authorization = "Basic Zm9vOmJhcg==" }
Invoke-WebRequest -Credential $credential -Headers $headers -Uri "https://dc01.test.local/"
This is what worked for our particular situation.
Notes are from Wikipedia on Basic Auth from the Client Side. Thank you to #briantist's answer for the help!
Combine the username and password into a single string username:password
$user = "shaunluttin"
$pass = "super-strong-alpha-numeric-symbolic-long-password"
$pair = "${user}:${pass}"
Encode the string to the RFC2045-MIME variant of Base64, except not limited to 76 char/line.
$bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
$base64 = [System.Convert]::ToBase64String($bytes)
Create the Auth value as the method, a space, and then the encoded pair Method Base64String
$basicAuthValue = "Basic $base64"
Create the header Authorization: Basic QWxhZGRpbjpvcGVuIHNlc2FtZQ==
$headers = #{ Authorization = $basicAuthValue }
Invoke the web-request
Invoke-WebRequest -uri "https://api.github.com/user" -Headers $headers
The PowerShell version of this is more verbose than the cURL version is. Why is that? #briantist pointed out that GitHub is breaking the RFC and PowerShell is sticking to it. Does that mean that cURL is also breaking with the standard?
another way is to use certutil.exe
save your username and password in a file e.g. in.txt as username:password
certutil -encode in.txt out.txt
Now you should be able to use auth value from out.txt
$headers = #{ Authorization = "Basic $((get-content out.txt)[1])" }
Invoke-WebRequest -Uri 'https://whatever' -Headers $Headers
I know this is a little off the OPs original request but I came across this while looking for a way to use Invoke-WebRequest against a site requiring basic authentication.
The difference is, I did not want to record the password in the script. Instead, I wanted to prompt the script runner for credentials for the site.
Here's how I handled it
$creds = Get-Credential
$basicCreds = [pscredential]::new($Creds.UserName,$Creds.Password)
Invoke-WebRequest -Uri $URL -Credential $basicCreds
The result is the script runner is prompted with a login dialog for the U/P then, Invoke-WebRequest is able to access the site with those credentials. This works because $Creds.Password is already an encrypted string.
I hope this helps someone looking for a similar solution to the above question but without saving the username or PW in the script
I am performing the following POST in a Tower server:
http://<my-tower-url>/api/v2/job_templates/10/launch/
Headers:
Content-Type:application/json
Authorization:sometokenhere
And getting back the error:
{"detail":"Authentication credentials were not provided."}
Have also tried the following:
Headers:
Content-Type:application/json
Authorization:Token sometokenhere
as suggested here.
Same happens when passing raw username/password in the POST body as follows (and skipping the Authorization header):
{
"username": "myusername",
"password": "mypass",
"inventory": "inventoryname",
"verbosity": 0,
"extra_vars": {
"var1": "somevar1",
"var2": "somevar2",
"var3": "somevar3",
"var4": "somevar4",
"var5": "somevar5"
}
}
Any idea why this is not working?
Authorization: Bearer <oauth2-token-value>
See here, Section "3. OAuth 2 Token Authentication", part "Curl Example".
I ended up using basic auth as follows:
1.create the user which you want to run your ci jobs with
2.perform the following post at the respective CI job:
curl -o /dev/null -s -w \"%{http_code}\n\" -X POST http://<my-tower-url>/api/v2/job_templates/10/launch/ \
-H \"authorization: Basic $MY_AUTH_TOKEN\" \
-H \"content-type: application/json\" \
-d \"#awx_data.json
Where
awx_data.json is a file holding the actual POST body
MY_AUTH_TOKEN is the tyical base64 encoded username+password of the above user
You can also assign the above result and check it against 201 which is what AWX returns upon successful job creation.
Polling the AWX server to check if the job was successfully finished is another story of course.
I'm using the following code to read an RSS feed and output the results.
function home_page_parser($feedURL) {
$rss = simplexml_load_file($feedURL);
$i = 0;
echo "<ul>";
foreach ($rss->channel->item as $feedItem) {
$i++;
$myDate = ($feedItem->pubDate);
$dateForm = explode(" ", $myDate);
echo "<li class=\"rss-feed\">".$feedItem->title."<br />" .$feedItem->pubDate. "</li>";
if($i >= 3) break;
echo "</ul>";
}
}
It is working fine on my testing site at Rackspace Cloud running PHP 5.2
On the live site at Media Temple running PHP 5.3, I get the following errors:
Warning: simplexml_load_file() [function.simplexml-load-file]: http:// wrapper is disabled in the server configuration by allow_url_fopen=0 in /.../html/includes/functions.php on line 39
Warning: simplexml_load_file(http://www.chinaknowledge.com/Newswires/RSS_News/RSS_News.xml) [function.simplexml-load-file]: failed to open stream: no suitable wrapper could be found in /.../html/includes/functions.php on line 39
Warning: simplexml_load_file() [function.simplexml-load-file]: I/O warning : failed to load external entity "http://www.chinaknowledge.com/Newswires/RSS_News/RSS_News.xml" in /.../html/includes/functions.php on line 39
Warning: Invalid argument supplied for foreach() in /.../html/includes/functions.php on line 44
Line 39 is this:
$rss = simplexml_load_file($feedURL);
What am I doing wrong or needs to change to work on 5.3?
The error is pretty descriptive dont you think?
http:// wrapper is disabled in the server configuration by
allow_url_fopen=0
You will need to edit the PHP configuration file and change the configuration allow_url_fopen. If you cant do this directly try ini_set()
Edit: As #evanmcd pointed out in the comments, this configuration can only be set in php.ini. PHP documentation reference.
This error comes due to "http:// wrapper is disabled in the server configuration by allow_url_fopen=0" .For avoiding this issue we need to override this setting to On instead off.In my view most of shared hosting servers do not allow you to do these setting through either ini_set('allow_url_fopen', 'on'); or htaccess overriding.So instead of trying these methods I suggest a way to fetch that feed is as follows.Using CURL we need to fetch the content of feed xml to a variable.Then process our simplexml file operations .
Example
$feed ='http://api.twitter.com/1/statuses/user_timeline.rss?screen_name=mytwittername';
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $feed);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// get the result of http query
$output = curl_exec($ch);
curl_close($ch);
$xml = simplexml_load_file($output);
If you are not allowed to edit php.ini in server you can use curl to get xml and read xml stirng as below.
function home_page_parser($feedURL) {
$rss = simplexml_load_file(curlXML($feedURL);
$i = 0;
echo "<ul>";
foreach ($rss->channel->item as $feedItem) {
$i++;
$myDate = ($feedItem->pubDate);
$dateForm = explode(" ", $myDate);
echo "<li class=\"rss-feed\">".$feedItem->title."<br />" .$feedItem->pubDate. "</li>";
if($i >= 3) break;
echo "</ul>";
}
}
function curlXML($url){
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
// get the result of http query
$output = curl_exec($ch);
curl_close($ch);
return $output;
}
ini_set("allow_url_fopen", 1);
This will set allow url open = On in php.ini file but you need to restart php in easyphp or xamp or wamp or in hosting.