I was on almaconnect.com, on home page there is a textbox which auto-suggest some results of universities when you type (load content by making an ajax call). I did make a curl request of same ajax call but request resulted in some encrypted lines on terminal
curl 'https://www.almaconnect.com/suggestions/portaled_institute?q=am' -H 'Host: www.almaconnect.com' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: en-US,en;q=0.5' -H 'Accept-Encoding: gzip, deflate, br' -H 'X-Requested-With: XMLHttpRequest' -H 'Referer: https://www.almaconnect.com/' -H 'Cookie: Almaconnect=; _ga=GA1.2.315358219.1489989532; __utma=117457241.315358219.1489989532.1490871434.1492414070.3; __utmz=117457241.1490871434.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _gat=1; __utmb=117457241.1.10.1492414070; __utmc=117457241; __utmt=1'
I want exactly the same functionality for my website so that if any user try to fetch my website data , he would not be able to.
Whatever binary data you see in the terminal when you make the curl call is not encrypted content. It is just compressed content. You can verify it by running
curl $params > output
You can check if the file matches any known file formats by running
file output
You will see that the result as something similar to
output: gzip compressed data, from Unix
Running gzip -d -c output will decompress and print the plaintext content to the terminal screen.
Reason
The reason why this happens is because, you send the accept-encoding header with the curl call. Unlike the browser, curl does not decompress the result automatically. That is the reason for this confusion.
-H 'Accept-Encoding: gzip, deflate, br'
Removing this particular header from the curl call will get you the response in an uncompressed plaintext format directly. You can try the following command for that.
curl 'https://www.almaconnect.com/suggestions/portaled_institute?q=am' -H 'Host: www.almaconnect.com' -H 'User-Agent: Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:44.0) Gecko/20100101 Firefox/44.0' -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Accept-Language: en-US,en;q=0.5' -H 'X-Requested-With: XMLHttpRequest' -H 'Referer: https://www.almaconnect.com/' -H 'Cookie: Almaconnect=; _ga=GA1.2.315358219.1489989532; __utma=117457241.315358219.1489989532.1490871434.1492414070.3; __utmz=117457241.1490871434.2.2.utmcsr=google|utmccn=(organic)|utmcmd=organic|utmctr=(not%20provided); _gat=1; __utmb=117457241.1.10.1492414070; __utmc=117457241; __utmt=1'
Summary
almaconnect.com does not really take any extra steps to obfuscate their AJAX responses. And it is generally a bad idea to do so. Whatever method you employ to obfuscate your responses (like using HTTP Referrer field), people can always come up with counter-measures to defeat them.
It is simply not worth the time to put in effort and time to come up with a mechanism which would eventually be broken by a determined attacker.
It is not possible.
The answer from gtux well explains the reasons why you are seeing binary characters of compressed content, not of encrypted content.
Note that this very simple version works:
curl 'https://www.almaconnect.com/suggestions/portaled_institute?q=am'
The answer from gaganshera may show you a way to obfuscate content, but that doesn't mean to really protect content, just to make a little harder for people to see it, since the decryption code is in public pages.
Your site can be protected by security (login + set cookie) or be public. If is protected, the security code checks the cookie header. If is public there are only ways to obfuscate content, not to protect it.
https://stackoverflow.com/a/14570971/1536382
https://www.quora.com/How-can-we-hide-JSON-data-from-tools-like-Chrome-development-tools-and-Firebug-etc-as-a-security-beyond-https
The server is returning an encrypted response which after its receipt is decrypted at the client end using javascript. You can do the same by sending your server response in some encryption that will be then decrypted at the client side. An example of the same can be crypto js.
Have a look at this: Encrypt with PHP, Decrypt with Javascript (cryptojs)
Maybe they are checking the HTTP_REFERER value for the ajax request. If the HTTP_REFERER is not the website, then it can return an encrypted response.
The ajax calls can also be secured using time based tokens. For example when a web page is requested, a random string may be generated on the server and stored in database. This string is sent to the client, which uses it in the ajax request. The server can then check if the token has expired or not. This method allows the ajax call to be valid for a certain time.
Related
I have a 3rd party REST, which I am successfully able to call like this using CURL (shell). This API return JSON. I tried calling same API, by changing content type to application/x-www-form-urlencoded but it doesn't work. I think I am forced to use content-type: multipart/form-data
curl --request POST --url https://************************* --header 'Authorization: Bearer ********' --header 'content-type: multipart/form-data; ' --form cluster_id=0717-035521-puny598 --form start_time=1534357800000 --form end_time=1534444199999 --form order=ASC --form limit=500
Now I want to call same API using InvokeHTTP processor (NiFi). So I configured it as follows. But I am not able to do a successful call. (it is not a proxy issue).
Following is how I am creating POST body (by FF)
I have tried replacing "enter" by \r\n etc, or changing body as name1=val1&name2=val2&.... etc. nothing worked.
This is the response I am getting.
[
I am able to run CURL (shell, from same server where Nifi is running). Also I am able to access url via postman.
[
finally, it worked. flow is somewhat like this.
GenerateFlowFile->UpdateAttribute->AttributesToJSON->InvokeHTTP
Only change, I made to InvokeHTTP. reverted content-type back to ${mime.type}.
I was struggling with this for a few hours. I've got a backend API that has
ResponseEntity<ByteArrayResource> post (#RequestPart("file") MultipartFile file) and a NiFi processor that sends XLSX data to this. I was getting the same issue... The key for me were the bottom two properties:
FlowFile Form Data Name -> file (this is the name of the variable in the API call)
Set Flowfile Form Data File Name -> true
I am trying to open a url with open-uri and when I open it from my browser, Safari, it takes me to the page in a second. However, when I try to open it with open-uri, it doesn't work. It says Net::ReadTimeout: Net::ReadTimeout, after a minute. It takes one second to open the url with my browser, but it doesn't work with open-uri ever. I have tried to increase the max timeout time but it doesn't work.
open(url).read
This is the code I use to open the url, and it doesn't work when I do it in the code.
Looks like they're protecting their API against really vague requests.
`curl 'http://stats.nba.com/stats/commonallplayers?LeagueID=00&Season=2016-17&IsOnlyCurrentSeason=1' -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,ru;q=0.6' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36'`
will do it just fine.
we are migrating our apps to Nodechef parse server with DBs hosted on Object Rocket.
While we are able to send pushes via Nodechef dashboard, we are unable to send pushes via REST api as we currently do with Parse.
This is the request sent from Nodechef dashboard:
POST /parse/push HTTP/1.1
Host: test-appname-1067.nodechef.com
Connection: keep-alive
Content-Length: 446
Origin: https://dashboard.nodechef.com
User-Agent: Mozilla/5.0 (Linux; Android 6.0; Nexus 5 Build/MRA58N) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.76 Mobile Safari/537.36
Content-Type: text/plain
Accept: /
Referer: https://dashboard.nodechef.com/apps/test-appname/push/new
Accept-Encoding: gzip, deflate, br
Accept-Language: it-IT,it;q=0.8,en-US;q=0.6,en;q=0.4
{"where": {"deviceType":{"$in":["ios"]}},"data":{"alert":"Text of the alert","category":"category of the alert","title":"Title of the alert","u":"2b30Yuh"},"_ApplicationId":"applicationid","_ClientVersion":"js1.6.14","_MasterKey":"masterkey","_InstallationId":"56205783-4f18-7427-919a-6e03ed80443a"}
This is the request we are trying to send (same structure used with Parse):
curl -X POST \
-H "X-Parse-Application-Id: applicationid" \
-H "X-Parse-Master-Key: masterkey" \
-H "Content-Type: application/json" \
-d
{"where": {"deviceType":{"$in":["ios"]}},"data": "{\"alert\":\"Text of the alert\", \"category\": \"category of the alert\", \"title\": \"Title of the alert\", \"u\": \"2b30Yuh\"}"
} \
https://test-appname-1067.nodechef.com/parse/push
We get no error (result:true) when trying this curl but no push is sent at all.
We noticed several differences between the two requests so we tried to reproduce the same request sent via dashboard and still we were not able to send anything:
we changed the content type in the header
we specified the various keys in the body instead of the header
we specified that "installation id" we have no clue of what is it
The only other difference we saw is that Nodechef dashboard makes use of Parse SDK for javascript, while we are using the php SDK, but honestly we don't think this could be the reason why requests from REST api fail
Already opened an issue to Nodechef, they couldn't help and suggested to post on the parse server repo on github for further assistance, where they asked to open a issue here.
Thank you in advance for any help
currently sending push notifications can be done only with master key. So the only way to do it is from cloud code.
In order to send a push you need to do the following steps:
Create cloud code function (let's call it sendPush for instance)
The cloud code function can receive parameters so you can send any parameters that you like in order to know what exactly needs to be done in cloud code before you sending the push. For example: to which users you like to send, or to which channel etc.
You need to add useMasterKey to where you send the push so at the end your code should look like this:
Parse.Cloud.define("sendPush", function(request, response) {
// You can get parameters in here... You can access to specific parameter like this:
// request.params.{PARAM_NAME}
// build the query for the push notification
// the query can be built by your parameters (e.g. to which userId or channel id etc.)
var query = new Parse.Query(Parse.Installation);
query.exists("deviceToken");
// this is the push payload
var payload = {
alert: "after save push",
sound: "default"
// ... add more here if required
};
Parse.Push.send({
data: payload,
where: query
}, {
useMasterKey: true
}) // useMasterKey is required currently
.then(function() {
response.success("Push Sent!");
}, function(error) {
response.error("Error while trying to send push " + error.message);
});
});
More info about cloud code and how to use it can be found here
Laravel 5.2 docs state:
AJAX Requests & Validation
In this example, we used a traditional form to send data to the
application. However, many applications use AJAX requests. When using
the validate method during an AJAX request, Laravel will not generate
a redirect response. Instead, Laravel generates a JSON response
containing all of the validation errors. This JSON response will be
sent with a 422 HTTP status code.
Src: https://laravel.com/docs/5.2/validation#quick-ajax-requests-and-validation
However, this doesn't seem correct. A simple CURL call to a route using validation, still returns HTML/a redirect:
curl -X POST -H "Content-Type: application/json"
-H "Cache-Control: no-cache"
-d '{}' "http://yours.com/user/register"
The response of the above is HTML, and a redirect. It's not JSON.
Even this page: https://laravel.com/docs/5.2/requests#retrieving-input states:
Retrieving JSON Input Values
When sending JSON requests to your application, you may access the
JSON data via the input method as long as the Content-Type header of
the request is properly set to application/json.
So what have we missed? What is the solution?
You also seem to need the following header:
"X-Requested-With: XMLHttpRequest"
Thus:
curl -X POST -H "Content-Type: application/json"
-H "X-Requested-With: XMLHttpRequest"
-H "Cache-Control: no-cache"
-d '{}' "http://yours.com/user/register"
And now $this->validate(...) is properly returning a JSON response.
This is a regular funded account - It will insist there is no From Number - I also tried as request parameters rather than request body, SID and TOKEN are correct, tested by modifying them and getting appropriate error. Works fine with the twilio libraries, not as a standalone POST
POST /2010-04-01/Accounts/ACCOUNT_SID/Messages.json HTTP/1.1
Host: ACCOUNT_SID:AUTH_TOKEN#api.twilio.com
Content-Type: text/json
Cache-Control: no-cache
{ "body": "Jenny please?! I love you <3", "From": VALID_TWILIO_NUMBER, "to": MY_CELL }
Also I would like to specify the ACCOUNT_SID and AUTH_TOKEN as header parameters, but Twilio does not recognize them.
The following works (from POSTMAN history)
POST /2010-04-01/Accounts/ACCOUND_SID/Messages.json HTTP/1.1
Host: api.twilio.com
Authorization: Basic Base64_encoding(ACCOUNT_SID:AUTH_TOKEN)
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Body=%22Test+1+-+urlencoded%22&From=%22%2BVALID_TWILIO_NUMBER%22&To=%22%2BMY_CELL%22
A few observations:
This worked using POSTMAN in Chrome and HTTPRequester in Firefox (Authorization entered interactively through user/passwd dialog)
The Body, From, and To parameters MUST be spelled that way, if the first letter is lowercase, it will return the above error or similar.
That was the only Content-Type that worked - could not get it to work with a JSON request body (the way I posed it on the original question) - that remains an open issue.
I suggest that you use request parameters:
$ curl -XPOST https://api.twilio.com/2010-04-01/Accounts/AC5ef8732a3c49700934481addd5ce1659/Messages.json \
-d "Body=Jenny%20please%3F%21%20I%20love%20you%20<3" \
-d "To=%2B15558675309" \
-d "From=%2B14158141829" \
-d "MediaUrl=http://www.example.com/hearts.png" \
-u 'AC5ef8732a3c49700934481addd5ce1659:{AuthToken}'
Additionally, I have seen that "No From Parameter" error when the phone number is not properly formatted.
I was trying to do this with Twilio's new WhatsApp messaging and was running into the same issue. Content-type was important, but also how the body of the POST request was formatted.
From=whatsapp:%2B441618507453&To=whatsapp:%2B17796758684&Body=Your appointment is coming up on {{July 21}} at {{3PM}}
Note the use of URL encoding for "+" symbols