My script usually takes ~4min to finish, but whenever I execute cURL in SSH, it shows output immediately, which is an old attempt.
This is happening in SSH & crontab only, while wget and using my browser always get fresh content.
I tried adding 'Cache-Control: no-cache' parameter to the command, but didn't help.
root#server [~]# /usr/bin/curl -H 'Cache-Control: no-cache' http://full-path/curl
What else should I try?
You could append ?xyz=1469279729 to get unique url strings and avoid caching. But it's unlikely to be curl that is caching results, either a proxy/server returning stale data, or some alias to curl with local caching.
Related
I am trying to test the Sumo Logic API by updating the information of my collector. The second curl command is the one that is causing the issue 'curl: (55) Failed sending PUT request'. It works in my terminal but not in the bash script.
#!/bin/bash
readonly etag=$(curl -u '<accessId>:<accessKey>' -I -X GET https://api.sumologic.com/api/v1/collectors/<id> | grep -Fi etag | awk '{print $2}' | tr -d \''"\')
echo ${etag}
curl -vvv -u '<accessId>:<accessKey>' -X PUT -H "Content-Type: application/json" -H "If-Match: \"${etag}\"" -T updated_collector.json https://api.sumologic.com/api/v1/collectors/<id>
set -x
The first curl command is assigned to the variable called 'etag' which stores the necessary etag. The etag is used in the second curl command to make a request to update the information stored in the 'updated_collector.json'. The updated_collector.json file is not the issue as I have successfully updated the information via the terminal with it. I suspect the content-type is not being sent in the header because someone ran the script on their end and it was not showing that information with the -vvv tag.
Here you can find the Sumo Logic Collector API Methods and Examples from which I got the curl commands to test the API: https://help.sumologic.com/APIs/Collector-Management-API/Collector-API-Methods-and-Examples
Update: I retieved the etag and then ran the second command in a bash script. I manually inserted the etag into the ${etag} portion of the second curl command. I then ran the script and it worked. Therefore, the etag variable isn't correctly formatted inside the second curl command. I do not know how to fix this.
The issue was partially the syntax but after fixing that, I was still getting an error. "If-Match: \"${etag}\" in my command should be "If-Match: ${etag}" instead. I had to add the --http1.1 flag for it to work. I'm sure this is a sumo logic issue. I am able to execute GET requests no problem using http2.0.
i am new to curl and i started using it, so i have a situation where i have to post using curl to a url and get json response back, everything works fine upto this. So what i wanted to do is have this json response send to a laravel api which in turn parses out and saves to database. this is curl script i use
curl -X POST \
-H "Authorization: Basic Sjg2eFZzdVFkOVFUMTlScEd0==" \
-d "grant_type=client_credentials" \
--cert /etc/letsencrypt/live/www.sample.com/fullchain.pem \
--key /etc/letsencrypt/live/www.sample.com/privkey.pem \
https://connect.innovative.com/auth/token
i am not sure if curl can save the response directly to mysql and also how can i setup a scheduling the above script to run every 2 hours in ubuntu?
i figured it out by creating bash script and storing it to variable and then subsequent post request to api, also did a crob job and it worked like charm
I am trying to do a script to get me access of advance scan option of nessus in localhost. So I want advance scan operation through shell script without GUI. I want all operations like login, advance scan and export report are performed through shell script without GUI access.
Why do you want to do it with bash script?
You can do this much easier with the nessus API.
Have a look at the link below
https://github.com/jfalken/nessus_enterprise_rest_client
the simplest way of doing automatisation in nessus is to use the nessus API.
its located at https://NessusServerIP:8834/ - if you visit it, you will be greeted by the API-Documentation.
There are various API-Implementations available - if you google 'Nessus API client' you'll get a glimpse.
If you, as you said, want to to run bash-skripts than the simplest way is probably using CURL for the API-Requests.
A typical workflow will look like this:
authorize yourself to the NessusAPI (either via TOKEN or API-Key)
launch or configure a scan (and wait until it finished)
export a report (and wait until it finished)
download the exported report
CURL #1 (authorize using token):
curl -X POST --data '{"username":"NessusUser","password":"YourPassword"}' -k "https://NessusServerIp:8834/session"
--header "Content-Type:application/json" | python -m json.tool
..which will yield you following JSON yielding an Token which you need for the other API-Calls:
{"token": "e411e443521adee4496d79823a510cc68c5bf05aeda6e6eb"}
CURL #2 (launch a scan):
curl -X POST -H 'X-Cookie: token=e411e443521adee4496d79823a510cc68c5bf05aeda6e6eb' -H 'Content-Type:application/json'
--data '{"scan_id":"21", "alt_targets":[127.0.0.1]}'
-k "https://NessusServerIp:8834/scans/21/launch" | python -m json.tool
...which will be answered with a JSON like this, containing the ID of the just startet scan:
{"scan_uuid":"c1c30d8f-5f79-2e4b-2d03-05b8b3c595f1e768e03195abdfa2"}
CURL #3 (exporting a scan):
curl -X POST -H 'X-Cookie: token=766ef7a2302780c189ba563b89c5eb3706140c0ef1e4de8b' -H
'Content-Type:application/json' --data '{"scan_id":"33", "format":"html"}' -k
"https://NessusServerIP:8834/scans/33/export" | python -m json.tool
...which will yield this JSON response, containing a token to the exported file and the file_id:
{"token":"3e13ab381c480caa1e377411c0b561970c46e5d78894c5a0cb2be0e7f00fefe0","file":1434780027}
...so now we are ready to download the report. in this case, since i have specified "format: html" in the last call, its a .html you will need to safe the outcome into.
Curl #4 (download exported report):
curl -X GET -H 'X-Cookie: token=7d155aef4359d02addea29d8d56bca4a5045ca61efeb38ee' -H 'Content-Type:application/json'
--data '{"scan_id":"21", "alt_targets":127.0.0.1}'
-k "https://NessusServerIP:8834/scans/17/export/945237343/download" > report.html
...which should leave you with a report.html in the folder you started your script.
Now... how do you automatize this? Well write a Bash-Skript, put in this calls, parse the answers to extract the information you need - and then enjoy! :)
ps: i use the python -m json.tool to beautify the otherwise not very beautiful output of CURL.
Hope i have helped,
Gewure
I want to aoutomate logging in to a website and downloading a file.
Since it is session based and I have to send a token from inside the html with the login credentials as POST, I was wondering if there was a solution to this in bashscript?!
Thanks in advance!
curl may be the answer to your problem. It supports saving and loading cookies to / from a file.
The --cookie-jar <file> option causes curl to dump cookies to the given file after the request.
And the --cookie <file> option causes curl to load cookies from the given file before the request.
So you can use these options like this:
curl --cookie cookies.txt --cookie-jar cookies.txt http://www.example.com/
Posting can be done in many ways, but the --data-urlencode option is the easier:
curl --data-urlencode "username=john.doe" --data-urlencode "password=dummy" --cookie [...] http://www.example.com/
A simple example of Snaplet and Session here
$ curl http://localhost:8000/sessioin -d "key=k&value=v"
k
getter
$ curl http://localhost:8000/sessioin
getter
I cannot get session in the 2nd request.
Depending on your definition of with, you probably forgot to commit your session after you set values in it.
Also, you're using curl to test this system. curl doesn't preserve cookies by default, which is why your session cookie is lost. Try using this (works on my machine):
curl -c cookies.txt http://localhost:8000/session -d "key=k&value=v"
curl -b cookies.txt http://localhost:8000/session
See also the revised answer to your previous question.