I'm using BASH and I need to download a TXT file, which is generated on server-side by request. This means the URL is something like:
http://1.1.1.1:4884/page.aspx?fileID=123456&lang=en&Export=1
Export=1 is caught by the .NET application and I'm provided with a TXT file, based on fileID.
In case I haven't logged in, I'm redirected to a login form with ?ReturnUrl in the URL, redirecting me back to my requested page upon login.
How can I successfully download this file using BASH, cURL/wget/lynx. It has to be non-interactive.
I've tried using the --cookie options for curl and wget and lynx automation (cmd-log). Lynx worked best, but for some reason, the file download prompt could not be automated.
Please help. If any additional info is required, I will provide.
Use curl.
Code following approach:
try to download the file
if failed (redirected to login page), log in and go to begin
You always need to use -c option of curl to store the cookies between curl calls
To log in using curl you need to know the form on the server, that means: names of fields where you usually type your login and password.
To send the data to server use -d option of curl. To send the cookie to server use -b (or --cookie).
Related
I am attempting to get an OAUTH token for bitbucket via a bash script.
At the moment, I'm able to use the following URL:
https://bitbucket.org/site/oauth2/authorize?client_id={key}&response_type=token
I simply visit this via a web browser and hit authenticate, I am then redirected to the callback website, And I can see the token in the URL. For example, if my callback url was stackoverflow.com, My url bar would now contain stackoverflow.com/#access_token=XYZ
What I need to do, is figure out how to hit this url but get this access token in a bash script rather than from a URL.
Relevant doc: https://bitbucket.org/site/oauth2/authorize?client_id=kdBELaEX6HkUexPMRS&response_type=token
Using curl (can be used inside your bash script)
curl -X POST https://bitbucket.org/site/oauth2/access_token \
-d grant_type=client_credentials \
-u key:secret
Is it possible to bash script the setup process of jenkins, for example i have jenkins container set-up on my local machine, i would like to complete jenkins set-up entirely on bash for the purpose of scripting this entirely from a dockerfile.
I need to be able to pass initialAdminPassword without the use of a browser and just from the terminal.
Is it possible to complete the set-up from the terminal?
Yes it is possible to skip the manual setup.
I don't know your particular setup, but let's assume you retrieve the password from jenkins instance :
cat /var/jenkins_home/secrets/initialAdminPassword
or
docker exec "myjenkinscontainer" bash -c 'cat $JENKINS_HOME/secrets/initialAdminPassword'
You could then connect to Jenkins as admin, using that password.
curl --silent -u "admin:$mypassword" http://localhost:8080/manage
If you've configured Jenkins Security to not allow API calls, then you might require to generate first a crumb token that you would use in every request, instead of the password. To issue a crumb token, you might do something similar :
curl -s 'http://admin:$mypassword#localhost:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)'
Then you might need pass this crumb value instead of password in further requests.
Depending on your situation / Jenkins configuration, I might provide more details.
I have a paid twilio account with verified number and am trying to use the bash script supplied by twilio entitled "twilio-call" on Ubuntu 16.04.3 LTS. The response is: Failed to call 941-8XX-XXXX: curl (22): The requested URL returned: 404 NOT FOUND.
I know that I have the credentials set correctly as the bash script entitled "twilio-sms" works flawlessly.
The bash script "twilio-call" has been downloaded directly from twilio.com/labs/bash and the permissions have been set correctly.
The actual curl command is:
RESPONSE=curl -fSs -u "$ACCOUNTSID:$AUTHTOKEN" -d "Caller=$CALLERID" -d "Called=$PHONE" -d "Url=http://twimlets.com/message?Message=$MSG" "https://api.twilio.com/2008-08-01/Accounts/$ACCOUNTSID/Calls" 2>&1
The variables $ACCOUNTSID, $AUTHTOKEN, $CALLERID, $PHONE, and $MSG have all been verified to be populated correctly.
What could be causing this 404 response? Am I correct in my understanding that a verified number from twilio which works correctly for sms should also work for call?
It must be some old example at Twilio. Where did you find it?
The API endpoint URL it's not
https://api.twilio.com/2008-08-01/Accounts/$ACCOUNTSID/Calls
it is
https://api.twilio.com/2010-04-01/Accounts/$ACCOUNTSID/Calls
I am trying to use wget to download the yahoo stock history file,
https://query1.finance.yahoo.com/v7/finance/download/ARDM?period1=1504030392&period2=1506708792&interval=1d&events=history&crumb=TKe9axyOsuR
but it always reports an error code":
Username/password authentication Failed.
However, if I visit that stock page
https://finance.yahoo.com/quote/ARDM/history?p=ARDM
I can click the datafile link, and download the CSV file without giving any user or password.
Any clue?
When you are going in via the browser the system is using your cached yahoo login details to go straight to the information using your cookie.
Set your authentication in your wget by using the following:
wget --user user --password pass http://example.com/
If you are not logging in to the server as seems to be the case here then you could try using two WGET using the first one to grab a cookie and the second one to download the data as follows:
wget -qO- --keep-session-cookies --save-cookies cookies.txt https://finance.yahoo.com/quote/ARDM/history?p=ARDM
followed by
wget -qO- --load-cookies cookies.txt https://query1.finance.yahoo.com/v7/finance/download/ARDM?period1=1504030392&period2=1506708792&interval=1d&events=history&crumb=TKe9axyOsuR
So I want to call a REST API from Bamboo after a deployment has completed.
This API needs a username and password but it can't be stored in Bamboo as it seems it can be viewed in the Bash History of the Build agent.
I intended to use a script task and execute something like
curl -f -v -k --user "${bamboo.user}":"${bamboo.password}" -X POST https://bamboo.url/builds/rest/api/latest/queue/project_name"/
This would make the REST call. But the username and password is a problem.
I do have the option, however of using a PEM file. It can be provided so does anyone know if this can be used in conjunction with the cURL?
--OR--
One other thought- could I encrypt a password within a file in my source control, and somehow decrypt it on the build agent, and then have curl use the file instead of reading the password from the command line? How would this look in cURL?
Any ideas how this could be achieved?
Your command seems to have an extra quote at the end of your command
Using a pem file to authenticate with curl:
curl -E /path/to/user-cert.pem -X POST https://bamboo.url/builds/rest/api/latest/queue/project_name
The file should have both private key and public key inside.