I want to write a script to automate a repetitive task I do.
I issue this command:
heroku pgbackups:capture --expire
And I get this response
MY_DB_URL (DATABASE_URL) ----backup---> b677
I then issue this command with the capture number (b677 in the above)
curl -o latest.dump `heroku pgbackups:url b677`
How can I parse the text and place the value into the next command?
It's only a matter of parsing the output of the first command and saving it into a variable:
capnum=$(heroku pgbackups:capture --expire | grep -- "--->" | awk '{print $NF}')
And then execute,
curl -o latest.dump $(heroku pgbackups:url ${capnum})
Alternatively, you may say:
curl -o latest.dump $(heroku pgbackups:url $(heroku pgbackups:capture --expire | grep -- "--->" | awk '{print $NF}'))
More ways:
read __ __ __ URL < <(exec heroku pgbackups:capture --expire)
curl -o latest.dump "$URL"
curl -o latest.dump $(exec heroku pgbackups:capture --expire | cut -f 4 -d ' ')
curl -o latest.dump $(exec heroku pgbackups:capture --expire | sed 's|.* ||')
Its doable using sed and a shell script.
#!/bin/sh
s=`heroku pgbackups:capture --expire`
id=`echo ${s} | sed -e "s/^.*> \(b[0-9]*\)$/\1/"`
url=`echo heroku pgbackups:url ${id}`
curl -o last.dump "${url}"
This is not the most beautiful shell script, but it can work as prove of concept.
Well, if the format of the capture number is always the same, you can use
captureNumber=`heroku pgbackups:capture --expire | grep -o [a-Z][0-9]*$`
curl -o latest.dump `heroku pgbackups:url ${captureNumber}`
or, perhaps more general, detect the last space and use the remainder of the line:
captureNumber=`heroku pgbackups:capture --expire | grep -o \s.*$`
Adjust the regex if there are different formats I don't know about, or go with devnull's answer if the format is not well-captured by regex.
Related
I have tried this command
curl -Ik https://dev.mydomain.com/
and it does print everything. And now what I want is to print out content-security-policy only.
Do I need to use jq or is there any other helpful tool that I can use?
curl -sIk https://stackoverflow.com/ | grep content-security-policy | cut -d ' ' -f 2-
Will curl the url, grep only the line with content-security-policy, cut on a space, and get all the fields from 2 onwards.
Example:
➜ ~ curl -sIk https://stackoverflow.com/ | grep content-secur | cut -d ' ' -f 2-
upgrade-insecure-requests; frame-ancestors 'self' https://stackexchange.com
If you use cURL >= 7.84.0, you can use the syntax %header{name} :
curl -Iks https://stackoverflow.com -o /dev/null -w "%header{content-security-policy}"
If you want to try it without installing a new version, you can run the Docker image :
docker run --rm curlimages/curl:7.85.0 -Iks https://stackoverflow.com -o /dev/null -w "%header{content-security-policy}"
I trying to add text (predefined) between a sorted output and saved to a new file.
I'm using a curl command to gather my info.
$ curl --user XXX:1234!## "http://......"
Then using grep to find IP addresses and sorting so they only appear once.
$ curl --user XXX:1234!## "http://......" | grep -E -o -m1 '([0-9]{1,3}[\.]){3}[0-9]{1,3}' | sort -u
I need to add <my_text_predefined> ([0-9]{1,3}[\.]){3}[0-9]{1,3} <my_text_predefined> between the regex ip address and then saved to a new file.
The script below only get my the ip address
$ curl --user XXX:1234!## "http://......" | grep -E -o -m1 '([0-9]{1,3}[\.]){3}[0-9]{1,3}' | sort -u
123.12.0.12
123.56.98.76
$ curl --user some_user:password "http://...." | grep -E -o -m1 '([0-9]{1,3}[\.]){3}[0-9]{1,3}' | sort -u | sed 's/.*/<prefix> -s & <suffix>/'
So if we need print some text for each IP ... try xargs
for i in {1..100}; do echo $i; done | xargs -n1 echo "Values are:"
if based on IP you would need to take decision put in a loop
for file $(curl ...) do ...
and check $file or do something with it ...
I have an output of a curl command as below,
comand: curl https://application.com/api/projectcreator
Output:
{"Table":[{"key":"projectA","name":"Jhon"},
{"key":"projectB","name":"Sam"},
{"key":"ProjectC","name":"Jack"}]}
I would like to cut this output to get only names. is there a way to do this in shell?
Eg:
Jhon
Sam
Jack
I tried below but doesnt seem to be promissing for me.
for Table in `curl -s -k http://application.com/api/projectcreator grep "name"`
do
echo "$Table"
done
Thanks for your help and insights!
Using jq you can do this easily:
curl -s -k 'http://application.com/api/projectcreator' |
jq -r '.Table[].name' | paste -s -d ' '
Jhon Sam Jack
If jq cannot be installed then use gnu grep:
curl -s -k 'http://application.com/api/projectcreator' |
grep -oP '"name":"\K[^"]+' | paste -s -d ' '
Is there a shell command to get the latest stable helm release .
For kubernetes we have something like this
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
Try:
wget -qO- https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' |head -1
Result:
v2.8.2
And, for those without wget:
HVER=$(curl -sSL https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1)
echo $HVER
Result (currently):
v2.9.1
To download the gz that contains the latest Helm executable:
Linux
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-linux-amd64.tar.gz
OSX
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-darwin-amd64.tar.gz
Windows (bash ell)
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-windows-amd64.tar.gz
if in-case you use Dockerfile & Linux
RUN wget "https://storage.googleapis.com/kubernetes-helm/helm-$(wget -qO- https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' |head -1)-linux-amd64.tar.gz"
For Linux, OSX, and Windows targets:
HELM_INSTALL_DIR=[‘desired path’]
USE_SUDO=[‘true’|’false’]
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
Other install options: Helm Install Documentation
Recommend migration away from V2.x as soon as you can.
The way I do it for v2:
curl -L0 "https://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz" | tar xzO linux-amd64/helm > /usr/local/bin/helm
And for v3:
curl -L0 "https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz" | tar xzO linux-amd64/helm > /usr/local/bin/helm
Do not forget to chmod +x /usr/local/bin/helm afterwards.
Here's the command I'm currently running:
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")'
The response from this command is a URL, like this:
$ curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")'
http://google.com
I want to use whatever that URL is to essentially do this:
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | curl 'http://google.com'
Is there any simple way to do this all in one line?
Use xargs with a place holder for the output from stdin with the -I{} flag as below. The -r flag is to ensure the curl command is not invoked on a empty output from previous grep output.
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | xargs -r -I{} curl {}
A small description about the flags, -I and -r from the GNU xargs man page,
-I replace-str
Replace occurrences of replace-str in the initial-arguments with
names read from standard input.
-r, --no-run-if-empty
If the standard input does not contain any nonblanks, do not run
the command. Normally, the command is run once even if there is
no input. This option is a GNU extension
(or) if you are looking for a bash approach without other tools,
curl 'http://test.com/?id=12345' | grep -o -P '(?<=content="2;url=).*?(?=")' | while read line; do [ ! -z "$line" ] && curl "$line"; done