Has anyone been able to successfully use GNU Parallel on Windows 10 with git-bash? Is it possible? - If so, how?
Background:
I'm having trouble installing GNU Parallel and using it, and it got me thinking - maybe git-bash is holding me back? I'm sure if I installed Ubuntu through WSL I wouldn't have any problems running GNU Parallel. But I wanted to know if I could do this in git-bash first.
I just installed git-bash on a Microsoft Windows 10 machine and had no problems installing GNU Parallel.
It is by no means well tested on git-bash, but basic functionality clearly works.
I'm having trouble installing GNU Parallel
Maybe you can post the error you get when running:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
Related
I'm trying to get this shell script to work in make:
$ VERSION=$(echo 'ThisBuild / version := "1.20"' | grep -e 'This.*version' | grep -Eo '[0-9]+\.[0-9]+')
$ echo $VERSION
1.20
make:
$ make -v
GNU Make 4.2.1
Built for x86_64-pc-linux-gnu
Makefile:
version:
VERSION=$(echo 'ThisBuild / version := "1.20"' | grep -e 'This.*version' | grep -Eo '[0-9]+\.[0-9]+') && \
echo "VERSION: ${VERSION}"
Doesn't work:
$ make version
VERSION= && \
echo "VERSION: "
VERSION:
Can this be done?
If you want to pass $ to the shell you have to escape it from make, by doubling it: $$.
version:
VERSION=$$(echo 'ThisBuild / version := "1.20"' | grep -e 'This.*version' | grep -Eo '[0-9]+\.[0-9]+') && \
echo "VERSION: $${VERSION}"
Otherwise, make will think that you're referencing a make variable and expand it.
I use the following to install the most recent docker compose for my ubuntu server:
curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
How to do I make this more version agnostic. For instance, so that I do not have to go in and keep changing the version -which in this case is 1.21.2. How do I change the command so it gets the most latest stable release?
How do I change the command so it gets the most latest stable release?
You could try following:
curl -L https://github.com/docker/compose/releases/download/`curl -Ls -o /dev/null -w %{url_effective} https://github.com/docker/compose/releases/latest | awk -F / '{print $NF}'`/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
This is same as your script only replacing actual version (1.21.2 in your case) with latest tag over several steps:
First we get redirection url for latest stable:
curl -Ls -o /dev/null -w %{url_effective} https://github.com/docker/compose/releases/latest
currently it resolves to https://github.com/docker/compose/releases/tag/1.21.2
Then we get version tag out of redirection url:
| awk -F / '{print $NF}'
currently resolving to 1.21.2
Finally we execute it in place of version number using your original curl statement. Note that this can break if latest tag is not properly redirected and ads some extra complexity, but automates version pulling as requested.
Accepted answer isn't the latest stable version according to https://docs.docker.com/compose/release-notes/ (returns v2 instead of the latest v1 which I was looking for)
This is the monstrosity I went with
rm -Rf /usr/local/bin/docker-compose && version=$(curl -s https://docs.docker.com/compose/release-notes/ | grep "Docker Compose release notes" | grep "Estimated reading time" | sed 's/.*id=//g' | sed 's/<.*$//g' | sed 's/.*>//g') && curl -L https://github.com/docker/compose/releases/download/${version}/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose && chmod +x /usr/local/bin/docker-compose
Is there a shell command to get the latest stable helm release .
For kubernetes we have something like this
curl -LO https://storage.googleapis.com/kubernetes-release/release/$(curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt)/bin/linux/amd64/kubectl
Try:
wget -qO- https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' |head -1
Result:
v2.8.2
And, for those without wget:
HVER=$(curl -sSL https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' | head -1)
echo $HVER
Result (currently):
v2.9.1
To download the gz that contains the latest Helm executable:
Linux
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-linux-amd64.tar.gz
OSX
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-darwin-amd64.tar.gz
Windows (bash ell)
curl -LO https://storage.googleapis.com/kubernetes-helm/helm-${HVER}-windows-amd64.tar.gz
if in-case you use Dockerfile & Linux
RUN wget "https://storage.googleapis.com/kubernetes-helm/helm-$(wget -qO- https://github.com/kubernetes/helm/releases | sed -n '/Latest release<\/a>/,$p' | grep -oE 'v[0-9]+\.[0-9]+\.[0-9]+' |head -1)-linux-amd64.tar.gz"
For Linux, OSX, and Windows targets:
HELM_INSTALL_DIR=[‘desired path’]
USE_SUDO=[‘true’|’false’]
curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash
Other install options: Helm Install Documentation
Recommend migration away from V2.x as soon as you can.
The way I do it for v2:
curl -L0 "https://storage.googleapis.com/kubernetes-helm/helm-v${HELM_VERSION}-linux-amd64.tar.gz" | tar xzO linux-amd64/helm > /usr/local/bin/helm
And for v3:
curl -L0 "https://get.helm.sh/helm-v${HELM_VERSION}-linux-amd64.tar.gz" | tar xzO linux-amd64/helm > /usr/local/bin/helm
Do not forget to chmod +x /usr/local/bin/helm afterwards.
I have a text file ( Input.txt ) containing domains and that is total of about 35 Millions domains.
#Input.txt
google.com
cnn.com
bbc.com
........
Now ,I have a python script to check the status code of each and every domains associated with in the text file ( Input.txt ). For smaller set, I do
for i in $(cat Input.txt);do python status_check.py $i;done > out_file.txt
If i process in this manner,It might take ages to check the status code for all 35 million domains.
I'm not familiar in parallel processing. Can some one help me on,How to achieve the task by saving time using shell/bash/any ?
You are looking for GNU Parallel:
cat Input.txt | parallel -j 100 python status_check.py > out_file.txt
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 883c667e01eed62f975ad28b6d50e22a
12345678 883c667e 01eed62f 975ad28b 6d50e22a
$ md5sum install.sh | grep cc21b4c943fd03e93ae1ae49e28573c0
cc21b4c9 43fd03e9 3ae1ae49 e28573c0
$ sha512sum install.sh | grep da012ec113b49a54e705f86d51e784ebced224fdf
79945d9d 250b42a4 2067bb00 99da012e c113b49a 54e705f8 6d51e784 ebced224
fdff3f52 ca588d64 e75f6033 61bd543f d631f592 2f87ceb2 ab034149 6df84a35
$ bash install.sh
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
Put an ampersand after your $1 and it will run each "concurrently"
Bash is probably not the right tool to do this. Each fork is very expensive resource-wise. You'd be better off using Ruby or Python, reading this into an array and then processing it inside the interpreter's VM.
Why not alter your python script to read the URLs itself and then distribute the processing?
It seems a bit pointless having a bash for-loop when you could just do that in python.
There are a number of modules in python for handling parallel processing listed here.
I want to make a bash script to take the system os and the version as a simple string.
Possible ways to get these info is from
/etc/issue
cat /etc/*-release
lsb_release -a
and probably some others which i dont know. The problem is that i want the bash script to work on Ubuntu 12,13,14 and CentOS. Some of the above does not work in these systems. For example the lsb_release does not work on CentOS and sometimes the /etc/issue is empty so i'm little confused about it.
As for the string i want to get it in this way (and save it to var). I will give examples.
If OS is Ubuntu 12.x i want to take it as ubuntu12
If OS is Ubuntu 13.x i want to take it as ubuntu13
If OS is CentOS 7.x i want to take it as centos7
Is that easy?
THANK YOU
Here is a bash solution. I tested on Ubuntu, but not on CentOS (I only have RHEL available now). But you can test the CentOS part and modify as needed.
#!/usr/bin/env bash
RELEASE=unknown
version=$( lsb_release -r | grep -oP "[0-9]+" | head -1 )
if lsb_release -d | grep -q "CentOS"; then
RELEASE=centos$version
elif lsb_release -d | grep -q "Ubuntu"; then
RELEASE=ubuntu$version
fi
echo $RELEASE
Or, without lsb_release on CentOS:
#!/usr/bin/env bash
RELEASE=unknown
if [ -f /etc/redhat-release ]; then
version=$( cat /etc/redhat-release | grep -oP "[0-9]+" | head -1 )
RELEASE=centos$version
elif [ -n $(which lsb_release 2> /dev/null) ] && lsb_release -d | grep -q "Ubuntu"; then
version=$( lsb_release -d | grep -oP "[0-9]+" | head -1 )
RELEASE=ubuntu$version
fi
echo $RELEASE
In any case, there's more than one way to skin this cat.