I've used 'uniq -d -c file' in many shell scripts on linux machines, and it works.
On my MAC (OS X 10.6.7 with developer tools installed) it doesn't seems to work:
$ uniq -d -c testfile.txt
usage: uniq [-c | -d | -u] [-i] [-f fields] [-s chars] [input [output]]
It would be nice if anyone could checks this.
Well, it's right there in the Usage message. [ -c | -d | -u] means you can use one of those possibilities, not two.
Since OSX is based on BSD, you can check that here or, thanks to Ignacio, the more Apple-specific one here.
If you want to achieve a similar output, you could use:
do_your_thing | uniq -c | grep -v '^ *1 '
which will strip out all those coalesced lines that have a count of one.
You can try this awk solution
awk '{a[$0]++}END{for(i in a)if(a[i]>1){ print i ,a[i] } }' file
Related
Running a nx affected:apps command gives me this output:
> NX NOTE Affected criteria defaulted to --base=master --head=HEAD
> NX Affected apps:
- app-backend
- app-frontend
- app-something
- app-anything
I need to get all the application names and use them again for a command call.
So I started with that
output=$(nx affected:apps)
echo "$output" | grep -E "^\W+app-(\w+)"
This gives me
- app-backend
- app-frontend
- app-something
- app-anything
But I need to get the names only instead to run foo --name={appname} four times.
Also not quite sure how to use it in a loop. Quite new to bash scripting :-(
You may use -o (show matches only) with -P (perl regex moode) in gnu-grep:
nx affected:apps |
grep -oP "^\W+app-\K\w+" |
xargs -I {} docker build -t {} .
If gnu-grep isn't available then use this awk command:
nx affected:apps |
awk -F- '/app-/{print $3}' |
xargs -I {} docker build -t {} .
I don't have nx command here but you can try using xargs:
nx affected:apps | grep '^ -' | cut -d' ' -f4 | xargs -I{} echo docker build -t {} ./dist/{}
Remove echo to actually run the command.
You can use the --plain option:
nx affected:apps --plain
the command should return all the affected apps with space as a divider. You can then store those to a bash array and cycle through them in a for loop, running the command you need:
#!/bin/bash
AFFECTED=($(./node_modules/.bin/nx affected:apps --plain))
for t in ${AFFECTED[#]}; do
echo $t
done
I am trying to escape the following script to run it via shell command-string option ( /bin/sh -c ).
privateIP=$(ifconfig eth0 | grep "inet " | awk \'{print $2}\');
sed -i "s/http:\/\/:/http:\/\/$privateIP:/g" init.conf
Please elaborate on the answer.
You're question is not clear, but perhaps you are looking for:
sh -c 'privateIP=$(ifconfig eth0 | awk "/inet/{print \$2}");
sed -i "s#http://:#http://$privateIP:#g" init.conf'
I'm trying to create script to be run by cron to create multiple folders with subfolders.
DATE=`date +%Y-%m-%d`
IP_ADDR=`ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p'`
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
If i run this script manually everything is created as expected. When script is ran by cron subdirectory $IP_ADDR is not created and there is no errors.
I suspect that /sbin is not part of the PATH for the environment that the cron job runs under. You should specify the full path for the ifconfig command:
IP_ADDR=$(/sbin/ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p')
It's also better practice (in general) to use $() for command substitution.
Try to use debug mode :
set -x
DATE=`date +%Y-%m-%d`
IP_ADDR=`ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p'`
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
set +x
Then, redirect the output of your cron to a file and have a look, you should find useful information in it.
You are not far off, but there are several ordering caveats that could cause problems. Many systems have different formats for the ifconfig output line. Some with inet xxx.xxx.xxx.xxx, others with inet addr:xxx.xxx.xxx.xxx. (those are the two most common). You may also need to handle the case where there are multiple wired inet interfaces (2+ NICs in the box). However, if you have only 1 NIC, you could try the following to handle the common ifconfig formats:
DATE=`date +%Y-%m-%d`
IP_ADDR=$(ifconfig |
grep -v '127.0.0.1' |
grep -E 'inet[ ](addr:)*[0-9]{1,3}([.][0-9]{1,3}){3}' |
sed -e 's/^.*inet \(addr:\)*//' -e 's/ .*$//')
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
or with IP_ADDR written as one line:
IP_ADDR=$(ifconfig | grep -v '127.0.0.1' | grep -E 'inet[ ](addr:)*[0-9]{1,3}([.][0-9]{1,3}){3}' | sed -e 's/^.*inet \(addr:\)*//' -e 's/ .*$//')
On my Fedora machine I sometimes need to find out certain components of the kernel name, e.g.
VERSION=3.18.9-200.fc21
VERSION_ARCH=3.18.9-200.fc21.x86_64
SHORT_VERSION=3.18
DIST_VERSION=fc21
EXTRAVERSION = -200.fc21.x86_64
I know uname -a/-r/-m but these give me not all the components I need.
Of course I can just disassemble uname -r e.g.
KERNEL_VERSION_ARCH=$(uname -r)
KERNEL_VERSION=$(uname -r | cut -d '.' -f 1-4)
KERNEL_SHORT_VERSION=$(uname -r | cut -d '.' -f 1-2)
KERNEL_DIST_VERSION=$(uname -r | cut -d '.' -f 4)
EXTRAVERSION="-$(uname -r | cut -d '-' -f 2)"
But this seems very cumbersome and not future-safe to me.
Question: is there an elegant way (i.e. more readable and distribution aware) to get all kernel version/name components I need?
Nice would be s.th. like
kernel-ver -f "%M.%m.%p-%e.%a"
3.19.4-200.fc21.x86_64
kernel-ver -f "%M.%m"
3.19
kernel-ver -f "%d"
fc21
Of course the uname -r part would need a bit sed/awk/grep magic. But there are some other options you can try:
cat /etc/os-release
cat /etc/lsb-release
Since it's fedora you can try: cat /etc/fedora-release
lsb_release -a is also worth a try.
cat /proc/version, but that nearly the same output as uname -a
In the files /etc/*-release the format is already VARIABLE=value, so you could source the file directly and access the variables later:
$ source /etc/os-release
$ echo $ID
fedora
To sum this up a command that should work on every system that combines the above ideas:
cat /etc/*_ver* /etc/*-rel* 2>/dev/null
So I have the equivalent of a list of files being output by another command, and it looks something like this:
http://somewhere.com/foo1.xml.gz
http://somewhere.com/foo2.xml.gz
...
I need to run the XML in each file through xmlstarlet, so I'm doing ... | xargs gzip -d | xmlstarlet ..., except I want xmlstarlet to be called once for each line going into gzip, not on all of the xml documents appended to each other. Is it possible to compose 'gzip -d' 'xmlstarlet ...', so that xargs will supply one argument to each of their composite functions?
Why not read your file and process each line separately in the shell? i.e.
fileList=/path/to/my/xmlFileList.txt
cat ${fileList} \
| while read fName ; do
gzip -d ${fName} | xmlstartlet > ${fName}.new
done
I hope this helps.
Although the right answer is the one suggested by shelter (+1), here is a one-liner "divertimento" providing that the input is the proposed by Andrey (a command that generates the list of urls) :-)
~$ eval $(command | awk '{a=a "wget -O - "$0" | gzip -d | xmlstartlet > $(basename "$0" .gz ).new; " } END {print a}')
It just generates a multi command line that does wget http://foo.xml.gz | gzip -d | xmlstartlet > $(basenname foo.xml.gz .gz).new for each of the urls in the input; after the resulting command is evaluated
Use GNU Parallel:
cat filelist | parallel 'zcat {} | xmlstarlet >{.}.out'
or if you want to include the fetching of urls:
cat urls | parallel 'wget -O - {} | zcat | xmlstarlet >{.}.out'
It is easy to read and you get the added benefit of having on job per CPU run in parallel. Watch the intro video to learn more: http://www.youtube.com/watch?v=OpaiGYxkSuQ
If xmlstarlet can operate on stdin instead of having to pass it a filename, then:
some command | xargs -i -n1 sh -c 'zcat "{}" | xmlstarlet options ...'
The xargs option -i means you can use the "{}" placeholder to indicate where the filename should go. Use -n 1 to indicate xargs should only one line at a time from its input.