From a file I'm retrieving the last line using the following cmd;
tail -n 1 build.log
The output looks like this:
1477101542,,ui,say,--> amazon-ebs: AMIs were created:\n\nus-east-1: ami-63237174\nus-west-1: ami-21236841\nus-west-2: ami-27872347
I'm trying to fetch the string after us-east-1:, us-west-1: & us-west-2 using the following grep commands:
echo | tail -n 1 build.log | egrep -m1 -oe 'us-east-1: ami-.{8}' | egrep -m1 -oe 'ami-.{8}'
I run this cmd three times for each condition. Is there a better way to do this?
If the order in which the regions appear is fixed, you can simply do:
$ echo | tail -n 1 build.log | egrep -o 'ami-.{8}'
ami-63237174
ami-21236841
ami-27872347
If you want to extract the region names and you have GNU grep, try:
$ echo | tail -n 1 build.log | grep -Po 'us-[^:]+(?=: ami-.{8})'
us-east-1
us-west-1
us-west-2
To get both region names and associated values:
$ echo | tail -n 1 build.log | egrep -o 'us-[^:]+: ami-.{8}'
us-east-1: ami-63237174
us-west-1: ami-21236841
us-west-2: ami-27872347
Related
I was trying to filter all the files from the URLs and get only paths.
echo -e "http://sub.domain.tld/secured/database_connect.php\nhttp://sub.domain.tld/section/files/image.jpg\nhttp://sub.domain.tld/.git/audio-files/top-secret/audio.mp3" | grep -Ei "(http|https)://[^/\"]+" | sort -u
http://sub.domain.tld
But I want the result like this
http://sub.domain.tld/secured/
http://sub.domain.tld/section/files/
http://sub.domain.tld/.git/audio-files/top-secret/
Is there any way to do it with sed or grep
Using grep
$ echo ... | grep -o '.*/'
http://sub.domain.tld/secured/
http://sub.domain.tld/section/files/
http://sub.domain.tld/.git/audio-files/top-secret/
with grep
If your grep has the -o option:
... | grep -Eio 'https?://.*/'
If there could be multiple URLs per line:
... | grep -Eio 'https?://[^[:space:]]+/'
with sed
If the input is always precisely one URL per line and nothing else, you can just delete the filename part:
... | sed 's/[^/]*$//'
You could use match function of awk, will work in any version of awk. Simple explanation would be, passing echo command's output to awk program. Using match matching everything till last occurrence of / and then printing the sub-string to print just before /(with -1 to RLENGTH).
your_echo_command | awk 'match($0,/.*\//){print substr($0,RSTART,RLENGTH-1)}'
GNU Awk
$ echo ... | awk 'match($0,/.*\//,a){print a[0]}'
$ echo ... | awk '{print gensub(/(.*\/).*/,"\\1",1)}'
$ echo ... | awk 'sub(/[^/]*$/,"")'
http://sub.domain.tld/secured/
http://sub.domain.tld/section/files/
http://sub.domain.tld/.git/audio-files/top-secret/
xargs
$ echo ... | xargs -i sh -c 'echo $(dirname "{}")/'
http://sub.domain.tld/secured/
http://sub.domain.tld/section/files/
http://sub.domain.tld/.git/audio-files/top-secret/
..poky/build$ for SUBPATH in $(bitbake -e alsa-lib | grep -P -e '(?<=^)FILES_alsa-lib(?==)' | cut -d= -f2 | tr -d \") ; do ls ./tmp-glibc/work/armv7a-vfp-neon-oe-linux-gnueabi/alsa-lib/1.0.29-r0/package$SUBPATH 2>&1 ; done | grep -e "No such file or directory" | wc -l
2855
..poky/build$ for SUBPATH in $(bitbake -e alsa-lib | grep -P -e '(?<=^)FILES_alsa-lib(?==)' | cut -d= -f2 | tr -d \") ; do ls ./tmp-glibc/work/armv7a-vfp-neon-oe-linux-gnueabi/alsa-lib/1.0.29-r0/package$SUBPATH 2>&1 ; done | grep -v -e "No such file or directory" | wc -l
15
Here one of all those No such file or directory
ls: cannot access ./tmp-glibc/work/armv7a-vfp-neon-oe-linux-gnueabi/alsa-lib/1.0.29-r0/package/usr/lib/libicalss.so.1.0.0: No such file or directory
where
..poky/build$ bitbake -e alsa-lib | grep -P -e '(?<=^)FILES_alsa-lib(?==)' | cut -d= -f2 | tr -d \"
/usr/bin/* /usr/sbin/* /usr/lib/alsa-lib/* /usr/lib/lib*.so.* /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d /usr/lib/udev/rules.d /usr/share/alsa-lib /usr/lib/alsa-lib/* /usr/share/pixmaps /usr/share/applications /usr/share/idl /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers /usr/lib/alsa-lib/smixer/*.so
and
poky/build$ echo $SHELL
/bin/bash
Apparently Bash file name expansion finds 2855 items in identified sub-paths the called ls command can't identify.
Actually in every iteration instead of ls ... I need to do find with search root point set to ./tmp-glibc/work/armv7a-vfp-neon-oe-linux-gnueabi/alsa-lib/1.0.29-r0/package$SUBPATH and -nameargument set few times (logical OR) to some patterns.
Where is my mistake?
Is this that file name expansion takes place in the for-loop instead of on invoking ls command (as programmer wishes it)?
#
Following alternative found under my limited expertise level and time resources
cd ./tmp-glibc/work/armv7a-vfp-neon-oe-linux-gnueabi/alsa-lib/1.0.29-r0/package && echo "/usr/bin/* /usr/sbin/* /usr/lib/alsa-lib/* /usr/lib/lib*.so.* /etc /com /var /bin/* /sbin/* /lib/*.so.* /lib/udev/rules.d /usr/lib/udev/rules.d /usr/share/alsa-lib /usr/lib/alsa-lib/* /usr/share/pixmaps /usr/share/applications /usr/share/idl /usr/share/omf /usr/share/sounds /usr/lib/bonobo/servers" | sed -r 's/(^\/)/.\//g' | sed -r 's/( \/)/ .\//g' | ls -Ralh $(awk '{print $0}') ; cd -
Glue for pasting long string with sub-paths to this command pipe from building block presented earlier out of scope as for this Q.
If possible please review. Thanks.
Does this qualify to be this Q's answer?
I am searching a word in a file through grep command. Now I need to store the status in a variable V1 with 0 or 1. how can i do it?
tail -n 2 test.s | grep -q "FA|"$(date "+%m/%d/%Y")
tail -n 2 test1.s | grep -q "FA|"$(date "+%m/%d/%Y")
tail -n 2 test2.s | grep -q "FA|"$(date "+%m/%d/%Y")
If the above searching word is found then variable V1 value should be 0 else 1.
file content :
keytran|20160111|test.s
submKeyqwqwqw|NDM|Jan 11 01:34|test.s|6666666|sdgdh-RB|ltd.ET.CTS00.act
loadstatus|thunnnB|6666666|FA|01/16/2016|01:34:57|01/16/2016
|01:37:13|load|test.s
please suggest
Depending on your shell, after each command execution the status of the previous command is available in a special variable: bash family $?, csh family $status$:
#/bin/bash
tail -n 2 test.s | grep -q "FA|"$(date "+%m/%d/%Y")
V1=$?
or
#/bin/csh
tail -n 2 test.s | grep -q "FA|"$(date "+%m/%d/%Y")
set V1=$status
I need to move the whole content of a file (test.log) which is constantly in use. Moving the file could result in an error for the application which is writing to the file.
My approach is to redirect the output to test.log_bck, copy the original file (test.log) to test.log_cp, clear it and then check if there was any data added in the same time the test.log file was cleared. If any data was missing from the _cp file then merge it with the _bck file without redundant data. It is a lot of effort for such an easy task and my question is: is there another, easier / more efficient way to do it.
#/usr/bin/bash
#redirect the output to /tmp/logging/test.log_bck
bck(){
`tail -f /tmp/logging/test.log &> /tmp/logging/test.log_bck`
}
#run it in background
bck &
#copy the original file to test.log_cp
`cp /tmp/logging/test.log /tmp/logging/test.log_cp` | echo "Copped"
#clear the original file
echo "" > /tmp/logging/test.log | echo "Cleared"
#get the PID of the redirection process and check if there are other running and kill them
bck_pid=`ps -ef | grep "tail -f /tmp/logging/test.log" | grep -v grep | awk '{print$2}' | head -1`
while [ "$bck_pid" != "" ]
do
echo $bck_pid
kill $bck_pid | echo "Killed"
bck_pid=`ps -ef | grep "tail -f /tmp/logging/test.log" | grep -v grep | awk '{print$2}' | head -1`
done
date=$(date '+%Y_%m_%d_%H_%M')
cat /tmp/logging/test.log_cp /tmp/logging/test.log_bck > /tmp/logging/test.log_$date
cat -n /tmp/logging/test.log_$date | sort -uk2 | sort -nk1 | cut -f2-
I have text coming from some command (in example it's echo -e "10 ABC \n5 DEF \n87 GHI"). This text goes through the pipeline and I get wanted output (in example it's GHI). Wanted output is sent to the following pipeline step (in example it's | xargs -I {} grep -w {} FILES |).
My question is:
I want to append a variable to an "inter pipe" output before it's sent to a following step - How can I do this?
Example:
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 | # Wanted output comes here. I want to append it to a variable before it goes to `grep`
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
You can't set a shell variable in the middle of the pipeline, but you can send the output to a file using the tee command, and then read that file later.
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 |
tee intermediate.txt |
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
# intermediate.txt now contains 87 GHI
How about something like this
echo -e "10 ABC \n5 DEF \n87 GHI" | sort -nr -k1 | head -n1 | cut -f 2 | while read MYVAR; do echo "intermediate value: $MYVAR"; echo $MYVAR | xargs -I {} grep -w {} FILES; done
Insert it to a stream. so I think your looking just to add the ?contents? of a variable to every 'line' from the stream? This prepends the contents of $example
ie
example="A String"
echo -e "10 ABC \n5 DEF \n87 GHI" |
sort -nr -k1 |
head -n1 |
cut -f 2 |
sed s/^/$example/ |
xargs -I {} grep -w {} FILES |
# FOLLOWING ANALYSIS
sed s/$/$example/ to append
NB I tend to do lot of things this way in bash, but a long pipeline of cuts, seds and heads etc does suggest maybe its time to break out awk or perl.