how can I create an array in one step instead of two stages, like shown below?'
The example below was executed on a live Linux system.
POSITION=`volt |grep ate |awk '{print $4}'` #returns three integers
declare -a POSITION_ARRAY=($POSITION) #create an array
You don't need the intermediate variable, as wjandrea said. These two snippets are equivalent:
POSITION=$(volt | grep ate | awk '{print $4}')
declare -a POSITION_ARRAY=($POSITION)
# declare -a also works, but isn't needed in modern Bash
POSITION_ARRAY=( $(volt | grep ate | awk '{print $4}') )
If you know the output of the pipeline is witespace-delimited integers this will do what you want. But it isn't a safe way to populate an array from arbitrary command output, because unquoted expansions will be word-split and globbed.
The proper way to read a command's output into an array, split by lines, is with the readarray builtin, like so:
readarray -t POSITION_ARRAY < <(volt | grep ate | awk '{print $4}')
Simply put the command in the parentheses.
By the way, declare -a is not needed, and backticks are deprecated in favour of $().
POSITION_ARRAY=( $(volt | grep ate | awk '{print $4}') )
And FWIW you can merge the grep and AWK commands:
POSITION_ARRAY=( $(volt | awk '/ate/ {print $4}') )
Related
I am writing this bash script that is supposed to print out all the users that have never logged in with an option to sort them. I have managed to get all the input working, however, I am encountering issues when it comes to printing the output. The loop goes as follows:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
done
of course, this loop doesn't sort the output, however, from my limited understanding of shells and shell scripting it should only be a matter of putting a ' | sort' after the first "awk '{print $1}'". my problem is that the output of this loop prints every user at least twice, and in some instances, four times. Why is that and how can I fix it?
Well, let's try to debug it:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
This outputs:
The user 'daemon' matches these lines:
daemon 1
colord 112
The user 'bin' matches these lines:
root 0
daemon 1
bin 2
sys 3
sync 4
games 5
man 6
(...)
And indeed, the entry for colord does contain daemon:
colord:x:112:120:colord colour management daemon,,,:/var/lib/colord:/bin/false
^-- Here
And the games entry does match bin:
games:x:5:60:games:/usr/games:/usr/sbin/nologin
^-- Here
So instead of matching the username string anywhere, we just want to match it from the start of the line until the first colon:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep "^$user:" /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
And now each entry only shows the singly entry it was supposed to, so you can remove the echos and keep going.
If you're interested in finesse and polish, here's an alternative solution that works efficiently across language settings, weird usernames, network auth, large lists, etc:
LC_ALL=C lastlog |
awk -F ' ' '/Never logged in/ {printf "%s\0", $1}' |
xargs -0 getent passwd |
awk -F : '{print $1,$3}' |
sort
Just think what happens with a user named sh. How many users would grep sh match? Probably all of them, since each is using some shell in the shell field.
You should think about
awk -F ':' '$1 == "'"$user"'" {print $1, $3}' /etc/passwd
or with an awk variable for user:
awk -F ':' -vuser="$user" '$1 == user {print $1, $3}' /etc/passwd
Your grep will match multiple lines, (man will match manuel and norman etc.) anchor it to the beginning of the line and add a trail :.
grep "^${user}:" /etc/passwd | awk -F ':' '{print $1, $3}'
A better option might be to forget about grepping /etc/passwd completely and use the id command to get the user id:
id=$(id -u "${user}" 2>/dev/null) && printf "%s %d\n" "${user}" "${id}"
If the id command fails nothing is printed, or it could be modified to be:
id=$(id -u "${user}" 2>/dev/null)
printf "%s %s\n" "${user}" "${id:-(User not found)}"
In gnu linux I'm pretty sure that the found users id not existing isn't possible as lastlog will only report existing users so the second example may be pointless.
Ok so with the new High Sierra, I am trying to write a script to automatically delete there local snapshots that eat up HDD space. I know you can shrink using thinlocalsnapshots / 1000000000 4 but I feel like that is only a band-aid.
So what I am trying to do is extract the date 2018-02-##-###### from:
sudo tmutil listlocalsnapshots /
com.apple.TimeMachine.2018-02-15-170531
com.apple.TimeMachine.2018-02-15-181655
com.apple.TimeMachine.2018-02-15-223352
com.apple.TimeMachine.2018-02-16-000403
com.apple.TimeMachine.2018-02-16-013400
com.apple.TimeMachine.2018-02-16-033621
com.apple.TimeMachine.2018-02-16-063811
com.apple.TimeMachine.2018-02-16-080812
com.apple.TimeMachine.2018-02-16-090939
com.apple.TimeMachine.2018-02-16-100459
com.apple.TimeMachine.2018-02-16-110325
com.apple.TimeMachine.2018-02-16-122954
com.apple.TimeMachine.2018-02-16-141223
com.apple.TimeMachine.2018-02-16-151309
com.apple.TimeMachine.2018-02-16-161040
I have tried variations of
| awk '{print $ } (insert number after $)
along with
| cut -d ' ' -f 10-.
Please if you know what I am missing here I would greatly appreciate it
edit: Here is the script that will get rid of those pesky Local snapshots.If anyone is interested, Thanks again:
#! /bin/bash
dates=`tmutil listlocalsnapshots / | awk -F "." 'NR++1{print $4}'`
for dates in $dates
do
tmutil deletelocalsnapshots $dates
done
You were close:
somecommand | cut -d"." -f4-
# or
somecommand | awk -F"." '{print $4}'
You can also try sed, but cut is made for this.
1- awk: you can either specify the field separator with the -F option, or print a substring
awk -F. '{print $4}'
awk '{print substr($0,23)}'
2- cut: equivalently.
cut -d. -f4
cut -c23-
3- Pure bash (sloooooow!): same as above.
while IFS=. read s1 s2 s3 d; do echo "$d"; done
while read line; do echo "${line:23}"; done
In practice, with a small number of records as in your use case, speed is not an issue and even pure bash or regexps (as in other aswers) can be used. As the number of records grows, the higher speed of awk and cut becomes noticeable.
Using grep and a regex :
$ grep -oP '\d{4}-\d{2}-\d{2}-\d{6}$'
2018-02-15-170531
2018-02-15-181655
2018-02-15-223352
2018-02-16-000403
2018-02-16-013400
2018-02-16-033621
2018-02-16-063811
2018-02-16-080812
2018-02-16-090939
2018-02-16-100459
2018-02-16-110325
2018-02-16-122954
2018-02-16-141223
2018-02-16-151309
2018-02-16-161040
I am new to Bash scripting, but do understand most of the basics. My scenario is as follows:
I have a server from which I get a load of data via cURL. This is parsed properly (XML format) and from these results I then extract the data I want. The cURL statement writes its output to a file called temp-rec-schedule.txt. The below code is what I use to get the values I want to use in further calculation.
MP=`cat temp-rec-schedule.txt | grep "<ns3:mediapackage" | awk -F' ' '{print $3}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}'`
REC_TIME=`cat temp-rec-schedule.txt | grep "<ns3:mediapackage" | awk -F' ' '{print $2}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}'`
So this all still work perfectly. The output of the above code is respectively (if written to two separate files):
MP output:
b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f
fd578fcc-342c-4f6c-986a-794ccb1abd0c
ce9f40e9-8e2c-4654-ba1c-7f79d35a69fd
c31a2354-6f4b-4bfe-b51e-2bac80889897
df342d88-c660-490e-9da6-9c91a7966536
49083f88-4264-4629-80fb-fae480d0bb25
946121c7-4948-4254-9cb5-2457e1b99685
f7bd0cad-e8f5-4e3d-a219-650d07a4bb34
REC_TIME output:
2014-09-15T07:30:00Z
2014-09-19T08:58:00Z
2014-09-22T07:30:00Z
2014-10-13T07:30:00Z
2014-10-17T08:58:00Z
2014-10-20T07:30:00Z
2014-10-22T13:28:00Z
2014-10-27T07:30:00Z
What I want to do now is create a file where line1 from file1 is appended with line1 from file2. i.e. :
b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T07:30:00Z
fd578fcc-342c-4f6c-986a-794ccb1abd0c 2014-09-19T08:58:00Z
and so on.
I am not really familiar with Perl, but do know a little bit about Bash, so if it is possible, I would like to do this in Bash.
Further, from here, I want to compare two files that contain the same MP variable, but have two different TIME values assigned: subtract the one value from the other, and calculate the amount of hours that have passed between. This is all to calculate the amount of hours that have passed between publishing a video on our system, and the start time of the recording. Basically:
File1's output: b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T07:30:00Z
File2's output: b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T09:30:00Z
The output of my script should yield a value of 2 hours.
How can I do this with Bash?
You're probably better off just using awk for the whole thing. Something like:
awk '/<ns3:medipacakge/{gsub("\"","");
split($3,mp,"=");
split($2,rt,"="); print mp[2],rt[2]}' temp-rec-schedule.txt
The answer to the first question is to write the output to two different files and then use paste.
grep "<ns3:mediapackage" temp-rec-schedule.txt | awk -F' ' '{print $3}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}' > MP_out.txt
grep "<ns3:mediapackage" temp-rec-schedule.txt | awk -F' ' '{print $2}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}' > REC_out.txt
paste MP_out.txt REC_out.txt
That being said (and as #WilliamPursell says in his comment on the OP) there is never a reason to string this series of commands together since awk can do all the things you are doing there with significantly less overhead and more flexibility.
Why do these do different things?
ENTRY="banana#Apple"
HOST_ID=$ENTRY | awk -F '#' '{print $2}'
echo $HOST_ID
echo $ENTRY | awk -F '#' '{print $2}'
In the echo command, the data is displayed as expected. In the save to variable command, the data is not, and the variable is left empty.
This:
HOST_ID=$ENTRY | awk -F '#' '{print $2}'
means this:
HOST_ID='banana#Apple' | awk -F '#' '{print $2}'
In other words, you're running two commands in separate subshells — HOST_ID='banana#Apple', and awk -F '#' '{print $2}' — and piping the output of one to the other. This doesn't accomplish anything: HOST_ID='banana#Apple' produces no output, so the awk command gets no input, so it doesn't print anything. (And because of the subshells, even the HOST_ID='banana#Apple' part has no effect: it's setting that variable in a subshell, rather than in the parent shell that's running the overall command. So the value disappears almost immediately)
Instead, you want to write:
HOST_ID="$(echo "$ENTRY" | awk -F '#' '{print $2}')
or:
HOST_ID="$(awk -F '#' '{print $2}' <<< "$ENTRY")
or perhaps (if you're only ever expecting $ENTRY to have two fields):
HOST_ID="${ENTRY#*#}"
I need to grep multiple strings, but i don't know the exact number of strings.
My code is :
s2=( $(echo $1 | awk -F"," '{ for (i=1; i<=NF ; i++) {print $i} }') )
for pattern in "${s2[#]}"; do
ssh -q host tail -f /some/path |
grep -w -i --line-buffered "$pattern" > some_file 2>/dev/null &
done
now, the code is not doing what it's supposed to do. For example if i run ./script s1,s2,s3,s4,.....
it prints all lines that contain s1,s2,s3....
The script is supposed to do something like grep "$s1" | grep "$s2" | grep "$s3" ....
grep doesn't have an option to match all of a set of patterns. So the best solution is to use another tool, such as awk (or your choice of scripting languages, but awk will work fine).
Note, however, that awk and grep have subtly different regular expression implementations. It's not clear from the question whether the target strings are literal strings or regular expression patterns, and if the latter, what the expectations are. However, since the argument comes delimited with commas, I'm assuming that the pieces are simple strings and should not be interpreted as patterns.
If you want the strings to be interpreted as patterns, you can change index to match in the following little program:
ssh -q host tail -f /some/path |
awk -v STRINGS="$1" -v IGNORECASE=1 \
'BEGIN{split(STRINGS,strings,/,/)}
{for(i in strings)if(!index($0,strings[i]))next}
{print;fflush()}'
Note:
IGNORECASE is only available in gnu awk; in (most) other implementations, it will do nothing. It seems that is what you want, based on the fact that you used -i in your grep invocation.
fflush() is also an extension, although it works with both gawk and mawk. In Posix awk, fflush requires an argument; if you were using Posix awk, you'd be better off printing to stderr.
You can use extended grep
egrep "$s1|$s2|$s3" fileName
If you don't know how many pattern you need to grep, but you have all of them in an array called s, you can use
egrep $(sed 's/ /|/g' <<< "${s[#]}") fileName
This creates a herestring with all elements of the array, sed replaces the field separator of bash (space) with | and if we feed that to egrep we grep all strings that are in the array s.
test.sh:
#!/bin/bash -x
a=" $#"
grep ${a// / -e } .bashrc
it works that way:
$ ./test.sh 1 2 3
+ a=' 1 2 3'
+ grep -e 1 -e 2 -e 3 .bashrc
(here is lots of text that fits all the arguments)