Loop that prints twice in bash - bash

I am writing this bash script that is supposed to print out all the users that have never logged in with an option to sort them. I have managed to get all the input working, however, I am encountering issues when it comes to printing the output. The loop goes as follows:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
done
of course, this loop doesn't sort the output, however, from my limited understanding of shells and shell scripting it should only be a matter of putting a ' | sort' after the first "awk '{print $1}'". my problem is that the output of this loop prints every user at least twice, and in some instances, four times. Why is that and how can I fix it?

Well, let's try to debug it:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
This outputs:
The user 'daemon' matches these lines:
daemon 1
colord 112
The user 'bin' matches these lines:
root 0
daemon 1
bin 2
sys 3
sync 4
games 5
man 6
(...)
And indeed, the entry for colord does contain daemon:
colord:x:112:120:colord colour management daemon,,,:/var/lib/colord:/bin/false
^-- Here
And the games entry does match bin:
games:x:5:60:games:/usr/games:/usr/sbin/nologin
^-- Here
So instead of matching the username string anywhere, we just want to match it from the start of the line until the first colon:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep "^$user:" /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
And now each entry only shows the singly entry it was supposed to, so you can remove the echos and keep going.
If you're interested in finesse and polish, here's an alternative solution that works efficiently across language settings, weird usernames, network auth, large lists, etc:
LC_ALL=C lastlog |
awk -F ' ' '/Never logged in/ {printf "%s\0", $1}' |
xargs -0 getent passwd |
awk -F : '{print $1,$3}' |
sort

Just think what happens with a user named sh. How many users would grep sh match? Probably all of them, since each is using some shell in the shell field.
You should think about
awk -F ':' '$1 == "'"$user"'" {print $1, $3}' /etc/passwd
or with an awk variable for user:
awk -F ':' -vuser="$user" '$1 == user {print $1, $3}' /etc/passwd

Your grep will match multiple lines, (man will match manuel and norman etc.) anchor it to the beginning of the line and add a trail :.
grep "^${user}:" /etc/passwd | awk -F ':' '{print $1, $3}'
A better option might be to forget about grepping /etc/passwd completely and use the id command to get the user id:
id=$(id -u "${user}" 2>/dev/null) && printf "%s %d\n" "${user}" "${id}"
If the id command fails nothing is printed, or it could be modified to be:
id=$(id -u "${user}" 2>/dev/null)
printf "%s %s\n" "${user}" "${id:-(User not found)}"
In gnu linux I'm pretty sure that the found users id not existing isn't possible as lastlog will only report existing users so the second example may be pointless.

Related

how to pass values from stdout as parameter for the next command

i want to svn blame lines of code which include "todo | fixme"
i have the general flow of the script but struggle to combine it into one
finding the lines with "todo"
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" .
blame the line of code
svn blame ${file} | cat -n |grep ${linenumber}
i could get $file and $linenumber from the first command with awk, but i dont know how to pipe the values i extract with awk into the second command.
i am missing the glue to combine these commands into one "script" (- :
You can build the command with awk and then pipe it to bash:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}'
That prints one command per input line with the following format:
svn blame "${file}" | cat -n | grep "${linenumber}"
The varibales are replaces. When you execute the command as above they are only printed to the shell, that you can comfirm if everything is right. If yes add a last pipe to the in of the command that the ouput is redirected to bash. The complete command would look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | grep \"%s\"\n", $1, $2}' | bash
A small notice: I think you want to print the line number extracterd in the first command, aren't you? But grep ${linenumber} just gives the line containing the string ${linenumber}. To print only the linenumber use that command: sed -n "2p" to print line number 2 for example. The complete command would then look like this:
grep --color -Ern --include=*.{php,html,phtml} --exclude-dir=vendor "todo|TODO|FIXME" . |\
awk -F: '{printf "svn blame \"%s\" | cat -n | sed -n \"%sp\"\n", $1, $2}' | bash

awk for different delimiters piped from xargs command

I run an xargs command invoking bash shell with multiple commands. I am unable to figure out how to print two columns with different delimiters.
The command is ran is below
cd /etc/yp
cat "$userlist" | xargs -I {} bash -c "echo -e 'For user {} \n'
grep -w {} auto_*home|sed 's/:/ /' | awk '{print \$1'\t'\$NF}'
grep -w {} passwd group netgroup |cut -f1 -d ':'|sort|uniq;echo -e '\n'"
the output I get is
For user xyz
auto_homeabc.jkl.com:/rtw2kop/xyz
group
netgroup
passwd
I need a tab after the auto_home(since it is a filename) like in
auto_home abc.jkl.com:/rtw2kop/xyz
The entry from auto_home file is below
xyz -rw,intr,hard,rsize=32768,wsize=32768 abc.jkl.com:/rtw2kop/xyz
How do I awk for the first field(auto_home) and the last field abc.jkl.com:/rtw2kop/xyz? As I have put a pipe from grep command to awk.'\t' isnt working in the above awk command.
If I understand what you are attempting correctly, then I suggest this approach:
while read user; do
echo "For user $user"
awk -v user="$user" '$1 == user { print FILENAME "\t" $NF }' auto_home
awk -F: -v user="$user" '$1 == user { print FILENAME; exit }' passwd group netgroup | sort -u
done < "$userlist"
The basic trick is the read loop, which will read a line into the variable $user from the file named in $userlist; after that, it's all straightforward awk.
I took the liberty of changing the selection criteria slightly; it looked as though you wanted to select for usernames, not strings anywhere in the line. This way, only lines will be selected in which the first token is equal to the currently inspected user, and lines in which other tokens are equal to the username but not the first are discarded. I believe this to be what you want; if it is not, please comment and we can work it out.
In the 1st awk command, double-escape the \t to \\t. (You may also need to double-escape the \n.)

How to ensure that gid in /etc/passwd also exist in /etc/group

Background: The Unclassified RHEL 6 Security Technical Implementation Guide (STIG), a DoD guide, specifies in (STID-ID) RHEL-06-000294 that all user primary GIDs appearing in /etc/passwd must exist in /etc/group.
Instead of running the recommended 'pwck -rq' command and piping to a log then forcing the admin to manually remediate, it makes more sense to programmatically check if the users GID exists and, if not, simply set it to something which does exist such as "users" or "nobody"
I've been beating my head against this and can't quite get it. I've failed at sed, awk, and some piping of grep into sed or awk. The problem seems to be when I attempt to nest commands. I learned the hard way that awk won't nest in one-liners (or possibly at all).
The pseudo-code for a more traditional loop looks something like:
# do while not EOF /etc/passwd
# GIDREF = get 4th entry, sepparator ":" from line
# USERNAMEFOUND = get first entry, separator ":" from line
# grep ":$GIDREF:" /etc/group
# if not found then
# set GID for USERNAMEFOUND to "users"
# fi
# end-do.
This seems like it should be quite simple but I'm apparently missing something.
Thanks for the help.
-Kirk
List all GIDs from /etc/passwd that don't exist in /etc/group:
comm -23 <(awk -F: '{print $4}' /etc/passwd | sort -u) \
<(awk -F: '{print $3}' /etc/group | sort -u)
Fix them:
nogroup=$(awk -F: '($1=="nobody") {print $3}' /etc/group)
for gid in $(
comm -23 <(awk -F: '{print $4}' /etc/passwd | sort -u) \
<(awk -F: '{print $3}' /etc/group | sort -u)); do
awk -v gid="$gid" -F: '($4==gid) {print $1}' /etc/passwd |
xargs -n 1 usermod -g "$nogroup"
done
I'm thinking along these lines:
if ! grep "^$(groups $USERNAME | cut -d\ -f 1):" /etc/group > /dev/null; then
usermod -g users $USERNAME
fi
Where
groups $USERNAME | cut -d\ -f 1
gives the primary group of $USERNAME by splitting the output of groups $USERNAME at the first space (before the :) and
grep ^foo: /etc/group
checks if group foo exists.
EDIT: Fix in the code: quoting and splitting at space instead of colon to allow the appended colon in the grep pattern (otherwise, grep ^foo would have also said that group foo existed if there was a group foobar).

Read two files simultaneously and create one from them

I am new to Bash scripting, but do understand most of the basics. My scenario is as follows:
I have a server from which I get a load of data via cURL. This is parsed properly (XML format) and from these results I then extract the data I want. The cURL statement writes its output to a file called temp-rec-schedule.txt. The below code is what I use to get the values I want to use in further calculation.
MP=`cat temp-rec-schedule.txt | grep "<ns3:mediapackage" | awk -F' ' '{print $3}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}'`
REC_TIME=`cat temp-rec-schedule.txt | grep "<ns3:mediapackage" | awk -F' ' '{print $2}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}'`
So this all still work perfectly. The output of the above code is respectively (if written to two separate files):
MP output:
b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f
fd578fcc-342c-4f6c-986a-794ccb1abd0c
ce9f40e9-8e2c-4654-ba1c-7f79d35a69fd
c31a2354-6f4b-4bfe-b51e-2bac80889897
df342d88-c660-490e-9da6-9c91a7966536
49083f88-4264-4629-80fb-fae480d0bb25
946121c7-4948-4254-9cb5-2457e1b99685
f7bd0cad-e8f5-4e3d-a219-650d07a4bb34
REC_TIME output:
2014-09-15T07:30:00Z
2014-09-19T08:58:00Z
2014-09-22T07:30:00Z
2014-10-13T07:30:00Z
2014-10-17T08:58:00Z
2014-10-20T07:30:00Z
2014-10-22T13:28:00Z
2014-10-27T07:30:00Z
What I want to do now is create a file where line1 from file1 is appended with line1 from file2. i.e. :
b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T07:30:00Z
fd578fcc-342c-4f6c-986a-794ccb1abd0c 2014-09-19T08:58:00Z
and so on.
I am not really familiar with Perl, but do know a little bit about Bash, so if it is possible, I would like to do this in Bash.
Further, from here, I want to compare two files that contain the same MP variable, but have two different TIME values assigned: subtract the one value from the other, and calculate the amount of hours that have passed between. This is all to calculate the amount of hours that have passed between publishing a video on our system, and the start time of the recording. Basically:
File1's output: b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T07:30:00Z
File2's output: b1706f0d-2cf1-4fd6-ab60-ae4d08608f1f 2014-09-15T09:30:00Z
The output of my script should yield a value of 2 hours.
How can I do this with Bash?
You're probably better off just using awk for the whole thing. Something like:
awk '/<ns3:medipacakge/{gsub("\"","");
split($3,mp,"=");
split($2,rt,"="); print mp[2],rt[2]}' temp-rec-schedule.txt
The answer to the first question is to write the output to two different files and then use paste.
grep "<ns3:mediapackage" temp-rec-schedule.txt | awk -F' ' '{print $3}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}' > MP_out.txt
grep "<ns3:mediapackage" temp-rec-schedule.txt | awk -F' ' '{print $2}' | cut -d '=' -f 2 | awk -F\" '{print $(NF-1)}' > REC_out.txt
paste MP_out.txt REC_out.txt
That being said (and as #WilliamPursell says in his comment on the OP) there is never a reason to string this series of commands together since awk can do all the things you are doing there with significantly less overhead and more flexibility.

Shell Scripting: Echo vs save to variable

Why do these do different things?
ENTRY="banana#Apple"
HOST_ID=$ENTRY | awk -F '#' '{print $2}'
echo $HOST_ID
echo $ENTRY | awk -F '#' '{print $2}'
In the echo command, the data is displayed as expected. In the save to variable command, the data is not, and the variable is left empty.
This:
HOST_ID=$ENTRY | awk -F '#' '{print $2}'
means this:
HOST_ID='banana#Apple' | awk -F '#' '{print $2}'
In other words, you're running two commands in separate subshells — HOST_ID='banana#Apple', and awk -F '#' '{print $2}' — and piping the output of one to the other. This doesn't accomplish anything: HOST_ID='banana#Apple' produces no output, so the awk command gets no input, so it doesn't print anything. (And because of the subshells, even the HOST_ID='banana#Apple' part has no effect: it's setting that variable in a subshell, rather than in the parent shell that's running the overall command. So the value disappears almost immediately)
Instead, you want to write:
HOST_ID="$(echo "$ENTRY" | awk -F '#' '{print $2}')
or:
HOST_ID="$(awk -F '#' '{print $2}' <<< "$ENTRY")
or perhaps (if you're only ever expecting $ENTRY to have two fields):
HOST_ID="${ENTRY#*#}"

Resources