Please help!
I have a little bit of a time sensitive issue here and so I am turning to Stackoverflow in the hope that I can get a fairly quick reply. I'm not a bash expert, especially when having to think quickly. I am trying to awk in two columns from a user database into an OpenStack API. The database (flat file I created) is really simple. It looks something like this:
| ID | username | True |
The database is pretty large so I am trying to loop through this. Here is the command I am trying to run:
for user in $(cat users.txt); do keystone user-role-add --user $user --role blah --tenant $tenant; done
This works great assuming users.txt list had a single username on each line.
I need to modify this command to pull the username out of users.txt for $user and also pull the corresponding ID out of users.txt for $tenant
* SOLUTION *
while read line;do
tenant=$(echo $line|awk '{print $2}')
user=$(echo $line|awk '{print $4}')
keystone user-role-add --user $user --role blah --tenant $tenant
done <users.txt
Thanks again!
This way avoids using awk, it shoud be more optimized:
while read id username tenant; do
echo "---DEBUG --- id=${id} username=${username} tenant=${tenant}"
# do other action here
# calling variables as in DEBUG line above
done <users.txt
You probably want something like...
while read line;do
id=$(echo $line|awk '{print $2}')
username=$(echo $line|awk '{print $4}')
bool=$(echo $line|awk '{print $6}')
# do other
# action here
done <users.txt
awk -F' *[|] *' \
'{print "keystone user-role-add --user \47" $3 "\47 --role blah --tenant \47" $2 "\47"}' users.txt \
| bash
Explanation line by line:
Pass the file through awk using pipe with zero or more leading/trailing spaces as a delimiter.
For each line of file users.txt print the required command as string. $2 and $3 mean second and third field from delimited file. \47 will be interpolated to a single quote in the output, in case we have spaces in values.
Pipe the printed commands from the previous step to bash for execution.
Related
I am writing this bash script that is supposed to print out all the users that have never logged in with an option to sort them. I have managed to get all the input working, however, I am encountering issues when it comes to printing the output. The loop goes as follows:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
done
of course, this loop doesn't sort the output, however, from my limited understanding of shells and shell scripting it should only be a matter of putting a ' | sort' after the first "awk '{print $1}'". my problem is that the output of this loop prints every user at least twice, and in some instances, four times. Why is that and how can I fix it?
Well, let's try to debug it:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep $user /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
This outputs:
The user 'daemon' matches these lines:
daemon 1
colord 112
The user 'bin' matches these lines:
root 0
daemon 1
bin 2
sys 3
sync 4
games 5
man 6
(...)
And indeed, the entry for colord does contain daemon:
colord:x:112:120:colord colour management daemon,,,:/var/lib/colord:/bin/false
^-- Here
And the games entry does match bin:
games:x:5:60:games:/usr/games:/usr/sbin/nologin
^-- Here
So instead of matching the username string anywhere, we just want to match it from the start of the line until the first colon:
for user in $(lastlog | grep -i 'never' | awk '{print $1}'); do
echo "The user '$user' matches these lines:"
grep "^$user:" /etc/passwd | awk -F ':' '{print $1, $3}'
echo
done
And now each entry only shows the singly entry it was supposed to, so you can remove the echos and keep going.
If you're interested in finesse and polish, here's an alternative solution that works efficiently across language settings, weird usernames, network auth, large lists, etc:
LC_ALL=C lastlog |
awk -F ' ' '/Never logged in/ {printf "%s\0", $1}' |
xargs -0 getent passwd |
awk -F : '{print $1,$3}' |
sort
Just think what happens with a user named sh. How many users would grep sh match? Probably all of them, since each is using some shell in the shell field.
You should think about
awk -F ':' '$1 == "'"$user"'" {print $1, $3}' /etc/passwd
or with an awk variable for user:
awk -F ':' -vuser="$user" '$1 == user {print $1, $3}' /etc/passwd
Your grep will match multiple lines, (man will match manuel and norman etc.) anchor it to the beginning of the line and add a trail :.
grep "^${user}:" /etc/passwd | awk -F ':' '{print $1, $3}'
A better option might be to forget about grepping /etc/passwd completely and use the id command to get the user id:
id=$(id -u "${user}" 2>/dev/null) && printf "%s %d\n" "${user}" "${id}"
If the id command fails nothing is printed, or it could be modified to be:
id=$(id -u "${user}" 2>/dev/null)
printf "%s %s\n" "${user}" "${id:-(User not found)}"
In gnu linux I'm pretty sure that the found users id not existing isn't possible as lastlog will only report existing users so the second example may be pointless.
I'm trying to achieve some goal here and while I do know partials steps, I am not successful in putting it all together. I'm looking for an inline command for single usage on multiple hosts. Let's have SW repository file organized like this:
# comments
PROD_NAME:INSTALL_DIR:OPTIONS
PROD_NAME:INSTALL_DIR:OPTIONS
PROD_NAME:INSTALL_DIR:OPTIONS
Now, let's say we want to process the file and do some copy action on every one of the products. So, I can pipe grep getting rid of comment lines into while do cycle, where I use awk to break down each line to product name and it's path and complete it into copy commands. And that's too much of nesting for my skill level, I'm afraid. Anyone who'd care to share?
you can use a bash loop to do the same
$ while IFS=: read -r p i o;
do echo "cp $o $p $i";
done < <(grep -v '^#' file)
cp OPTIONS PROD_NAME INSTALL_DIR
cp OPTIONS PROD_NAME INSTALL_DIR
cp OPTIONS PROD_NAME INSTALL_DIR
remove echo to run as given.
Comments can be removed by
grep -v '^#'
For awk you have to specify the field delimiter:
awk -f: '{print $1, $2, $3}'
In order to craft copy commands you have to pipe the result to a shell.
echo -e '# comments\nNAME:DIR:OPT' |
grep -v '^#' |
awk -F: '{print "cp", $3, $2, $1}' |
sh
Even better: read a book.
Or this:
http://linuxcommand.org/learning_the_shell.php
https://en.wikibooks.org/wiki/Bourne_Shell_Scripting
I run an xargs command invoking bash shell with multiple commands. I am unable to figure out how to print two columns with different delimiters.
The command is ran is below
cd /etc/yp
cat "$userlist" | xargs -I {} bash -c "echo -e 'For user {} \n'
grep -w {} auto_*home|sed 's/:/ /' | awk '{print \$1'\t'\$NF}'
grep -w {} passwd group netgroup |cut -f1 -d ':'|sort|uniq;echo -e '\n'"
the output I get is
For user xyz
auto_homeabc.jkl.com:/rtw2kop/xyz
group
netgroup
passwd
I need a tab after the auto_home(since it is a filename) like in
auto_home abc.jkl.com:/rtw2kop/xyz
The entry from auto_home file is below
xyz -rw,intr,hard,rsize=32768,wsize=32768 abc.jkl.com:/rtw2kop/xyz
How do I awk for the first field(auto_home) and the last field abc.jkl.com:/rtw2kop/xyz? As I have put a pipe from grep command to awk.'\t' isnt working in the above awk command.
If I understand what you are attempting correctly, then I suggest this approach:
while read user; do
echo "For user $user"
awk -v user="$user" '$1 == user { print FILENAME "\t" $NF }' auto_home
awk -F: -v user="$user" '$1 == user { print FILENAME; exit }' passwd group netgroup | sort -u
done < "$userlist"
The basic trick is the read loop, which will read a line into the variable $user from the file named in $userlist; after that, it's all straightforward awk.
I took the liberty of changing the selection criteria slightly; it looked as though you wanted to select for usernames, not strings anywhere in the line. This way, only lines will be selected in which the first token is equal to the currently inspected user, and lines in which other tokens are equal to the username but not the first are discarded. I believe this to be what you want; if it is not, please comment and we can work it out.
In the 1st awk command, double-escape the \t to \\t. (You may also need to double-escape the \n.)
I have a CSV list that is two columns (col1 is Share Name, col2 is file system path). I need two variables for either everything BEFORE the comma, or everything AFTER the column. My issue is that either column potentially has spaces, and even though these are quoted in the output, my script isn't handling them properly.
CSV:
ShareName,/path/to/sharename
"Share with spaces",/path/to/sharewithspaces
ShareWithSpace,"/path/to/share with spaces"
I was using this awk statement to get either field 1 or field 2:
echo $line | awk -F "\"*,\"*" '{print $2}'
BUT, I soon realized that it wasn't handling the spaces properly, even when passing that command to a variable and quoting the variable.
So, then after googling my brain out, I was trying this:
echo $line | cut -d, -f2
Which works, EXCEPT when echoing the variable $line. If I echo the string, it works perfectly, but unfortunately I'm using this in a while/read/do.
I am fairly certain my issue is having to define fields and having whitespace, but I really only need before or after a comma.
Here's the stripped down version so there's no sensitive data.
#!/usr/bin/bash
ssh <ip> <command> > "2_shares.txt"
<command> > "1_shares.txt"
file1="1_shares.txt"
file2="2_shares.txt"
while read -r line
do
share=`echo "$line" | awk -F "\"*,\"*" '{print $1}'`
path=`echo "$line" | awk -F "\"*,\"*" '{print $2}'`
if grep "$path" $file2 > /dev/null;
then
:
else
echo "SHARE NEEDS CREATED FOR $line"
case $path in
*)
blah blah blah
;;
esac
fi
done < "$file1"
You could simply do like this,
awk -F',' '{print $2}' file
To skip the first line.
awk -F',' 'NR>1{print $2}' file
Your issue is simply that you aren't quoting your shell variables. ALWAYS quote shell variables unless you have a very specific reason not to and are fully aware of all of the consequences.
I strongly suspect the rest of your script is completely wrong in it's approach since you apparently didn't know to quote variables and are talking about shell loops and echoing one line at time to awk so please do post a followup question if you'd like help.
Background: The Unclassified RHEL 6 Security Technical Implementation Guide (STIG), a DoD guide, specifies in (STID-ID) RHEL-06-000294 that all user primary GIDs appearing in /etc/passwd must exist in /etc/group.
Instead of running the recommended 'pwck -rq' command and piping to a log then forcing the admin to manually remediate, it makes more sense to programmatically check if the users GID exists and, if not, simply set it to something which does exist such as "users" or "nobody"
I've been beating my head against this and can't quite get it. I've failed at sed, awk, and some piping of grep into sed or awk. The problem seems to be when I attempt to nest commands. I learned the hard way that awk won't nest in one-liners (or possibly at all).
The pseudo-code for a more traditional loop looks something like:
# do while not EOF /etc/passwd
# GIDREF = get 4th entry, sepparator ":" from line
# USERNAMEFOUND = get first entry, separator ":" from line
# grep ":$GIDREF:" /etc/group
# if not found then
# set GID for USERNAMEFOUND to "users"
# fi
# end-do.
This seems like it should be quite simple but I'm apparently missing something.
Thanks for the help.
-Kirk
List all GIDs from /etc/passwd that don't exist in /etc/group:
comm -23 <(awk -F: '{print $4}' /etc/passwd | sort -u) \
<(awk -F: '{print $3}' /etc/group | sort -u)
Fix them:
nogroup=$(awk -F: '($1=="nobody") {print $3}' /etc/group)
for gid in $(
comm -23 <(awk -F: '{print $4}' /etc/passwd | sort -u) \
<(awk -F: '{print $3}' /etc/group | sort -u)); do
awk -v gid="$gid" -F: '($4==gid) {print $1}' /etc/passwd |
xargs -n 1 usermod -g "$nogroup"
done
I'm thinking along these lines:
if ! grep "^$(groups $USERNAME | cut -d\ -f 1):" /etc/group > /dev/null; then
usermod -g users $USERNAME
fi
Where
groups $USERNAME | cut -d\ -f 1
gives the primary group of $USERNAME by splitting the output of groups $USERNAME at the first space (before the :) and
grep ^foo: /etc/group
checks if group foo exists.
EDIT: Fix in the code: quoting and splitting at space instead of colon to allow the appended colon in the grep pattern (otherwise, grep ^foo would have also said that group foo existed if there was a group foobar).