I'm trying to get this script to basically read input from a file on a command line, match the user id in the file using grep and output these lines with line numbers starting from 1)...n in a new file.
so far my script looks like this
#!/bin/bash
linenum=1
grep $USER $1 |
while [ read LINE ]
do
echo $linenum ")" $LINE >> usrout
$linenum+=1
done
when i run it ./username file
i get
line 4: [: read: unary operator expected
could anyone explain the problem to me?
thanks
Just remove the [] around read line - they should be used to perform tests (file exists, string is empty etc.).
How about the following?
$ grep $USER file | cat -n >usrout
Leave off the square brackets.
while read line; do
echo $linenum ")" $LINE
done >> usrout
just use awk
awk -vu="$USER" '$0~u{print ++d") "$0}' file
or
grep $USER file |nl
or with the shell, (no need to use grep)
i=1
while read -r line
do
case "$line" in
*"$USER"*) echo $((i++)) $line >> newfile;;
esac
done <"file"
Why not just use grep with the -n (or --line-number) switch?
$ grep -n ${USERNAME} ${FILE}
The -n switch gives the line number that the match was found on in the file. From grep's man page:
-n, --line-number
Prefix each line of output with the 1-based line number
within its input file.
So, running this against the /etc/passwd file in linux for user test_user, gives:
31:test_user:x:5000:5000:Test User,,,:/home/test_user:/bin/bash
This shows that the test_user account appears on line 31 of the /etc/passwd file.
Also, instead of $foo+=1, you should write foo=$(($foo+1)).
Related
for example our bash script's name is masodik and there is a text.txt with these lines:
qwer
qw2qw
12345
qwert432
Then I write ./masodik text.txt and i got
qw2qw
12345
I tried it many ways and I dont know why this is not working
#!/bin/bash
for i in read u ; do
echo $i $u | grep '^[a-zA-Z0-9][a-zA-Z0-9][0-9]'
done
$ grep -E '^.{2}[0-9]' text.txt
qw2qw
12345
, and in script it could be something like:
#!/bin/sh
grep -E '^.{2}[0-9]' "$1"
To print lines whose third character is a digit:
grep ^..[0-9] text.txt
^ matches the start of the line. The dot . matches any character. [0-9] matches any digit.
You can do it with awk quite easily as well:
awk '/^..[0-9]/' file
Result
With your input in file:
$ awk '/^..[0-9]/' file
qw2qw
12345
(sed works as well, sed -n '/^..[0-9]/p' file)
The problem with the code here:
#!/bin/bash
for i in read u ; do
echo $i $u | grep '^[a-zA-Z0-9][a-zA-Z0-9][0-9]'
done
...is that the for syntax is wrong:
read u is treated as a word list. So the $u variable is never set, so $u stays empty.
The for loop will run twice -- the 1st time $i will be set to the string "read", the 2nd time $i will be set to the string "u". Since neither string contains a number, the grep returns nothing.
The code never reads text.txt.
See Sasha Khapyorsky's answer for actual working code.
If for some odd reason all external utils, (grep, awk, etc.), are forbidden, this pure POSIX code would work:
#!/bin/sh
while read u ; do
case "$u" in
[a-zA-Z0-9][a-zA-Z0-9][0-9]*) echo "$u" ;;
esac
done
If perl is installed into the system then shell script will look like
#!/bin/bash
perl -e 'print if /^.{2}\d/' text.txt
I have a log file with a lot of lines with the following format:
IP - - [Timestamp Zone] 'Command Weblink Format' - size
I want to write a script.sh that gives me the number of times each website has been clicked.
The command:
awk '{print $7}' server.log | sort -u
should give me a list which puts each unique weblink in a separate line. The command
grep 'Weblink1' server.log | wc -l
should give me the number of times the Weblink1 has been clicked. I want a command that converts each line created by the Awk command above to a variable and then create a loop that runs the grep command on the extracted weblink. I could use
while IFS='' read -r line || [[ -n "$line" ]]; do
echo "Text read from file: $line"
done
(source: Read a file line by line assigning the value to a variable) but I don't want to save the output of the Awk script in a .txt file.
My guess would be:
while IFS='' read -r line || [[ -n "$line" ]]; do
grep '$line' server.log | wc -l | ='$variabel' |
echo " $line was clicked $variable times "
done
But I'm not really familiar with connecting commands in a loop, as this is my first time. Would this loop work and how do I connect my loop and the Awk script?
Shell commands in a loop connect the same way they do without a loop, and you aren't very close. But yes, this can be done in a loop if you want the horribly inefficient way for some reason such as a learning experience:
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
n=$(grep -c "$line" server.log)
echo "$line" clicked $n times
done
# you only need the read || [ -n ] idiom if the input can end with an
# unterminated partial line (is illformed); awk print output can't.
# you don't really need the IFS= and -r because the data here is URLs
# which cannot contain whitespace and shouldn't contain backslash,
# but I left them in as good-habit-forming.
# in general variable expansions should be doublequoted
# to prevent wordsplitting and/or globbing, although in this case
# $line is a URL which cannot contain whitespace and practically
# cannot be a glob. $n is a number and definitely safe.
# grep -c does the count so you don't need wc -l
or more simply
awk '{print $7}' server.log |
sort -u |
while IFS= read -r line; do
echo "$line" clicked $(grep -c "$line" server.log) times
done
However if you just want the correct results, it is much more efficient and somewhat simpler to do it in one pass in awk:
awk '{n[$7]++}
END{for(i in n){
print i,"clicked",n[i],"times"}}' |
sort
# or GNU awk 4+ can do the sort itself, see the doc:
awk '{n[$7]++}
END{PROCINFO["sorted_in"]="#ind_str_asc";
for(i in n){
print i,"clicked",n[i],"times"}}'
The associative array n collects the values from the seventh field as keys, and on each line, the value for the extracted key is incremented. Thus, at the end, the keys in n are all the URLs in the file, and the value for each is the number of times it occurred.
I am trying to capture and read into $line the line or lines in file that have only del in them (line 2 is an example). Line 3 has del in it but it also has ins and the bash when executed currently captures both. I am not sure how to exclude anything but del and only capture those lines. Thank you :).
file
NM_003924.3:c.765_779dupGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCinsGGCGGCAGC
NM_003924.3:c.765_779insGGCAGCGGCGGCAGC
desired output
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
bash w/ current output
while read line; do
if [[ $line =~ del ]] ; then echo $line; fi
done < file
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
NM_003924.3:c.765_779delGGCAGCinsGGCGGCAGC
Could you please try following(if ok with awk).
awk '/del/ && !/ins/' Input_file
Try:
while read -r line; do
[[ $line =~ del && ! $line =~ ins ]] && printf '%s\n' "$line"
done < file
The revised code is also ShellCheck clean and avoids BashPitfall #14.
This solution may fail if the last line in the file does not have a terminating newline. If that is a concern, see the accepted answer to Read last line of file in bash script when reading file line by line for a fix.
Here is a sed solution. It negates the match del followed by ins and prints everything that has a del in it. -n to silent every other output.
$ sed -n -e '/del.*ins/!{/.*del.*/p}' inputFile
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
Here is another answer using PCRE enabled grep. This should work with -P option in GNU grep
$ grep -P 'del\.*(?!.*ins)' inputFile
NM_003924.3:c.765_779delGGCAGCGGCGGCAGC
Split it in 2 steps. You do not need a loop:
grep "del" file | grep -v "ins"
How do i iterate through a file and print the first word only. The line is colon separated. example
root:01:02:toor
the file contains several lines. And this is what i've done so far but it does'nt work.
FILE=$1
k=1
while read line; do
echo $1 | awk -F ':'
((k++))
done < $FILE
I'm not good with bash-scripting at all. So this is probably very trivial for one of you..
edit: variable k is to count the lines.
Use cut:
cut -d: -f1 filename
-d specifies the delimiter
-f specifies the field(s) to keep
If you need to count the lines, just
count=$( wc -l < filename )
-l tells wc to count lines
awk -F: '{print $1}' FILENAME
That will print the first word when separated by colon. Is this what you are looking for?
To use a loop, you can do something like this:
$ cat test.txt
root:hello:1
user:bye:2
test.sh
#!/bin/bash
while IFS=':' read -r line || [[ -n $line ]]; do
echo $line | awk -F: '{print $1}'
done < test.txt
Example of reading line by line in bash: Read a file line by line assigning the value to a variable
Result:
$ ./test.sh
root
user
A solution using perl
%> perl -F: -ane 'print "$F[0]\n";' [file(s)]
change the "\n" to " " if you don't want a new line printed.
You can get the first word without any external commands in bash like so:
printf '%s' "${line%%:*}"
which will access the variable named line and delete everything that matches the glob :* and do so greedily, so as close to the front (that's the %% instead of a single %).
Though with this solution you do need to do the loop yourself. If this is the only thing you want to do with the variable the cut solution is better so you don't have to do the file iteration yourself.
I bumped into the following problem: I'm writing a Linux bash script which does the following:
Read line from file
Strip the \n character from the end of the line just read
Execute the command that's in there
Example:
commands.txt
ls
ls -l
ls -ltra
ps as
The execution of the bash file should get the first line, and execute it, but while the \n present, the shell just outputs "command not found: ls"
That part of the script looks like this
read line
if [ -n "$line" ]; then #if not empty line
#myline=`echo -n $line | tr -d '\n'`
#myline=`echo -e $line | sed ':start /^.*$/N;s/\n//g; t start'`
myline=`echo -n $line | tr -d "\n"`
$myline #execute it
cat $fname | tail -n+2 > $fname.txt
mv $fname.txt $fname
fi
Commented you have the things I tried before asking SO. Any solutions? I'm smashing my brains for the last couple of hours over this...
I always like perl -ne 'chomp and print' , for trimming newlines. Nice and easy to remember.
e.g. ls -l | perl -ne 'chomp and print'
However
I don't think that is your problem here though. Although I'm not sure I understand how you're passing the commands in the file through to the 'read' in your shell script.
With a test script of my own like this (test.sh)
read line
if [ -n "$line" ]; then
$line
fi
and a sample input file like this (test.cmds)
ls
ls -l
ls -ltra
If I run it like this ./test.sh < test.cmds, I see the expected result, which is to run the first command 'ls' on the current working directory.
Perhaps your input file has additional non-printable characters in it ?
mine looks like this
od -c test.cmds
0000000 l s \n l s - l \n l s - l t
0000020 r a \n
0000023
From your comments below, I suspect you may have carriage returns ( "\r" ) in your input file, which is not the same thing as a newline. Is the input file originally in DOS format ? If so, then you need to convert the 2 byte DOS line ending "\r\n" to the single byte UNIX one, "\n" to achieve the expected results.
You should be able to do this by swapping the "\n" for "\r" in any of your commented out lines.
Someone already wrote a program which executes shell commands: sh file
If you really only want to execute the first line of a file: head -n 1 file |sh
If your problem is carriage-returns: tr -d '\r' <file |sh
I tried this:
read line
echo -n $line | od -x
For the input 'xxxx', I get:
0000000 7878 7878
As you can see, there is no \n at the end of the contents of the variable. I suggest to run the script with the option -x (bash -x script). This will print all commands as they are executed.
[EDIT] Your problem is that you edited commands.txt on Windows. Now, the file contains CRLF (0d0a) as line delimiters which confuses read (and ls\r is not a known command). Use dos2unix or similar to turn it into a Unix file.
You may also try to replace carriage returns with newlines only using Bash builtins:
line=$'a line\r'
line="${line//$'\r'/$'\n'}"
#line="${line/%$'\r'/$'\n'}" # replace only at line end
printf "%s" "$line" | ruby -0777 -n -e 'p $_.to_s'
you need eval command
#!/bin/bash -x
while read cmd
do
if [ "$cmd" ]
then
eval "$cmd"
fi
done
I ran it as
./script.sh < file.txt
And file.txt was:
ls
ls -l
ls -ltra
ps as
though not working for ls, I recommend having a look at find’s -print0 option
The following script works (at least for me):
#!/bin/bash
while read I ; do if [ "$I" ] ; then $I ; fi ; done ;