pass shell parameters to awk does not work - bash

Why does this work
for myfile in `find . -name "R*VER" -mtime +1`
do
SHELLVAR=`grep ^err $myfile || echo "No error"`
ECHO $SHELLVAR
done
and outputs
No error
err ->BIST Login Fail 3922 err
No error
err ->IR Remote Key 1 3310 err
But this does not
for myfile in `find . -name "R*VER" -mtime +1`
do
SHELLVAR=`grep ^err $myfile || echo "No error"`
awk -v awkvar=${SHELLVAR} '{print awkvar}'
done
and outputs
awk: cmd. line:1: fatal: cannot open file `{print awkvar}' for reading (No such file or directory)
What am I missing?

Does $SHELLVAR contain a space? If so, your awk script is getting misparsed, and the {print awkvar} is being assumed to be a file name and not the actual AWK program.
You also have a problem where both your for loop and the awk program are both slurping STDIN. In fact, your for loop would only be executed once since the AWK will read in all the STDIN until it finishes. In the end, you'll get the same line over and over, and then your program will stop running as the awk awaits for more STDIN.
I think you want to do something like this...
find . -name "R*VER" -mtime +1 | while read myfile
do
echo "$SHELLVAR" | awk '{print $0}'
done
This way, your echo command feeds into your awk which will prevent awk from reading from the find statement.
As an aside, you're better off doing this:
find . -name "R*VER" -mtime +1 | while read myfile
do
...
done
Rather than this:
for myfile in `find . -name "R*VER" -mtime +1`
do
...
done
This is for several reasons:
The command line buffer could overflow, and you'll lose file names, but you'll never see an error.
The find command in the second example must first complete before the for loop can start to execute. In the first example, the find feeds into the while loop.
ADDENDUM
Now that I saw just-my-correct-opinion's answer, I realize what you've really done wrong: You forgot the file name in the awk command:
find . -name "R*VER" -mtime +1 | while read myfile
do
SHELLVAR=`grep ^err $myfile || echo "No error"`
awk -v awkvar=${SHELLVAR} '{print awkvar}' $myfile
done
Now, the question is what exactly are you doing with the awk. You're not printing anything except the value of $SHELVAR for each and every line in the file. That's probably not what you want to do. In fact, why not simply do this:
find . -name "R*VER" -mtime +1 | while read myfile
do
SHELLVAR=$(grep -q "^err" $myfile")
if [ "x$SHELLVAR" != "x" ]
then
echo "$SHELLVAR"
fi
done
That way, you print out $SHELLVAR, but only if $SHELLVAR is empty.
Or, you can use awk to print out only those lines that match your regex:
find . -name "R*VER" -mtime +1 | while read myfile
do
awk '/^err/ {print $0}' $myfile
done

What you're trying to do is possible but ... a bit quirky. Here's an alternative for you:
for f in $(find . -name "R*VER" -mtime +1)
do
echo "$f"
awk 'BEGIN{ec=0} /^err/{ec+=1;print} END{if(ec==0){print "No error"}}' "$f"
done
This way you don't have to worry about grep, don't have to worry about shell variables and can keep your logic all in one place in one language.
Note that this code is only partially tested since I don't have your data files, but the testing I did worked fine.
If you'd like you can even go a step farther and write the whole thing in awk. It is, after all, a full, general-purpose programming language (with, IMO, a far cleaner syntax than bash). This way you avoid the need for find, grep and bash entirely. I won't be writing that script, however. You'll have to pick up the awk man page and read up on file I/O yourself.

You need to quote $SHELLVAR, to prevent the shell from splitting it.
awk -v awkvar="${SHELLVAR}" '{print awkvar}'

You have already got alternative solutions. Regardless of that, I just want to answer your question, ie the thing that you are missing:
awk -v awkvar=${SHELLVAR} '{print awkvar}'
Here awk is seeking to read the input from STDIN. And that's the problem. Awk seeks to read input from STDIN, unless you specify input file(s). The given commands to awk are executed for each RECORD in the input (and by default a record is a line). See man awk to read more on awk.
But here is a hack, if you want it to proceed without any input:
awk -v awkvar=${SHELLVAR} 'BEGIN{print awkvar}'
BEGIN block is executed as soon as the awk is called. And if awk doesn't find any other block except BEGIN, it executes those BEGIN blocks and exits.
I hope you got the problem behind the error, and a quick solution for that as well.

Related

Rename files named foobar(12345).txt to 12345.txt

All:
Quickly and succinctly, I have many many files named as such:
lorem(12312315).txt
ipsum(578938-12-315-13-416-4).txt
amet(ran-dom-guid).txt
And I want to rename them to what's inside the parentheses dot text, like so:
12312315.txt
578938-12-315-13-416-4.txt
randomguid.txt
I'm sure a mix of sed, awk, grep, etc will do it, but commenting out the parentheses from the shell is throwing me. I cant come up with a string that will do it.
If anyone is kind enough to share a few thought cycles and help me, it would be a lovely Karma gesture!
Thanks for reading!
-Jim
Another flavor:
find . -type f -name \*\(\*\).txt -print0 | xargs -0 sh -c '
for filename ; do
basename_core="${filename##*(}"
basename_core="${basename_core%%)*}"
mv "${filename}" "${basename_core}".txt
done' dummy
This might work for you (GNU sed and shell);
sed -n 's/.*(\(.*\)).*/mv '\''&'\'' '\''\1.txt'\''/p' *.txt
This will print out a list of move commands, after you have validated they are correct, pipe to shell:
sed -n 's/.*(\(.*\)).*/mv '\''&'\'' '\''\1.txt'\''/p' *.txt | sh
find and mv can handle this, with a bash rematch to find your names;
#!/bin/bash
touch lorem\(12312315\).txt
touch ipsum\(578938-12-315-13-416-4\).txt
touch amet\(ran-dom-guid\).txt
pat=".*\((.*)\).txt"
for f in $(find . -type f -name "*.txt" ); do
if [[ $f =~ $pat ]]; then
mv $f ${BASH_REMATCH[1]}.txt
fi
done
ls *.txt
A for loop and Parameter Expansion.
#!/usr/bin/env bash
for f in *\(*\)*.txt; do
temp=${f#*\(}
value=${temp%\)*}
case $value in
*[!0-9-]*) value="${value//-}";;
esac
echo mv -v "$f" "$value.txt"
done
Remove the echo if you're satisfied with the output, so mv can rename/move the files.
Thank you everyone for the responses! I ended up using a mishmash of your suggestions and doing something else entirely, but I'm posting here for posterity...
The files all had one thing in common, the GUID contained in the filename was also always contained in line 2 of the accompanying file, so I yank lane two, strip out the things that are NOT the guid, and rename the file to that string, for any .xml file in the directory where the script is run.
as such:
for i in ./*xml
do
GUID=`cat "$i" | sed -n '2p' | awk '{print $1}' | sed 's/<id>//g' | sed 's/<\/id>//'`
echo File "|" $i "|" is "|" $GUID
done
In the actual script, I do a MV instead of an ECHO and the files are renamed to the guid.
Hopefully this helps someone else in the future, and yes, I know it's wasteful to call sed three times. If I were better with regular expressions, I'm sure I could get that down to one! :)
Thanks!

How to get the nth recent file in the nth last modified subdirectory using pipes

I'm doing an exercise for OS exam. It requires to get the 3rd recent file of the 2nd last modified sub-directory inside current directory. Then I have to print its lines in reverse order. I can not use tac command. The text suggest to use (other than awk and sed): head, tails, wc.
I've succeded getting filename of the requested file (but in a too complex way I think). Now I have to print it in reverse. I think I can use this awk solution https://stackoverflow.com/a/744093/11614625.
This is how I'm getting the filename:
ls -t | head | awk '{system("test -d \"" $0 "\" && echo \"" $0 "\"")}' | awk 'NR==2 {system("ls \"" $0 "\" | head")}' | awk 'NR==1'
How can I do better? And what if 3rd directory or 2nd file doesn't exists?
See https://mywiki.wooledge.org/ParsingLs and awk '{system("test -d \"" $0 "\" && echo \"" $0 "\"")}' is calling shell to call awk to call system to call shell to call test which is clearly a worse approach than just having shell call test in the first place if you were going to do that. Also, any solution that reads the whole file into memory (as any sed or a naive awk solution would) will fail for large files as they'll exceed available memory.
Unfortunately this is how to do what you want robustly:
dir="$(find . -mindepth 1 -maxdepth 1 -type d -printf '%T+\t%p\0' |
sort -rz |
awk -v RS='\0' 'NR==2{sub(/[^\t]+\t/,""); print; exit}')" &&
file="$(find "$dir" -mindepth 1 -maxdepth 1 -type f -printf '%T+\t%p\0' |
sort -z |
awk -v RS='\0' 'NR==3{sub(/[^\t]+\t/,""); print; exit}')" &&
cat -n "$file" | sort -rn | cut -f2-
If any of the commands in any of the pipes fail then the error message from the command that failed will be printed and then no other command will execute and the overall exit status will be the failure one from that failing command.
I used cat | sort | cut rather than awk or sed to print the file in reverse because awk (unless you write demand paging in it) or sed would have to read the whole file into memory at once and so would fail for very large files while sort is designed to handle large files by using paging with tmp files as necessary and only keeping parts of the file in memory at a time so it's limited only by how much free disk space you have on your device.
The above requires GNU tools to provide/handle NUL line-endings - if you don't have those then change \0 to \n in the find command, remove the z from sort options, and remove -v RS='\0' from the awk command and be aware that the result will only work if your directory or file names don't contain newlines.

syntax error not sure what its saying or how to fix it

I am trying to write a shell script for school that searches your entire home directory for all files with the .java extension. For each such file, list the number of lines in the file along with its location (that is, its full path).
my script looks like
#!/bin/bash
total=0
for currfile in $(find ~ -name "*.java" -print)
do
total=$[total+($(wc -l $currfile| awk '{print $1}'))]
echo -n 'total=' $total
echo -e -n '\r'
done
echo 'total=' $total
when i run it from the konsole i get error
./fileQuest.sh: line 5: total+(): syntax error: operand expected (error token is ")")
I am a novice and cannot figure out what the error is telling me. Any help would be appreciated
total+()
This is the expression that's being evaluated inside of $[...]. Notice that the parentheses are empty. There should be a number there. It indicates that the $(wc | awk) bit is yielding an empty string.
total=$[total+($(wc -l $currfile| awk '{print $1}'))]
# ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If that part is blank then you get:
total=$[total+()]
Note that wc can handle multiple file names natively. You don't need to write your own loop. You could use find -exec to call it directly instead.
find ~ -name "*.java" -exec wc {} +

Get index of argument with xargs?

In bash, I have list of files all named the same (in different sub directories) and I want to order them by creation/modified time, something like this:
ls -1t /tmp/tmp-*/my-file.txt | xargs ...
I would like to rename those files with some sort of index or something so I can move them all into the same folder. My result would ideally be something like:
my-file0.txt
my-file1.txt
my-file2.txt
Something like that. How would I go about doing this?
You can just loop through these files and keep appending an incrementing counter to desired file name:
for f in /tmp/tmp-*/my-file.txt; do
fname="${f##*/}"
fname="${fname%.*}"$((i++)).txt
mv "$f" "/dest/dir/$fname"
done
EDIT: In order to sort listed files my modification time as is the case with ls -1t you can use this script:
while IFS= read -d '' -r f; do
f="${f#* }"
fname="${f##*/}"
fname="${fname%.*}"$((i++)).txt
mv "$f" "/dest/dir/$fname"
done < <(find /tmp/tmp-* -name 'my-file.txt' -printf "%T# %p\0" | sort -zk1nr)
This handles filenames with all special characters like white spaces, newlines, glob characters etc since we are ending each filename with NUL or \0 character in -printf option. Note that we are also using sort -z to handle NUL terminated data.
So I found an answer to my own question, thoughts on this one?
ls -1t /tmp/tmp-*/my-file.txt | awk 'BEGIN{ a=0 }{ printf "cp %s /tmp/all-the-files/my-file_%03d.txt\n", $0, a++ }' | bash;
I found this from another stack overflow question looking for something similar that my search didn't find at first. I was impressed with the awk line, thought that was pretty neat.

Individually Execute All Files in a Directory

I've been trying to write a shell script to travel a directory tree and play every mp3 file it finds. afplay is my utility of choice, given that I am on a mac. However, afplay only takes one argument at a time, so you have to call it over and over again if you want it to keep playing. It seems like the simplest solution would be as follows:
$(`find . -name *.mp3 | awk '{ print "afplay \047" $0 "\047"; }' | tr '\n' ';' | sed 's/;/; /g'`)
...but something keeps getting caught up in the escaping of quotes. For quick reference, \047 is octal for ' (the single quote character), which should encapsulate arguments into one, but for some reason it is not. I have no clue what is going wrong here.
Why not just find . -name '*.mp3' -exec afplay '{}' \;?
If all of your songs look like this:
1. song_name.mp3
2. song_name.mp3
3. song_name.mp3
...
20. song_name.mp3
to play all 20 of them you could just loop
for ((i=1; i<=20; i++)); do afplay $i* ; done

Resources