Bash Shell Scripting assigning new variables for output of a grep search - bash

EDIT 2:
I've decided to re-write this in order to better portray my outcome.
I'm currently using this code to output a list of files within various directories:
for file in /directoryX/*.txt
do
grep -rl "Annual Compensation" $file
done
The output shows all files that have a certain table I'm trying to extract in a layout like this:
txtfile1.txt
txtfile2.txt
txtfile3.txt
I have been using this awk command on each individual .txt file to extract the table and then send it to a .csv:
awk '/Annual Compensation/{f=1} f{print; if (/<\/TABLE>/) exit}' txtfile1.txt > txtfile1.csv
My goal is to find a command that will run my awk command against each file in the list all at once. Thank you to those that have provided suggestions already.

If I understand what you're asking, I think what you want to do is add a line after the grep, or instead of the grep, that says:
awk '/Annual Compensation/{f=1} f{print; if (/<\/TABLE>/) exit}' $file > ${file}_new.csv
When you say ${file}_new.csv, it expands the file variable, then adds the string "_new.csv" to it. That's what you're shooting for, right?

Modifying your code:
for file in /directoryX/*.txt
do
files+=($(grep -rl "Annual Compensation" $file))
done
for f in "${files[#]}";do
awk '/Annual Compensation/{f=1} f{print; if (/<\/TABLE>/) exit}' "$f" > "$f"_new.csv
done
Alternative code:
files+=($(grep -rl "Annual Compensation" /directoryX/*))
for f in "${files[#]}";do
awk '/Annual Compensation/{f=1} f{print; if (/<\/TABLE>/) exit}' "$f" > "$f"_new.csv
In both cases, the grep results and awk results are not verified by me - it is just a copy - paste of your code.

Related

Combine multiple files into one including the file name

I have been looking around trying to combine multiple text files into including the name of the file.
My current file content is:
1111,2222,3333,4444
What I'm after is:
File1,1111,2222,3333,4444
File1,1111,2222,3333,4445
File1,1111,2222,3333,4446
File1,1111,2222,3333,4447
File2,1111,2222,3333,114444
File2,1111,2222,3333,114445
File2,1111,2222,3333,114446
I found multiple example how to combine them all but nothing to combine them including the file name.
Could you please try following. Considering that your Input_file names extensions are .csv.
awk 'BEGIN{OFS=","} {print FILENAME,$0}' *.csv > output_file
After seeing OP's comments if file extensions are .txt then try:
awk 'BEGIN{OFS=","} {print FILENAME,$0}' *.txt > output_file
Assuming all your files have a .txt extension and contain only one line as in the example, you can use the following code:
for f in *.txt; do echo "$f,$(cat "$f")"; done > output.log
where output.log is the output file.
Well, it works:
printf "%s\n" *.txt |
xargs -n1 -d $'\n' bash -c 'xargs -n1 -d $'\''\n'\'' printf "%s,%s\n" "$1" <"$1"' --
First output a newline separated list of files.
Then for each file xargs execute sh
Inside sh execute xargs for each line of file
and it executes printf "%s,%s\n" <filename> for each line of input
Tested in repl.
Solved using grep "" *.txt -I > $filename.

How to rename a CSV file from a value in the CSV file

I have 100 1-line CSV files. The files are currently labeled AAA.txt, AAB.txt, ABB.txt (after I used split -l 1 on them). The first field in each of these files is what I want to rename the file as, so instead of AAA, AAB and ABB it would be the first value.
Input CSV (filename AAA.txt)
1234ABC, stuff, stuff
Desired Output (filename 1234ABC.csv)
1234ABC, stuff, stuff
I don't want to edit the content of the CSV itself, just change the filename
something like this should work:
for f in ./* ; do new_name=$(head -1 $f | cut -d, -f1); cp $f dir/$new_name
move them into a new dir just in case something goes wrong, or you need the original file names.
starting with your original file before splitting
$ awk -F, '{print > ($1".csv")}' originalFile.csv
and do all in one shot.
This will store the whole input file into the colum1.csv of the inputfile.
awk -F, '{print $0 > $1".csv" }' aaa.txt
In a terminal, changed directory, e.g. cd /path/to/directory that the files are in and then use the following compound command:
for f in *.txt; do echo mv -n "$f" "$(awk -F, '{print $1}' "$f").cvs"; done
Note: There is an intensional echo command that is there for you to test with, and it will only print out the mv command for you to see that it's the outcome you wish. You can then run it again removing just echo from the compound command to actually rename the files as desired via the mv command.

awk Getting ALL line but last field with the delimiters

I have to make a one-liner that renames all files in the current directory
that end in ".hola" to ".txt".
For example:
sample.hola and name.hi.hola will be renamed to sample.txt and name.hi.txt respectively
I was thinking about something like:
ls -1 *.hola | awk '{NF="";print "$0.hola $0.txt"}' (*)
And then passing the stdin to xargs mv -T with a |
But the output of (*) for the example would be sample and name hi.
How do I get the output name.hi for name.hi.hola using awk?
Why would you want to involve awk in this?
$ for f in *.hola; do echo mv "$f" "${f%hola}txt"; done
mv name.hi.hola name.hi.txt
mv sample.hola sample.txt
Remove the echo when you're happy with the output.
Well, for your specific problem, I recommend the rename command. Depending on the version on your system, you can do either rename -s .hola .txt *.hola, or rename 's/\.hola$/.txt/' *.hola.
Also, you shouldn't use ls to get filenames. When you run ls *.hola, the shell expands *.hola to a list of all the filenames matching that pattern, and ls is just a glorified echo at that point. You can get the same result using e.g. printf '%s\n' *.hola without running any program outside the shell.
And your awk is missing any attempt to remove the .hola. If you have GNU awk, you can do something like this:
awk -F. '{old=$0; NF-=1; new=$0".txt"; print old" "new}'
That won't work on BSD/MacOS awk. In that case you can do something like this:
awk -F. '{
old=$0; new=$1;
for (i=2;i<NF;++i) { new=new"."$i };
print old" "new".txt"; }'
Either way, I'm sure #EdMorton probably has a better awk-based solution.
How about this? Simple and straightforward:
for file in *.hola; do mv "$file" "${file/%hola/txt}"; done

Using awk to extract specific line from all text files in a directory

I have a folder with 50 text files and I want to extract the first line from each of them at the command line and output this to a result.txt file.
I'm using the following command within the directory that contains the files I'm working with:
for files in *; do awk '{if(NR==1) print NR, $0}' *.txt; done > result.txt
When I run the command, the result.txt file contains 50 lines but they're all from a single file in the directory rather than one line per file. The common appears to be looping over a single 50 times rather than over each of the 50 files.
I'd be grateful if someone could help me understand where I'm going wrong with this.
try this -
for i in *.txt;do head -1 $i;done > result.txt
OR
for files in *.txt;do awk 'NR==1 {print $0}' $i;done > result.txt
Your code has two problems:
You have an outer loop that iterates over *, but your loop body doesn't use $files. That is, you're invoking awk '...' *.txt 50 times. This is why any output from awk is repeated 50 times in result.txt.
Your awk code checks NR (the number of lines read so far), not FNR (the number of lines read within the current file). NR==1 is true only at the beginning of the very first file.
There's another problem: result.txt is created first, so it is included among *.txt. To avoid this, give it a different name (one that doesn't end in .txt) or put it in a different directory.
A possible fix:
awk 'FNR==1 {print NR, $0}' *.txt > result
Why not use head? For example with find:
find midir/ -type f -exec head -1 {} \; >> result.txt
If you want to follow your approach you need to specify the file and not use the wildcard with awk:
for files in *; do awk '{if(NR==1) print NR, $0}' "$files"; done > result.txt

awk execute same command on different files one by one

Hi I have 30 txt files in a directory which are containing 4 columns.
How can I execute a same command on each file one by one and direct output to different file.
The command I am using is as below but its being applied on all the files and giving single output. All i want is to call each file one by one and direct outputs to a new file.
start=$1
patterns=''
for i in $(seq -43 -14); do
patterns="$patterns /cygdrive/c/test/kpi/SIGTRAN_Load_$(exec date '+%Y%m%d' --date="-${i} days ${start}")*"; done
cat /cygdrive/c/test/kpi/*$patterns | sed -e "s/\t/,/g" -e "s/ /,/g"| awk -F, 'a[$3]<$4{a[$3]=$4} END {for (i in a){print i FS a[i]}}'| sed -e "s/ /0/g"| sort -t, -k1,2> /cygdrive/c/test/kpi/SIGTRAN_Load.csv
Sth like this
for fileName in /path/to/files/foo*.txt
do
mangleFile "$fileName"
done
will mangle a list of files you give via globbing. If you want to generate the file name patterns as in your example, you can do it like this:
for i in $(seq -43 -14)
do
for fileName in /cygdrive/c/test/kpi/SIGTRAN_Load_"$(exec date '+%Y%m%d' --date="-${i} days ${start}")"*
do
mangleFile "$fileName"
done
done
This way the code stays much more readable, even if shorter solutions may exist.
The mangleFile of course then will be the awk call or whatever you would like to do with each file.
Use the following idiom:
for file in *
do
./your_shell_script_containing_the_above.sh $file > some_unique_id
done
You need to run a loop on all the matching files:
for i in /cygdrive/c/test/kpi/*$patterns; do
tr '[:space:]\n' ',\n' < "$i" | awk -F, 'a[$3]<$4{a[$3]=$4} END {for (i in a){print i FS a[i]}}'| sed -e "s/ /0/g"| sort -t, -k1,2 > "/cygdrive/c/test/kpi/SIGTRAN_Load-$i.csv"
done
PS: I haven't tried much to refactor your piped commands that can probably be shortened too.

Resources