I have simple script that looks like
for file in `ls -rlt *.rules | awk '{print $9}'`
do
cat $file | awk -F"|" -v DATE=$(date +%Y"_"%m"_"%d) '!$3{$3=DATE} !$4{$4=DATE} 1' OFS="|" $file
done
How can i redirect output of awk to the same file which it is reading to perform action.
files have data before running above script
123|test||
After running script files should have data like
123|test|2017_04_05|2017_04_05
You cannot replace your files on the fly like this, mostly because you increase their size.
The way is to use temporary file, then replace the current:
for file in `ls -1 *.rules `
do
TMP_FILE=/tmp/${file}_$$
awk -F"|" -v DATE=$(date +%Y"_"%m"_"%d) '!$3{$3=DATE} !$4{$4=DATE} 1' OFS="|" $file > ${TMP_FILE}
mv ${TMP_FILE} $file
done
I would modify Michael Vehrs otherwise good answer as follows:
ls -rt *.rules | while read file
do
TMP_FILE="/tmp/${file}_$$"
awk -F"|" -v DATE=$(date +%Y"_"%m"_"%d) \
'!$3{$3=DATE} !$4{$4=DATE} 1' OFS="|" "$file" > "$TMP_FILE"
mv "$TMP_FILE" "$file"
done
Your question uses ls(1) to sort the files by time, oldest first. The above preserves that property. I removed the {} braces because they add nothing in a shell script if the variable name isn't being interpolated, and quotes to cope with filenames that include whitespace.
If time-order doesn't matter, I'd consider an inside-out solution: in awk, write to a temporary file instead of standard output, and then rename it with system in an END block. Then if something goes wrong your input is preserved.
First of all, it is silly to use a combination of ls -rlt and awk when the only thing you need is the file name. You don't even need ls because the shell glob is expanded by the shell, not ls. Simply use for file in *.rules. Since the date would seem to be the same for every file (unless you run the command at midnight), it is sufficient to calculate it in advance:
date=$(date +%Y"_"%m"_"%d)
for file in *.rules
do
TMP_FILE=$(mktemp ${file}_XXXXXX)
awk -F"|" -v DATE=${date} '!$3{$3=DATE} !$4{$4=DATE} 1' OFS="|" $file > ${TMP_FILE}
mv ${TMP_FILE} $file
done
However, since awk also knows which file it is reading, you could do something like this:
awk -F"|" -v DATE=$(date +%Y"_"%m"_"%d) \
'!$3{$3=DATE} !$4{$4=DATE} { print > FILENAME ".tmp" }' OFS="|" *.rules
rename .tmp "" *.rules.tmp
Related
I have been looking around trying to combine multiple text files into including the name of the file.
My current file content is:
1111,2222,3333,4444
What I'm after is:
File1,1111,2222,3333,4444
File1,1111,2222,3333,4445
File1,1111,2222,3333,4446
File1,1111,2222,3333,4447
File2,1111,2222,3333,114444
File2,1111,2222,3333,114445
File2,1111,2222,3333,114446
I found multiple example how to combine them all but nothing to combine them including the file name.
Could you please try following. Considering that your Input_file names extensions are .csv.
awk 'BEGIN{OFS=","} {print FILENAME,$0}' *.csv > output_file
After seeing OP's comments if file extensions are .txt then try:
awk 'BEGIN{OFS=","} {print FILENAME,$0}' *.txt > output_file
Assuming all your files have a .txt extension and contain only one line as in the example, you can use the following code:
for f in *.txt; do echo "$f,$(cat "$f")"; done > output.log
where output.log is the output file.
Well, it works:
printf "%s\n" *.txt |
xargs -n1 -d $'\n' bash -c 'xargs -n1 -d $'\''\n'\'' printf "%s,%s\n" "$1" <"$1"' --
First output a newline separated list of files.
Then for each file xargs execute sh
Inside sh execute xargs for each line of file
and it executes printf "%s,%s\n" <filename> for each line of input
Tested in repl.
Solved using grep "" *.txt -I > $filename.
I have 100 1-line CSV files. The files are currently labeled AAA.txt, AAB.txt, ABB.txt (after I used split -l 1 on them). The first field in each of these files is what I want to rename the file as, so instead of AAA, AAB and ABB it would be the first value.
Input CSV (filename AAA.txt)
1234ABC, stuff, stuff
Desired Output (filename 1234ABC.csv)
1234ABC, stuff, stuff
I don't want to edit the content of the CSV itself, just change the filename
something like this should work:
for f in ./* ; do new_name=$(head -1 $f | cut -d, -f1); cp $f dir/$new_name
move them into a new dir just in case something goes wrong, or you need the original file names.
starting with your original file before splitting
$ awk -F, '{print > ($1".csv")}' originalFile.csv
and do all in one shot.
This will store the whole input file into the colum1.csv of the inputfile.
awk -F, '{print $0 > $1".csv" }' aaa.txt
In a terminal, changed directory, e.g. cd /path/to/directory that the files are in and then use the following compound command:
for f in *.txt; do echo mv -n "$f" "$(awk -F, '{print $1}' "$f").cvs"; done
Note: There is an intensional echo command that is there for you to test with, and it will only print out the mv command for you to see that it's the outcome you wish. You can then run it again removing just echo from the compound command to actually rename the files as desired via the mv command.
I'm trying to get something like basename of second field in a file and replace it:
$ myfile=/var/lib/jenkins/myjob/myfile
$ sha512sum "$myfile" | tee myfile-checksum
$ cat myfile-checksum
deb32b1c7122fc750a6742765e0e54a821 /var/lib/jenkins/myjob/myfile
Desired output:
deb32b1c7122fc750a6742765e0e54a821 myfile
So people can easily do sha512sum -c myfile-checksum with no manual edits.
With sed or awk, that is how far i made it for now :)
awk -F/ '{print $NF}' myfile-checksum
sed -i "s|${value}|$(basename $value)|" myfile-checksum
Thanks.
You can set the field separators to both spaces and slashes and print the first and last fields:
awk -F" |/" '{print $1, $NF}'
With your input:
$ awk -F" |/" '{print $1, $NF}' <<< "deb32b1c7122fc750a6742765e0e54a821 /var/lib/jenkins/myjob/myfile"
deb32b1c7122fc750a6742765e0e54a821 myfile
In case your filename contain spaces, do remove everything from the first field up to the last slash, as indicated by Ed Morton:
$ awk '{hash=$1; gsub(/^.*\//,""); print hash, $0}' <<< "deb32b1c7122fc750a6742765e0e54a821 /var/lib/jenkins/myjob/myfile with spaces"
deb32b1c7122fc750a6742765e0e54a821 myfile with spaces
$ awk 'sub(".*/",$1" ")' <<< "deb32b1c7122fc750a6742765e0e54a821 /var/lib/jenkins/myjob/myfile"
deb32b1c7122fc750a6742765e0e54a821 myfile
The will work for any file name except one that contains newlines. If you have that case let us know.
sha512sum will simply use the file name you've passed to it - unchanged.
If you pass
sha512sum /path/to/file
it will give you:
123456.. /path/to/file
But if you:
pushd /path/to
sha512sum file
popd
it will give you
123456.. file
If the filename is a variable you can use parameter expansion like this:
pushd "${file%/*}"
sha256sum "${file##*/}"
popd
or even
# cd will not change the PWD of the current shell since
# the command runs in a sub shell
(cd "${file%/*}"; sha256sum "${file##*/}")
Having that $file contains the filename, ${file%/*} expands to the path without the filename and ${file##*/} expands to the filename without the path.
I had written an awk code for deleting all the lines ending in a colon from a file. But now I want to run this particular awk action on a whole folder containing similar files.
awk '!/:$/' qs.txt > fin.txt
awk '{print $3 " " $4}' fin.txt > out.txt
You could wrap your awk command in a loop in your shell such as bash.
myfiles=mydirectory/*.txt
for file in $myfiles
do
b=$(basename "$file" .txt)
awk '!/:$/' "$b.txt" > "$b.out"
done
EDIT: improved quoting as commenters suggested
If you like it better, you can use "${file%.txt}" instead of $(basename "$file" .txt).
Aside: My own preference runs to basename just because man basename is easier for me than man -P 'less -p "^ Param"' bash (when that is the relevant heading on the particular system). Please accept this quirk of mine and let's not discuss info and http://linux.die.net/man/ and whatever.
You could use sed. Just run the below command on the directory in which the files you want to change was actually stored.
sed -i '/:$/d' *.*
This will create new files in an empty directory, with the same name.
mkdir NEWFILES
for file in `find . -name "*name_pattern*"`
do
awk '!/:$/' $file > fin.txt
awk '{print $3 " " $4}' fin.txt > NEWFILES/$file
done
After that you just need to
cp -fr NEWFILES/* .
Hi I have 30 txt files in a directory which are containing 4 columns.
How can I execute a same command on each file one by one and direct output to different file.
The command I am using is as below but its being applied on all the files and giving single output. All i want is to call each file one by one and direct outputs to a new file.
start=$1
patterns=''
for i in $(seq -43 -14); do
patterns="$patterns /cygdrive/c/test/kpi/SIGTRAN_Load_$(exec date '+%Y%m%d' --date="-${i} days ${start}")*"; done
cat /cygdrive/c/test/kpi/*$patterns | sed -e "s/\t/,/g" -e "s/ /,/g"| awk -F, 'a[$3]<$4{a[$3]=$4} END {for (i in a){print i FS a[i]}}'| sed -e "s/ /0/g"| sort -t, -k1,2> /cygdrive/c/test/kpi/SIGTRAN_Load.csv
Sth like this
for fileName in /path/to/files/foo*.txt
do
mangleFile "$fileName"
done
will mangle a list of files you give via globbing. If you want to generate the file name patterns as in your example, you can do it like this:
for i in $(seq -43 -14)
do
for fileName in /cygdrive/c/test/kpi/SIGTRAN_Load_"$(exec date '+%Y%m%d' --date="-${i} days ${start}")"*
do
mangleFile "$fileName"
done
done
This way the code stays much more readable, even if shorter solutions may exist.
The mangleFile of course then will be the awk call or whatever you would like to do with each file.
Use the following idiom:
for file in *
do
./your_shell_script_containing_the_above.sh $file > some_unique_id
done
You need to run a loop on all the matching files:
for i in /cygdrive/c/test/kpi/*$patterns; do
tr '[:space:]\n' ',\n' < "$i" | awk -F, 'a[$3]<$4{a[$3]=$4} END {for (i in a){print i FS a[i]}}'| sed -e "s/ /0/g"| sort -t, -k1,2 > "/cygdrive/c/test/kpi/SIGTRAN_Load-$i.csv"
done
PS: I haven't tried much to refactor your piped commands that can probably be shortened too.