when redirecting output from a bash script to a file, I get special characters in the file. For example,
for file in *; do echo $file; done > output.txt
then if I cat output.txt
cat output.txt
I get
file1.txt
file2.txt
file3.txt
output.txt
but when editing the file, I see this:
^[[0m^[[0mfile1.txt
^[[0m^[[0mfile2.txt
^[[0m^[[0mfile3.txt
^[[0m^[[0moutput.txt
How do I avoid those nasty characters?
Solution:
I had the following line in the .bashrc:
trap 'echo -ne "\e[0m"' DEBUG
by removing it, I solved the problem.
Thank you all for your help.
These are ANSI escape codes, used for formatting text in a terminal. Rather than trying to remove them, you should prevent them from being written in the first place.
Are you sure you're getting this from the exact code you posted? If so, your files actually have these characters in your names, and you should simply rename them.
The far more common way of seeing this is having tools that output ANSI escapes sequences. This is a reproducible way of showing the same issue:
ls --color=always > file
If your posted code was an untested example, you should go through and find the tool responsible for the ANSI codes and make it stop (make especially sure you're not looping over ls output).
Here's an example of the problem you're seeing, with touch as stand-in for some process/script that accidentally created filenames with ANSI escapes:
# Reproduce the problem
$ touch $'\x1B[0m\x1B[0mfile.txt'
# Symptoms of the problem
$ ls *.txt
?[0m?[0mfile.txt
$ for f in *.txt; do echo "$f"; done
file.txt
$ for f in *.txt; do echo "$f"; done | cat -v
^[[0m^[[0mfile.txt
# Fix the problem by renaming the bad files
$ crud=$'\x1B[0m'; for f in *"$crud"*; do mv "$f" "${f//$crud/}"; done
# Now works as expected
$ ls *.txt
file.txt
$ for f in *.txt; do echo "$f"; done
file.txt
$ for f in *.txt; do echo "$f"; done | cat -v
file.txt
Run the raw ls command
/bin/ls
as
/bin/ls > your_file
and you will avoid the special characters in the output file your_file.
You can avoid those nasty characters with the ls -f option (which disables color) like so:
for file in $(ls -f *); do echo $file; done > output.txt
Paul
Related
I have some strings like:
dimension\u003d1920x1024:format\u003djpg
In a file. I want to decode them so they will look like:
dimension=1920x1024:format=jpg
I know that:
$ echo -e dimension\u003d1920x1024:format\u003djpg
dimensionu003d1920x1024:formatu003djpg
$ echo -e 'dimension\u003d1920x1024:format\u003djpg'
dimension=1920x1024:format=jpg
$ echo -e "dimension\u003d1920x1024:format\u003djpg"
dimension=1920x1024:format=jpg
So I tried this to get what I want:
$ cat file | xargs -L1 echo -e
dimensionu003d1920x1024:formatu003djpg
But as you can see it doesn't work. How can I get this to work? How can I make xargs pass parameters to echo as if they were quoted?
You are actually asking how to convert the sequence \uXXXX into the corresponding Unicode code point. That's quite different from other backslash escapes, or handling backslashes in general. Neither echo -e nor xargs is particularly suited for this task.
Here is one way:
perl -CSD -pe 's/\\u(\X{4})/chr(oct("0x$1"))/ge' <<<"string"
Obscurely, oct("0xff") actually performs hex decoding, because of the "0x" prefix.
Obviously, if your input is the text in a file rather than just a string in the shell, simply pass that as the argument to Perl.
For small files:
Bash:
cat file | echo -e "$(cat -)"
Zsh:
cat file | { echo -e "$(cat -)"; }
For large files in both bash and zsh:
cat file | while read -r LINE; do echo -e "$LINE"; done
(loses spaces at the beginning of the line)
This is a try with ruby where the changes are written to the file
$ cat ./file
dimension\u003d1920x1024:format\u003djpg
dimension=800x600:format\u003djpg
The example above is made a bit more real-world.
$ cat ./script.rb
#!/usr/bin/ruby
contents=File.read("#{ARGV[0]}")
file=File.open("#{ARGV[0]}","w")
if file
file.syswrite(contents.gsub(/\\[uU]\{?([0-9A-F]{4})\}?/i) { $1.hex.chr(Encoding::UTF_8) })
file.close()
else
puts "No file with name #{ARGV[0]} present, Usage script <filename>"
end
$ ./script file
# The changes are written to the file with nothing printed to stdout
$ cat ./file
dimension=1920x1024:format=jpg
dimension=800x600:format=jpg
I have a list of filenames like this in bash
UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R1.fq.gz
UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R2.fq.gz
UTSHoS11_Other_AGGCCT-TTAGGA_R_160418.R2.fq.gz
UTSHoS11_Other_AGGCCT-TTAGGA_R_160418.R2.fq.gz
UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R1.fq.gz
UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R2.fq.gz
And I want them to look like this
UTSHoS10_R1.fq.gz
UTSHoS10_R2.fq.gz
UTSHoS11_R1.fq.gz
UTSHoS11_R2.fq.gz
UTSHoS12_R1.fq.gz
UTSHoS12_R2.fq.gz
I do not have the perl rename command and sed 's/_Other*160418./_/' *.gz
is not doing anything. I've tried other rename scripts on here but either nothing occurs or my shell starts printing huge amounts of code to the console and freezes.
This post (Removing Middle of Filename) is similar however the answers given do not explain what specific parts of the command are doing so I could not apply it to my problem.
Parameter expansions in bash can perform string substitutions based on glob-like patterns, which allows for a more efficient solution than calling an extra external utility such as sed in each loop iteration:
for f in *.gz; do echo mv "$f" "${f/_Other_*-TTAGGA_R_160418./_}"; done
Remove the echo before mv to perform actual renaming.
You can do something like this in the directory which contains the files to be renamed:
for file_name in *.gz
do
new_file_name=$(sed 's/_[^.]*\./_/g' <<< "$file_name");
mv "$file_name" "$new_file_name";
done
The pattern (_[^.]*\.) starts matching from the FIRST _ till the FIRST . (both inclusive). [^.]* means 0 or more non-dot (or non-period) characters.
Example:
AMD$ ls
UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R1.fq.gz UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R1.fq.gz
UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R2.fq.gz UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R2.fq.gz
UTSHoS11_Other_AGGCCT-TTAGGA_R_160418.R2.fq.gz
AMD$ for file_name in *.gz
> do new_file_name=$(sed 's/_[^.]*\./_/g' <<< "$file_name")
> mv "$file_name" "$new_file_name"
> done
AMD$ ls
UTSHoS10_R1.fq.gz UTSHoS10_R2.fq.gz UTSHoS11_R2.fq.gz UTSHoS12_R1.fq.gz UTSHoS12_R2.fq.gz
Pure Bash, using substring operation and assuming that all file names have the same length:
for file in UTS*.gz; do
echo mv -i "$file" "${file:0:9}${file:38:8}"
done
Outputs:
mv -i UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R1.fq.gz UTSHoS10_R1.fq.gz
mv -i UTSHoS10_Other_CAAGCC-TTAGGA_R_160418.R2.fq.gz UTSHoS10_R2.fq.gz
mv -i UTSHoS11_Other_AGGCCT-TTAGGA_R_160418.R2.fq.gz UTSHoS11_R2.fq.gz
mv -i UTSHoS11_Other_AGGCCT-TTAGGA_R_160418.R2.fq.gz UTSHoS11_R2.fq.gz
mv -i UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R1.fq.gz UTSHoS12_R1.fq.gz
mv -i UTSHoS12_Other_GGCAAG-TTAGGA_R_160418.R2.fq.gz UTSHoS12_R2.fq.gz
Once verified, remove echo from the line inside the loop and run again.
Going with your sed command, this can work as a bash one-liner:
for name in UTSH*fq.gz; do newname=$(echo $name | sed 's/_Other.*160418\./_/'); echo mv $name $newname; done
Notes:
I've adjusted your sed command: it had an * without a preceeding . (sed takes a regular expression, not a globbing pattern). Similarly, the dot needs escaping.
To see if it works, without actually renaming the files, I've left the echo command in. Easy to remove just that to make it functional.
It doesn't have to be a one-liner, obviously. But sometimes, that makes editing and browsing your command-line history easier.
I have a config file which contains some ENV_VARIABLE styled variables.
This is my file.
It might contain $EXAMPLES of text.
Now I want that variable replaced with a value which is saved in my actual environment variables. So I'm trying this:
export EXAMPLES=lots
envsubst < file.txt > file.txt
But it doesn't work when the input file and output file are identical. The result is an empty file of size 0.
There must be a good reason for this, some bash basics that I'm not aware of?
How do I achieve what I want to do, ideally without first outputting to a different file and then replacing the original file with it?
I know that I can do it easily enough with sed, but when I discovered the envsubst command I thought that it should be perfect for my use case, so I'd like to use that.
Here is the solution that I use:
originalfile="file.txt"
tmpfile=$(mktemp)
cp --attributes-only --preserve $originalfile $tmpfile
cat $originalfile | envsubst > $tmpfile && mv $tmpfile $originalfile
Be careful with other solutions that do not use a temporary file. Pipes are asynchronous, so the file will occasionally be read after it has already been truncated.
Redirects are handled by the shell, not the program being executed, and they are set up before the program is invoked.
The redirect >output.file has the effect of creating output.file if it doesn't exist and emptying it if it does. Either way, you end up with an empty file, and that is what the program's output is redirected to.
Programs like sed which are capable of "in-place" modification must take the filename as a command-line argument, not as a redirect.
In your case, I would suggest using a temporary file and then renaming it if all goes OK.
envsubst < file.txt | tee file.txt
I found another shortcut to put into temp file and then rename it to original file.
envsubst < in.txt > out.txt && mv out.txt in.txt
To avoid creating a temporary file, use sponge not tee:
envsubst < file.txt | sponge file.txt
From https://linux.die.net/man/1/sponge:
sponge reads standard input and writes it out to the specified file. Unlike a shell redirect, sponge soaks up all its input before opening the output file. This allows constricting pipelines that read from and write to the same file.
You can achieve in-place substitution by calling envsubst from gnu sed with the "e" command:
EXAMPLES=lots sed -i 's/.*/echo & | envsubst/e' file.txt
It's worth noting that the mv solution won't maintain file permissions. Using cp -pf would be preferable in the case that you're modifying an executable file.
tmpfile=$(mktemp)
cat file.txt | envsubst > "$tmpfile" && cp -pf "$tmpfile" file.txt
rm -f "$tmpfile"
This answer was framed from two other answers. I guess this is the best solution.
originalFile=file.txt
tmpfile=$(mktemp)
cat $originalFile | envsubst > "$tmpfile" && cp -pf "$tmpfile" $originalFile
rm -f "$tmpfile"
Updated 20221011 - Using 1 sed command
sed -i -r 's/["`]|\$\(/\\&/g; s/.*/echo "&"/ e' ./input.txt
Updated 20221007 - Using 2 sed commands
sed -i -r 's/["`]|\$\(/\\&/g' input.txt
sed -i -r 's/.*/echo "&"/ e' input.txt
Do it without envsubst
envsubst_file () {
local original_file=$1
local temp_file=$(mktemp)
trap "rm -f ${temp_file}" 0 2 3 15
cp -p ${original_file} ${temp_file}
cat ${original_file} | sed -r 's/["`]|\$\(/\\&/g' | sed -r 's/.*/echo "&"/g' | sh > ${temp_file}
mv ${temp_file} ${original_file}
}
envsubst_file 'input.txt'
First using sed to escapes double quotes("), backtick(`) and command $( by prefixing with backslash(\),then using sed again replace with
echo "&"
Finally executing the shell script and redirecting to ${temp_file}
If you use bash, check this:
a=`<file.txt` && envsubst <<<"$a" >file.txt
Tested on 500mb file, works as expected.
In the end I found that using envsubst was too dangerous after all. My files might contain dollar signs in places where I don't want any substitution to happen, and envsubst will just replace them with empty strings if no corresponding environment variable is defined. Not cool.
I am trying to modify a bash script to remove a glob of malicious code from a large number of files.
The community will benefit from this, so here it is:
#!/bin/bash
grep -r -l 'var createDocumentFragm' /home/user/Desktop/infected_site/* > /home/user/Desktop/filelist.txt
for i in $(cat /home/user/Desktop/filelist.txt)
do
cp -f $i $i.bak
done
for i in $(cat /home/user/Desktop/filelist.txt)
do
$i | sed 's/createDocumentFragm.*//g' > $i.awk
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
This is where the script bombs out with this message:
+ for i in '$(cat /home/user/Desktop/filelist.txt)'
+ sed 's/createDocumentFragm.*//g'
+ /home/user/Desktop/infected_site/index.htm
I get 2 errors and the script stops.
/home/user/Desktop/infected_site/index.htm: line 1: syntax error near unexpected token `<'
/home/user/Desktop/infected_site/index.htm: line 1: `<html><head><script>(function (){ '
I have the first 2 parts done.
The files containing createDocumentfragm have been enumerated in a text file correctly.
The files in the textfile.txt have been duplicated, in their original location with a .bak added to them IE: infected_site/some_directory/infected_file.htm and infected_file.htm.bak
effectively making sure we have a backup.
All I need to do now is write an AWK command that will use the list of files in filelist.txt, use the entire glob of malicious text as a pattern, and remove it from the files. Using just the uppercase script as the starting point, and the lower case script is too generic and could delete legitimate text
I suspect this may help me, but I don't know how to use it correctly.
http://backreference.org/2010/03/13/safely-escape-variables-in-awk/
Once I have this part figured out, and after you have verified that the files weren't mangled you can do this to clean out the bak files:
for i in $(cat /home/user/Desktop/filelist.txt)
do
rm -f $i.bak
done
Several things:
You have:
$i | sed 's/var createDocumentFragm.*//g' > $i.awk
You should probably meant this (using your use of cat which we'll talk about in a moment):
cat $i | sed 's/var createDocumentFragm.*//g' > $i.awk
You're treating each file in your file list as if it was a command and not a file.
Now, about your use of cat. If you're using cat for almost anything but concatenating multiple files together, you probably are doing something not quite right. For example, you could have done this:
sed 's/var createDocumentFragm.*//g' "$i" > $i.awk
I'm also a bit confused about the awk statement. Exactly what file are you using awk on? Your awk statement is using STDIN and STDOUT, so it's reading file names from the for loop and then printing the output on the screen. Is the sed statement suppose to feed into the awk statement?
Note that I don't have to print out my file to STDOUT, then pipe that into sed. The sed command can take the file name directly.
You also want to avoid for loops over a list of files. That is very inefficient, and can cause problems with the command line getting overloaded. Not a big issue today, but can affect you when you least suspect it. What happens is that your $(cat /home/user/Desktop/filelist.txt) must execute first before the for loop can even start.
A little rewriting of your program:
cd ~/Desktop
grep -r -l 'var createDocumentFragm' infected_site/* > filelist.txt
while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" > "$i.awk"
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
done < filelist.txt
We can use one loop, and we made it a while loop. I could even feed the grep into that while loop:
grep -r -l 'var createDocumentFragm' infected_site/* | while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" > "$i.awk"
awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p'
done < filelist.txt
and then I don't even have to create a temporary file.
Let me know what's going on with the awk. I suspect you wanted something like this:
grep -r -l 'var createDocumentFragm' infected_site/* | while read file
do
cp -f "$file" "$file.bak"
sed 's/var createDocumentFragm.*//g' "$file" \
| awk '/<\/SCRIPT>/{p=1;print}/<\/script>/{p=0}!p' > "$i.awk"
done < filelist.txt
Also note I put quotes around file names. This helps prevent problems if file name has a space in it.
I am using sed in a script to do a replace and I want to have the replaced file overwrite the file. Normally I think that you would use this:
% sed -i 's/cat/dog/' manipulate
sed: illegal option -- i
However as you can see my sed does not have that command.
I tried this:
% sed 's/cat/dog/' manipulate > manipulate
But this just turns manipulate into an empty file (makes sense).
This works:
% sed 's/cat/dog/' manipulate > tmp; mv tmp manipulate
But I was wondering if there was a standard way to redirect output into the same file that input was taken from.
I commonly use the 3rd way, but with an important change:
$ sed 's/cat/dog/' manipulate > tmp && mv tmp manipulate
I.e. change ; to && so the move only happens if sed is successful; otherwise you'll lose your original file as soon as you make a typo in your sed syntax.
Note! For those reading the title and missing the OP's constraint "my sed doesn't support -i": For most people, sed will support -i, so the best way to do this is:
$ sed -i 's/cat/dog/' manipulate
Yes, -i is also supported in FreeBSD/MacOSX sed, but needs the empty string as an argument to edit a file in-place.
sed -i "" 's/old/new/g' file # FreeBSD sed
If you don't want to move copies around, you could use ed:
ed file.txt <<EOF
%s/cat/dog/
wq
EOF
Kernighan and Pike in The Art of Unix Programming discuss this issue. Their solution is to write a script called overwrite, which allows one to do such things.
The usage is: overwrite file cmd file.
# overwrite: copy standard input to output after EOF
opath=$PATH
PATH=/bin:/usr/bin
case $# in
0|1) echo 'Usage: overwrite file cmd [args]' 1>&2; exit 2
esac
file=$1; shift
new=/tmp/overwr1.$$; old=/tmp/overwr2.$$
trap 'rm -f $new $old; exit 1' 1 2 15 # clean up
if PATH=$opath "$#" >$new
then
cp $file $old # save original
trap '' 1 2 15 # wr are commmitted
cp $new $file
else
echo "overwrite: $1 failed, $file unchanged" 1>&2
exit 1
fi
rm -f $new $old
Once you have the above script in your $PATH, you can do:
overwrite manipulate sed 's/cat/dog/' manipulate
To make your life easier, you can use replace script from the same book:
# replace: replace str1 in files with str2 in place
PATH=/bin:/usr/bin
case $# in
0|2) echo 'Usage: replace str1 str2 files' 1>&2; exit 1
esac
left="$1"; right="$2"; shift; shift
for i
do
overwrite $i sed "s#$left#$right#g" $i
done
Having replace in your $PATH too will allow you to say:
replace cat dog manipulate
You can use sponge from the moreutils.
sed "s/cat/dog/" manipulate | sponge manipulate
Perhaps -i is gnu sed, or just an old version of sed, but anyways. You're on the right track. The first option is probably the most common one, the third option is if you want it to work everywhere (including solaris machines)... :) These are the 'standard' ways of doing it.
To change multiple files (and saving a backup of each as *.bak):
perl -p -i -e "s/oldtext/newtext/g" *
replaces any occurence of oldtext by newtext in all files in the current folder. However you will have to escape all perl special characters within oldtext and newtext using the backslash
This is called a “Perl pie” (mnemonic: easy as a pie)
The -i flag tells it do do in-place replacement, and it should be ok to use single (“'”) as well as double (“””) quotes.
If using ./* instead of just *, you should be able to do it in all sub-directories
See man perlrun for more details, including how to take a backup file of the original.
using sed:
sed -i 's/old/new/g' ./* (used in GNU)
sed -i '' 's/old/new/g' ./* (used in FreeBSD)
-i option is not available in standard sed.
Your alternatives are your third way or perl.
A lot of answers, but none of them is correct. Here is the correct and simplest one:
$ echo "111 222 333" > file.txt
$ sed -i -s s/222/444/ file.txt
$ cat file.txt
111 444 333
$
Workaround using open file handles:
exec 3<manipulate
Prevent open file from being truncated:
rm manipulate
sed 's/cat/dog/' <&3 > manipulate