The idea is that I want to read any .txt file in a specific folder and do something. So I tried this code:
#!/bin/bash
#Read the file line by line
while read line
do
if [ $i -ne 0 ]; then
#do something...
fi
done < "*.txt"
echo "Finished!"
I think you got my idea now. Thanks for any advice.
After doing some stuff, I want to move the file to another folder.
Not sure what $i is in your if statement.. but you can read all the .txt files in a dir line by line like this:
while read line; do
# your code here, eg
echo "$line"
done < <(cat *.txt)
For a "specific directory" (ie not the directory you are currently in):
DIR=/example/dir
while read line; do
# your code here, eg
echo "$line"
done < <(cat "$DIR"/*.txt)
To avoid using cat unnecessarily, you could use a for loop:
for file in *.txt
do
while read line
do
# whatever
mv -i "$file" /some/other/place
done < "$file"
done
This treats each file separately so you can perform actions on each one individually. If you wanted to move all the files to the same place, you could do that outside the loop:
for file in *.txt
do
while read line
do
# whatever
done < "$file"
done
mv -i *.txt /some/other/place
As suggested in the comments, I have added the -i switch to mv, which prompts before overwriting files. This is probably a good idea, especially when you are expanding a * wildcard. If you would rather not be prompted, you could instead use the -n switch which will not overwrite any files.
Related
I have a template script with some analysis and the only thing that I need to change in it is a case.
#!/bin/bash
CASE=XXX
... the rest of the script where I use $CASE
I created a list of all my cases, that I saved into file: list.txt.
So my list.txt file may contain cases as XXX, YYY, ZZZ.
Now I would run a loop over list.txt content and fill my template_script.sh with a case from the list.txt and then saved the file with a new name - script_CASE.sh
for case in `cat ./list.txt`;
do
# open template_script.sh
# use somehow the line from template_script.sh (maybe substitute CASE=$case)
# save template_script with a new name script_$case
done
In pure bash :
#!/bin/bash
while IFS= read -r casevalue; do
escaped=${casevalue//\'/\'\\\'\'} # escape single quotes if any
while IFS= read -r line; do
if [[ $line = CASE=* ]]; then
echo "CASE='$escaped'"
else
echo "$line"
fi
done < template_script.sh > "script_$casevalue"
done < list.txt
Note that saving to "script_$casevalue" may not work if the case contains a / character.
If it is guaranteed that case values (lines in list.txt) needn't to be escaped then using sed is simpler:
while IFS= read -r casevalue; do
sed -E "s/^CASE=(.*)/CASE=$casevalue/" template_script.sh > "script_$casevalue"
done < list.txt
But this approach is fragile and will fail, for instance, if a case value contains a & character. The pure bash version, I believe, is very robust.
Converting my comment to answer so that solution is easy to find for future visitors.
You may use this bash script:
while read -r c; do
sed "s/^CASE=.*/CASE=$c/" template_script.sh > "script_${c}.sh"
done < list.txt
I am working in a directory with file names ending with fastq.gz. with using a loop like the following, I will be running a tool.
for i inls; do if [[ "$i" == *".gz" ]]; then bwa aln ../hg38.fa $i > $i | sed 's/fastq.gz/sai/g'; fi; done
My question is, I want my output filename to end with .sai instead of fastq.gz with keeping the rest of the filename the same. yet, as it first sees $i after >, it modifies the input file itself. I tried using it like <($i | sed 's/fastq.gz/sai/g') but that does not work either. what is the right way of writing this?
You can use string replacements to compute the filename and the extension.
Moreover, you shouldn't rely on the ls output but loop directly over the expression you are looking for.
for file in *.gz; do
name="${file%.*}"
file_output="${name}.sai"
bwa aln ../hg38.fa ${file} > ${file_output}
done
I have got a directory with files in which some of then end with an underscore.
I would like to test each file to see if it ends with an underscore and then strip off the underscore.
I am currently running the following code:
for file in *;do
echo $file;
if [[ "${file:$length:1}" == "_" ]];then
mv $file $(echo $file | sed "s/.$//g");
fi
done
But it does not seem to be renaming the files with underscore. For example if i have a file called all_indoors_ I expect it to give me all_indoors.
You could use built-in string substitution:
for file in *_; do
mv "$file" "${file%_}"
done
Just use a regex to check the string:
for file in *
do
[[ $file =~ "_$" ]] && echo mv "$file" "${file%%_}"
done
Once you are sure it works as intended, remove the echo so that the mv command executes!
It may even be cleaner to use *_ so that the for will just loop over the files with a name ending with _, as hek2mgl suggests in comments.
for file in *_
do
echo mv "$file" "${file%%_}"
done
You can use which will be recursive:
while read f; do
mv "$f" "${f:0:-1}"; # Remove last character from $f
done < <(find . -type f -name '*_')
Although not a pure bash approach, you can use rename.ul (written by Larry Wall, the person behind perl). Rename is not part of the default linux environment, but is part of util-linux.
You use rename with:
rename perlexpr files
(some flags ommitted).
So you could use:
rename 's/_$//' *
if you want to remove all characters including and after the underscore.
As #hek2mgl points out, there are multiple rename commands (see here), so first test if you have picked the right one.
I am trying to run a simple bash script but I am struggling on how to incoperate a condition. any pointers. the loop says. I would like to incoperate a conditions such that when gdalinfo cannot open the image it copies that particular file to another location.
for file in `cat path.txt`; do gdalinfo $file;done
works fine in opening the images and also shows which ones cannot be opened.
the wrong code is
for file in `cat path.txt`; do gdalinfo $file && echo $file; else cp $file /data/temp
Again, and again and again - zilion th again...
Don't use contsructions like
for file in `cat path.txt`
or
for file in `find .....`
for file in `any command what produces filenames`
Because the code will BREAK immediatelly, when the filename or path contains space. Never use it for any command what produces filenames. Bad practice. Very Bad. It is incorrect, mistaken, erroneous, inaccurate, inexact, imprecise, faulty, WRONG.
The correct form is:
for file in some/* #if want/can use filenames directly from the filesystem
or
find . -print0 | while IFS= read -r -d '' file
or (if you sure than no filename contains a newline) can use
cat path.txt | while read -r file
but here the cat is useless, (really - command what only copies a file to STDOUT is useless). You should use instead
while read -r file
do
#whatever
done < path.txt
It is faster (doesn't fork a new process, as do in case of every pipe).
The above whiles will fill the corect filename into the variable file in cases when the filename contains a space too. The for will not. Period. Uff. Omg.
And use "$variable_with_filename" instead of pure $variable_with_filename for the same reason. If the filename contains a white-space any command will misunderstand it as two filenames. This probably not, what you want too..
So, enclose any shell variable what contain a filename with double quotes. (not only filename, but anything what can contain a space). "$variable" is correct.
If i understand right, you want copy files to /data/temp when the gdalinfo returns error.
while read -r file
do
gdalinfo "$file" || cp "$file" /data/temp
done < path.txt
Nice, short and safe (at least if your path.txt really contains one filename per line).
And maybe, you want use your script more times, therefore dont out the filename inside, but save the script in a form
while read -r file
do
gdalinfo "$file" || cp "$file" /data/temp
done
and use it like:
mygdalinfo < path.txt
more universal...
and maybe, you want only show the filenames for what gdalinfo returns error
while read -r file
do
gdalinfo "$file" || printf "$file\n"
done
and if you change the printf "$file\n" to printf "$file\0" you can use the script in a pipe safely, so:
while read -r file
do
gdalinfo "$file" || printf "$file\0"
done
and use it for example as:
mygdalinfo < path.txt | xargs -0 -J% mv % /tmp/somewhere
Howgh.
You can say:
for file in `cat path.txt`; do gdalinfo $file || cp $file /data/temp; done
This would copy the file to /data/temp if gdalinfo cannot open the image.
If you want to print the filename in addition to copying it in case of failure, say:
for file in `cat path.txt`; do gdalinfo $file || (echo $file && cp $file /data/temp); done
I need to extract data from files in directory /tmp/log.
I have no problem extract from single file.
#!/bin/bash
while read line;
do
echo $line
done < /tmp/log/file1
I want try it with multiple files /tmp/log/* but it returned error ambiguous redirect.
Any idea how can I around it?
You could read the files in a for loop as follows:
for file in /tmp/log/*; do
while read -r line; do
echo "$line"
done < "$file"
done
The strategy is just wrap your while loop with a for loop that takes care of processing each of the files one at a time.
Dont know exactly waht you need..
probably you are looking for this:
cat /tmp/log/*
Is this what you need?
for line in `cat /tmp/log/*`
do
echo $line
done