I need to extract data from files in directory /tmp/log.
I have no problem extract from single file.
#!/bin/bash
while read line;
do
echo $line
done < /tmp/log/file1
I want try it with multiple files /tmp/log/* but it returned error ambiguous redirect.
Any idea how can I around it?
You could read the files in a for loop as follows:
for file in /tmp/log/*; do
while read -r line; do
echo "$line"
done < "$file"
done
The strategy is just wrap your while loop with a for loop that takes care of processing each of the files one at a time.
Dont know exactly waht you need..
probably you are looking for this:
cat /tmp/log/*
Is this what you need?
for line in `cat /tmp/log/*`
do
echo $line
done
Related
I have a template script with some analysis and the only thing that I need to change in it is a case.
#!/bin/bash
CASE=XXX
... the rest of the script where I use $CASE
I created a list of all my cases, that I saved into file: list.txt.
So my list.txt file may contain cases as XXX, YYY, ZZZ.
Now I would run a loop over list.txt content and fill my template_script.sh with a case from the list.txt and then saved the file with a new name - script_CASE.sh
for case in `cat ./list.txt`;
do
# open template_script.sh
# use somehow the line from template_script.sh (maybe substitute CASE=$case)
# save template_script with a new name script_$case
done
In pure bash :
#!/bin/bash
while IFS= read -r casevalue; do
escaped=${casevalue//\'/\'\\\'\'} # escape single quotes if any
while IFS= read -r line; do
if [[ $line = CASE=* ]]; then
echo "CASE='$escaped'"
else
echo "$line"
fi
done < template_script.sh > "script_$casevalue"
done < list.txt
Note that saving to "script_$casevalue" may not work if the case contains a / character.
If it is guaranteed that case values (lines in list.txt) needn't to be escaped then using sed is simpler:
while IFS= read -r casevalue; do
sed -E "s/^CASE=(.*)/CASE=$casevalue/" template_script.sh > "script_$casevalue"
done < list.txt
But this approach is fragile and will fail, for instance, if a case value contains a & character. The pure bash version, I believe, is very robust.
Converting my comment to answer so that solution is easy to find for future visitors.
You may use this bash script:
while read -r c; do
sed "s/^CASE=.*/CASE=$c/" template_script.sh > "script_${c}.sh"
done < list.txt
I am working in a directory with file names ending with fastq.gz. with using a loop like the following, I will be running a tool.
for i inls; do if [[ "$i" == *".gz" ]]; then bwa aln ../hg38.fa $i > $i | sed 's/fastq.gz/sai/g'; fi; done
My question is, I want my output filename to end with .sai instead of fastq.gz with keeping the rest of the filename the same. yet, as it first sees $i after >, it modifies the input file itself. I tried using it like <($i | sed 's/fastq.gz/sai/g') but that does not work either. what is the right way of writing this?
You can use string replacements to compute the filename and the extension.
Moreover, you shouldn't rely on the ls output but loop directly over the expression you are looking for.
for file in *.gz; do
name="${file%.*}"
file_output="${name}.sai"
bwa aln ../hg38.fa ${file} > ${file_output}
done
I have a script that extracts filenames from an input file and is supposed to read each line (filename) and unzip the specified file, saving the unzipped content as individual files. However, I can't get my counter to work and just get all the unzipped files in one large file.
Input file contains a list:
ens/484/59/traj.pdb 0.001353
ens/263/39/traj.pdb 0.004178
ens/400/35/traj.pdb 0.004191
I'm using the regex /.*?/.*?/ to extract the file that I'd like to unzip and name each output{1..40}.pdb -- instead I get one output file: output1.pdb which contains all the contents of the 40 unzipped files.
My question is: how do I correct my counter in order to achieve the desired naming scheme?
#!/bin/bash
file="/home/input.txt"
grep -Po '/.*?/.*?/' $file > filenames.txt
i=$((i+1))
structures='filenames.txt'
while IFS= read line
do
gunzip -c 'ens'$line'traj.pdb.gz' >> 'output'$i'.pdb'
done <"$structures"
rm "$structures"
file="/home/input.txt"
grep -Po '/.*?/.*?/' $file > filenames.txt
structures='filenames.txt'
i=1
while IFS= read "line"
do
gunzip -c 'ens'$line'traj.pdb.gz' >> 'output'$i'.pdb'
i=$(expr $i + 1)
done <$structures
rm $structures
couple of logical mistakes, the counter has to be fined as one out of the while loop and the counter +1 should be inside the loop, also for the counter to work you have to use expr, in this case i made the counter start from 1, so the first entry will get this value. Also on the parameter for the while loop i dont really understand what you are doing, if it works as you have it then cool or else use a test statement after while and before the parameters.
Using a Bash script, I'd like to move a list of files by using a for-loop, not a while-loop (for testing purpose). Can anyone explain to me why mv always acts as file rename rather than file move under this for loop? How can I fix it to move the list of files?
The following works:
for file in "/Volumes/HDD1/001.jpg" "/Volumes/HDD1/002.jpg"
do
mv "$file" "/Volumes/HDD2/"
done
UPDATE#1:
However, suppose that I have a sample_pathname.txt
cat sample_pathname.txt
"/Volumes/HDD1/001.jpg" "/Volumes/HDD1/002.jpg"
Why the following for-loop will not work then?
array=$(cat sample_path2.txt)
for file in "${array[#]}"
do
mv "$file" "/Volumes/HDD2/"
done
Thanks.
System: OS X
Bash version: 3.2.53(1)
cat sample_pathname.txt
"/Volumes/HDD1/001.jpg" "/Volumes/HDD1/002.jpg"
The quotation marks here are the problem. Unless you need to cope with file names with newlines in them, the simple and standard way to do this is to list one file name per line, with no quotes or other metainformation.
vbvntv$ cat sample_pathname_fixed.txt
/Volumes/HDD1/001.jpg
/Volumes/HDD1/002.jpg
vbvntv$ while read -r file; do
> mv "$file" "/Volumes/HDD2/"
> done <sample_pathname_fixed.txt
In fact, you could even
xargs mv -t /Volumes/HDD2 <sample_pathname_fixed.txt
(somewhat depending on how braindead your xargs is).
The syntax used in your example will not create an array... It is just storing the file contents in a variable named array.
IFS=$'\n' array=$(cat sample_path2.txt)
If you have a text file containing filenames (each on separate line would be simplest), you can load it into an array and iterate over it as follows. Note the use of $(< file ) as a better alternative to cat and the parenthesis to initialize the contents into an array. Each line of the file corresponds to an index.
array=($(< file_list.txt ))
for file in "${array[#]}"; do
mv "$file" "/absolute/path"
done
Update: Your IFS was probably not set correctly if the command at the top of the post didn't work. I updated it to reflect that. Also, there are a couple of other reliable ways to initialize an array from a file. But like you mentioned, if you are just piping the file directly into a while loop, you may not need it.
This is a shell builtin in Bash 4+ and a synonym of mapfile. This works great if its available.
readarray -t array < file
The 'read' command can also initialize an array for you:
IFS=$'\n' read -d '' -r -a array < file
use this:
for file in "/Volumes/HDD1/001.jpg" "/Volumes/HDD1/002.jpg"
do
f=$(basename $file)
mv "$file" "/Volumes/HDD2/$f"
done
The idea is that I want to read any .txt file in a specific folder and do something. So I tried this code:
#!/bin/bash
#Read the file line by line
while read line
do
if [ $i -ne 0 ]; then
#do something...
fi
done < "*.txt"
echo "Finished!"
I think you got my idea now. Thanks for any advice.
After doing some stuff, I want to move the file to another folder.
Not sure what $i is in your if statement.. but you can read all the .txt files in a dir line by line like this:
while read line; do
# your code here, eg
echo "$line"
done < <(cat *.txt)
For a "specific directory" (ie not the directory you are currently in):
DIR=/example/dir
while read line; do
# your code here, eg
echo "$line"
done < <(cat "$DIR"/*.txt)
To avoid using cat unnecessarily, you could use a for loop:
for file in *.txt
do
while read line
do
# whatever
mv -i "$file" /some/other/place
done < "$file"
done
This treats each file separately so you can perform actions on each one individually. If you wanted to move all the files to the same place, you could do that outside the loop:
for file in *.txt
do
while read line
do
# whatever
done < "$file"
done
mv -i *.txt /some/other/place
As suggested in the comments, I have added the -i switch to mv, which prompts before overwriting files. This is probably a good idea, especially when you are expanding a * wildcard. If you would rather not be prompted, you could instead use the -n switch which will not overwrite any files.