I have a file called file_names_list.txt which contains absolute file names, for example, the first line is:
~/Projects/project/src/files/file.mm
I run a script to grep each of these files,
for file in $(cat file_names_list.txt); do
echo "doing file: $file"
grep '[ \t]*if (.* = .*) {' $file | while read -r line ; do ...
and I get the output:
doing file: ~/Projects/project/src/files/file.mm
grep: ~/Projects/project/src/files/file.mm: No such file or directory
But if I go to the terminal and enter
grep '[ \t]*if (.* = .*) {' ~/Projects/project/src/files/file.mm
I get the proper grep output
What's the problem here? I'm out of ideas
The problem is with the ~ character. That character gets expanded to your home directory when you use it in bash but in this case, it is just another character stored in the variable $file. To see the difference, try this:
file='~'
echo $file
echo ~
So now you have to either recreate the file file_names_list.txt or try to fix it, e.g. with sed:
sed -i -e "s|^~/|$HOME/|" file_names_list.txt
Also note that it would be prefereable to use a while loop instead of a for loop:
while IFS= read -r file; do
# write your code here
done < file_names_list.txt
You can use your script like this:
while IFS= read -r f; do
grep '[ \t]*if .* = .* {' "${f/#\~/\$HOME}"
done < file_names_list.txt
Since ~ cannot be stored in a variable and expanded we are replacing starting ~ by $HOME in each line in this BASH expression: "${f/#\~/\$HOME}"
Related
I have the following file min.conf as shown below. I need to change the path directory to the current one in the file.
Input file (min.conf)
#!/bin/bash
#
#SBATCH --workdir=/work/LAS/input
myNAMD2=$(which namd2)
myWork=/work/LAS/input
I have the following code :
#!/bin/bash
filename="min.conf"
#While loop to read line by line
while IFS= read -r line; do
#If the line starts with ST then set var to yes.
if [[ $line == '#SBATCH --workdir='* ]] || [[ $line == 'myWork='* ]] ; then
fi
done < "$filename"
Desired output(min.conf) :
#!/bin/bash
#
#SBATCH --workdir=/work/input23
myNAMD2=$(which namd2)
myWork=/work/input23
Could you please try following. Point to be noted here PWD is a shell variable and then passing it to awk by assigning it to an awk variable.
PWD=`pwd`
awk -v pwd="$PWD" '
/mywork/{
sub(/=.*/,"=" pwd)
}
/SBATCH/{
sub(/workdir=.*/,"workdir=" pwd)
}
1
' Input_file
In case you want to save output into Input_file itself append > temp && mv temp Input_file
sed could do somewhat you want:
sed -E '/^(a|b)/s:c:d:g'
/^(a|b)/: for all lines starting with a or b.
s:c:d:g : replace all c with d.
So, as for changing the path to the current one:
sed -E '/^(#SBATCH --workdir=|myWork=)/s:=\/.*\/:='$PWD'\/:'
/^(#SBATCH --workdir=|myWork=)/: for all lines starting with #SB..
s:=\/.*\/:=\/'$PWD'\/:: replace dir name with current working dir
$PWD: in bash, it is your current working dir; $(pwd) is also the same
Finally, if you want sed edit your min.conf file:
sed -i~ -E '/^(#SBATCH --workdir=|myWork=)/s:=\/.*\/:='$PWD'\/:' min.conf
sed -i~: edit the file, and make backup
In a bash script, I am trying to remove the directory name in filenames :
documents/file.txt
direc/file5.txt
file2.txt
file3.txt
So I try to first see if there is a "/" and if yes delete everything before :
for i in **/*.scss *.scss; do
echo "$i" | sed -n '^/.*\// s/^.*\///p'
done
But it doesn't work for files in the current directory, it gives me a blank string.
I get :
file.txt
file5.txt
When you only want the filename, use basename instead of sed.
# basename /path/to/file
returns file
here is the man page
Your sed attempt is basically fine, but you should print regardless of whether you performed a substitution; take out the -n and the p at the end. (Also there was an unrelated syntax error.)
Also, don't needlessly loop over all files.
printf '%s\n' **/*.scss *.scss |
sed -n 's%^.*/%%p'
This also can be done with awk bash util.
Example:
echo "1/2/i.py" | awk 'BEGIN {FS="/"} {print $NF}'
output: i.py
Eventually, I did :
for i in **/*.scss *.scss; do
# for i in *.scss; do
# for i in _hm-globals.scss; do
name=${i##*/} # remove dir name
name=${name%.scss} # remove extension
name=`echo "$name" | sed -n "s/^_hm-//p"` # remove _hm-
if [[ $name = *"."* ]]; then
name=`echo "$name" | sed -n 's/\./-/p'` #replace . to --
fi
echo "$name" >&2
done
I have been working in bash, and need to create a string argument. bash is a newish for me, to the point that I dont know how to build a string in bash from a list.
// foo.txt is a list of abs file names.
/foo/bar/a.txt
/foo/bar/b.txt
/delta/test/b.txt
should turn into: a.txt,b.txt,b.txt
OR: /foo/bar/a.txt,/foo/bar/b.txt,/delta/test/b.txt
code
s = ""
for file in $(cat foo.txt);
do
#what goes here? s += $file ?
done
myShellScript --script $s
I figure there was an easy way to do this.
with for loop:
for file in $(cat foo.txt);do echo -n "$file",;done|sed 's/,$/\n/g'
with tr:
cat foo.txt|tr '\n' ','|sed 's/,$/\n/g'
only sed:
sed ':a;N;$!ba;s/\n/,/g' foo.txt
This seems to work:
#!/bin/bash
input="foo.txt"
while IFS= read -r var
do
basename $var >> tmp
done < "$input"
paste -d, -s tmp > result.txt
output: a.txt,b.txt,b.txt
basename gets you the file names you need and paste will put them in the order you seem to need.
The input field separator can be used with set to create split/join functionality:
# split the lines of foo.txt into positional parameters
IFS=$'\n'
set $(< foo.txt)
# join with commas
IFS=,
echo "$*"
For just the file names, add some sed:
IFS=$'\n'; set $(sed 's|.*/||' foo.txt); IFS=,; echo "$*"
I have piped the output of ls command into a file. The contents are like so:
[Chihiro]_Grisaia_no_Kajitsu_-_01_[1920x816_Blu-ray_FLAC][D2B961D6].mkv
[Chihiro]_Grisaia_no_Kajitsu_-_02_[1920x816_Blu-ray_FLAC][38F88A81].mkv
[Chihiro]_Grisaia_no_Kajitsu_-_03_[1920x816_Blu-ray_FLAC][410F74F7].mkv
My attempt to rename these episodes according to episode number is as follows:
cat grisaia | while read line;
#get the episode number
do EP=$(echo $line | egrep -o "_([0-9]{2})_" | cut -d "_" -f2)
if [[ $EP ]]
#escape special characters
then line=$(echo $line | sed 's/\[/\\[/g' | sed 's/\]/\\]/g')
mv "$line" "Grisaia_no_Kajitsu_${EP}.mkv"
fi
done
The mv commands exit with code 1 with the following error:
mv: cannot stat
'\[Chihiro\]_Grisaia_no_Kajitsu_-01\[1920x816_Blu-ray_FLAC\]\[D2B961D6\].mkv':
No such file or directory
What I really don't get is that if I copy the file that could not be stat and attempt to stat the file, it works. I can even take the exact same string that is output and execute the mv command individually.
If you surround your variable ($line) with double quotes (") you don't need to escape those special characters. So you have two options there:
Remove the following assignation completely:
then # line=$(echo $line | sed 's/\[/\\[/g' | sed 's/\]/\\]/g')`
or
Remove the double quotes in the following line:
mv $line "Grisaia_no_Kajitsu_${EP}.mkv"
Further considerations
Parsing the output of ls is never a good idea. Think about filenames with spaces. See this document for more information.
The cat here is unnecessary:
cat grisaia | while read line;
...
done
Use this instead to avoid an unnecessary pipe:
while read line;
...
done < grisaia
Why is good to avoid pipes in some scenarios? (answering comment)
Pipes create subshells (which are expensive), and you can also make some mistakes as the following:
last=""
cat grisaia | while read line; do
last=$line
done
echo $last # surprise!! it outputs an empty string
The reason is that $last inside the loop belongs to another subshell.
Now, see the same approach wothout pipes:
while read line; do
last=$line
done < grisaia
echo $last # it works as expected and prints the last line
I would like to remove a file name only from the following configuration file.
Configuration File -- test.conf
knowledgebase/arun/test.rf
knowledgebase/arunraj/tester/test.drl
knowledgebase/arunraj2/arun/test/tester.drl
The above file should be read. And removed contents should went to another file called output.txt
Following are my try. It is not working to me at all. I am getting empty files only.
#!/bin/bash
file=test.conf
while IFS= read -r line
do
# grep --exclude=*.drl line
# awk 'BEGIN {getline line ; gsub("*.drl","", line) ; print line}'
# awk '{ gsub("/",".drl",$NF); print line }' arun.conf
# awk 'NF{NF--};1' line arun.conf
echo $line | rev | cut -d'/' -f 1 | rev >> output.txt
done < "$file"
Expected Output :
knowledgebase/arun
knowledgebase/arunraj/tester
knowledgebase/arunraj2/arun/test
There's the dirname command to make it easy and reliable:
#!/bin/bash
file=test.conf
while IFS= read -r line
do
dirname "$line"
done < "$file" > output.txt
There are Bash shell parameter expansions that will work OK with the list of names given but won't work reliably for some names:
file=test.conf
while IFS= read -r line
do
echo "${line%/*}"
done < "$file" > output.txt
There's sed to do the job — easily with the given set of names:
sed 's%/[^/]*$%%' test.conf > output.txt
It's harder if you have to deal with names like /plain.file (or plain.file — the same sorts of edge cases that trip up the shell expansion).
You could add Perl, Python, Awk variants to the list of ways of doing the job.
You can get the path like this:
path=${fullpath%/*}
It cuts away the string after the last /
Using awk one liner you can do this:
awk 'BEGIN{FS=OFS="/"} {NF--} 1' test.conf
Output:
knowledgebase/arun
knowledgebase/arunraj/tester
knowledgebase/arunraj2/arun/test