Why am I getting this file error and a cat error even though I am passing the text file in the argument? - bash

The program calls for us to read in the directory full of text files, parse data from those files into their respective attributes.
Then once the data is set, load a general template which has those attributes in the text.
I'm using a sed command to replace the specific attributes, only if the number of students is greater than 50. If so it runs the sed command and writes to a file, and into a directory.
But i am getting this error when I'm passing
test3.sh ./data assign4.template 12/16/2021 ./output
Error
cat: assign4.template: No such file or directory
test3.sh: line 62: output/MAT3103.crs: No such file or directory
The current file is MAT4353.crs
Now what I am thinking is that, for the file or directory error, it is looking in that folder and searching for a file named that
But Not entirely sure how to resolve that.
As for the cat: template error, I don't get that since I am passing the template in the terminal
As for the other paramaters being passed, the Date which is also substituted in the sed command, All output files should be written to the directory defined by the last argument. This directory may or may not already exist. Each file should be named by the course’s department code and number,and with the extension.warn
Here is the total code
#!/bin/bash
# checking if user has passed atleast four arguments are passed
if [ $# -ne 4 ]
then
echo "Atleast 4 argument should be passed"
exit 1
fi
# if output directory exits check
if [ -d output ]
then
# if output directory exists will get deleted
echo "output directory already exists. So removing its contents"
rm -f output/*
else
# output directory does not exist, so gets created here
echo "output directory does not exist. So creating a new directory"
mkdir output
fi
max_students=50
template=$2
dt=$3
cd $1
for i in *; do
echo The current file is ${i}
dept_code=$(awk 'NR==2
{print $1 ; exit}' $i)
echo $dept_code
dept_name=$(awk 'NR==2
{print $2 ; exit}' $i)
echo $dept_name
course_name=$(awk 'FNR==2' $i)
echo $course_name
course_sched=$(awk 'FNR==3' $i | awk '{print $1}')
course_sched=$(awk 'FNR==3' $i | awk '{print $1}')
echo $course_sched
course_start=$(awk 'FNR==3' $i | awk '{print $2}')
echo $course_start
course_end=$(awk 'FNR==3' $i | awk '{print $3}')
echo $course_end
credit_hours=$(awk 'FNR==4' $i)
echo $credit_hours
num_students=$(awk 'FNR==5' $i)
echo $num_students
# checking if number of students currently enrolled > max students
if (( $(echo "$num_students > $max_students" |bc -l) ))
then
# output filename creation
out_file=${i}
# using example Template and sed command to replace the variables
cat $template | sed -e "s/\[\\[\dept_code\]\]/$dept_code/" | sed -e "s/\[\\[\dept_name\]\]/$dept_name/" | sed -e "s|\[\[course_name\]\]|$course_name|" | sed -e "s|\[\[course_start\]\]|$$
fi
done

You define the variable as
template=$2
and since your second parameter is assign4.template, this is what the variable template is set to. Then you do a
cat $template
which is, first of all, unnecessary, since you can do an input redirection on sed instead, but most of all requires, that the file exists in your working directory. Since you have done before a
cd $1
it means that the file data/assign4.template does not exist. You have to create this file before you can use your script.

use single quotes in your positional arguments.
test3.sh './data' 'assign4.template' '12/16/2021' './output'
or
test3.sh data assign4.template '12/16/2021' output

Related

In a bash pipe, take the output of the previous command as a variable to the next command (Eg. if statement)

I wanted to write a command to compare the hash of a file. I wrote the below single line command. Wanted to understand as to how I can take the output of the previous command as a variable for the current command, in a pipe.
Eg. below command I wanted to compare the output of 1st command "Calculated hash" to the original hash. In the last command, I wanted to refer to the output of the previous command. How do I do that in the if statement? (Instead of $0)
sha256sum abc.txt | awk '{print $1}' | if [ "$0" = "8237491082roieuwr0r9812734iur" ]; then
echo "match"
fi
Following your narrow request looks like:
sha256sum abc.txt |
awk '{print $1}' |
if [ "$(cat)" = "8237491082roieuwr0r9812734iur" ]; then echo "match"; fi
...as cat with no arguments reads the command's stdin, and in a pipeline, content generated from prior stages are streamed into their successors.
Alternately:
sha256sum abc.txt |
awk '{print $1}' |
if read -r line && [ "$line" = "8237491082roieuwr0r9812734iur" ]; then echo "match"; fi
...wherein we read only a single line from stdin instead of using cat. (To instead loop over all lines given on stdin, see BashFAQ #1).
However, I would strongly suggest writing this instead as:
if [ "$(sha256sum abc.txt | awk '{print $1}')" = "8237491082roieuwr0r9812734iur" ]; then
echo "match"
fi
...which, among other things, keeps your logic outside the pipeline, so your if statement can set variables that remain set after the pipeline exits. See BashFAQ #24 for more details on the problems inherent in running code in pipelines.
Consider using sha256sum's check mode. If you save the output of sha256sum to a file, you can check it with sha256sum -c.
$ echo foo > file
$ sha256sum file > hash.txt
$ cat hash.txt
b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c file
$ sha256sum -c hash.txt
file: OK
$ if sha256sum -c --quiet hash.txt; then echo "match"; fi
If you don't want to save the hashes to a file you could pass them in via a here-string:
if sha256sum -c --quiet <<< 'b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c file'; then
echo "match"
fi

How to use variable with awk when being read from a file

I have a file with the following entries:
foop07_bar2_20190423152612.zip
foop07_bar1_20190423153115.zip
foop08_bar2_20190423152612.zip
foop08_bar1_20190423153115.zip
where
foop0* = host
bar* = fp
I would like to read the file and create 3 variables, the whole file name, host and fp (which stands for file_path_differentiator).
I am using read to take the first line and get my whole file name variable, I though I could then feed this into awk to grab the next two variables, however the first method of variable insertion creates an error and the second gives me all the variables.
I would like to loop each line, as I wish to use these variables to ssh to the host and grab the file
#!/bin/bash
while read -r FILE
do
echo ${FILE}
host=`awk 'BEGIN { FS = "_" } ; { print $1 }'<<<<"$FILE"`
echo ${host}
path=`awk -v var="${FILE}" 'BEGIN { FS = "_" } ; { print $2 }'`
echo ${path}
done <zips_not_received.csv
Expected Result
foop07_bar2_20190423152612.zip
foop07
bar2
foop07_bar1_20190423153115.zip
foop07
bar1
Actual Result
foop07_bar2_20190423152612.zip
/ : No such file or directoryfoop07_bar2_20190423152612.zip
bar2 bar1 bar2 bar1
You can do this alone with bash, without using any external tool.
while read -r file; do
[[ $file =~ (.*)_(.*)_.*\.zip ]] || { echo "invalid file name"; exit 1; }
host="${BASH_REMATCH[1]}"
path="${BASH_REMATCH[2]}"
echo "$file"
echo "$host"
echo "$path"
done < zips_not_received.csv
typical...
Managed to work a solution after posting...
#!/bin/bash
while read -r FILE
do
echo ${FILE}
host=`echo "$FILE" | awk -F"_" '{print $1}'`
echo $host
path=`echo "$FILE" | awk -F"_" '{print $2}'`
echo ${path}
done <zips_not_received.csv
not sure on the elegance or its correctness as i am using echo to create variable...but i have it working..
Assuming there is no space or _ in your "file name" that are part of the host or path
just separate line before with sed, awk, ... if using default space separator (or use _ as argument separator in batch). I add the remove of empty line value as basic security seeing your sample.
sed 's/_/ /g;/[[:blank:]]\{1,\}/d' zips_not_received.csv \
| while read host path Ignored
do
echo "${host}"
echo "${path}"
done

Sed replace substring only if expression exist

In a bash script, I am trying to remove the directory name in filenames :
documents/file.txt
direc/file5.txt
file2.txt
file3.txt
So I try to first see if there is a "/" and if yes delete everything before :
for i in **/*.scss *.scss; do
echo "$i" | sed -n '^/.*\// s/^.*\///p'
done
But it doesn't work for files in the current directory, it gives me a blank string.
I get :
file.txt
file5.txt
When you only want the filename, use basename instead of sed.
# basename /path/to/file
returns file
here is the man page
Your sed attempt is basically fine, but you should print regardless of whether you performed a substitution; take out the -n and the p at the end. (Also there was an unrelated syntax error.)
Also, don't needlessly loop over all files.
printf '%s\n' **/*.scss *.scss |
sed -n 's%^.*/%%p'
This also can be done with awk bash util.
Example:
echo "1/2/i.py" | awk 'BEGIN {FS="/"} {print $NF}'
output: i.py
Eventually, I did :
for i in **/*.scss *.scss; do
# for i in *.scss; do
# for i in _hm-globals.scss; do
name=${i##*/} # remove dir name
name=${name%.scss} # remove extension
name=`echo "$name" | sed -n "s/^_hm-//p"` # remove _hm-
if [[ $name = *"."* ]]; then
name=`echo "$name" | sed -n 's/\./-/p'` #replace . to --
fi
echo "$name" >&2
done

How to get unique string from variable in bash

Hello I would like to ask how can I get unique string from variable.
while read -r line
do
route=$(echo $line | awk -F'[:]' '{print $2}') #get path from log file
if [ "`dirname "$route"`" == "`xrealpath "$PWD"`" ]; then #compare path from log file with $PWD
name=$(echo $line | awk -F'[:]' '{print $1}') #take name from 1st column in log file
fi
if ! [ "$name" == "$help_name" ]; then
echo $name
help_name=$name
pom=$pom:$name
fi
done < $WEDI_RC
Sample logfile:
proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:1
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:1
proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:2
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:2
proj.sh:/Users/Tom/Documents/proj.sh:2015-03-21:3
proj1.sh:/Users/Tom/Documents/proj.sh:2015-03-21:3
How can I echo each unique just one time?
My output now looks something like this:
proj.sh
proj1.sh
proj.sh
proj1.sh
proj.sh
:proj.sh:proj1.sh:proj.sh:proj1.sh:proj.
Expecting output:
proj.sh
proj1.sh
I don't know how much files can be readed in while cycle.
We cannot use any temporary files
Thank you
Answer for Original Version of This Question
This uses the associative array seen to keep track of what names have been seen:
declare -A seen
while read -r line
... blabla ...
do
if [ -z "${seen[$name]}" ]; then
echo $name
seen["$name"]=1
pom=$pom:$name
fi
done < "$WEDI_RC"
Working Example (No blabla)
Let us start with this file:
$ cat file
proj.sh
proj1.sh
proj.sh
proj1.sh
proj.sh
We will run this code (note that ...blabla... has been removed and the loop now reads in name directly):
$ cat script.sh
declare -A seen
while read -r name
do
if [ -z "${seen[$name]}" ]; then
echo $name
seen["$name"]=1
pom=$pom:$name
fi
done < file
declare -p pom
This is the result:
$ bash script.sh
proj.sh
proj1.sh
declare -- pom=":proj.sh:proj1.sh"
Answer for Revised Question
In the revised question, the following code appears:
route=$(echo $line | awk -F'[:]' '{print $2}') #get path from log file
if [ "`dirname "$route"`" == "`xrealpath "$PWD"`" ]; then #compare path from log file with $PWD
name=$(echo $line | awk -F'[:]' '{print $1}') #take name from 1st column in log file
This means that, as the code runs, name may never be set depending on the current directory when the script is run. This would explain the error messages reported in the comments.

Bash script read specifc value from files of an entire folder

I have a problem creating a script that reads specific value from all the files of an entire folder
I have a number of email files in a directory and I need to extract from each file, 2 specific values.
After that I have to put them into a new file that looks like that:
--------------
To: value1
value2
--------------
This is what I want to do, but I don't know how to create the script:
# I am putting the name of the files into a temp file
`ls -l | awk '{print $9 }' >tmpfile`
# use for the name of a file
`date=`date +"%T"
# The first specific value from file (phone number)
var1=`cat tmpfile | grep "To: 0" | awk '{print $2 }' | cut -b -10 `
# The second specific value from file(subject)
var2=cat file | grep Subject | awk '{print $2$3$4$5$6$7$8$9$10 }'
# Put the first value in a new file on the first row
echo "To: 4"$var1"" > sms-$date
# Put the second value in the same file on the second row
echo ""$var2"" >>sms-$date
.......
and do the same for every file in the directory
I tried using while and for functions but I couldn't finalize the script
Thank You
I've made a few changes to your script, hopefully they will be useful to you:
#!/bin/bash
for file in *; do
var1=$(awk '/To: 0/ {print substr($2,0,10)}' "$file")
var2=$(awk '/Subject/ {for (i=2; i<=10; ++i) s=s$i; print s}' "$file")
outfile="sms-"$(date +"%T")
i=0
while [ -f "$outfile" ]; do outfile="sms-$date-"$((i++)); done
echo "To: 4$var1" > "$outfile"
echo "$var2" >> "$outfile"
done
The for loop just goes through every file in the folder that you run the script from.
I have added added an additional suffix $i to the end of the file name. If no file with the same date already exists, then the file will be created without the suffix. Otherwise the value of $i will keep increasing until there is no file with the same name.
I'm using $( ) rather than backticks, this is just a personal preference but it can be clearer in my opinion, especially when there are other quotes about.
There's not usually any need to pipe the output of grep to awk. You can do the search in awk using the / / syntax.
I have removed the cut -b -10 and replaced it with substr($2, 0, 10), which prints the first 10 characters from column 2.
It's not much shorter but I used a loop rather than the $2$3..., I think it looks a bit neater.
There's no need for all the extra " in the two output lines.
I sugest to try the following:
#!/bin/sh
RESULT_FILE=sms-`date +"%T"`
DIR=.
fgrep -l 'To: 0' "$DIR" | while read FILE; do
var1=`fgrep 'To: 0' "$FILE" | awk '{print $2 }' | cut -b -10`
var2=`fgrep 'Subject' "$FILE" | awk '{print $2$3$4$5$6$7$8$9$10 }'`
echo "To: 4$var1" >>"$RESULT_FIL"
echo "$var2" >>"$RESULT_FIL"
done

Resources