I wanted to write a command to compare the hash of a file. I wrote the below single line command. Wanted to understand as to how I can take the output of the previous command as a variable for the current command, in a pipe.
Eg. below command I wanted to compare the output of 1st command "Calculated hash" to the original hash. In the last command, I wanted to refer to the output of the previous command. How do I do that in the if statement? (Instead of $0)
sha256sum abc.txt | awk '{print $1}' | if [ "$0" = "8237491082roieuwr0r9812734iur" ]; then
echo "match"
fi
Following your narrow request looks like:
sha256sum abc.txt |
awk '{print $1}' |
if [ "$(cat)" = "8237491082roieuwr0r9812734iur" ]; then echo "match"; fi
...as cat with no arguments reads the command's stdin, and in a pipeline, content generated from prior stages are streamed into their successors.
Alternately:
sha256sum abc.txt |
awk '{print $1}' |
if read -r line && [ "$line" = "8237491082roieuwr0r9812734iur" ]; then echo "match"; fi
...wherein we read only a single line from stdin instead of using cat. (To instead loop over all lines given on stdin, see BashFAQ #1).
However, I would strongly suggest writing this instead as:
if [ "$(sha256sum abc.txt | awk '{print $1}')" = "8237491082roieuwr0r9812734iur" ]; then
echo "match"
fi
...which, among other things, keeps your logic outside the pipeline, so your if statement can set variables that remain set after the pipeline exits. See BashFAQ #24 for more details on the problems inherent in running code in pipelines.
Consider using sha256sum's check mode. If you save the output of sha256sum to a file, you can check it with sha256sum -c.
$ echo foo > file
$ sha256sum file > hash.txt
$ cat hash.txt
b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c file
$ sha256sum -c hash.txt
file: OK
$ if sha256sum -c --quiet hash.txt; then echo "match"; fi
If you don't want to save the hashes to a file you could pass them in via a here-string:
if sha256sum -c --quiet <<< 'b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c file'; then
echo "match"
fi
Related
The program calls for us to read in the directory full of text files, parse data from those files into their respective attributes.
Then once the data is set, load a general template which has those attributes in the text.
I'm using a sed command to replace the specific attributes, only if the number of students is greater than 50. If so it runs the sed command and writes to a file, and into a directory.
But i am getting this error when I'm passing
test3.sh ./data assign4.template 12/16/2021 ./output
Error
cat: assign4.template: No such file or directory
test3.sh: line 62: output/MAT3103.crs: No such file or directory
The current file is MAT4353.crs
Now what I am thinking is that, for the file or directory error, it is looking in that folder and searching for a file named that
But Not entirely sure how to resolve that.
As for the cat: template error, I don't get that since I am passing the template in the terminal
As for the other paramaters being passed, the Date which is also substituted in the sed command, All output files should be written to the directory defined by the last argument. This directory may or may not already exist. Each file should be named by the course’s department code and number,and with the extension.warn
Here is the total code
#!/bin/bash
# checking if user has passed atleast four arguments are passed
if [ $# -ne 4 ]
then
echo "Atleast 4 argument should be passed"
exit 1
fi
# if output directory exits check
if [ -d output ]
then
# if output directory exists will get deleted
echo "output directory already exists. So removing its contents"
rm -f output/*
else
# output directory does not exist, so gets created here
echo "output directory does not exist. So creating a new directory"
mkdir output
fi
max_students=50
template=$2
dt=$3
cd $1
for i in *; do
echo The current file is ${i}
dept_code=$(awk 'NR==2
{print $1 ; exit}' $i)
echo $dept_code
dept_name=$(awk 'NR==2
{print $2 ; exit}' $i)
echo $dept_name
course_name=$(awk 'FNR==2' $i)
echo $course_name
course_sched=$(awk 'FNR==3' $i | awk '{print $1}')
course_sched=$(awk 'FNR==3' $i | awk '{print $1}')
echo $course_sched
course_start=$(awk 'FNR==3' $i | awk '{print $2}')
echo $course_start
course_end=$(awk 'FNR==3' $i | awk '{print $3}')
echo $course_end
credit_hours=$(awk 'FNR==4' $i)
echo $credit_hours
num_students=$(awk 'FNR==5' $i)
echo $num_students
# checking if number of students currently enrolled > max students
if (( $(echo "$num_students > $max_students" |bc -l) ))
then
# output filename creation
out_file=${i}
# using example Template and sed command to replace the variables
cat $template | sed -e "s/\[\\[\dept_code\]\]/$dept_code/" | sed -e "s/\[\\[\dept_name\]\]/$dept_name/" | sed -e "s|\[\[course_name\]\]|$course_name|" | sed -e "s|\[\[course_start\]\]|$$
fi
done
You define the variable as
template=$2
and since your second parameter is assign4.template, this is what the variable template is set to. Then you do a
cat $template
which is, first of all, unnecessary, since you can do an input redirection on sed instead, but most of all requires, that the file exists in your working directory. Since you have done before a
cd $1
it means that the file data/assign4.template does not exist. You have to create this file before you can use your script.
use single quotes in your positional arguments.
test3.sh './data' 'assign4.template' '12/16/2021' './output'
or
test3.sh data assign4.template '12/16/2021' output
I need to read a file that has lines like
user=username1
pass=password1
How can I read multiple lines like this into separate variables like username and password?
Would I use awk or grep? I have found ways to read lines into variables with grep but would I need to read the file for each individual item?
The end result is to use these variables to access a database via the command line. So I need to be able to read, store and use these values in other commands.
if the process which generates the file is safe and has shell syntax just source the file.
. ./file
Otherwise the file can be processes before to add quotes
perl -ne 'if (/^([A-Za-z_]\w*)=(.*)/) {$k=$1;$v=$2;$v=~s/\x27/\x27\\\x27\x27/g;print "$k=\x27$v\x27\n";}' <file >file2
. ./file2
If you want to use awk then
Input
$ cat file
user=username1
pass=password1
Reading
$ user=$(awk -F= '$1=="user"{print $2;exit}' file)
$ pass=$(awk -F= '$1=="pass"{print $2;exit}' file)
Output
$ echo $user
username1
$ echo $pass
password1
You could use a loop for your file perhaps, but this is probably the functionality you're looking for.
$ echo 'user=username1' | awk -F= '{print $2}'
username1
Using the -F flag sets the delimiter to = and we select the 2nd item from the row.
file.txt:
user=username1
pass=password1
user=username2
pass=password2
user=username3
pass=password3
Do to avoid browsing several times the file file.txt:
#!/usr/bin/env bash
func () {
echo "user:$1 pass:$2"
}
i=0
while IFS='' read -r line; do
if [ $i -eq 0 ]; then
i=1
user=$(echo ${line} | cut -f2 -d'=')
else
i=0
pass=$(echo ${line} | cut -f2 -d'=')
func "$user" "$pass"
fi
done < file.txt
Output:
user:username1 pass:password1
user:username2 pass:password2
user:username3 pass:password3
How can I validate that this replace command succeeded:
perl -pi -e 's/contoso/'"$hostname"'/g' /etc/inet/hosts
I have tried checking the return value:
if [ $? -eq 0 ]; then
echo "OK"
else
echo "Error."
fi
But the return value is not being set when the command fails.
Thanks
One option is to check, if file has been modified. You can achieve with adding extension of backup file to -i option:
perl -pi.orig -e 's/contoso/'"$hostname"'/g' /etc/inet/hosts
This command will store original content of /etc/inet/hosts into /etc/inet/hosts.orig. Then run the specified command. Then you can check if the files are different with, for example cmp command:
if ! cmp -s foo.txt foo.txt.orig; then
echo OK
else
echo ERROR
fi
Remove the .orig file after that.
The other option is to modify the script to read the content of the file, replace required entry, check is change actually happened and return proper status at the end to verify in the shell using $?. You have been given solution in this answer.
I don't know Perl, but you can manage multiple case of "error" (no match/no way to write in file) with a little Bash script like that :
#!/bin/bash
FILE="/etc/inet/hosts"
SEARCH="contoso"
REPLACE="$hostname"
NB=$(grep -c $SEARCH $FILE)
if [ $NB -ne 0 ]; then
perl -pi -e 's/${SEARCH}/'${REPLACE}'/g' "$FILE" && echo "${NB} replaced" || echo "Error (permission maybe)"
else
echo "No match in file"
fi
I think there is a better way by improving the Perl code or by using the sed command. But it should works.
If you expect your perl script to return a value that has some meaning, you will need to write your perl script to return a meaningful value. In your case, perhaps something as simple as:
perl -p -e 's/contoso/'"$hostname"'/g; $rv=1 if $&; END{ exit !$rv }'
Generally checksums is a very efficient way to detect changes in files.
md5sum [filename]
root#miaoulis:~# echo 'line 1' >>1.txt
root#miaoulis:~# md5sum 1.txt
5c2ce561e1e263695dbd267271b86fb8 1.txt
root#miaoulis:~# echo 'line 2' >>1.txt
root#miaoulis:~# md5sum 1.txt
c7253b64411b3aa485924efce6494bb5 1.txt
I guess the sum could be extracted from the output with AWK
root#miaoulis:~# echo $(md5sum 1.txt) | awk 'BEGIN{FS=" *"}{print "MD5:",$1}'
MD5: c7253b64411b3aa485924efce6494bb5
root#miaoulis:~# echo $(md5sum 1.txt) | awk 'BEGIN{FS=" *"}{print "filename:",$2}'
filename: 1.txt
FS=" *" instructs AWK to split the string on the occurrence of one or more spaces. $1 will be the MD5, $2 will be the filename.
MD5 checksum works fast for any size of file. The downside is that you don't really detect what exactly changed in the file, only the fact that it has changed. Should be good enough for most scenarios.
I have written a shell script in ksh to convert a CSV file into Spreadsheet XML file. It takes an existing CSV file (the path to which is a variable in the script), and then creates a new output file .xls. The script has no positional parameters. The file name of the CSV is currently hardcoded into the script.
I would like to amend the script so it can take the input CSV data from a pipe, and so that the .xls output data can also be piped or redirected (>) to a file on the command line.
How is this achieved?
I am struggling to find documentation on how to write a shell script to take input from a pipe. It appears that 'read' is only used for std input from kb.
Thanks.
Edit : script below for info (now amended to take input from a pipe via the cat, as per the answer to the question.
#!/bin/ksh
#Script to convert a .csv data to "Spreadsheet ML" XML format - the XML scheme for Excel 2003
#
# Take CSV data as standard input
# Out XLS data as standard output
#
DATE=`date +%Y%m%d`
#define tmp files
INPUT=tmp.csv
IN_FILE=in_file.csv
#take standard input and save as $INPUT (tmp.csv)
cat > $INPUT
#clean input data and save as $IN_FILE (in_file.csv)
grep '.' $INPUT | sed 's/ *,/,/g' | sed 's/, */,/g' > $IN_FILE
#delete original $INPUT file (tmp.csv)
rm $INPUT
#detect the number of columns and rows in the input file
ROWS=`wc -l < $IN_FILE | sed 's/ //g' `
COLS=`awk -F',' '{print NF; exit}' $IN_FILE`
#echo "Total columns is $COLS"
#echo "Total rows is $ROWS"
#create start of Excel File
echo "<?xml version=\"1.0\"?>
<?mso-application progid=\"Excel.Sheet\"?>
<Workbook xmlns=\"urn:schemas-microsoft-com:office:spreadsheet\"
xmlns:o=\"urn:schemas-microsoft-com:office:office\"
xmlns:x=\"urn:schemas-microsoft-com:office:excel\"
xmlns:ss=\"urn:schemas-microsoft-com:office:spreadsheet\"
xmlns:html=\"http://www.w3.org/TR/REC-html40\">
<DocumentProperties xmlns=\"urn:schemas-microsoft-com:office:office\">
<Author>Ben Hamilton</Author>
<LastAuthor>Ben Hamilton</LastAuthor>
<Created>${DATE}</Created>
<Company>MCC</Company>
<Version>10.2625</Version>
</DocumentProperties>
<ExcelWorkbook xmlns=\"urn:schemas-microsoft-com:office:excel\">
<WindowHeight>6135</WindowHeight>
<WindowWidth>8445</WindowWidth>
<WindowTopX>240</WindowTopX>
<WindowTopY>120</WindowTopY>
<ProtectStructure>False</ProtectStructure>
<ProtectWindows>False</ProtectWindows>
</ExcelWorkbook>
<Styles>
<Style ss:ID=\"Default\" ss:Name=\"Normal\">
<Alignment ss:Vertical=\"Bottom\" />
<Borders />
<Font />
<Interior />
<NumberFormat />
<Protection />
</Style>
<Style ss:ID=\"AcadDate\">
<NumberFormat ss:Format=\"Short Date\"/>
</Style>
</Styles>
<Worksheet ss:Name=\"Sheet 1\">
<Table>
<Column ss:AutoFitWidth=\"1\" />"
#for each row in turn, create the XML elements for row/column
r=1
while (( r <= $ROWS ))
do
echo "<Row>\n"
c=1
while (( c <= $COLS ))
do
DATA=`sed -n "${r}p" $IN_FILE | cut -d "," -f $c `
if [[ "${DATA}" == [0-9][0-9]\.[0-9][0-9]\.[0-9][0-9][0-9][0-9] ]]; then
DD=`echo $DATA | cut -d "." -f 1`
MM=`echo $DATA | cut -d "." -f 2`
YYYY=`echo $DATA | cut -d "." -f 3`
echo "<Cell ss:StyleID=\"AcadDate\"><Data ss:Type=\"DateTime\">${YYYY}-${MM}-${DD}T00:00:00.000</Data></Cell>"
else
echo "<Cell><Data ss:Type=\"String\">${DATA}</Data></Cell>"
fi
(( c+=1 ))
done
echo "</Row>"
(( r+=1 ))
done
echo "</Table>\n</Worksheet>\n</Workbook>"
rm $IN_FILE > /dev/null
exit 0
Commands inherit their standard input from the process that starts them. In your case, your script provides its standard input for each command that it runs. A simple example script:
#!/bin/bash
cat > foo.txt
Piping data into your shell script causes cat to read that data, since cat inherits its standard input from your script.
$ echo "Hello world" | myscript.sh
$ cat foo.txt
Hello world
The read command is provided by the shell for reading text from standard input into a shell variable if you don't have another command to read or process your script's standard input.
#!/bin/bash
read foo
echo "You entered '$foo'"
$ echo bob | myscript.sh
You entered 'bob'
There is one problem here. If you run the script without first checking to ensure there is input on stdin, then it will hang till something is typed.
So, to get around this, you can check to ensure there is stdin first, and if not, then use a command line argument instead if given.
Create a script called "testPipe.sh"
#!/bin/bash
# Check to see if a pipe exists on stdin.
if [ -p /dev/stdin ]; then
echo "Data was piped to this script!"
# If we want to read the input line by line
while IFS= read line; do
echo "Line: ${line}"
done
# Or if we want to simply grab all the data, we can simply use cat instead
# cat
else
echo "No input was found on stdin, skipping!"
# Checking to ensure a filename was specified and that it exists
if [ -f "$1" ]; then
echo "Filename specified: ${1}"
echo "Doing things now.."
else
echo "No input given!"
fi
fi
Then to test:
Let's add some stuff to a test.txt file and then pipe the output to our script.
printf "stuff\nmore stuff\n" > test.txt
cat test.txt | ./testPipe.sh
Output:
Data was piped to this script!
Line: stuff
Line: more stuff
Now let's test if not providing any input:
./testPipe.sh
Output:
No input was found on stdin, skipping!
No input given!
Now let's test if providing a valid filename:
./testPipe.sh test.txt
Output:
No input was found on stdin, skipping!
Filename specified: test.txt
Doing things now..
And finally, let's test using an invalid filename:
./testPipe.sh invalidFile.txt
Output:
No input was found on stdin, skipping!
No input given!
Explanation:
Programs like read and cat will use the stdin if it is available within the shell, otherwise they will wait for input.
Credit goes to Mike from this page in his answer showing how to check for stdin input: https://unix.stackexchange.com/questions/33049/check-if-pipe-is-empty-and-run-a-command-on-the-data-if-it-isnt?newreg=fb5b291531dd4100837b12bc1836456f
If the external program (that you are scripting) already takes input from stdin, your script does not need to do anything. For example, awk reads from stdin, so a short script to count words per line:
#!/bin/sh
awk '{print NF}'
Then
./myscript.sh <<END
one
one two
one two three
END
outputs
1
2
3
I'm trying to create a shell script and the fact is, I want to change the output if the variable $output is filled. I was thinking about checking the variable with an if inside the command but I don't know if it's the correct syntax. Here is an exemple (of course that doesn't work):
ls -lisa | awk '$5 == own' own="$owner" | sort -k$column -n if [
$output ]; then print > out.txt fi
I don't know if it's going to work that way and if it's possible.
The exec built-in can change the default standard output for the rest of the running shell script. So, in this case, you would do:
if [ -n "$output" ]; then
exec >out.txt
fi
ls -lisa | awk '$5 == own' own="$owner" | sort -k$column
I'm not entirely sure what you're trying to do with the awk part, so this is just verbatim from your question.
Another option is to put the part of your script that you want to redirect into a function, and then call the function in one of two ways, redirecting the output. Example:
do_work() {
ls -lisa | awk '$5 == own' own="$owner" | sort -k$column
}
if [ -n "$output" ]; then
do_work >out.txt
else
do_work
fi
You can use the shell's "use default value" option (${variable:-default}, with /dev/stdout as the default) to do this:
ls -lisa | awk '$5 == own' own="$owner" | sort -k$column -n > "${output:-/dev/stdout}"