I have two scripts. Script A includes script B and calls a function in script B.
The setup looks like this:
Test file - ~/file.txt
one==1.0.0
two==2.0.0
three==3.0.0
four==4.0.0
Script A - ~/script_a.sh
#!/bin/bash
source script_b.sh
func_one
Script B - ~/script_b.sh
#!/bin/bash
# Note: don't forget to change the spaces to tabs else heredoc won't work
my_user=$USER
func_two() {
# Here, I need run everything in the heredoc as user $my_user
sudo su - $my_user -s /bin/bash <<- EOF
while read -r line || [[ -n "$line" ]];
do
# **This is the problem line**
# I can confirm that all the lines are being
# read but echo displays nothing
echo "$line"
# The line below will be printed 4 times as there are 4 lines in the file of interest
echo "Test"
done < "/home/$my_user/file.txt"
EOF
}
func_one() {
func_two
}
To run
cd ~
bash script_a.sh
Question: Why is the line echo "$line" not producing any output?
The problem is that bash is substituting $line with its value (nothing) before it gets passed to su. Escaping the dollar sign should fix it. So $line should be changed to \$line in both places in script_b.sh.
Related
I have a shell script that uses the sed command to transform an input file to a new output file, this script works when I call :
./sed_command.sh input_file > output_file
here is my script :
#!/usr/bin/env sh
while read line
do
# Split each line based on ; separator
transposed=$(echo "$line" | sed -e "s/;/\\n/g")
res=$(echo "$transposed" | sed '/${GDATEF(\([^,]*\),ddMMyyyy)}/{
# Split the string in two part
s//\1/g
# # parse the input for GNU date input format
s/D/ day/g
# Handle shell quoting
'"s/'/\\''\\'/"'
# pass and execute date
s/.*/date -d "now "'\''&'\'' +%d%m%Y/e
}')
oneline=$(echo "$res" | sed -z "s/\n/;/g" | sed 's/.$//')
echo $oneline
done < $1
Now I need to call that script inside a double loop that browse all directories in a folder :
cd /etc/newman/Newman
for team in *;do
if [ -d "$team" ]; then
echo "team=$team"
cd $team
for scope in *;do
if [ -d "$scope" ]; then
echo "scope=$scope"
CSV_FILE=$(ls /etc/newman/Newman/$team/$scope/*.csv -1 || true)
echo "Transforming : ${CSV_FILE}"
if [ ! -z "${CSV_FILE}" ] ; then sh sed_command.sh ${CSV_FILE} > ${CSV_FILE}_Newman; else echo "No CSV files to transform"; fi
fi
done
cd ..
fi
done
I have tested the second script looping works fine the only issue is when I call sed_command.sh inside the second script I get theses two errors :
: not found.sh: line 2:
sed_command.sh: line 19: syntax error: unexpected "done" (expecting "do")
I guess redirection does not work the same way inside a loop? thanks for help
I wrote a Shell script to replace text in the Jenkinsfiles of a large number of repos. I had used this to add on a parameter to a single line of text. However, now I need to insert a line of text before and after an existing command in the Jenkinsfiles. I don't have too much shell experience and could use some help.
Here is the Jenkinsfile text before:
sh "chmod +x increment.sh"
def result = sh returnstdout:true, script: "./increment.sh '${Version}' '${ReleaseVersion}' '${GitRepoURL}' '${CutRelease}' '${Branch}' '${JiraID}'"
//echo "$result"
I need to add the following before the "def result" line:
sshagent(['gitssh']) {
and then add a closing curly bracket after the "def result" line:
}
I need the end result to be:
sh "chmod +x increment.sh"
sshagent(['gitssh']) {
def result = sh returnstdout:true, script: "./increment.sh '${Version}' '${ReleaseVersion}' '${GitRepoURL}' '${CutRelease}' '${Branch}' '${JiraID}'"
}
//echo "$result"
I actually don't care about keeping the commented out echo command if that makes it more difficult, but it is just to show what I have surrounding the "def result" line.
How can I accomplish this end result?
If it helps, I previously was adding new parameters at the end of the "def result" line with this code:
if [ -e Jenkinsfile ]
then
sed -i -e "s/\${Branch}/\${Branch}\' \'\${JiraID}/g" Jenkinsfile
fi
Note: I am on a Mac.
Code so far:
file=repos_remaining.txt
while IFS="," read -r repoURL repoName; do
echo $repoURL
cd ..
echo $repoName
cd $(echo $repoName | tr -d '\r')
file=repos_remaining.txt
if [ -e Jenkinsfile ]
then
# sed -i -e $"s/def result/sshagent([\'gitssh\']) {\
# def result/g" Jenkinsfile
fi
# git add "Jenkinsfile"
# git commit -m "Added JiraID parameter to Jenkinsfile"
# git push origin master
done < "$file"
As with most cases where people want to automate the editing of a file, I suggest using ed:
ed -s Jenkinsfile <<'EOF'
/^def result/i
sshagent(['gitssh']) {
.
.+1a
}
.
w
EOF
The commands in the heredoc tell ed to move the cursor to the first line starting with def result, insert a line above it, append a line after it, and finally write the modified file back to disc.
I am trying to read a text file which has few commented starts with '#', my bash script should read the lines of the text file which doesn't start with '#'.
Also im trying to capture the output of echo statements in both logs and to show it console window for the user understanding.
I have tried to use the below query for capturing logs and printing in console
exec 2>&1 1>>$logfile
For reading each line of the file and calling the function, i have declared an array and to eliminate lines which starts with '#' , i have used the below query.
declare -a cmd_array
while read -r -a cmd_array | grep -vE '^(\s*$|#)'
do
"${cmd_array[#]}"
done < "$text_file"
Note : I need to eliminate the line starts with '#' and remaining lines to be read and place in array as declared.
Bash script
***********
#! /bin/bash
Function_1()
{
now=$( date '+%Y%m%d%H%M' )
eval logfile="$1"_"$now".log
exec 2>&1 1>>$logfile ### Capture echo output in log and printing in console
#exec 3>&1 1>>$logfile 2>&1
echo " "
echo "############################"
echo "Function execution Begins"
echo "############################"
echo "Log file got created with file name as $1.log"
eval number=$1
eval path=$2
echo "number= $number"
ls -lR $path >> temp.txt
if [ $? -eq 0 ]; then
echo " Above query executed."
else
echo "Query execution failed"
fi
echo "############################"
echo "Function execution Ends"
echo "############################"
echo " "
}
text_file=$1
echo $text_file
declare -a cmd_array ### declaring a array
while read -r -a cmd_array | grep -vE '^(\s*$|#)' ### Read each line in the file with doesnt starts with '#' & keep it in array
do
"${cmd_array[#]}"
done < "$text_file"
Text file
*********
####################################
#Test
#Line2
####################################
Function_1 '125' '' ''
Function_1 '123' '' ''
Consider piping the grep output into the read:
declare -a cmd_array ### declaring a array
### Read each line in the file with doesnt starts with '#' & keep it in array
grep -vE '^(\s*$|#)' < "$text_file" | while read -r -a cmd_array
do
"${cmd_array[#]}"
done
I'm not clear about the output/logging comment. If you need the output appended to a file, in addition to stdout/console), consider using the 'tee' (probably 'tee -a')
I tested with the input file inputfile
echo a
Function_1 '125' '' ''
# skip me
Function_1 '123' '' ''
echo b
and wrote this script:
declare -a cmd_array ### declaring a array
while read -r -a cmd_array
do
echo "${cmd_array[#]}"
"${cmd_array[#]}"
echo
done < <(grep -vE '^(\s*$|#)' inputfile)
For showing output in log and console, see https://unix.stackexchange.com/a/145654/57293
As #GordonDavisson suggested in a comment, you get a simular result with
source inputfile
ignoring comments and empty lines, and calling functions, so I am not sure why you would want an array. This command can be included in your master script, you do not need to modify the inputfile.
Another advantage of sourcing the input is the handling of multi-line input and # in strings:
Function_1 '123' 'this is the second parameter, the third will be on the next line' \
'third parameter for the Function_1 call'
echo "This echo continues
on the next line."
echo "Don't delete # comments in a string"
Function_1 '124' 'Parameter with #, interesting!' ''
My professor gave me this confusing assignment :
write a program/script which asks the user for a filename to get input
from, a filename to send errors and a filename to write output if the
filename for output and errors are the same, use the proper redirects
input file will have instructions for your program such as: CHANGE:
this command will tell your program to change the redirects like this:
CHANGE STDIN newfilename CHANGE STOUT newoutfile CHANGE STDERR
newerorfile STOP: stop the program and exit STOP
any other input should be written to the output file Write the number
of lines copied to standard error
your error file might look like this:
1
2
ERROR: Input file not found
filename
CHANGE: redirecting output to filename
3
4
CHANGE: redirecting errors to
**at this point the new error file would start with
5
5
6
STOP requested
**
here are some sample files:
> fileio.test.1
line 1 of first input file
line 2 of first inputt file
CHANGE STDIN stdin.1
since input has changed this line should never
get read EXIT
> stdin.1
line 1 of stdin.1
line 2 of stdin.1
CHANGE STDERR stderr.1
line 3 of stdin.1
CHANGE STDOUT stdout.1
line 4 of stdin.1
CHANGE STDOUT stdout.2
line 5 of stdin.1
CHANGE STDIN stdin.2
since input has changed this line should never get read EXIT
> stdin.2
this should go wherever it's supposed to
EXIT
this line shouldn't be read since our program should have exited by
now
However, I am not quite sure how to interpret these directions. As a stab in the dark, I've done:
#!/bin/bash
read -p "Provide filename to receive input from: " input
read -p "Provide filename to send errors to: " errorFile
read -p "Provide filename to write output to: " writeFile
#set -x
if [ -z $input ]; then
echo "Input file needed."
exit=1
elif [ -z $errorFile ] || [ -z $writeFile ]; then
echo "Supply the correct number of filenames."
exit=1
elif [ $errorFile == $writeFile ]; then
exec 2> $errorFile
exec >&2
else
exec 2> $errorFile
exec 1> $writeFile
fi
while read -r line
do
eval $(echo "$line")
done< $input
Edit: For a function to interpret the input, I was thinking I could match for the specific commands in the testfiles and then trigger the corresponding bash command. Anything that doesn't match would be echo'd and directed to its appropriate file.
ChangeFD () {
if [[ "CHANGE STDE" == ${line:0:11} ]]; then
exec $(echo "2>${line:14}")
elif [[ "CHANGE STDO" == ${line:0:11} ]]; then
exec $(echo "1>${line:14}")
elif [[ "CHANGE STDI" == ${line:0:11} ]]; then
exec $(echo "0<${line:13}")
else
echo $1
fi
}
to count the number of lines (and output that count at regular intervals) you will need to create another variable.
Before your while loop
count=0
and within the while loop (probably just before the 'done < $input' line)
# increment the counter
count=$(( count + 1 ))
# output the counter var to stderr (wherever it is pointing)
echo $count >>&2
exec $(echo "0<${line:13}")
This doesn't work, because redirection operators are not recognized in the replacement of $(…); just use
exec <${line:13}
instead.
This question already has answers here:
here-document gives 'unexpected end of file' error
(10 answers)
Closed 6 years ago.
I have the following file and I have chmod a+x on the file.
When I try to run it, I get a line 75: syntax error: unexpected end of file. What is the error with my script? What do I need to do to fix it?
#!/bin/sh
# log directory for ascp transfer
logdirectory=/tmp/log20079
# log for this script
baselog=$logdirectory/sent4files.log
#directory of where the files to be transferred are located
basedirectory=/tmp/test20079/
#remote host data
REMOTE_DIR=/Upload
REMOTE_HOST=xxx
REMOTE_USER=xxx
# extensions of file
FEATURE_EXT=feature
KEYART_EXT=keyart
TRAILER_EXT=trailer
METADATA_EXT=metadata
# files to be excluded, must match exclude_file_suffix
COMPLETE_EXT=complete
SENT_EXT=sent
# file to send on completion
FILE_ON_COMPLETE="$basedirectory"delivery.complete
if [ "$TYPE" == "File" ]; then
if [ "$STARTSTOP" == "Stop" ]; then
if [ "$STATE" == "success" ]; then
filename=$(basename "$FILE")
extension="${filename##*.}"
# check the extension here, and create files ending in .sent as a flag
case "$extension" in
"$FEATURE_EXT")
cat > "$basedirectory$FEATURE_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Feature sent" >> $baselog
;;
"$KEYART_EXT")
cat > "$basedirectory$KEYART_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Keyart sent" >> $baselog
;;
"$TRAILER_EXT")
cat > "$basedirectory$TRAILER_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Trailer sent" >> $baselog
;;
"$METADATA_EXT")
cat > "$basedirectory$METADATA_EXT.$SENT_EXT" << 'EOF'
EOF
echo "Metadata sent" >> $baselog
;;
esac
# check that all four files exists
if [ -e "$basedirectory$FEATURE_EXT.$SENT_EXT" ] && [ -e "$basedirectory$KEYART_EXT.$SENT_EXT" ] && [ -e "$basedirectory$TRAILER_EXT.$SENT_EXT" ] && [ -e "$basedirectory$METADATA_EXT.$SENT_EXT" ]; then
echo "All files sent" >> $baselog
echo>$FILE_ON_COMPLETE
# $FILE_ON_COMPLETE "$REMOTE_USER#$REMOTE_HOST:$REMOTE_DIR"
rm "$basedirectory$FEATURE_EXT.$SENT_EXT"
rm "$basedirectory$KEYART_EXT.$SENT_EXT"
rm "$basedirectory$TRAILER_EXT.$SENT_EXT"
rm "$basedirectory$METADATA_EXT.$SENT_EXT"
rm $FILE_ON_COMPLETE
fi
fi
fi
fi
Heredocs are tricky beasts to get right. If you use 'EOF' that's exactly what the closing line needs to be, with no whitespace at the front like you have.
Alternatively, you can use the <<- variant which strips off all leading tab characters from the lines in the heredoc and the closing line as per the following transcript (where <tab> is the TAB character):
pax> cat <<-'eof'
...> 1
...< 2
...> <tab>eof
...> 4
...> eof
1
2
<tab>eof
4
pax> cat <<-'eof'
...> 1
...> 2
...> <tab>eof
1
2
Using the <<- variant allows for neater files, thoug it's no good if you want to preserve leading tabs of course. From the bash manpage:
If the redirection operator is <<-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.
Of course, if you're just wanting to use those files as flag files, there's a better way than cat with a heredoc. Just use:
touch "$basedirectory$FEATURE_EXT.$SENT_EXT"
This will create the file if it doesn't exist and update the modification time if it does, just like the cat but without messing about with heredocs. It won't empty the file but, if you need that for some reason:
rm -f "$basedirectory$FEATURE_EXT.$SENT_EXT"
touch "$basedirectory$FEATURE_EXT.$SENT_EXT"
will do the trick.
However, since the heredoc does actually output a single empty line (one \n character), you can opt for:
echo >"$basedirectory$FEATURE_EXT.$SENT_EXT"
instead.
The end of your heredocs EOF can not have any whitespace in front of them. Remove those spaces.