Shell Script to Replace or Insert Lines of Text in File - shell

I wrote a Shell script to replace text in the Jenkinsfiles of a large number of repos. I had used this to add on a parameter to a single line of text. However, now I need to insert a line of text before and after an existing command in the Jenkinsfiles. I don't have too much shell experience and could use some help.
Here is the Jenkinsfile text before:
sh "chmod +x increment.sh"
def result = sh returnstdout:true, script: "./increment.sh '${Version}' '${ReleaseVersion}' '${GitRepoURL}' '${CutRelease}' '${Branch}' '${JiraID}'"
//echo "$result"
I need to add the following before the "def result" line:
sshagent(['gitssh']) {
and then add a closing curly bracket after the "def result" line:
}
I need the end result to be:
sh "chmod +x increment.sh"
sshagent(['gitssh']) {
def result = sh returnstdout:true, script: "./increment.sh '${Version}' '${ReleaseVersion}' '${GitRepoURL}' '${CutRelease}' '${Branch}' '${JiraID}'"
}
//echo "$result"
I actually don't care about keeping the commented out echo command if that makes it more difficult, but it is just to show what I have surrounding the "def result" line.
How can I accomplish this end result?
If it helps, I previously was adding new parameters at the end of the "def result" line with this code:
if [ -e Jenkinsfile ]
then
sed -i -e "s/\${Branch}/\${Branch}\' \'\${JiraID}/g" Jenkinsfile
fi
Note: I am on a Mac.
Code so far:
file=repos_remaining.txt
while IFS="," read -r repoURL repoName; do
echo $repoURL
cd ..
echo $repoName
cd $(echo $repoName | tr -d '\r')
file=repos_remaining.txt
if [ -e Jenkinsfile ]
then
# sed -i -e $"s/def result/sshagent([\'gitssh\']) {\
# def result/g" Jenkinsfile
fi
# git add "Jenkinsfile"
# git commit -m "Added JiraID parameter to Jenkinsfile"
# git push origin master
done < "$file"

As with most cases where people want to automate the editing of a file, I suggest using ed:
ed -s Jenkinsfile <<'EOF'
/^def result/i
sshagent(['gitssh']) {
.
.+1a
}
.
w
EOF
The commands in the heredoc tell ed to move the cursor to the first line starting with def result, insert a line above it, append a line after it, and finally write the modified file back to disc.

Related

Refactoring my shell scipts : calling a script and redirect output to a new file inside a loop

I have a shell script that uses the sed command to transform an input file to a new output file, this script works when I call :
./sed_command.sh input_file > output_file
here is my script :
#!/usr/bin/env sh
while read line
do
# Split each line based on ; separator
transposed=$(echo "$line" | sed -e "s/;/\\n/g")
res=$(echo "$transposed" | sed '/${GDATEF(\([^,]*\),ddMMyyyy)}/{
# Split the string in two part
s//\1/g
# # parse the input for GNU date input format
s/D/ day/g
# Handle shell quoting
'"s/'/\\''\\'/"'
# pass and execute date
s/.*/date -d "now "'\''&'\'' +%d%m%Y/e
}')
oneline=$(echo "$res" | sed -z "s/\n/;/g" | sed 's/.$//')
echo $oneline
done < $1
Now I need to call that script inside a double loop that browse all directories in a folder :
cd /etc/newman/Newman
for team in *;do
if [ -d "$team" ]; then
echo "team=$team"
cd $team
for scope in *;do
if [ -d "$scope" ]; then
echo "scope=$scope"
CSV_FILE=$(ls /etc/newman/Newman/$team/$scope/*.csv -1 || true)
echo "Transforming : ${CSV_FILE}"
if [ ! -z "${CSV_FILE}" ] ; then sh sed_command.sh ${CSV_FILE} > ${CSV_FILE}_Newman; else echo "No CSV files to transform"; fi
fi
done
cd ..
fi
done
I have tested the second script looping works fine the only issue is when I call sed_command.sh inside the second script I get theses two errors :
: not found.sh: line 2:
sed_command.sh: line 19: syntax error: unexpected "done" (expecting "do")
I guess redirection does not work the same way inside a loop? thanks for help

How to import bash aliases into fish

I am missing my bash aliases in fish, and don't want to manually convert all of them to fish functions.
How to get them all from bash to fish?
Bonus points if:
the solution supports an iterative process, as in: i can easily change the aliases in bash, and re-convert/re-import them into fish
the solution also imports bash functions
Convert bash aliases to bash scripts
I decided to do this instead of approach below, and putting the scripts into ~/bin/, which is in my PATH. This allows me to use them from whatever shell i am currently executing, and it prevents potential problems with quoting.
use like so:
# converts all bash aliases to script files
convert_bash_aliases_to_scripts
# removes all scripts previously converted by this script
convert_bash_aliases_to_scripts clean
bash conversion script:
#!/bin/bash
# Convert bash aliases to bash scripts.
#
# Copyright 2018 <hoijui.quaero#gmail.com>, licensed under the GPL-3.0+
#
# Usage:
# convert_bash_aliases_to_scripts # converts all bash aliases to script files
# convert_bash_aliases_to_scripts clean # removes all scripts previously converted by this script
COLOR_RED=$'\e[0;31m'
COLOR_ORANGE=$'\e[0;33m'
COLOR_BLUE=$'\e[0;34m'
COLOR_BLUE_LIGHT=$'\e[1;34m'
COLOR_GREEN=$'\e[0;32m'
COLOR_BROWN=$'\e[0;33m'
COLOR_YELLOW=$'\e[1;33m'
COLOR_WHITE=$'\e[1;37m'
COLOR_CYAN=$'\e[0;36m'
COLOR_PURPLE=$'\e[0;35m'
COLOR_GRAY=$'\e[1;30m'
COLOR_GRAY_LIGHT=$'\e[0;37m'
COLOR_NONE=$'\e[m' # No Color
OUTPUT_DIR=~/bin/converted/aliases
LINKS_DIR=~/bin
README_FILE_NAME="README.md"
README_FILE="$OUTPUT_DIR/$README_FILE_NAME"
if [ "$1" = "clean" ]
then
for script_file in $(find "$LINKS_DIR" -maxdepth 1 -type l)
do
conv_script_file="$OUTPUT_DIR/$(basename $script_file)"
if [ -e $conv_script_file ] && [ "$(readlink --canonicalize $script_file)" = "$(realpath $conv_script_file)" ]
then
script_name=$(basename $script_file)
echo "removing converted bash alias-script: $script_name"
rm $conv_script_file \
&& rm $script_file
fi
done
rm $README_FILE 2> /dev/null
rmdir $OUTPUT_DIR 2> /dev/null
exit 0
fi
SOURCE_FILES="${HOME}/.bashrc ${HOME}/.bash_aliases"
mkdir -p $OUTPUT_DIR
echo -e "# Bash alias conversion scripts\n\nsee $0\n\nWARNING: Do NOT manually edit files in this directory. instead, copy them to $LINKS_DIR (replacing the symbolic link that already exists there), and edit that new file.\nIf you edit the files in this dir, it will be replaced on the next (re)conversion from aliases." \
> $README_FILE
AUTO_IMPORT_WARNING="# WARNING Do NOT edit this file by hand, as it was auto-generated from a bash alias, and may be overwritten in the future. please read ${README_FILE}"
function _is_link_to {
local file_link=$1
local file_target=$2
test -e $file_target \
&& test "$(readlink --canonicalize $file_link)" = "$(realpath $file_target)"
return $?
}
for source_file in $SOURCE_FILES
do
IFS=$'\n'
for a in $(cat $source_file | grep "^alias")
do
a_name="$(echo "$a" | sed -e 's/alias \([^=]*\)=.*/\1/')"
a_command="$(echo "$a" | sed -e 's/alias \([^=]*\)=//' -e 's/[ \t]*#.*$//')"
if echo "${a_command:0:1}" | grep -q -e "[\'\"]"
then
# unquote
a_command_start=1
let a_command_end="${#a_command} - 2"
else
# leave as is
a_command_start=0
let a_command_end="${#a_command}"
fi
script_file="$LINKS_DIR/$a_name"
conv_script_file="$OUTPUT_DIR/$a_name"
# Check whether the script already exists.
# If so, we skip importing it, unless it is just a link to a previously imported script.
log_action="none"
log_action_color="${COLOR_NONE}"
log_content=""
if [ -e $script_file ] && ! $(_is_link_to $script_file $conv_script_file)
then
log_action="skipped (exists)"
log_action_color="${COLOR_ORANGE}"
log_content=""
else
if [ -e $script_file ]
then
log_action="reimporting"
log_action_color="${COLOR_BLUE}"
else
log_action="importing"
log_action_color="${COLOR_GREN}"
fi
# write the script file to a temporary location
conv_script_file_tmp="${conv_script_file}_BAK"
echo "#!/bin/bash" > $conv_script_file_tmp
echo -e "$AUTO_IMPORT_WARNING" >> $conv_script_file_tmp
echo -e "#\n# Imported bash alias '$a_name' from file '$source_file'" >> $conv_script_file_tmp
cat >> "${conv_script_file_tmp}" <<EOF
${a_command:${a_command_start}:${a_command_end}} \${#}
EOF
if diff -N ${conv_script_file_tmp} ${conv_script_file} > /dev/null
then
log_content="no change"
log_content_color="${COLOR_NONE}"
else
log_content="changed"
log_content_color="${COLOR_GREEN}"
fi
log_content=$(printf "%s %10s -> %s${COLOR_NONE}" "${log_content_color}" "${log_content}" "$a_command")
mv "${conv_script_file_tmp}" "${conv_script_file}"
# make the script executable
chmod +x $conv_script_file
# remove the link if it already exists (in case of reimport)
rm $script_file 2> /dev/null
# .. and re-create it as local symbolic link
# to the function in the imports dir
ln --symbolic --relative $conv_script_file $script_file
fi
printf "%s%20s: %-25s${COLOR_NONE}%s\n" "${log_action_color}" "${log_action}" "$a_name" "${log_content}"
done
done
Deprecated: Creating fish wrappers that execute bash code
Below is a script that creates fish script wrappers for the local bash aliases: For each bash alias, it takes the contents, and creates a fish alias/script that executes the code in bash sub-shell.
It is not optimal, but is sufficient for most of my aliases.
WARNING It might happen that the imported function acts differently then in bash. You may loose data or accidentally DDOS your coworkers when using them.
use like so:
# imports (or reimports) all bash aliases into fish functions, permanently
import_bash_aliases
# removes all fish functions previously imported by this script
import_bash_aliases clean
save this in ~/.config/fish/functions/import_bash_aliases.fish:
#!/usr/bin/fish
# Fish function to import bash aliases
#
# Copyright 2018 <hoijui.quaero#gmail.com>, licensed under the GPL-3.0+
#
# This script is based on a script from Malte Biermann,
# see: https://glot.io/snippets/efh1c4aec0
#
# WARNING: There is no guarantee that the imported aliases work the same way
# as they do in bash, so be cautious!
#
# Usage:
# import_bash_aliases # imports (or reimports) all bash aliases into fish functions, permanently
# import_bash_aliases clean # removes all fish functions previously imported by this script from bash aliases
function import_bash_aliases --description 'Converts bash aliases to .fish functions.\nThis might be called repeatedly, and will not override functions that are already defined in fish, except they are merely an older import from this script.'
set -l FISH_FUNCTIONS_DIR ~/.config/fish/functions
set -l BASH_IMPORTS_DIR_NAME bash-imports
set -l BASH_IMPORTS_DIR $FISH_FUNCTIONS_DIR/$BASH_IMPORTS_DIR_NAME
set -l README_FILE $BASH_IMPORTS_DIR/README.md
if test "$argv[1]" = "clean"
for fun_file in (find $FISH_FUNCTIONS_DIR -maxdepth 1 -name '*.fish')
set -l imp_fun_file $BASH_IMPORTS_DIR/(basename $fun_file)
if test -e $imp_fun_file ; and test (readlink --canonicalize $fun_file) = (realpath $imp_fun_file)
set -l fun_name (basename $fun_file '.fish')
echo "removing imported bash alias/function $fun_name"
rm $imp_fun_file
and rm $fun_file
and functions --erase $fun_name
end
end
rm $README_FILE ^ /dev/null
rmdir $BASH_IMPORTS_DIR ^ /dev/null
return 0
end
set -l SOURCE_FILES ~/.bashrc ~/.bash_aliases
mkdir -p $BASH_IMPORTS_DIR
echo -e "# Bash alias imports\n\nsee `$argv[0]`\n\nWARNING: Do NOT manually edit files in this directory. instead, copy them to $FISH_FUNCTIONS_DIR (replacing the symbolic link that already exists there), and edit that new file.\nIf you edit the files in this dir, it will be replaced on the next (re)import from bash aliases." \
> $README_FILE
set -l UNUSED_STUB_MSG "The bash alias corresponding to this function was NOT imported, because a corresponding function already exists at %s\n"
set -l AUTO_IMPORT_WARNING "# WARNING Do NOT edit this file by hand, as it was auto-generated from a bash alias, and may be overwritten in the future. please read {$README_FILE}"
function _fish_func_exists
set -l fun_name $argv[1]
# This also detects in-memory functions
functions --query $fun_name
# This also detects script files in the functions dir
# that do not contain a function wiht the same name
or test -e "$FISH_FUNCTIONS_DIR/$fun_name.fish"
return $status
end
function _is_link_to
set -l file_link $argv[1]
set -l file_target $argv[2]
test -e $file_target
and test (readlink --canonicalize $file_link) = (realpath $file_target)
return $status
end
for source_file in $SOURCE_FILES
for a in (cat $source_file | grep "^alias")
set -l a_name (echo $a | sed -e 's/alias \([^=]*\)=.*/\1/')
set -l a_command (echo $a | sed -e 's/alias \([^=]*\)=//' -e 's/[ \t]*#[^\'\"]\+$//')
set -l fun_file "$FISH_FUNCTIONS_DIR/$a_name.fish"
set -l imp_fun_file "$BASH_IMPORTS_DIR/$a_name.fish"
# Check whether the function already exists.
# If so, we skip importing it, unless it is just a link to a previously imported function.
if _fish_func_exists $a_name; and not _is_link_to $fun_file $imp_fun_file
set_color red
printf "%20s: %-25s\n" "skipping (exists)" $a_name
set_color normal
#printf $UNUSED_STUB_MSG $fun_file > $imp_fun_file
else
set_color green
printf "%20s: %-25s -> %s\n" "(re-)importing" $a_name $a_command
set_color normal
# remove the link, in case of re-importing
rm $fun_file ^ /dev/null
# write the function file
echo "#!/usr/bin/fish" > $imp_fun_file
echo "\
$AUTO_IMPORT_WARNING
function $a_name -d 'bash alias "$a_name" import'
bash -c $a_command' '\$argv''
end
" \
>> $imp_fun_file
# make the script executable
chmod +x $imp_fun_file
# .. and re-create it as local symbolic link
# to the function in the imports dir
ln --symbolic --relative $imp_fun_file $fun_file
end
end
end
# (re-)load all the functions we just defined
exec fish
end

Read file line by line in sourced bash script within heredoc

I have two scripts. Script A includes script B and calls a function in script B.
The setup looks like this:
Test file - ~/file.txt
one==1.0.0
two==2.0.0
three==3.0.0
four==4.0.0
Script A - ~/script_a.sh
#!/bin/bash
source script_b.sh
func_one
Script B - ~/script_b.sh
#!/bin/bash
# Note: don't forget to change the spaces to tabs else heredoc won't work
my_user=$USER
func_two() {
# Here, I need run everything in the heredoc as user $my_user
sudo su - $my_user -s /bin/bash <<- EOF
while read -r line || [[ -n "$line" ]];
do
# **This is the problem line**
# I can confirm that all the lines are being
# read but echo displays nothing
echo "$line"
# The line below will be printed 4 times as there are 4 lines in the file of interest
echo "Test"
done < "/home/$my_user/file.txt"
EOF
}
func_one() {
func_two
}
To run
cd ~
bash script_a.sh
Question: Why is the line echo "$line" not producing any output?
The problem is that bash is substituting $line with its value (nothing) before it gets passed to su. Escaping the dollar sign should fix it. So $line should be changed to \$line in both places in script_b.sh.

How to verify the commands to be run by shell script

If I use set -x, then the commands are displayed just before they are executed. That way they are printed.
But I want to have a debug mode for my script where the user can actually see what commands are going to get printed but those are not executed.
I also tried using : (null command) to print the current command but that does not propagate the result. e.g.,
find /my/home -name "test*" | while read -r i;
do
rm -f $i
done
For this purpose, the out put expected is:
+find /my/home -name "test"
+ rm -f test1
+ rm -f test2 ...
and so on
Is there any way I can achieve this without repeating code (the obvious way being have 2 sections in the batch script for debug and normal mode)?
You can maybe create a wrapper function that either prints or evaluates the command you give to it:
#!/bin/bash
run_command () {
printf '%q ' "$#"
"$#"
}
run_command ls -l
run_command touch /tmp/hello
run_command rm /tmp/hello
This way, you prepend run_command to any single thing you want to do and comment the execution or the echo action as you wish.
You could also provide a parameter to the script that switches to either echo or execute mode:
debug_mode=$1
run_command () {
if [ "$debug_mode" = true ]; then
printf '%q ' "$#"
else
"$#"
fi
}
run_command ...
Most simplest way to run preview/dryrun:
for f in *; do echo "these files will be removed with rm -f $f"; done

SSH commands via bash script

I've been trying several fails to perform the following:
Basically, what I need is to execute several sequenced commands on a remote unix shell, such as setting environment variables with variables that I have on the script, move to a particular directory and run a script there and so on.
I've tried using a printf with the portion of the script and then piped the ssh command, but it didn't work quite well, also, I've read about the "ssh ... >> END" marker, which is great but since I'm using functions, it doesn't work well.
Do you have any thoughts?
Here's an excerpt of the code:
deployApp() {
inputLine=$1;
APP_SPECIFIC_DEPLOY_SCRIPT="$(echo $inputLine | cut -d ' ' -s -f1)";
BRANCH="$(echo $inputLine | cut -d ' ' -s -f2)";
JBOSS_HOME="$(echo $inputLine | cut -d ' ' -s -f3)";
BASE_PORT="$(echo $inputLine | cut -d ' ' -s -f4)";
JAVA_HOME_FOR_JBOSS="$(echo $inputLine | cut -d ' ' -s -f5)";
JAVA_HEAP="$(echo $inputLine | cut -d ' ' -s -f6)";
echo "DEPLOYING $APP_SPECIFIC_DEPLOY_SCRIPT"
echo "FROM BRANCH $BRANCH"
echo "IN JBOSS $JBOSS_HOME"
echo "WITH BASE PORT $BASE_PORT"
echo "USING $JAVA_HOME_FOR_JBOSS"
if [[ -n "$JAVA_HEAP" ]]; then
echo "WITH $JAVA_HEAP"
fi
echo
echo "Exporting jboss to $JBOSS_HOME"
ssh me#$SERVER <<END
cleanup() {
rm -f $JBOSS_SERVER/log/*.log
rm -Rf $JBOSS_SERVER/deploy/
rm -Rf $JBOSS_SERVER/tmp/
mkdir $JBOSS_SERVER/deploy
}
startJboss() {
cd $JBOSS_SERVER/bin
./jbossctl.sh start
return 0;
}
export JBOSS_HOME
export JBOSS_SERVER=$JBOSS_HOME/server/default
END
return 0;
}
With that "HERE" approach, I'm getting this error: "syntax error: unexpected end of file"
Thanks a lot in advance!
Just put the functions in your here document, too:
var="Hello World"
ssh user#host <<END
x() {
print "x function with args=$*"
}
x "$var"
END
[EDIT] Some comments:
You say "export JBOSS_HOME" but you never define a value for the variable in the here document. You should use export JBOSS_HOME="$JBOSS_HOME". BASH will take all text between the two END, replace all variables, and send the result to SSH for processing.
That also means the other side will see rm -f /path/to/jboss/server/*.log; the assignment to JBOSS_SERVER in the last line of the here document has no effect (at least not to the code in cleanup()).
If you want to pass $ unmodified to the remote server, you have to escape it with \: rm -f \$JBOSS_SERVER/log/*.log
You never call cleanup()
There is a } missing after return 0 to finish the definition of deployapp()
There may be other problems as well. Run the script with bash -x to see what the shell actually executes. You can also add echo commands in the here document to see what the values of the variables are or you can add set -x before cleanup() to get the same output as with bash -x but from the remote side.
I don't understand why you're using cut to split the arguments to your function. Just do
APP_SPECIFIC_DEPLOY_SCRIPT=$1
BRANCH=$2
JBOSS_HOME=$3
# etc.
If you don't quote your here document delimiter, the contents are expanded before they're sent to the server. That may be what you want. If you don't and you want all expansion to be done on the server side, then quote it like this:
ssh me#$SERVER <<'END'
# etc.
END
If you wan't a mixture, don't quote the delimiter, but do escape those things that you want delayed expansion for:
ssh me#$SERVER <<END
echo $EXPAND_ME_NOW \$EXPAND_ME_LATER
END
What are the export statements supposed to do? I can't see that they would have any effect at all.

Resources