I would like to remotely change a Jenkins build description. I have my script all set and ready except for one tiny problem: Multiple line descriptions.
I am using the REST API and JSON in Jenkins to download the old description:
old_description=$(curl -s --user "$USER:$PASSWORD" --data-urlencode "tree=description" \
"$jenkins_url/job/$job_name/$build_number/api/json")
old_description=${old_description#*:\"} #Remove JSON garbage
old_description=${old_description%\"\}} #Remove JSON garbage
The `curl command pulls out:
<font color=blue><b>At first you don't succeed. Try again</b></font><br/>
\r\n<font color=gold><b>At first you don't succeed. Try again</b></font><br/>
\r\n<font color=green><b>At first you don't succeed. Try again</b></font>
(Note: I added line breaks to make the above easier to read. This is pulled out as a single line).
The \r\n are separate lines, so I do this:
old_description=$(sed 's/\\r\\n/\
/g' <<<$old_description)
And that changes $old_description to:
font color=blue><b>At first you don't succeed. Try again</b></font><br/>
<font color=gold><b>At first you don't succeed. Try again</b></font><br/>
<font color=green><b>At first you don't succeed. Try again</b></font>
(NOTE: The new lines are part of the value. This is a three line description.)
My program (depending upon the command line parameters) can replace, append, or prepend a new description to the build:
if [ "$prepend_flag" -a -n "$old_description" ] #Prepend new description to old description
then
new_description="$new_description<br/>
$old_description"
elif [ "$append_flag" -a -n "$old_description" ] #Append new description to old description
then
new_description="$old_description<br/>
$new_description"
fi
Now, I'll redo the description:
if curl -u $USER:$PASSWORD --data-urlencode "description=$new_description" \
--data-urlencode "Submit=Submit" \
"$jenkins_url/job/$job_name/$build_number/submitDescription"
then
echo "Description successfully changed on Build #$build_number in Jenkins job $job_name"
else
echo "WARNING: Description was not set. Manually change the descripiton of the build"
echo " for Build #$build_number in Jenkins job $job_name"
fi
If I am prepending or appending the new description the first time, I get this in Jenkins:
<font color=blue><b>At first you don't succeed. Try again</b></font><br/>
<font color=gold><b>At first you don't succeed. Try again</b></font><br/>
<font color=green><b>At first you don't succeed. Try again</b></font><br/>
<font color=red><b>My new description</b></font><br/>
Looks nice. The next time, it doesn't work. I get this:
<font color=blue><b>At first you don't succeed. Try again</b></font><br/>\n<font color=gold><b>At first you don't succeed. Try again</b></font><br/>\n<font color=green><b>At first you don't succeed. Try again</b></font><br/>\n<font color=red><b>My new description</b></font><br/>
<font color=blue><b>My new new description</b></font>
Note the \n showing up.
How can I fix this issue?
I've put the whole program in pastebin.
I played around with this for a long while...
First, instead of doing this:
new_description="$new_description<br/>
$old_description"
to append or prepend the line, I used printf:
new_description="$(printf "$new_description\r\n$old_description")"
By using printf, I put a <CR><LF> instead of just a <LF> in my description line separator. This way, I don't have a jumble of <NL> and <CR><NL> and I'm no longer dependent upon the operating system's definition of the line break.
The sed command took me a long, long time to figure out. I tried all sorts of things:
old_description=$(sed 's/\\r\\n/\r\n/g' <<<$old_description)
But, nothing seemed to work... I tried the -E flag which allows me to use the extended regular expressions, but it kept interpreting \r\n as replacing \\r\\n with literal 'rn.
After several hours of this, I finally tried double quotation marks instead of single quotation marks:
old_description=$(sed "s/\\r\\n/\r\n/g" <<<$old_description)
That worked! You normally use single quotation marks with sed to protect the regular expression from interpolation. However, the single quotes were also killing the interpolation of \r\n as <CR><LF>. Changing them with double quotes solved the problem.
Related
Situation: Using a shell script (bash/ksh), there is a message that should be shown in the console log, and subsequently sent via email.
Problem: There are newline characters in the message.
Example below:
ErrMsg="File names must be unique. Please correct and rerun.
Duplicate names are listed below:
File 1.txt
File 1.txt
File 2.txt
File 2.txt
File 2.txt"
echo "${ErrMsg}"
# OK. After showing the message in the console log, send an email
Question: How can these newline characters be translated into HTML line breaks for the email?
Constraint: We must use HTML email. Downstream processes (such as Microsoft Outlook) are too inconsistent for anything else to be of use. Simple text email is usually a good choice, but off the table for this situation.
To be clear, the newlines do not need to be completely removed, but HTML line breaks must be inserted wherever there is a newline character.
This question is being asked because I have already attempted to use several commands, such as sed, tr, and awk with varying degrees of success.
TL;DR: The following snippet will do the job:
ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`
Just make sure there are double quotes around the variable when using echo.
This turned out to be a tricky situation. Some notes of explanation are below.
Using sed
Turns out, sed reads through input line by line, which makes finding and replacing those newlines somewhat outside the norm. There were several clever tricks that appeared to work, but I felt they were far too complicated to apply appropriately to this rather simple situation.
Using tr
According to this answer the tr command should work. Unfortunately, this only translates character by character. The two character strings are not the same length, and I am limited to translating the newline into a space or other single character.
For the following:
ErrMsg="Line 1
Line 2
"
ErrMsg=`echo $ErrMsg| tr '\n' 'BREAK'`
# You might expect:
# "Line 1BREAKLine 2BREAK"
# But instead you get:
# "Line 1BLine 2B"
echo "${ErrMsg}"
Using awk
Using awk according to this answer initially appeared to work, but due to some other circumstances with echo there was a subtle problem. The solution is noted in this forum.
You must have double-quotes around your variable, or echo will strip out all newlines.(Of course, awk will receive the characters with a newline at the end, because that's what echo does after it echos stuff.)
This snippet is good: (line breaks in the middle are preserved and replaced correctly)
ErrMsg=`echo "$ErrMsg"|awk 1 ORS='<br/>'`
This snipped is bad: (newlines converted to spaces by echo, one line break at end)
ErrMsg=`echo $ErrMsg|awk 1 ORS='<br/>'`
You can wrap your message in HTML using <pre>, something like
<pre>
${ErrMsg}
and more.
</pre>
I'm trying to call rrdtool xport command on arbitrary number of files, so I'm writing a script that reads in the rrd file names and builds the DEF argument. The problem is some of the rrd files have whitespaces in them, i.e. "foo bar.rrd" (-_-)...and when the DEF argument is generated, it looks something like this:
DEF:a=foo bar.rrd:sum:AVERAGE
and when this is passed in to the rrdtool command, it generates an error saying "problems reading database name". I also have tried inserting the escape character ("\") before whitespace so it would look like "foo\ bar.rrd", but when this is run in bash, it still produces same error, whereas when I echo the command and copy paste it on the prompt and run it then it works fine...
Just put quotes around the whole thing
"DEF:a=foo bar.rrd:sum:AVERAGE"
rrdtool should be fine with the spaces.
I am currently trying to install GCC-4.1.2 on my machine: Fedora 20.
In the instruction, the first three commands involve using 'sed' commands, for Makefile modification. However, I am having difficulty in using those commands properly for my case. The website link for GCC-4.1.2.
The commands are:
sed -i 's/install_to_$(INSTALL_DEST) //' libiberty/Makefile.in &&
sed -i 's#\./fixinc\.sh#-c true#' gcc/Makefile.in &&
sed -i 's/#have_mktemp_command#/yes/' gcc/gccbug.in &&
I am trying to understand them by reading the 'sed' man page, but it is not so easy to do so. Any help/tip would be appreciated!
First, the shell part: &&. That just chains the commands together, so each subsequent line will only be run if the prior one is run successfully.
sed -i means "run these commands inline on the file", that is, modify the file directly instead of printing the changed contents to STDOUT. Each sed command here (the string) is a substitute command, which we can tell because the command starts with s.
Substitute looks for a piece of text in the file, and then replaces it. So the order is always s/needle/replacement/. See how the first and last lines have those same forward-slashes? That's the traditional delimiter between the command (substitute), the needle to find in the haystack (install_to_$(INSTALL_DEST), and the text to replace it with ().
So, the first one looks for the string and deletes it (the empty replacement). The last one looks for #have_mktemp_command# and replaces it with yes.
The middle one is a bit weird. See how it starts with s# instead of s/? Well, sed will let you use any delimiter you like to separate the needle from the replacement. Since this needle had a / in it (\./fixinc\.sh), it made sense to use a different delimiter than /. It will replace the text ./fixinc.sh with -c true.
Last note: Why does the second needle have \. instead of .? Well, in a Regular Expression like the needle is (but not used in your example), some characters are magical and do magical fairy dust operations. One of those magic characters is .. To avoid the magic, we put a \ in front of it, escaping away from the magic. (The magic is "match any character", and we want a literal period. That's why.)
I'm passing some TeamCity parameters to the command line build step. The problem comes when the parameter value contains spaces, e.g.:
%env.TEAMCITY_BUILDCONF_NAME% ---> My TC Project
Is there a way to replace white spaces with some other character, for example underscore?
%env.TEAMCITY_BUILDCONF_NAME% ---> My_TC_Project
You can typically keep using the white spaces if you wrap the parameter in double quotes:
%program.files.dir% => C:\Program Files (x86)
Executable: dir
Parameters: "%program.files.dir%"
I don't know how to replace spaces with underscore, but I had an issue with whitespaces.
In a TeamCity build step, I was trying to run an sqlcmd as Executable with Parameters
-S %sql_server% -U %sql_username% -P %sql_password%
-i "custom_script.sql" -d "%custom_db%"
-v DealerName="%DealerName%"
where DealerName was "Great Dealer Ltd" but it didn't work with white spaces, even with double quotes.
It fixed the issue by setting it as a Custom Script like
sqlcmd -S %sql_server% -U %sql_username% -P %sql_password%
-i "custom_script.sql" -d "%custom_db%"
-v DealerName="%DealerName%"
and (thanks to my boss suggestion) it worked like charm.
Even if is not the precise answer to your question, it could be useful for similar issues.
String given below worked for me.
%env.TEAMCITY_BUILDCONF_NAME% ---> "My\ TC\ Project"
I would need to read certain data using curl. I'm basically reading keywords from file
while read line
do
curl 'https://gdata.youtube.com/feeds/api/users/'"${line}"'/subscriptions?v=2&alt=json' \
> '/home/user/archive/'"$line"
done < textfile.txt
Anyway I haven't found a way to form the url to curl so it would work. I've tried like every possible single and double quoted versions. I've tried basically:
'...'"$line"'...'
"..."${line}"..."
'...'$line'...'
and so on.. Just name it and I'm pretty sure that I've tried it.
When I'm printing out the URL in the best case it will be formed as:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE
or something similar. If you know what could be the cause of this I would appreciate the information. Thanks!
It's not a quoting issue. The problem is that your keyword file is in DOS format -- that is, each line ends with carriage return & linefeed (\r\n) rather than just linefeed (\n). The carriage return is getting read into the line variable, and included in the URL. The giveaway is that when you echo it, it appears to print:
/subscriptions?v=2&alt=jsoneeds/api/users/KEYWORD FROM FILE"
but it's really printing:
https://gdata.youtube.com/feeds/api/users/KEYWORD FROM FILE
/subscriptions?v=2&alt=json
...with just a carriage return between them, so the second overwrites the first.
So what can you do about it? Here's a fairly easy way to trim the cr at the end of the line:
cr=$'\r'
while read line
do
line="${line%$cr}"
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/$line"
done < textfile.txt
Your current version should work, I think. More elegant is to use a single pair of double quotes around the whole URL with the variable in ${}:
"https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
Just use it like this, should be sufficient enough:
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" > "/home/user/archive/${line}"
If your shell gives you issues with & just put \&, but it works fine for me without it.
If the data from the file can contain spaces and you have no objection to spaces in the file name in the /home/user/archive directory, then what you've got should be OK.
Given the contents of the rest of the URL, you could even just write:
while read line
do
curl "https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json" \
> "/home/user/archive/${line}"
done < textfile.txt
where strictly the ${line} could be just $line in both places. This works because the strings are fixed and don't contain shell metacharacters.
Since you're code is close to this, but you claim that you're seeing the keywords from the file in the wrong place, maybe a little rewriting for ease of debugging is in order:
while read line
do
url="https://gdata.youtube.com/feeds/api/users/${line}/subscriptions?v=2&alt=json"
file="/home/user/archive/${line}"
curl "$url" > "$file"
done < textfile.txt
Since the strings may end up containing spaces, it seems (do you need to expand spaces to + in the URL?), the quotes around the variables are strongly recommended. You can now run the script with sh -x (or add a line set -x to the script) and see what the shell thinks it is doing as it is doing it.