I'm trying to create a multi-line string cleanly which includes backslashes and variable substitution. Originally, I did not need backslashes and was able to achieve what I needed with this snippet:
description=$(cat <<EOF
Creation time: $(date)
Creator: $test_group_creator
Test group: $test_group
Test execution: $test_execution_id - $test_name
Adapter: $initiator1_mac_addr
http://...
EOF
)
It's not ideal as it looks a little clunky at the end and requires I break the indentation level of the code, but it's pretty clear and represents how the description will appear on a webpage later. (I am aware of the <<-EOF syntax, but we are using 4-space indents in code and that syntax requires tabs.) Here is the variable echoed out:
Creation time: Tue Jun 27 15:22:27 PDT 2017
Creator: langlorx
Test group: all
Test execution: 13 - go
Adapter: eth0
link...
However, recently, the text has been also dropped into a Wiki parser and now line-breaks are collapsed to a single space. To force a line-break mid-paragraph, the line must end in a double backslash. Knowing that bash treats backslashes specially, I knew I had to double up on the double-backslashes, but it still didn't come out right. Using this:
description=$(cat <<EOF
Creation time: $(date)\\\\
Creator: $test_group_creator\\\\
Test group: $test_group\\\\
Test execution: $test_execution_id - $test_name\\\\
Adapter: $initiator1_mac_addr\\\\
http://...
EOF
)
I find that the newlines are completely gone in the resulting variable, and last \\\\ sequence before the blank line results in a single backslash in the variable with the blank line completely gone. Without any backslashes, the resulting string comes out exactly as written above with newlines, blank lines and with the variables substituted. Once I include any backslashes at the end of a line, it triggers a different set of rules in bash for encoding the variable.
Creation time: Tue Jun 27 15:20:17 PDT 2017\\Creator: user\\Test group: all\\Test execution: 12 - go\\Adapter: eth0\link...
Is there a better way to do this in bash than trying to abuse cat and command-substitution while still having the code somewhat clean?
Store the data in the format you want it, and then just postprocess as necessary. If you want to ignore leading spaces and append \\ to each line, just do that:
embed() { sed -e 's/^ *//; s/$/\\\\/'; }
description=$(embed << eof
Creation time: $(date)
Creator: foo
Test group: bar
etc
eof
)
printf '%s\n' "$description"
will give you:
Creation time: Tue Jun 27 15:33:31 PDT 2017\\
Creator: foo\\
Test group: bar\\
etc\\
Related
We have a project using Azure Pipeline, relying on azure-pipelines.yml file at the repo's root.
When implementing a script step, it is possible to execute successive commands in the same step simply writing them on different lines:
- script: |
ls -la
pwd
echo $VALUE
Yet, if we have a single command that is very long, we would like to be able to break it on several lines in the YAML file, but cannot find the corresponding syntax?
You didn't specify your agent OS so I tested on both windows-latest and ubuntu-latest. Note that the script task runs a bit differently on these 2 environments. On Windows, it uses cmd.exe. On Ubuntu, it uses bash. Therefore, you have to use the correct syntax.
On Windows:
pool:
vmImage: 'windows-latest'
steps:
- script: |
mkdir ^
test ^
-p ^
-v
On Ubuntu:
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
mkdir \
test \
-p \
-v
Those two files above work on my Azure DevOps.
At the moment, the only way we found for to break a single command on multiple line is using YAML folded style:
- script: >
echo
'hello world'
It is all about replacing | with >.
Notes:
It is not possible to introduce extra indentation on the following lines! For example, trying to align all arguments given to a command would break the behaviour.
This style will replace newlines in the provided value with a simple white space. This means the script now can only contain a single command (maybe adding literal \n at the end of the line would actually introduce a linebreak in the string, but it feels backward compared to the usual approach of automatice linebreak unless an explicit continuation is added).
You can use '^' to break your command line into multiple lines. Check below exmaple. Below script will output 'hello world' like a single line command echo 'hello world'
- script: |
echo ^
'hello ^
world'
I realized in bash, and perhaps in other shells too, multiline commands can be pasted within
{
# some bash command here
}
How does this work?
What is the bash language syntax or property coming into picture here?
I realize this has to be newline separated
For e.g.
$ { date }
> -bash: syntax error: unexpected end of file
$ {
> date
> }
Sat Aug 24 06:02:03 PDT 2019
{...} is one of several compound commands described in the man page. Neither { nor } is a reserved keyword; they are only treated specially when they appear in the command position of a simple command. For that reason, the last command in the group has to be properly terminated (either with a newline or a semicolon), so that } isn't treated simply as another argument of the last command.
$ { date; }
Sat Aug 24 06:02:03 PDT 2019
From the man page:
Compound Commands
A compound command is one of the following. In most cases a list in a
command's description may be separated from the rest of the command by
one or more newlines, and may be followed by a newline in place of a
semicolon.
[...]
{ list; }
list is simply executed in the current shell environment. list
must be terminated with a newline or semicolon. This is known
as a group command. The return status is the exit status of
list. Note that unlike the metacharacters ( and ), { and } are
reserved words and must occur where a reserved word is permitted
to be recognized. Since they do not cause a word break, they
must be separated from list by whitespace or another shell
metacharacter.
[...]
I'm using the CI Lint tester to try and figure out how to store an expected JSON result, which I later compare to a curl response. Neither of these work:
Attempt 1
---
image: ruby:2.1
script:
- EXPECT_SERVER_OUTPUT='{"message": "Hello World"}'
Fails with:
did not find expected key while parsing a block mapping at line 4 column 5
Attempt 2
---
image: ruby:2.1
script:
- EXPECT_SERVER_OUTPUT="{\"message\": \"Hello World\"}"
Fails with:
jobs:script config should be a hash
I've tried using various combinations of echo as well, without a working solution.
You could use literal block scalar1 style notation and put the variable definition and subsequent script lines on separate lines2 without worrying about quoting:
myjob:
script:
- |
EXPECT_SERVER_OUTPUT='{"message": "Hello World"}'
or you can escape the nested double quotes:
myjob:
script:
- "EXPECT_SERVER_OUTPUT='{\"message\": \"Hello World\"}'"
but you may also want to just use variables like:
myjob:
variables:
EXPECT_SERVER_OUTPUT: '{"message": "Hello World"}'
script:
- dothething.sh
Note: variables are by default expanded inside variable definitions so take care with any $ characters inside the variable value (they must be written as $$ to be literal). This feature can also be turned off.
1See this answer for an explanation of this and related notation
2See this section of the GitLab docs for more info on multi-line commands
I made it work like this:
script: |
"EXPECT_SERVER_OUTPUT='{\"message\": \"Hello World\"}'"
echo $EXPECT_SERVER_OUTPUT
I found a great answer for how to comment in bash script (by #sunny256):
#!/bin/bash
echo before comment
: <<'END'
bla bla
blurfl
END
echo after comment
The ' and ' around the END delimiter are important, otherwise things inside the block like for example $(command) will be parsed and executed.
This may be ugly, but it works and I'm keen to know what it means. Can anybody explain it simply? I did already find an explanation for : that it is no-op or true. But it does not make sense to me to call no-op or true anyway....
I'm afraid this explanation is less "simple" and more "thorough", but here we go.
The goal of a comment is to be text that is not interpreted or executed as code.
Originally, the UNIX shell did not have a comment syntax per se. It did, however, have the null command : (once an actual binary program on disk, /bin/:), which ignores its arguments and does nothing but indicate successful execution to the calling shell. Effectively, it's a synonym for true that looks like punctuation instead of a word, so you could put a line like this in your script:
: This is a comment
It's not quite a traditional comment; it's still an actual command that the shell executes. But since the command doesn't do anything, surely it's close enough: mission accomplished! Right?
The problem is that the line is still treated as a command beyond simply being run as one. Most importantly, lexical analysis - parameter substitution, word splitting, and such - still takes place on those destined-to-be-ignored arguments. Such processing means you run the risk of a syntax error in a "comment" crashing your whole script:
: Now let's see what happens next
echo "Hello, world!"
#=> hello.sh: line 1: unexpected EOF while looking for matching `''
That problem led to the introduction of a genuine comment syntax: the now-familiar # (which was first introduced in the C shell created at BSD). Everything from # to the end of the line is completely ignored by the shell, so you can put anything you like there without worrying about syntactic validity:
# Now let's see what happens next
echo "Hello, world!"
#=> Hello, world!
And that's How The Shell Got Its Comment Syntax.
However, you were looking for a multi-line (block) comment, of the sort introduced by /* (and terminated by */) in C or Java. Unfortunately, the shell simply does not have such a syntax. The normal way to comment out a block of consecutive lines - and the one I recommend - is simply to put a # in front of each one. But that is admittedly not a particularly "multi-line" approach.
Since the shell supports multi-line string-literals, you could just use : with such a string as an argument:
: 'So
this is all
a "comment"
'
But that has all the same problems as single-line :. You could also use backslashes at the end of each line to build a long command line with multiple arguments instead of one long string, but that's even more annoying than putting a # at the front, and more fragile since trailing whitespace breaks the line-continuation.
The solution you found uses what is called a here-document. The syntax some-command <<whatever causes the following lines of text - from the line immediately after the command, up to but not including the next line containing only the text whatever - to be read and fed as standard input to some-command. Here's an alternate shell implementation of "Hello, world" which takes advantage of this feature:
cat <<EOF
Hello, world
EOF
If you replace cat with our old friend :, you'll find that it ignores not only its arguments but also its input: you can feed whatever you want to it, and it will still do nothing (and still indicate that it did that nothing successfully).
However, the contents of a here-document do undergo string processing. So just as with the single-line : comment, the here-document version runs the risk of syntax errors inside what is not meant to be executable code:
#!/bin/sh -e
: <<EOF
(This is a backtick: `)
EOF
echo 'In modern shells, $(...) is preferred over backticks.'
#=> ./demo.sh: line 2: bad substitution: no closing "`" in `
The solution, as seen in the code you found, is to quote the end-of-document "sentinel" (the EOF or END or whatever) on the line introducing the here document (e.g. <<'EOF'). Doing this causes the entire body of the here-document to be treated as literal text - no parameter expansion or other processing occurs. Instead, the text is fed to the command unchanged, just as if it were being read from a file. So, other than a line consisting of nothing but the sentinel, the here-document can contain any characters at all:
#!/bin/sh -e
: <<'EOF'
(This is a backtick: `)
EOF
echo 'In modern shells, $(...) is preferred over backticks.'
#=> In modern shells, $(...) is preferred over backticks.
(It is worth noting that the way you quote the sentinel doesn't matter - you can use <<'EOF', <<E"OF", or even <<EO\F; all have the same result. This is different from the way here-documents work in some other languages, such as Perl and Ruby, where the content is treated differently depending on the way the sentinel is quoted.)
Notwithstanding any of the above, I strongly recommend that you instead just put a # at the front of each line you want to comment out. Any decent code editor will make that operation easy - even plain old vi - and the benefit is that nobody reading your code will have to spend energy figuring out what's going on with something that is, after all, intended to be documentation for their benefit.
It is called a Here Document. It is a code block that lets you send a list of commands to another command or program
The string following the << is the marker determining the end of the block. If you send commands to no-op, nothing happens, which is why you can use it as a comment block.
That's heredoc syntax. It's a way of defining multi-line string literals.
As the answer at your link explains, the single quotes around the END disables interpolation, similar to the way single-quoted strings disable interpolation in regular bash strings.
I'm trying to match the From line all the way to the end of the Subject line in the following:
....
From: XXXXXX
Date: Tue, 8 Mar 2011 10:52:42 -0800
To: XXXXXXX
Subject: XXXXXXX
....
So far I have:
/From:.*Date:.*To:.*Subject/m
But that doesn't match to the end of the subject line. I tried adding $ but that had no effect.
You can use the /m modifier to enable multiline mode (i.e. to allow . to match newlines), and you can use ? to perform non-greedy matching:
message = <<-MSG
Random Line 1
Random Line 2
From: person#example.com
Date: 01-01-2011
To: friend#example.com
Subject: This is the subject line
Random Line 3
Random Line 4
MSG
message.match(/(From:.*Subject.*?)\n/m)[1]
=> "From: person#example.com\nDate: 01-01-2011\nTo: friend#example.com\nSubject: This is the subject line"
See http://ruby-doc.org/core/Regexp.html and search for "multiline mode" and "greedy by default".
If you are using ruby, you can try :
Regexp.new("some reg", Regexp::MULTILINE)
If you are not using ruby, I suggest you hack this question:
replace all the "\n" with SOME_SPECIAL_TOKEN
search the regexp, and do other operations...
restore: replace SOME_SPECIAL_TOKEN with "\n"
If you want to match across linebreaks, one possibility is to first replace all newline characters with some other character (or character sequence) that wouldn't otherwise appear in the text. For example, if you have all of the text in one string variable you can do something like aString.split("\n").join("|") to replace all newlines in the string with pipe characters.
Also, look at Alan Moore's answer to your previous question regarding how to match the newline character in a regular expression.
Try:
/...^Subject:[^\n]*/m
Using the following data:
From: XXXXXX
Date: Tue, 8 Mar 2011 10:52:42 -0800
To: XXXXXXX
Subject: XXXXXXX
The following regex will do the magic:
From:([^\r\n]+)[\r\n]+Date:([^\r\n]+)[\r\n]+To:([^\r\n]+)[\r\n]+Subject:([^\r\n]+)[\r\n]+
But I would recommend that you don't try and do this in 1 regex. Push into a regex "^(\w+):(.+)$" line by line, unless you are sure that the sequence of the FROM/DATE/TO/SUBJECT is not going to change ;)