I've made a mistake in the file below, but I cannot see where my mistake is. I have this command in my .gitlab-ci.yml configuration file.
- sed "s/use_scm_version=True/use_scm_version={'write_to': '..\/version.txt', 'root': '..'},\/"setup.py
It seems that the ":" are interpreted as a semicolon even if I surround the entire sed between double quotes.
(<unknown>): did not find expected key while parsing a block mapping at line 109 column 11
Any ideas ?
Since your double quotes are not at the beginning of the scalar node, they don't have special meaning in YAML and the colon is seen as the normal value indicator (and both the key and value have an embedded double quote).
I recommend you quote the whole scalar:
- "sed s/use_scm_version=True/use_scm_version={'write_to': '..\/version.txt', 'root': '..'},\/setup.py"
And optionally add \" (backslash escaped double quotes) as necessary within there if that doesn't work.
Related
I have the following code:
script_list=$'1 a\n2\n3'
old_IFS="${IFS}"
IFS=$'\n'
for line in "${script_list}"; do
echo "__${line}__";
done
IFS="${old_IFS}"
which shall produce:
__1 a__
__2__
__3__
However, instead it gives me:
__1 a
2
3__
I double quoted "${script_list}" because it can contain spaces (see: https://stackoverflow.com/a/10067297). But I believe this is also where the problem lies, because when I remove the double quotes around it, it works as expected.
What am I missing?
edit:
As Cyrus suggested, I ran the code at ShellCheck and it tells me:
$ shellcheck myscript
Line 5:
for line in "${script_list}"; do
^-- SC2066: Since you double quoted this, it will not word split, and the loop will only run once.
Is it safe to simply remove the double quotes or do I need to be careful with that?
I double quoted "${script_list}" because it can contain spaces
This is only done to prevent the shell from splitting the string at spaces. However you explicitly tell the shell (by setting IFS) that your IFS is now a newline, not a space. The shell would split by default here on newlines, not on spaces, unless you quote it. Hence, remove the quotes.
this is my first post here (maybe more later ^^)
So, here we are :
I'm using Gitlab-CI, docker, makefile, firebase.
So in my Makefile :
firebase-deploy: ## Deploy the application
$(FIREBASE) deploy -m "$(comment)"
in my CI :
variables:
DOCKER_DRIVER: overlay2
MESSAGE: $CI_COMMIT_MESSAGE
deploy-firebase:
stage: deploy
script:
- make install
- make deploy comment="$MESSAGE"
The problem is actually, i have
docker-compose exec -T js node_modules/firebase-tools/lib/bin/firebase.js deploy -m "test ci ?
/bin/sh: syntax error: unterminated quoted string
So, as you can see, the double quote at the end is ... not here.
I tested a few things
\"Comment\"
\'Comment\'
"Comment\"
And more, but none of them work.
One thing interesting is that if i commit FROM gitlab.com it work as expected, but from my computer on push : nop.
Do you have any advice, or something to look for ?
Thanks!
If your commit message contains end of lines you cannot pass it to your script as it is. Try, maybe, to substitute end of lines by another character, e.g. -. Of course the way to do it depends on the shell that executes your CI scripts. If it is bash, for instance, the following should work:
make deploy comment="${MESSAGE//$'\n'/-}"
If you must also escape other characters you can chain substitutions. Example if you want to substitute end of lines and double quotes by - and ':
c="${MESSAGE//$'\n'/-}"; c="${c//\"/\'}"; make deploy comment="$c"
To learn more about these bash substitutions just type man bash and read section EXPANSION, sub-section Parameter Expansion. The parameter expansion I used is:
${parameter/pattern/string}
that replaces pattern by string in the value of variable parameter (in bash parlance parameter is another name for variable). When the pattern begins with a / all matches of pattern are replaced with string, else only the first match is replaced. So the first parameter expansion:
c="${MESSAGE//$'\n'/-}"
substitutes all end-of-line characters in the value of variable MESSAGE by - (more on $'\n' later) and assigns the result to variable c. Note the enclosing double quotes, they are needed because you can have spaces in the message. And the second:
c="${c//\"/\'}"
substitutes all " in the value of variable c by ' and assigns again the result to variable c.
In this case the most complicated is to put a end-of-line character in the first pattern, a double quote in the second and a single quotes in the second string. The $'string' construct is explained in the QUOTING section of the bash manual:
Words of the form $'string' are treated specially. The word expands to
string, with backslash-escaped characters replaced as specified by the
ANSI C standard.
So $'\n' is replaced by the end-of-line character. In the second pattern and string you just need to escape the double and single quotes with a backslash to have them interpreted literally.
Finally the 3 commands of this little shell script are chained by ;, the bash sequence operator. They are executed one after the other in the same bash invocation.
This question already has answers here:
Quotes in ansible lineinfile
(4 answers)
Closed 4 years ago.
ERROR! Syntax Error while loading YAML.
did not find expected key
The error appears to have been in '/etc/ansible/main.yml': line 73, column 50, but may
be elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:
- name: Ensure IP forwarding is disabled
shell: "sysctl net.ipv4.ip_forward ; grep "net\.ipv4\.ip_forward" /etc/sysctl.conf /etc/sysctl.d/*"
^ here
We could be wrong, but this one looks like it might be an issue with
unbalanced quotes. If starting a value with a quote, make sure the
line ends with the same set of quotes. For instance this arbitrary
example:
I using grep with "" into shell module in ansible playbook and got an ERROR message as follow "ERROR! Syntax Error while loading YAML."
- name: Ensure IP forwarding is disabled
shell: "sysctl net.ipv4.ip_forward ; grep "net\.ipv4\.ip_forward" /etc/sysctl.conf /etc/sysctl.d/*"
register: CIS_3.1.1
ignore_errors: True
That configuration file for ansible is in the YAML format and in YAML a scalar that represents a string can be in multiple formats:
plain: no quotes, has restrictions on the start character and internal character sequences, no escapes
single quoted: can contain double quotes, no escapes except for repeating single quotes
double quoted: backslash escapes in string, in string double quotes need to be escaped
literal: newlines are preserved, no escapes
folded: newlines are converted to spaces, no escapes
You are using double quoted style, and in that you would need to escape the internal double quotes (") and backslashes (\). That gets ugly and unreadble real soon. It is much more useful to use literal style in such cases:
- name: Ensure IP forwarding is disabled
shell: |-
sysctl net.ipv4.ip_forward ; grep "net\.ipv4\.ip_forward" /etc/sysctl.conf /etc/sysctl.d/*
I.e. you put |- (the minus is to strip the final newline of the following line), then put the line without starting or end quotes, indented, on the next line.
As per this question How to input a path with a white space? I have declared a directory path like that:
startup='/cygdrive/c/Users/Me/AppData/Roaming/Microsoft/Windows/Start Menu/Programs/Startup'; I tried to wrap the path in to double quotes but it is not working either.
But for some reason when I am typing $startup I am getting an error:
$ $startup
bash: /cygdrive/c/Users/Alex/AppData/Roaming/Microsoft/Windows/Start: No such file or directory
How would u fix that?
Surround your variable with quotes:
$ "$startup"
You should always quote variables to be safe. You can refer to this page for more details.
As it states in the first paragraph: "When referencing a variable, it is generally advisable to enclose its name in double quotes. This prevents reinterpretation of all special characters within the quoted string -- except $, ` (backquote), and \ (escape)."
You can also refer to this page which states: The basic rule of thumb is that you should double-quote every expansion. This prevents unwanted word splitting and globbing. When in doubt, quote it.
I am trying to execute script with commands:
sed -i "USER/c\export USER=${signumid}" .bashrc
sed -i "DEVENVHOME=$/c\export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x}" .bashrc
I want to replace the line with string "USER" in .bashrc with export USER=${signumid} where $signumid variable is being provided through Cygwin prompt. Similarly I want to replace line with string DEVENVHOME=$ with export DEVENVHOME=${DEVENVHOME:-/home/${signumid}/CPM_WORKAREA/devenv.x} in .bashrc file, where $signumid variable is provided through Cygwin prompt.
But I am getting following errors on Cygwin termminal.:
sed: -e expression #1, char 1: unknown command: `U'
sed: -e expression #1, char 3: extra characters after command
The general syntax of a sed script is a sequence of address command arguments statements (separated by newline or semicolon). The most common command is the s substitution command, with an empty address, so we can perhaps assume that that is what you want to use here. You seem to be attempting to interpolate a shell variable $signumid which adds a bit of a complication to this exposition.
If your strings were simply static text, it would make sense to use single quotes; then, the shell does not change the text within the quotes at all. The general syntax of the s command is s/regex/replacement/ where the slash as the argument separator is just a placeholder, as we shall soon see.
sed -i 's/.*USER.*/export USER=you/
s% DEVENVHOME=\$%export DEVENVHOME=${DEVENVHOME:-/home/you/CPM_WORKAREA/devenv.x}%' .bashrc
This will find any line with USER and substitute the entire line with export USER=you; then find any line which contains DEVENVHOME=$ (with a space before, and a literal dollar character) and replace the matched expression with the long string. Because the substitution string uses slashes internally, we use a different regex separator % -- alternatively, we could backslash-escape the slashes which are not separators, but as we shall see, that quickly becomes untenable when we add the following twist. Because the dollar sign has significance as the "end of line" metacharacter in regular expressions, we backslash-escape it.
I have ignored the c\ in your attempt on the assumption that it is simply a misunderstanding of sed syntax. If it is significant, what do you hope to accomplish with it? c\export is not a valid Bash command, so you probably mean something else, but I cannot guess what.
Now, to interpolate the value of the shell variable signumid into the replacement, we cannot use single quotes, because those inhibit interpolation. You have correctly attempted to use double quotes instead (in your edited question), but that means we have to make some additional changes. Inside double quotes, backslashes are processed by the shell, so we need to double all backslashes, or find alternative constructs. Fortunately for us, the only backslash is in \$ which can equivalently be expressed as [$], so let's switch to that notation instead. Also, where a literal dollar sign is wanted in the replacement string, we backslash-escape it in order to prevent the shell from processing it.
sed -i "s/.*USER.*/export USER=$signumid/
s% DEVENVHOME=[$]%export DEVENVHOME=\${DEVENVHOME:-/home/$signumid/CPM_WORKAREA/devenv.x}%" .bashrc
Equivalenty, you could use single quotes around the parts of the script which are meant to be untouched by the shell, and then put an adjacent double-quoted string around the parts which need interpolation, like
'un$touched*by$(the!shell)'"$signumid"'more$[complex]!stuff'
This final script still rests on a number of lucky or perhaps rather unlucky guesses about what you actually want. On the first line, I have changed just USER to a regular expression which matches the entire line -- maybe that's not what you want? On the other hand, the second line makes the opposite assumption, just so you can see the variations -- it only replaces the actual text we matched. Probably one or the other needs to be changed.
Finally, notice how the two separate sed commands have been conflated into a single script. Many newcomers do not realize that sed is a scripting language which accepts an arbitrary number of commands in a script, and simply treat it as a "replace" program with a funny syntax.
Another common source of confusion is the evaluation order. The shell processes the double-quoted string even before sed starts to execute, so if you have mistakes in the quoting, you can easily produce syntax errors in the sed script which lead to rather uninformative error messages (because what sed tells you in the error message is based on what the script looks like after the shell's substutions). For example, if signumid contains slashes, it will produce syntax errors, because sed will see those as terminating separators for the s/// command. An easy workaround is to switch to a separator which does not occur in the value of signumid.