sed -i "s|{{PLACEHOLDER}}|${KEY_B64}|g" <path>
The command above is executed in a Gitlab CI runner and throws the following:
sed: unmatched '|'
I have double-checked the KEY_B64 environment variable, it is set and looks valid.
This variable is a base-64 encoded JWT token (Kubernetes secrets expect to be base-64 encoded.
What is really strange though is that this command works fine if I run it locally (Ubuntu 22.04) and replace the env variable with the output from echo -n <JWT_TOKEN> | base64.
Based on the error message, it seems that the env value might contain the delimiter, but changing it to anything else doesn't solve the problem. On top of that, the encoded value for sure doesn't include such symbols.
What could be the cause of the issue?
Updates:
Running sed --version outputs:
$ sed --version
This is not GNU sed version 4.0
Looking with the set -x option on, I can see that the encoded string includes newlines (outputted the variable in the pipeline logs).
Using printf %s $VAR did not solve the issue
Surprisingly, base64 doesn't support -w0
Using set -x revealed that for some reason the base64 command added line breaks to the output.
As base64 -w0 is not supported, I had to use the command below to remove the newlines in the base64 output, which solved the problem.
export MY_KEY_B64=$(echo -n $MY_KEY | base64 | tr -d \\n)
Note: as I was fairly told in the comments, using printf is preferable but in this case, it was not the cause of the issue so I did not modify this command to emphasize what cause the problem.
Related
I tried to update this line in pubspec.yaml on macOS
version: 0.2.6-alpha+26
#!/bin/bash
version=$(grep -oE '(?\<=version: )\[^ \]+' pubspec.yaml)
version=$(echo $version | sed 's/(\[0-9\].\[0-9\].\[0-9\])-alpha+(\[0-9\])/\\1-alpha+\\1/')
sed -i '' "s/version:.\*/version: $version/" pubspec.yaml
This script is intended to increment the patch number of the version number in the file pubspec.yaml. However, the script contains errors that cause it to throw the following error message:
grep: repetition-operator operand invalid
sed: 1: "s/([0-9].[0-9].[0-9])-a ...": \1 not defined in the RE
I'd like it to simply work
It looks like your grep statement derives from this bump_version.sh.
But the macOS grep does not recognize the GNU-grep -P option.
(Aside: The ?<= syntax comes from Perl regular expressions. See "Lookbehind assertions" here.)
But you can drop the regex gymnastics by translating dots & dashes into spaces, which are easily digested by the Bash "read" command:
#!/usr/bin/env bash
grep version pubspec.yaml | (
IFS=' .-' read label major minor patch build;
sed -i '' "s/version:.*/version: $major.$minor.$((patch+1))-$build/" pubspec.yaml
)
Notes:
This script favors piped output over variables & redirection.
I didn't know that IFS can take multiple characters & applied C.Duffy's tip (to omit tr)!
Reusing variables through Process Substitution is left as an exercise for the reader.
I have created a build pipeline where I need to pull the Images from private ACR repo.
The version to be pulled is stored in a file which I read using a regex expression and sed command of Linux
- bash :
MONGO=$(sed -n -e '/Mongo_Service_Version/ s/.*\= *//p' $(Build.SourcesDirectory)/deployment/.env |
sed 's/ *$//g')
echo $MONGO
The above prints the correct version that is present in the .env file (Eg: 4.0)
But now when I use the same variable to pull the image from Private repo I get an error of invalid reference
- bash :
MONGO=$(sed -n -e '/Mongo_Service_Version/ s/.*\= *//p' $(Build.SourcesDirectory)/deployment/.env |
sed 's/ *$//g')
echo $MONGO
docker pull my.repo/mongo:$MONGO
An interesting point to note is that if I give a hardcoded value to the MONGO variable like 4.0 the above thing works. Also, echo command writes the correct output for the variable but I am not sure why I am not able to use it with docker pull command.
I tried docker pull command in different syntax as mentioned below, but I get the same error
docker pull klingelnberg.azurecr.io/gearengine/mongo:${MONGO}
docker pull "klingelnberg.azurecr.io/gearengine/mongo:$MONGO"
MONGO_VERSION=$(sed -n -e '/Mongo_Service_Version/ s/.*\= *//p' $(Build.SourcesDirectory)/deployment/.env | sed 's/ *$//g')
MONGO="my.repo/mongo:$MONGO_VERSION"
docker pull $MONGO
What I think is the variable value my be having some escape characters or something that does not get printed but used when pulling the image. (Maybe color code values or as such)
The above is just my guess.
I have a docker-compose file, in which I have a docker images with its versioning tag, example myrepo.com/myimage:1.0.1.0. I want to write a bash script that receives the version number as a parameter and updates the version tag of the relevant image.
Currently I am using the below command to replace the version tag with the help of grep and sed, but I believe there should be a more efficient way for this. Any suggestions?
sedval='1.1.1.1' # <-- new value
imageval=$(grep -i 'image: myrepo.com/myimage:' docker-compose.yml | cut -d ':' -f2,3)
imagename=$(grep -i 'image: myrepo.com/myimage:' docker-compose.yml | cut -d ':' -f2)
imageversion=$(grep -i 'image: myrepo.com/myimage:' docker-compose.yml | cut -d ':' -f3)
sed -i -e "s+$imageval+$imagename:$sedval+g" docker-compose.yml
docker-compose supports variable substitution. You could do the following:
version: "3"
services:
myrepo:
image: myrepo.com/myimage:${VERSION}
docker-compose looks for the VERSION environment variable and substitutes its value in the file. Run with:
export VERSION=1.1.1.1; docker-compose up
Manipulating YAML with line-oriented regex tools is generally not advisable. You should probably switch to a YAML-aware tool like yq for this.
There doesn't seem to be a reason you extract imageversion; you assign this variable, but never use it. In fact, if you forgo the requirement to match case-insensitively, all you really need is
sed -i "s+\\(image: myrepo\\.com/myimage:\\)[.0-9]*+\\1$1+" docker-compose.yml
assuming (like you say in your question) the new version number is in $1, which is the first command-line argument to your script.
I don't think Docker allows you to use upper case; and if it does, you can replace i with [Ii], m with [Mm], etc. Notice also how the literal dot needs to be backslash-escaped, strictly speaking.
Generally speaking, you can often replace grep 'x' | sed 'y' with sed '/x/y'; see also useless use of grep. Also, the repeated grep could easily be refacored into a single grep with something like
imageval=$(grep -i 'image: myrepo.com/myimage:' docker-compose.yml)
imagever=${imageval##*:}
imagename=${imageval%:$imagever}
where the shell parameter substitutions are both simpler and more elegant as well as more efficient than repeatedly spawning cut in a separate process for this very simple string extraction task.
I'm getting the size of a file from a remote webserver and saving the results to a var called remote I get this using:
remote=`curl -sI $FILE | grep -i Length | awk '/Content/{print $(NF-0)}'`
Once I've downloaded the file I'm getting the local files size with:
local=`stat --print="%s" $file`
If I echo remote and local they contain the same value.
I'm trying to run an if statement for this
if [ "$local" -ne "$remote" ]; then
But it always shows the error message, and never advises they match.
Can someone advise what I'm doing wrong.
Thanks
curl's output uses the network format for text, meaning that lines are terminated by a carriage return followed by linefeed; unix tools (like the shell) expect lines to end with just linefeed, so they treat the CR as part of the content of the line, and often get confused. In this case, what's happening is that the remote variable is getting the content length and a CR, which isn't valid in a numeric expression, hence errors. There are many ways to strip the CR, but in this case it's probably easiest to have awk do it along with the field extraction:
remote=$(curl -sI "$remotefile" | grep -i Length | awk '/Content/{sub("\r","",$NF); print $NF}')
BTW, I also took the liberty of replacing backticks with $( ) -- this is easier to read, and doesn't have some oddities with escapes that backticks have, so it's the preferred syntax for capturing command output. Oh, and (NF-0) is equivalent to just NF, so I simplified that. As #Jason pointed out in a comment, it's safest to use lower- or mixed-case for variable names, and put double-quotes around references to them, so I did that by changing $FILE to "$remotefile". You should do the same with the local filename variable.
You could also drop the grep command and have awk search for /^Content-Length:/ to simplify it even further.
If I base64 encode a string which consists of seven characters e.g. abcdefg with the website https://www.base64encode.org/ the result is YWJjZGVmZw==. The trailing "==" characters are padding because the number of input characters cannot be divided by 7.
I've to reproduce this result in bash. So I've tried the following command:
echo "abcdefg" | base64
However, the result is different now:
YWJjZGVmZwo=
I'm using Ubuntu where base64 (GNU coreutils) 8.25 is installed.
I would be glad if someone could give me a hint.
I've just noticed that the reason for the described behaviour is the newline which echo writes at the end. So the correct command is the following which suppress the newline
echo -n "abcdefg" | base64
Then the output is like I expect it:
YWJjZGVmZw==
It is also tricky how a here-string will produce unexpected output. It is probably missing the null character \0.
$ base64 <<<"abcdefg"
YWJjZGVmZwo=
$ printf 'abcdefg' | base64
YWJjZGVmZw==