i've discovered that instead of unzipping a .tar.gz file, i am better off loading it:
- name: Load image
run: docker load -i pga.tar.gz
for some reason, it does not come with a name - so i need to extract the id...the only way i've seen that is possible:
- name: Get image id
run: imageid=$(docker images --format="{{.ID}}" | head -n 1); echo "$imageid"
what i get for that part looks good:
Run imageid=$(docker images --format="{{.ID}}" | head -n 1); echo "$imageid"
fc6e040841a1
but when i try to tag....
- name: Tag image
run:
docker tag $imageid
registry.digitalocean.com/abda-containers/abd_db:${{github.event.inputs.version }}
it gives me:
Run docker tag $imageid registry.digitalocean.com/abda-containers/abd_db:kk
docker tag $imageid registry.digitalocean.com/abda-containers/abd_db:kk
shell: /usr/bin/bash -e {0}
"docker tag" requires exactly 2 arguments.
See 'docker tag --help'.
Usage: docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]
Create a tag TARGET_IMAGE that refers to SOURCE_IMAGE
Error: Process completed with exit code 1.
not sure what to do..
Related
Say I have a number of jobs that all do similar series of scripts, but need a few variables that change between them:
test a:
stage: test
tags:
- a
interruptible: true
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t a -f Dockerfile.a .
test b:
stage: test
tags:
- b
interruptible: true
rules:
- if: $CI_PIPELINE_SOURCE == 'merge_request_event'
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t b -f Dockerfile.b .
All I need is to be able to define e.g.
- docker build -t ${WHICH} -f Dockerfile.${which} .
If only I could make an anchor like:
.x: &which_ref
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t $WHICH -f Dockerfile.$WHICH .
And include it there:
test a:
script:
- export WHICH=a
<<: *which_ref
This doesn't work and in a yaml validator I get errors like
Error: YAMLException: cannot merge mappings; the provided source object is unacceptable
I also tried making an anchor that contains some entries under script inside of it:
.x: &which_ref
script:
- echo "env is $(env)"
- echo etcetera
- echo and so on
- docker build -t $WHICH -f Dockerfile.$WHICH .
This means I have to include it from one step higher up. So this does not error, but all this accomplishes is cause the later declared script section to override the first one.
So I'm losing hope. It seems like I will just need to abstract the sections away into their own shell scripts and call them with arguments or whatever.
The YAML merge key << is a non-standard extension for YAML 1.1, which has been superseded by YAML 1.2 about 14 years ago. Usage is discouraged.
The merge key works on mappings, not on sequences. It cannot deep-merge. Thus what you want to do is not possible to implement with it.
Generally, YAML isn't designed to process data, it just loads it. The merge key is an outlier and didn't find its way into the standard for good reasons. You need a pre- or postprocessor to do complex processing, and Gitlab CI doesn't offer anything besides simple variable expension, so you're out of luck.
I have written custom golang kong plugin called go-wait following the example from the github repo https://github.com/redhwannacef/youtube-tutorials/tree/main/kong-gateway-custom-plugin
The only difference is I created a custom docker image so kong would have the mentioned plugin by default in it's /usr/local/bin directory
Here's the dockerfile
FROM golang:1.18.3-alpine as pluginbuild
COPY ./charts/custom-plugins/ /app/custom-plugins
RUN cd /app/custom-plugins && \
for d in ./*/ ; do (cd "$d" && go mod tidy && GOOS=linux GOARCH=amd64 go build .); done
RUN mkdir /app/all-plugin-execs && cd /app/custom-plugins && \
find . -type f -not -name "*.*" | xargs -i cp {} /app/all-plugin-execs/
FROM kong:2.8
COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/
COPY --from=pluginbuild /app/all-plugin-execs/ /usr/local/bin/plugin-ref/
# Loop through the plugin-ref directory and create an entry for all of them in
# both KONG_PLUGINS and KONG_PLUGINSERVER_NAMES env vars respectively
# Additionally append `bundled` to KONG_PLUGINS list as without it any unused plugin will case Kong to error out
#### Example Env vars for a plugin named `go-wait`
# ENV KONG_PLUGINS=go-wait
# ENV KONG_PLUGINSERVER_NAMES=go-wait
# ENV KONG_PLUGINSERVER_GO_WAIT_QUERY_CMD="/usr/local/bin/go-wait -dump"
####
RUN cd /usr/local/bin/plugin-ref/ && \
PLUGINS=$(ls | tr '\n' ',') && PLUGINS=${PLUGINS::-1} && \
echo -e "KONG_PLUGINS=bundled,$PLUGINS\nKONG_PLUGINSERVER_NAMES=$PLUGINS" >> ~/.bashrc
# Loop through the plugin-ref directory and create an entry for QUERY_CMD entries needed to load the plugin
# format KONG_PLUGINSERVER_EG_PLUGIN_QUERY_CMD if the plugin name is `eg-plugin` and it should point to the
# plugin followed by `-dump` argument
RUN cd /usr/local/bin/plugin-ref/ && \
for f in *; do echo "$f" | tr "[:lower:]" "[:upper:]" | tr '-' '_' | \
xargs -I {} sh -c "echo 'KONG_PLUGINSERVER_{}_QUERY_CMD=' && echo '\"/usr/local/bin/{} -dump\"' | tr [:upper:] [:lower:] | tr '_' '-'" | \
sed -e '$!N;s/\n//' | xargs -i echo "{}" >> ~/.bashrc; done
This works fine in the docker-compose file and docker container. But when I tried to use the same image in the kubernetes environment along with kong-ingress-controller, I started running into errors "failed to fill-in defaults for plugin: go-wait" and/or a bunch of other errors including "plugin 'go-wait' enabled but not installed" in the logs and I ended up not being able to enable it.
Has anyone tried including go plugins in their kubernetes/helm kong setup. If so please shed some light on this
Update: Found the answer I was looking for, along with setting the environment variables generated by the image, there's modifications in the _helpers.tpl file of the kong helm chart itself.
The reason is that in the deployment charts, the configuration expects plugins to be configured in values-custom.yml used to override the default settings.
But the helm chart seems to be specific to values and plugins being loaded via configMaps which turned out to be a huge bottleneck, as any binary plugin you will generate in golang for kong is going to exceed the maximum allowed limit of the configMaps in kubernetes.
That's the whole reason I had set out on this endeavor to make the plugins part of my image.
TL;dr
I was able to clone the repo to my local system, make the changes for the following patch for loading the plugins from values without having to club them with the lua plugins. (Credits : Answer of thatbenguy from the discussion https://discuss.konghq.com/t/how-to-load-go-plugins-using-kong-helm-chart/5717/10)
--- a/charts/kong/templates/_helpers.tpl
+++ b/charts/kong/templates/_helpers.tpl
## -530,6 +530,9 ## The name of the service used for the ingress controller's validation webhook
{{- define "kong.plugins" -}}
{{ $myList := list "bundled" }}
+{{- range .Values.plugins.goPlugins -}}
+{{- $myList = append $myList .pluginName -}}
+{{- end -}}
{{- range .Values.plugins.configMaps -}}
{{- $myList = append $myList .pluginName -}}
{{- end -}}
Add the following block to my values-custom.yml and I was good to go.
Hopefully this helps anyone else also trying to write custom plugins for kong in golang for use in helm charts.
env:
database: "off"
plugins: bundled,go-wait
pluginserver_names: go-wait
pluginserver_go_wait_query_cmd: "/usr/local/bin/go-wait -dump"
plugins:
goPlugins:
- pluginName: "go-wait"
NOTE : Please remember all this still depends on you having the prebuilt custom kong plugins in your image, in my case I had built an image from the above dockerfile contents (in question) and pushed that to my own docker hub repo and replaced the image in the values-custom.yml using the following block
image:
repository: chalukyaj/kong-custom-image
tag: "1.0.1"
PS: As you guys might have noticed, the only disappointment I have with this is that the environment variables couldn't just be picked from the docker image's ~/.bashrc, which would have made this awesome. But nonetheless, this works, and I couldn't find a single post which showed how to use the new go-pdk (instead of the older go-pluginserver) to build the go plugins and use them in helm.
I am trying to deploy code from GitLab to the EC2 instance. However, I am getting the following errors when I run the pipeline
/home/gitlab-runner/.ssh/config: line 1: Bad configuration option: \342\200\234host
/home/gitlab-runner/.ssh/config: terminating, 1 bad configuration options
Here is my .gitlab-ci.yml file that I am using.
stages:
- QAenv
- Prod
Deploy to Staging:
stage: QAenv
tags:
- QA
before_script:
# Generates to connect to the AWS unit the SSH key.
- mkdir -p ~/.ssh
- echo -e “$SSH_PRIVATE_KEY” > ~/.ssh/id_rsa
# Sets the permission to 600 to prevent a problem with AWS
# that it’s too unprotected.
- chmod 600 ~/.ssh/id_rsa
- 'echo -e “Host *\n\tStrictHostKeyChecking no\n\n” > ~/.ssh/config'
script:
- bash ./gitlab-deploy/.gitlab-deploy.staging.sh
environment:
name: QAenv
# Exposes a button that when clicked take you to the defined URL:
url: https://your.url.com
Below is my .gitlab-deploy.staging.sh file that I have set up to deploy to my server.
# !/bin/bash
# Get servers list:
set — f
# Variables from GitLab server:
# Note: They can’t have spaces!!
string=$DEPLOY_SERVER
array=(${string//,/ })
for i in "${!array[#]}"; do
echo "Deploy project on server ${array[i]}"
ssh ubuntu#${array[i]} "cd /opt/bau && git pull origin master"
done
I checked my .ssh/config file contents and below is what I can see.
ubuntu#:/home/gitlab-runner/.ssh$ cat config
“Host *ntStrictHostKeyChecking nonn”
Any ideas about what I am doing wrong and what changes I should make?
The problem is with.
ubuntu#ip-172-31-42-114:/home/gitlab-runner/.ssh$ cat config
“Host *ntStrictHostKeyChecking nonn”
Because there are some Unicode characters here, which usually comes when we copy paste code from a document or a webpage.
In your case this “ char specifically you can see in the output as well.
replace that with " and check for others in your config and update should work.
There are more details in this question getting errors stray ‘\342’ and ‘\200’ and ‘\214’
enviornment: 'dev'
acr-login: $(enviornment)-acr-login
acr-secret: $(enviornment)-acr-secret
dev-acr-login and dev-acr-secret are secrets stored in keyvault for acr login and acr secret.
In Pipeline, getting secrets with this task
- task: AzureKeyVault#1
inputs:
azureSubscription: $(connection)
KeyVaultName: $(keyVaultName)
SecretsFilter: '*'
This task will create task variables with name 'dev-acr-login' and 'dev-acr-secret'
Not if I want to login in docker I am not able to do that
Following code works and I am able to login into acr.
- bash: |
echo $(dev-acr-secret) | docker login \
$(acrName) \
-u $(dev-acr-login) \
--password-stdin
displayName: 'docker login'
Following doesnot work. Is there a way that I can use variable names $(acr-login) and $(acr-secret) rather than actual keys from keyvault?
- bash: |
echo $(echo $(acr-secret)) | docker login \
$(acrRegistryServerFullName) \
-u $(echo $(acr-login)) \
--password-stdin
displayName: 'docker login'
You could pass them as environment variables:
- bash: |
echo $(echo $ACR_SECRET) | ...
displayName: docker login
env:
ACR_SECRET: $(acr-secret)
But what is the purpose, as opposed to just echoing the password values as you said works in the other example? As long as the task is creating secure variables, they will be protected in logs. You'd need to do that anyway, since they would otherwise show up in diagnostic logs if someone enabled diagnostics, which anyone can do.
An example to do that:
- bash: |
echo "##vso[task.setvariable variable=acr-login;issecret=true;]$ACR_SECRET"
env:
ACR_SECRET: $($(acr-secret)) # Should expand recursively
See Define variables : Set secret variables for more information and examples.
I have an Nx workspace with multiple Angular apps included. When master is updated in my GitHub repo, I want a build to kick off. That part is easy enough with GCB's triggers. But what I want to happen is to run this command:
npm run affected:apps
on the trigger, and build a Docker image and push it to Google Container registry for each affected app. My cloudbuild.yaml file looks like this so far:
steps:
- name: 'gcr.io/cloud-builders/git'
args: ['fetch', '--unshallow']
- name: node:10.15.1
entrypoint: npm
args: ['run affected:apps --base=origin/master --head=HEAD']
That command returns a result like this:
> project-name#0.0.0 affected:apps /Users/username/projects/project-folder
> nx affected:apps
Note: Nx defaulted to --base=master --head=HEAD
my-app-1
I'm not sure what to do with Google Cloud with that result. With a node script, I could do the following to print out an array of affected apps:
const { exec } = require('child_process');
function getApps() {
exec('npm run affected:apps', (err, out) => {
if (err) {
console.log(null);
} else {
const lines = out.split('\n');
const apps = lines[lines.length - 2].split(' ');
console.log(JSON.stringify(apps));
}
});
}
getApps();
That returns an array of the affected apps, and null if an error. Still, even with that, I'm not sure what I would do for the next step in Google Cloud build. With the results of either the command or that script, ideally I'd be able to run a docker build command, like this:
docker build --file ./:loop variable:/Dockerfile
where :loop variable: is the name of an affected app. I'd like to do that for each value in the array, and not do anything if, for some reason, the command returns no affected apps.
Any ideas on how to use Google Cloud Build with Nx Workspaces, or if you've just got Google Cloud build experience and know what my next step should be, that'd be great.
Continue #chinoche comment there is an example of how you could save the list of affected apps to the affected.txt file
- name: 'gcr.io/cloud-builders/npm'
entrypoint: 'bash'
args:
- '-c'
- |
IFS=' ' read -a apps <<< $(npx nx affected:apps --base=origin/master~1 --plain)
for app in "${apps[#]}"
do
echo $app
done >> affected.txt
The next step could read the file and call any other commands, e.g. create docker image
- name: 'gcr.io/cloud-builders/docker'
entrypoint: 'bash'
args:
- '-c'
- |
while IFS= read -r project; do
docker build -t gcr.io/$PROJECT_ID/$project -f <path-to>/Dockerfile .
done < affected.txt
One of the tricks might be to create separate cloudbuild.yaml file for each project and then trigger a new cloud build process for each affected project. That allows having a completely different build process for each project.
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args:
- '-c'
- |
while IFS= read -r project; do
gcloud builds submit --config ./<path-to>/$project/project-specific-cloudbuild.yaml .
done < affected.txt
if you are able to get the affected apps with the node script I'd suggest you to write a file with the affected apps in a Cloud Build custom steps, this file will be written at the "/workspace" directory and will be able to any other custom step that may be executed in later steps, with this you should be able to run the docker build command