I'm using gitlab runner on a mac mini server.
While using user named "runner" I manage to use this command:
gsutil ls -l gs://tests/ |grep staging | sort -k 2 | tail -n 3| head -n 2 |awk '{print $3}' | gsutil -m cp -I .
I manage to get the files, but while using the same command in gitlab-ci.yml like this:
stages:
- test
test:
stage: test
when: always
script:
- gsutil ls -l gs://tests/ |grep staging | sort -k 2 | tail -n 3| head -n 2 |awk '{print $3}' | gsutil -m cp -I .
I get the error:
bash: line 141: gsutil: command not found
Also I checked and gitlab runner is using the same user I used.
The gitlab runner is configured with shell executor.
Changing the command to hold the full path of gsutil didn't help either.
I added whoami to the gitlab-ci.yml and got the result of the same user "runner"
I managed to solve this issue by using this solution:
gcloud-command-not-found-while-installing-google-cloud-sdk
I included this 2 line into my gitlab-ci.yml before using the gsutil command.
source '[path-to-my-home]/google-cloud-sdk/path.bash.inc'
source '[path-to-my-home]/google-cloud-sdk/completion.bash.inc'
Related
Introduction
Currently, I'm writing a customized GitHub workflow inside it I use a curl and grep command.
GitHub repo
action.yml
- name: get new tag
id: get_new_tag
shell: bash
run: |
temp_result=$(curl -s https://api.github.com/repos/${{inputs.github_repository}}/tags | grep -h "name" | grep -h "${{inputs.selector}}" | head -1 | grep -ho "${{inputs.regex}}")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
Full code here
test for action.yml
uses: ./
with:
files: tests/test1.txt tests/test2.txt
github_repository: MathieuSoysal/file-updater-for-release
prefix: MathieuSoysal/file-updater-for-release#
Full code are is here
Problem
I don't understand why but my GitHub Actions don't work with MacOS.
The given error:
temp_result=$(curl -s https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags | grep -h "name" | grep -h "" | head -1 | grep -ho "v\?[0-9.]*")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
shell: /bin/bash --noprofile --norc -e -o pipefail {0}
Error: Process completed with exit code 1.
Full log error is here
Question
Does someone know how we can fix this issue?
Alternatively, you can simply use the jq command line utility for this which is already installed and available on the GitHub runners. See the preinstalled software.
$ export URL='https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags'
$ curl -s $URL | jq -r .[0].name
v1.0.3
The issue is from this command grep -h "", this command is not supported on MacOS.
The empty string is not supported, the solution is to add something inside it.
I want to set up a target which downloads the latest s3 file containing _id_config within a path. So I know I can get the name of file I am interested in by
FILE=$(shell aws s3 ls s3:blah//xyz/mno/here --recursive | sort | tail -n 2 | awk '{print $4}' | grep id_config)
Now, I want to download the file to local with something like
download_stuff:
aws s3 cp s3://prod_an.live.data/$FILE .
But when I run this, my $FILE has some extra stuff like
aws s3 cp s3://blah/2022-02-17 16:02:21 2098880 blah//xyz/mno/here54fa8c68e41_id_config.json .
Unknown options: 2098880,blah/xyz/mno/here54fa8c68e41_id_config.json,.
Please can someone help me understand why 2098880 and the spaces are there in the output and how to resolve this. Thank you in advance.
Suggesting a trick with ls options -1 and -t to get the latest files in a folder:
FILE=$(shell aws s3 ls -1t s3:blah//xyz/mno/here |head -n 2 | grep id_config)
Running a nx affected:apps command gives me this output:
> NX NOTE Affected criteria defaulted to --base=master --head=HEAD
> NX Affected apps:
- app-backend
- app-frontend
- app-something
- app-anything
I need to get all the application names and use them again for a command call.
So I started with that
output=$(nx affected:apps)
echo "$output" | grep -E "^\W+app-(\w+)"
This gives me
- app-backend
- app-frontend
- app-something
- app-anything
But I need to get the names only instead to run foo --name={appname} four times.
Also not quite sure how to use it in a loop. Quite new to bash scripting :-(
You may use -o (show matches only) with -P (perl regex moode) in gnu-grep:
nx affected:apps |
grep -oP "^\W+app-\K\w+" |
xargs -I {} docker build -t {} .
If gnu-grep isn't available then use this awk command:
nx affected:apps |
awk -F- '/app-/{print $3}' |
xargs -I {} docker build -t {} .
I don't have nx command here but you can try using xargs:
nx affected:apps | grep '^ -' | cut -d' ' -f4 | xargs -I{} echo docker build -t {} ./dist/{}
Remove echo to actually run the command.
You can use the --plain option:
nx affected:apps --plain
the command should return all the affected apps with space as a divider. You can then store those to a bash array and cycle through them in a for loop, running the command you need:
#!/bin/bash
AFFECTED=($(./node_modules/.bin/nx affected:apps --plain))
for t in ${AFFECTED[#]}; do
echo $t
done
I am trying to execute a command based on extracting it from README file.
I was able to extract it using grep and sed:
cat README.md | grep -i "docker build" | grep -vi "dockerfile.debug" | sed 's/.*\(d[a-z]\).*/\1/'
This script would give a result something like 'docker build .'
I want to execute that command.
But I am not sure how to execute the extracted text. I thought 'exec' would work but I couldn't apply it. Please help me find a way to execute the text extracted through the above script.
Set your command in
$(CommandToExecute)
or back-ticks
`CommandToExecute`
As Example:
$(cat README.md | grep -i "docker build" | grep -vi "dockerfile.debug" | sed 's/.*\(d[a-z]\).*/\1/'
);
try:
$(grep -i "docker build" README.md | grep -vi "dockerfile.debug" | sed 's/.*\(d[a-z]\).*/\1/')
I'm trying to create script to be run by cron to create multiple folders with subfolders.
DATE=`date +%Y-%m-%d`
IP_ADDR=`ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p'`
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
If i run this script manually everything is created as expected. When script is ran by cron subdirectory $IP_ADDR is not created and there is no errors.
I suspect that /sbin is not part of the PATH for the environment that the cron job runs under. You should specify the full path for the ifconfig command:
IP_ADDR=$(/sbin/ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p')
It's also better practice (in general) to use $() for command substitution.
Try to use debug mode :
set -x
DATE=`date +%Y-%m-%d`
IP_ADDR=`ifconfig | grep -v '127.0.0.1' | sed -n 's/.*inet addr:\([0-9.]\+\)\s.*/\1/p'`
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
set +x
Then, redirect the output of your cron to a file and have a look, you should find useful information in it.
You are not far off, but there are several ordering caveats that could cause problems. Many systems have different formats for the ifconfig output line. Some with inet xxx.xxx.xxx.xxx, others with inet addr:xxx.xxx.xxx.xxx. (those are the two most common). You may also need to handle the case where there are multiple wired inet interfaces (2+ NICs in the box). However, if you have only 1 NIC, you could try the following to handle the common ifconfig formats:
DATE=`date +%Y-%m-%d`
IP_ADDR=$(ifconfig |
grep -v '127.0.0.1' |
grep -E 'inet[ ](addr:)*[0-9]{1,3}([.][0-9]{1,3}){3}' |
sed -e 's/^.*inet \(addr:\)*//' -e 's/ .*$//')
/bin/mkdir -p /mnt/db-backup/12/$DATE/$IP_ADDR/
or with IP_ADDR written as one line:
IP_ADDR=$(ifconfig | grep -v '127.0.0.1' | grep -E 'inet[ ](addr:)*[0-9]{1,3}([.][0-9]{1,3}){3}' | sed -e 's/^.*inet \(addr:\)*//' -e 's/ .*$//')