How do I cache dependencies without having knowledge of its version or when it changed? - caching

Question

I have written some code that uses a shell script to download and install a toolchain (that is not mine, but is released on GitHub itself in a .tar.gz form). What I wanted to do was that all the artifacts in the latest release of that toolchain be downloaded and saved to a directory which is cached through GitHub actions. And when there is any release newer than the one the cache already has, i.e. tag of latest release is not equal to current downloaded release, download newer one and update the existing cache. But reading more of the docs, I found that caches can not be changed. Here is what I currently have in the script:
#!/bin/sh
LATEST_API_URL="https://api.github.com/repos/.../.../releases/latest"
LATEST_TOOLCHAINS_TAG=$(curl -s "$LATEST_API_URL" | grep "tag_name.*" | cut -d: -f2 | cut -d\" -f2)
TOOLCHAINS_PACKED_SAVE_DIR="../toolchains_packed"
ROOT_DIR="$(dirname "$(readlink -f "$0")")"
cd "$TOOLCHAINS_PACKED_SAVE_DIR"
if [ ! -f "./toolchains_tag.metadata" ]; then
echo 'NEW FILE' > "./toolchains_tag.metadata"
fi
if [ "$LATEST_TOOLCHAINS_TAG" != "$(cat "./toolchains_tag.metadata")" ]; then
DOWNLOADS_URLS=$(curl -s "$LATEST_API_URL" | grep "browser_download_url.*tar.gz" | cut -d: -f2,3 | tr -d \")
wget $DOWNLOADS_URLS || { printf "wget failed: Please read above errors.\nExiting!\n"; exit 1; }
echo "$LATEST_TOOLCHAINS_TAG" > "./toolchains_tag.metadata"
fi
for tar_pkg in *.tar.gz; do
[ ! -f "$tar_pkg" ] && { printf "No *.tar.gz found.\nExiting\n"; exit 1; }
CURR_COMP=$(echo "$tar_pkg" | cut -d. -f1)
COMPILERS_LIST="$COMPILERS_LIST$CURR_COMP\n"
sudo tar -xzvf "$tar_pkg" -C /
done
printf "$COMPILERS_LIST" > "./compilers_list.metadata"
cd $ROOT_DIR
And in the workflow file:
...
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Create toolchains Directory
run: mkdir ../toolchains_packed
- name: Get Cached toolchains Directory
uses: actions/cache#v3.2.3
with:
path: ../toolchains_packed
key: toolchains_packed
- name: Setup toolchains
run: ./path/to/the_script.sh
...
NOTE: This code works successfully in fetching the latest release, and as intended also doesn't fetch one when there is no update available. I only ask for a way using which I can replace older cache with newer one.
How I previously understood caching on GitHub was that caches automatically updated over different runs of a workflow. But now, what I read is not only that caches do not update automatically, they can never update. As a result, the current scenario is that once the GitHub cache is generated, it will never be updated again, and the script will have to download the packages every time, making the entire point of caching null. So, I usually also have to keep an eye on latest release of the toolchain and delete the cache manually if update available.
Personal Thoughts on the Issue: I have thoughts whether I can name my caches using the tag present at ../toolchains/toolchains_tag.metadata. But that too is present in the cache, and hence I don't know how can I do this.
Final Question: How can I update my code such that I do not have to do any manual stuff for updating the cache every time there is a new release available for the toolchain?

Answer 1

I got my answer. I tried viewing some real life examples from different repos on GitHub and taking inspiration from #Azeem's comments, made some changes which worked well. Here is the edited script:-
#!/bin/sh
# The code is split into three parts:-
# 1. Code to check for available update ('-c' cmd line option)
# 2. Code to download available update ('-d' cmd line option) -- ran only
# when '-c' returned with code '1'.
# 3. Code to install toolchains ('-i' cmd line option)
LATEST_API_URL="https://api.github.com/repos/.../releases/latest"
TOOLCHAINS_PACKED_SAVE_DIR="../toolchains_packed"
ROOT_DIR="$(dirname "$(readlink -f "$0")")"
LATEST_TOOLCHAINS_TAG="$(curl -s "$LATEST_API_URL" | grep "tag_name" | cut -d: -f2 | cut -d\" -f2)"
cd "$TOOLCHAINS_PACKED_SAVE_DIR"
case $1 in
-c)
UPDATE_AVAILABLE=0
if [ ! -f ./toolchains_tag.metadata ]; then
echo 'NEW FILE' > ./toolchains_tag.metadata
UPDATE_AVAILABLE=1
fi
if [ "$LATEST_TOOLCHAINS_TAG" != "$(cat ./toolchains_tag.metadata)" ]; then
rm -rf ./*
UPDATE_AVAILABLE=1
fi
exit $UPDATE_AVAILABLE
;;
-d)
TC_DOWNLOADS_URLS="$(curl -s "$LATEST_API_URL" | grep "browser_download_url" | cut -d: -f2,3 | tr -d \")"
echo "wget -t 3 $TC_DOWNLOADS_URLS"
wget -t 3 $TC_DOWNLOADS_URLS || { printf "wget failed: Please read above errors.\nExiting!\n"; exit 1; }
echo "$LATEST_TOOLCHAINS_TAG" > ./toolchains_tag.metadata
;;
-i)
for tar_pkg in *.tar.gz; do
[ ! -f "$tar_pkg" ] && { printf "No *.tar.gz found.\nExiting\n"; exit 1; }
CURR_COMP=$(echo "$tar_pkg" | cut -d. -f1)
COMPILERS_LIST="$COMPILERS_LIST$CURR_COMP\n"
sudo tar -xzf "$tar_pkg" -C /
done
printf "$COMPILERS_LIST" > "./compilers_list.metadata"
;;
esac
cd $ROOT_DIR
exit 0
and the edited workflow file:-
...
jobs:
release:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout#v3
- name: Create toolchains Directory
run: |
mkdir -p toolchains_packed
ln -s `pwd`/toolchains_packed ../toolchains_packed
- name: Get Cached toolchains Directory
uses: actions/cache#v3.2.3
with:
path: toolchains_packed
key: toolchains_packed-${{ github.sha }}
restore-keys: toolchains_packed-
- name: Check For toolchains' Update
id: toolchains_update_check
run: |
if ! ./build_helper/github/setup_toolchains.sh -c; then
echo "update_available=true" >> $GITHUB_OUTPUT
else
echo "update_available=false" >> $GITHUB_OUTPUT
fi
- name: Delete toolchains' Cache
if: steps.toolchains_update_check.outputs.update_available == 'true'
uses: snnaplab/delete-branch-cache-action#v1.0.0
- name: Download toolchains' Update
if: steps.toolchains_update_check.outputs.update_available == 'true'
run: ./build_helper/github/setup_toolchains.sh -d
- name: Install toolchains
run: ./build_helper/github/setup_toolchains.sh -i
...
Explanation:
I changed my script to make it capable of performing three distinct tasks:-
Check if the cache was latest (or if there was even any cache)
Download update (the workflow runs it only if (1) returns something which would tell it that update is available -- exit code 1 in this case)
Install toolchain (ran everytime -- caching is done for packed .tar.gzs only).
Also, I changed my workflow to make it do these things (in order):-
Get cache [NOTE: As ...-${{ github.sha }} is used as primary key of the cache, the code will fail to update cache if workflow is run without any new commit.]
Check for available update availability [by making use of the script]
(3) and (4) are run only if (2) succeeded, i.e. the script returned with exit code 1.
Delete cache [NOTE: For sake of simplicity, I used snnaplab/delete-branch-cache-action#v1.0.0 which deletes all existing caches in the branch. If you want to preserve any cache, please use gh-actions-cache.]
Download update
Install update