Introduction
Currently, I'm writing a customized GitHub workflow inside it I use a curl and grep command.
GitHub repo
action.yml
- name: get new tag
id: get_new_tag
shell: bash
run: |
temp_result=$(curl -s https://api.github.com/repos/${{inputs.github_repository}}/tags | grep -h "name" | grep -h "${{inputs.selector}}" | head -1 | grep -ho "${{inputs.regex}}")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
Full code here
test for action.yml
uses: ./
with:
files: tests/test1.txt tests/test2.txt
github_repository: MathieuSoysal/file-updater-for-release
prefix: MathieuSoysal/file-updater-for-release#
Full code are is here
Problem
I don't understand why but my GitHub Actions don't work with MacOS.
The given error:
temp_result=$(curl -s https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags | grep -h "name" | grep -h "" | head -1 | grep -ho "v\?[0-9.]*")
echo "new-tag=${temp_result}" >> $GITHUB_OUTPUT
shell: /bin/bash --noprofile --norc -e -o pipefail {0}
Error: Process completed with exit code 1.
Full log error is here
Question
Does someone know how we can fix this issue?
Alternatively, you can simply use the jq command line utility for this which is already installed and available on the GitHub runners. See the preinstalled software.
$ export URL='https://api.github.com/repos/MathieuSoysal/file-updater-for-release/tags'
$ curl -s $URL | jq -r .[0].name
v1.0.3
The issue is from this command grep -h "", this command is not supported on MacOS.
The empty string is not supported, the solution is to add something inside it.
I'm trying to understand what's happening here out of curiosity, even though I can just copy and paste the output of the terminal to do what I need to do. The following command does not print anything.
ls /opt/local/var/macports/registry/portfiles -1 | sed 's/-.*//g' | sort -u | parallel "sudo port -N install" {} 2>&1 | grep -Po "Use '\K.*(?=')" | parallel "{}"
The directory I call ls on contains a bunch of filenames starting with the string I want to extract that ends at the first dash (so stringexample-4.2009 pipes stringexample into parallel (like xargs but to run each line separately). After running the command sudo port install <stringexample>, I get error outputs like so:
Unable to activate port <stringexample>. Use 'port -f activate <stringexample>' to force the activation.
Now, I wish to run port -f activate <stringexample>. However, I cannot seem to do anything with the output port -f activate gettext that I get to the terminal.
I cannot even do ... | grep -Po "Use '\K.*(?=')" | xargs echo or ... | grep -Po "Use '\K.*(?=')" >> commands_to_run.txt (the output stream to file only creates an empty file), despite the shorter part of the command:
ls /opt/local/var/macports/registry/portfiles -1 | sed 's/-.*//g' | sort -u | parallel "sudo port -N install {}" 2>&1 | grep -Po "Use '\K.*(?=')"
printing the commands to the terminal. Why does the pipe operator not work here? If the commands I wish to run are outputting to the terminal, surely there's got to be a way to capture them.
It works ok as a single tool:
curl "someURL"
curl -o - "someURL"
but it doesn't work in a pipeline:
curl "someURL" | tr -d '\n'
curl -o - "someURL" | tr -d '\n'
it returns:
(23) Failed writing body
What is the problem with piping the cURL output? How to buffer the whole cURL output and then handle it?
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page.
In curl "url" | grep -qs foo, as soon as grep has what it wants it will close the read stream from curl. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
E.g.
curl "url" | tac | tac | grep -qs foo
tac is a simple Unix program that reads the entire input page and reverses the line order (hence we run it twice). Because it has to read the whole input to find the last line, it will not output anything to grep until cURL is finished. Grep will still close the read stream when it has what it's looking for, but it will only affect tac, which doesn't emit an error.
For completeness and future searches:
It's a matter of how cURL manages the buffer, the buffer disables the output stream with the -N option.
Example:
curl -s -N "URL" | grep -q Welcome
Another possibility, if using the -o (output file) option - the destination directory does not exist.
eg. if you have -o /tmp/download/abc.txt and /tmp/download does not exist.
Hence, ensure any required directories are created/exist beforehand, use the --create-dirs option as well as -o if necessary
The server ran out of disk space, in my case.
Check for it with df -k .
I was alerted to the lack of disk space when I tried piping through tac twice, as described in one of the other answers: https://stackoverflow.com/a/28879552/336694. It showed me the error message write error: No space left on device.
You can do this instead of using -o option:
curl [url] > [file]
So it was a problem of encoding. Iconv solves the problem
curl 'http://www.multitran.ru/c/m.exe?CL=1&s=hello&l1=1' | iconv -f windows-1251 | tr -dc '[:print:]' | ...
If you are trying something similar like source <( curl -sS $url ) and getting the (23) Failed writing body error, it is because sourcing a process substitution doesn't work in bash 3.2 (the default for macOS).
Instead, you can use this workaround.
source /dev/stdin <<<"$( curl -sS $url )"
Trying the command with sudo worked for me. For example:
sudo curl -O -k 'https url here'
note: -O (this is capital o, not zero) & -k for https url.
I had the same error but from different reason. In my case I had (tmpfs) partition with only 1GB space and I was downloading big file which finally filled all memory on that partition and I got the same error as you.
I encountered the same problem when doing:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
The above query needs to be executed using root privileges.
Writing it in following way solved the issue for me:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo apt-key add -
If you write sudo before curl, you will get the Failed writing body error.
For me, it was permission issue. Docker run is called with a user profile but root is the user inside the container. The solution was to make curl write to /tmp since that has write permission for all users , not just root.
I used the -o option.
-o /tmp/file_to_download
In my case, I was doing:
curl <blabla> | jq | grep <blibli>
With jq . it worked: curl <blabla> | jq . | grep <blibli>
I encountered this error message while trying to install varnish cache on ubuntu. The google search landed me here for the error (23) Failed writing body, hence posting a solution that worked for me.
The bug is encountered while running the command as root curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
the solution is to run apt-key add as non root
curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
The explanation here by #Kaworu is great: https://stackoverflow.com/a/28879552/198219
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
I believe the more correct implementation would be to use sponge, as already suggested by #nisetama in the comments:
curl "url" | sponge | grep -qs foo
I got this error trying to use jq when I didn't have jq installed. So... make sure jq is installed if you're trying to use it.
In Bash and zsh (and perhaps other shells), you can use process substitution (Bash/zsh) to create a file on the fly, and then use that as input to the next process in the pipeline chain.
For example, I was trying to parse JSON output from cURL using jq and less, but was getting the Failed writing body error.
# Note: this does NOT work
curl https://gitlab.com/api/v4/projects/ | jq | less
When I rewrote it using process substitution, it worked!
# this works!
jq "" <(curl https://gitlab.com/api/v4/projects/) | less
Note: jq uses its 2nd argument to specify an input file
Bonus: If you're using jq like me and want to keep the colorized output in less, use the following command line instead:
jq -C "" <(curl https://gitlab.com/api/v4/projects/) | less -r
(Thanks to Kowaru for their explanation of why Failed writing body was occurring. However, their solution of using tac twice didn't work for me. I also wanted to find a solution that would scale better for large files and tries to avoid the other issues noted as comments to that answer.)
I was getting curl: (23) Failed writing body . Later I noticed that I did not had sufficient space for downloading an rpm package via curl and thats the reason I was getting issue. I freed up some space and issue for resolved.
I had the same question because of my own typo mistake:
# fails because of reasons mentioned above
curl -I -fail https://www.google.com | echo $?
curl: (23) Failed writing body
# success
curl -I -fail https://www.google.com || echo $?
I added flag -s and it did the job. eg: curl -o- -s https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
I want to execute cql queries from bash command.
[cqlsh 3.1.8 | Cassandra 1.2.19 | CQL spec 3.0.5 | Thrift protocol 19.36.2]
[root#hostname ~]# /opt/apache-cassandra-1.2.19/bin/cqlsh -k "some_keyspace" -e "SELECT column FROM Users where key=value"
I got:
cqlsh: error: no such option: -e
Options:
--version show program's version number and exit
-h, --help show this help message and exit
-C, --color Always use color output
--no-color Never use color output
-u USERNAME, --username=USERNAME
Authenticate as user.
-p PASSWORD, --password=PASSWORD
Authenticate using password.
-k KEYSPACE, --keyspace=KEYSPACE
Authenticate to the given keyspace.
-f FILE, --file=FILE Execute commands from FILE, then exit
-t TRANSPORT_FACTORY, --transport-factory=TRANSPORT_FACTORY
Use the provided Thrift transport factory function.
--debug Show additional debugging information
--cqlversion=CQLVERSION
Specify a particular CQL version (default: 3.0.5).
Examples: "2", "3.0.0-beta1"
-2, --cql2 Shortcut notation for --cqlversion=2
-3, --cql3 Shortcut notation for --cqlversion=3
Any suggestions ?
First of all, you should seriously consider upgrading. You are missing out on a lot of new features and bug fixes.
Secondly, with cqlsh in Cassandra 1.2 you can use the -f flag to specify a file containing cql statements:
$ echo "use system_auth; SELECT role,is_superuser FROM roles WHERE role='cassandra';" > userQuery.cql
$ bin/cqlsh -u aploetz -p reindeerFlotilla -f userQuery.cql
role | is_superuser
-----------+--------------
cassandra | True
(1 rows)
You can use -f to execute from a file or SOURCE once you start CQLSH. I don't think -e is a valid option with that version.
It's bit dirty and unstable, but here is the answer:
/opt/apache-cassandra-1.2.19/bin/cqlsh -k "keyspace" -f /path/to/file.cql > /path/to/output.txt
tail -2 /path/to/output.txt | head -1 > /path/to/output-value.txt
I'm executing the following command in a bash script
curl --silent "https://$URL” | xpath "//connections/ip/text()"
It's working great, but the output includes additional information i.e.
Found 1 nodes:
-- NODE --
192.123.234.11
How can I clean the output so I'm only getting the IP address?
Thanks
xpath lists this flag:
-q quiet. Only output the resulting PATH
Therefore, you can use:
curl --silent "https://$URL" | xpath -q -e "//connections/ip/text()"
You can use grep
curl --silent "https://$URL” | xpath "//connections/ip/text()" | grep "[0-9]*\.[0-9]*\.[0-9]*\.[0-9]*"
Try piping to grep:
curl --silent "https://$URL” | xpath "//connections/ip/text()" | grep -oP "\d+\.\d+\.\d+\.\d+\"