I have multiple sets two of Docker services running simultaneously, so my docker ps logs look something like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0a8e26682915 image_name:latest "bash -c 'pip instal…" About a minute ago Up About a minute process_name_1_actions_1
ed8b56ff29b1 image_name:latest "bash -c 'cd live-bo…" About a minute ago Up About a minute 0.0.0.0:5005->5005/tcp, :::5005->5005/tcp process_name_1_model_1
6d8016412d12 image_name:latest "bash -c 'pip instal…" About a minute ago Up About a minute process_name_2_actions_1
128d3605297a image_name:latest "bash -c 'cd live-bo…" About a minute ago Up About a minute 0.0.0.0:5006->5005/tcp, :::5006->5005/tcp process_name_2_model_1
Note that the names of these services have tags. I would like to create a shell script which can loop through these sets of processes by their tags, and restart the actions_1 service. Something like:
declare -a arr=("process_1" "process_2")
for i in "${arr[#]}"
do
# here, restart only the 'actions_1' service of process i.
done
Justification for this is that the processes titled _model_1 takes a long time to spawn. The process titled _actions_1 needs to be restarted often, and does not take a long time to spawn. For this reason, running docker-compose down followed by docker-compose up is a very tedious process.
Use this command
docker ps --format '{{.Names}} {{.ID}}' | awk '{if ($1 ~ "_actions_1") print $2;}' | xargs -P 10 docker restart
Description ::
docker ps --format '{{.Names}} {{.ID}}: Lists current containers running with only 2 fields - name & id
awk: Checks $1 i.e. image name via regex, if matches, then prints corresponding $2 i.e. container id
xargs Executes specified command one by one upon the output. -P 10 executes 10 restarts in parallel at most, to speed things up.
Team,
On bash I run below to view logs but when they are huge the terminal all is all occupied dumping them out and i can't stop it for minutes.
Is there a way i can execute kubectl on bash in a a way that it shows logs only till shell screen size and then i use return key or spacebar to see more ? just like journalctl? but i want to achieve this when using kubectl.
kubectl logs test-pod -n namespace-test
Above displans 10K lines at a time which i don't want. Neither I want to exec on to the pod and see log file physically. any hints? or is there something like which displays last 100lines? or first with xargs?
There are a few ways to obtain that.
1) less
kubectl logs test-pod -n namespace-test | less
This command will allow you to read logs page by page from top to the bottom. You can use arrows to go up or down.
2) --tail=
kubectl logs test-pod -n namespace-test --tail=500
Will display last 500 rows of logs
3) pipeline with grep
kubectl logs test-pod -n namespace-test | grep <some_phrase>
It will allow you to find logs with given phrases. Please note that you can join commands via pipeline i.e
kubectl logs test-pod -n namespace-test --tail=500 | grep <some_phrase>
4) --since=
kubectl logs test-pod -n namespace-test --since=60
It display logs from last 60 seconds
5) --since-time=''
Similar to previous one. It will display logs from provided time. Format in this command is
'YYYY-MM-DDTHH:MM:ssZ'
kubectl logs test-pod -n namespace-test --since-time='2019-04-23T12:00:00Z'
You could hear about more option like but it is old format. You can scroll only from top to the bottom, you cannot go up.
kubectl logs test-pod -n namespace-test | more
You can do:
kubectl logs --tail=100 test-pod -n namespace-test
I am trying to find a command or a sample shell snippet where I can wait until the list of Kubernetes pods is successful. I have checked the answer but it was not giving any output. Can someone guide me or suggest an approach, I am completely new to kubernetes.
kubectl -n test-ns get jobs -w
NAME DESIRED SUCCESSFUL AGE
test-1 1 1 2d
test-2 1 1 2d
test-3 1 1 2d
test-4 1 1 2d
until kubectl get jobs -n test-ns -o jsonpath='{.status.conditions[?(#.type=="Complete")].status}' | grep True ; do sleep 1 ; done
This is not giving any output
To wait until your pods are running, check for "condition=ready" and filter by app label, for example:
$ kubectl wait --for=condition=ready pod -l app=netshoot
pod/netshoot-58785d5fc7-xt6fg condition met
you need to use this command
kubectl rollout status
If you want to use kubectl as described here where it gets all the jobs, you need to use .items[*]... in your JSONpath (That answer is for just one specific job). For example:
kubectl -n test-ns get jobs \
-o jsonpath='{.items[*].status.conditions[?(#.type=="Complete")].status}' \
| grep True
It works ok as a single tool:
curl "someURL"
curl -o - "someURL"
but it doesn't work in a pipeline:
curl "someURL" | tr -d '\n'
curl -o - "someURL" | tr -d '\n'
it returns:
(23) Failed writing body
What is the problem with piping the cURL output? How to buffer the whole cURL output and then handle it?
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page.
In curl "url" | grep -qs foo, as soon as grep has what it wants it will close the read stream from curl. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
E.g.
curl "url" | tac | tac | grep -qs foo
tac is a simple Unix program that reads the entire input page and reverses the line order (hence we run it twice). Because it has to read the whole input to find the last line, it will not output anything to grep until cURL is finished. Grep will still close the read stream when it has what it's looking for, but it will only affect tac, which doesn't emit an error.
For completeness and future searches:
It's a matter of how cURL manages the buffer, the buffer disables the output stream with the -N option.
Example:
curl -s -N "URL" | grep -q Welcome
Another possibility, if using the -o (output file) option - the destination directory does not exist.
eg. if you have -o /tmp/download/abc.txt and /tmp/download does not exist.
Hence, ensure any required directories are created/exist beforehand, use the --create-dirs option as well as -o if necessary
The server ran out of disk space, in my case.
Check for it with df -k .
I was alerted to the lack of disk space when I tried piping through tac twice, as described in one of the other answers: https://stackoverflow.com/a/28879552/336694. It showed me the error message write error: No space left on device.
You can do this instead of using -o option:
curl [url] > [file]
So it was a problem of encoding. Iconv solves the problem
curl 'http://www.multitran.ru/c/m.exe?CL=1&s=hello&l1=1' | iconv -f windows-1251 | tr -dc '[:print:]' | ...
If you are trying something similar like source <( curl -sS $url ) and getting the (23) Failed writing body error, it is because sourcing a process substitution doesn't work in bash 3.2 (the default for macOS).
Instead, you can use this workaround.
source /dev/stdin <<<"$( curl -sS $url )"
Trying the command with sudo worked for me. For example:
sudo curl -O -k 'https url here'
note: -O (this is capital o, not zero) & -k for https url.
I had the same error but from different reason. In my case I had (tmpfs) partition with only 1GB space and I was downloading big file which finally filled all memory on that partition and I got the same error as you.
I encountered the same problem when doing:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | apt-key add -
The above query needs to be executed using root privileges.
Writing it in following way solved the issue for me:
curl -L https://packagecloud.io/golang-migrate/migrate/gpgkey | sudo apt-key add -
If you write sudo before curl, you will get the Failed writing body error.
For me, it was permission issue. Docker run is called with a user profile but root is the user inside the container. The solution was to make curl write to /tmp since that has write permission for all users , not just root.
I used the -o option.
-o /tmp/file_to_download
In my case, I was doing:
curl <blabla> | jq | grep <blibli>
With jq . it worked: curl <blabla> | jq . | grep <blibli>
I encountered this error message while trying to install varnish cache on ubuntu. The google search landed me here for the error (23) Failed writing body, hence posting a solution that worked for me.
The bug is encountered while running the command as root curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
the solution is to run apt-key add as non root
curl -L https://packagecloud.io/varnishcache/varnish5/gpgkey | apt-key add -
The explanation here by #Kaworu is great: https://stackoverflow.com/a/28879552/198219
This happens when a piped program (e.g. grep) closes the read pipe before the previous program is finished writing the whole page. cURL doesn't expect this and emits the "Failed writing body" error.
A workaround is to pipe the stream through an intermediary program that always reads the whole page before feeding it to the next program.
I believe the more correct implementation would be to use sponge, as already suggested by #nisetama in the comments:
curl "url" | sponge | grep -qs foo
I got this error trying to use jq when I didn't have jq installed. So... make sure jq is installed if you're trying to use it.
In Bash and zsh (and perhaps other shells), you can use process substitution (Bash/zsh) to create a file on the fly, and then use that as input to the next process in the pipeline chain.
For example, I was trying to parse JSON output from cURL using jq and less, but was getting the Failed writing body error.
# Note: this does NOT work
curl https://gitlab.com/api/v4/projects/ | jq | less
When I rewrote it using process substitution, it worked!
# this works!
jq "" <(curl https://gitlab.com/api/v4/projects/) | less
Note: jq uses its 2nd argument to specify an input file
Bonus: If you're using jq like me and want to keep the colorized output in less, use the following command line instead:
jq -C "" <(curl https://gitlab.com/api/v4/projects/) | less -r
(Thanks to Kowaru for their explanation of why Failed writing body was occurring. However, their solution of using tac twice didn't work for me. I also wanted to find a solution that would scale better for large files and tries to avoid the other issues noted as comments to that answer.)
I was getting curl: (23) Failed writing body . Later I noticed that I did not had sufficient space for downloading an rpm package via curl and thats the reason I was getting issue. I freed up some space and issue for resolved.
I had the same question because of my own typo mistake:
# fails because of reasons mentioned above
curl -I -fail https://www.google.com | echo $?
curl: (23) Failed writing body
# success
curl -I -fail https://www.google.com || echo $?
I added flag -s and it did the job. eg: curl -o- -s https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.1/install.sh | bash
I went through the documentation to edit kubernetes resource using kubectl edit command. Once I execute the command, the file in YAML-format is opened in the editor where I can change the values as per requirement and save it. I am trying to execute these steps by means of sed. How can the following steps be achieved?
Execute kubectl edit for a deployment resource
Set a value from true to false (using sed)
Save the changes
I tried to achieve this in the following way :
$ kubectl edit deployment tiller-deploy -n kube-system | \
sed -i "s/\(automountServiceAccountToken:.*$\)/automountServiceAccountToken: true/g"`
Your command is missing a backtick. But even though you put it there, it won't work. The reason is because when you do kubectl edit ..., it edits the file on vim. I am not sure sed would work on vim though. Even though if it does, the output goes to a file, so you get the Vim: Warning: Output is not to a terminal error, which I don't know how to solve.
I would recommend you to get the file and save it. Replace the desired parameters and run it again:
kubectl get deploy tiller-deploy -n kube-system -o yaml > tiller.yaml && sed -i "s/automountServiceAccountToken:.*$/automountServiceAccountToken: true/g" tiller.yaml && kubectl replace -f tiller.yaml
I tried the command above and it worked.
Note: no need to add -n kube-system as the yaml file already contains the namespace.
I just found a less convoluted way of doing this:
KUBE_EDITOR="sed -i s/SOMETHING TO CHANGE/CHANGED/g" kubectl edit resource -n your-ns
I automate through piping the commands through sed command without creating a temporary file. Take the below example, where I am replacing nameserver 8.8.8.8 with 1.1.1.1
$ kubectl -n kube-system get configmap/kube-dns -o yaml | sed "s/8.8.8.8/1.1.1.1/" | kubectl replace -f -
Thanks, #suren for giving what I really looking for, but you don't need to save it in a file. you can directly do kubectl replace using pipe operations
kubectl get deploy test-deploy -o yaml | sed "s/find/replace/g" | kubectl replace -f -
An easy way to do this, just use kubectl-patch instead of sed.
$ kubectl patch deployment tiller-deploy -n kube-system --patch '{"map": {"to": {"the": {"key": {"automountServiceAccountToken": "true"}}}}}'
I don't know kubectl but doc seems to explain that it extract data, edit from an editor than send back, not sure sed pipe work in this case
if piping wokrs
Don't use -i, you don't change a file in a pipe
kubectl edit deployment tiller-deploy -n kube-system | \
sed 's/automountServiceAccountToken:.*$/automountServiceAccountToken: true/g'
if editing a file (and using group in sed)
kubectl edit deployment tiller-deploy -n kube-system > YourCOnfigFile && \
sed -i 's/\(automountServiceAccountToken:\).*$/\1 true/g' YourConfigFile \
&& Some kubectl to send back YourConfigFile