i have the following command to curl a list of urls but I want to rename the output files to something else, as the original name is really long and results in the error below:
command:
jq -r '… | .[]' | xargs -I{} curl -O {}
error:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0Warning: Failed to create the file
Warning: 270230751_944073806238863_7984852070388291566_n.jpg?stp=dst-jpg_e35&_n
Warning: c_ht=example.njrsnnkjnrjgngkrnngrggk&_nc_cat=111&_nc_ohc=Ch26T4U5kDIAX
Warning: 8viWsL&edm=AABBvjUBAAAA&ccb=7-4&ig_cache_key=MjczOTA2MDk4MDU3MTIyNjQ1N
0 92492 0 1 0 0 1 0 25:41:32 --:--:-- 25:41:32 1
curl: (23) Failed writing body (0 != 1)
how to pass some argument to curl so that it renames the file to a smaller name eg 16 digit uuid or something like njfnjsnf48u8 but not too common as there can be upto 1000 files being downloaded at once.
You can manipulate the {} variable that xargs uses in curl command to make it more unique. If you are sure that there are no duplicate urls in the input file then the following might work.
curl {} -o $(echo {}|base64)
Related
I am using ElasticSearch 7.17.
I am trying to create a snapshot of an index:
(I know I shouldn't have a single shard, but for now, that's how it is) :
$ curl -s -k "http://localhost:9200/_cat/indices"
yellow open myIndex vVr6ojDCQTi9ASOUGkkRBA 1 1 679161903 0 140.8gb 140.8gb
I have already registered an S3 bucket for snapshots, which I named backups.
I ran the following command:
$ curl -s -k -X PUT "http://localhost:9200/_snapshot/backups/myIndex?pretty&wait_for_completion=false" -H "content-type:application/json" -d'{"indices": "myIndex"}'
{
"accepted" : true
}
Now, I want to have a look at the progress of that backup's upload :
$ curl -s -k "http://localhost:9200/_cat/snapshots/backups/myIndex"
myIndex IN_PROGRESS 1676385605 14:40:05 0 00:00:00 8.6m 1 0 0 0
$ curl -s -k "http://localhost:9200/_cat/recovery"
myIndex 0 37ms empty_store done n/a n/a 172.24.0.3 7529c7447620 n/a n/a 0 0 0.0% 0 0 0 0.0% 0 0 0 100.0%
It's been in this state, with no change, for the past 1 hour.
I don't understand why 0 bytes are transfered. Am I missing something obvious ?
I don't know what empty_store refers to - shouldn't it be existing_store ?
Other people were right - it just took its time.
The snapshot ended in "SUCCESS" status, but the repository remains in empty_store.
Hi, I am trying to create a Jira issue through Jenkins on a windows slave. The console output is not showing any error, however, the Jira issue is not getting created. Below is the code:
pipeline {
agent { label 'windows'}
stages {
stage('Build') {
steps {
bat script {"""curl -u ${jira_username}:${jira_password} -X POST -H 'Content- Type:application/json' -d '{"fields":{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":{"key":"KEY"},"summary":"summary","description":"description","issuetype":{"name":"Test"}}}' http://localhost:8080/rest/api/2/issue/"""}
}
}
}
}
The console output is:
D:\workspace\TestJob>curl -u username:password -X POST -H 'Content-Type:application/json' -d
'{"fields":{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":
{"key":"KEY"},"summary":"summary","description":"description","issuetype":{"name":"Test"}}}'
http://localhost:8080/rest/api/2/issue/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 136 0 0 100 136 0 323 --:--:-- --:--:-- --:--:-- 326
It doesn't throw any errors but the issue is also not getting created.
However, The same code if I run on Linux slave then I get the below response and the issue is getting created.
+ curl -u 'username:password' -X POST -H Content-Type:application/json -d '{"fields":
{"components":[{"id":"1"}],"fixVersions":[{"id":"2"}],"project":
{"key":"KEY"},"summary":"summary","description":"description","issuetype":
{"name":"Test"}}}' http://localhost:8080/rest/api/2/issue/
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 255 0 87 100 168 115 223 --:--:-- --:--:-- --:--:-- 223
100 255 0 87 100 168 115 223 --:--:-- --:--:-- --:--:-- 223
{"id":"648","key":"KEY-35","self":"http://localhost:8080/rest/api/2/issue/648"}
As you can see the issue is getting created on Linux but not on windows. Requesting help to resolve this issue.
Resolved the issue
Firstly 'Content-Type: application/json' should be in double-quotes.
Putting it in single quote 'Content-Type: application/json' was actually giving 415 error code which I got to know after using -k -D- in the curl command
Secondly, I had to use double backslash in the data.
The command which actually worked is:
bat script {"""curl -u username:password --header "Content-Type:application/json" -X POST --data "{\\"fields\\":{\\"project\\":{\\"key\\":\\"KEY\\"},\\"summary\\":\\"summary\\",\\"description\\":\\"description\\",\\"issuetype\\":{\\"name\\":\\"Test\\"},\\"components\\":[{\\"id\\":\\"1\\"}],\\"fixVersions\\":[{\\"id\\":\\"2\\"}]}}" http://localhost:8080/rest/api/2/issue/"""}
I am writing a small bash script that gets the profile of a Minecraft user from their username. However, when I run the following script, the profile_json variable comes out blank while the uuid_json and uuid variables have the content they should.
#!/bin/bash
# settings
username="Notch"
# script
uuid_json=$(curl https://api.mojang.com/users/profiles/minecraft/$username)
echo $uuid_json
uuid=$(jq '.id' <<< "$uuid_json")
echo $uuid
profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid)
echo $profile_json
Moreover, the output from running the script in a terminal shows the Current Speed of profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid) as 0. Any ideas as to why this would be?
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 56 100 56 0 0 160 0 --:--:-- --:--:-- --:--:-- 160
{"name":"Notch","id":"069a79f444e94726a5befca90e38aaf5"}
"069a79f444e94726a5befca90e38aaf5"
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
AS you can see by executing echo $uuid you got "069a79f444e94726a5befca90e38aaf5"
instead of 069a79f444e94726a5befca90e38aaf5 there should be no quotes.
jq has option to remove them it's -r with that everything works as expected(I think).
So change it to:
#!/bin/bash
# settings
username="Notch"
# script
uuid_json=$(curl https://api.mojang.com/users/profiles/minecraft/$username)
echo $uuid_json
uuid=$(jq -r '.id' <<< "$uuid_json") echo $uuid
profile_json=$(curl https://sessionserver.mojang.com/session/minecraft/profile/$uuid)
echo $profile_json
Your echo $uuid outputs this:
"069a79f444e94726a5befca90e38aaf5"
But it really should be:
069a79f444e94726a5befca90e38aaf5
for the curl command to work. So, tweak your jq line to strip off quotation marks.
I am trying to make a bash script which basically takes a bunch of .debs, unpacks them and place binaries and libs in /usr/local/opt/{lib}bin.
The script checks whether / is mounted as ro or rw, and if mounted as ro to remount it as rw.
On chromebooks however, in order to mount / as rw you need to remove_rootfs_verification for the partition in question. The script fails to echo what stated above when rootfs_verification is enabled for /, and should exit 1, instead it carries on.
Here is the part of the script I am referring to
### ChromeOS's Specific!!!
# The following assumes rootfs_verification for / has already been removed
if grep $rootfs /proc/mounts | grep ro; then
mount -o remount,rw / &> mount.out
elif
grep -iw 'read-write' mount.out; then
echo '\nrootfs_verification for the root partition must to be removed in order to remount,rw /
To remove rootfs_verification run the following command and than reboot the system:
"sudo /usr/share/vboot/bin/make_dev_ssd.sh --remove_rootfs_verification --partitions 4"'
else exit 1
fi
The entire WIP script can be found here https://pastebin.com/ekEPSvYy
This is what happen when I execute it
localhost /usr/local # ./kvm_install.sh
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 124 0 124 0 0 322 0 --:--:-- --:--:-- --:--:-- 345
100 135 100 135 0 0 170 0 --:--:-- --:--:-- --:--:-- 170
100 60384 100 60384 0 0 57950 0 0:00:01 0:00:01 --:--:-- 344k
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 143 0 143 0 0 407 0 --:--:-- --:--:-- --:--:-- 412
100 154 100 154 0 0 202 0 --:--:-- --:--:-- --:--:-- 202
100 1298k 100 1298k 0 0 929k 0 0:00:01 0:00:01 --:--:-- 3020k
/dev/root / ext2 ro,seclabel,relatime,block_validity,barrier,user_xattr,acl 0 0
./kvm_install.sh: line 31: /etc/env.d/30kvm: Read-only file system
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 66802 100 66802 0 0 69657 0 --:--:-- --:--:-- --:--:-- 74555
./kvm_install.sh: line 39: ar: command not found
tar (child): control.tar.gz: Cannot open: No such file or directory
tar (child): Error is not recoverable: exiting now
tar: Child returned status 2
tar: Error is not recoverable: exiting now
md5sum: md5sums: No such file or directory
Basically what happens here is ar cannot be found as the script was unable to add the PATH variables to /etc/env.d/30kvm since the root partition cannot be mounted because roots_verification is enabled on /.
I tried adding the elif "grep" command in [[ as some suggested here, but that didn't work and adds further syntax issues.
I am in the process of learnign the basics of bash scripting. I apologize if the script is written poorly.
Thanks
I ultimately ended up doing this.
if grep $rootfs /proc/mounts | grep 'ro,'; then
mount -o remount,rw / &> mount.out
if
grep 'read-write' mount.out; then
echo 'something to echo' && exit 1
fi
fi
It is not pretty, but it works until I find/ learn a better way to implement the loop.
to make the output of a command a variable do:
$variable="$(command)"
if you want to use grep use the syntax:
command | grep text
if you want to do if statements use the syntax:
if [ some text ]; text
commands
elif [ some text ]
commands
else
commands
fi
for the some text check this cart
grep -iw is not valid on chromebook
rootfs has a different path depending on a lot of things if you want to save it as $rootfs do this command
rootfs="$(rootdev -s)"
also you made some mistake is the "|" and "&"
command1 | command to edit command1
command1 || run this command if command1 fails
command1 && run this command if command1 succeeds
Here's an example ip proxy checker shell script:
#!/bin/sh
while read IP
do
CURL=$(curl -x http://$IP -L http://icanhazip.com)
echo "$CURL"
done < ip.txt
But instead of a simple result like:
0.0.0.0
1.1.1.1
I get:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
0.0.0.0
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
1.1.1.1
How can I quiet the extra stuff?
-s/--silent
Silent mode. Don't show progress meter or error messages. Makes Curl mute.
If this option is used twice, the second will again disable mute.
CURL=$(curl -s -x http://$IP -L http://icanhazip.com)