How do I download a file to a newly created directory with curl on OS X? - bash

I am trying to download my Heroku backups to a folder.
Downloading to the current folder like this works:
curl -o latest.dump `heroku pg:backups public-url`
But when I tried adding a folders path to latest.dump it looks like this:
$ curl -o /db-bkups/latest.dump `heroku pg:backups public-url`
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 44318 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0Warning: Failed to create the file
Warning: /db-bkups/latest.dump: No such file
Warning: or directory
36 44318 36 16384 0 0 9626 0 0:00:04 0:00:01 0:00:03 9626
curl: (23) Failed writing body (0 != 16384)
Ideally, I would like it be saved and downloaded like this:
/db-bkups/nov-1-2016/timestamp-db.dump
Where the folder nov-1-2016 is created dynamically when the cron is run, and the filename is the timestamp when the bkup was run.

You could try using the --create-dirs argument which was added in curl 7.10.3:
Here is an example that will create the directory hierarchy, (if it doesn't already exist), and will name the subdirectory you require renamed with the output of the datecommand:
curl -o /db-bkups/$(date +"%b-%d-%Y")/timestamp-db.dump --create-dirs http://www.w3schools.com/xml/simple.xml
The result is a file stored in a directory like so /db-bkups/Nov-04-2016/timestamp-db.dump.

Related

Download tsv file from link which generates the file upon request, using bash

I need to download .txt files which are generated from links like this one:
https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0
but I need to download it in the bash shell. It works perfectly fine on Firefox, on the shell I tried wget and curl to no avail. I read lots of similar question in Stack Overflow and other pages, tried everything I could find, but couldn't find a solution.
For example:
curl https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0
This is the output, and no file is downloaded:
[1] 1094
[2] 1095
[3] 1096
[4] 1097
[5] 1098
[2] Done result=read_run
[3] Done fields=fastq_ftp
[4]- Done format=tsv
(base) user#DESKTOP-LV4SKHQ:/mnt/c/Users/conog/Desktop/prova$ curl: (6) Could not resolve host: www.ebi.ac.uk
[1]- Exit 6 curl https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480
[5]+ Done download=true
Another example, after I read a couple of posts here:
curl -O -L https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0
[1] 1056
[2] 1057
[3] 1058
[4] 1059
[5] 1060
[2] Done result=read_run
[3] Done fields=fastq_ftp
[4] Done format=tsv
[5]+ Done download=true
(base) gsoletta#DESKTOP-LV4SKHQ:/mnt/c/Users/conog/Desktop/prova$ % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 49 100 49 0 0 68 0 --:--:-- --:--:-- --:--:-- 67
[1]+ Done
this last one downloads a 49 byte file with no extension, called filereportaccession=SRP002480, with the content: "Required String parameter 'result' is not present".
I'll also add I'm a novice at bash.
What could I do?
Thank you!
It works for me:
$ curl -s 'https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0'
run_accession fastq_ftp
SRR1620013 ftp.sra.ebi.ac.uk/vol1/fastq/SRR162/003/SRR1620013/SRR1620013_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR162/003/SRR1620013/SRR1620013_2.fastq.gz
SRR1620014 ftp.sra.ebi.ac.uk/vol1/fastq/SRR162/004/SRR1620014/SRR1620014_1.fastq.gz;ftp.sra.ebi.ac.uk/vol1/fastq/SRR162/004/SRR1620014/SRR1620014_2.fastq.gz
...
$ wget -O filereport.tsv 'https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0'
--2021-11-15 17:51:48-- https://www.ebi.ac.uk/ena/portal/api/filereport?accession=SRP002480&result=read_run&fields=fastq_ftp&format=tsv&download=true&limit=0
Resolving www.ebi.ac.uk (www.ebi.ac.uk)... 193.62.193.80
Connecting to www.ebi.ac.uk (www.ebi.ac.uk)|193.62.193.80|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: ‘filereport.tsv’
...
2021-11-15 17:51:51 (831 KB/s) - ‘filereport.tsv’ saved [675136]
Your problem is that you didn't put quotes around the URL. When you don't quote the URL the &s in it cause each URL parameter to be interpreted as a separate command by bash.

Google dataproc Intialization action error: 'sh: 0: Can't open SparkBuildPanelBatch.sh'

I am trying to use Initialization actions using Dataproc (GCP).
A very short intro:
We are running processes constantly on Dataproc clusters with GCP using Spark jobs to transform our data in many different ways.
I am trying the last few days to build an initialization action file (.sh) that will run a few Linux commands when the cluster is up. Everything works well except the last command (the one that supposes to run the Spark app engine & execute the job).
So I am now adding all the related information (My .sh file + the error log) and I would love to hear your suggestions while I still looking around the internet to figure it out. Thanks.
My .sh File:
#!/bin/bash
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [ $ROLE = "Master" ]; then
echo $ROLE
sudo mkdir /opt/ads
sudo mkdir /opt/ads/apps
sudo gcloud compute scp --recurse 35613250742-compute#brainservice:/opt/ads/apps/ /opt/ads/apps/ --zone=europe-west2-c --internal-ip
sudo sh SparkBuildPanelBatch.sh /opt/ads/apps/apps/SparkBuildPanelBatch_Latest/DeployPack_2203.txt 20201201 20201210 20201210
fi
My error output log:
Master
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/google_compute_engine.
Your public key has been saved in /root/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:QEN6xC2DB8+lUmFPpvxilMQmPYMBXSPkaTEgNuvqP8c root#analytics-m
The key's randomart image is:
+---[RSA 2048]----+
|.oo=B#%o+ |
|..o.=/##. |
| . *+#=. |
|. . + o |
| . o S |
|. . . |
|. . |
|. . E |
| ...o |
+----[SHA256]-----+
Updating project ssh metadata...
.............................................................................................................................................................Updated [https://www.googleapis.com/compute/v1/projects/supersal].
..done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.4114551798115890446' (ECDSA) to the list of known hosts.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 19 100 19 0 0 3022 0 --:--:-- --:--:-- --:--:-- 3166
scp: /opt/ads/apps/Profile360_Latest/serviceLog/Profile360Service.out: Permission denied
ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1].
sh: 0: Can't open SparkBuildPanelBatch.sh
I am not worried about that line:'scp: /opt/ads/apps/Profile360_Latest/serviceLog/Profile360Service.out: Permission denied
'
because it out of use and will be fixed later. I don't understand why it cant open the .sh file.
I figured it out I just added a line of command 'cd /opt/ads/apps/CurrectFolder' between the last two and it works lol. Thanks, #tripleee for your time.

Piping raw code from github to ruby not working?

I am doing some basic piping of some simple raw code from github to terminal as shown here i.e.
curl https://raw.github.com/leachim6/hello-world/master/r/ruby.rb | ruby
When I try it, it doesn't produce "Hello World", but instead I just see
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
use
curl -sSL https://raw.github.com/leachim6/hello-world/master/r/ruby.rb | ruby
this should work
Update to explain
this URL is redirecting to
https://raw.githubusercontent.com/leachim6/hello-world/master/r/ruby.rb
so -L option was required to follow the redirection (-L, --location)
this option will make curl redo the request on the new place
sS to hide the progress bar and show errors if happened
to debug curl request you can use -v option which will make you see exactly what is happening

executing .sh file - "./gradlew no such file or directory" and misc errors

I'm trying to run a .sh file on Windows. More exactly I'm trying to run the build scripts in the following GitHub repo: https://github.com/facebook/facebook-sdk-for-unity.
But I'm extremely novice when it comes to .sh files. And came as far as executing the file by writing the following in "Git Shell": sh ./setup.sh.
I'm unsure which errors is actually "errors", but I've bolded what I think is the important issue. It's also worth mentioning that I've downloaded Gradle 3.1 and added it to my path variable. gradle -v does work (in cmd), but gradlew as a command does not. What am I missing? I feel stupid. :/
I get the following output/errors:
D:\GitHub\facebook-sdk-for-unity [tmp-test-bashscripts]> sh .\scripts\setup.sh
/d/GitHub/facebook-sdk-for-unity/scripts/local.properties: line 1: sdk.dir=D:\Android\android-sdk\sdk: No such file or directory
sed: can't read /FacebookSdkVersion.cs: No such file or directory
.\scripts\setup.sh: line 25: buildAndCopyCore: command not found
.\scripts\setup.sh: line 26: buildAndCopyPlatformDLLs: command not found
/d/GitHub/facebook-sdk-for-unity/scripts/local.properties: line 1: sdk.dir=D:\Android\android-sdk\sdk: No such file or directory
sed: can't read /FacebookSdkVersion.cs: No such file or directory
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 75.1M 100 75.1M 0 0 359k 0 0:03:33 0:03:33 --:--:-- 372k
**/d/GitHub/facebook-sdk-for-unity/scripts/local.properties: line 1: sdk.dir=D:\Android\android-sdk\sdk: No such file or directory**
sed: can't read /FacebookSdkVersion.cs: No such file or directory
\033[0;36mStarting build \033[0m
\033[0;36mStep 1 - Cleaning wrapper libs folder \033[0m
/d/GitHub/facebook-sdk-for-unity/facebook-android-wrapper/libs /d/GitHub/facebook-sdk-for-unity
find: warning: you have specified the -maxdepth option after a non-option argument !, but options are not positional (-maxdepth affects tests specified before it as well as those specified after it). Please specify options before other arguments.
/d/GitHub/facebook-sdk-for-unity
\033[0;36mStep 2 - Get depenancies for android wrapper \033[0m
\033[0;36mStep 2.1.0 - Download bolts-android-1.4.0.jar \033[0m
\033[0;36mbolts-android-1.4.0.jar already exists. Skipping download. \033[0m
\033[0;36mStep 2.1.1 - Download bolts-tasks-1.4.0.jar \033[0m
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 178 100 178 0 0 546 0 --:--:-- --:--:-- --:--:-- 642
100 190 100 190 0 0 212 0 --:--:-- --:--:-- --:--:-- 690
100 38092 100 38092 0 0 28892 0 0:00:01 0:00:01 --:--:-- 28892
\033[0;36mStep 2.1.2 - Download bolts-applinks-1.4.0.jar \033[0m
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 178 100 178 0 0 650 0 --:--:-- --:--:-- --:--:-- 661
100 196 100 196 0 0 233 0 --:--:-- --:--:-- --:--:-- 233
100 25020 100 25020 0 0 20337 0 0:00:01 0:00:01 --:--:-- 228k
\033[0;36mStep 2.2 - Download facebook-android-sdk-4.14.0.aar \033[0m
\033[0;36mfacebook-android-sdk-4.14.0.aar already exists. Skipping download \033[0m
\033[0;36mStep 2.3 - Coping support lib \033[0m
\033[0;36mStep 3 - Build android wrapper \033[0m
/d/GitHub/facebook-sdk-for-unity/facebook-android-wrapper /d/GitHub/facebook-sdk-for-unity
**/d/GitHub/facebook-sdk-for-unity/scripts/setup_android_unity_plugin.sh: line 118: ./gradlew: No such file or directory**
\033[0;31mFATAL: Failed to perform gradle clean \033[0m
\033[0;31mFATAL: Failed to build the android sdk plugin \033[0m
It looks like you are running sh .\scripts\setup.sh in a cmd or Git CMD window. Open a Git Bash window instead. It's important to see "Bash" in its name. In that shell, go to the project directory with the cd command, and run sh ./setup.sh as it was written in the documentation.

Elasticsearch 1.7.1 not creating a snapshot, but not reporting error either

I'm trying to create a snapshot of all indexes in my local Elasticsearch instance.
I've set path.repo as follows in elasticsearch.yml:
path.repo: ["F:\\backup\\elasticsearch"]
And here's the command I'm using to create the snapshot:
curl -XPUT http://localhost:9200/_snapshot/my_test_backup -d '
{
"type": "fs",
"settings": {
"location": "F:\\backup\\elasticsearch\\my_test_backup"
}
}'
Executing this generates the following output from Elasticsearch:
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 136 100 21 100 115 269 1474 --:--:-- --:--:-- --:--:-- 1854{"acknowledged":true}
Note, no error.
There are a couple of indexes set up on my local instance, which aren't particularly large so, when I check the snapshot status, it shows no in progress snapshots:
$ curl -XGET http://localhost:9200/_snapshot/_status
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 16 100 16 0 0 1000 0 --:--:-- --:--:-- --:--:-- 1000{"snapshots":[]}
As I say, this isn't necessarily a worry because the snapshot would be small anyway. I can see the snapshot I've just created by executing the following, but it appears to have hung:
$ curl -XGET http://localhost:9200/_snapshot
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 100 100 100 0 0 100 0 0:00:01 --:--:-- 0:00:01 97k{"my_test_backup":{"type":"fs","settings":{"location":"F:\\backup\\elasticsearch\\my_test_backup"}}}
Consistent with this, when I navigate to F:\backup\elasticsearch\my_test_backup the folder is empty.
Could somebody tell me why my snapshot isn't working? What am I doing wrong?
Many thanks,
Bart
All you have done there is to create a repository, not a snapshot. Creating a repository is a necessary artifact that will store all future snapshots you will create.
So now that you have your repository, you can simply kick off the snapshot creation as follows:
curl -XPUT "localhost:9200/_snapshot/my_test_backup/snapshot_1"
If you run the following command instead, it will only return when the snapshot is done:
curl -XPUT "localhost:9200/_snapshot/my_test_backup/snapshot_1?wait_for_completion=true"

Resources