I am quite new with wget and I have done my research on Google but I found no clue.
I need to save a single HTML file of a webpage:
wget yahoo.com -O test.html
and it works, but, when I try to be more specific:
wget http://search.yahoo.com/404handler?src=search&p=food+delicious -O test.html
here comes the problem, wget recognizes &p=food+delicious as a syntax, it says: 'p' is not recognized as an internal or external command
How can I solve this problem? I really appreciate your suggestions.
The & has a special meaning in the shell. Escape it with \ or put the url in quotes to avoid this problem.
wget http://search.yahoo.com/404handler?src=search\&p=food+delicious -O test.html
or
wget "http://search.yahoo.com/404handler?src=search&p=food+delicious" -O test.html
In many Unix shells, putting an & after a command causes it to be executed in the background.
Wrap your URL in single quotes to avoid this issue.
i.e.
wget 'http://search.yahoo.com/404handler?src=search&p=food+delicious' -O test.html
if you are using a jubyter notebook, maybe check if you have downloaded
pip install wget
before warping from URL
Related
I need to download and run a firefox through a bash script, so I tried running the commands below:
curl -o ~/firefox.tar.bz2 https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64
tar xjf ~/firefox.tar.bz2
~/firefox/firefox
Yet already the first command fails to download the tar file.
Note: The OS is Ubuntu 16, and I don't want to use apt-get.
Quote the address, otherwise the shell interprets the ampersand as a shell order and it ends up trying to download something different to what you expect. Also, add the -L parameter to tell cURL to follow the links:
curl -L -o ~/firefox.tar.bz2 "https://download.mozilla.org/?product=firefox-latest-ssl&os=linux64"
I'm trying to set up a Bash Script (shl) that will use curl to download a file.
I really can't find a good bash script tutorial. I need assistance.
I've tried testing it with a windows bat file that has something like
: curl ${url} > file name [trying to see it work from windows]
and getting
Protocol "https" not supported or disabled in libcurl
the URL that I can use to extract the file would look something like this {example only)
https://bigstate.academicworks.com/api/v1/disbursements.csv?per_page=3&fields=id,disbursement_amount,portfolio_name,user_uid,user_display_name,portfolio_code,category_name&token=fcc28431bcb6771437861378aefe4a4474dbf9e503c78fd9a4db05924600c03b
I'm trying to put the file here \aiken\ProdITFileTrans\cofc_aw_disbursement.csv
so my bat file looks
#Echo On
curl --verbose -g ${https://bigstate.academicworks.com/api/v1/disbursements.csv?per_page=3&fields=id,disbursement_amount,portfolio_name,user_uid,user_display_name,portfolio_code,category_name&token=fcc28431bcb6771437861378aefe4a4474dbf9e503c78fd9a4db05924600c03b} >\\aiken\ProdITFileTrans\cofc_aw_disbursement.csv
PAUSE
Again the goal is to take a working version of this call in put in in a Bash shell that I can call forom ATOMIC/UC4
Once I have the bash script I want to be able to do a daily download of my file.
Well, perhaps something like:
#!/bin/bash
curl --verbose -g yourlongurlhere -o /path/to/your/file.csv
Make the file executable (chmod +x).
EDIT: check Advanced Bash Scripting Guide for tons of examples. It covers just about everything.
I want to download a program. For example "package.zip"
I want to download it in, for example "~/programs/downloaded"
I want to name it, for example "new.zip"
So I tried:
wget -P ~/programs/downloaded \
-O new.zip https://somewebsite.com/package.zip
But it only downloaded the package in the terminal's current directory and renamed it. The -P command does not work. Any idea how to make it work?
This is a feature of wget that has bitten many people. Unfortunately, it was a design decision taken many years ago and cannot be changed now for fear of breaking existing scripts. The crucial thing to understand here is that -O acts like shell redirection and hence is unaffected by the -P option.
The way to do what you want would be to directly provide the filename:
wget -O ~/programs/downloaded/new.zip <url>
The filename of my curl download target is unpredictable and globbing with an asterisk isn't possible. I can download the file using the following command, but only after I've determined its' name in advance:
curl -O -vvv -k -u user:password https://myURL/ws/myfile.zip
How can I tailor my curl command to succeed with an unpredictable target name?
There's no easy way to get a directory listing using HTTP. You can use curl to just print the HTML generated by the site. If there's an index with links to the files on that server, simply running
curl -s -u user:password https://myURL/ws/ | grep .zip
will print HTML-formatted links to the zip files available for download on that page.
Intro:
Like the OP, I had a similar issue scripting the download of a binary- for docker-compose- from Github because the version number keeps iterating making the file name unpredictable.
This is how I solved it. Might not be the tidiest solution, but if you have a more elegant way, ping me a comment and I'll update the answer.
Solution:
I merely used an auto-populating variable that takes the output of curl, prints the 1st line- which will be the most recent release- and thengrep for the release number prefaced by a "v". The result is saved to the the path /home/ubuntu as the arbitrary file name "docker-compose-latest"
curl -L "https://github.com/docker/compose/releases/download/$(curl https://github.com/docker/compose/releases | grep -m1 '<a href="/docker/compose/releases/download/' | grep -o 'v[0-9:].[0-9].[0-9]')/docker-compose-$(uname -s)-$(uname -m)" -o /home/ubuntu/docker-compose-latest
And we validate that we received the correct binary (I'm downloading to a Raspberry Pi which has an ARM processor on 64 bit Ubuntu 20.04 LTS:
file /home/ubuntu/docker-compose-latest
Produces the following feedback on the file:
/home/ubuntu/docker-compose-latest: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, Go BuildID=QqyJMzYMWOofWehXt3pb/T7U4zg-t8Xqz_11RybNZ/ukJOlZCpzQuZzBcwSK3b/d6ecQ2m2VfqKb_EQRUZA, stripped
To validate this solution works, just execute the above commands remembering to change the path of the file command if not using Ubuntu.
Conclusion:
Again, might not be the most elegant solution, but it's a solution for how one can download a target with curl that has an unpredictable filename.
I wrote a bash script that fetches lyrics from a website. The script is here --> http://scrippets.wordpress.com/2011/02/01/fetching-lyrics-of-songs-from-the-terminal/ (the indentations in the script are correct unlike how it looks on the blog)
This script works perfectly well when executed from the terminal. Now i created a custom keyboard shortcut using compiz commands, that executes the following command when the right key combination is pressed :
gnome-terminal --working-directory="/home/tapan/sandbox/bash/" --window-with-profile=lyrics -e "/home/tapan/sandbox/bash/lyrics.sh" -t "`rhythmbox-client --print-playing`"
I created a new profile called "lyrics" to give the terminal that opens up a custom look. When i open up a terminal with this profile and run the script, it works perfectly fine again. However, when i use the keyboard shortcut to run the custom command, i get the following error:
Pink Floyd - Is There Anybody Out There?
wget: missing URL
Usage: wget [OPTION]... [URL]...
Try `wget --help' for more options.
cat: 3.txt: No such file or directory
I cannot figure out whats wrong. I mean if it works perfectly well in the terminal normally, why shouldn't this work? Any suggestions?
PS: The script i have written is pretty elementary and noobish, so any suggestions to improve it are also welcome in the comments :)
EDIT: The output has changed a little, now it just shows the name of the song playing and nothing else. Though sometimes it still shows the wget error.
EDIT2: When i run that gnome terminal command from a terminal, it works. The problem is only when running it with the keyboard shortcut using compiz commands or if i use the run dialog (the alt+f2 one).
The two wget commands should probably have the url variables in double quotes, for example: wget -q -U Mozilla -O 1.txt $link should be wget -q -U Mozilla -O 1.txt "$link"
You need to uriencode your song title so that special characters like '?', '&', '%', and '+' are not passed literally in your URL.
name3=${name2//\?/%3F}
searchq=${name3// /+}
will handle the ?'s. I don't know of a more general solution in bash without resorting to one-line Perl or Python scripts.