I have a problem with using the curl command: I have to download a file from a site that is of "www.example.com:8000/get.php?username=xxxx&password=xxxx" form.
Now what happens is this: If I open the link from the browser, if that file exists, part of the automatic download, but if the link is not correct, nothing happens and the displayed page is white.
My problem is that by using the command
curl -o file.txt "www.example.com:8000/get.php?username=xxxx&password=xxxx", the file is generated to me is if the link is correct, by downloading the file correctly, is that the link is not right, generating the 0 byte .txt file.
How can I do that, if not corrected the link (and therefore there is no file to download), no file is generated to me?
Related
I have access to a remote server over ssh. I have only read (no write) access on the server. There is a zipped log file that I want to read.
But because I have only read access, I cannot first extract the file and then read it, because when I try to unzip, I get the message Read-only file system.
My idea was to redirect the output of the gunzip-command to some other command that can read from the standrat input and display the content in the console. So I do not write the unzipped file on the file system (that I do not have the right) but display it directly in the console. Until now I couldn't successfully do it.
How to achieve this goal? And is there any better way to do it?
Since you do not have permission to unzip the file, you will first need to view the list of contents and their path. Once you get that then you can view content using -p option of unzip command.
View contents
zipinfo your.zip
View file contents
unzip -p latest.zip wordpress/wp-config-sample.php
In case it is a .gz file then use: gunzip -c wordpress/wp-config-sample.php
Hope this helps!
I am trying to automate a data downloading process. For this purpose, my goal is to extract (using bash commands) the .zip from a redirection link that could be seen on display here: https://journals.sagepub.com/doi/suppl/10.1177/0022002706289303
I have seen that people suggest the -L tag with curl for redirections, but it doesn't seem to work for my case. The specific command I have tried is:
curl -L -o output.zip https://journals.sagepub.com/doi/suppl/10.1177/0022002706289303/suppl_file/Sambanis_Aug_06.zip
The command file output.zip shows that the extracted .zip file is actually a HTML document text. On the other hand, clicking the redirection link (used inside curl command) downloads the extracted folder automatically via a browser.
Any ideas, tips, or suggestions on what I should try (or whether this is possible or not) will be highly appreciated!
If you execute curl with the --verbose option you can see that it is a cookie related problem. The cookie engine needs to be enabled. You can download the desired file as follows:
curl -b cookies.txt -L https://journals.sagepub.com/doi/suppl/10.1177/0022002706289303/suppl_file/Sambanis_Aug_06.zip -o test.zip
It doesn't matter if the file provided with the -b option doesn't exist. We just need to activate the cookie engine.
Refer to Send cookies with curl and Save cookies between two curl requests for futher information.
You can download that file with wget on Linux
$ wget https://journals.sagepub.com/doi/suppl/10.1177/0022002706289303/suppl_file/Sambanis_Aug_06.zip
$ unzip Sambanis_Aug_06.zip
Archive: Sambanis_Aug_06.zip
inflating: Sambanis (Aug 06).dta
inflating: Sambanis Appendix (Aug 06).pdf
I do not have much experiance with the command line and I have done my research and still wasn't able to solve my problem.
I need to download a .txt file from a folder on box.com.
I attempted using:
$ curl -o FILE URL
However, all I got was a empty text file that was named random numbers. I assumed the reason this happened is because the url of the file location does not end in .txt since it is in a file on box.com.
I also attempted:
$ wget FILE URL
However, my mac terminal doesn't seem to find that command
Is there a different command that can download the file from box.com? Or am I missing something?
You need to put your URL in quotes to avoid shell trying to parse it:
curl -o myfile.txt "http://example.com/"
Update: If the URL requires authentication
Modern browsers allow you to export requests as curl commands.
For example, in Chrome, you can:
open your file URL in a new tab
open Developer tools (View -> Developer -> Developer Tools)
switch to Network tab in the tools
refresh the page, a request should appear in the "Network" tab
Right-click the request, choose "Copy -> Copy as cURL"
paste the command in the shell
Here's how it looks for this page for example:
Dropbox makes it easy to programmatically download a single file via curl (EX: curl -O https://dl.dropboxusercontent.com/s/file.ext). It is a little bit trickier for a folder (regular directory folder, not zipped). The shared link for a folder, as opposed to a file, does not link directly to the zipped folder (Dropbox automatically zips the folder before it is downloaded). It would appear that you could just add ?dl=1 to the end of the link, as this will directly start the download in a browser. This, however, points to an intermediary html document that redirects to the actual zip folder and does not seem to work with curl. Is there anyway to use curl to download a folder via a shared link? I realize that the best solution would be to use the Dropbox api, but for this project it is important to keep it as simple as possible. Additionally, the solution must be incorporated into a bash shell script.
It does appear to be possible with curl by using the -L option. This forces curl to follow the redirect. Additionally, it is important to specify an output name with a .zip extension, as the default will be a random alpha-numeric name with no extension. Finally, do not forget to add the ?dl=1 to the end of the link. Without it, curl will never reach the redirect page.
curl -L -o newName.zip https://www.dropbox.com/sh/[folderLink]?dl=1
Follow redirects (use -L). Your immediate problem is that Curl is not following redirects.
Set a filename. (Optional)
Dropbox already sends a Content-Disposition Header with its Dropbox filename. There is no reason to specify the filename if you use the correct curl flags.
Conversely, you can force a filename using something of your choosing.
Use one of these commands:
curl https://www.dropbox.com/sh/AAbbCCEeFF123?dl=1 -O -J -L
Preserve/write the remote filename (-O,-J) and follows any redirects (-L).
This same line works for both individually shared files or entire folders.
Folders will save as a .zip automatically (based on folder name).
Don't forget to change the parameter ?dl=0 to ?dl=1 (see comments).
OR:
curl https://www.dropbox.com/sh/AAbbCCEeFF123?dl=1 -L -o [filename]
Follow any redirects (-L) and set a filename (-o) of your choosing.
NOTE: Using the -J flag in general:
WARNING: Exercise judicious use of this option, especially on Windows. A rogue server could send you the name of a DLL or other file that could possibly be loaded automatically by Windows or some third party software.
Please consult: https://curl.haxx.se/docs/manpage.html#OPTIONS (See: -O, -J, -L, -o) for more.
I am trying to download some videos from a website using wget. I used the syntax of wget.
The command is as follows:
wget https://some_link/download.mp4?lecture_id=5
The problem is that it is downloading an HTML file. However, when I right click on this link in the website and select Save target as, I get the video file which I want to save. Similarily, when I click on the link, it shows a video file that can be saved or opened.
I tried the following command but to no avail:-
wget -O vid.mp4 https://some_link/download.mp4?lecture_id=5
It created a .mp4 file but it didn't have any video in it. The size of this file was also equal to the size of the HTML file that was created before.
You can use wget with parameters to download files recursively (you can download html page with all resources needed by this page like graphics).
Alternatively you can download HTML page via wget, then look for the mp4 file using grep with regular expression and then again use wget do download only mp4 file.