return filename after downloading file with curl - macos

I would like to capture the filename to a variable of a file that is downloaded using curl. I am using the following flag to preserve the filename as in the below using --remote-name
My code:
file1=$(curl -O --remote-name 'https://url.com/download_file.tgz')
echo $file1


You can use the -w|--write-out switch of curl:
file1="$(curl -O --remote-name -s \
-w "%{filename_effective}" "https://url.com/download_file.tgz")"
echo "$file1"


file1=download_file.tgz
url=https://url.com/$file1 #encoding this might be necessary
curl -O --remote-name $url
echo $file1
If, to construct the URL, you need to know the filename that you want to download, then you don't need anything from curl to identify the file that it downloaded unless there is not a 1:1 relationship between the basename of the URL and file that was downloaded.

Related

How to use curl get image right filename and extension

My links just like below
https://cdn.sspai.com/2022/06/22/article/a88df95f3401d5b6c9d716bf31eeef33?imageView2/2/w/1120/q/90/interlace/1/ignore-error/1
If I use chrome to open this like and cmd + s
I will get the right filename and right extension png.
But if I use bash below, then it will no extension:
curl -J -O https://cdn.sspai.com/2022/06/22/article/a88df95f3401d5b6c9d716bf31eeef33?imageView2/2/w/1120/q/90/interlace/1/ignore-error/1
I just want to download image with right filename and extension.
a88df95f3401d5b6c9d716bf31eeef33.png
Same error include different image links below:
https://cdn.sspai.com/article/fa848601-4cdf-38b0-b020-7afd6efc4a7e.jpg?imageMogr2/auto-orient/quality/95/thumbnail/!800x400r/gravity/Center/crop/800x400/interlace/1

You can get the name from the URL itself.
url="YOUR-URL"
file="`echo "${url}" | sed 's|\?.*|.jpg|' | xargs basename`"
curl -o "${file}.tmp" "${url}"
mv "${file}.tmp" "${file}"
Hope it helps

Why does curl -o output contain sequences like "^[[38;5;250m", when "surf" output looks fine?

I want to output wttr.in in to a file with curl. The problem is that the output isn't how it would be when i just surf wttr.in.
What i did is:
curl wttr.in -o ~/wt.tex and curl wttr.in -o ~/wt
The output is like: <output>
It should be https://wttr.in.

I solved my self:
less -r -f -L wt.tex
-r controlls the binary characters
-f forces to open the the file with out asking.

curl script to extract tar files

I have a curl script that i use to get read files from a url. I now need to extract the tar files after they are downloaded.
for file in $(/usr/bin/curl 'https://linktofile.com/yourfile.txt' );
do
echo $file;
/usr/bin/curl -L -J -O "$file";
done;
output to the screen is
curl: Saved to filename 'your_file_1.tar'
How do I extract the file that was saved?
I tried adding tar -xvf $file but nothing happens.
How do I get the name of the file that was just saved?

Using -J makes the file's location depend on the Content-Disposition header, so you need to retrieve the filename it specifies. You can do so by specifying through -w the output of the curl command, which you will want to output the filename :
for url in $(/usr/bin/curl 'https://linktofile.com/yourfile.txt' ); do
filename=$(/usr/bin/curl -sLJOw '%{filename_effective}' "$url")
tar -xvf "$filename"
done

curl: (26) couldn't open file when the file is a variable

I am trying to upload a list of files to a server. This is the script that I have
files=$(shopt -s nullglob dotglob; echo /media/USB/*) > /dev/null 2>&1
if (( ${#files} ))
then
for file in $files
do
echo "Filename"
echo $file
curl -i -X POST -F files=#$file 192.168.1.122:5000/upload
done
Basically I am trying to take all of the files on a USB drive and upload them to my local server. The curl command is giving me trouble. I can move these files to drives that I mount on this system but I haven't been able to send them with the curl command. I have tried variations on #"$file" and #\"$file\" based on other related questions but I haven't been able to get this to work. However what is annoying is that when I do this:
curl -i -X POST -F files=#/absolute/path/to/my/file.txt 192.168.1.122:5000/upload
It works as I expect. How can I get this to work in my loop?

So I ended up figuring out a solution that I will share in case anyone else is having this problem. I am not sure exactly why this fixed it but I simply had to put quotes around the files=#$file in the curl command:
curl -i -X POST -F "files=#$file" 192.168.1.122:5000/upload
Leaving this here in case it is useful to someone down the line.

Save file to specific folder with curl command

In a shell script, I want to download a file from some URL and save it to a specific folder. What is the specific CLI flag I should use to download files to a specific folder with the curl command, or how else do I get that result?

I don't think you can give a path to curl, but you can CD to the location, download and CD back.
cd target/path && { curl -O URL ; cd -; }
Or using subshell.
(cd target/path && curl -O URL)
Both ways will only download if path exists. -O keeps remote file name. After download it will return to original location.
If you need to set filename explicitly, you can use small -o option:
curl -o target/path/filename URL

The --output-dir option is available since curl 7.73.0:
curl --create-dirs -O --output-dir /tmp/receipes https://example.com/pancakes.jpg

curl doesn't have an option to that (without also specifying the filename), but wget does. The directory can be relative or absolute. Also, the directory will automatically be created if it doesn't exist.
wget -P relative/dir "$url"
wget -P /absolute/dir "$url"

it works for me:
curl http://centos.mirror.constant.com/8-stream/isos/aarch64/CentOS-Stream-8-aarch64-20210916-boot.iso --output ~/Downloads/centos.iso
where:
--output allows me to set up the path and the naming of the file and extension file that I want to place.

Use redirection:
This works to drop a curl downloaded file into a specified path:
curl https://download.test.com/test.zip > /tmp/test.zip
Obviously "test.zip" is whatever arbitrary name you want to label the redirected file- could be the same name or a different name.
I actually prefer #oderibas solution, but this will get you around the issue until your distro supports curl version 7.73.0 or later-

For powershell in Windows, you can add relative path + filename to --output flag:
curl -L http://github.com/GorvGoyl/Notion-Boost-browser-extension/archive/master.zip --output build_firefox/master-repo.zip
here build_firefox is relative folder.

Use wget
wget -P /your/absolut/path "https://jdbc.postgresql.org/download/postgresql-42.3.3.jar"

For Windows, in PowerShell, curl is an alias of the cmdlet Invoke-WebRequest and this syntax works:
curl "url" -OutFile file_name.ext
For instance:
curl "https://airflow.apache.org/docs/apache-airflow/2.2.5/docker-compose.yaml" -OutFile docker-compose.yaml
Source: https://krypted.com/windows-server/its-not-wget-or-curl-its-iwr-in-windows/

Here is an example using Batch to create a safe filename from a URL and save it to a folder named tmp/. I do think it's strange that this isn't an option on the Windows or Linux Curl versions.
#echo off
set url=%1%
for /r %%f in (%url%) do (
set url=%%~nxf.txt
curl --create-dirs -L -v -o tmp/%%~nxf.txt %url%
)
The above Batch file will take a single input, a URL, and create a filename from the url. If no filename is specified it will be saved as tmp/.txt. So it's not all done for you but it gets the job done in Windows.

Resources