Okay, so I've got this small problem with a bash script that I'm writing.
This script is supposed to be run like this:
bash script.sh https://www.youtube.com/user/<channel name>
OR
bash script.sh https://www.youtube.com/user/<random characters that make up a youtube channel ID>
It downloads an entire YouTube channel to a folder named
<uploader>{<uploader_id>}/
Or, at least it SHOULD...
the problem I'm getting is that the archive.txt file that youtube-dl creates is not created in the same directory as the videos. It's created in the directory from which the script is run.
Is there a grep or sed command that I could use to get the archive.txt file to the video folder?
Or maybe create the folder FIRST, then cd into it, and run the command from there?
I dunno
Here is my script:
#!/bin/bash
pwd
sleep 1
echo "You entered: $1 for the URL"
sleep 1
echo "Now downloading all videos from URL "$1""
youtube-dl -iw \
--no-continue $1 \
-f bestvideo+bestaudio --merge-output-format mkv \
-o "%(uploader)s{%(uploader_id)s}/[%(upload_date)s] %(title)s" \
--add-metadata --download-archive archive.txt
exit 0
I ended up solving it with this:
uploader="$(youtube-dl -i -J $URL --playlist-items 1 | grep -Po '(?<="uploader": ")[^"]*')"
uploader_id="$(youtube-dl -i -J $URL --playlist-items 1 | grep -Po '(?<="uploader_id": ")[^"]*')"
uploaderandid="$uploader{$uploader_id}"
echo "Uploader: $uploader"
echo "Uploader ID: $uploader_id"
echo "Folder Name: $uploaderandid"
echo "Now downloading all videos from URL "$URL" to the folder "$DIR/$uploaderandid""
Basically I had to parse the JSON with grep, since the youtube-dl devs said that implementing -o type variables into any other variable would clog up the code and make it bloated.
Related
ok, I have these urls
https://www.ppppppppppp.com/it/yyyy/911-omicidio-al-telefono/stagione-1-appesa-a-un-filo
https://www.ppppppppppp.com/it/yyyy/avamposti-dispacci-dal-confine/stagione-1-cerignola
https://www.ppppppppppp.com/it/yyyy/belle-da-morire/stagione-1-bellezza-stalking
I try to create these folder with these names
911-omicidio-al-telefono
avamposti-dispacci-dal-confine
belle-da-morire
extracting the name from urls
for example I would like the file from the url
https://www.ppppppppppp.com/it/yyyy/911-omicidio-al-telefono/stagione-1-appesa-a-un-filo
to download directly inside the folder name extracted from the url
911-omicidio-al-telefono
but this seems problematic because no folder names are extracted and each file is downloaded outside their folderURL name
To solve this problem I created a script.sh with this code
#!/bin/bash
# Extract the folder name from the URL
url=$1
folder_name=$(echo $url | cut -d "/" -f6)
echo "folder_name: $folder_name"
if [[ "$folder_name" == "NA" ]]
then
echo "Can't extract folder name from $url"
exit
fi
# Create the folder if it doesn't exist
mkdir -p "$folder_name"
echo "file_path: $file_path"
# Download the video and audio files
ffmpeg -i "$file_path.fdash-video=6157520.mp4" -i "$file_path.fdash-audio_eng=160000.m4a" -c copy "$file_path.mp4"
# Move the file to the correct folder and rename it with .mp4 extension
mv "$file_path.mp4" "$folder_name/$file_path.mp4"
and then from bash terminal I call it in this way
yt-dlp --referer "https://ppppppppppp.com/" --add-header "Cookie:COOKIE" --batch-file links_da_scaricare.txt -o '%(playlist)s/%(title)s.%(ext)s' --exec "~/script.sh {}"
I use cygwin and script.sh is in C:\cygwin64\home\Administrator but I test also with ubuntu and problem is the same: it creates a folder called NA and download inside that folder.
All files are downloaded in the same NA folder and not in their folders, in other word are not downloaded in the folders having names extracted from the url from which the files are downloaded
EDIT
I use SpellCheck and I fix code of script.sh and now I haven't issues
#!/bin/bash
url=$1
file_path=$2
# Extract the folder name from the URL
folder_name=$(echo "$url" | cut -d "/" -f4)
echo "folder_name: $folder_name"
# Create the folder if it doesn't exist
mkdir -p "$folder_name"
echo "The script is running and creating folder: $folder_name" > ~/script.log
# Move the file to the correct folder and rename it with .mp4 extension
mv "$file_path" "$folder_name/$folder_name.mp4"
but when I try to run this command from Cygwin terminal
yt-dlp --referer "https://pppppppppp.com" --add-header "Cookie:COOKIE" --batch-file links_da_scaricare.txt -o '%(playlist)s/%(title)s.%(ext)s' --exec "C:\cygwin64\home\Administrator\script.sh {} $file_path"
NA folder is still created and no other folders are created so files are downloaded only into NA and not in their folders
I think you are trying to do something like this:
#!/bin/bash
#QUESTION: https://stackoverflow.com/questions/75088710/files-are-not-downloaded-in-the-folders-having-names-extracted-from-the-url
# Extract the folder name from the URL
#url=$1
url="https://www.ppppppppppp.com/it/yyyy/911-omicidio-al-telefono/stagione-1-appesa-a-un-filo"
folder_name=$(echo $url | cut -d "/" -f6 )
### Missing
if [ "${folder_name}" = "" ] ; then folder_name="NA" ; fi
echo "folder_name: $folder_name"
if [[ "$folder_name" == "NA" ]]
then
echo "Can't extract folder name from $url"
exit
fi
### Missing
file_path="$(pwd)/${folder_name}"
# Create the folder if it doesn't exist
mkdir -p "${file_path}"
echo "file_path: ${file_path}"
download="arbitrary_name"
wget -O "${file_path}/${download}" "${url}"
# Download the video and audio files
ffmpeg -i "${file_path}/${download}.fdash-video=6157520.mp4" -i "${file_path}/${download}.fdash-audio_eng=160000.m4a" -c copy "${file_path}/${download}.mp4"
Some of that can't be verified by myself, because I don't have an account to access primevideo, at
https://www.primevideo.com/detail/0S0UEN2OCD7CTY5TQF3N6KB1ET/ref=atv_dp_season_select_s1
Also, I don't think you can download the two, video and audio, separately.
I have a short bash script that is meant to convert a folder full of .mp3 files to .m4r ringtones. I am having trouble figuring out why it is throwing the following error:
"Error: ExtAudioFileCreateWithURL failed ('fmt?')"
#!/bin/bash
cd "/path/to/directory/containing/mp3s"
for i in *.mp3; do
baseFilename=$( basename ${i} .mp3 )
afconvert -f m4af ${i} -o "/path/to/new/location/${baseFilename}.m4r"
done
exit 0
The issue was that I had not specified the output data format. Found a helpful page that lead me to the answer:
http://support.moonpoint.com/os/os-x/audio/afconvert.php
#!/bin/bash
cd "/path/to/directory/containing/mp3s"
for i in *.mp3; do
baseFilename=$( basename "${i}" .mp3 )
afconvert -f m4af "${i}" -d aac "/path/to/new/location/${baseFilename}.m4r"
done
exit 0
I'm trying to set up a loop to monitor a file being written by Apple Compressor. Once the file is no longer being written, I'd like to change the name of the directory it's in. However, when I feed a variable containing the filepath to lsof it comes out garbled.
For instance, "/Users/leftright/Desktop/Output/${process##*/}_PROCESSING/" stored as $output is interpreted by lsof as DHt\x96?\x7f. I can't see anything in the lsof manpages to suggest why that's happening. It's being called in a if/then statement inside a function.
#!/bin/bash
compressor() {
filepath="$1"
process="${filepath%.*}"
output="/Users/leftright/Desktop/Output/${process##*/}_PROCESSING/"
filename="${filepath##*/}"
moving="${output}""${filepath##*/}"
cleanname="${filename%.*}"
final= "${output}""${cleanname}".mp4
completed="/Users/leftright/Desktop/Output/${process##*/}_COMPLETED/${cleanname}.mp4"
mkdir -m 777 "$output";
mv "$filepath" "$moving";
/Applications/Compressor.app/Contents/MacOS/Compressor -computergroup "This Computer" -jobpath "$output""$filename" -settingpath /Users/leftright/Documents/CONVERTHQTEST.cmprstng -locationpath "$final";
while true; do
if lsof "$final" > /dev/null; then
sleep 1
else
mv "$final" "$completed"
fi
done
}
export -f compressor
fswatch -0 -v --event Created /Users/leftright/Desktop/Watch | xargs -0 -n1 -I filepath bash -c 'compressor "filepath"'
What am I doing wrong here?
For those of you playing at home, I left whitespace in front of final= "${output}""${cleanname}".mp4 which caused it to be evaluated incorrectly. Cyrus' suggestion of shellcheck.net found the issue.
Im currently using 3 different commands to achieve my goal of downloading a zip, extracting it, converting the txt file to utf8 and then converting the csv to json!
First I have:
wget https://www.example.com/example.zip -O temp.zip; unzip -o temp.zip; rm temp.zip
Which is good, but the problem to start with is how do I rename the file that is extracted so its the same every time for the next processes as it can be a different name within the zip every day? Next I run this script depending on the filename that converts the ISO to utf8:
sh dir_iconv.sh example1.txt ISO8859-1 UTF-8
Which is this script:
#!/bin/bash
ICONVBIN='/usr/bin/iconv' # path to iconv binary
if [ $# -lt 3 ]
then
echo "$0 dir from_charset to_charset"
exit
fi
for f in $1/*
do
if test -f $f
then
echo -e "\nConverting $f"
/bin/mv $f $f.old
$ICONVBIN -f $2 -t $3 $f.old > $f
rm -f $f.old
else
echo -e "\nSkipping $f - not a regular file";
fi
done
And then finally I run a ruby script csv2json - https://github.com/darwin/csv2json - that is called as follows (pipe delimited) to give me a json output:
csv2json -s '|' example1.txt > example1.json
Is there a simple way to roll this into one command or script that can be called?
Pipe all your commands one after another and, if necessary, throw them in a shell script file.
wget -qO- https://www.example.com/example.zip | unzip | iconv -f ISO8859-1 -t UTF-8 | csv2json > example.json
I want to create a shellscript that reads files from a .diz file, where information about various source files are stored, that are needed to compile a certain piece of software (imagemagick in this case). i am using Mac OSX Leopard 10.5 for this examples.
Basically i want to have an easy way to maintain these .diz files that hold the information for up-to-date source packages. i would just need to update these .diz files with urls, version information and file checksums.
Example line:
libpng:1.2.42:libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:http://downloads.sourceforge.net/project/libpng/00-libpng-stable/1.2.42/libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks:9a5cbe9798927fdf528f3186a8840ebe
script part:
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
curl -L -v -O $url 2>> logfile.txt
$calculated_md5=`/sbin/md5 $file | /usr/bin/cut -f 2 -d "="`
echo $calculated_md5
done < "files.diz"
Actually I have more than just one question concerning this.
how to calculate and compare the checksums the best? i wanted to store md5 checksums in the .diz file and compare it with string comparison with "cut"ting out the string
is there a way to tell curl another filename to save to? (in my case the filename gets ugly libpng-1.2.42.tar.bz2?use_mirror=biznetnetworks)
i seem to have issues with the backticks that should direct the output of the piped md5 and cut into the variable $calculated_md5. is the syntax wrong?
Thanks!
The following is a practical one-liner:
curl -s -L <url> | tee <destination-file> |
sha256sum -c <(echo "a748a107dd0c6146e7f8a40f9d0fde29e19b3e8234d2de7e522a1fea15048e70 -") ||
rm -f <destination-file>
wrapping it up in a function taking 3 arguments:
- the url
- the destination
- the sha256
download() {
curl -s -L $1 | tee $2 | sha256sum -c <(echo "$3 -") || rm -f $2
}
while IFS=: read app version file url md5
do
echo "Downloading $app Version: $version"
#use -o for output file. define $outputfile yourself
curl -L -v $url -o $outputfile 2>> logfile.txt
# use $(..) instead of backticks.
calculated_md5=$(/sbin/md5 "$file" | /usr/bin/cut -f 2 -d "=")
# compare md5
case "$calculated_md5" in
"$md5" )
echo "md5 ok"
echo "do something else here";;
esac
done < "files.diz"
My curl has a -o (--output) option to specify an output file. There's also a problem with your assignment to $calculated_md5. It shouldn't have the dollar sign at the front when you assign to it. I don't have /sbin/md5 here so I can't comment on that. What I do have is md5sum. If you have it too, you might consider it as an alternative. In particular, it has a --check option that works from a file listing of md5sums that might be handy for your situation. HTH.