sometime ago I asked what was wrong about a bash script I was trying to do and I got a great solution: What's wrong with this youtube-dl automatic script?
I kept modifying the script to work with different youtube-dl command combinations and to deal with my very unstable Internet connection (that's the reason for the while/do loop) and it kept working flawlessly, but when I tried to use that same script structure to download Youtube playlists starting from a specific item in the list (e.g.: item number 15) that's when I get an error. I'm still pretty much a newbie in bash script (obviously), so I don't know what's wrong.
The script in question is this:
#!/bin/bash
function video {
youtube-dl --no-warnings -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' --socket-timeout 15 --hls-use-mpegts -R 64 --fragment-retries 64 --prefer-free-formats --all-subs --embed-subs -f 'bestvideo[height<=720]+bestaudio/best[height<=720]' "$#"
}
read -p "url: " url
video "$url"
while [ $? -ne 0 ]; do sleep 5 && video "$url" ; done
clear && echo completed!
So, for example, if I try to download a playlist, I just write in my Terminal:
printf https://www.youtube.com/playlist?list=PLS1QulWo1RIYmaxcEqw5JhK3b-6rgdWO_ | list720
("list720" is the name of the script, of course) The script runs without problems and does exactly what I expect it to do.
But if I run in the Terminal:
printf --playlist-start=15 https://www.youtube.com/playlist?list=PLS1QulWo1RIYmaxcEqw5JhK3b-6rgdWO_ | list720
I get the following error:
bash: printf: --: invalid option
printf: usage: printf [-v var] format [arguments]
ERROR: '' is not a valid URL. Set --default-search "ytsearch" (or run youtube-dl "ytsearch:" ) to search YouTube
If I invert the order (1st the youtube URL and then the --playlist-start=15 command), the script downloads the whole playlist and omits the "--playlist-start" command.
I tried just running the youtube-dl command string directly in the terminal and added the "--playlist-start" and URL at the end and it runs perfectly:
youtube-dl --no-warnings -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' --socket-timeout 15 --hls-use-mpegts -R 64 --fragment-retries 64 --prefer-free-formats --all-subs --embed-subs -f 'bestvideo[height<=720]+bestaudio/best[height<=720]' --playlist-start=15 https://www.youtube.com/playlist?list=PLS1QulWo1RIYmaxcEqw5JhK3b-6rgdWO_
...so I assume the problem is with the script.
Any help is welcome, thanks!
A much better design is to accept any options and the URL as command-line arguments. Scripts which require interactive I/O are pesky to include in bigger scripts and generally harder to use (you lose the ability to use your shell's tab completion and command-line history etc).
#!/bin/bash
# Don't needlessly use Bash-only syntax for declaring a function
# Indent the code
video () {
youtube-dl --no-warnings \
-o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' \
--socket-timeout 15 --hls-use-mpegts -R 64 --fragment-retries 64 \
--prefer-free-formats --all-subs --embed-subs \
-f 'bestvideo[height<=720]+bestaudio/best[height<=720]' "$#"
}
until video "$#"; do
sleep 5
done
Clearing the screen after finishing seems hostile so I took that out, too.
Now if you want to pass additional parameters to youtube-dl just include them as parameters to your script:
list720 --playlist-start=15 'https://www.youtube.com/playlist?list=PLS1QulWo1RIYmaxcEqw5JhK3b-6rgdWO_'
You should also usually quote any URLs in case they contain shell metacharacters. See also When to wrap quotes around a shell variable?
Notice how we always take care to use double quotes around "$#"; omitting them in this case is simply an error.
Notice also how inside the function, "$#" refers to the function's arguments, whereas in the main script, it refers to the script's command-line arguments.
Tangentially, using printf without a format string is problematic, too. If you pass in a string which contains a per-cent character, that will get interpreted as a format string.
bash$ printf 'http://example.com/%7Efnord'
http://example.com/0.000000E+00fnord
The proper solution is to always pass a format string as the first argument.
bash$ printf '%s\n' 'http://example.com/%7Efnord'
http://example.com/%7Efnord
But you don't need printf to pass something as standard input. Bash has "here strings":
list720 <<<'http://example.com/%7Efnord'
(This would of course only work with your old version which read the URL from standard input; the refactored script in this answer doesn't work that way.)
SOLVED!
My brother (a "retired" programmer) took some time to evaluate how Bash script works and we figured a way of making the script work in a simpler way by just adding the youtube-dl commands and the Youtube URL as arguments.
The script changed a little bit, now it looks like this:
#!/bin/bash
function video() {
youtube-dl --no-warnings -o '%(playlist)s/%(playlist_index)s - %(title)s.%(ext)s' --socket-timeout 15 --hls-use-mpegts -R 64 --fragment-retries 64 --prefer-free-formats --all-subs --embed-subs -f 'bestvideo[height<=720]+bestaudio/best[height<=720]' "$#"
}
video $#
while [ $? -ne 0 ]; do sleep 5 && video $# ; done
clear && echo completed!
Now I just have to write in my terminal:
list720 --playlist-start=15 https://www.youtube.com/playlist?list=PLS1QulWo1RIYmaxcEqw5JhK3b-6rgdWO_
And it works exactly as I want it to.
Thank you very much for your help and suggestions!
In bash the character '-' at the begin of a command is used to set an option, if you wants to print --playlist... you should use the escape character '\'.
Try something like printf "\-\-playlist..."
correction : printf '%s' '--playlist...'
Related
My Goal
I'm writing a small Bash script, which uses entr, which is a utility to re-run arbitrary commands when it detects file-system events. My immediate goal is to pass entr a command which converts a given markdown file to HTML. entr will run this command every time the markdown file changes. A simplified but working script looks like:
# script 1
in="$1"
out="${in%.md}.html"
echo "$in" | entr pandoc "${in}" -o "${out}"
This works fine. The filename to be watched is supplied to entr on stdin. On detecting changes in that file, entr runs the command specified by its args. In this example that is pandoc, and all the args after it, to convert the markdown file to an HTML file.
For future reference, set -x shows that entr was invoked as we'd expect. (Throughout, lines starting with + show the output from set -x):
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
The problem
I want to look-up the command given to entr depending on the file-type of the
given input file. So the file-conversion command ends up in a variable, and I want to use that variable as the command-line args to entr. But I can't get the quoting right.
Again, simplified:
# script 2
in="$1"
out="${in%.md}.html"
cmd="pandoc \"${in}\" -o \"${out}\""
echo "$in" | entr "$cmd"
(shellcheck.net detects no issues on the above)
This fails. Because "$cmd" in the final line is in quotes, the entirety of $cmd
is treated as a single arg to entr:
+ entr 'pandoc "READ ME.md" -o "READ ME.html"'
entr tries to interpret the whole thing as the name of an executable, which
it cannot find:
entr: exec pandoc "READ ME.md" -o "READ ME.html": No such file or directory
So how should I modify script 2, to use the content of $cmd as the args to
entr?
What have I tried?
Check that $cmd is being formed as I expect? If I echo "$cmd" right after
it is defined in script 2, it looks exactly how I'd hope:
pandoc "READ ME.md" -o "READ ME.html"
I tried messing around with alternate ways of constructing cmd, such as:
cmd='pandoc "'"${in}"'" -o "'"${out}"'"'
but variations like this produce identical values of $cmd, and identical
behavior as script2.
Try not quoting the use of $cmd?
Since the final line of script 2 erroneously treats the whole of "$cmd"
as a single arg, and we want it to split up the words into seprate args
instead, maybe removing the quotes and using a bare $cmd is a step in the
right direction?
echo "$in" | entr $cmd
Predictably enough though, this splits $cmd up on every space, even the
ones inside our double-quotes:
+ entr pandoc '"READ' 'ME.md"' -o '"READ' 'ME.html"'
This makes Pandoc try, and fail, to open a file called "READ:
pandoc: "READ: openBinaryFile: does not exist (No such file or directory)
Try constructing $cmd using printf?
I notice printf -v can store output in a variable. How about using that
instead of assiging to cmd?
printf -v cmd 'pandoc "%s" -o "%s"' "$in" "$out"
Predictably enough, this produces the same results as script2. I tried some
speculative variations, such as %q in the format string, or using $in
and $out directly in the format string, but didn't stumble on anything
that seemed to help.
Try using the ${var#Q} form of parameter expansion.
echo "$in" | entr ${cmd#Q}
Tried with and without double quotes around the use of ${cmd#q}. No joy,
I guess I'm misunderstanding what #Q is for.
+ entr ''\''pandoc' '"READ' 'ME.md"' -o '"READ' 'ME.html"'\'''
entr: exec 'pandoc: No such file or directory
Details
I'm using Bash v5.1.16, in Pop!_OS 22.04, derived from Ubuntu 22.04 (Jammy).
The current 'apt' version of entr (v5.1) in Ubuntu Jammy (22.04) is too old
for my needs (e.g. the -z flag doesn't work.) so I'm compiling my own from
the latest v5.3 source release.
I know there are a lot of questions about quoting in Bash, but I don't see any that seem to match this. Apologies if I'm wrong.
Assemble the command as an array, instead of a string.
I read somewhere that maybe $# might do what I need, so I put the parts of $cmd into an array:
in="$1"
out="${in%.md}.html"
cmd=(pandoc "$in" -o "$out")
echo "$in" | entr "${cmd[#]}"
This correctly quotes the items in ${cmd[#]} which require it (e.g. have spaces in.)
+ entr pandoc 'READ ME.md' -o 'READ ME.html'
So ‘entr’ successfully calls ‘pandoc’, which successfully converts the documents. It works! I confess I did not expect that.
This approach seems viable for other similar situations, not just when invoking entr.
So I have a solution. It doesn't seem completely ideal for my future plans. I had visions of these 'file conversion commands' being configurable, and hence defined in a text file somewhere, so that users (==me, probably) could override them and define their own, and I'm not fluent enough with Bash to be sure how to go about that when commands are defined as arrays instead of strings.
I can't help but feel I've overlooked something simpler.
Use a shell to interpret the value of "$cmd":
echo "$in" | entr sh -c "$cmd"
This approach seems viable for other similar situations, not just when invoking entr.
Similarly, entr has a -s option which invokes a shell for you (chosen using the first word in $SHELL):
echo "$in" | entr -s "$cmd"
These both work well, at the minor cost of spawning an extra shell process.
I'm facing some problems to pass some environment parameters to docker run in a relatively generic way.
Our first iteration was to load a .env file into the environment via these lines:
set -o allexport;
. "${PROJECT_DIR}/.env";
set +o allexport;
And then manually typing the --env VARNAME=$VARNAME as options for the docker run command. But this can be quite annoying when you have dozens of variables.
Then we tried to just pass the file, with --env-file .env, and it seems to work, but it doesn't, because it does not play well with quotes around the variable values.
Here is where I started doing crazy/ugly things. The basic idea was to do something like:
set_docker_parameters()
{
grep -v '^$' "${PROJECT_DIR}/.env" | while IFS= read -r LINE; do
printf " -e %s" "${LINE}"
done
}
docker run $(set_docker_parameters) --rm image:label command
Where the parsed lines are like VARIABLE="value", VARIABLE='value', or VARIABLE=value. Blank lines are discarded by the piped grep.
But docker run complains all the time about not being called properly. When I expand the result of set_docker_parameters I get what I expected, and when I copy its result and replace $(set_docker_parameters), then docker run works as expected too, flawless.
Any idea on what I'm doing wrong here?
Thank you very much!
P.S.: I'm trying to make my script 100% POSIX-compatible, so I'll prefer any solution that does not rely on Bash-specific features.
Based on the comments of #jordanm I devised the following solution:
docker_run_wrapper()
{
# That's not ideal, but in any case it's not directly related to the question.
cmd=$1
set --; # Unset all positional arguments ($# will be emptied)
# We don't have arrays (we want to be POSIX compatible), so we'll
# use $# as a sort of substitute, appending new values to it.
grep -v '^$' "${PROJECT_DIR}/.env" | while IFS= read -r LINE; do
set -- "$#" "--env";
set -- "$#" "${LINE}";
done
# We use $# in a clearly non-standard way, just to expand the values
# coming from the .env file.
docker run "$#" "image:label" /bin/sh -c "${cmd}";
}
Then again, this is not the code I wrote for my particular use case, but a simplification that shows the basic idea. If you can rely on having Bash, then it could be much cleaner, by not overloading $# and using arrays.
I'm trying to use enscript to print PDFs from Mutt, and hitting character encoding issues. One way around them seems to be to just use sed to replace the problem characters: sed -ir 's/[“”]/"/g' {input}
My test input file is this:
“very dirty”
we’re
I'm hoping to get "very dirty" and we're but instead I'm still getting
â\200\234very dirtyâ\200\235
weâ\200\231re
I found a nice little post on printing to PDFs from Mutt that I used as a starting point. I have a bash script that I point to from my .muttrc with set print_command="$HOME/.mutt/print.sh" -- the script currently reads about like this:
#!/bin/bash
input="$1" pdir="$HOME/Desktop" open_pdf=evince
# Straighten out curly quotes
sed -ir 's/[“”]/"/g' $input
sed -ir "s/[’]/'/g" $input
tmpfile="`mktemp $pdir/mutt_XXXXXXXX.pdf`"
enscript --font=Courier8 $input -2r --word-wrap --fancy-header=mutt -p - 2>/dev/null | ps2pdf - $tmpfile
$open_pdf $tmpfile >/dev/null 2>&1 &
sleep 1
rm $tmpfile
It does a fine job of creating a PDF (and works fine if you give it a file as an argument) but I can't figure out how to fix the curly quotes.
I've tried a bunch of variations on the sed line:
input=sed -r 's/[“”]/"/g' $input
$input=sed -ir "s/[’]/'/g" $input
Per the suggestion at Can I use sed to manipulate a variable in bash? I also tried input=$(sed -r 's/[“”]/"/g' <<< $input) and I get an error: "Syntax error: redirection unexpected"
But none manages to actually change $input -- what is the correct syntax to change $input with sed?
Note: I accepted an answer that resolved the question I asked, but as you can see from the comments there are a couple of other issues here. enscript is taking in a whole file as a variable, not just the text of the file. So trying to tweak the text inside the file is going to take a few extra steps. I'm still learning.
On Editing Variables In General
BashFAQ #21 is a comprehensive reference on performing search-and-replace operations in bash, including within variables, and is thus recommended reading. On this particular case:
Use the shell's native string manipulation instead; this is far higher performance than forking off a subshell, launching an external process inside it, and reading that external process's output. BashFAQ #100 covers this topic in detail, and is well worth reading.
Depending on your version of bash and configured locale, it might be possible to use a bracket expression (ie. [“”], as your original code did). However, the most portable thing is to treat “ and ” separately, which will work even without multi-byte character support available.
input='“hello ’cruel’ world”'
input=${input//'“'/'"'}
input=${input//'”'/'"'}
input=${input//'’'/"'"}
printf '%s\n' "$input"
...correctly outputs:
"hello 'cruel' world"
On Using sed
To provide a literal answer -- you almost had a working sed-based approach in your question.
input=$(sed -r 's/[“”]/"/g' <<<"$input")
...adds the missing syntactic double quotes around the parameter expansion of $input, ensuring that it's treated as a single token regardless of how it might be string-split or glob-expanded.
But All That May Not Help...
The below is mentioned because your test script is manipulating content passed on the command line; if that's not the case in production, you can probably disregard the below.
If your script is invoked as ./yourscript “hello * ’cruel’ * world”, then information about exactly what the user entered is lost before the script is started, and nothing you can do here will fix that.
This is because $1, in that scenario, will only contain “hello; ’cruel’ and world” are in their own argv locations, and the *s will have been replaced with lists of files in the current directory (each such file substituted as a separate argument) before the script was even started. Because the shell responsible for parsing the user's command line (which is not the same shell running your script!) did not recognize the quotes as valid at the time when it ran this parsing, by the time the script is running, there's nothing you can do to recover the original data.
Abstract: The way to use sed to change a variable is explored, but what you really need is a way to use and edit a file. It is covered ahead.
Sed
The (two) sed line(s) could be solved with this (note that -i is not used, it is not a file but a value):
input='“very dirty”
we’re'
sed 's/[“”]/\"/g;s/’/'\''/g' <<<"$input"
But it should be faster (for small strings) to use the internals of the shell:
input='“very dirty”
we’re'
input=${input//[“”]/\"}
input=${input//[’]/\'}
printf '%s\n' "$input"
$1
But there is an underlying problem with your script, you are trying to clean an input received from the command line. You are using $1 as the source of the string. Once somebody writes:
./script “very dirty”
we’re
That input is lost. It is broken into shell's tokens and "$1" will be “very only.
But I do not believe that is what you really have.
file
However, you are also saying that the input comes from a file. If that is the case, then read it in with:
input="$(<infile)" # not $1
sed 's/[“”]/\"/g;s/’/'\''/g' <<<"$input"
Or, if you don't mind to edit (change) the file, do this instead:
sed -i 's/[“”]/\"/g;s/’/'\''/g' infile
input="$(<infile)"
Or, if you are clear and certain that what is being given to the script is a filename, like:
./script infile
You can use:
infile="$1"
sed -i 's/[“”]/\"/g;s/’/'\''/g' "$infile"
input="$(<"$infile")"
Other comments:
Then:
Quote your variables.
Do not use the very old `…` syntax, use $(…) instead.
Do not use variables in UPPER case, those are reserved for environment variables.
And (unless you actually meant sh) use a shebang (first line) that targets bash.
The command enscript most definitively requires a file, not a variable.
Maybe you should use evince to open the PS file, there is no need of the step to make a pdf, unless you know you really need it.
I believe that is better use a file to store the output of enscript and ps2pdf.
Do not hide the errors printed by the commands until everything is working as desired, then, just call the script as:
./script infile 2>/dev/null
Or as required to make it less verbose.
Final script.
If you call the script with the name of the file that enscript is going to use, something like:
./script infile
Then, the whole script will look like this (runs both in bash or sh):
#!/usr/bin/env bash
Usage(){ echo "$0; This script require a source file"; exit 1; }
[ $# -lt 1 ] && Usage
[ ! -e $1 ] && Usage
infile="$1"
pdir="$HOME/Desktop"
open_pdf=evince
# Straighten out curly quotes
sed -i 's/[“”]/\"/g;s/’/'\''/g' "$infile"
tmpfile="$(mktemp "$pdir"/mutt_XXXXXXXX.pdf)"
outfile="${tmpfile%.*}.ps"
enscript --font=Courier10 "$infile" -2r \
--word-wrap --fancy-header=mutt -p "$outfile"
ps2pdf "$outfile" "$tmpfile"
"$open_pdf" "$tmpfile" >/dev/null 2>&1 &
sleep 5
rm "$tmpfile" "$outfile"
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
We don’t allow questions seeking recommendations for books, tools, software libraries, and more. You can edit the question so it can be answered with facts and citations.
Closed 5 years ago.
Improve this question
Of course a bash file cannot be truly obfuscated and will always be readable. And I don't want to wrap them in some binary package.
And renaming local variables wouldn't be worth the trouble.
But is there a reliable simple bash obfuscator or minifier which at least removes all the indentation, all the empty lines and all whitespace without breaking anything? And especially the comments and commented out portions of the script which might contain sensitive documents or information?
I would be afraid of simple grep/sed-lines to do this because "HEREDOCs" must not be modified of course, so a little bit of real parsing is necessary.
Maybe there's a tool to do this, that would be great!
:P here is something funny.
say your script is named origin and the obfuscated one is named obsf.
here is origin:
#!/bin/sh
echo "fooo"
here is obsf
$ echo "echo $(base64 origin)" > obsf
$ cat obsf
echo IyEvYmluL3NoCmVjaG8gImZvb28iCg==
$ chmod +x obsf
now rm origin and run obsf like this:
$ sh obsf | base64 -d | sh
fooo
heh :3
Here is a tool that I created for bash scripts minification: https://github.com/precious/bash_minifier – it tries to remove all comments and as many spaces/tabs/newlines as possible. It is also available as a service here https://bash-minifier.appspot.com/.
To minify your bash script run this command:
python minifier.py /path/to/shell/script.sh
Even though this is an old question, it seems to be popular on Google. I was also looking for a bash minifer/obfuscator, and didn't like the one in the answer.
I didn't want to add gibberish either, or compile the script. So I wrote one that did what I wanted in Perl, and put it on GitHub at https://github.com/Aralhach/bashobfus/tree/master
It can:
Flatten indentation
Remove full-line comments (except the initial hashbang (#!)) and blank lines
Renames all lowercase variables in basic declarations (to avoid renaming variables like PATH), for loops, array access, and 'read' statements.
There might be some case where it fails, but I tested it with a fairly big bash script, and the listed cases of variable replacing popped up. It also leaves variables within single quotes (') alone --this popped up when printing an AWK script-- but replaces between single quotes (') when the statement is already inside double quotes (") --this popped up when printing an MySQL statement.
This makes me think I've covered all the big use cases, but I've been wrong before. If you find any bugs, feel free to report them (or fix them! :D). I was also thinking of adding a feature to join short lines into one with ";" but the cases were too many to analyze while making my deadline.
I hope people find it useful!
Original file script.sh:
#!/usr/bin/env bash
echo "foo"
Create other.sh
$ echo '#!/usr/bin/env bash' > other.sh
$ echo "echo '$(base64 script.sh)' | base64 -d | sh" >> other.sh
$ chmod +x other.sh
Result (cat other.sh):
#!/usr/bin/env bash
echo 'IyEvdXNyL2Jpbi9lbnYgYmFzaAplY2hvICJmb28iCg==' | base64 -d | sh
Try:
$ ./other.sh
foo
Minification and Obfuscation of a shell script are two different things.
Minification means reducing the size of a script by removing all unnecessary characters from source code without changing its functionality. Obfuscation on the other hand means making the script difficult, if not impossible, to read.
Minification:
To minify a big script, you can run the following code against the actual script you want to minify:
#!/bin/sh
Script=${1}
if [ ! -z "${Script}" ] && [ -f ${Script} ] ; then
CurrenTime=$(date | sed -e 's~ ~_~g' -e 's~:~~g')
cp ${Script} ${Script}_${CurrenTime}
#### Remove all empty lines
#### Remove lines that begin with spaces and a comment sign #
#### Remove all comment lines (meaning, lines that begin with a "#")
awk '
(/.*/ || /#!/) && (!/^#$/) &&
(!/^#[[:blank:]]/) && (!/^#[a-z]/) &&
(!/^#[A-Z]/) && (!/^##/) &&
(!/^\t#/) && (!/^[[:space:]]*$/) &&
( /^#.*!/ || !/^[[:space:]]*#/)
' ${Script} | sed 's_^[[:space:]]*__g' > ${Script}.tmp 2>/dev/null
#' ${Script} > ${Script}.tmp 2>/dev/null (comment out the above line and uncomment this line if your HEREDOCS are affected)
ExitCode=$?
if [ ${ExitCode} -eq 0 ] && [ -s ${Script}.tmp ] ; then
echo
echo "SUCCESS: Your newly [ minified ] script can be found here [ ${Script}.tmp ]."
echo "Review the script [ ${Script}.tmp ] and if happy with it, replace your original script with it!"
echo "NOTE: Your original script [ ${Script} ] was backed up as [ ${Script}_${CurrenTime} ]!"
echo
exit 0
else
echo
echo "FAILURE: Unable to [ minify ] the specified script [ ${Script} ]!!!"
echo
exit 2
fi
else
echo
echo "USAGE: ${0} <your-script>"
echo
exit 3
fi
Note, minification tends to make a difference only if the script being minified is big...with several hundred or even thousands of lines. I was able to trim off a few Megabytes from a script using the above code.
Obfuscation:
After the above minification is completed, you can just stop right there if size reduction is what you're going for. If however, after the minification, you also wish to obfuscate your script as well, you have options.
The simplest way to obfuscate your script is through the use of encryption tools such as as Openssl.
To encrypt your script using Openssl:
1. cat <your-script> | openssl aes-128-cbc -a -salt -k "specify-a-password" > yourscript.enc
OR
2. openssl aes-128-cbc -a-salt -in <path-to-your-script> -k "yourpassword"
To decrypt a script using Openssl (notice the '-d'):
1. cat yourscript.enc | openssl aes-128-cbc -a -d -salt -k "specify-a-password" > yourscript.dec
OR
2. openssl aes-128-cbc -a -d -salt -in <path-to-your-script> -k "yourpassword" > yourscript.dec
Encryption/Obfuscation:
If your ultimate goal is to make it difficult to others to read your script, try pasting it here to have an encrypted copy generated for you.
In case you change your mind about SHC, the latest version can be downloaded here.
A Tool to obfuscate shell scripts:
http://www.comp.eonworks.com/scripts/obfuscate_shell_script-20011012.html
Sort of silly to do, but that's up to you. There are also ways to "compile" your shell script into an executable. This post's accepted answer gives several links with tools to do that.
Based on c00kiemon5ter idea, here you have THE script
Your twisted recursive minds will love it, since this is not the original script, but the obfuscated(obfuscated(original))
#!/bin/bash
#
# Usage:
# obfuscate scrript.sh > script_obfuscated.sh
#
PIXIE=$(mktemp)
base64 -d >${PIXIE}<<DIXIE
IyEvYmluL2Jhc2ggClBJWElFPSQobWt0ZW1wKQpiYXNlNjQgLWQgID4ke1BJWElFfTw8RElYSUUK
SXlFdlltbHVMMkpoYzJnS2FXWWdXeUFnTFdZZ0lpUXhJaUJkSUFwMGFHVnVDbU5oZENBOFBGQkpX
RWxGSUFvaklTOWlhVzR2WW1GegphQ0FLVUVsWVNVVTlYQ1FvYld0MFpXMXdLUXBpWVhObE5qUWdM
V1FnSUQ1Y0pIdFFTVmhKUlgwOFBFUkpXRWxGQ2lRb1ltRnpaVFkwCklDUXhLUXBFU1ZoSlJRcHpi
M1Z5WTJVZ1hDUjdVRWxZU1VWOUNuSnRJQzF5WmlCY0pIdFFTVmhKUlgwS1VFbFlTVVVLWlhocGRD
QXcKQ21acENtTmhkRHc4VGtWU1JBb2dJQ0IxYzJGblpUb2diMkoxYzJOaGRHVWdjMk55YVhCMENn
b2dJQ0JYYVd4c0lHZGxibVZ5WVhSbApJQ0p6WTNKcGNIUXViMkp6YUNJS1RrVlNSQW89CkRJWElF
CnNvdXJjZSAke1BJWElFfQpybSAtcmYgJHtQSVhJRX0K
DIXIE
source ${PIXIE}
rm -rf ${PIXIE}
I have a web server that saves the logs files of a web application numbered. A file name example for this would be:
dbsclog01s001.log
dbsclog01s002.log
dbsclog01s003.log
The last 3 digits are the counter and they can get sometime up to 100.
I usually open a web browser, browse to the file like:
http://someaddress.com/logs/dbsclog01s001.log
and save the files. This of course gets a bit annoying when you get 50 logs.
I tried to come up with a BASH script for using wget and passing
http://someaddress.com/logs/dbsclog01s*.log
but I am having problems with my the script.
Anyway, anyone has a sample on how to do this?
thanks!
#!/bin/sh
if [ $# -lt 3 ]; then
echo "Usage: $0 url_format seq_start seq_end [wget_args]"
exit
fi
url_format=$1
seq_start=$2
seq_end=$3
shift 3
printf "$url_format\\n" `seq $seq_start $seq_end` | wget -i- "$#"
Save the above as seq_wget, give it execution permission (chmod +x seq_wget), and then run, for example:
$ ./seq_wget http://someaddress.com/logs/dbsclog01s%03d.log 1 50
Or, if you have Bash 4.0, you could just type
$ wget http://someaddress.com/logs/dbsclog01s{001..050}.log
Or, if you have curl instead of wget, you could follow Dennis Williamson's answer.
curl seems to support ranges. From the man page:
URL
The URL syntax is protocol dependent. You’ll find a detailed descrip‐
tion in RFC 3986.
You can specify multiple URLs or parts of URLs by writing part sets
within braces as in:
http://site.{one,two,three}.com
or you can get sequences of alphanumeric series by using [] as in:
ftp://ftp.numericals.com/file[1-100].txt
ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
ftp://ftp.letters.com/file[a-z].txt
No nesting of the sequences is supported at the moment, but you can use
several ones next to each other:
http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html
You can specify any amount of URLs on the command line. They will be
fetched in a sequential manner in the specified order.
Since curl 7.15.1 you can also specify step counter for the ranges, so
that you can get every Nth number or letter:
http://www.numericals.com/file[1-100:10].txt
http://www.letters.com/file[a-z:2].txt
You may have noticed that it says "with leading zeros"!
You can use echo type sequences in the wget url to download a string of numbers...
wget http://someaddress.com/logs/dbsclog01s00{1..3}.log
This also works with letters
{a..z} {A..Z}
Not sure precisely what problems you were experiencing, but it sounds like a simple for loop in bash would do it for you.
for i in {1..999}; do
wget -k http://someaddress.com/logs/dbsclog01s$i.log -O your_local_output_dir_$i;
done
You can use a combination of a for loop in bash with the printf command (of course modifying echo to wget as needed):
$ for i in {1..10}; do echo "http://www.com/myurl`printf "%03d" $i`.html"; done
http://www.com/myurl001.html
http://www.com/myurl002.html
http://www.com/myurl003.html
http://www.com/myurl004.html
http://www.com/myurl005.html
http://www.com/myurl006.html
http://www.com/myurl007.html
http://www.com/myurl008.html
http://www.com/myurl009.html
http://www.com/myurl010.html
Interesting task, so I wrote full script for you (combined several answers and more). Here it is:
#!/bin/bash
# fixed vars
URL=http://domain.com/logs/ # URL address 'till logfile name
PREF=logprefix # logfile prefix (before number)
POSTF=.log # logfile suffix (after number)
DIGITS=3 # how many digits logfile's number have
DLDIR=~/Downloads # download directory
TOUT=5 # timeout for quit
# code
for((i=1;i<10**$DIGITS;++i))
do
file=$PREF`printf "%0${DIGITS}d" $i`$POSTF # local file name
dl=$URL$file # full URL to download
echo "$dl -> $DLDIR/$file" # monitoring, can be commented
wget -T $TOUT -q $dl -O $file
if [ "$?" -ne 0 ] # test if we finished
then
exit
fi
done
At the beggiing of the script you can set URL, log file prefix and suffix, how many digits you have in numbering part and download directory. Loop will download all logfiles it found, and automaticaly exit on first non-existant (using wget's timeout).
Note that this script assumes that logfile indexing starts with 1, not zero, as you mentioned in example.
Hope this helps.
Here you can find a Perl script that looks like what you want
http://osix.net/modules/article/?id=677
#!/usr/bin/perl
$program="wget"; #change this to proz if you have it ;-)
my $count=1; #the lesson number starts from 1
my $base_url= "http://www.und.nodak.edu/org/crypto/crypto/lanaki.crypt.class/lessons/lesson";
my $format=".zip"; #the format of the file to download
my $max=24; #the total number of files to download
my $url;
for($count=1;$count<=$max;$count++) {
if($count<10) {
$url=$base_url."0".$count.$format; #insert a '0' and form the URL
}
else {
$url=$base_url.$count.$format; #no need to insert a zero
}
system("$program $url");
}
I just had a look at the wget manpage discussion of 'globbing':
By default, globbing will be turned on if the URL contains a globbing character. This option may be used to turn globbing on or off permanently.
You may have to quote the URL to protect it from being expanded by your shell. Globbing makes Wget look for a directory listing, which is system-specific. This is why it currently works only with Unix FTP servers (and the ones emulating Unix "ls" output).
So wget http://... won't work with globbing.
Check to see if your system has seq, then it would be easy:
for i in $(seq -f "%03g" 1 10); do wget "http://.../dbsclog${i}.log"; done
If your system has the jot command instead of seq:
for i in $(jot -w "http://.../dbsclog%03d.log" 10); do wget $i; done
Oh! this is a similar problem I ran into when learning bash to automate manga downloads.
Something like this should work:
for a in `seq 1 999`; do
if [ ${#a} -eq 1 ]; then
b="00"
elif [ ${#a} -eq 2 ]; then
b="0"
fi
echo "$a of 231"
wget -q http://site.com/path/fileprefix$b$a.jpg
done
Late to the party, but a real easy solution that requires no coding is to use the DownThemAll Firefox add-on, which has the functionality to retrieve ranges of files. That was my solution when I needed to download 800 consecutively numbered files.