What is the best way to override an existing shell command? - bash

If one were to have a need to do modify an existing shell command (built-in or otherwise), what is the best way to go about this?
Let me state that I understand there are a number of potential risks involved in attempting a procedure such as this, but that is the very reason I ask this question.
I would like to create a shell function (called ping) to use in place of the common ping command. For the sake of this topic, let's say that it should at the very least be compatible in both Bash and Zsh shell environments.
Specifically, I would like to allow for ping to accept full URLs in the hostname (with protocol, trailing slash/pathname, query parameters, etc). The string manipulation is not what is stumping me, but rather how to properly execute the original ping executable without calling the function something other than ping.
For example, the following two commands should yield the same result:
# EXAMPLE 2 (What `ping` will accept)
ping who.is
# EXAMPLE 1 (What I would like to be able to do with `ping`)
ping https://who.is/my/ping/?example=this

A shim might look like the following (if your real ping is in /usr/bin):
#!/usr/bin/env bash
uri_re='^[[:alnum:]]+://([^/]+)/'
if [[ $1 =~ $uri_re ]]; then
exec /usr/bin/ping "${BASH_REMATCH[1]}" "${#:2}"
else
exec /usr/bin/ping "$#"
fi
Put it somewhere like /opt/overrides/bin, and put that earlier in the PATH than /usr/bin (so PATH=/opt/overrides/bin:/bin:/usr/bin:/usr/local/bin or such).
Or, for a portable function that will work on all POSIX-compliant shells (and zsh as well, even though it doesn't try to be one):
ping() {
local _ping_addr >/dev/null 2>&1 ||: "ignore failure on shells that don't support local"
case $1 in
*://*/)
_ping_addr=${1#*://}
_ping_addr=${_ping_addr%%/*}
shift
command ping "$_ping_addr" "$#"
;;
*)
command ping "$#" ;;
esac
}

I'd like to address #blizzrdof77's original question (paraphrased below):
"I would like to create a shell function (called 'ping') to use in place of the common ping🛈 command." [ℹ]
This drop-in portable shell function [ℹ] works well on Mac & Linux in zsh,bash,sh, and all POSIX-compliant shells.
# ---------------------------------------------------------
# Better `ping` support for various URL formats/protocols
#
# #param $1 - hostname
# EXAMPLE USAGE: `ping http://google.com/`
# ---------------------------------------------------------
ping() {
local pingdomain="$1"
shopt -s nocasematch # allow lowercase
pingdomain=${pingdomain/#*:\/\/} # strip protocol
pingdomain=${pingdomain/#*:*#/} # strip leading 'user:pass#'
pingdomain=$(echo "${pingdomain//"?"//}") # remove '?'
pingdomain="$(echo "$pingdomain" | cut -d/ -f 1)" # clear last '/'
command ping $pingdomain
}

Related

Bash command completion with full path expansion injected into history for vim

i've spent a solid week searching online and trying many different ways to solve a tricky problem. basically i would like to use vim to edit custom commands / scripts that are in my $PATH without having to actually cd to their given directories first or manually type their full paths on the command line.
in essence, i'd love to be able to combine stock bash command completion (compgen -c) with simultaneous path expansion when specifying scripts in my $PATH as vim FILE ARGUMENTS. btw i'm using the caps to make clear what can be a tricky subject and not shouting.
it's probably easier to show you what i'm trying to do then explain it. lets say i have scripts in directories that are on my $PATH
~/bin/x/y/cmd1.sh
~/bin/a/b/cmd2.sh
/ppp/n/m/cmd3.sh
sometimes these scripts provide functionality on files that exist in other directories so i'd like to be able to edit them easily from anywhere in the file system. sometimes i just want to be able to edit those scripts from other directories because it's more convenient. lets say i'm currently in the following directory.
/completely/different/dir
but now i need to vim edit
~/bin/a/b/cmd2.sh
my options to achieve this solely with default bash functionality is to do one of the following which takes a long time
cd ~/bin/a/b/; vim cmd.sh
vim ~/<tab-complete-my-way-to-file>
open a new terminal window plus some combination of the above
since i know the names of my custom scripts it would be soooo much easier to just do the following which requires no tab completion of the full path to the file or directory as well as no cd'ing to a different directory to change my context!!!
vim cmd2.sh
but this won't work by default b/c vim needs the full path to the script
my first thought was to write a vim wrapper function which basically uses which to do the $PATH expansion for me and then tie bash command completion to my vc function like this:
vc () { vim $(which "$#"); }
complete -c vc
i can run the following in the shell to complete partial script names that start with "c" from the choices of cmd1.sh, cmd2.sh, cmd3.sh
vc c<tab>
until i get what i want here which is great
vc cmd2.sh
when i hit enter and execute the command it all works fine BUT it doesn't inject the expanded path into the READLINE command line and thus the FULL EXAPANDED PATH of 'cmd2.sh' never winds up in my command history! my history will show this
vc cmd2.sh
instead of
vc ~/bin/a/b/cmd2.sh
or
vim ~/bin/a/b/cmd2.sh
i want that expanded path in my command history because it makes future operations on that script file super easy when reusing command history. ie i can ls, file, diff, mv, cp that expanded path much easier reusing history than writing more wrapper scripts for ls, file, diff, mv, cp etc.. like i had to do with vc above.
QUESTIONS :
OPTION 1
is there a way to reinject the full expanded path provided by which in my vc function directly back into the original vc READLINE or just inject the entire "vim " command that actually gets executed in vc as a replacement for the original vc command into READLINE? any method that allows me to get the expanded vim command into the history even if it is in addition to the original vc command is ok by me.
basically how do you access and edit the current READLINE programmatically in bash?
OPTION 2
note i can also do something like this DIRECTLY on the command line in real-time
vim $(which cmd2.sh) C-x-e
which gives me what i want (it expands the path which will then put it into history) but i have to always type the extra subshell and which text as well as the C-x-e (to expand the line) on every iteration of the command while losing the command completion functionality which basically makes this useless. put another way, is there anyway to automate the above using a bind key so that
vc cmd2.sh
is automatcially transformed first into
vim $(which cmd2.sh)
and then automatically follows up with C-x-e so that it gets expanded to
vim ~/bin/a/b/cmd2.sh
but have all the editing movement, text insertion and final command line expansion happen all in the same bindkey macro? this might be the best solution of all.
OPTION 3
alternatively, since bash command completion automatically winds up in the READLINE and thus the history, a custom completion function would solve my problem. is there a way to make vc use a completion function that would BOTH complete commands in $PATH when used as vim arguments as described above AND ALSO SIMULTANEOUSLY EXPAND THEM TO THEIR FULL PATHS?
i know how to write a basic completion function. countless hours of attempts (which i am choosing not to put here to keep confusion / post length down) are failing for the simple reason that i'm not sure command completion is compatible with simultaneous full path expansion b/c it breaks traditional completion.
with a custom completion function, here's what happens when i try to find one of my scripts "cmd2.sh" living in "vim ~/bin/a/b/cmd2.sh" but start with a "c" and hit "".
vim c<tab>
instead of getting me these completions to choose from
cmd1.sh
cmd2.sh
cmd3.sh
it completes the first one it finds in the $PATH and inserts it into the READLINE which might be
/ppp/n/m/cmd3.sh
when i really want
~/bin/a/b/cmd2.sh
this effectively kills the completion lookup because the word before my cursor in the READLINE now starts with /ppp/n/m/cmd3.sh and there's no way of getting back to cmd2.sh
i hope that's clear.
thanks
This requires some boilerplate in your .bashrc file, but might work for you. It makes use of the directory stack (some might say it abuses the directory stack, but if you aren't using it for anything else, it might be OK).
In your .bashrc, add each directory of interest to your directory stack. End the list with your home directory, as pushd also changes your current working directory.
pushd ~/bin/x/y/cmd1.sh
pushd ~/bin/a/b/cmd2.sh
pushd /ppp/n/m/cmd3.sh
pushd ~
Yes, it duplicates your PATH entry a bit, but I contend you don't really need access to every directory in your PATH, just the ones where you have files you intend to edit. (Are you really going to try to edit anything in /bin or /usr/bin?)
Now, in your interactive shell, you can run dirs -v to see, along with its index, the directories in your stack:
$ dirs -v
0 ~
1 /ppp/n/m
2 ~/bin/a/b
3 ~/bin/x/y
4 ~
Now, no matter where you are, if you want to edit ~/bin/x/y/cmd1.sh, you can use
$ vi ~3/cmd3.sh
As long as you don't use popd or pushd elsewhere to modify the stack, the indices will stay the same. (Using pushd will add a new directory to the top of the stack, increasing each index; popd will decrease each index after it removes the top directory.)
A much simpler process would be to simply define some variables whose values are the desired directories:
binab=~/bin/a/b
binxy=~/bin/x/y
ppp=/ppp/n/m
and simply expand them
$ vi $ppp/cmd3.sh
The shell performs parameter name completion, so the variable names don't have to be particularly short, but the dirstack approach guarantees you only need 2 or 3 characters. (Also, it doesn't pollute the global namespace with additional varibles.)
Interestingly, I've found myself wanting to do something similar a while back. I hacked together the following bash script. It's pretty self-explanatory. If I want to edit one of my scripts (this one, for example is ~/bin/vm), I just run vm vm. I can open several files in my path, either in buffers, or vertical/horizontal splits etc...
Do with it what you like, pasting it here because it's all ready to use:
#!/usr/bin/env bash
Usage() {
cat <<-__EOF_
${0##*/} Opens scripts in PATH from any location (vim -O)
Example: ${0##*/} ${0##*/}
opens this script in vim editor
-o: Change default behaviour (vim -O) to -o
-b: Change default behaviour to open in buffers (vim file1 file2)
-h: Display this message
__EOF_
}
flag="O"
vimopen() {
local wrapped
local located
local found
found=false
[ $# -lt 1 ] && echo "No script given" && return
wrapped=""
for arg in "$#"; do
if located=$(which "${arg}" 2> /dev/null); then
found=true
wrapped="${wrapped} ${located}"
else
echo "${arg} not found!"
fi
done
$found || return
# We WANT word splitting to occur here
# shellcheck disable=SC2086
case ${flag} in
O)
vim $wrapped -O
;;
o)
vim $wrapped -o
;;
*)
vim $wrapped
esac
}
while getopts :boh f; do
case $f in
h)
Usage
exit 0
;;
o)
flag="o"
shift
;;
b)
flag=""
shift
;;
*)
echo "Unknown option ${f}-${OPTARG}"
Usage
exit 1
;;
esac
done
vimopen "$#"
Let me share something that answers OPTION3 part of your answer:
Behavior of this solution
The solutions that I will show will offer up basenames of commands (i.e. what compgen -c ${cur} returns where cur is last word on the command line) until there is only one candidate in which case it will be replaced by the full path of the command.
$ vc c<TAB><TAB>
Display all 216 possibilities? (y or n)
$ vc cm<TAB>
cmake cmake-gui cmcprompt cmd1.sh cmd2.sh cmd3.sh cmp cmpdylib cmuwmtopbm
$ vc cmd<TAB>
cmd1.sh cmd2.sh cmd3.sh
$ vc cmd1<TAB>
$ vc /Users/pcarphin/vc/bin/cmd1.sh
which I think is what you want.
And for your vc function, you can still do
vc(){
vim "$(which "${1}")
}
since which /Users/pcarphin/vc/bin/cmd3.sh returns /Users/pcarphin/vc/bin/cmd3.sh and so it will work whether you do vc cmd3.sh<ENTER> or if you do vc cmd3.sh<TAB><ENTER>
Basic solution
So here it is, it's as simple as using compgen -c to get command basename candidates and checking if you only have a single candidate and if so, replacing it with the full path.
_vc(){
local cur prev words cword
_init_completion || return;
COMPREPLY=( $(compgen -c ${cur}) )
#
# If there is only one candidate for completion, replace it with the
# full path returned by which.
#
if ((${#COMPREPLY[#]} == 1)) ; then
COMPREPLY[0]=$(which ${COMPREPLY[0]})
fi
}
complete -F _vc vc
Solution that filters out shell functions
The compgen -c command will include the names of shell functions and if you want to leave those out (maybe because your vc function would fail which would be inelegant for an argument supplied by a completion function), here is what you can do:
_vc(){
local cur prev words cword
_init_completion || return;
local candidates=($(compgen -c ${cur}))
#
# Put in COMPREPLY only the command names that are files in PATH
# and leave out shell functions
#
local i=0
for cmd in "${candidates[#]}" ; do
if which $cmd 2>/dev/null ; then
COMPREPLY[i++]=${cmd}
fi
done
#
# If there is only one candidate for completion, replace it with the
# full path returned by which.
#
if ((${#COMPREPLY[#]} == 1)) ; then
COMPREPLY[0]=$(which ${COMPREPLY[0]})
fi
}
Solution that handles shell functions
If we want to handle shell functions, then we can get rid of the part that filters them out and enhance the part that replaces the command name by a full path when COMPREPLY contains only one candidate. This is based on turning on extdebug which causes declare -F shell_function to output the file where shell_function was defined:
cmd_location(){
local location
if location=$(which "${1}" 2>/dev/null) ; then
echo "${location}"
else
# If extdebug is off, remember that and turn it on
local user_has_extdebug
if ! shopt extdebug ; then
user_has_extdebug=no
shopt -s extdebug
fi
info=$(declare -F ${COMPREPLY[0]})
if [[ -n "${info}" ]] ; then
echo ${info} | cut -d ' ' -f 3
fi
# Turn extdebug back off if it was off before
if [[ "${user_has_extdebug}" == no ]] ; then
shopt -u extdebug
fi
fi
}
_vc(){
local cur prev words cword
_init_completion || return;
COMPREPLY=( $(compgen -c ${cur}) )
if ((${#COMPREPLY[#]} == 1)) ; then
COMPREPLY[0]=$(cmd_location ${COMPREPLY[0]})
fi
}
And in this case, your vc function would need the same kind of logic or you could just remember to always use the shell completion to end up calling it with a full path.
That's why I factored out the cmd_location function
vc(){
if [[ "${1}" == /* ]] ; then
vim "${1}"
else
vim $(cmd_location "${1}")
fi
}
I was looking for something else but I found this question which inspired me to do this for myself so thank you, now I'll have a neat vc function with a cool completion function. Personally, I'm going to use the last version which handles shell functions.
The declare -F command with extdebug prints out the function name, the line number, and the file, so I'll see if I can adapt the solution so that in the case of shell functions, it opens the file at the location.
For that, I'd have to get rid of the part that puts a full path on the command line. So what I'm going to do for myself won't be an answer to your question. Note the use of parentheses for open_shell_function which makes it run in a subshell so I don't have to do the whole thing with user_has_extdebug.
open_shell_function()(
# Use subshell so as not to turn on extdebug in the user's shell
# and avoid doing this remembering stuff
shopt -s extdebug
local info=$(declare -F ${1})
if [[ -z "${info}" ]] ; then
echo "No info from 'declare -F' for '${1}'"
return 1
fi
local lineno
if ! lineno=$(echo ${info} | cut -d ' ' -f 2) ; then
echo "Error getting line number from info '${info}' on '${1}'"
return 1
fi
local file
if ! file=$(echo ${info} | cut -d ' ' -f 3) ; then
echo "Error getting filename from info '${info}' on '${1}'"
return 1
fi
vim ${file} +${lineno}
)
vc(){
local file
if file=$(which ${1} 2>/dev/null) ; then
vim ${file}
else
echo "no '${1}' found in path, looking for shell function"
open_shell_function "${1}"
fi
}
complete -c vc

How to properly escape spaces form multiple files in an scp command in a sourced function in bash

I've built a function in my .bashrc that breaks when it tries to scp files with spaces in their names, but if I run the generated command output from the function on in the shell directly it seems to work fine.
I've tried escaping spaces, and several variations of single and double quotes, the version below is the closest I've gotten to working and I don't understand why it fails.
From .bashrc
push2() {
# parse args, build file list, get suffix from final arg
files=""
i=1
orig_IFS=$IFS; IFS=":"
for arg in $*; do
if [ "$i" = "$#" ]; then
suffix=$arg
else
files="$files $(echo $arg | sed -r 's/ /\\ /')" #escape spaces
fi
i=$(($i+1))
done
IFS=$orig_IFS
# determine prefix and proxy
gen_prefix $suffix
# output generated command for debugging
echo "scp $scp_proxy$files testuser#$prefix$suffix:"
# run command
scp $scp_proxy$files testuser#$prefix$suffix:
}
Running the function still seems to fail even though the output command string appears properly escaped
root#DHCP-137:~$ push2 temp* 42
scp temp temp\ file testuser#10.3.3.42:
temp 100% 6008 1.1MB/s 00:00
temp\: No such file or directory
file: No such file or directory
Running the command it generates works as expected
root#DHCP-137:~$ scp temp temp\ file testuser#10.3.3.42:
temp 100% 6008 896.4KB/s 00:00
temp file 100% 0 0.0KB/s 00:00
root#DHCP-137:~$
Additional Info: GNU bash, version 4.4.12(1)-release (x86_64-pc-linux-gnu) - running on Debian 9
First, change your calling signature so that the suffix comes first:
push2 42 ./temp*
Then the function should be defined simply as
push2 () {
local -a scpProxy
local prefix suffix
suffix=$1
shift
gen_prefix "$suffix"
scp "${scpProxy[#]}" "$#" "testuser#$prefix.$suffix:"
}
where gen_prefix looks something like
gen_prefix () {
case $1 in
42) scpProxy=()
prefix=10.3.3
;;
89) scpProxy=(-o ProxyJump=user#server)
prefix=123.456.789
;;
esac
}
After calling shift, $# contains just the files you want to transfer. scpProxy is an array that holds multiple individual arguments to pass to scp; if it is empty, then "${scpProxy[#]}" will expand to 0 arguments, not the empty string.
(Using ./temp* instead of temp* guards against matches that contain : and could thus be mistaken for a remote file name.)
Although gen_prefix appears to define its variables "globally", it's really just defining them in whatever scope gen_prefix is called from (bash uses dynamic scoping, not lexical scoping, like most other common languages). The two calls to local ensure that whatever gen_prefix assigns stays inside push2, and not visible after push2 exits.
As an additional note, much of this function can go away with a suitable ssh configuration. Consider this in your .ssh/config file:
Host somehost
User testuser
Hostname 10.3.3.42
Host someotherhost
User testuser
Hostname 123.456.789.89
ProxyJump user#server
Now you don't need push2 at all; just run
scp temp* somehost:
or
scp temp* someotherhost:
and the correct addresses and options will be used automatically. The ssh configuration replaces everything gen_prefix did, and without the need to call gen_prefix, there's no longer any need to wrap scp.
The whole thing was fixed by changing the last line
scp $scp_proxy$files testuser#$prefix$suffix:
and wrapping it in an eval like this
eval "scp $scp_proxy$files testuser#$prefix$suffix:"

Expand An Alias That Executes Another Alias (Nested Alias)

I have two aliases:
alias ls="ls -G"
alias la="ls -aFhlT"
I know that after you type your alias, but before you execute, you can type Meta-Control-e (probably Alt-Control-e, but possibly Esc-Control-e) to expand what you've typed.
So, if I expand my alias la using this method I get:
ls -aFhlT
However, what I really want is to see:
ls -G -aFhlT
Is there any way to achieve this besides typing Meta-Control-e a second time?
--OR--
Is there any way to confirm that my execution of la actually executed ls -G -aFhlT (other than knowing how nested aliases work and trusting that it did what I think it did)?
I'm trying to do this on macOS, but a general bash solution will also be accepted.
This question rides the fine line between using an alias and using a function. When aliases get even slightly complicated, it is generally better to write a function instead. That being said, I did find a solution for this question that allows for expanding aliases as desired.
I wrote a bash function for this:
xtrace() {
local eval_cmd
printf -v eval_cmd '%q ' "${#}"
{ set -x
eval "${eval_cmd}"
} 2>&1 | grep '^++'
return "${PIPESTATUS[0]}"
}
The -v flag of printf will store the output of printf in the specified variable.
The printf format string %q will print the associated argument ($# in this case) shell-quoted, reusable as input. This eliminates the dangers associated with passing arbitrary code/commands to eval.
I then use a command group { ... } so I can control the functionality of set -x, which tells bash to print a trace of all executed commands. For my purposes, I do not care about any output except for the fully expanded command, so I redirect stderr and grep for the output line that starts with "++". This will be the line that shows the fully expanded command.
Finally, I return the value of PIPESTATUS[0], which contains the return code of the last command executed in the command group (i.e. the eval command).
Thus, we will get something like the following:
$ xtrace la; echo $?
++ ls -G -aFhlT
0
Much thanks to #CharlesDuffy for the set -x recommendation as well as the input sanitation for eval.

Programmatically dereference/resolve aliases in bash

I need to determine which command a shell alias resolves to in bash, programmatically; i.e., I need to write a bash function that will take a name potentially referring to an alias and return the "real" command it ultimately refers to, recursing through chains of aliases where applicable.
For example, given the following aliases:
alias dir='list -l'
alias list='ls'
where my function is dereference_alias,
dereference_alias list # returns "ls"
dereference_alias dir # also returns "ls"
Is there some builtin I don't know about that does this neatly, or shall I resign myself to scraping the output of alias?
Here's a version I wrote that does not rely on any external commands and also handles recursive aliases without creating an infinite loop:
# Arguments:
#
# $1 Command to compact using aliases
#
function command-to-alias()
{
local alias_key
local expansion
local guess
local command="$1"
local search_again="x"
local shortest_guess="$command"
while [[ "${search_again:-}" ]]; do
unset search_again
for alias_key in "${!BASH_ALIASES[#]}"; do
expansion="${BASH_ALIASES[$alias_key]}"
guess="${command/#"$expansion"/$alias_key}"
test "${#guess}" -lt "${#shortest_guess}" || continue
shortest_guess="$guess"
search_again="x"
done
command="$shortest_guess"
done
echo "$command"
}
Here's how I'm doing it, though I'm not sure it's the best way:
dereference_alias () {
# recursively expand alias, dropping arguments
# output == input if no alias matches
local p
local a="$1"
if [[ "alias" -eq $(type -t $a) ]] && p=$(alias "$a" 2>&-); then
dereference_alias $(sed -re "s/alias "$a"='(\S+).*'$/\1/" <<< "$p")
else
echo $a
fi
}
The major downsides here are that I rely on sed, and my means of dropping any arguments in the alias stops at the first space, expecting that no alias shall ever point to a program which, for some reason, has spaces in its name (i.e. alias badprogram='A\ Very\ Bad\ Program --some-argument'), which is a reasonable enough assumption, but still. I think that at least the whole sed part could be replaced by maybe something leveraging bash's own parsing/splitting/tokenization of command lines, but I wouldn't know where to begin.

BASH script: Downloading consecutive numbered files with wget

I have a web server that saves the logs files of a web application numbered. A file name example for this would be:
dbsclog01s001.log
dbsclog01s002.log
dbsclog01s003.log
The last 3 digits are the counter and they can get sometime up to 100.
I usually open a web browser, browse to the file like:
http://someaddress.com/logs/dbsclog01s001.log
and save the files. This of course gets a bit annoying when you get 50 logs.
I tried to come up with a BASH script for using wget and passing
http://someaddress.com/logs/dbsclog01s*.log
but I am having problems with my the script.
Anyway, anyone has a sample on how to do this?
thanks!
#!/bin/sh
if [ $# -lt 3 ]; then
echo "Usage: $0 url_format seq_start seq_end [wget_args]"
exit
fi
url_format=$1
seq_start=$2
seq_end=$3
shift 3
printf "$url_format\\n" `seq $seq_start $seq_end` | wget -i- "$#"
Save the above as seq_wget, give it execution permission (chmod +x seq_wget), and then run, for example:
$ ./seq_wget http://someaddress.com/logs/dbsclog01s%03d.log 1 50
Or, if you have Bash 4.0, you could just type
$ wget http://someaddress.com/logs/dbsclog01s{001..050}.log
Or, if you have curl instead of wget, you could follow Dennis Williamson's answer.
curl seems to support ranges. From the man page:
URL
The URL syntax is protocol dependent. You’ll find a detailed descrip‐
tion in RFC 3986.
You can specify multiple URLs or parts of URLs by writing part sets
within braces as in:
http://site.{one,two,three}.com
or you can get sequences of alphanumeric series by using [] as in:
ftp://ftp.numericals.com/file[1-100].txt
ftp://ftp.numericals.com/file[001-100].txt (with leading zeros)
ftp://ftp.letters.com/file[a-z].txt
No nesting of the sequences is supported at the moment, but you can use
several ones next to each other:
http://any.org/archive[1996-1999]/vol[1-4]/part{a,b,c}.html
You can specify any amount of URLs on the command line. They will be
fetched in a sequential manner in the specified order.
Since curl 7.15.1 you can also specify step counter for the ranges, so
that you can get every Nth number or letter:
http://www.numericals.com/file[1-100:10].txt
http://www.letters.com/file[a-z:2].txt
You may have noticed that it says "with leading zeros"!
You can use echo type sequences in the wget url to download a string of numbers...
wget http://someaddress.com/logs/dbsclog01s00{1..3}.log
This also works with letters
{a..z} {A..Z}
Not sure precisely what problems you were experiencing, but it sounds like a simple for loop in bash would do it for you.
for i in {1..999}; do
wget -k http://someaddress.com/logs/dbsclog01s$i.log -O your_local_output_dir_$i;
done
You can use a combination of a for loop in bash with the printf command (of course modifying echo to wget as needed):
$ for i in {1..10}; do echo "http://www.com/myurl`printf "%03d" $i`.html"; done
http://www.com/myurl001.html
http://www.com/myurl002.html
http://www.com/myurl003.html
http://www.com/myurl004.html
http://www.com/myurl005.html
http://www.com/myurl006.html
http://www.com/myurl007.html
http://www.com/myurl008.html
http://www.com/myurl009.html
http://www.com/myurl010.html
Interesting task, so I wrote full script for you (combined several answers and more). Here it is:
#!/bin/bash
# fixed vars
URL=http://domain.com/logs/ # URL address 'till logfile name
PREF=logprefix # logfile prefix (before number)
POSTF=.log # logfile suffix (after number)
DIGITS=3 # how many digits logfile's number have
DLDIR=~/Downloads # download directory
TOUT=5 # timeout for quit
# code
for((i=1;i<10**$DIGITS;++i))
do
file=$PREF`printf "%0${DIGITS}d" $i`$POSTF # local file name
dl=$URL$file # full URL to download
echo "$dl -> $DLDIR/$file" # monitoring, can be commented
wget -T $TOUT -q $dl -O $file
if [ "$?" -ne 0 ] # test if we finished
then
exit
fi
done
At the beggiing of the script you can set URL, log file prefix and suffix, how many digits you have in numbering part and download directory. Loop will download all logfiles it found, and automaticaly exit on first non-existant (using wget's timeout).
Note that this script assumes that logfile indexing starts with 1, not zero, as you mentioned in example.
Hope this helps.
Here you can find a Perl script that looks like what you want
http://osix.net/modules/article/?id=677
#!/usr/bin/perl
$program="wget"; #change this to proz if you have it ;-)
my $count=1; #the lesson number starts from 1
my $base_url= "http://www.und.nodak.edu/org/crypto/crypto/lanaki.crypt.class/lessons/lesson";
my $format=".zip"; #the format of the file to download
my $max=24; #the total number of files to download
my $url;
for($count=1;$count<=$max;$count++) {
if($count<10) {
$url=$base_url."0".$count.$format; #insert a '0' and form the URL
}
else {
$url=$base_url.$count.$format; #no need to insert a zero
}
system("$program $url");
}
I just had a look at the wget manpage discussion of 'globbing':
By default, globbing will be turned on if the URL contains a globbing character. This option may be used to turn globbing on or off permanently.
You may have to quote the URL to protect it from being expanded by your shell. Globbing makes Wget look for a directory listing, which is system-specific. This is why it currently works only with Unix FTP servers (and the ones emulating Unix "ls" output).
So wget http://... won't work with globbing.
Check to see if your system has seq, then it would be easy:
for i in $(seq -f "%03g" 1 10); do wget "http://.../dbsclog${i}.log"; done
If your system has the jot command instead of seq:
for i in $(jot -w "http://.../dbsclog%03d.log" 10); do wget $i; done
Oh! this is a similar problem I ran into when learning bash to automate manga downloads.
Something like this should work:
for a in `seq 1 999`; do
if [ ${#a} -eq 1 ]; then
b="00"
elif [ ${#a} -eq 2 ]; then
b="0"
fi
echo "$a of 231"
wget -q http://site.com/path/fileprefix$b$a.jpg
done
Late to the party, but a real easy solution that requires no coding is to use the DownThemAll Firefox add-on, which has the functionality to retrieve ranges of files. That was my solution when I needed to download 800 consecutively numbered files.

Resources