The below script download file using CURL, I'm trying inside the loop to save the file and also to insert the saved file name into a variable and then to print it.
My script downloads the script and saved the file but can't echo the saved file name:
for link in $url2; do
cd /var/script/twitter/html_files/ && file1=$({ curl -O $link ; cd -; })
echo $file1
done
Script explanation:
$url2 contains one or more URLs
curl -O write output to a file named as the remote file
Your code has several problems. Assuming $url2 is a list of valid URLs which do not require shell quoting, you can make curl print the output variable directly.
cd /var/script/twitter/html_files
for link in $url2; do
curl -s -w '%{filename_effective}\n' -O "$link"
done
Without the -w formatstring option, the output of curl does not normally contain the output file name in a conveniently machine-readable format (or actually at all). I also added an -s option to disable the download status output it prints by default.
There is no point in doing cd to the same directory over and over again inside the loop, or capturing the output into a variable which you only use once to print to standard output the string which curl by itself would otherwise print to standard output.
Finally, the cd - does not seem to do anything useful here; even if it did something useful per se, you are doing it in a subshell, which doesn't change the current working directory of the script which contains the $(cd -) command substitution.
If your task is to temporarily switch to that directory, then switch back to where you started, just cd once. You can use cd - in Bash but a slightly more robust and portable solution is to run the fetch in a subshell.
( cd directory;
do things ...
)
# you are now back to where you were before the cd
If you genuinely need the variable, you can trivially use
for link in $url2; do
file1=$(curl -s -w '%{filename_effective}' -O "$link")
echo "$file1"
done
but obviously the variable will only contain the result from the last iteration after the loop (in the code after done). (The format string doesn't need the final \n here because the command substitution will trim off any trailing newline anyway.)
Related
Trying to write a bash script to copy a large number of files from an external drive into separate directories based on a subject id.
I've included the script I've written below.
I get the following error:
cat: /Volumes/Seagate: No such file or directory
cat: Backup: No such file or directory
cat: Plus: No such file or directory
cat: Drive/Subject_List.txt: No such file or directory
When I try to copy a single file at a time using the terminal, it copies using the exact command I've put in this script. I'm not sure why it's not recognizing the directory when I try and use it in the script below. Any help is greatly appreciated!
#!/bin/bash
#A bash script to copy the structural and functional files into the HCP_Entropy folder
#subject list
SUBJECT_LIST="/Volumes/Seagate/Backup/Plus/Drive/Subject_List.txt
for j in $(cat ${SUBJECT_LIST}); do
echo ${j}
cp /Volumes/Seagate\ Backup\ Plus\ Drive/HCP_DATA/Structural/{j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz /Users/myname/Box/HCP_Entropy/BEN/${j}/anat
done
the line
$SUBJECT_LIST=/Volumes/Seagate\ Backup\ Plus\ Drive/Subject_List.txt
is bogus.
to assign values to a variable, you must not add the $ specifier.
a token starting with $ will be expanded, so $SUBJECT_LIST=... will first be expanded to =... (since you haven't assigned anything to the SUBJECT_LIST variable yet it is empty).
the proper way would be:
SUBJECT_LIST="/Volumes/Seagate Backup Plus Drive/Subject_List.txt"
(this uses quotes instead of escaping each space, which i find much more readable)
you also need to quote variables in case they contain spaces, else they might be interpreted by the command (cp) as multiple arguments.
for j in $(cat "${SUBJECT_LIST}"); do
# ...
done
and of course, you should check whether the source file actually exists, just like the destination directory.
indir="/Volumes/Seagate Backup Plus Drive"
SUBJECT_LIST="${indir}/Subject_List.txt"
cat "${SUBJECT_LIST}" | while read j; do
infile="${indir}/HCP_DATA/Structural/${j}/unprocessed/3T/T1w_MPR1/${j}_3T_T1w_MPR1.nii.gz"
outdir="/Users/myname/Box/HCP_Entropy/BEN/${j}/anat"
mkdir -p "${outdir}"
if [ -e "${infile}" ]; then
cp -v "${infile}" "${outdir}"
else
echo "missing file ${infile}" 1>&2
fi
done
I am new to programming and just starting in bash.
I'm trying to print a list of directories and files to a txt file, and remove some of the path that gets printed to make it cleaner.
It works with this:
TODAY=$(date +"%Y-%m-%d")
cd
cd Downloads
ls -R ~/Music/iTunes/iTunes\ Media/Music | sed 's/\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\///g' > music-list-$TODAY.txt
But to clean it up I want to use variables like so,
# Creates a string of the date, format YYYY-MM-DD
TODAY="$(date +"%Y-%m-%d")"
# Where my music folders are
MUSIC="$HOME/Music/iTunes/iTunes\ Media/Music/"
# Where I want it to go
DESTINATION="$HOME/Downloads/music-list-"$TODAY".txt"
# Path name to be removed from text file
REMOVED="\/Users\/BilPaLo\/Music\/iTunes\/iTunes\ Media\/Music\/"
ls -R "$MUSIC" > "$DESTINATION"
sed "s/$REMOVED//g" > "$DESTINATION"
but it gives me a 'no such file or directory' error that I can't seem to get around.
I'm sure there are many other problems with this code but this one I don't understand.
Thank you everyone! I followed the much needed formatting advice and #amo-ej1's answer and now this works:
# Creates a string of the date format YYYY-MM-DD
today="$(date +"%Y-%m-%d")"
# Where my music folders are
music="$HOME/Music/iTunes/iTunes Media/Music/"
# Where I want it to go
destination="$HOME/Downloads/music-list-$today.txt"
# Temporary file
temp="$HOME/Downloads/temp.txt"
# Path name to be removed of text file to only leave artist name and album
remove="\\/Users\\/BilPaLo\\/Music\\/iTunes\\/iTunes\\ Media\\/Music\\/"
# lists all children of music and writes it in temp
ls -R "$music" > "$temp"
# substitutes remove by nothing and writes it in destination
sed "s/$remove//g" "$temp" > "$destination"
rm $temp #deletes temp
First when debugging bash it can be helpful to start bash with the -x flags (bash -x script.sh) or within the script enter set -x, that way bash will print out the commands it is executing (with the variable expansions) and you can more easily spot errors that way.
In this specific snippet our ls output is being redirected to a file called $DESTINATION and and sed will read from standard input and write also to $DESTINATION. So however you wanted to replace the pipe in your oneliner is wrong. As a result this will look as if your program is blocked but sed will simply wait for input arriving on standard input.
As for the 'no such file or directory', try executing with set -x and doublecheck the paths it is trying to access.
I have made a directory with lots of files with:
samplefile_111222015_reporting_{1..13}
I am trying to create a vi script where when I enter the directory as an argument to the command e.g.
sh myScript $HOME/theDir/*
then it copies all the files in that directory to a new one I made. Although right now, I'm having problems with the for loop alone.
This is what I have in my script:
for f in $1;
do
echo "$f"
done
but when i enter sh myScript $HOME/theDir, I get back the name of the first file only (samplefile_111222015_reporting_1). why the first file? Is this not a for loop>
# Because of the wild card expansion, all the files in the directory are
# already made available to the script through arguments
# So do the following to get all the file listing
for f ; do echo $f; done
This is because each file is passed as a separate argument and you're only looping over $1, which is the first argument.
Instead, you most likely want to loop over "$#", which is every argument starting from $1.
The man page for bash, under the Special Parameters section, details the special parameters available in more detail.
These lines work when copy-pasted to the shell but don't work in a script:
ls -l file1 > /path/`echo !#:2`.txt
ls -l file2 > /path/`echo !#:2`.txt
ls -l file1 > /path/$(echo !#:2).txt
ls -l file2 > /path/$(echo !#:2).txt
What's the syntax for doing this in a bash script?
If possible, I would like to know how to do this for one file and for all files with the same extension in a folder.
Non-interactive shell has history expansion disabled.
Add the following two lines to your script to enable it:
set -o history
set -o histexpand
(UPDATE: I misunderstood the original question as referring to arguments to the script, not arguments to the current command within the script; this is a rewritten answer.)
As #choroba said, history is disabled by default in scripts, because it's not really the right way to do things like this in a script.
The preferred way to do things like this in a script is to store the item in question (in this case the filename) in a variable, then refer to it multiple times in the command:
fname=file1
ls -l "$fname" > "/path/$fname.txt"
Note that you should almost always put variable references inside double-quotes (as I did above) to avoid trouble if they contain spaces or other shell metacharacters. If you want to do this for multiple files, use a for loop:
for fname in *; do # this will repeat for each file (or directory) in the current directory
ls -l "$fname" > "/path/$fname.txt"
done
If you want to operate on files someplace other than the current directory, things are a little more complicated. You can use /inputpath/*, but it'll include the path along with each filename (e.g. it'd run the loop with "/inputpath/file1", "/inputpath/file2", etc), and if you use that directly in the output redirect you'll get something like > /path/inputpath/file1.txt (i.e. the two different paths will get appended together), probably not what you want. In this case, you can use the basename command to strip off the unwanted path for output purposes:
for fpath in /inputpath/*; do
ls -l "$fpath" > "/path/$(basename "$fpath").txt"
done
If you want a list of files with a particular extension, just use *.foo or /inputpath/*.foo as appropriate. However, in this case you'll wind up with the output going to files named e.g. "file1.foo.txt"; if you don't want stacked extensions, basename has an option to trim that as well:
for fpath in /inputpath/*.foo; do
ls -l "$fpath" > "/path/$(basename "$fpath" .foo).txt"
done
Finally, it might be neater (depending how complex the actual operation is, and whether it occurs multiple times in the script) to wrap this in a function, then use that:
doStuffWithFile() {
ls -l "$1" > "/path/$(basename "$1" "$2").txt"
}
for fpath in /inputpath/*.foo; do
doStuffWithFile "$fpath" ".foo"
done
doStuffWithFile /otherpath/otherfile.bar .bar
I'm thinking of writing a script for cygwin to cd into a windows directory which is copied from Windows explorer.
e.g.
cdw D:\working\test
equals to
cd /cygdrive/d/working/test
But it seems for shell script, all backslashs in parameters are ignored unless using single quote 'D:\working\test' or double backslashs D:\\working\\test.
But in my case it would be very inconvenience because I can't simply paste the directory name in the command line to execute the script.
Is there any way to make cdw D:\working\test working?
Well, you can do it, but you want something strange :)
cdw()
{
set $(history | tail -1 )
shift 2
path="$*"
cd $(cygpath "$path")
}
Example of usage:
$ cdw D:\working\test
$ pwd
/cygdrive/d/working/test
The main point here is the usage of history.
You don't use an argument directly, but get it from the history in the form it was typed.
$ rawarg() { set $(history | tail -1 ); shift 2; echo "$#"; }
$ rawarg C:\a\b\c\d
C:\a\b\c\d
Of course, you can use this trick in a interactive shell only (for obvious reasons).
The problem you deal with is related to the shell. Any argument you add to cdw on the command line, will be processed by the shell before cdw gets executed.
In order to prevent that processing to happen, you need at least one level of quoting,
either by enclosing the whole string in single quotes:
cd 'D:\working\test'
or with double backslashses:
cd D:\\working\test
A separate program will not help, because the damage is already done before it runs. ;-)
However, I have a possible function for cdw, which works in my AST UWIN ksh:
function cdw { typeset dir
read -r dir?"Paste Directory Path: "
cd ${dir:?}
}
And this one works in Bash (which does not support read var?prompt):
function cdw {
typeset dir
printf "Paste Directory Path: "
read -r dir || return
cd ${dir:?}
}
For me, I just type the two single quotes around the Pasted value.
The solution to add single quotes allows to copy paste