I want my script to find a file (in the current directory) with the first line equal to START. Then that file should have FILE <file_name> as the last line. So I want to extract the <file_name> - I use tail for this. It works ok for standard file names but cracks for nonstandard file names like a a or a+b-c\ = e with tail reporting tail option used in invalid context -- 1
Here is the beginning of the script:
#!/bin/bash
next_stop=0;
# find the first file
start_file=$(find . -type f -exec sed '/START/F;Q' {} \;)
mv "$start_file" $start_file # << that trick doesn't work
if [ ! -f "$start_file" ]
then
echo "File with 'START' head not found."
exit 1
else
echo "Found $start_file"
fi
# parse the last line of the start file
last_line=$(tail -1 $start_file) # << here it crashes for hacky names
echo "last line: $last_line"
if [[ $last_line == FILE* ]] ; then
next_file=${last_line#* }
echo "next file from last line: $next_file"
elif [[ $last_line == STOP ]] ; then
next_stop=true;
else
echo "No match for either FILE or STOP => exit"
exit 1
fi
I tried to embrace the find output with braces this way
mv "$start_file" $start_file
but it doesn't help
This error is occur to the character of the escape.
You should write it start_file variable in quotes.
last_line=$(tail -1 $start_file) --> last_line=$(tail -1 "$start_file")
For you two examples, you need to escape space and egual in file name (with \ character), and escape escape character too.
So a a have to be a\ a when passing to tail, and a+b-c\ = e have to be a+b-c\\\ \=\ e.
You can use sed to make this replacement.
This example give you an better and easier way to make this replacement :
printf '%q' "$Strange_filename"
Related
subjects_list=$(ls -l /Volumes/Backup_Plus/PPMI_10 | awk '{ print $NF }')
filepath="/Volumes/Backup_Plus/PPMI_10/$subjects/*/*/S*/"
for subjects in $subjects_list; do
if [[ -f "${filepath}/*.bval" && -f "${filepath}/*.bvec" && -f "${filepath}/*.json" && -f "${filepath}/*.nii.gz" ]]; then
echo "${subjects}" >> /Volumes/Backup_Plus/PPMI_10/keep_subjects.txt
else
echo "${subjects}" >> /Volumes/Backup_Plus/PPMI_10/not_keep_subjects.txt
fi
done
problem is supposedly in the if statement, I tried this...
bvalfile = (*.bval)
bvecfile =(*.bvec)
jsonfile =(*.json)
niigzfile =(*.nii.gz)
if [[ -f "$bvalfile" && -f "$bvecfile" && -f "$jsonfile" && -f "$niigzfile" ]]; then
however that didn't work. Any help with syntax or errors or does it need to be changed completely. Trying to separate the files that have .^file types from those that don't by making two lists.
thanks
You're assigning filepath outside the for-subject loop but using the unset variable $subjects in it. You want to move that inside the loop.
Double-quoted wildcards aren't expanded, so both $filepath and your -f test will be looking for filenames with literal asterisks in them.
-f only works on a single file, so even if you fix the quotes, you'll have a syntax error if there's more than one file matching the pattern.
So I think what you want is something like this:
# note: array assignment -
# shell does the wildcard expansion, no ls required
prefix_list=( /Volumes/Backup_Plus/PPMI_10/* )
# and array expansion
for prefix in "${prefix_list[#]}"; do
# the subject is just the last component of the path
subject=${prefix##*/}
# start by assuming we're keeping this one
decision=keep
# in case filepath pattern matches more than one directory, loop over them
for filepath in "$prefix"/*/*/S*/; do
# if any of the files don't exist, switch to not keeping it
for file in "$filepath"/{*.bval,*.bvec,*.json,*.nii.gz}; do
if [[ ! -f "$file" ]]; then
decision=not_keep
# we have our answer and can stop looping now
break 2
fi
done
done
# now append to the correct list
printf '%s\n' "$subject" >>"/Volumes/Backup_Plus/PPMI_10/${decision}_subjects.txt"
done
I'm hoping this is a simple question, since I've never done shell scripting before. I'm trying to filter certain files out of a list of results. While the script executes and prints out a list of files, it's not filtering out the ones I don't want. Thanks for any help you can provide!
#!/bin/bash
# Purpose: Identify all *md files in H2 repo where there is no audit date
#
#
#
# Example call: no_audits.sh
#
# If that call doesn't work, try ./no_audits.sh
#
# NOTE: Script assumes you are executing from within the scripts directory of
# your local H2 git repo.
#
# Process:
# 1) Go to H2 repo content directory (assumption is you are in the scripts dir)
# 2) Use for loop to go through all *md files in each content sub dir
# and list all file names and directories where audit date is null
#
#set counter
count=0
# Go to content directory and loop through all 'md' files in sub dirs
cd ../content
FILES=`find . -type f -name '*md' -print`
for f in $FILES
do
if [[ $f == "*all*" ]] || [[ $f == "*index*" ]] ;
then
# code to skip
echo " Skipping file: " $f
continue
else
# find audit_date in file metadata
adate=`grep audit_date $f`
# separate actual dates from rest of the grepped line
aadate=`echo $adate | awk -F\' '{print $2}'`
# if create date is null - proceed
if [[ -z "$aadate" ]] ;
then
# print a list of all files without audit dates
echo "Audit date: " $aadate " " $f;
count=$((count+1));
fi
fi
done
echo $count " files without audit dates "
First, to address the immediate issue:
[[ $f == "*all*" ]]
is only true if the exact contents of f is the string *all* -- with the wildcards as literal characters. If you want to check for a substring, then the asterisks shouldn't be quoted:
[[ $f = *all* ]]
...is a better-practice solution. (Note the use of = rather than == -- this isn't essential, but is a good habit to be in, as the POSIX test command is only specified to permit = as a string comparison operator; if one writes [ "$f" == foo ] by habit, one can get unexpected failures on platforms with a strictly compliant /bin/sh).
That said, a ground-up implementation of this script intended to follow best practices might look more like the following:
#!/usr/bin/env bash
count=0
while IFS= read -r -d '' filename; do
aadate=$(awk -F"'" '/audit_date/ { print $2; exit; }' <"$filename")
if [[ -z $aadate ]]; then
(( ++count ))
printf 'File %q has no audit date\n' "$filename"
else
printf 'File %q has audit date %s\n' "$filename" "$aadate"
fi
done < <(find . -not '(' -name '*all*' -o -name '*index*' ')' -type f -name '*md' -print0)
echo "Found $count files without audit dates" >&2
Note:
An arbitrary list of filenames cannot be stored in a single bash string (because all characters that might otherwise be used to determine where the first name ends and the next name begins could be present in the name itself). Instead, read one NUL-delimited filename at a time -- emitted with find -print0, read with IFS= read -r -d ''; this is discussed in [BashFAQ #1].
Filtering out unwanted names can be done internal to find.
There's no need to preprocess input to awk using grep, as awk is capable of searching through input files itself.
< <(...) is used to avoid the behavior in BashFAQ #24, wherein content piped to a while loop causes variables set or modified within that loop to become unavailable after its exit.
printf '...%q...\n' "$name" is safer than echo "...$name..." when handling unknown filenames, as printf will emit printable content that accurately represents those names even if they contain unprintable characters or characters which, when emitted directly to a terminal, act to modify that terminal's configuration.
Nevermind, I found the answer here:
bash script to check file name begins with expected string
I tried various versions of the wildcard/filename and ended up with:
if [[ "$f" == *all.md ]] || [[ "$f" == *index.md ]] ;
The link above said not to put those in quotes, and removing the quotes did the trick!
For an assignment, I'm supposed to create a script called my_which.sh that will "do the same thing as the Unix command, but do it using a for loop over an if." I am also not allowed to call which in my script.
I'm brand new to this, and have been reading tutorials, but I'm pretty confused on how to start. Doesn't which just list the path name of a command?
If so, how would I go about displaying the correct path name without calling which, and while using a for loop and an if statement?
For example, if I run my script, it will echo % and wait for input. But then how do I translate that to finding the directory? So it would look like this?
#!/bin/bash
path=(`echo $PATH`)
echo -n "% "
read ans
for i in $path
do
if [ -d $i ]; then
echo $i
fi
done
I would appreciate any help, or even any starting tutorials that can help me get started on this. I'm honestly very confused on how I should implement this.
Split your PATH variable safely. This is a general method to split a string at delimiters, that is 100% safe regarding any possible characters (including newlines):
IFS=: read -r -d '' -a paths < <(printf '%s:\0' "$PATH")
We artificially added : because if PATH ends with a trailing :, then it is understood that current directory should be in PATH. While this is dangerous and not recommended, we must also take it into account if we want to mimic which. Without this trailing colon, a PATH like /bin:/usr/bin: would be split into
declare -a paths='( [0]="/bin" [1]="/usr/bin" )'
whereas with this trailing colon the resulting array is:
declare -a paths='( [0]="/bin" [1]="/usr/bin" [2]="" )'
This is one detail that other answers miss. Of course, we'll do this only if PATH is set and non-empty.
With this split PATH, we'll use a for-loop to check whether the argument can be found in the given directory. Note that this should be done only if argument doesn't contain a / character! this is also something other answers missed.
My version of which handles a unique option -a that print all matching pathnames of each argument. Otherwise, only the first match is printed. We'll have to take this into account too.
My version of which handles the following exit status:
0 if all specified commands are found and executable
1 if one or more specified commands is nonexistent or not executable
2 if an invalid option is specified
We'll handle that too.
I guess the following mimics rather faithfully the behavior of my which (and it's pure Bash):
#!/bin/bash
show_usage() {
printf 'Usage: %s [-a] args\n' "$0"
}
illegal_option() {
printf >&2 'Illegal option -%s\n' "$1"
show_usage
exit 2
}
check_arg() {
if [[ -f $1 && -x $1 ]]; then
printf '%s\n' "$1"
return 0
else
return 1
fi
}
# manage options
show_only_one=true
while (($#)); do
[[ $1 = -- ]] && { shift; break; }
[[ $1 = -?* ]] || break
opt=${1#-}
while [[ $opt ]]; do
case $opt in
(a*) show_only_one=false; opt=${opt#?} ;;
(*) illegal_option "${opt:0:1}" ;;
esac
done
shift
done
# If no arguments left or empty PATH, exit with return code 1
(($#)) || exit 1
[[ $PATH ]] || exit 1
# split path
IFS=: read -r -d '' -a paths < <(printf '%s:\0' "$PATH")
ret=0
# loop on arguments
for arg; do
# Check whether arg contains a slash
if [[ $arg = */* ]]; then
check_arg "$arg" || ret=1
else
this_ret=1
for p in "${paths[#]}"; do
if check_arg "${p:-.}/$arg"; then
this_ret=0
"$show_only_one" && break
fi
done
((this_ret==1)) && ret=1
fi
done
exit "$ret"
To test whether an argument is executable or not, I'm checking whether it's a regular file1 which is executable with:
[[ -f $arg && -x $arg ]]
I guess that's close to my which's behavior.
1 As #mklement0 points out (thanks!) the -f test, when applied against a symbolic link, tests the type of the symlink's target.
#!/bin/bash
#Get the user's first argument to this script
exe_name=$1
#Set the field separator to ":" (this is what the PATH variable
# uses as its delimiter), then read the contents of the PATH
# into the array variable "paths" -- at the same time splitting
# the PATH by ":"
IFS=':' read -a paths <<< $PATH
#Iterate over each of the paths in the "paths" array
for e in ${paths[*]}
do
#Check for the $exe_name in this path
find $e -name $exe_name -maxdepth 1
done
This is similar to the accepted answer with the difference that it does not set the IFS and checks if the execute bits are set.
#!/bin/bash
for i in $(echo "$PATH" | tr ":" "\n")
do
find "$i" -name "$1" -perm +111 -maxdepth 1
done
Save this as my_which.sh (or some other name) and run it as ./my_which java etc.
However if there is an "if" required:
#!/bin/bash
for i in $(echo "$PATH" | tr ":" "\n")
do
# this is a one liner that works. However the user requires an if statment
# find "$i" -name "$1" -perm +111 -maxdepth 1
cmd=$i/$1
if [[ ( -f "$cmd" || -L "$cmd" ) && -x "$cmd" ]]
then
echo "$cmd"
break
fi
done
You might want to take a look at this link to figure out the tests in the "if".
For a complete, rock-solid implementation, see gniourf_gniourf's answer.
Here's a more concise alternative that makes do with a single invocation of find [per name to investigate].
The OP later clarified that an if statement should be used in a loop, but the question is general enough to warrant considering other approaches.
A naïve implementation would even work as a one-liner, IF you're willing to make a few assumptions (the example uses 'ls' as the executable to locate):
find -L ${PATH//:/ } -maxdepth 1 -type f -perm -u=x -name 'ls' 2>/dev/null
The assumptions - which will hold in many, but not all situations - are:
$PATH must not contain entries that when used unquoted result in shell expansions (e.g., no embedded spaces that would result in word splitting, no characters such as * that would result in pathname expansion)
$PATH must not contain an empty entry (which must be interpreted as the current dir).
Explanation:
-L tells find to investigate the targets of symlinks rather than the symlinks themselves - this ensures that symlinks to executable files are also recognized by -type f
${PATH//:/ } replaces all : chars. in $PATH with a space each, causing the result - due to being unquoted - to be passed as individual arguments split by spaces.
-maxdepth 1 instructs find to only look directly in each specified directory, not also in subdirectories
-type f matches only files, not directories.
-perm -u=x matches only files and directories that the current user (u) can execute (x).
2>/dev/null suppresses error messages that may stem from non-existent directories in the $PATH or failed attempts to access files due to lack of permission.
Here's a more robust script version:
Note:
For brevity, only handles a single argument (and no options).
Does NOT handle the case where entries or result paths may contain embedded \n chars - however, this is extremely rare in practice and likely leads to bigger problems overall.
#!//bin/bash
# Assign argument to variable; error out, if none given.
name=${1:?Please specify an executable filename.}
# Robustly read individual $PATH entries into a bash array, splitting by ':'
# - The additional trailing ':' ensures that a trailing ':' in $PATH is
# properly recognized as an empty entry - see gniourf_gniourf's answer.
IFS=: read -r -a paths <<<"${PATH}:"
# Replace empty entries with '.' for use with `find`.
# (Empty entries imply '.' - this is legacy behavior mandated by POSIX).
for (( i = 0; i < "${#paths[#]}"; i++ )); do
[[ "${paths[i]}" == '' ]] && paths[i]='.'
done
# Invoke `find` with *all* directories and capture the 1st match, if any, in a variable.
# Simply remove `| head -n 1` to print *all* matches.
match=$(find -L "${paths[#]}" -maxdepth 1 -type f -perm -u=x -name "$name" 2>/dev/null |
head -n 1)
# Print result, if found, and exit with appropriate exit code.
if [[ -n $match ]]; then
printf '%s\n' "$match"
exit 0
else
exit 1
fi
If I need to start grepping a file from line num 1293 all the way to the end of the file how can I do that?
More detailed info in case it helps:
I am trying to whip a quick function in my bashrc that lets me quickly search vim snippet files for a particular snippet echoing the snippet name and associated command(s) to screen. So I have no probs getting the line num for the snippet name and even printing out the command on the following line num. But if the snippet is a multi-line command then I need to grep for the next line beginning with snippet "^snippet " and then return all lines between, but I cannot find any details how I can go about getting grep to start its search starting from a particular line num.
A secondary question is how in a .bashrc function can I exit the function early? When I use the 'exit' command
currently commented out in the funct below the terminal itself exits/closes rather than just exiting the funct.
function vsls() {
if [[ "$2" =~ ^(html|sh|vim)$ ]] ; then
sPath="$2".snippets
elif [[ "$2" =~ ^(html|sh|vim).snippets$ ]] ; then
sPath="$2"
else
echo "\nExiting. You did not enter a recognized vim snippets file name."
# exit 69
fi
lnN=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 $1|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$lnN: ${lnN}"
cntr="$lnN"
sed -n "$cntr"p "$HOME/.vim/snippets/$sPath"
((cntr++))
sed -n "$cntr"p "$HOME/.vim/snippets/$sPath"
}
#chepner
I don't know why (lack of know-how likely) but without specifying 'more' I get a permissions error:
03:43 ~ $ fLNum=$($HOME/.vim/snippets/"$sPath"|grep -nm 1 tdotti|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}"
bash: /home/user/.vim/snippets/html.snippets: Permission denied
$fLNum:
03:43 ~ $ fLNum=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 tdotti|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}"
$fLNum: 1293
Now working as desired:
I stuck with sed since I feel most comfortable using sed. I have used the -n print opt before, but not too often so it totally escaped my mind to try something like that.
function vsls() {
if [[ "$2" =~ ^(html|sh|vim)$ ]] ; then
sPath="$2".snippets
elif [[ "$2" =~ ^(html|sh|vim).snippets$ ]] ; then
sPath="$2"
else
echo "\nExiting. You did not enter a recognized vim snippets file name."
# exit 69
fi
fLNum=$(more $HOME/.vim/snippets/"$sPath"|grep -nm 1 "snippet $1"|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$fLNum: ${fLNum}" #get line number of the snippet name searched, entered as input $1
((tLNum1 = fLNum+=1)) ; echo "\$tLNum1: ${tLNum1}" # tmpLineNum is next line num from which to start next grep search for lineNum of next snippet entry to determine where commands of desired snippet end
tLNum2=$(sed -n "${tLNum1},$ p" $HOME/.vim/snippets/"$sPath"|grep -nm 1 "snippet"|sed -r 's/^([0-9]*):.*$/\1/') ; echo "\$tLNum2: ${tLNum2}" #lineNum of next 'snippet entry'
let sLNum=tLNum2+fLNum sLNum-=1 ; let sLNum-=1 ; echo "\$sLNum: ${sLNum}" #tmpLineNum2 is not actual line num in file, but rather the number of lines since the start of the second search, that is necessarily somewhere within the file: so if second search begins on line 1294, for all intents and purpose actual line num 1294 is line 1 of the new (second) search; therefore I need to add the tLNum2 with fLNum to determine actual lineNum in the of the next snippet entry
echo ""
sed -n "${fLNum},${sLNum} p" "$HOME/.vim/snippets/$sPath"
echo ""
}
But it is curious why I needed to do:
let sLNum=tLNum2+fLNum sLNum-=1 ; let sLNum-=1
to get the correct line number of the second grep search. I only got lucky fooling around, b/c I would have thought:
let sLNum=tLNum2+fLNum sLNum-=1
or:
let sLNum=tLNum2+fLNum ; let sLNum-=1
should have done the trick; that is, secondLineNum = tmpLNum2 + firstLineNum and then secondLineNum - 1. But the result would never end up 1 less but always equal to tLNum+fLNum. It would be good to learn why that did not work as expected.
But its working. so thanks.
Or with sed like this:
sed -n "1293,$ p" yourfile | grep xyz
Or, if the line number is in a variable called line:
sed -n "${line},$ p" yourfile | grep xyz
Or, if you want your grep to find nothing in the first 1292 lines, but still report the correct line number if you are using grep -n, you can just get the (empty) hold buffer for grep to look at for lines 1 to 1292
sed "1,1292g" yourfile | grep -n xyz
awk is better suited for this
awk '/search_pattern/ && NR > 1292' filename
tail -n +1293 file | grep ....
I wrote a bash script that uploads a file on my home server. It gets activated from a folder action script using applescript. The setup is the folder on my desktop is called place_on_server. Its supposed to have an internal file structure exactly like the folder I want to write to: /var/www/media/
usage goes something like this:
if directory etc added to place_on_server: ./upload DIR etc
if directory of directory: etc/movies ./upload DIR etc movies //and so on
if file to place_on_server: ./upload F file.txt
if file in file in place_on_server ./upload F etc file.txt //and so on
for creating a directory its supposed to execute a command like:
ssh root#192.168.1.1<<EOF
cd /var/www/media/wherever
mkdir newdirectory
EOF
and for file placement:
rsync -rsh='ssh -p22' file root#192.168.1.1:/var/www/media/wherever
script:
#!/bin/bash
addr=$(ifconfig -a | ./test)
if ($# -le "1")
then
exit
elif ($1 -eq "DIR")
then
f1="ssh -b root#$addr<<EOF"
list = "cd /var/www/media\n"
if($# -eq "2")
then
list=list+"mkdir $2\nEOF\n"
else
num=2
i=$(($num))
while($num < $#)
do
i=$(($num))
list=list+"mkdir $i\n"
list=list+"cd $i\n"
$num=$num+1
done
fi
echo $list
elif ($1 -eq "F")
then
#list = "cd /var/www/media\n"
f2="rsync -rsh=\'ssh -p22\' "
f3 = "root#$addr:/var/www/media"
if($# -eq "2")
then
f2=f2+$2+" "+f3
else
num=3
i=$(($num))
while($num < $#)
do
i=$(($num))
f2=f2+"/"+$i
$num=$num+1
done
i=$(($num))
f2=f2+$i+" "+$f3
fi
echo $f2
fi
exit
output:
(prompt)$ ./upload2 F SO test.txt
./upload2: line 3: 3: command not found
./upload2: line 6: F: command not found
./upload2: line 25: F: command not found
So as you can see I'm having issues handling input. Its been awhile since I've done bash. And it was never extensive to begin with. Looking for a solution to my problem but also suggestions. Thanks in advance.
For comparisons, use [[ .. ]]. ( .. ) is for running commands in subshells
Don't use -eq for string comparisons, use =.
Don't use < for numerical comparisons, use -lt
To append values, f2="$f2$i $f3"
To add line feeds, use $'\n' outside of double quotes, or a literal linefeed inside of them.
You always need "$" on variables in strings to reference them, otherwise you get the literal string.
You can't use spaces around the = in assignments
You can't use $ before the variable name in assignments
To do arithmetics, use $((..)): result=$((var1+var2))
For indirect reference, such as getting $4 for n=4, use ${!n}
To prevent word splitting removing your line feeds, double quote variables such as in echo "$line"
Consider writing smaller programs and checking that they work before building out.
Here is how I would have written your script (slightly lacking in parameter checking):
#!/bin/bash
addr=$(ifconfig -a | ./test)
if [[ $1 = "DIR" ]]
then
shift
( IFS=/; echo ssh "root#$addr" mkdir -p "/var/www/media/$*"; )
elif [[ $1 = "F" ]]
then
shift
last=$#
file=${!last}
( IFS=/; echo rsync "$file" "root#$addr:/var/www/media/$*" )
else
echo "Unknown command '$1'"
fi
$* gives you all parameters separated by the first character in $IFS, and I used that to build the paths. Here's the output:
$ ./scriptname DIR a b c d
ssh root#somehost mkdir -p /var/www/media/a/b/c/d
$ ./scriptname F a b c d somefile.txt
rsync somefile.txt root#somehost:/var/www/media/a/b/c/d/somefile.txt
Remove the echos to actually execute.
The main problem with your script are the conditional statements, such as
if ($# -le "1")
Despite what this would do in other languages, in Bash this is essentially saying, execute the command line $# -le "1" in a subshell, and use its exit status as condition.
in your case, that expands to 3 -le "1", but the command 3 does not exist, which causes the error message
./upload2: line 3: 3: command not found
The closest valid syntax would be
if [ $# -le 1 ]
That is the main problem, there are other problems detailed and addressed in that other guy's post.
One last thing, when you're assigning value to a variable, e.g.
f3 = "root#$addr:/var/www/media"
don't leave space around the =. The statement above would be interpreted as "run command f3 with = and "root#$addr:/var/www/media" as arguments".