I am trying to push some variables into a bash array. For some reasons I cant understand, my script find the variable templates_age directly but not in the loop.
You can try the code on BASH Shell Online.
script:
templates_age="42"
templates_name="foo"
echo "age=${templates_age}"
echo "name=${templates_name}"
readarray GREPPED < <($(compgen -A variable | grep "templates_"))
for item in "${GREPPED[#]}"
do
echo "${item}"
done
output:
age=42
name=foo
./main.sh: line 32: templates_age: command not found
I tried different kind of echo "${item}" without success.
To convert from grep to array, I am using this logic.
To correctly populate array from a command's output use process substitution without $(...) which is called command substitution:
readarray -t grepped < <(compgen -A variable | grep "templates_")
Also note use of -t to trim newlines.
Full script:
templates_age="42"
templates_name="foo"
echo "age=${templates_age}"
echo "name=${templates_name}"
readarray -t grepped < <(compgen -A variable | grep "templates_")
declare -p grepped
for item in "${grepped[#]}"
do
printf "%s=%s\n" "${item}" "${!item}"
done
I'm not sure why you want to use compgen and grep here. Wouldn't this be enough?
for item in "${!templates_#}"; do
printf '%s=%s\n' "$item" "${!item}"
done
If you really want to populate an array, it's as simple as:
grepped=( "${!templates_#}" )
See Shell Parameter Expansion in the reference manual.
Related
Hi guys I got this bash one line that i wish to make a script
for i in 'ls *.fastq.gz'; do echo $(zcat ${i} | wc -l)/4|bc; done
I would like to make it as a script to read from a data dir and print out the result with the name of the file.
I tried to put the dir in front of the 'data/*.fastq.gz' but got am error No such dir exist...
I would like some like this:
name1.fastq.gz 1898516
name2.fastq.gz 2467421
namen.fastq.gz 1234532
I am not experienced in bash.
Could you guys give a help?
Thanks
Take the dir as an argument, but default to the current dir if it's not set.
dir="${1-.}"
Then put it in the glob: "$dir"/*.fastq.gz
As well:
Quote variables and command expansions.
Don't parse ls.
Don't trust echo with arbitrary data (filenames). Use printf instead.
Use an end-of-options flag -- when giving filenames to commands.
I prefer to not have any inline command expansions, but that's just personal preference
Putting it together:
#!/bin/bash
dir="${1-.}"
for file in "$dir"/*.fastq.gz; do
printf '%s ' "$file"
lines="$(zcat -- "$file" | wc -l)"
bc <<< "$lines/4" # Using a here-string (Bash feature)
done
There is no need to escape to bc for integer math (divide by 4), or to use 'ls' to enumerate the files. The original version will do with minor changes:
#!/bin/bash
dir="${1-.}"
for i in "$dir"/*.fastq.gz; do
lines=$(zcat "${i}" | wc -l)
printf '%s %d\n' "$i" "$((lines/4))"
done
This question already has answers here:
Bash doesn't parse quotes when converting a string to arguments
(5 answers)
Closed 6 years ago.
Say I have a variable $ARGS which contains the following:
file1.txt "second file.txt" file3.txt
How can I pass the contents of $ARGS as arguments to a command (say cat $ARGS, for example), treating "second file.txt" as one argument and not splitting it into "second and file.txt"?
Ideally, I'd like to be able to pass arguments to any command exactly as they are stored in a variable (read from a text file, but I don't think that's pertinent).
Thanks!
It's possible to do this without either bash arrays or eval: This is one of the few places where the behavior of xargs without either -0 or -d extensions (a behavior which mostly creates bugs) is actually useful.
# this will print each argument on a different line
# ...note that it breaks with arguments containing literal newlines!
xargs printf '%s\n' <<<"$ARGS"
...or...
# this will emit arguments in a NUL-delimited stream
xargs printf '%s\0' <<<"$ARGS"
# in bash 4.4, you can read this into an array like so:
readarray -t -d '' args < <(xargs printf '%s\0' <<<"$ARGS")
yourprog "${args[#]}" # actually run your programs
# in bash 3.x or newer, it's just a bit longer:
args=( );
while IFS= read -r -d '' arg; do
args+=( "$arg" )
done < <(xargs printf '%s\0' <<<"$ARGS")
yourprog "${args[#]}" # actually run your program
# in POSIX sh, you can't safely handle arguments with literal newlines
# ...but, barring that, can do it like this:
set --
while IFS= read -r arg; do
set -- "$#" "$arg"
done < <(printf '%s\n' "$ARGS" | xargs printf '%s\n')
yourprog "$#" # actually run your program
...or, letting xargs itself do the invocation:
# this will call yourprog with ARGS given
# ...but -- beware! -- will cause bugs if there are more arguments than will fit on one
# ...command line invocation.
printf '%s\n' "$ARGS" | xargs yourprog
As mentioned by Jonathan Leffler you can do this with an array.
my_array=( "file1.txt" "second file.txt" "file3.txt" )
cat "${my_array[1]}"
An array's index starts at 0. So if you wanted to cat the first file in your array you would use the index number 0. "${my_array[0]}". If you wanted to run your command on all elements, replace the index number with # or *. For instance instead of "${my_arryay[0]}" you would use "${my_array[#]}"Make sure you quote the array or it will treat any filename with spaces as separate files.
Alternatively if for some reason quoting the array is a problem, you can set IFS (which stands for Internal Field Separator) to equal a newline. If you do this, it's a good idea to save the default IFS to a variable before changing it so you can set it back to the way it was once the script completes. For instance:
# save IFS to a variable
old_IFS=${IFS-$' \t\n'}
#set IFS to a newline
IFS='$\n'
# run your script
my_array=( "file1.txt" "second file.txt" "file3.txt" )
cat ${my_array[1]}
# restore IFS to its default state
IFS=$old_IFS
It's probably better to not mess around with IFS unless you have to. If you can quote the array to make your script work then you should do that.
For a much more in depth look into using arrays see:
http://mywiki.wooledge.org/BashGuide/Arrays
http://wiki.bash-hackers.org/syntax/arrays
http://mywiki.wooledge.org/BashFAQ/005
Without bashisms, plain shell code might need an eval:
# make three temp files and list them.
cd /tmp ; echo ho > ho ; echo ho ho > "ho ho" ; echo ha > ha ;
A='ho "ho ho" ha' ; eval grep -n '.' $A
Output:
ho:1:ho
ho ho:1:ho ho
ha:1:ha
Note that eval is powerful, and if not used responsibly can lead to mischief...
To get started, here's the script I'm running to get the offending string:
# sed finds all sourced file paths from inputted file.
#
# while reads each match output from sed to $SOURCEFILE variable.
# Each should be a file path, or a variable that represents a file path.
# Any variables found should be expanded to the full path.
#
# echo and calls are used for demonstractive purposes only
# I intend to do something else with the path once it's expanded.
PATH_SOME_SCRIPT="/path/to/bash/script"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
"$SOURCEFILE"
$SOURCEFILE
done < <(cat $PATH_SOME_SCRIPT | sed -n -e "s/^\(source\|\.\|\$include\) //p")
You may also wish to use the following to test this out as mock data:
[ /path/to/bash/script ]
#!/bin/bash
source "$HOME/bash_file"
source "$GLOBAL_VAR_SCRIPT_PATH"
echo "No cow powers here"
For the tl;dr crew, basically the while loop spits out the following on the mock data:
"$HOME/bash_file"
bash: "$HOME/bash_file": no such file or directory
bash: "$HOME/bash_file": no such file or directory
"$GLOBAL_VAR_SCRIPT_PATH"
"$GLOBAL_VAR_SCRIPT_PATH": command not found
"$GLOBAL_VAR_SCRIPT_PATH": command not found
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"? I should also state that although eval works I do not intend to use it because of its potential insecurities.
Protip that any variable value used in cat | sed would be available globally, including to the calling script, so it's not because the script cannot call the variable value.
FIRST SOLUTION ATTEMPT
Using anubhava's envsubst solution:
SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
This outputs the following:
"$SOMEVARIABLE"
""
"/home/nick/.another_file"
"/home/nick/.another_file"
Unfortunately, it does not expand the variable! Oh dear :(
SECOND SOLUTION ATTEMPT
Based upon the first attempt:
export SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
unset SOMEVARIABLE
which produces the results we wanted without eval and without messing with global variables (for too long anyway), hoorah!
Good runner-ups were further suggested using eval (although potentially unsafe) which can be found in this answer and here (link courtesy of anubhava's extended comments).
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"?
Yes you can use envsubst program, that substitutes the values of environment variables:
while read -r sourceFile; do
envsubst <<< "$sourceFile"
done < <(sed -n "s/^\(source\|\.\|\$include\) //p" "$PATH_SOME_SCRIPT")
I think you are asking how to recursively expand variables in bash. Try
expanded=$(eval echo $SOURCEFILE)
inside your loop. eval runs the expanded command you give it. Since $SOURCEFILE isn't in quotes, it will be expanded to, e.g., $HOME/whatever. Then the eval will expand the $HOME before passing it to echo. echo will print the result, and expanded=$(...) will put the printed result in $expanded.
Somewhere I found this command that sorts lines in an input file by number of characters(1st order) and alphabetically (2nd order):
while read -r l; do echo "${#l} $l"; done < input.txt | sort -n | cut -d " " -f 2- > output.txt
It works fine but I would like to use the command in a bash script where the name of the file to be sorted is an argument:
& cat numbersort.sh
#!/bin/sh
while read -r l; do echo "${#l} $l"; done < $1 | sort -n | cut -d " " -f 2- > sorted-$1
Entering numbersort.sh input-txt doesn't give the desired result, probably because $1 is already in using as an argument for something else.
How do I make the command work in a shell script?
There's nothing wrong with your original script when used with simple arguments that don't involve quoting issues. That said, there are a few bugs addressed in the below version:
#!/bin/bash
while IFS= read -r line; do
printf '%d %s\n' "${#line}" "$line"
done <"$1" | sort -n | cut -d " " -f 2- >"sorted-$1"
Use #!/bin/bash if your goal is to write a bash script; #!/bin/sh is the shebang for POSIX sh scripts, not bash.
Clear IFS to avoid pruning leading and trailing whitespace from input and output lines
Use printf rather than echo to avoid ambiguities in the POSIX standard (see http://pubs.opengroup.org/onlinepubs/009604599/utilities/echo.html, particularly APPLICATION USAGE and RATIONALE sections).
Quote expansions ("$1" rather than $1) to prevent them from being word-split or glob-expanded
Note also that this creates a new file rather than operating in-place. If you want something that operates in-place, tack a && mv -- "sorted-$1" "$1" on the end.
I have a loop in a bash file to show me all of the files in a directory, each as its own variable. I need to take that variable (filename) and parse out only a section of it.
Example:
92378478234978ehbWHATIWANT#98712398712398723
Now, assuming "ehb" and the pound symbol never change, how can I just capture WHATIWANT into its own variable?
So far I have:
#!/bin/bash
for FILENAME in `dir -d *` ; do
done
You can use sed to edit out the parts you don't want.
want=$(echo "$FILENAME" | sed -e 's/.*ehb\(.*\)#.*/\1/')
Or you can use Bash's parameter expansion to strip out the tail and head.
want=${FILENAME%#*}; want=${want#*ehb}
One possibility:
for i in '92378478234978ehbWHATIWANT#98712398712398723' ; do
j=$(echo $i | sed -e 's/^.*ehb//' -e 's/#.*$//')
echo $j
done
produces:
WHATIWANT
using only the bash shell, no need external tools
$ string=92378478234978ehbWHATIWANT#98712398712398723
$ echo ${string#*ehb}
WHATIWANT#98712398712398723
$ string=${string#*ehb}
$ echo ${string%#*}
WHATIWANT