As we know, in bash programming the way to pass arguments is $1, ..., $N. However, I found it not easy to pass an array as an argument to a function which receives more than one argument. Here is one example:
f(){
x=($1)
y=$2
for i in "${x[#]}"
do
echo $i
done
....
}
a=("jfaldsj jflajds" "LAST")
b=NOEFLDJF
f "${a[#]}" $b
f "${a[*]}" $b
As described, function freceives two arguments: the first is assigned to x which is an array, the second to y.
f can be called in two ways. The first way use the "${a[#]}" as the first argument, and the result is:
jfaldsj
jflajds
The second way use the "${a[*]}" as the first argument, and the result is:
jfaldsj
jflajds
LAST
Neither result is as I wished. So, is there anyone having any idea about how to pass array between functions correctly?
You cannot pass an array, you can only pass its elements (i.e. the expanded array).
#!/bin/bash
function f() {
a=("$#")
((last_idx=${#a[#]} - 1))
b=${a[last_idx]}
unset a[last_idx]
for i in "${a[#]}" ; do
echo "$i"
done
echo "b: $b"
}
x=("one two" "LAST")
b='even more'
f "${x[#]}" "$b"
echo ===============
f "${x[*]}" "$b"
The other possibility would be to pass the array by name:
#!/bin/bash
function f() {
name=$1[#]
b=$2
a=("${!name}")
for i in "${a[#]}" ; do
echo "$i"
done
echo "b: $b"
}
x=("one two" "LAST")
b='even more'
f x "$b"
You can pass an array by name reference to a function in bash (since version 4.3+), by setting the -n attribute:
show_value () # array index
{
local -n myarray=$1
local idx=$2
echo "${myarray[$idx]}"
}
This works for indexed arrays:
$ shadock=(ga bu zo meu)
$ show_value shadock 2
zo
It also works for associative arrays:
$ declare -A days=([monday]=eggs [tuesday]=bread [sunday]=jam)
$ show_value days sunday
jam
See also nameref or declare -n in the man page.
You could pass the "scalar" value first. That would simplify things:
f(){
b=$1
shift
a=("$#")
for i in "${a[#]}"
do
echo $i
done
....
}
a=("jfaldsj jflajds" "LAST")
b=NOEFLDJF
f "$b" "${a[#]}"
At this point, you might as well use the array-ish positional params directly
f(){
b=$1
shift
for i in "$#" # or simply "for i; do"
do
echo $i
done
....
}
f "$b" "${a[#]}"
This will solve the issue of passing array to function:
#!/bin/bash
foo() {
string=$1
array=($#)
echo "array is ${array[#]}"
echo "array is ${array[1]}"
return
}
array=( one two three )
foo ${array[#]}
colors=( red green blue )
foo ${colors[#]}
Try like this
function parseArray {
array=("$#")
for data in "${array[#]}"
do
echo ${data}
done
}
array=("value" "value1")
parseArray "${array[#]}"
Pass the array as a function
array() {
echo "apple pear"
}
printArray() {
local argArray="${1}"
local array=($($argArray)) # where the magic happens. careful of the surrounding brackets.
for arrElement in "${array[#]}"; do
echo "${arrElement}"
done
}
printArray array
Here is an example where I receive 2 bash arrays into a function, as well as additional arguments after them. This pattern can be continued indefinitely for any number of bash arrays and any number of additional arguments, accommodating any input argument order, so long as the length of each bash array comes just before the elements of that array.
Function definition for print_two_arrays_plus_extra_args:
# Print all elements of a bash array.
# General form:
# print_one_array array1
# Example usage:
# print_one_array "${array1[#]}"
print_one_array() {
for element in "$#"; do
printf " %s\n" "$element"
done
}
# Print all elements of two bash arrays, plus two extra args at the end.
# General form (notice length MUST come before the array in order
# to be able to parse the args!):
# print_two_arrays_plus_extra_args array1_len array1 array2_len array2 \
# extra_arg1 extra_arg2
# Example usage:
# print_two_arrays_plus_extra_args "${#array1[#]}" "${array1[#]}" \
# "${#array2[#]}" "${array2[#]}" "hello" "world"
print_two_arrays_plus_extra_args() {
i=1
# Read array1_len into a variable
array1_len="${#:$i:1}"
((i++))
# Read array1 into a new array
array1=("${#:$i:$array1_len}")
((i += $array1_len))
# Read array2_len into a variable
array2_len="${#:$i:1}"
((i++))
# Read array2 into a new array
array2=("${#:$i:$array2_len}")
((i += $array2_len))
# You can now read the extra arguments all at once and gather them into a
# new array like this:
extra_args_array=("${#:$i}")
# OR you can read the extra arguments individually into their own variables
# one-by-one like this
extra_arg1="${#:$i:1}"
((i++))
extra_arg2="${#:$i:1}"
((i++))
# Print the output
echo "array1:"
print_one_array "${array1[#]}"
echo "array2:"
print_one_array "${array2[#]}"
echo "extra_arg1 = $extra_arg1"
echo "extra_arg2 = $extra_arg2"
echo "extra_args_array:"
print_one_array "${extra_args_array[#]}"
}
Example usage:
array1=()
array1+=("one")
array1+=("two")
array1+=("three")
array2=("four" "five" "six" "seven" "eight")
echo "Printing array1 and array2 plus some extra args"
# Note that `"${#array1[#]}"` is the array length (number of elements
# in the array), and `"${array1[#]}"` is the array (all of the elements
# in the array)
print_two_arrays_plus_extra_args "${#array1[#]}" "${array1[#]}" \
"${#array2[#]}" "${array2[#]}" "hello" "world"
Example Output:
Printing array1 and array2 plus some extra args
array1:
one
two
three
array2:
four
five
six
seven
eight
extra_arg1 = hello
extra_arg2 = world
extra_args_array:
hello
world
For further examples and detailed explanations of how this works, see my longer answer on this topic here: Passing arrays as parameters in bash
You can also create a json file with an array, and then parse that json file with jq
For example:
my-array.json:
{
"array": ["item1","item2"]
}
script.sh:
ARRAY=$(jq -r '."array"' $1 | tr -d '[],"')
And then call the script like:
script.sh ./path-to-json/my-array.json
I have a function that creates an array and I want to return the array to the caller:
create_array() {
local my_list=("a", "b", "c")
echo "${my_list[#]}"
}
my_algorithm() {
local result=$(create_array)
}
With this, I only get an expanded string. How can I "return" my_list without using anything global?
With Bash version 4.3 and above, you can make use of a nameref so that the caller can pass in the array name and the callee can use a nameref to populate the named array, indirectly.
#!/usr/bin/env bash
create_array() {
local -n arr=$1 # use nameref for indirection
arr=(one "two three" four)
}
use_array() {
local my_array
create_array my_array # call function to populate the array
echo "inside use_array"
declare -p my_array # test the array
}
use_array # call the main function
Produces the output:
inside use_array
declare -a my_array=([0]="one" [1]="two three" [2]="four")
You could make the function update an existing array as well:
update_array() {
local -n arr=$1 # use nameref for indirection
arr+=("two three" four) # update the array
}
use_array() {
local my_array=(one)
update_array my_array # call function to update the array
}
This is a more elegant and efficient approach since we don't need command substitution $() to grab the standard output of the function being called. It also helps if the function were to return more than one output - we can simply use as many namerefs as the number of outputs.
Here is what the Bash Manual says about nameref:
A variable can be assigned the nameref attribute using the -n option
to the declare or local builtin commands (see Bash Builtins) to create
a nameref, or a reference to another variable. This allows variables
to be manipulated indirectly. Whenever the nameref variable is
referenced, assigned to, unset, or has its attributes modified (other
than using or changing the nameref attribute itself), the operation is
actually performed on the variable specified by the nameref variable’s
value. A nameref is commonly used within shell functions to refer to a
variable whose name is passed as an argument to the function. For
instance, if a variable name is passed to a shell function as its
first argument, running
declare -n ref=$1 inside the function creates a nameref variable ref
whose value is the variable name passed as the first argument.
References and assignments to ref, and changes to its attributes, are
treated as references, assignments, and attribute modifications to the
variable whose name was passed as $1.
What's wrong with globals?
Returning arrays is really not practical. There are lots of pitfalls.
That said, here's one technique that works if it's OK that the variable have the same name:
$ f () { local a; a=(abc 'def ghi' jkl); declare -p a; }
$ g () { local a; eval $(f); declare -p a; }
$ f; declare -p a; echo; g; declare -p a
declare -a a='([0]="abc" [1]="def ghi" [2]="jkl")'
-bash: declare: a: not found
declare -a a='([0]="abc" [1]="def ghi" [2]="jkl")'
-bash: declare: a: not found
The declare -p commands (except for the one in f() are used to display the state of the array for demonstration purposes. In f() it's used as the mechanism to return the array.
If you need the array to have a different name, you can do something like this:
$ g () { local b r; r=$(f); r="declare -a b=${r#*=}"; eval "$r"; declare -p a; declare -p b; }
$ f; declare -p a; echo; g; declare -p a
declare -a a='([0]="abc" [1]="def ghi" [2]="jkl")'
-bash: declare: a: not found
-bash: declare: a: not found
declare -a b='([0]="abc" [1]="def ghi" [2]="jkl")'
-bash: declare: a: not found
Bash can't pass around data structures as return values. A return value must be a numeric exit status between 0-255. However, you can certainly use command or process substitution to pass commands to an eval statement if you're so inclined.
This is rarely worth the trouble, IMHO. If you must pass data structures around in Bash, use a global variable--that's what they're for. If you don't want to do that for some reason, though, think in terms of positional parameters.
Your example could easily be rewritten to use positional parameters instead of global variables:
use_array () {
for idx in "$#"; do
echo "$idx"
done
}
create_array () {
local array=("a" "b" "c")
use_array "${array[#]}"
}
This all creates a certain amount of unnecessary complexity, though. Bash functions generally work best when you treat them more like procedures with side effects, and call them in sequence.
# Gather values and store them in FOO.
get_values_for_array () { :; }
# Do something with the values in FOO.
process_global_array_variable () { :; }
# Call your functions.
get_values_for_array
process_global_array_variable
If all you're worried about is polluting your global namespace, you can also use the unset builtin to remove a global variable after you're done with it. Using your original example, let my_list be global (by removing the local keyword) and add unset my_list to the end of my_algorithm to clean up after yourself.
You were not so far out with your original solution. You had a couple of problems, you used a comma as a separator, and you failed to capture the returned items into a list, try this:
my_algorithm() {
local result=( $(create_array) )
}
create_array() {
local my_list=("a" "b" "c")
echo "${my_list[#]}"
}
Considering the comments about embedded spaces, a few tweaks using IFS can solve that:
my_algorithm() {
oldIFS="$IFS"
IFS=','
local result=( $(create_array) )
IFS="$oldIFS"
echo "Should be 'c d': ${result[1]}"
}
create_array() {
IFS=','
local my_list=("a b" "c d" "e f")
echo "${my_list[*]}"
}
Use the technique developed by Matt McClure:
http://notes-matthewlmcclure.blogspot.com/2009/12/return-array-from-bash-function-v-2.html
Avoiding global variables means you can use the function in a pipe. Here is an example:
#!/bin/bash
makeJunk()
{
echo 'this is junk'
echo '#more junk and "b#d" characters!'
echo '!#$^%^&(*)_^&% ^$##:"<>?/.,\\"'"'"
}
processJunk()
{
local -a arr=()
# read each input and add it to arr
while read -r line
do
arr+=('"'"$line"'" is junk')
done;
# output the array as a string in the "declare" representation
declare -p arr | sed -e 's/^declare -a [^=]*=//'
}
# processJunk returns the array in a flattened string ready for "declare"
# Note that because of the pipe processJunk cannot return anything using
# a global variable
returned_string="$(makeJunk | processJunk)"
# convert the returned string to an array named returned_array
# declare correctly manages spaces and bad characters
eval "declare -a returned_array=${returned_string}"
for junk in "${returned_array[#]}"
do
echo "$junk"
done
Output is:
"this is junk" is junk
"#more junk and "b#d" characters!" is junk
"!#$^%^&(*)_^&% ^$##:"<>?/.,\\"'" is junk
A pure bash, minimal and robust solution based on the 'declare -p' builtin — without insane global variables
This approach involves the following three steps:
Convert the array with 'declare -p' and save the output in a variable.
myVar="$( declare -p myArray )"
The output of the declare -p statement can be used to recreate the array.
For instance the output of declare -p myVar might look like this:
declare -a myVar='([0]="1st field" [1]="2nd field" [2]="3rd field")'
Use the echo builtin to pass the variable to a function or to pass it back from there.
In order to preserve whitspaces in array fields when echoing the variable, IFS is temporarly set to a control character (e.g. a vertical tab).
Only the right-hand-side of the declare statement in the variable is to be echoed - this can be achieved by parameter expansion of the form ${parameter#word}. As for the example above: ${myVar#*=}
Finally, recreate the array where it is passed to using the eval and the 'declare -a' builtins.
Example 1 - return an array from a function
#!/bin/bash
# Example 1 - return an array from a function
function my-fun () {
# set up a new array with 3 fields - note the whitespaces in the
# 2nd (2 spaces) and 3rd (2 tabs) field
local myFunArray=( "1st field" "2nd field" "3rd field" )
# show its contents on stderr (must not be output to stdout!)
echo "now in $FUNCNAME () - showing contents of myFunArray" >&2
echo "by the help of the 'declare -p' builtin:" >&2
declare -p myFunArray >&2
# return the array
local myVar="$( declare -p myFunArray )"
local IFS=$'\v';
echo "${myVar#*=}"
# if the function would continue at this point, then IFS should be
# restored to its default value: <space><tab><newline>
IFS=' '$'\t'$'\n';
}
# main
# call the function and recreate the array that was originally
# set up in the function
eval declare -a myMainArray="$( my-fun )"
# show the array contents
echo ""
echo "now in main part of the script - showing contents of myMainArray"
echo "by the help of the 'declare -p' builtin:"
declare -p myMainArray
# end-of-file
Output of Example 1:
now in my-fun () - showing contents of myFunArray
by the help of the 'declare -p' builtin:
declare -a myFunArray='([0]="1st field" [1]="2nd field" [2]="3rd field")'
now in main part of the script - showing contents of myMainArray
by the help of the 'declare -p' builtin:
declare -a myMainArray='([0]="1st field" [1]="2nd field" [2]="3rd field")'
Example 2 - pass an array to a function
#!/bin/bash
# Example 2 - pass an array to a function
function my-fun () {
# recreate the array that was originally set up in the main part of
# the script
eval declare -a myFunArray="$( echo "$1" )"
# note that myFunArray is local - from the bash(1) man page: when used
# in a function, declare makes each name local, as with the local
# command, unless the ‘-g’ option is used.
# IFS has been changed in the main part of this script - now that we
# have recreated the array it's better to restore it to the its (local)
# default value: <space><tab><newline>
local IFS=' '$'\t'$'\n';
# show contents of the array
echo ""
echo "now in $FUNCNAME () - showing contents of myFunArray"
echo "by the help of the 'declare -p' builtin:"
declare -p myFunArray
}
# main
# set up a new array with 3 fields - note the whitespaces in the
# 2nd (2 spaces) and 3rd (2 tabs) field
myMainArray=( "1st field" "2nd field" "3rd field" )
# show the array contents
echo "now in the main part of the script - showing contents of myMainArray"
echo "by the help of the 'declare -p' builtin:"
declare -p myMainArray
# call the function and pass the array to it
myVar="$( declare -p myMainArray )"
IFS=$'\v';
my-fun $( echo "${myVar#*=}" )
# if the script would continue at this point, then IFS should be restored
# to its default value: <space><tab><newline>
IFS=' '$'\t'$'\n';
# end-of-file
Output of Example 2:
now in the main part of the script - showing contents of myMainArray
by the help of the 'declare -p' builtin:
declare -a myMainArray='([0]="1st field" [1]="2nd field" [2]="3rd field")'
now in my-fun () - showing contents of myFunArray
by the help of the 'declare -p' builtin:
declare -a myFunArray='([0]="1st field" [1]="2nd field" [2]="3rd field")'
I recently discovered a quirk in BASH in that a function has direct access to the variables declared in the functions higher in the call stack. I've only just started to contemplate how to exploit this feature (it promises both benefits and dangers), but one obvious application is a solution to the spirit of this problem.
I would also prefer to get a return value rather than using a global variable when delegating the creation of an array. There are several reasons for my preference, among which are to avoid possibly disturbing an preexisting value and to avoid leaving a value that may be invalid when later accessed. While there are workarounds to these problems, the easiest is have the variable go out of scope when the code is finished with it.
My solution ensures that the array is available when needed and discarded when the function returns, and leaves undisturbed a global variable with the same name.
#!/bin/bash
myarr=(global array elements)
get_an_array()
{
myarr=( $( date +"%Y %m %d" ) )
}
request_array()
{
declare -a myarr
get_an_array "myarr"
echo "New contents of local variable myarr:"
printf "%s\n" "${myarr[#]}"
}
echo "Original contents of global variable myarr:"
printf "%s\n" "${myarr[#]}"
echo
request_array
echo
echo "Confirm the global myarr was not touched:"
printf "%s\n" "${myarr[#]}"
Here is the output of this code:
When function request_array calls get_an_array, get_an_array can directly set the myarr variable that is local to request_array. Since myarr is created with declare, it is local to request_array and thus goes out of scope when request_array returns.
Although this solution does not literally return a value, I suggest that taken as a whole, it satisfies the promises of a true function return value.
Useful example: return an array from function
function Query() {
local _tmp=`echo -n "$*" | mysql 2>> zz.err`;
echo -e "$_tmp";
}
function StrToArray() {
IFS=$'\t'; set $1; for item; do echo $item; done; IFS=$oIFS;
}
sql="SELECT codi, bloc, requisit FROM requisits ORDER BY codi";
qry=$(Query $sql0);
IFS=$'\n';
for row in $qry; do
r=( $(StrToArray $row) );
echo ${r[0]} - ${r[1]} - ${r[2]};
done
I tried various implementations, and none preserved arrays that had elements with spaces ... because they all had to use echo.
# These implementations only work if no array items contain spaces.
use_array() { eval echo '(' \"\${${1}\[\#\]}\" ')'; }
use_array() { local _array="${1}[#]"; echo '(' "${!_array}" ')'; }
Solution
Then I came across Dennis Williamson's answer. I incorporated his method into the following functions so they can a) accept an arbitrary array and b) be used to pass, duplicate and append arrays.
# Print array definition to use with assignments, for loops, etc.
# varname: the name of an array variable.
use_array() {
local r=$( declare -p $1 )
r=${r#declare\ -a\ *=}
# Strip keys so printed definition will be a simple list (like when using
# "${array[#]}"). One side effect of having keys in the definition is
# that when appending arrays (i.e. `a1+=$( use_array a2 )`), values at
# matching indices merge instead of pushing all items onto array.
echo ${r//\[[0-9]\]=}
}
# Same as use_array() but preserves keys.
use_array_assoc() {
local r=$( declare -p $1 )
echo ${r#declare\ -a\ *=}
}
Then, other functions can return an array using catchable output or indirect arguments.
# catchable output
return_array_by_printing() {
local returnme=( "one" "two" "two and a half" )
use_array returnme
}
eval test1=$( return_array_by_printing )
# indirect argument
return_array_to_referenced_variable() {
local returnme=( "one" "two" "two and a half" )
eval $1=$( use_array returnme )
}
return_array_to_referenced_variable test2
# Now both test1 and test2 are arrays with three elements
I needed a similar functionality recently, so the following is a mix of the suggestions made by RashaMatt and Steve Zobell.
echo each array/list element as separate line from within a function
use mapfile to read all array/list elements echoed by a function.
As far as I can see, strings are kept intact and whitespaces are preserved.
#!bin/bash
function create-array() {
local somearray=("aaa" "bbb ccc" "d" "e f g h")
for elem in "${somearray[#]}"
do
echo "${elem}"
done
}
mapfile -t resa <<< "$(create-array)"
# quick output check
declare -p resa
Some more variations…
#!/bin/bash
function create-array-from-ls() {
local somearray=("$(ls -1)")
for elem in "${somearray[#]}"
do
echo "${elem}"
done
}
function create-array-from-args() {
local somearray=("$#")
for elem in "${somearray[#]}"
do
echo "${elem}"
done
}
mapfile -t resb <<< "$(create-array-from-ls)"
mapfile -t resc <<< "$(create-array-from-args 'xxx' 'yy zz' 't s u' )"
sentenceA="create array from this sentence"
sentenceB="keep this sentence"
mapfile -t resd <<< "$(create-array-from-args ${sentenceA} )"
mapfile -t rese <<< "$(create-array-from-args "$sentenceB" )"
mapfile -t resf <<< "$(create-array-from-args "$sentenceB" "and" "this words" )"
# quick output check
declare -p resb
declare -p resc
declare -p resd
declare -p rese
declare -p resf
Here is a solution with no external array references and no IFS manipulation:
# add one level of single quotes to args, eval to remove
squote () {
local a=("$#")
a=("${a[#]//\'/\'\\\'\'}") # "'" => "'\''"
a=("${a[#]/#/\'}") # add "'" prefix to each word
a=("${a[#]/%/\'}") # add "'" suffix to each word
echo "${a[#]}"
}
create_array () {
local my_list=(a "b 'c'" "\\\"d
")
squote "${my_list[#]}"
}
my_algorithm () {
eval "local result=($(create_array))"
# result=([0]="a" [1]="b 'c'" [2]=$'\\"d\n')
}
[Note: the following was rejected as an edit of this answer for reasons that make no sense to me (since the edit was not intended to address the author of the post!), so I'm taking the suggestion to make it a separate answer.]
A simpler implementation of Steve Zobell's adaptation of Matt McClure's technique uses the bash built-in (since version == 4) readarray as suggested by RastaMatt to create a representation of an array that can be converted into an array at runtime. (Note that both readarray and mapfile name the same code.) It still avoids globals (allowing use of the function in a pipe), and still handles nasty characters.
For some more-fully-developed (e.g., more modularization) but still-kinda-toy examples, see bash_pass_arrays_between_functions. Following are a few easily-executable examples, provided here to avoid moderators b!tching about external links.
Cut the following block and paste it into a bash terminal to create /tmp/source.sh and /tmp/junk1.sh:
FP='/tmp/source.sh' # path to file to be created for `source`ing
cat << 'EOF' > "${FP}" # suppress interpretation of variables in heredoc
function make_junk {
echo 'this is junk'
echo '#more junk and "b#d" characters!'
echo '!#$^%^&(*)_^&% ^$##:"<>?/.,\\"'"'"
}
### Use 'readarray' (aka 'mapfile', bash built-in) to read lines into an array.
### Handles blank lines, whitespace and even nastier characters.
function lines_to_array_representation {
local -a arr=()
readarray -t arr
# output array as string using 'declare's representation (minus header)
declare -p arr | sed -e 's/^declare -a [^=]*=//'
}
EOF
FP1='/tmp/junk1.sh' # path to script to run
cat << 'EOF' > "${FP1}" # suppress interpretation of variables in heredoc
#!/usr/bin/env bash
source '/tmp/source.sh' # to reuse its functions
returned_string="$(make_junk | lines_to_array_representation)"
eval "declare -a returned_array=${returned_string}"
for elem in "${returned_array[#]}" ; do
echo "${elem}"
done
EOF
chmod u+x "${FP1}"
# newline here ... just hit Enter ...
Run /tmp/junk1.sh: output should be
this is junk
#more junk and "b#d" characters!
!#$^%^&(*)_^&% ^$##:"<>?/.,\\"'
Note lines_to_array_representation also handles blank lines. Try pasting the following block into your bash terminal:
FP2='/tmp/junk2.sh' # path to script to run
cat << 'EOF' > "${FP2}" # suppress interpretation of variables in heredoc
#!/usr/bin/env bash
source '/tmp/source.sh' # to reuse its functions
echo '`bash --version` the normal way:'
echo '--------------------------------'
bash --version
echo # newline
echo '`bash --version` via `lines_to_array_representation`:'
echo '-----------------------------------------------------'
bash_version="$(bash --version | lines_to_array_representation)"
eval "declare -a returned_array=${bash_version}"
for elem in "${returned_array[#]}" ; do
echo "${elem}"
done
echo # newline
echo 'But are they *really* the same? Ask `diff`:'
echo '-------------------------------------------'
echo 'You already know how to capture normal output (from `bash --version`):'
declare -r PATH_TO_NORMAL_OUTPUT="$(mktemp)"
bash --version > "${PATH_TO_NORMAL_OUTPUT}"
echo "normal output captured to file # ${PATH_TO_NORMAL_OUTPUT}"
ls -al "${PATH_TO_NORMAL_OUTPUT}"
echo # newline
echo 'Capturing L2AR takes a bit more work, but is not onerous.'
echo "Look # contents of the file you're about to run to see how it's done."
declare -r RAW_L2AR_OUTPUT="$(bash --version | lines_to_array_representation)"
declare -r PATH_TO_COOKED_L2AR_OUTPUT="$(mktemp)"
eval "declare -a returned_array=${RAW_L2AR_OUTPUT}"
for elem in "${returned_array[#]}" ; do
echo "${elem}" >> "${PATH_TO_COOKED_L2AR_OUTPUT}"
done
echo "output from lines_to_array_representation captured to file # ${PATH_TO_COOKED_L2AR_OUTPUT}"
ls -al "${PATH_TO_COOKED_L2AR_OUTPUT}"
echo # newline
echo 'So are they really the same? Per'
echo "\`diff -uwB "${PATH_TO_NORMAL_OUTPUT}" "${PATH_TO_COOKED_L2AR_OUTPUT}" | wc -l\`"
diff -uwB "${PATH_TO_NORMAL_OUTPUT}" "${PATH_TO_COOKED_L2AR_OUTPUT}" | wc -l
echo '... they are the same!'
EOF
chmod u+x "${FP2}"
# newline here ... just hit Enter ...
Run /tmp/junk2.sh # commandline. Your output should be similar to mine:
`bash --version` the normal way:
--------------------------------
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
`bash --version` via `lines_to_array_representation`:
-----------------------------------------------------
GNU bash, version 4.3.30(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
But are they *really* the same? Ask `diff`:
-------------------------------------------
You already know how to capture normal output (from `bash --version`):
normal output captured to file # /tmp/tmp.Ni1bgyPPEw
-rw------- 1 me me 308 Jun 18 16:27 /tmp/tmp.Ni1bgyPPEw
Capturing L2AR takes a bit more work, but is not onerous.
Look # contents of the file you're about to run to see how it's done.
output from lines_to_array_representation captured to file # /tmp/tmp.1D6O2vckGz
-rw------- 1 me me 308 Jun 18 16:27 /tmp/tmp.1D6O2vckGz
So are they really the same? Per
`diff -uwB /tmp/tmp.Ni1bgyPPEw /tmp/tmp.1D6O2vckGz | wc -l`
0
... they are the same!
There's no need to use eval or to change IFS to \n. There are at least 2 good ways to do this.
1) Using echo and mapfile
You can simply echo each item of the array in the function, then use mapfile to turn it into an array:
outputArray()
{
for i
{
echo "$i"
}
}
declare -a arr=( 'qq' 'www' 'ee rr' )
mapfile -t array < <(outputArray "${arr[#]}")
for i in "${array[#]}"
do
echo "i=$i"
done
To make it work using pipes, add (( $# == 0 )) && readarray -t temp && set "${temp[#]}" && unset temp to the top of output array. It converts stdin to parameters.
2) Using declare -p and sed
This can also be done using declare -p and sed instead of mapfile.
outputArray()
{
(( $# == 0 )) && readarray -t temp && set "${temp[#]}" && unset temp
for i; { echo "$i"; }
}
returnArray()
{
local -a arr=()
(( $# == 0 )) && readarray -t arr || for i; { arr+=("$i"); }
declare -p arr | sed -e 's/^declare -a [^=]*=//'
}
declare -a arr=( 'qq' 'www' 'ee rr' )
declare -a array=$(returnArray "${arr[#]}")
for i in "${array[#]}"
do
echo "i=$i"
done
declare -a array=$(outputArray "${arr[#]}" | returnArray)
echo
for i in "${array[#]}"
do
echo "i=$i"
done
declare -a array < <(outputArray "${arr[#]}" | returnArray)
echo
for i in "${array[#]}"
do
echo "i=$i"
done
This can also be done by simply passing array variable to the function and assign array values to this var then use this var outside of function. For example.
create_array() {
local __resultArgArray=$1
local my_list=("a" "b" "c")
eval $__resultArgArray="("${my_list[#]}")"
}
my_algorithm() {
create_array result
echo "Total elements in the array: ${#result[#]}"
for i in "${result[#]}"
do
echo $i
done
}
my_algorithm
The easest way y found
my_function()
{
array=(one two three)
echo ${array[#]}
}
result=($(my_function))
echo ${result[0]}
echo ${result[1]}
echo ${result[2]}
You can also use the declare -p method more easily by taking advantage of declare -a's double-evaluation when the value is a string (no true parens outside the string):
# return_array_value returns the value of array whose name is passed in.
# It turns the array into a declaration statement, then echos the value
# part of that statement with parentheses intact. You can use that
# result in a "declare -a" statement to create your own array with the
# same value. Also works for associative arrays with "declare -A".
return_array_value () {
declare Array_name=$1 # namespace locals with caps to prevent name collision
declare Result
Result=$(declare -p $Array_name) # dehydrate the array into a declaration
echo "${Result#*=}" # trim "declare -a ...=" from the front
}
# now use it. test for robustness by skipping an index and putting a
# space in an entry.
declare -a src=([0]=one [2]="two three")
declare -a dst="$(return_array_value src)" # rehydrate with double-eval
declare -p dst
> declare -a dst=([0]="one" [2]="two three") # result matches original
Verifying the result, declare -p dst yields declare -a dst=([0]="one" [2]="two three")", demonstrating that this method correctly deals with both sparse arrays as well as entries with an IFS character (space).
The first thing is to dehydrate the source array by using declare -p to generate a valid bash declaration of it. Because the declaration is a full statement, including "declare" and the variable name, we strip that part from the front with ${Result#*=}, leaving the parentheses with the indices and values inside: ([0]="one" [2]="two three").
It then rehydrates the array by feeding that value to your own declare statement, one where you choose the array name. It relies on the fact that the right side of the dst array declaration is a string with parentheses that are inside the string, rather than true parentheses in the declare itself, e.g. not declare -a dst=( "true parens outside string" ). This triggers declare to evaluate the string twice, once into a valid statement with parentheses (and quotes in the value preserved), and another for the actual assignment. I.e. it evaluates first to declare -a dst=([0]="one" [2]="two three"), then evaluates that as a statement.
Note that this double evaluation behavior is specific to the -a and -A options of declare.
Oh, and this method works with associative arrays as well, just change -a to -A.
Because this method relies on stdout, it works across subshell boundaries like pipelines, as others have noted.
I discuss this method in more detail in my blog post
If your source data is formatted with each list element on a separate line, then the mapfile builtin is a simple and elegant way to read a list into an array:
$ list=$(ls -1 /usr/local) # one item per line
$ mapfile -t arrayVar <<<"$list" # -t trims trailing newlines
$ declare -p arrayVar | sed 's#\[#\n[#g'
declare -a arrayVar='(
[0]="bin"
[1]="etc"
[2]="games"
[3]="include"
[4]="lib"
[5]="man"
[6]="sbin"
[7]="share"
[8]="src")'
Note that, as with the read builtin, you would not ordinarily* use mapfile in a pipeline (or subshell) because the assigned array variable would be unavailable to subsequent statements (* unless bash job control is disabled and shopt -s lastpipe is set).
$ help mapfile
mapfile: mapfile [-n count] [-O origin] [-s count] [-t] [-u fd] [-C callback] [-c quantum] [array]
Read lines from the standard input into an indexed array variable.
Read lines from the standard input into the indexed array variable ARRAY, or
from file descriptor FD if the -u option is supplied. The variable MAPFILE
is the default ARRAY.
Options:
-n count Copy at most COUNT lines. If COUNT is 0, all lines are copied.
-O origin Begin assigning to ARRAY at index ORIGIN. The default index is 0.
-s count Discard the first COUNT lines read.
-t Remove a trailing newline from each line read.
-u fd Read lines from file descriptor FD instead of the standard input.
-C callback Evaluate CALLBACK each time QUANTUM lines are read.
-c quantum Specify the number of lines read between each call to CALLBACK.
Arguments:
ARRAY Array variable name to use for file data.
If -C is supplied without -c, the default quantum is 5000. When
CALLBACK is evaluated, it is supplied the index of the next array
element to be assigned and the line to be assigned to that element
as additional arguments.
If not supplied with an explicit origin, mapfile will clear ARRAY before
assigning to it.
Exit Status:
Returns success unless an invalid option is given or ARRAY is readonly or
not an indexed array.
You can try this
my_algorithm() {
create_array list
for element in "${list[#]}"
do
echo "${element}"
done
}
create_array() {
local my_list=("1st one" "2nd two" "3rd three")
eval "${1}=()"
for element in "${my_list[#]}"
do
eval "${1}+=(\"${element}\")"
done
}
my_algorithm
The output is
1st one
2nd two
3rd three
I'd suggest piping to a code block to set values of an array. The strategy is POSIX compatible, so you get both Bash and Zsh, and doesn't run the risk of side effects like the posted solutions.
i=0 # index for our new array
declare -a arr # our new array
# pipe from a function that produces output by line
ls -l | { while read data; do i=$i+1; arr[$i]="$data"; done }
# example of reading that new array
for row in "${arr[#]}"; do echo "$row"; done
This will work for zsh and bash, and won't be affected by spaces or special characters. In the case of the OP, the output is transformed by echo, so it is not actually outputting an array, but printing it (as others mentioned shell functions return status not values). We can change it to a pipeline ready mechanism:
create_array() {
local my_list=("a", "b", "c")
for row in "${my_list[#]}"; do
echo "$row"
done
}
my_algorithm() {
i=0
declare -a result
create_array | { while read data; do i=$i+1; result[$i]="$data"; done }
}
If so inclined, one could remove the create_array pipeline process from my_algorithm and chain the two functions together
create_array | my_algorithm
A modern Bash implementation using #Q to safely output array elements:
#!/usr/bin/env bash
return_array_elements() {
local -a foo_array=('1st one' '2nd two' '3rd three')
printf '%s\n' "${foo_array[#]#Q}"
}
use_array_elements() {
local -a bar_array="($(return_array_elements))"
# Display declareation of bar_array
# which is local to this function, but whose elements
# hahaves been returned by the return_array_elements function
declare -p bar_array
}
use_array_elements
Output:
declare -a bar_array=([0]="1st one" [1]="2nd two" [2]="3rd three")
While the declare -p approach is elegant indeed, you can still create a global array using declare -g within a function and have it visible outside the scope of the function:
create_array() {
declare -ag result=("a", "b", "c")
}
my_algorithm() {
create_array
echo "${result[#]}"
}
I have a bash script that is being used in a CGI. The CGI sets the $QUERY_STRING environment variable by reading everything after the ? in the URL. For example, http://example.com?a=123&b=456&c=ok sets QUERY_STRING=a=123&b=456&c=ok.
Somewhere I found the following ugliness:
b=$(echo "$QUERY_STRING" | sed -n 's/^.*b=\([^&]*\).*$/\1/p' | sed "s/%20/ /g")
which will set $b to whatever was found in $QUERY_STRING for b. However, my script has grown to have over ten input parameters. Is there an easier way to automatically convert the parameters in $QUERY_STRING into environment variables usable by bash?
Maybe I'll just use a for loop of some sort, but it'd be even better if the script was smart enough to automatically detect each parameter and maybe build an array that looks something like this:
${parm[a]}=123
${parm[b]}=456
${parm[c]}=ok
How could I write code to do that?
Try this:
saveIFS=$IFS
IFS='=&'
parm=($QUERY_STRING)
IFS=$saveIFS
Now you have this:
parm[0]=a
parm[1]=123
parm[2]=b
parm[3]=456
parm[4]=c
parm[5]=ok
In Bash 4, which has associative arrays, you can do this (using the array created above):
declare -A array
for ((i=0; i<${#parm[#]}; i+=2))
do
array[${parm[i]}]=${parm[i+1]}
done
which will give you this:
array[a]=123
array[b]=456
array[c]=ok
Edit:
To use indirection in Bash 2 and later (using the parm array created above):
for ((i=0; i<${#parm[#]}; i+=2))
do
declare var_${parm[i]}=${parm[i+1]}
done
Then you will have:
var_a=123
var_b=456
var_c=ok
You can access these directly:
echo $var_a
or indirectly:
for p in a b c
do
name="var$p"
echo ${!name}
done
If possible, it's better to avoid indirection since it can make code messy and be a source of bugs.
you can break $QUERY down using IFS. For example, setting it to &
$ QUERY="a=123&b=456&c=ok"
$ echo $QUERY
a=123&b=456&c=ok
$ IFS="&"
$ set -- $QUERY
$ echo $1
a=123
$ echo $2
b=456
$ echo $3
c=ok
$ array=($#)
$ for i in "${array[#]}"; do IFS="=" ; set -- $i; echo $1 $2; done
a 123
b 456
c ok
And you can save to a hash/dictionary in Bash 4+
$ declare -A hash
$ for i in "${array[#]}"; do IFS="=" ; set -- $i; hash[$1]=$2; done
$ echo ${hash["b"]}
456
Please don't use the evil eval junk.
Here's how you can reliably parse the string and get an associative array:
declare -A param
while IFS='=' read -r -d '&' key value && [[ -n "$key" ]]; do
param["$key"]=$value
done <<<"${QUERY_STRING}&"
If you don't like the key check, you could do this instead:
declare -A param
while IFS='=' read -r -d '&' key value; do
param["$key"]=$value
done <<<"${QUERY_STRING:+"${QUERY_STRING}&"}"
Listing all the keys and values from the array:
for key in "${!param[#]}"; do
echo "$key: ${param[$key]}"
done
I packaged the sed command up into another script:
$cat getvar.sh
s='s/^.*'${1}'=\([^&]*\).*$/\1/p'
echo $QUERY_STRING | sed -n $s | sed "s/%20/ /g"
and I call it from my main cgi as:
id=`./getvar.sh id`
ds=`./getvar.sh ds`
dt=`./getvar.sh dt`
...etc, etc - you get idea.
works for me even with a very basic busybox appliance (my PVR in this case).
To converts the contents of QUERY_STRING into bash variables use the following command:
eval $(echo ${QUERY_STRING//&/;})
The inner step, echo ${QUERY_STRING//&/;}, substitutes all ampersands with semicolons producing a=123;b=456;c=ok which the eval then evaluates into the current shell.
The result can then be used as bash variables.
echo $a
echo $b
echo $c
The assumptions are:
values will never contain '&'
values will never contain ';'
QUERY_STRING will never contain malicious code
While the accepted answer is probably the most beautiful one, there might be cases where security is super-important, and it needs to be also well-visible from your script.
In such a case, first I wouldn't use bash for the task, but if it should be done on some reason, it might be better to avoid these new array - dictionary features, because you can't be sure, how exactly are they escaped.
In this case, the good old primitive solutions might work:
QS="${QUERY_STRING}"
while [ "${QS}" != "" ]
do
nameval="${QS%%&*}"
QS="${QS#$nameval}"
QS="${QS#&}"
name="${nameval%%=*}"
val="${nameval#$name}"
val="${nameval#=}"
# and here we have $name and $val as names and values
# ...
done
This iterates on the name-value pairs of the QUERY_STRING, and there is no way to circumvent it with any tricky escape sequence - the " is a very strong thing in bash, except a single variable name substitution, which is fully controlled by us, nothing can be tricked.
Furthermore, you can inject your own processing code into "# ...". This enables you to allow only your own, well-defined (and, ideally, short) list of the allowed variable names. Needless to say, LD_PRELOAD shouldn't be one of them. ;-)
Furthermore, no variable will be exported, and exclusively QS, nameval, name and val is used.
Following the correct answer, I've done myself some changes to support array variables like in this other question. I added also a decode function of which I can not find the author to give some credit.
Code appears somewhat messy, but it works. Changes and other recommendations would be greatly appreciated.
function cgi_decodevar() {
[ $# -ne 1 ] && return
local v t h
# replace all + with whitespace and append %%
t="${1//+/ }%%"
while [ ${#t} -gt 0 -a "${t}" != "%" ]; do
v="${v}${t%%\%*}" # digest up to the first %
t="${t#*%}" # remove digested part
# decode if there is anything to decode and if not at end of string
if [ ${#t} -gt 0 -a "${t}" != "%" ]; then
h=${t:0:2} # save first two chars
t="${t:2}" # remove these
v="${v}"`echo -e \\\\x${h}` # convert hex to special char
fi
done
# return decoded string
echo "${v}"
return
}
saveIFS=$IFS
IFS='=&'
VARS=($QUERY_STRING)
IFS=$saveIFS
for ((i=0; i<${#VARS[#]}; i+=2))
do
curr="$(cgi_decodevar ${VARS[i]})"
next="$(cgi_decodevar ${VARS[i+2]})"
prev="$(cgi_decodevar ${VARS[i-2]})"
value="$(cgi_decodevar ${VARS[i+1]})"
array=${curr%"[]"}
if [ "$curr" == "$next" ] && [ "$curr" != "$prev" ] ;then
j=0
declare var_${array}[$j]="$value"
elif [ $i -gt 1 ] && [ "$curr" == "$prev" ]; then
j=$((j + 1))
declare var_${array}[$j]="$value"
else
declare var_$curr="$value"
fi
done
I would simply replace the & to ;. It will become to something like:
a=123;b=456;c=ok
So now you need just evaluate and read your vars:
eval `echo "${QUERY_STRING}"|tr '&' ';'`
echo $a
echo $b
echo $c
A nice way to handle CGI query strings is to use Haserl which acts as a wrapper around your Bash cgi script, and offers convenient and secure query string parsing.
To bring this up to date, if you have a recent Bash version then you can achieve this with regular expressions:
q="$QUERY_STRING"
re1='^(\w+=\w+)&?'
re2='^(\w+)=(\w+)$'
declare -A params
while [[ $q =~ $re1 ]]; do
q=${q##*${BASH_REMATCH[0]}}
[[ ${BASH_REMATCH[1]} =~ $re2 ]] && params+=([${BASH_REMATCH[1]}]=${BASH_REMATCH[2]})
done
If you don't want to use associative arrays then just change the penultimate line to do what you want. For each iteration of the loop the parameter is in ${BASH_REMATCH[1]} and its value is in ${BASH_REMATCH[2]}.
Here is the same thing as a function in a short test script that iterates over the array outputs the query string's parameters and their values
#!/bin/bash
QUERY_STRING='foo=hello&bar=there&baz=freddy'
get_query_string() {
local q="$QUERY_STRING"
local re1='^(\w+=\w+)&?'
local re2='^(\w+)=(\w+)$'
while [[ $q =~ $re1 ]]; do
q=${q##*${BASH_REMATCH[0]}}
[[ ${BASH_REMATCH[1]} =~ $re2 ]] && eval "$1+=([${BASH_REMATCH[1]}]=${BASH_REMATCH[2]})"
done
}
declare -A params
get_query_string params
for k in "${!params[#]}"
do
v="${params[$k]}"
echo "$k : $v"
done
Note the parameters end up in the array in reverse order (it's associative so that shouldn't matter).
why not this
$ echo "${QUERY_STRING}"
name=carlo&last=lanza&city=pfungen-CH
$ saveIFS=$IFS
$ IFS='&'
$ eval $QUERY_STRING
$ IFS=$saveIFS
now you have this
name = carlo
last = lanza
city = pfungen-CH
$ echo "name is ${name}"
name is carlo
$ echo "last is ${last}"
last is lanza
$ echo "city is ${city}"
city is pfungen-CH
#giacecco
To include a hiphen in the regex you could change the two lines as such in answer from #starfry.
Change these two lines:
local re1='^(\w+=\w+)&?'
local re2='^(\w+)=(\w+)$'
To these two lines:
local re1='^(\w+=(\w+|-|)+)&?'
local re2='^(\w+)=((\w+|-|)+)$'
For all those who couldn't get it working with the posted answers (like me),
this guy figured it out.
Can't upvote his post unfortunately...
Let me repost the code here real quick:
#!/bin/sh
if [ "$REQUEST_METHOD" = "POST" ]; then
if [ "$CONTENT_LENGTH" -gt 0 ]; then
read -n $CONTENT_LENGTH POST_DATA <&0
fi
fi
#echo "$POST_DATA" > data.bin
IFS='=&'
set -- $POST_DATA
#2- Value1
#4- Value2
#6- Value3
#8- Value4
echo $2 $4 $6 $8
echo "Content-type: text/html"
echo ""
echo "<html><head><title>Saved</title></head><body>"
echo "Data received: $POST_DATA"
echo "</body></html>"
Hope this is of help for anybody.
Cheers
Actually I liked bolt's answer, so I made a version which works with Busybox as well (ash in Busybox does not support here string).
This code will accept key1 and key2 parameters, all others will be ignored.
while IFS= read -r -d '&' KEYVAL && [[ -n "$KEYVAL" ]]; do
case ${KEYVAL%=*} in
key1) KEY1=${KEYVAL#*=} ;;
key2) KEY2=${KEYVAL#*=} ;;
esac
done <<END
$(echo "${QUERY_STRING}&")
END
One can use the bash-cgi.sh, which processes :
the query string into the $QUERY_STRING_GET key and value array;
the post request data (x-www-form-urlencoded) into the $QUERY_STRING_POST key and value array;
the cookies data into the $HTTP_COOKIES key and value array.
Demands bash version 4.0 or higher (to define the key and value arrays above).
All processing is made by bash only (i.e. in an one process) without any external dependencies and additional processes invoking.
It has:
the check for max length of data, which can be transferred to it's input,
as well as processed as query string and cookies;
the redirect() procedure to produce redirect to itself with the extension changed to .html (it is useful for an one page's sites);
the http_header_tail() procedure to output the last two strings of the HTTP(S) respond's header;
the $REMOTE_ADDR value sanitizer from possible injections;
the parser and evaluator of the escaped UTF-8 symbols embedded into the values passed to the $QUERY_STRING_GET, $QUERY_STRING_POST and $HTTP_COOKIES;
the sanitizer of the $QUERY_STRING_GET, $QUERY_STRING_POST and $HTTP_COOKIES values against possible SQL injections (the escaping like the mysql_real_escape_string php function does, plus the escaping of # and $).
It is available here:
https://github.com/VladimirBelousov/fancy_scripts
This works in dash using for in loop
IFS='&'
for f in $query_string; do
value=${f##*=}
key=${f%%=*}
# if you need environment variable -> eval "qs_$key=$value"
done