Exporting function with xargs parallel & psql in bash - bash

Question

I'm trying to run SQL against one-or-many psql-compatible hosts in parallel, with SQL run in sequence on each host, using xargs.
The bash script, which I'm sourcing from another script:
# Define the count of hosts (also the number of parallel processes)
export pe_fpe_hosts_line_count=$(cat $pe_fpe_hosts_file_loc | wc -l)
# Define the function that runs SQL from a file
function func_pe_exec_sql {
while read pe_sql_file; do
psql -q -t -c "\"$pe_sql_file"\"
done <$pe_fpe_sql_file_loc
}
export -f func_pe_exec_sql
# Define the xargs parallel function
function func_pe_parallel {
while read pe_hosts_file; do
echo $pe_hosts_file | xargs -d '\n' -P $pe_fpe_hosts_line_count func_pe_exec_sql
done <$pe_fpe_hosts_file_loc
}
The error I get: xargs: func_pe_exec_sql: No such file or directory. This is weird - I've exported the function!
Example SQL file:
INSERT INTO public.psql_test SELECT 1 as myint, now() as mytime;
INSERT INTO public.psql_test SELECT 2 as myint, now() as mytime;
INSERT INTO public.psql_test SELECT 3 as myint, now() as mytime;
INSERT INTO public.psql_test SELECT 4 as myint, now() as mytime;
INSERT INTO public.psql_test SELECT 5 as myint, now() as mytime;
Example SQL Host file:
--host=myhost1 --port=5432 --dbname=postgres --username=cooluser
--host=myhost2 --port=5432 --dbname=postgres --username=cooluser
pe_fpe_sql_file_loc is the path to the SQL file, and pe_fpe_hosts_file_loc is the path to the SQL Host file.
The SQL must always be run in separate transactions, and each row in the SQL file needs to be inserted separately, one after another. 5 should be in the same row as the greatest of the mytime values.
I am using it as an ETL framework with functions defined in the database though, and not for simple inserts :)

Answer 1

I think your invocation of xargs is incorrect. You are not actually passing the line from pe_hosts_file to the function func_pe_exec_sql.
You need to pass the input from the pipe to the function, to do that; you need to have a place-holder which -I flag in xargs provides.
-I replace-str
Replace occurrences of replace-str in the initial-arguments with names
read from standard input. Also, unquoted blanks do not terminate input items;
instead the separator is the newline character. Implies -x and -L 1.
Using that something like below needs to be used.
| xargs -d '\n' -I {} -P "$pe_fpe_hosts_line_count" bash -c 'func_pe_exec_sql "{}"'
where the {} is the place-holder for the value piped and we are passing it to the sub-shell spawned by bash -c directly to the function func_pe_exec_sql. The special double quotes around {} is to ensure, the shell to expand the value before the function is invoked.