bash - loop and nested variable to store api response - bash

I am learning to work with Bash and APIs, and am currently stuck on a problem related to nested variables.
The basic action I wish to do looks like this:
name="peter"; country="uk";
gender=$(http -b GET "https://api.genderize.io/?$name=peter&country_id=$uk" | jq -e '.gender');
echo $gender
output is
"male"
The complete action I want to take looks like this:
while read name <&3 && read country <&4
do gender=$(http -b GET "https://api.genderize.io/?name=$name&country_id=$country" | jq -e '.gender')
echo "$name : $gender" >> namesorted.txt
sleep 0.1
done 3<namelist.txt 4<countrylist.txt
but the output is null, which means that the request wasn't sent correctly:
Peter : null
Edouard : null
Henri : null
Anabelle : null
Nonso : null
Tom : null
What's wrong with my code? And is there a way for me to see the query sent by the code for debugging purposes?
edit: it ended up being a formatting mistake, as pointed out in the comments. Thanks!

As pointed in the comments, it ended up being a formatting mistake:
the use of double quotes in the initial script is correct, because it allows variables to be exanded:
gender=$(http -b GET "https://api.genderize.io/?$name=peter&country_id=$uk" | jq -e '.gender')
But in my full script I got mixed up with the single quotes, which kept the variables from being expanded. I fixed it in the original question, and the code works.
Thanks to #Gordon Davidsson for identifying the issue.

Related

Create new bash var from value of dict bash var

My environment created a variable that looks like this:
SM_TRAINING_ENV={"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
EDIT by Ed Morton: per the OPs comment below, this is what (s)he is trying to describe above as the sample input:
$ SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
$ echo "$SM_TRAINING_ENV"
{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}
How can I create a new bash variable that is equal to the value of SM_TRAINING_ENV["hyperparameters"]["model_dir"]?
For completeness, I was trying simple things like echo ${SM_TRAINING_ENV} | jq . and kept getting errors with everything I tried.
Edit: I've been informed that this value isn't a proper json, so rewording the question. I think the environment sets it to the value of a python dictionary, so jq seems not usable. Removed json tag. Maybe this is a job for awk?
It looks like I can match the value I want if I assume the structure doesn't change with the regex pattern s3.*?model, but not sure how to set a regex pattern to a new variable.
First, you need to quote the JSON value so that the double quotes will be included in the value.
SM_TRAINING_ENV='{"additional_framework_parameters":{},"channel_input_dirs":{"training":"/opt/ml/input/data/training"},"current_host":"algo-1","framework_module":"sagemaker_tensorflow_container.training:main","hosts":["algo-1"],"hyperparameters":{"bool_param":true,"float_param":1.25,"int_param":5,"model_dir":"s3://bucket/detection/prefix/testing-2019-04-06-02-24-20-194/model","str_param":"bla"},"input_config_dir":"/opt/ml/input/config","input_data_config":{"training":{"RecordWrapperType":"None","S3DistributionType":"FullyReplicated","TrainingInputMode":"File"}},"input_dir":"/opt/ml/input","is_master":true,"job_name":"testing-2019-04-06-02-24-20-194","log_level":20,"master_hostname":"algo-1","model_dir":"/opt/ml/model","module_dir":"s3://bucket/prefix/testing-2019-04-06-02-24-20-194/source/sourcedir.tar.gz","module_name":"launcher.sh","network_interface_name":"ethwe","num_cpus":8,"num_gpus":1,"output_data_dir":"/opt/ml/output/data","output_dir":"/opt/ml/output","output_intermediate_dir":"/opt/ml/output/intermediate","resource_config":{"current_host":"algo-1","hosts":["algo-1"],"network_interface_name":"ethwe"},"user_entry_point":"launcher.sh"}'
Then you can use the jq utility to extract the value you want.
new_var=$(echo "$SM_TRAINING_ENV" | jq '.hyperparameters.model_dir')
This doesn't really index but it works if order is always the same:
NEW_VAR=$(echo $SM_TRAINING_ENV | egrep -o s3.*?model | head -1)
Would much prefer something not dependent on order though.

How to convert a variable to string format? KSH shell scripting

I am trying to convert a variable to a string format so I can use java runsql utility to insert its value into a database later. The database needs the value to be in a char format, hence string.
This is a dumbed down version of my code so I can get to the heart of what I'm asking -
#!/bin/ksh -x
value1=2018-01-01
value2=2018-02-01
mystring=$value1,$value2
echo $mystring
stringify=${'value1'},${'value2'})
echo $stringify
What happens is I get no output for stringify or depending on how I switch up the arrangement of the symbols I get the literal string 'value1' or 'value2'.
What am I doing wrong? I feel like this is very simple, maybe I've just been staring at it too long, I dunno.
You can just escape quote like this:
mystring=\'$value1\',\'$value2\'
Output:
$ echo $mystring
'2018-01-01','2018-02-01'
A simpler option to get the same output (as suggested by #CharlesDuffy) is:
mystring="'$value1','$value2'"
You can just do like this, more simpler:
#!/bin/ksh -x
value1=2018-01-01
value2=2018-02-01
mystring=$value1,$value2
echo $mystring
stringify="'$value1','$value2'" #use double-quotes around the variables
echo $stringify
Output:
2018-01-01,2018-02-01
'2018-01-01','2018-02-01'

Expanding variables before bash call/exec

I know the basics of Bash but often miss the nuance and I'm having a problem using it to achieve what I had hoped would be a rather simple problem:
If I have the following in a bash script, which works exactly as I'd want it to:
cbType=`echo $configuration | jsawk -a 'return _.where(this,{name: "reference_data"})'`
It takes $configuration -- which is a JSON string -- and identifies the array element where name is "reference_data" and returns that object/hash definition only. Please note that this does use the very handy jsawk utility but it has been designed to be exhibit good command-line behaviour.
The problem is that when I remove the hard-coded "reference-data" with a variable it seems to not be able to reference the scope of the variable. So for instance, ...
myVar="\"reference_data\""
cbType=`echo $configuration | jsawk -a 'return _.where(this,{name: $myVar})'`
Does not work and instead returns a jsawk error of:
jsawk: js error: ReferenceError: $myVar is not defined
Is there anything I can do to enforce that first the variable is expanded, and then the command string is executed?
Declared variables won't be expanded if it's not within double quotes. So put your code inside double quotes instead of single quotes.
myVar="\"reference_data\""
cbType=$(echo "$configuration" | jsawk -a "return _.where(this,{name: $myVar})")

Unix Shell Script does not send intended email

I have a shell script like this. The purpose of this script is to tail+head out a certain amount of data from file.csv and then send it to email Bob#123.com. DataFunction seems to work fine alone however when I try to call DataFunction within the email function body. It seems it sends a empty email with the correct Title and destination. The body of the email is missing which should be the data from DataFunction. Is there a workaround for this ? Thank you in advance.
#!/bin/bash
DataFunction()
{
tail -10 /folder/"file.csv" | head -19
}
fnEmailFunction()
{
echo ${DataFunction}| mail -s Title Bob#123.com
}
fnEmailFunction
You are echoing an unset variable, $DataFunction (written ${DataFunction}), not invoking the function.
You should use:
DataFunction | mail -s Title Bob#123.com
You may have been trying to use:
echo $(DataFunction) | mail -s Title Bob#123.com
but that is misguided for several reasons. The primary problem is that it converts the 10 lines of output from the DataFunction function into a single line of input to mail. If you enclosed the $(DataFunction) in double quotes, that would preserve the 'shape' of the input, but it wastes time and energy compared to running the command (function) directly as shown.
Try this:
mail -s "Title" "bob#123com" <<EOF
${DataFunction}
EOF
I tried #0xAX's answer and couldnt get it to work properly within a bash script. You simply need to save the output of DataFunction() in some variable. You can simply use these three lines to achieve this.
#!/bin/bash
VAR1=`tail -10 "/folder/file.csv" | head -19`
echo $VAR1 | mail -s Title Bob#123.com

bash script getting the value from a variable of the form var$[id]

This is the first time Im using a shell script ( #!/bin/sh ) and Ive been working my way through it reading tutorials and the like but Im stuck on this reading and writing values of a key..
Im trying to read in key=value pairs from a config file of the form
key1_begin=abc
key1_end=def
key2_begin=123
key2_end=jkl
.. and so on
I would like the user to pass in parameters to the script like
something.sh 1 x y z
where the first parameter would serve as an that is used to modify the appropriate keys. So after I have checked that the directory exists and the file exists I source it using
source config.cfg
I then save the id using ID=$1 and access the keys using
echo key${ID}_begin
so a read to obtain the value of the key would be
echo key${ID}_begin = $[key${ID}_begin]
where I expect to get " key1_begin = abc " but instead keep getting " key1_begin = 0 ". The same command however seems to work work for numbers. For example using this command with an ID of 2 gives " key2_begin = 123 "
Could someone please point me in the right direction as to why this works fine for numbers but not alphabets?
And what do I use to change the value of the variable? I am currently using "eval" but this again seems to only work with numbers
[ ! -z $2 ] && eval key${ID}_end=$3
Would really appreciate any advice / pointers with this.\
Thank you
You most likely going to have to use an intermediary variable:
key1_begin=abc
var="key${ID}_begin"
echo "$var=${!var}"
Just as this (assume bash):
#! /bin/sh
ID=2
source mychild.sh
keyid="key${ID}_begin"
echo "key${ID}_begin = ${!keyid}"
In short, in bash, you can
var=${!i}
Or on other shells:
var=`eval echo \$$i`

Resources