in hql scripts we use "!sh echo ---new line---" for the same . Want to know the alternative for this in impala to print any line in impala scripts? - hql

Question

in hql scripts we use "!sh echo ---new line---" for the same .
Want to know the alternative for this in impala to print any line in impala scripts?

Answer 1

You can invoke the shell command line from impala scripts.
As an example of how it works.
script_impala.sql
-- set a variable containing the of the game
SET hivevar:game=Monopoly;
-- return the list of the game
SELECT list_price FROM fun.games WHERE name = '${hivevar:game}';
-- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = '${hivevar:game}';
shell hdfs dfs -ls /user;
shell ls -ltr;
shell echo I can echo from impala scripts;
shell cat hex_color.sql
$ impala-shell -f script_impala.sql
output
Starting Impala Shell without Kerberos authentication
Connected to localhost.localdomain:21000
Server version: impalad version 2.10.0-cdh5.13.3 RELEASE (build 15a453e15865344e75ce0fc6c4c760696d50f626)
Variable GAME set to Monopoly
Query: -- return the list of the game
SELECT list_price FROM fun.games WHERE name = 'Monopoly'
Query submitted at: 2020-06-04 23:29:19 (Coordinator: http://localhost.localdomain:25000)
Query progress can be monitored at: http://localhost.localdomain:25000/query_plan?query_id=a94afc7556843cf7:8b19809d00000000
+------------+
| list_price |
+------------+
| 19.99 |
+------------+
Fetched 1 row(s) in 0.15s
Query: -- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = 'Monopoly'
Query submitted at: 2020-06-04 23:29:19 (Coordinator: http://localhost.localdomain:25000)
Query progress can be monitored at: http://localhost.localdomain:25000/query_plan?query_id=c64d1c8950e5e1f8:15d410cb00000000
+-----------+-------+
| shop | price |
+-----------+-------+
| Dicey | 17.99 |
| Board 'Em | 25.00 |
+-----------+-------+
Fetched 2 row(s) in 0.14s
Found 5 items
drwxrwxrwt - mapred hadoop 0 2019-04-29 18:19 /user/history
drwxrwxrwx - hive hive 0 2019-04-29 18:19 /user/hive
drwxr-xr-x - hue hue 0 2019-11-25 10:19 /user/hue
drwxr-xr-x - spark spark 0 2019-04-29 18:19 /user/spark
drwxrwxrwx - training supergroup 0 2020-05-28 11:33 /user/training
--------
Executed in 2.10s
total 56
-rw-rw-r-- 1 training training 61 Sep 25 2019 hex_color.sql
-rw-rw-r-- 1 training training 115 Sep 25 2019 color_from_rgb.sql~
-rw-rw-r-- 1 training training 58 Sep 25 2019 hex_color_impala.sql
-rwxr-xr-x 1 training training 449 Sep 25 2019 email_results.sh
-rw-rw-r-- 1 training training 1166 Sep 25 2019 zero_air_time.csv
-rw-r--r-- 1 training training 261 Sep 26 2019 change_background.sh~
-rwxr-xr-x 1 training training 262 Sep 26 2019 change_background.sh
-rw------- 1 training training 2966 Sep 26 2019 ChangeVMDesktopColor.txt
-rw------- 1 training training 3279 Sep 26 2019 Hive&ImpalaInScripts&Applications.txt~
-rw------- 1 training training 3278 Sep 26 2019 Hive&ImpalaInScripts&Applications.txt
-rw-rw-r-- 1 training training 449 Sep 26 2019 email_resuts.sh
-rw-rw-r-- 1 training training 120 Apr 27 11:38 color_from_rgb.sql
-rw-rw-r-- 1 training training 397 Jun 4 23:24 game_prices.sql~
-rw-rw-r-- 1 training training 395 Jun 4 23:29 game_prices.sql
--------
Executed in 0.00s
I can echo from impala scripts
--------
Executed in 0.00s
SELECT hex FROM wax.crayons WHERE color = '${hivevar:color}'
--------
Executed in 0.01s
As you can see, you only have to write in your script
shell <command>;
You can even invoke hdfs dfs commands
shell hdfs dfs -<command>;
The above example prints the output to the console, if you want to print the output to a file you have to do as following to redirect all the output to a file:
$ impala-shell -f script_impala.sql >> /home/..../my_file.csv
Other option would be to redirect only the echo commands from inside the script. This would be as following:
script_impala.sql
-- set a variable containing the of the game
SET hivevar:game=Monopoly;
-- return the list of the game
SELECT list_price FROM fun.games WHERE name = '${hivevar:game}';
-- return the prices of the game ate game shops
SELECT shop, price FROM fun.inventory WHERE game = '${hivevar:game}';
shell hdfs dfs -ls /user;
shell ls -ltr;
shell echo I can echo from impala scripts >> /home/..../my_file.csv;
shell echo How are you? >> /home/..../my_file.csv;
shell echo I am feeling very good, how about you? >> /home/..../my_file.csv;
shell cat hex_color.sql
and don't forget the semicolon at the end of the line ";".
I hope it helps.