Insert json in postgres via bash script using kubectl - bash

I'm trying to update a field in one of my table using bash script. I'm trying to insert json values inside of this field in postgress. The field type is 'text'. Here is the query which I try to execute:
TOKEN="${ENV_TOKEN}"
kubectl exec -it postgres-d5b5794-7fxxm -n eclipse-che -- bash -c "psql keycloak -c UPDATE federated_identity SET token ={"access_token":"'$TOKEN'","expires_in":86400,"scope":"user:full","token_type":"Bearer"} WHERE federated_username = 'admin'"
After executing this code I got the following error:
psql: warning: extra command-line argument "SET" ignored
psql: warning: extra command-line argument "token" ignored
psql: warning: extra command-line argument "=access_token:sha256~g1KaHT3-RSa2G1oPfDboc7jI90OLy67VQ4mJsVjFYZw" ignored
psql: warning: extra command-line argument "=expires_in:86400" ignored
psql: warning: extra command-line argument "=scope:user:full" ignored
psql: warning: extra command-line argument "=token_type:Bearer" ignored
psql: warning: extra command-line argument "WHERE" ignored
psql: warning: extra command-line argument "federated_username" ignored
psql: warning: extra command-line argument "=" ignored
psql: warning: extra command-line argument "admin" ignored
psql: FATAL: role "federated_identity" does not exist
command terminated with exit code 2
I've tried everyting to avoid the quotes but nothing works...
Can someone please help me to make work this query?

I have not tried to add JSON object but here is working syntax for CREATE & INSERT with kubectl. Best way would be to put your SQL statement within file to avoid the pain of ensuring double/sungle quotes are escaped properly or not. Escaping quotes becomes very complex with few values only. My recommendation is to use *.sql file with all your SQL statements.
SQL file execution with kubectl
kubectl exec -i <pod-name> -- bash -c "psql -U postgres -f /tmp/create.sql"
Create table
kubectl exec -i <pod-name> -- bash -c "psql -U postgres -c \"CREATE TABLE account1 (user_id INT PRIMARY KEY, username VARCHAR(50) NOT NULL, Password VARCHAR(50) NOT NULL, email VARCHAR(255) NOT NULL);\""
Insert record into table
kubectl exec -i <pod-name> -- bash -c "psql -U postgres -c \"INSERT INTO account1 VALUES (1, 'dummyuser', 'dummypassword', 'dummy#gmail.com');\""

The fundamental problem is that the string after psql -c should be quoted as a single argument, but you are already using both single and double quotes in it, so attempting to just add more quotes is not sufficient.
Because you can't easily escape single quotes within single quotes, the simplest solution is probably to add double quotes around the string, and backslash-escape any existing double quotes within.
kubectl exec -it postgres-d5b5794-7fxxm -n eclipse-che -- bash -c "psql keycloak -c \"UPDATE federated_identity SET token ={\\\"access_token\\\":\\\"'$TOKEN'\\\",\\\"expires_in\\\":86400,\\\"scope\\\":\\\"user:full\\\",\\\"token_type\\\":\\\"Bearer\\\"} WHERE federated_username = 'admin'\""
Because I had to guess which quotes belong to which level of nested string, this probably needs some adjustments. Try to add printf '%s\\n' in front of psql keycloak to have the container simply print out the result. I would imagine that you might need more quoting around the JSON string, but I'm not familiar enough with psql to guess what that should look like. Also, the quoting around $TOKEN looks wrong (and it seems rather pointless to copy $ENV_TOKEN to a new identical variable which you do not change at all).
If kubectl exec -it postgres-d5b5794-7fxxm -n eclipse-che -- bash lets you read from standard input (not in a place where I can check that) you might be able to use a here document instead:
kubectl exec -it postgres-d5b5794-7fxxm -n eclipse-che -- bash <<____
psql keycloak -c "UPDATE federated_identity SET token ={\"access_token\":\"'$TOKEN'\",\"expires_in\":86400,\"scope\":\"user:full\",\"token_type\":\"Bearer\"} WHERE federated_username = 'admin';"
____
Again, I had to guess some things. If you can't get this to work, perhaps update your question to explain in more detail what the code is expected to do.

Related

Invoking a bash command with an argument '#' in it

Consider the following code:
sqls='/mnt/c/alias/Binn/SQLCMD.EXE -b -S HostName -U Username -P MyPW# -d dbName'
which is the prefix of the command to connect to the SQL Server DB dbName with username Username on the HostName
`$sqls -Q "select 1;"`
executes select 1 statement on it and it works as expected.
$($sqls -Q "select 1;")
fails. Why? Is it because of the '#' in the password? How should I change the sqls line above so it works with $(...) expression?
Storing a command in a string is prone to failure. We could work out why it's not working, but the better thing to do would be to use a function:
sql() {
/mnt/c/alias/Binn/SQLCMD.EXE -b -S HostName -U Username -P 'MyPW#' -d dbName -Q "$1"
}
This allows the password to be quoted, removing the possibility that it can break the command.
Use it like sql 'select 1'. Save the result using a standard command substitution:
foo=$(sql 'select 1')
Also bear in mind that you can use a config file (possibly $HOME/.my.cnf, but it looks like you're on Windows so not sure) to store default user, pass and database, enabling you to run queries without passing all those options every time.

Proper format for setting the variable to the output of a command

If I run this command from the Azure CLI, I get a true false depending on if the resource group actually exists. Works great.
az group exists –n MyResourceGroup
It also works in this form:
az group exists –n MyResourceGroup
If, in a bash script I try to set a variable to the output of those commands:
GROUPEXISTS1="$(az group exists --name MyResourceGroup)"
GROUPEXISTS2="$(az group exists –n MyResourceGroup)"
Only the first one works. When I use -n I get the following error:
az: error: unrecognized arguments: –n MyResourceGroup
However I know that command works with the -n parameter. Do I need to encode a single dash when using it inside "$()"? Why else would it not accept that form?
Looks like your editor replaces hyphen-minus - (U+002D) with dash –. These are different chars. Bash and other unix-like shells treat dash as common char, not a start of option.
If you have a lot of already written script files and don't know if there some dashes instead of minus, you can convert them automatically with command like
sed -i 's/–/-/g' your-script-file
You could check your script carefully. –n is different from –n. In Azure Cloud Shell, you can clearly see the difference.
Modify your script as below:
GROUPEXISTS2="$(az group exists -n MyResourceGroup)"

How to get text that is printed into console using sybase procedure into shell script

I have a sybase procedure which prints data like: print 'Hello World'.
How can I get that text into shell script. In other words how can a shell script read from the console of the database?
You can do this a couple of different ways. If you have a script file that you are running, do the following to specify an output file:
isql -U username -P password -S servername -i Inputscript.name -o outputfile.name
It will run the script, and output the results into the file specified by the -o
To run isql 'interactively' in your script you can do the following within your shell script:
isql -U username -P password -S servername -b << ENDSQL >> outputfile.name
set nocount on \*stops displaying rows affected count*\
go
select some, data from table
go
ENDSQL
You can also use the example above to set the results of a query to a variable:
myvar = `isql -U username -P password -S servername << ENDSQL
sp_my_procedure
go
ENDSQL`
A couple of notes. If you are executing SQL statements, and are going to use the results in other processing (e.g. not just trying to print them), you will likely want to set nocount on within your SQL to prevent affected rowcount from printing, and use the -b flag in your isql connection string to prevent headers from being printed.

mysqldump command works when typed directly into command line - but not in a shell script

I am just getting started with shell scripts to save me typing in the same commands over and over. This command is used to copy a database over to a slave server as part of setting up MySQL database replication.
It works when typed into the command prompt directly:
mysqldump --host=192.168.1.1 –uUSER –pPASSWORD --opt database_name | mysql --host=192.168.1.2 –uUSER –pPASSWORD -C database_name
USER, PASSWORD and database_name all are replaced with their actual values in the real script.
When I type this command into a scripts.sh file, give it the execute permission, and then run it with ./scripts.sh I get:
'RROR1102 (42000): Incorrect database name 'database_name
mysqldump: Got errno 32 on write
What could be causing this error? Do I need to modify the command somehow when it is contained in a shell script?
The variable your database name is in has a CR at the end. You may need to run your script through dos2unix, or use one of the solutions on this site for stripping CRs from data if you're getting the database name from an external source.

Shell script to execute pgsql commands in files

I am trying to automate a set of procedures that create TEMPLATE databases.
I have a set of files (file1, file2, ... fileN), each of which contains a set of pgsql commands required for creating a TEMPLATE database.
The contents of the file (createdbtemplate1.sql) looks roughly like this:
CREATE DATABASE mytemplate1 WITH ENCODING 'UTF8';
\c mytemplate1
CREATE TABLE first_table (
--- fields here ..
);
-- Add C language extension + functions
\i db_funcs.sql
I want to be able to write a shell script that will execute the commands in the file, so that I can write a script like this:
# run commands to create TEMPLATE db mytemplate1
# ./groksqlcommands.sh createdbtemplate1.sql
for dbname in foo foofoo foobar barbar
do
# Need to simply create a database based on an existing template in this script
psql CREATE DATABASE $dbname TEMPLATE mytemplate1
done
Any suggestions on how to do this? (As you may have guessed, I'm a shell scripting newbie.)
Edit
To clarify the question further, I want to know:
How to write groksqlcommands.sh (a bash script that will run a set of pgsql cmds from file)
How to create a database based on an existing template at the command line
First off, do not mix psql meta-commands and SQL commands. These are separate sets of commands. There are tricks to combine those (using the psql meta-commands \o and \\ and piping strings to psql in the shell), but that gets confusing quickly.
Make your files contain only SQL commands.
Do not include the CREATE DATABASE statement in the SQL files. Create the db separately, you have multiple files you want to execute in the same template db.
Assuming you are operating as OS user postgres and use the DB role postgres as (default) Postgres superuser, all databases are in the same DB cluster on the default port 5432 and the role postgres has password-less access due to an IDENT setting in pg_hba.conf - a default setup.
psql postgres -c "CREATE DATABASE mytemplate1 WITH ENCODING 'UTF8'
TEMPLATE template0"
I based the new template database on the default system template database template0. Basics in the manual here.
Your questions
How to (...) run a set of pgsql cmds from file
Try:
psql mytemplate1 -f file
Example script file for batch of files in a directory:
#! /bin/sh
for file in /path/to/files/*; do
psql mytemplate1 -f "$file"
done
The command option -f makes psql execute SQL commands in a file.
How to create a database based on an existing template at the command line
psql -c 'CREATE DATABASE my_db TEMPLATE mytemplate1'
The command option -c makes psql execute a single SQL command string. Can be multiple commands, terminated by ; - will be executed in one transaction and only the result of the last command returned.
Read about psql command options in the manual.
If you don't provide a database to connect to, psql will connect to the default maintenance database named "postgres". In the second answer it is irrelevant which database we connect to.
you can echo your commands to the psql input:
for dbname in foo foofoo foobar barbar
do
echo """
CREATE DATABASE $dbname TEMPLATE mytemplate1
""" | psql
done
If you're willing to go the extra mile, you'll probably have more success with sqlalchemy. It'll allow you to build scripts with python instead of bash, which is easier and has better control.
As requested in the comments: https://github.com/srathbun/sqlCmd
Store your sql scripts under a root dir
Use dev,tst,prd parametrized dbs
Use find to run all your pgsql scripts as shown here
Exit on errors
Or just git clone the whole tool from here
For that use case where you have to do it....
Here is a script I've used for importing JSON into PostgreSQL (WSL Ubuntu), which basically requires that you mix psql meta commands and SQL in the same command line. Note use of the somewhat obscure script command, which allocates a pseudo-tty:
$ more update.sh
#!/bin/bash
wget <filename>.json
echo '\set content `cat $(ls -t <redacted>.json.* | head -1)` \\ delete from <rable>; insert into <table> values(:'"'content'); refresh materialized view <view>; " | PGPASSWORD=<passwd> psql -h <host> -U <user> -d <database>
$

Resources