I am trying to execute a query on postgres pod in k8s via bash script but cannot get results when i select a large number of columns. Here is my query:
kubectl exec -it postgres-pod-dcd-wvd -- bash -c "psql -U postgres -c \"Select json_build_object('f_name',json_agg(f_name),'l_name',json_agg(l_name),'email',json_agg(email),'date_joined',json_agg(date_joined),'dep_name',json_agg(dep_name),'address',json_agg(address),'zip_code',json_agg(zip_code),'city',json_agg(city), 'country',json_agg(country)) from accounts WHERE last_name='ABC';\""
When i reduce the number of columns to be selected in the query, i get the results but if I use all the column names, the query just hangs indefinitely. What could be wrong here?
Update:
I tried using the query as :
kubectl exec -it postgres-pod-dcd-wvd -- bash -c "psql -U postgres -c \"Select last_name,first_name,...(other column names).. row_to_json(accounts) from register_account WHERE last_name='ABC';\""
But this also hangs.
When i try from inside the pod, It works but i need to execute it via bash script
Means it is almost certainly the results pagination; when you run exec -t it sets up a TTY in the Pod, just like you were connected interactively, so it is likely waiting for you to press space or "n" for the next page
You can disable the pagination with env PAGER=cat psql -c "select ..." or use the --pset pager=off as in psql --pset pager=off -c "Select ..."
Also, there's no need to run bash -c unless your .bashrc is setting some variables or otherwise performing work in the Pod. Using exec -- psql should work just fine, all other things being equal. You will need to use the env command if you want to go with the PAGER=cat approach, because $ ENV=var some_command is shell syntax, and thus cannot be fed directly into exec
As the resulting columns are having a lot of json processing, I think the time taken to execute these two queries are different.
Maybe you can login into the pod and execute the query and see.
kubectl exec -it postgres-pod-dcd-wvd -- bash
Now you are inside the pod. Then we can execute the query.
# psql -U postgres -c \"Select json_build_object('f_name',json_agg(f_name),'l_name',json_agg(l_name),'email',json_agg(email),'date_joined',json_agg(date_joined),'dep_name',json_agg(dep_name),'address',json_agg(address),'zip_code',json_agg(zip_code),'city',json_agg(city), 'country',json_agg(country)) from accounts WHERE last_name='ABC';\"
# psql -U postgres -c \"Select last_name,first_name,...(other column names).. row_to_json(accounts) from register_account WHERE last_name='ABC';\"
Now you we will be able to see whether one query is taking longer time to execute.
Also, kubectl exec pod command can be executed with a request timeout value (--request-timeout=5m) to see if there is a slowness.
Related
I have a repetitive task that I do while testing which entails connecting to a cassandra pod and running a couple of CQL queries.
Here's the "manual" approach:
On cluster controller node, I exec a shell on the pod using kubectl:
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /bin/bash
Once in the pod I execute cqlsh:
cqlsh $(hostname -i) -u myuser
and then enter password interactively
I execute my cql queries interactively
Now, I'd like to have a bash script to automate this. My intent is to run cqlsh directly, via kubectl exec.
The problem I have is that apparently I cannot use a shell variable within the "command" section of kubectl exec. And I will need shell variables to store a) the pod's IP, b) an id which is the input to my first query, and c) intermediate query results (the two latter ones are not added to script yet).
Here's what I have so far, using a dummy CQL query for now:
#!/bin/bash
CASS_IP=$(kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /usr/bin/hostname -i)
echo $CASS_IP # This prints out the IP address just fine, say 192.168.79.208
# The below does not work, errors provided below
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /opt/cassandra/bin/cqlsh $CASS_IP -u myuser -p 'mypass' -e 'SELECT now() FROM system.local;'
# The below works just fine and returns the CQL query output
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- /opt/cassandra/bin/cqlsh 192.168.79.208 -u myuser -p 'mypass' -e 'SELECT now() FROM system.local;'
The output from the above is as follows, where IP is echoed, first exec'd cqlsh breaks, and second succeeds:
192.168.79.208
Warning: Timezone defined and 'pytz' module for timezone conversion not installed. Timestamps will be displayed in UTC timezone.
Traceback (most recent call last):
File "/opt/cassandra/bin/cqlsh.py", line 2357, in <module>
main(*read_options(sys.argv[1:], os.environ))
File "/opt/cassandra/bin/cqlsh.py", line 2326, in main
encoding=options.encoding)
File "/opt/cassandra/bin/cqlsh.py", line 463, in __init__
load_balancing_policy=WhiteListRoundRobinPolicy([self.hostname]),
File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 425, in __init__
File "/opt/cassandra/bin/../lib/cassandra-driver-internal-only-3.25.0.zip/cassandra-driver-3.25.0/cassandra/policies.py", line 426, in <listcomp>
File "/usr/lib64/python3.6/socket.py", line 745, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
command terminated with exit code 1
Warning: Timezone defined and 'pytz' module for timezone conversion not installed. Timestamps will be displayed in UTC timezone.
system.now()
--------------------------------------
e78e75c0-0d3e-11ed-8825-1de1a1b1c128
(1 rows)
Any ideas how to get around this? I've been researching this for quite a while now, but I'm stuck...
This is a very, very FAQ: the kubectl exec is, as its name says, using exec(3) versus system(3) -- which in your case wouldn't work anyway because the $ in your kubectl exec would be interpreted by your shell not the pod's shell
but thankfully the solution is the same to both problems: create your own system(3) by wrapping the command in a sh -c invocation (or bash -c if you have bash-isms and bash is available inside the pod):
kubectl exec pod/my-app-cassandra-pod-name -it --namespace myns -- sh -c '/opt/cassandra/bin/cqlsh $(hostname -i) -u myuser -p "mypass" -e "SELECT now() FROM system.local;"'
as always, be cognizant of the "outer" versus "inner" quoting, especially if your "mypass" or the -e statement contains shell meta-characters
I'm trying to create a simple script which would:
Connect to docker container's BASH shell
Go into redis-cli
Perform a flushall command in redis-cli
So far, I have this in my docker_script.sh (this basically copies the manual procedure):
docker exec -it redis /bin/bash
redis-cli
flushall
However, when I run it, it only connects to the container's BASH shell and doesn't do anything else. Then, if I type exit into the container's BASH shell, it outputs this:
root#5ce358657ee4:/data# exit
exit
./docker_script.sh: line 2: redis-cli: command not found
./docker_script.sh: line 3: keys: command not found
Why is the command not found if commands redis-cli and flushall exist and are working in the container when I perform the same procedure manually? How do I "automate" it by creating such a small BASH script?
Thank you
Seems like you're trying to run /bin/bash inside the redis container, while the redis-cli and flushall commands are scheduled after in your current shell instance. Try passing in your redis-cli command to bash like this:
docker exec -it redis /bin/bash -c "redis-cli FLUSHALL"
The -c is used to tell bash to read a command from a string.
Excerpt from the man page:
-c string If the -c option is present, then commands are read from
string. If there are arguments after the string, they
are assigned to the positional parameters, starting with
$0.
To answer your further question in the comments, you want to run a single script, redis_flushall.sh to run that command. The contents of that file are:
docker exec -it redis /bin/bash -c redis-cli auth MyRedisPass; flushall
Breaking that down, you are calling redis-cli auth MyRedisPass as a bash command, and flushall as another bash command. The issue is, flushall is not a valid command, you'd want to call redis-cli flushall instead. Command chaining is something that has to be implemented in a CLI application deliberately, not something that falls out of the cracks.
If you replace the contents of your script with the following, it should work, i.e., after ; add a redis-cli call before specifying the flushall command.
docker exec -it redis /bin/bash -c redis-cli auth MYSTRONGPASSWORD; redis-cli FLUSHALL
The above proposed solution with auth still got me an error
(error) NOAUTH Authentication required
This worked for me:
docker exec -it redis /bin/sh -c 'redis-cli -a MYPASSWORD FLUSHALL'
I am using a docker-compose type of scenario. I would like to back up the database for partkeepr on a nightly basis. I can get a bash shell by doing a docker exec -it docker-partkeer_database_1 bash into the container and run the mysqldump command just fine but I can't run it successfully as a docker exec type of function.
I run docker exec -it docker-partkeepr_database_1 bash -c mysqldump --databases partkeepr -upartkeepr -pUSERPASSWORD > /var/lib/mysql/backup/partkeeprsql.$(date +%Y-%m-%d-%H.%M.%S).sql.
That date part on the end works just fine when I already have gotten in with a bash shell in the container but not outside of it. I can run the command as written without the date part just fine but I need the date so I can have multiple backups.
The system errors with -bash: /var/lib/mysql/backup/partkeeprsql.$(date +%Y-%m-%d-%H.%M.%S).sql: No such file or directory.
Have any ideas?
I found out the answer to my question. I didn't wrap the command after the bash -c in ""'s. Once I ran the command below it worked like a charm. I just wanted everyone to know the answer that I figured out.
docker exec -it docker-partkeepr_database_1 bash -c "mysqldump --databases partkeepr -upartkeepr -
pUSERPASSWORD > /var/lib/mysql/backup/partkeeprsql.$(date +%Y-%m-%d-%H.%M.%S).sql"
I'm working with a simple postgres database and docker. Currently, I have a docker-compose file which creates the container I need and loads in the SQL files. After loading in this data, I would like it to perform a simple query through a bash script that I"m going to use for some basic tests (i.e., confirm # of rows > 0, etc). To start, I'm just trying to make a simple script that will run and print the number of rows (then I can worry about implementing actual testing). Here is what I have so far:
docker-compose.yml:
services:
postgres:
image: postgres
environment:
POSTGRES_DB: test-db
POSTGRES_USER: postgres
POSTGRES_PASSWORD: postgres
volumes:
- ./database/create_table.sql:/docker-entrypoint-initdb.d/create_table.sql
- ./databases/data.sql:/docker-entrypoint-initdb.d/data.sql
- ./script/test.sh:/docker-entrypoint-initdb.d/test.sh
test.sh:
#!/bin/bash
echo "Accessing bash terminal of DB container"
docker exec -it postgres_1 bash
echo "Accessing psql terminal"
psql -U postgres
echo "Connecting to database"
\c database
echo "Checking number of rows"
numrows = $("SELECT count(*) FROM my_table")
echo numrows + " found."
Currently when I run docker-compose up, it creates the data from my SQL files and then stays idle. Is there something additional I need to run my script? I am able to do all of this myself through a separate terminal, but I would like this to all be automated so that I can just add tests to my test.sh and then run that rather than having to do it manually each time. What am I missing here? Shouldn't my script work since I really just recreated the commands I was executing manually? Thanks for any help!
By the time your bash script is executed you are already in the postgres container itself. So, you can simply query the database from there - as #DavidMaze already pointed out. Your script might look like this:
#!/bin/bash
db_name='test-db'
db_user='postgres'
function execute_sql() {
psql --tuples-only -U $db_user -d $db_name -c "$#"
}
function log() {
printf "Log [%s]: %s\n" "$(date --iso-8601=seconds)" "$*"
}
numrows=$(execute_sql "SELECT count(*) FROM my_table")
log "${numrows} rows found"
Output should be something like that:
postgres_1 | /usr/local/bin/docker-entrypoint.sh: sourcing /docker-entrypoint-initdb.d/test.sh
postgres_1 | [2020-03-31T23:02:14+00:00]: 4 rows found
Regarding the testing: If you only want to run SQL queries and don't do/need additional scripting you can simply put your SQL test queries into a .sql file (e.g. test.sql) as well.
One more important thing to mention - which I'm sure you already know: The files (*.sql, *.sh etc.) that you mount to the postgres container are executed in alphabetical order, i.e.
create_table.sql
data.sql
test.sh
So, you are good.
Basically, I want to switch to user postgres and get a listing of databases. This is with a Fabric script that reads command lines from a text file one by one, executes them and then saves their output to file.
su - postgres && psql -c '\l'
When I do this under bash directly:
(ssha)root ~$su - postgres && psql -c '\l'
postgres#localvm:~$
I saw a related question, linux - Executing multiple commands under as another username within a file in BASH shell, but that wouldn't work with my 1-line-per-command scheme and I don't need a full script, just 1 command.
You can use su -c:
su - postgres -c "psql -c '\l'"
Though often you'll also have sudo, which is more robust and easier to use:
sudo -u postgres psql -c '\l'