I'm trying to execute a script on the master node of AWS EMR cluster. The intention is to create a new conda env and link it to jupyter. I'm following this doc from AWS. Problem is, whatever be the content of the script, I'm getting the same error: bash: /home/hadoop/scripts/bootstrap.sh: No such file or directory while executing sudo docker exec jupyterhub bash /home/hadoop/scripts/bootstrap.sh. I've made sure the sh file is in the correct location.
But if I copy the bootstrap.sh file inside the container, and then run the same docker exec cmd, it's working fine. What am I missing here? I've tried with a simple script with the following entries, but it throws the same error:
#!/bin/bash
echo "Hello"
The doc clearly says:
Kernels are installed within the Docker container. The easiest way to
accomplish this is to create a bash script with installation commands,
save it to the master node, and then use the sudo docker exec
jupyterhub script_name command to run the script within the jupyterhub
container.
The docker exec command runs a command within the container's namespaces. One of those namespaces is the filesystem. So unless the command is part of the image, written into the container directly, or you have mounted a host volume to map a host directory into the container, you won't be able to execute it. A host volume could look like:
docker run -v /host/scripts:/container/scripts -n your_container $your_image
docker exec -it your_container /container/scripts/test.sh
That host volume could be the same path on both the host and the container.
If it is a shell script, you could use I/O redirection, e.g.:
docker exec -i $container_id /bin/bash <local_script.sh
but be aware that you cannot do interactive stuff this way since the script content has replaced your terminal as stdin. This works because the shell inside the container is just processing commands from stdin.
Other than those scenarios, I don't know what to tell you other than the documentation from AWS appears to be wrong.
Related
I am trying to copy a folder outside of a container using docker cp, but I am running in an unexpected issue: the command works perfectly outside of a shell script yet fails when running the script.
For example: copy_indices.sh
for x in "${find_container_id_arr[#]}"; do
CONTAINER_NAME="${x}"
CONTAINER_ID=$(docker ps -aqf "name=${x}")
idx_name=$(docker exec -it "$CONTAINER_ID" ls -1 /usr/share/elasticsearch/data/nodes/0/indices)
docker cp "$CONTAINER_ID":/usr/share/elasticsearch/data/nodes/0/indices/"$idx_name" "$ALL_INDICES"/"$idx_name"
done
I determine the container ID using CONTAINER_ID=$(docker ps -aqf "name=${x}"), find the name of folder I need using idx_name=$(docker exec -it "$CONTAINER_ID" ls -1 /usr/share/elasticsearch/data/nodes/0/indices) and then copy it on the host filesystem: docker cp "$CONTAINER_ID":/usr/share/elasticsearch/data/nodes/0/indices/"$idx_name" "$ALL_INDICES"/"$idx_name"
My issue is that every command evaluates and run as expected when not put inside this script. I can run the command docker cp <my_container>:/usr/share/elasticsearch/data/nodes/0/indices/<index_name> ./all_indices/<index_name> and the target folder is indeed found and copied onto the host.
Once these commands are inside a script however, I get an "Error: No such container:path:" error and I can't pinpoint what is going wrong, because the mentionned path indeed exists in the container and the container is correct as I tested it running the "final" command supposed to be executed (the docker cp one).
What could be the reason these commands suddenly stop working when put in a shell script?
This question already has answers here:
Pass commands as input to another command (su, ssh, sh, etc)
(3 answers)
Closed 3 years ago.
I want to create s sh script but it stops when "docker exec -it cli bash" is executed and do not go to the next line. How to run the other commands on root?
root#ee3abae377df:/opt/gopath/src/github.com/hyperledger/fabric/peer#
Stops here and i am not able do execute the next command
docker exec -it cli bash
export CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp
export CORE_PEER_ADDRESS=peer0.org1.example.com:7051
export CORE_PEER_LOCALMSPID="Org1MSP"
export CORE_PEER_TLS_ROOTCERT_FILE=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/peers/peer0.org1.example.com/tls/ca.crt
export CHANNEL_NAME=mychannel
docker exec -it creates an interactive docker container. It starts a new shell in your current terminal. This blocks the rest of the commands from running until you kill or exit the container. The rest of the commands you have will in fact be run, once you exit the container.
I assume this is not desired. You should look into creating an entrypoint in a custom Dockerfile for your docker container where you execute the remainder of the commands in your script.
If you haven't created a Dockerfile before, the getting started guide from Docker is a pretty good intro to everything docker-related.
Executing commands using CLI on any peer
As per the description you have given, cli is pointing to
peer0.org1.example.com:7051. (Please check your docker-compose file
and there would be one service with container name as "cli" and image
as hyperledger/fabric-tools)
When you are executing "docker exec -it cli bash", you are entering
into container of peer0.org1. It provides an interactive shell to
you.
Let us consider we want to install chaincode using cli on Pee1 Org1, create one test.sh file and write the following cammand inside test.sh file
docker exec -e "CORE_PEER_LOCALMSPID=Org1MSP" -e "CORE_PEER_ADDRESS=peer1.org1.example.com:7051" -e "CORE_PEER_MSPCONFIGPATH=/opt/gopath/src/github.com/hyperledger/fabric/peer/crypto/peerOrganizations/org1.example.com/users/Admin#org1.example.com/msp" cli peer chaincode install -n cc_name -v cc_version -p cc_path
Here we are passing environment varibles CORE_PEER_ADDRESS and CORE_PEER_MSPCONFIGPATH.
Then just execute test.sh file, the chaincode will be installed on peer1 provided container is already running and environment varibles are correct. (please provide correct chaincode path, name, and version)
You should leverage the option given by #jeremysprofile. But still, if you want to achieve it through shell script then you might want to write a expect script along with shell script or might merge both the scripts in one. It should look something like this :
#!/usr/bin/expect -f
expect "*root*" {
send -- "<your command>\r"
send -- "exit \r"
}
Expect is a program that "talks" to other interactive programs according to a script. Following the script, Expect knows what can be expected from a program and what the correct response should be.
Here is the Linux man Page.
What I am trying to do is setup a local development database and to prevent everyone having to go through all the steps I thought it would be useful to create a script.
What I have below stop once it is in the terminal, which looks like:
output
./dbSetup.sh
hash of container 0d1b182aa6f1
/ #
At which point I have to manually enter exit.
script
#!/bin/bash
command=$(docker ps | grep personal)
set $command
echo "hash of container ${1}"
docker exec -it ${1} sh
Is there a way I can inject a command via a script into a dockers container terminal?
In order to execute command inside a container, you can use something like this:
docker exec -ti my_container sh -c "echo a && echo b"
More information available at: https://docs.docker.com/engine/reference/commandline/exec/
Your script finds a running Docker container and opens a shell to it. The "-it" makes it interactive and allocates a tty which is why it continues to wait for input, e.g. "exit". If the plan is to execute some commands to initialize a local development database, I'd recommend looking at building an image with a Dockerfile instead. i.e. Once you figure out the commands to run, they would become RUN commands and the container after docker run would expose a local development database.
If you really want some commands to run within the shell after it is started and maintain the session, depending on the base image, you might be able to mount a bash profile that has the required commands, e.g. -v db_profile:/etc/profile.d where db_profile is a folder with the shell scripts you want to run. To get them to run you'd exec sh -l to have the login startup scripts to run.
I am just wondering is that possible to run one script (e.g. shell script, python script, etc.) in different environments?
For example, I want to run my script from Linux shell to docker container shell (which the container is created by the script)? In other words, keep the script executing the rest of commands on container (after into the container).
run.sh (#shell script)
sudo docker exec -it some_containers bash #this command will lead me to docker container environment
apt-get install curl # I want to also execute this command inside the docker container after I enter the docker container environment
# this is just one script
Your question is not very clear, but it sounds like this is a job requiring two scripts - the first script runs in your "Linux shell", and needs to cause the second script to be placed into the container (perhaps by way of the dockerfile), at which point you can have the first script use docker exec.
Please see the answers on this question for more information.
I have a container that is running with no issues. I added a bash script to compliment a couple other scripts already in the container. The docker image copy 2 scripts to /usr/local/bin and they can be accessed with docker exec -c container-name existingscript.
I added my own script to the same directory and when running the same command I get an error that exec cannot run the script: no file or directory,script not located in $PATH. I check path and sure enough, /usr/local/bin is listed. I checked permissions and the script is 755.
I then open an interactive shell with docker exec -it mycontainer bash and run /usr/local/bin/myscript and it runs with no problem.
Why can I not run the script from outside the container like I can the other two (that were included in the image). All three have almost the same functions a day do not use any special programs, one lists files, one adds files, one reads the file.
The base is Ubuntu.
EDIT: Found where I was running into the issue. Provided the answer in case anyone else happens to make the same mistake.
EDIT-2: So the script that came with the docker image to perform a couple common functions calls the image not the container so my adding the scripts to the container had no effect on the script which was why I kept getting the no file or directory error.
The line in the script in question was:
docker run --rm -v "$(pwd)/config":/path/to/file -ti image_name:latest" mynewscript $#
Of course that ran against the image and NOT the container.
Once I noticed that I tried running it with exec instead of run and it ran without error, like so:
docker exec -it container_name mynewscript
The reason is "/usr/local/bin" not in your script's $PATH, you can use /usr/local/bin/myscript explicitly in your script. Or export $PATH first in the script.
While I was adding snippets to help explain the issue I found the problem and the solution.
So I access the scripts inside the container from the host with another script that allows you to do different things based on switch case. The scripts are called against the docker image and not the container so the script I added does not actually exist in the image.
I modified the script to call the container instead of the image and it works as expected.
EDIT: I updated the question with the answer but I am adding it here as well:
So the script that came with the docker image to perform a couple common functions calls the image not the container so my adding the scripts to the container had no effect on the script which was why I kept getting the no file or directory error.
The line in the script in question was:
docker run --rm -v "$(pwd)/config":/path/to/file -ti image_name:latest" mynewscript $#
Of course that ran against the image and NOT the container.
Once I noticed that I tried running it with exec instead of run and it ran without error, like so:
docker exec -it container_name mynewscript