I have the following docker file
FROM confluentinc/cp-kafka-connect:5.3.1
ENV CONNECT_PLUGIN_PATH=/usr/share/java
# JDBC-MariaDB
RUN wget -nv -P /usr/share/java/kafka-connect-jdbc/ https://downloads.mariadb.com/Connectors/java/connector-java-2.4.4/mariadb-java-client-2.4.4.jar
# SNMP Source
RUN wget -nv -P /tmp/ https://github.com/KarthikDuggirala/kafka-connect-snmp/releases/download/0.0.1.11/kafka-connect-snmp-0.0.1.11.tar.gz
RUN mkdir /tmp/kafka-connect-snmp && tar -xf /tmp/kafka-connect-snmp-0.0.1.11.tar.gz -C /tmp/kafka-connect-snmp/
RUN mv /tmp/kafka-connect-snmp/usr/share/kafka-connect/kafka-connect-snmp /usr/share/java/
# COPY script and make it executable
COPY plugins-config.sh /usr/share/kafka-connect-script/plugins-config.sh
RUN ["chmod", "+x", "/usr/share/kafka-connect-script/plugins-config.sh"]
#entrypoint
ENTRYPOINT [ "./usr/share/kafka-connect-script/plugins-config.sh" ]
and the following bash script
#!/bin/bash
#script to configure kafka connect with plugins
#export CONNECT_REST_ADVERTISED_HOST_NAME=localhost
#export CONNECT_REST_PORT=8083
url=http://$CONNECT_REST_ADVERTISED_HOST_NAME:$CONNECT_REST_PORT/connectors
curl_command="curl -s -o /dev/null -w %{http_code} $url"
sleep_second=5
sleep_second_counter=0
max_seconds_to_wait=30
echo "Waiting for Kafka Connect to start listening on localhost"
echo "HOST: $CONNECT_REST_ADVERTISED_HOST_NAME , PORT: $CONNECT_REST_PORT"
while [[ $(eval $curl_command) -eq 000 ]]
do
echo "In"
echo -e $date " Kafka Connect listener HTTP state: " $(eval $curl_command) " (waiting for 200) $sleep_second_counter"
echo "Going to sleep for $sleep_second seconds"
# sleep $sleep_second
echo "Finished sleeping"
# ((sleep_second_counter+=$sleep_second))
echo "Finished counter"
done
echo "Out"
nc -vz $CONNECT_REST_ADVERTISED_HOST_NAME $CONNECT_REST_PORT
I try to run the docker and using docker logs to see whats happening, and I am expecting that the script would run and wait till the kafka connect is started. But apparently after say few seconds the script or (I dont know what is hanging) hangs and I do not see any console prints anymore.
I am a bit lost what is wrong, so I need some guidance on what is that I am missing or is this not the correct way
What I am trying to do
I want to have logic defined that I could wait for kafka connect to start then run the curl command
curl -X POST -H "Content-Type: application/json" --data '{"name":"","config":{"connector.class":"com.github.jcustenborder.kafka.connect.snmp.SnmpTrapSourceConnector","topic":"fm_snmp"}}' http://localhost:8083/connectors
PS: I cannot use docker-compose way to do it, since there are places I have to use docker run
The problem here is that ENTRYPOINT will run when the container starts and will prevent the default CMD to run since the script will loop waiting for the server to be up , that is the script will loop forever since the CMD will not run.
you need to do one of the following:
start the kafka connect server in your Entrypoint and your script in CMD or running your script outside the container ....
Related
I need send curl request to itself while docker container starting.
But after nginx starting any command not working because nginx stay in foreground.
Example of entrypoint.sh
echo "Starting memcached"
memcached memcache -m 1024 -d
service memcached restart
echo "Starting php-fpm"
php-fpm -D
echo "Starting Nginx"
nginx -g 'daemon off;'
!!this part not working!!!
check_robots=$(wp wpc check_robots)
echo "Starting check robots"
if [ "$check_robots" != "Robots checked successfully!" ]; then
echo ERROR: Robots checked failed
exit 0
fi
exec "$#"
Not sure if it would work, but lanching nginx to background directly could be a solution, like so:
nginx -g 'daemon off;' &
Have not tested in case of Docker entrypoint though.
I need to connect redis server from my local machine . I have installed redis on my machine under the path: /Users/huangkunlun/Work/soft/redis/redis-6.0.10
If I enter the redis installation and run the command like following from my terminal,it will be successful.
cd /Users/huangkunlun/Work/soft/redis/redis-6.0.10/src
redis-cli -h reids-host-server -p 6379 -a psw
but if i run the previous two command in shell script file,I got error displaying command redis-cli can not be found.
so the question is how can I enter a specific dir and execute the specific file under that dir from a shell script ?
the shell script file is like following:
#!/bin/bash
REDIS_SRC_DIR=/Users/huangkunlun/Work/soft/redis/redis-6.0.10/src
HOST_TEST=redis-test-in.shantaijk.cn
HOST_DEV=r-uf61x52g8b0cketmk7.redis.rds.aliyuncs.com
env=$1
echo "environment is ${env}"
cd ${REDIS_SRC_DIR}
echo "cd redis directory $(pwd)"
if [[ ${env} == "test" ]]; then
#/Users/huangkunlun/Work/soft/redis/redis-6.0.10/src/redis-cli -h ${HOST_TEST} -p 6379 -a Cloudhis1234
redis-6.0.10/src/redis-cli -h ${HOST_TEST} -p 6379 -a Cloudhis1234
elif [[ ${env} == "dev" ]]; then
#/Users/huangkunlun/Work/soft/redis/redis-6.0.10/src/redis-cli -h ${HOST_DEV} -p 6379 -a Cloudhis1234
redis-cli -h ${HOST_DEV} -p 6379 -a Cloudhis1234
else
echo "no env args is specified, exit..."
fi
Cloud-build is not showing build failure status
I created my own remote-builder which scp all files from /workspace to my Instance and running build on using gcloud compute ssh -- COMMAND
remote-builder
#!/bin/bash
USERNAME=${USERNAME:-admin}
REMOTE_WORKSPACE=${REMOTE_WORKSPACE:-/home/${USERNAME}/workspace/}
GCLOUD=${GCLOUD:-gcloud}
KEYNAME=builder-key
ssh-keygen -t rsa -N "" -f ${KEYNAME} -C ${USERNAME} || true
chmod 400 ${KEYNAME}*
cat > ssh-keys <<EOF
${USERNAME}:$(cat ${KEYNAME}.pub)
EOF
${GCLOUD} compute scp --compress --recurse \
$(pwd)/ ${USERNAME}#${INSTANCE_NAME}:${REMOTE_WORKSPACE} \
--ssh-key-file=${KEYNAME}
${GCLOUD} compute ssh --ssh-key-file=${KEYNAME} \
${USERNAME}#${INSTANCE_NAME} -- ${COMMAND}
below is the example of the code to run build(cloudbuild.yaml)
steps:
- name: gcr.io/$PROJECT_ID/remote-builder
env:
- COMMAND="docker build -t [image_name]:[tagname] -f Dockerfile ."
During docker build inside Dockerfile it got failure and show errors in log but status showing SUCCESS
can any help me how to resolve it.
Thanks in advance.
try adding
|| exit 1
at the end of your docker command... alternatively, you might just need to change the entrypoint to 'bash' and run the script manually
To confirm -- the first part was the run-on.sh script, and the second part was your cloudbuild.yaml right? I assume you trigger the build manually via UI and/or REST API?
I wrote all docker commands on bash script and add below error handling code to it.
handle_error() {
echo "FAILED: line $1, exit code $2"
exit 1
}
trap 'handle_error $LINENO $?' ERR
It works!
for testing I try to run the gcloud devserver inside docker with that comment:
sudo /usr/local/gcloud/google-cloud-sdk/bin/java_dev_appserver.sh --disable_update_check --port=8888 --help /app/target/qdacity-war/ 2>&1 | sudo tee /app/logs/devserver.log > /dev/null &
To check if the devserver has started successfully, I use this script:
#!/bin/bash
# This script waits until the port 8888 is open.
SERVER=localhost
PORT=8888
TIMEOUT=180
TIME_INTERVAL=2
PORT_OPEN=1
PORT_CLOSED=0
time=0
isPortOpen=0
while [ $time -lt $TIMEOUT ] && [ $isPortOpen -eq $PORT_CLOSED ];
do
# Connect to the port
(echo > /dev/tcp/$SERVER/$PORT) >/dev/null 2>&1
if [ $? -ne 0 ]; then
isPortOpen=$PORT_CLOSED
else
isPortOpen=$PORT_OPEN
fi
time=$(($time+$TIME_INTERVAL))
sleep $TIME_INTERVAL
done
if [ $isPortOpen -eq $PORT_OPEN ]; then
echo "Port is open after ${time} seconds."
# Give the server more time to properly start
sleep 10
else
echo "Reached the timeout (${TIMEOUT} seconds). The port ${SERVER}:${PORT} is not available."
exit 1
fi
After running all the test, I just got back:
Reached the timeout (180 seconds). The port localhost:8888 is not available.
I couldn't find out if there were any problems starting the devserver or querying the port.
Does anyone have an idea or solution?
Thanks!
By default, by only accepting localhost|loopback traffic, you're unable to access the server remotely.
Please try adding --address=0.0.0.0:
(link) to your java_dev_appserver.sh command.
Example
Used a variant of Google's HelloWorld sample.
Ran this with mvn appengine:run (to confirm it works and build the WAR).
Then /path/to/bin/java_dev_appserver.sh ./target/hellofreddie-0.1 (to confirm it works with the local development server).
Then used Google's Cloud SDK container image (link), mounted the previously generated WAR directory into it, and ran the server on :9999:
docker run \
--interactive \
--tty \
--publish=9999:9999 \
--volume=${PWD}/target:/target \
google/cloud-sdk \
/usr/lib/google-cloud-sdk/bin/java_dev_appserver.sh \
--address=0.0.0.0 \
--port=9999 \
./target/hellofreddie-0.1
Am able to curl the endpoint:
curl \
--silent \
--location \
--write-out "%{http_code}" \
--output /dev/null \
localhost:9999
returns 200
And, running your scripts adjusted with PORT=9999 returns:
Port is open after 2 seconds.
I have a jenkins job that will do a deployment in my CentOS machine by running the docker-compose file. This is how my shell script looks ?
#!/bin/sh
# Post steps for deployment
# Navigate to deployment scripts
cd /deployment/scripts/v1.0.784
#Execute the uninstall script
./dit_undeploy_all.sh
set +e
#Remove all docker images and containers
docker container rm $(docker ps -a -q) -f
docker rmi $(docker images -a -q)
docker volume rm $(docker volume ls -q)
set -e
#Remove and clear out the folder structure
rm -rf *.*
#Gets all the latest files from Artifactory by reading from teh input file
wget -B https://artifactory.gue.com -i /deployment/scripts/inputFile.txt
# Gives rea/write access
chmod +x *.*
# Execute docker compose file to get all the latest containers
./dit_deploy_all.sh
#Add wait time for the services to be up and running
sleep 60s #Wait 15 sec
# Need to update the URL
./dit_create_policies.sh
#Verify URL Status Code of 200
cd /deployment/scripts
sleep 60s #Wait time 60s
./verifyHttpCode
The script ./verifyHttpCode does the following:
#!/bin/bash
while read LINE; do
curl -o /dev/null --silent --head --write-out '%{http_code}' "$LINE"
echo " $LINE"
done < url-list.txt
So basically after the deployment it will verify the http status code...What's the equivalent of testNG in shell script that I can use in Jenkins to verify the http status code and generate reports ??
Have you tried this plugin : Audit2DB
https://wiki.jenkins.io/display/JENKINS/Audit+To+Database+Plugin
It will log all required job details into a DB. When you want to create a report, it will fetch it from DB.
So in this case, you'll need to fail the job (non-zero exit) on script failure. Also you can set $http_code as env variable on completion of your job and log the same into DB.