Why I cannot scale openshift cart having an environment variable defined? - cluster-computing

Question

I am trying to integrate Hazelcast in Tomcat cartridge (https://github.com/worldline/openshift-cartridge-tomcat). The problem is to retrieve ip:ports of gears of a scalable application. I looked at how vert.x does it and it is perfectly fine. It uses pub/sub mechanist to set ip and port in the future environment variable. I can see this in hook folder "set-vertex-cluster" file.
echo $list > $OPENSHIFT_VERTX_DIR/env/OPENSHIFT_VERTX_HAZELCAST_CLUSTER
I did id the same way replacing VERTX by TOMCAT (short name of cart).
But after creating the app there is no OPENSHIFT_TOMCAT_HAZELCAST_CLUSTER env variable.
I looked at how JbossEAP does it. It has
touch ${OPENSHIFT_JBOSSEAP_DIR}/env/OPENSHIFT_JBOSSEAP_CLUSTER
https://github.com/openshift/origin-server/blob/master/cartridges/openshift-origin-cartridge-jbosseap/bin/install
It worked for me, and finally I see OPENSHIFT_VERTX_HAZELCAST_CLUSTER env var and it is populated by gear_ip:gear_port.It's good. But when I scale the app I get the error:
Activation of new gears failed: 53d2ed31e0b8cd2bba00051f: Error activating gear: CLIENT_ERROR: Failed to execute: 'control start' for /var/lib/openshift/53d2ed31e0b8cd2bba00051f/tomcat
Unable to complete the requested operation due to: An invalid exit code (1) was returned from the server ex-std-node92.prod.rhcloud.com. This indicates an unexpected problem during the execution of your request.
Reference ID: 0b4e8a465d1901e8317a18739586e6d1
OPENSHIFT_VERTX_HAZELCAST_CLUSTER is populated by gear1_ip:gear1_port and gear2_ip:gear2_port but of course the second gear failed to start.
When I remove
touch ${OPENSHIFT_TOMCAT_DIR}/env/OPENSHIFT_TOMCAT_HAZELCAST_CLUSTER
from bin/install file everything is fine! Except that I don't have list of cluster members...
I am going mad, struggling with the problem all day long. Can anybody help me, please?
UPDATED:
• Customer A create an OpenShift Application A1 with the Git downloadable cartridge
• OpenShift install the downloadable cartridge into Node N1 and install it into the Application A1.
• Customer A now want to scale the application A1.
• OpenShift try to scale the application A1 by acquiring a new gear in Node N2 (notice that it is different from N1 above) and copy the content from A1 into N2 (but somehow will not copy all the environment variables and necessary settings). In .env folder of every gear
• The gear creation now fail on N2, because the downable cartridge content is not available on N2, because of following commands in bin/tomcat
# Filter user-owned configuration files through sed to replace all
# ${OPENSHIFT_*} variables with their actual values, and write the
# resulting filtered files to the live Tomcat configuration location.
sed_replace_env=$(print_sed_exp_replace_env_var)
replacement_conf_files=(
"server.xml"
"context.xml"
)
for conf_file in "${replacement_conf_files[#]}"; do
sed ${sed_replace_env} ${OPENSHIFT_REPO_DIR}/.openshift/config/${conf_file} > ${OPENSHIFT_TOMCAT_DIR}/conf/${conf_file}
done
The particular function
function print_sed_exp_replace_env_var {
sed_exp=""
for openshift_var in $(env | grep OPENSHIFT_ | awk -F '=' '{print $1}')
do
# environment variable values that contain " or / need to be escaped
# or they will cause problems in the sed command line.
variable_val=$(echo "${!openshift_var}" | sed -e "s#\/#\\\\/#g" | sed -e "s/\"/\\\\\"/g")
# the entire sed s/search/replace/g command needs to be quoted in case the variable value
# contains a space.
sed_exp="${sed_exp} -e \"s/\\\${env.${openshift_var}}/${variable_val}/g\""
done
printf "%s\n" "$sed_exp"
}
As you can see, there is a dangerous of the sed command that may replace entire file into blank file for server.xml and context.xml if the environment variables are not present.
The correct order that OpenShift should perform is:
Customer A create an OpenShift Application A1 with the Git downloadable cartridge
• OpenShift install the downloadable cartridge into Node N1 and install it into the Application A1.
• Customer A now want to scale the application A1.
• OpenShift try to scale the application A1 by acquiring a new gear in Node N2 (notice that it is different from N1 above). Copy all necessary environment variables into the new gears as well
• There is a script within your cartridge that require the server.xml and context.xml template from downloadable cartridge, it can now be successfully found and copy.