Spring Dataflow and Yarn: How to set properties properly? - spring

Question

How can one change the default appdeployappmaster properties ?
When I'm trying to deploy an application through Spring DataFlow YARN. I registered my app, created a stream, and click the "deploy" button. When doing so, I get the following error :
[XNIO-2 task-2] WARN o.s.c.d.s.c.StreamDeploymentController - Exception when deploying the app StreamAppDefinition [streamName=histo, name=my-app, registeredAppName=my-app, properties={spring.cloud.stream.bindings.input.destination=log, spring.cloud.stream.bindings.input.group=histo}]: java.util.concurrent.ExecutionException: org.springframework.yarn.YarnSystemException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost; nested exception is java.net.UnknownHostException: Invalid host name: local host is: (unknown); destination host is: "null":8032; java.net.UnknownHostException; For more details see: http://wiki.apache.org/hadoop/UnknownHost
As you can see, the deployer is unable to find the "Resource Manager" URI, Although it is well found when the Spring DataFlow Server starts.
So I only get the problem at the deployment time.
Which property should I set to fix this issue, and where would I do that ?
EDIT 1:
Following Janne Valkealahti's answer, I added the following properties in /dataflow/apps/stream/app/servers.yml, relaunched the server, and tried to re-deploy my stream.
spring:
cloud:
dataflow:
yarn:
version: 0.0.1-SNAPSHOT
deployer:
yarn:
version: 1.0.2.RELEASE
stream:
kafka:
binder:
brokers: kafka.my-domain.com:9092
zkNodes: zookeeper.my-domain.com:2181/node
# Configured for Hadoop single-node running on localhost. Replace with property values reflecting your
# actual Hadoop cluster when running in a distributed environment.
hadoop:
fsUri: hdfs://mapr.my-domain.com/referentiel/ca_category_2014/
resourceManagerHost: mapr.my-domain.com
resourceManagerPort: 8032
resourceManagerSchedulerAddress: mapr.my-domain.com:8030
session:
store-type: none
I still get the exact same message.
PS: I'm not using Ambari, I'd like to understand how it works manually first.
EDIT 2:
I solved the problem adding the -Dspring.config.location VM arg on the DataFlow Server. The given configuration is passed to the deployer, and the application is effectively deployed.
I'll write an answer for it.

Answer 1

You didn't tell if your installation was based on ambari or normal manual YARN install so I assume it was a latter(manual).
I think a problem is that in distribution you use the config/servers.yml has a wrong setting for resourceManagerHost as it defaults to localhost. This file is distribute only once into hdfs when streams are launched. If you have changed it after you redeploy/create stream, app in hdfs directory will not get updated. On default this file in hdfs is /dataflow/apps/stream/app/servers.yml.
This error makes sense as also dataflow yarn server controlling whole stuff also needs access to yarn resource manager to submit apps. Settings for server also comes from a same servers.yml file.

Answer 2

It turns out I needed to add the -Dspring.config.location JVM arg to make it work. -Dspring.config.location should point to the file containing the YARN configuration, i.e.:
spring:
cloud:
dataflow:
yarn:
version: 0.0.1-SNAPSHOT
deployer:
yarn:
version: 1.0.2.RELEASE
stream:
kafka:
binder:
brokers: kafka.my-domain.com:9092
zkNodes: zookeeper.my-domain.com:2181/node
# Configured for Hadoop single-node running on localhost. Replace with property values reflecting your
# actual Hadoop cluster when running in a distributed environment.
hadoop:
fsUri: hdfs://mapr.my-domain.com/referentiel/ca_category_2014/
resourceManagerHost: mapr.my-domain.com
resourceManagerPort: 8032
resourceManagerSchedulerAddress: mapr.my-domain.com:8030
session:
store-type: none
This configuration is then passed to the deployer app (appdeployerappmaster if I get it right).