I am using hadoop jar-tasklet:
<hdp:jar-tasklet id="testjob" jar="bhs_abhishek.jar">
</hdp:jar-tasklet>
This jar currently needs some config files on classpath which I was earlier setting through HADAOOP_CLASSPATH variable for invocation through hadoop jar command. But I could not find a way of setting HADOOP_CLASSPATH using spring xml. Please provide any suggestions on how this can be achieved or a better way of doing this. I am OK to make changes in jar.
You can try adding your config files to xd/config directory which should be on the classpath.
There is also a xd/config/hadoop-site.xml file where you could add Hadoop config properties. One more alternative is to modify xd/config/servers.yml and add Hadoop config properties under spring:hadoop:config: like we do for io.file.buffer.size in this example:
---
# Hadoop properties
spring:
hadoop:
fsUri: hdfs://hadoop.example.com:8020
resourceManagerHost: hadoop.example.com
resourceManagerPort: 8032
jobHistoryAddress: hadoop.example.com:10020
config:
io.file.buffer.size: 4096
---
Related
I'm trying to use the jasypt-spring-boot and deploy it to a Tomcat server as war.
How to pass the encryptor password, in this case, to ensure the encrypted values could be read?
All the provided example are about running a jar file or a Spring Boot app as follows:
java -Djasypt.encryptor.password={my-password-to-decrypt} -jar target/jasypt-spring-boot-demo-0.0.1-SNAPSHOT.jar
May be add some settings to catalina.properties file in the Tomcat conf folder as we do for example when defining active profile?
I figured out how to achieve that:
create a setenv.sh file in CATALINE_HOME/bin folder
add the following entry to set the environment variable on the Tomcat startup:
export JASYPT_ENCRYPTOR_PASSWORD=your-password
save and restart Tomcat.
I have spring boot application that I am running in jboss(instead of tomcat).
I am using spring profile for loading environment specific application.properties.
Profile specific application{env}.properties is working fine when placed under "src/main/resources/" however, it is not working when placed externally.
I tried setting property in jboss standalone.xml but application fails to start in jboss.
<property name="spring.config.location" value="C:/Dev/config/rt" />
Please suggest how to load the environment specific application.properties files that are not placed inside the war file.
I was expecting spring to pick the profile specific file from the folder but looks like that's not the case.
It appears that spring.config.location needs to have the absolute file location instead of folder location. so, updated spring.config.location in standalone.xml and it worked :
<property name="spring.config.location" value="C:/Dev/config/rt/application-dev.properties" />
You can provide externalized configuration file using below command when you are initially starting the application,
java -jar <your-jar-name.jar> --spring.config.location=<path><external property>
example :
java -jar mySample.jar --spring.config.location=./application-external.properties
read more
Generally it was picked very easily when it's placed under the 'src/main/resources' folder. suppose you have to different files for profiles like - application-dev.properties and application-prod.properties, you need to set only the current working profile in the application.properties like
spring.profiles.active=dev
and it will be picked easily. If it's doesn't you need to create a workaround by creating a new bat or sh file like run.bat and run.sh in the bin folder of the jboss and pass the file location like
--spring.config.location=
The complete command to be added in the bat/sh file will be
java -jar appName.jar --spring.config.name=application-dev --spring.config.location=c:/Dev/application-dev.properties
I have Spring Boot application deployed as WAR file on standalone Apache Tomcat.
In order to configure Logback on Windows machine I've created setenv.bat file with the following content:
set logging.config=<absolute path to logback.xml>
I was trying the same for Tomcat deployed on Mac, but the following setenv.sh doesn't work:
logging.config=<absolute path to logback.xml>
I've also tried set logging.config variable using export command:
export logging.config=<absolute path to logback.xml>
but result is the same - log files aren't created.
Any ideas what is wrong with my configuration?
I hope you already have found your answer somewhere. But what I have done is that in conf/catalina.properties, point to the logback config file as follows:
logging.config=${catalina.base}/conf/logback-spring.xml
I don't do anything in setenv.bat.
I have a client application that uses the hadoop conf files (hadoop-site.xml and hadoop-core.xml)
I don't want to check it in on the resources folders, so I try to add it via idea.
The problem is that the hadoop Confs ignores my HADOOP_CONF_DIR and loads the default confs from the hadoop package. Any ideia ?
I'm using gradle
I end up solving it by putting the configuration files on test resources folder. So when the jar gets build it does not take it.
I have a Spark application which I would like to configure using configuration files, such as Spark's spark-defaults.conf, HBase's hbase-site.xml and log4j's log4j.properties. I also want to avoid having to add the files programmatically.
I tried adding the files to my JAR (under both / and /conf paths) but when I ran spark-submit the configuration files did not seem to have any effect.
To further check my claim I tried running spark-shell with the same JARs and checking the contents of the files and I discovered that they were overridden by files from other locations: /spark-defaults.conf and /log4j.properties were completely different, and /conf/hbase-site.xml while staying intact has (probably) had its properties overridden by another JAR's hbase-default.xml.
I use CDH 5.4.0.
The files log4j.properties and spark-defaults.conf were loaded from /etc/spark/ and hbase-default.xml was loaded from /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/jars/hbase-common-1.0.0-cdh5.4.0.jar.
Is there a way to specify some sort of priority on my configuration files over the others? Should I just configure the files in /etc/spark (and perhaps add my hbase-site.xml too)? Is there a way to add a custom directory path to the classpath that could take priority over the others?
i don't think that it is possible to include the spark-defaults.conf into the jar. The only way i know is to edit the file on the server or to add the config settings programatically.
but for hbase-site.xml and other hadoop site configs it should work.
you can put every site.xml in the root of your resource directory and it should be loaded unless you have some other site.xml in the classpath of spark which are loaded at first.
i.e. if you are adding hadoop classpath or hbase classpath to the spark env on the server, then they are in the classloader loaded first, unless you are using the setting spark.files.userClassPathFirst