I am using Flux 1.0.0 and I have rewritten my topology into a YAML file. But I have some properties that used to be part of the configuration that I used the Storm driver to run with.
storm.Driver --config myConfig/config.conf
Now with Storm Flux, how can I inject the properties that are in config.conf into my topology?
I am currently doing java -cp myStormJar org.apache.sotrm.flux.Flux --local /src/main/resources/myTopology.yaml
I tried to use --resources option, followed by the path to the conf file, but it does not inject it.
Add the filter --resources placeholders ${resource.filter} to your yaml file.
To make the property available in the stormConf - re-declare the filter resource in config: properties.
name: "storm-topology"
config:
kafka.mapper.zkPort: ${kafka.mapper.zkPort}
kafka.mapper.zkServers: ${kafka.mapper.zkServers}
You can also review the simple_hdfs.yaml example at: https://github.com/ptgoetz/flux/tree/master/flux-examples
Related
I'm trying to build a Docker image of Tomcat that uses Elastic logging. I followed the ECS Logging Java Reference documentation and added the required jars to $CATALINA_HOME/lib, however, it seems that the classes are not loaded by Tomcat.
Tomcat's class loader documentation states the following about the common class loader:
This class loader contains additional classes that are made visible to both Tomcat internal classes and to all web applications.
Normally, application classes should NOT be placed here. The locations searched by this classloader are defined by the common.loader property in $CATALINA_BASE/conf/catalina.properties. The default setting will search the following locations in the order they are listed:
unpacked classes and resources in $CATALINA_BASE/lib
JAR files in $CATALINA_BASE/lib
unpacked classes and resources in $CATALINA_HOME/lib
JAR files in $CATALINA_HOME/lib
If I understand the above correctly, placing the jars in $CATALINA_HOME/lib should be enough for the common loader to find them. However the logging is not formatted by the Elastic formatter.
As a workaround I added a setenv.sh file with a CLASSPATH variable that points to the jars I added. The logging is then printed correctly in ECS format.
Can someone please explain to me how I can get this to work without using the workaround and make use of the common class loader instead?
Below is my Dockerfile:
FROM tomcat:9-jre17
COPY conf/logging.properties conf/logging.properties
COPY lib/ecs-logging-core-1.5.0.jar lib/ecs-logging-core.jar
COPY lib/jul-ecs-formatter-1.5.0.jar lib/jul-ecs-formatter.jar
COPY lib/logback-ecs-encoder-1.5.0.jar lib/logback-ecs-encoder.jar
# Workaround
COPY bin/setenv.sh bin
RUN chmod +x bin/setenv.sh
EXPOSE 8080
CMD ["catalina.sh", "run"]
This is my logging.properties file:
handlers = java.util.logging.ConsoleHandler
.handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = co.elastic.logging.jul.EcsFormatter
co.elastic.logging.jul.EcsFormatter.serviceName = tomcat
java.util.logging.ConsoleHandler.encoding = UTF-8
And this is the setenv.sh file that I use as a workaround, but that I would like to get rid of:
#!/bin/sh
export CLASSPATH="/usr/local/tomcat/lib/ecs-logging-core.jar:/usr/local/tomcat/lib/jul-ecs-formatter.jar:/usr/local/tomcat/lib/logback-ecs-encoder.jar"
The result that I'm expecting looks like this:
{"#timestamp":"2022-11-28T12:29:43.276Z","log.level": "INFO","message":"Server version name: Apache Tomcat/9.0.69","ecs.version": "1.2.0","service.name":"tomcat","event.dataset":"tomcat","process.thread.name":"main","log.logger":"org.apache.catalina.startup.VersionLoggerListener"}
But the result that I'm getting without adding setenv.sh looks like this:
2022-11-28 13:44:20 Nov 28, 2022 12:44:20 PM org.apache.catalina.startup.VersionLoggerListener log
2022-11-28 13:44:20 INFO: Server version name: Apache Tomcat/9.0.69
I am using test container with Janusgraph, but I need to use ConfiguredGraphFactory in my test. How to enable it in test container? This is how I am starting the test container:
GenericContainer<JanusGraph> janus = new JanusGraph()
.withNetwork(network)
.withExposedPorts(JANUS_PORT)
.waitingFor(Wait.forListeningPort())
.withStartupTimeout(Duration.ofMinutes(5))
.withEnv("janusgraph.storage.backend", "cql")
.withEnv("janusgraph.storage.hostname", storageName)
.withEnv("janusgraph.index.search.backend", "elasticsearch")
.withEnv("janusgraph.index.search.hostname", indexName)
.withEnv("janusgraph.schema.default", "tp3");
I was trying to add .withEnv("janusgraph.graph.graphname"."org.janusgraph.core.ConfiguredGraphFactory") but it doesn't work (image name: janusgraph/janusgraph:0.5.3).
The example yaml and properties files for trying ConfiguredGraphFactory can be found at:
conf/gremlin-server/gremlin-server-configuration.yaml
conf/janusgraph-cql-configurationgraph.properties
It is described in the janusgraph-docker project that you can define gremlin.* and janusgraph.* environment variables in your Dockerfile to override the default configs. Your approach with GenericContainer.withEnv() should be fine too.
From looking at this JanusGraph PR, I think this approach should work:
.withEnv("gremlin.graph"."org.janusgraph.core.ConfiguredGraphFactory")
However, I have no experience with JanusGraph and I think this is not a Testcontainers-specific question. This means you can conceptually configure it for Testcontainers the same way to would configure it for a Docker container with the Docker CLI.
I'm wondering if there's a nice clean way to set logging location based on OS just using the application.properties file in Spring Boot?
For instance is it possible to use a regex matcher on ${os.name} or would I just need to go ahead and create a groovy script or something?
My ideal solution is something like
logging:
file: ${os.name}.test(/*window*/gi) ? C:/ProgramData/Logs/ : /var/log/
You can take advantage of spring profiles and pick configurations according to the -Dspring.profile.active=some_profile system property or SPRING_PROFILES_ACTIVE=some_profile env variable.
Yaml file could be
# a safe default relative to app root
logging:
file: logs
----
spring:
profiles: nix
logging:
file: /var/log/myapp
----
spring:
profiles: win
logging:
file: C:/ProgramData/Logs/
App is executed as
java -Dspring.profile.active=nix <more opts> MyAppMain
or also:
SPRING_PROFILES_ACTIVE=nix java <more opts> MyAppMAin
I want to send a key-value pair to my spark application something like the following:
mapreduce.input.fileinputformat.input.dir.recursive=true
I understand I can do this from the code in the following way:
sc.hadoopConfiguration.set("mapreduce.input.fileinputformat.input.dir.recursive","true")
But I want to be able to send this property through spark-submit at runtime. Would this be possible?
Absolutely!
spark-submit (as well as spark-shell) support the --conf PROP=VALUE and --properties-file FILE options, which allow you specify such arbitrary configurations options. You can then get the values you pass by using the SparkConf .get function:
val conf = new SparkConf()
val mrRecursive =
conf.get("spark.mapreduce.input.fileinputformat.input.dir.recursive")
sc.hadoopConfiguration.set("spark.mapreduce.input.fileinputformat.input.dir.recursive", mrRecursive)
Spark-submit/spark-shell --help:
--conf PROP=VALUE Arbitrary Spark configuration property.
--properties-file FILE Path to a file from which to load extra properties. If not
specified, this will look for conf/spark-defaults.conf.
Spark docs regarding [dynamically] loading properties: https://spark.apache.org/docs/latest/configuration.html
Without code modification, such approach can be used.
Hadoop Configuration reads file "core-default.xml" during creation, description is here:
https://hadoop.apache.org/docs/r2.6.1/api/org/apache/hadoop/conf/Configuration.html
If put values in "core-default.xml", and include directory with file in classpath with spark-submit "driver-class-path" parameter, it can work.
I would like to specify location of property file, from which OSGi blueprint property placeholder should read properties values:
<cm:property-placeholder persistent-id="myBundle"/>
<bean
id="myCoolBean"
class="test.properties.MyCoolBean">
<property
name="echo"
value="${echo}"/>
</bean>
UPDATE:
Configuration felix.configadmin + felix.fileinstall works for me.
I installed:
org.apache.felix.configadmin-1.8.0.jar
org.apache.felix.fileinstall-3.1.4.jar
org.eclipse.equinox.cm-3.2.0.jar
I specified VM argument -Dfelix.fileinstall.dir=C:/eclipse/config
The myBundle.cfg file has value:
echo=Echo
The property placeholder in blueprint does not work with files. Instead it uses the persistent id to retreive a config from ConfigurationAdmin service.
So the solution is to install felix config admin together with felixfileinstall. So configs will be retrieved from a folder and updated in ConfigurationAdmin.
In apache karaf this is already configured but you can also do it on your own. See my karaf tutorial about config admin.
If you want to go with plain felix then you can take a look what karaf does to solve it. So for example in config.properties there are the settings for felix fileinstall. There you have to e.g. set he directory containing your configs. In plain felix that would be framework properties.
This is what karaf sets:
felix.fileinstall.enableConfigSave = true
felix.fileinstall.dir = ${karaf.etc}
felix.fileinstall.filter = .*\\.cfg
felix.fileinstall.poll = 1000
felix.fileinstall.noInitialDelay = true
felix.fileinstall.log.level = 3