Tomcat common class loader not working as expected - elasticsearch

I'm trying to build a Docker image of Tomcat that uses Elastic logging. I followed the ECS Logging Java Reference documentation and added the required jars to $CATALINA_HOME/lib, however, it seems that the classes are not loaded by Tomcat.
Tomcat's class loader documentation states the following about the common class loader:
This class loader contains additional classes that are made visible to both Tomcat internal classes and to all web applications.
Normally, application classes should NOT be placed here. The locations searched by this classloader are defined by the common.loader property in $CATALINA_BASE/conf/catalina.properties. The default setting will search the following locations in the order they are listed:
unpacked classes and resources in $CATALINA_BASE/lib
JAR files in $CATALINA_BASE/lib
unpacked classes and resources in $CATALINA_HOME/lib
JAR files in $CATALINA_HOME/lib
If I understand the above correctly, placing the jars in $CATALINA_HOME/lib should be enough for the common loader to find them. However the logging is not formatted by the Elastic formatter.
As a workaround I added a setenv.sh file with a CLASSPATH variable that points to the jars I added. The logging is then printed correctly in ECS format.
Can someone please explain to me how I can get this to work without using the workaround and make use of the common class loader instead?
Below is my Dockerfile:
FROM tomcat:9-jre17
COPY conf/logging.properties conf/logging.properties
COPY lib/ecs-logging-core-1.5.0.jar lib/ecs-logging-core.jar
COPY lib/jul-ecs-formatter-1.5.0.jar lib/jul-ecs-formatter.jar
COPY lib/logback-ecs-encoder-1.5.0.jar lib/logback-ecs-encoder.jar
# Workaround
COPY bin/setenv.sh bin
RUN chmod +x bin/setenv.sh
EXPOSE 8080
CMD ["catalina.sh", "run"]
This is my logging.properties file:
handlers = java.util.logging.ConsoleHandler
.handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = INFO
java.util.logging.ConsoleHandler.formatter = co.elastic.logging.jul.EcsFormatter
co.elastic.logging.jul.EcsFormatter.serviceName = tomcat
java.util.logging.ConsoleHandler.encoding = UTF-8
And this is the setenv.sh file that I use as a workaround, but that I would like to get rid of:
#!/bin/sh
export CLASSPATH="/usr/local/tomcat/lib/ecs-logging-core.jar:/usr/local/tomcat/lib/jul-ecs-formatter.jar:/usr/local/tomcat/lib/logback-ecs-encoder.jar"
The result that I'm expecting looks like this:
{"#timestamp":"2022-11-28T12:29:43.276Z","log.level": "INFO","message":"Server version name: Apache Tomcat/9.0.69","ecs.version": "1.2.0","service.name":"tomcat","event.dataset":"tomcat","process.thread.name":"main","log.logger":"org.apache.catalina.startup.VersionLoggerListener"}
But the result that I'm getting without adding setenv.sh looks like this:
2022-11-28 13:44:20 Nov 28, 2022 12:44:20 PM org.apache.catalina.startup.VersionLoggerListener log
2022-11-28 13:44:20 INFO: Server version name: Apache Tomcat/9.0.69

Related

Spring application war - Google App Engine - How to specify Log location, profile, environment variab;les

I have a spring boot application bundled as war file , and able to push to App Engine
But I am getting problems starting app (I suspect there could be an issue with DB too...but couldnt remember where I saw...a nightmare)
java.lang.RuntimeException: java.lang.IllegalStateException: Logback
configuration error detected: ERROR in
ch.qos.logback.core.rolling.RollingFileAppender[FILE] - Failed to
create parent directories for
[/base/data/home/apps/e~pemy/20210716t001812.436581063072799646/logs/pynew.log]
ERROR in ch.qos.logback.core.rolling.RollingFileAppender[FILE] -
openFile(logs/pynew.log,true) call failed.
java.io.FileNotFoundException: logs/pynew.log (No such file or
directory)
I am using the below properties in my application props
> logging.file.path=logs
> logging.file.name=${logging.file.path}/pynew.log
I am finding it very hard to include google specific dependencies and properties , and making a mess of my project...created app.yaml, web-inf>> appengine-web xml, logging.properties (not sure why but added as told in a tutorial)
Question: How can I create parent directory or link to cloud storage folder etc?
I also want to specify a profile and I see I can do it in yaml file. Is this used only
env_variables:
JAVA_USER_OPTS: '-Dspring.profiles.active=prod'
But I would like to know how to connect to Cloud SQL
spring.datasource.url=jdbc:mysql:///mydb?cloudSqlInstance=myapp:europe-west2:dBinstancename&socketFactory=com.google.cloud.sql.mysql.SocketFactory
spring.datasource.username=${dbuser}
spring.datasource.password=ENC(${dbencpwd})
spring.cloud.gcp.sql.database-name=mydb
spring.cloud.gcp.sql.instance-connection-name=myapp:europe-west2:dBinstancename
It is so confusing that I keep forgetting which connection needs password and which wont. and keep breaking my local
Question
Assuming that I need to supply credentials, How can I supply - ${dbuser}
I used the default spring logger with logback-spring.xml for all my development, and this is not working on AppEngine
So I followed https://cloud.google.com/logging/docs/setup/java
and added logback.xml and the dependency
implementation 'com.google.cloud:google-cloud-logging-logback:0.121.3-alpha'

Spring kafka not able to read truststore file from classpath

I am building an kafka consumer app which needs SASL_SSL config. Some how apache kafka is not recognizing truststore file located in classpath and looks like there is an open request to enhance it in kafka(KAFKA-7685).
In the mean time what would be the best way to solve this problem. Same app needs to deployed in PCF too so solution should work both during local windows based development and PCF (linux).
Any solution would be highly appreciated.
Here is the code which does file copy to java temp dir
String tempDirPath = System.getProperty("java.io.tmpdir");
System.out.println("Temp dir : " + tempDirPath);
File truststoreConf = ResourceUtils.getFile("classpath:Truststore.jks");
File truststoreFile = new File(tempDirPath + truststoreConf.getName());
FileUtils.copyFile(truststoreConf, truststoreFile);
System.setProperty("ssl.truststore.location", truststoreFile.getAbsolutePath());
You could use a ClassPathResource and FileCopyUtils to copy it from the jar to a file in a temporary directory in main() before creating the SpringApplication.
Root cause of this issue was resource filtering enabled. Maven during resource filtering corrupts the binary file. So if you have that enabled, disable it

Why doesn't my NiFi Nar show up in the processor list?

Just made my first NAR from a tutorial, put it in the /lib file and restarted NiFi. The Processor shows up as loaded in the logs but does not show up in the processor list. Any ideas?
Tutorial: http://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
#SideEffectFree
#Tags({"JSON", "NIFI ROCKS"})
#CapabilityDescription("Fetch value from json path.")
public class JsonProcessor extends AbstractProcessor {
...
}
Edit:
I see two lines in the logs with this in it:
2017-07-06 19:34:06,200 INFO [main] org.apache.nifi.nar.NarClassLoaders Loaded NAR file: C:\UserApps\NIFI-1~1.0\.\work\nar\extensions\examples-1.0-SNAPSHOT.nar-unpacked as class loader org.apache.nifi.nar.NarClassLoader[.\work\nar\extensions\examples-1.0-SNAPSHOT.nar-unpacked]
What should the processor be called in the list? JsonProcessor?
Edit:
Tried changing the <nifi.version>1.3.0</nifi.version> in the pom from 1.2.0 to 1.3.0. No joy.
So on top of coding up the program you must put a file called
org.apache.nifi.processor.Processor
in YOUR project to get it to show up. I would suspect NiFi would scan for the classes with the annotations but alas, it does not. Probably some security thing.
The tutorial above does have this in it. "Browsing" got me again!
I also used this tutorial to create a Customer Processor. The issue I had was that I mixed up the ./lib directory.
When you start Apache Nifi, you would notice the following at the beginning of the log:
Java home: /Library/Java/JavaVirtualMachines/jdk1.8.0_45.jdk/Contents/Home
NiFi home: /usr/local/Cellar/nifi/1.6.0/libexec
Confirm that the NAR file is the $NIFI_HOME/libexec/lib folder which would be /usr/local/Cellar/nifi/1.6.0/libexec/lib in my case.
I hope this helps
NB: I ran Apache Nifi on a Mac, logs might be different on another OS

jodconverter doesn't find soffice.bin

I'm trying to use JODConverter on top of jetty and sinatra. Whenever I fire up my Jetty server, which has the sinatra application deployed to it in a war. i get this exception (much cruft removed):
java.lang.IllegalStateException: invalid officeHome: it doesn't contain soffice.bin:
in my settings.yml file I have the following:
secret: Whatever_you_want_it_to_be
PARAMETER_OFFICE_PORT: 8100
PARAMETER_OFFICE_HOME: /Volumes/OpenOffice.org 3.3/OpenOffice.org.app/Contents/MacOS
PARAMETER_OFFICE_PROFILE: /Applications/OpenOffice.org.app/Contents/MacOS
and here are the contents of those last two directories:
>ls '/Volumes/OpenOffice.org 3.3/OpenOffice.org.app/Contents/MacOS'
about.png crashrep intro.png sbase sdraw simpress soffice sofficerc testtool unopkg urelibs
boot straprc fundamentalrc resource scalc setuprc smath soffice.bin swriter unoinfo unopkg.bin versionrc
>ls /Applications/OpenOffice.org.app/Contents/MacOS
about.png crashrep intro.png sbase sdraw simpress soffice sofficerc testtool unopkg urelibs
bootstraprc fundamentalrc resource scalc setuprc smath soffice.bin swriter unoinfo unopkg.bin versionrc
Notice that both ls's show "soffice.bin" in them
update:
I've also tried this:
> sudo chmod 777 '/Volumes/OpenOffice.org 3.3/OpenOffice.org.app/Contents/MacOS/soffice.bin'
Password:
chmod: Unable to change file mode on /Volumes/OpenOffice.org 3.3/OpenOffice.org.app/Contents/MacOS/soffice.bin: Read-only file system
I've also tried using the second directory for office_home and vice versa. no luck.
If anyone still has the same problem (as I did) the problem was, that the parameter OfficeHome for Mac has to specify the folder "Contents", not the folder "Contents/MacOs/".
The JODConverter is checking if the following exists:
new File(officeHome, "MacOS/soffice.bin")
So the OpenOffice home folder should have been /Applications/OpenOffice.org.app/Contents.
Sounds like you're either missing a JAR file or there may be a conflict in the JAR files being used. Although it's not JRuby, this post should shed some light: http://groups.google.com/group/jodconverter/browse_thread/thread/cdf6600288bfba5a/8ed4558cfde08e39

Embedded Glassfish v3: deploying sun-resources.xml programmatically fails

I would like to be able to package my jpa-ejb-web project as a standalone application, by using Glassfish embedded API.
To use the JPA layer, I need to deploy the sun-resource.xml configuration, which should be possible with the asadmin command add-resources path\to\sun-resources.xml. I've this code to do it:
String command = "add-resources";
ParameterMap params = new ParameterMap();
params.add("", "...\sun-resources.xml" );
CommandRunner runner = server.getHabitat().getComponent(CommandRunner.class);
ActionReport report = server.getHabitat().getComponent(ActionReport.class);
runner.getCommandInvocation(command, report).parameters(params).execute();
but Glassfish refuses it with:
15-Jul-2010 16:34:12 org.glassfish.admin.cli.resources.AddResources execute
SEVERE: Something went wrong in add-resources
java.lang.Exception: ...\gfembed6930201441546233570tmp\lib\dtds\sun-resources_1_4.dtd (The system cannot find the path specified)
at org.glassfish.admin.cli.resources.ResourcesXMLParser.initProperties(ResourcesXMLParser.java:163)
at org.glassfish.admin.cli.resources.ResourcesXMLParser.<init>(ResourcesXMLParser.java:109)
at org.glassfish.admin.cli.resources.ResourcesManager.createResources(ResourcesManager.java:67)
at org.glassfish.admin.cli.resources.AddResources.execute(AddResources.java:106)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$1.execute(CommandRunnerImpl.java:305)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:320)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1176)
at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$900(CommandRunnerImpl.java:83)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1235)
at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1224)
at javaapplication4.Main.main(Main.java:55)
and indeed, there is no lib directory on the indicated path ...
is there something wrong in my code? (I use glassfish-embedded-all-3.0.1.jar)
Thanks
I solved it by specifying an Embedded File System for the embedded Glassfish, and prepopulated the /path/to/my/glassfish/lib/dtds folder with the files which were missing.
EmbeddedFileSystem.Builder efsb = new EmbeddedFileSystem.Builder();
efsb.autoDelete(false);
efsb.installRoot(new File("/path/to/my/glassfish"), true);
EmbeddedFileSystem efs = efsb.build();
Server.Builder builder = new Server.Builder("test");
builder.embeddedFileSystem(efs);
builder.logger(true);
Server server = builder.build();
server.addContainer(ContainerBuilder.Type.all);
server.start();
and asking Glassfish not to delete the folder at the end of the execution.
I'm not sure this is possible, Running asadmin Commands Using the Sun GlassFish Embedded Server API doesn't mention such a use case (passing a sun-resources.xml).
But I would use a preconfigured domain.xml instead of trying to deploy a sun-resource.xml file, the result should be similar. From the Sun GlassFish Enterprise Server v3 Embedded Server Guide:
Using an Existing domain.xml File
Using an existing domain.xml file
avoids the need to configure embedded
Enterprise Server programmatically in
your application. Your application
obtains domain configuration data from
an existing domain.xml file. You can
create this file by using the
administrative interfaces of an
installation of nonembedded Enterprise
Server. To specify an existing
domain.xml file, invoke the
installRoot, instanceRoot, or
configurationFile method of the
EmbeddedFileSystem.Builder class or
a combination of these methods.
The documentation provides code samples showing how to do this (should be pretty straightforward).

Resources