Is there way how we can enable debug for logs in apache storm. Not at the topology level but for Master node(nimbus.log). Wanted to enable DEBUG level for nimbus.log to understand how exactly scheduling works.
I have already gone through dynamic logging using UI for topologies.
The non-worker logging is configured in the storm/log4j2/cluster.xml file https://github.com/apache/storm/blob/master/log4j2/cluster.xml#L86. This is a standard Log4j2 configuration file, so refer to the Log4j2 documentation for how this works.
You should be able to just add a new logger at the bottom there for the package(s) you want logs from, and set the level to DEBUG.
Related
I have a bug that occurs in my Storm topology during initialization. I would like to set the log level to DEBUG from when the topology is started.
I realize there is a mechanism to dynamically set the log level for a running topology using either the Storm UI or CLI, but I am not able to dynamically change this setting before the bug occurs in my topology during initialization.
How can I statically set the log level to DEBUG so that I can see more detailed logs when my topology is initialized?
The following only applies to Storm 2.0.0 and later.
You can include a log4j2 config file in your topology jar. You then need to set the topology.logging.config property in your topology configuration.
I'll include the documentation here for convenience:
Log file the user can use to configure Log4j2. Can be a resource in the jar (specified with classpath:/path/to/resource) or a file. This configuration is applied in addition to the regular worker log4j2 configuration. The configs are merged according to the rules here: https://logging.apache.org/log4j/2.x/manual/configuration.html#CompositeConfiguration
See https://github.com/apache/storm/blob/885ca981fc52bda6552be854c7e4af9c7a451cd2/storm-client/src/jvm/org/apache/storm/Config.java#L735
The "regular worker log4j2 configuration" is the log4j2/worker.xml file in your Storm deployment, assuming default settings.
OK, I spent quiet some time figuring out how to configure stuff to have DataDog trace ID in logs but couldn't get it working. To be clear what I'm looking for is to see trace IDs in logs message, the same way that adding spring-cloud-starter-sleuth to the classpath, automatically configure Slf4j/Logback to show trace IDs in log messages.
Where I've started:
We've got a simple web spring boot application running as a Docker container deployed as an AWS Elastic BeansTalk, whose logs go to CloudWatch and we read them there.
We have DataDog as a Java agent (thus no dependencies in pom.xml)
We have SLF4J/Logback in our dependencies list.
There's no other related depndencies (like dd-trace-ot or any opertracing libs)
What I did so far:
I found on SO that adding opentracing-spring-cloud-starter will add log integration automatically. But I couldn't get it working.
On DD website, it says configuring the pattern is enough to see the IDs, but in our case it didn't work. (is it because we don't have our logs a JSON?). Also, adding dd-trace-ot didn't help.
Notes:
We can't switch to JSON logs.
We can't switch to any other library (e.g. Slueth).
We can't go away from CloudWatch.
Can someone tell me how exactly I need to configure the application to see trace IDs in log messages? Is there any documentation or samples I can look at?
Do you have the ability to add some parameters in the logs sent. From the documentation you should be able to inject the trace id into your logs in a way that Datadog will interpret them.
You can also look at a parser to extract the trace id and span id from the raw log. This documentation should help you out on that.
From the documentation, if you don't have JSON logs, you need to include dd.trace_id and dd.span_id in your formatter:
If your logs are raw formatted, update your formatter to include
dd.trace_id and dd.span_id in your logger configuration:
<Pattern>"%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L -
%X{dd.trace_id:-0} %X{ dd.span_id:-0} - %m%n"</Pattern> ```
So if you add %X{dd.trace_id:-0} %X{ dd.span_id:-0}, it should work.
I hope you can solve this very mysterious puzzle!
After migrating to Java8 and Tomcat8, the logging info on the servers is ok, but there is no logging information on the developer laptops. Adding Spring-boot-starter-log4j2 there is logging info on the laptops but NOT on the servers. How come? How to fix?
In the parent.pom both the dependencies slf4j-api, the slf4j-log4j binding and the 'slf4j : jcl-over-slf4j' are used. In the project there are no log4j.properites files. In all places where spring boot is used, the spring-boot-starter-logging is excluded.
Scenario 1:
On laptops - the SLF4J combination produces NO logging. I guess that is because there are no log4j.properties files. On the servers this combination shows logging. How can this be? Can there be a provided Tomcat configuration that allows this logging?
Scenario 2:
When I add the spring-boot-starter-log4j2, then I get logging on my laptop. On the server there is no logging at all.
You can define general logging settings in your tomcat environment. This is for example shown in this article.
In short: (1) add an extra library and (2) add a default log4j.properties file.
I have a Spring boot application which use logback.xml for logging configurations.I am looking for options to dynamically change log level.
For instance if I have deployed an app with loglevel as ERROR,Let say I want to change this to INFO but I don't want to redeploy/restart my JVM.
Is there any possibility we can configure logback.xml like config server to achieve this
You can configure Logback to Automatically reloading configuration file upon modification
Yes, this is quite possible. Expose a rest endpoint where you supply the className and log level. With slf4j you can get the LoggerContext and change the level.
LoggerContext context = (LoggerContext) LoggerFactory.getILoggerFactory();
context.getLogger(className).setLevel(Level.valueOf(level));
Apache Commons logging and others have similar features.
If you are using spring cloud then you can have this in your yml file
logging:
level:
root: INFO
Then you can change it and refresh the configuration using actuator refresh to fetch new configuration changes no need to restart the service.
Also if you need some sort of UI to do this stuff you can explore the Spring-cloud-dashboard It is pretty cool and uses the features from the actuator to do and show you a lot of stuff not only changing log levels.
I cannot control logging levels for my code in Websphere Liberty Profile server.
I have configured the server.xml on the server not to log hibernate and spring, since my logs will get flooded with activity from those two frameworks. I commonly do this using log4j and it works fine in standalone WAS.
<logging consoleLogLevel="INFO" copySystemStreams="true" traceFormat="ENHANCED" traceSpecification="org.springframework.*=off:com.ibm.ws.*=off:org.hibernate*=off"/>
In Liberty this does not work.
I get the following log when liberty updates the configuration (when I save server.xml with the changes):
[INFO ] TRAS0040I: The configured trace state included the following specifications that do not match any loggers currently registered in the server: org.hibernate*=off:org.springframework.*=off
Basically this message applies to any of my code and any third party code (Spring, Hibernate, etc).
However the traceSpecification levels work fine for the IBM classes, and I'm able to specify *=off, which effectively turns off all logging.
Has anyone experienced this?
IBM's documentation for TRAS0040I seems simple enough, but I can't seem to figure out why my loggers are not getting registered with the server.
Liberty doesn't have a rich control on logging. You should understand the difference between "logging" and "tracing". Check description of console.log, messages.log, trace.log files at the beggining of: http://www.ibm.com/support/knowledgecenter/SSEQTP_8.5.5/com.ibm.websphere.wlp.doc/ae/rwlp_logging.html
Your configuration in "traceSpecification" - actually will do nothing, as spring and hibernate are logs from JVM and they doesn't go to trace, so trace configuration doesn't affect them.
All you can configure in liberty for jvm logs is consoleLogLevel (INFO, AUDIT, WARNING, ERROR, and OFF)
If you want to configure log levels for specific components in Liberty - you should use for example log4j with own configuration