NLog/Elasticsearch writes to DataStream not index - elasticsearch

I have NLog configuration that pushes data to Elastic. When the index pattern is static everything works fine. When I change it to be time-based, I can see messages coming to datastream (with the name of the index from configuration), but this datastream is creating .ds-* index which does not follow the name from the pattern. Therefore logs are not visible in Kibana. I have action.auto_create_index even set to true. What am I missing to have this working properly?
Here is NLog.config part related to elastic:
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true">
<!-- enable asp.net core layout renderers -->
<extensions>
<add assembly="NLog.Web.AspNetCore"/>
<add assembly="NLog.Targets.ElasticSearch"/>
<add assembly="NLog.StructuredLogging.Json"/>
</extensions>
<variable name="AspData" value="${when:when=length('${aspnet-request-url}') > 0:inner=|url\: ${aspnet-request-url}}${when:when=length('${aspnet-mvc-action}') > 0:inner=|action\: ${aspnet-mvc-action}}" />
<variable name="ExceptionLayout" value="${onexception:${newline}EXCEPTION - ${exception:format=type,message,method:maxInnerExceptionLevel=5:innerFormat=shortType,message,method}}" />
<variable name="LoggingLayout" value="${longdate}|EV[${event-properties:item=EventId_Id}]|TH[${threadname:whenEmpty=${threadid}}]|${mdc:userId}|${uppercase:${level}}|${logger}|${message} ${AspData} ${ExceptionLayout}" />
<!-- the targets to write to -->
<targets>
<!--<default-wrapper xsi:type="BufferingWrapper" bufferSize="100"/>-->
<!-- write logs to file -->
<target name="asyncElastic" xsi:type="AsyncWrapper">
<target name="elasticLog" xsi:type="ElasticSearch" index="logs-${configsetting:item=ElasticLog.ServiceName}-${date:format=yyyy-MM-dd}"
uri="${configsetting:item=ElasticLog.ServerUrl}" requireAuth="true" username="${configsetting:item=ElasticLog.User}" password="${configsetting:item=ElasticLog.Password}"
layout ="${LoggingLayout}" >
<field name="data" layout="${structuredlogging.json}" layoutType="System.Object" />
<field name="serviceName" layout="${configsetting:item=ElasticLog.ServiceName}" layoutType="System.String" />
<field name="userId" layout="${mdc:userId}" layoutType="System.String" />
</target>
</target>
</targets>
<!-- rules to map from logger name to target -->
<rules>
<!--Skip non-critical Microsoft logs and so log only own logs-->
<logger name="Microsoft.EntityFrameworkCore.Database.Command" maxlevel="Debug" final="true" />
<logger name="Microsoft.EntityFrameworkCore.Database.Command" maxlevel="Warning" writeTo="asyncElastic" final="true" />
<logger name="Elastic.Apm" maxlevel="Info" final="true" />
<logger name="*" minlevel="Trace" writeTo="asyncElastic" />
</rules>
</nlog>
Edit:
After further investigation I found out that default Kibana installation has index template called logs with pattern logs-*-*. This template treats this indice with that pattern as datastream and not index. So if I wanted to have indexes all I had to do is change my index name to not follow that pattern.

Related

NLog is not logging on Oracle database, Please how can I get it right

I want to log into Oracle database table using NLog but nothing gets logged. Below is NLog.config file am using. Please what am I doing wrongly?
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
autoReload="true"
throwConfigExceptions="true"
internalLogLevel="info"
internalLogFile="Logtxt.txt">
<extensions>
<add assembly="NLog.Web.AspNetCore"/>
</extensions>
<targets>
<target name="database" xsi:type="Database" keepConnection="false" useTransactions="true"
dbProvider="System.Data.OracleClient.OracleConnection,System.Data.OracleClient, Version=1.0.5000.0, Culture=neutral, PublicKeyToken=b77a5c561934e089"
connectionString="Data Source=#####;User ID=######;password=******;Integrated Security=no;"
commandText="insert into DBO_ERRORLOGS(LOGID,LEVEL,CALLSITE,TYPE,MESSAGE,STACKTRACE,INNEREXCEPTION,ADDITINALINFO,LOGGEDONDATE,USERNAME) values('',:LEVEL,:CALLSITE,:TYPE,:MESSAGE,:STACKTRACE,:INNEREXCEPTION,:ADDITIONALINFO,:DATE,:USERNAME)">
<parameter name="LEVEL" layout="${Level}" />
<parameter name="CALLSITE" layout="${Callsite}" />
<parameter name="DATE" layout="${date}" />
<parameter name="TYPE" layout="${exception:format=Type}" />
<parameter name="MESSAGE" layout="${exception:format=Message}" />
<parameter name="STACKTRACE" layout="${exception:format=StackTrace}" />
<parameter name="INNEREXCEPTION" layout="${exception:format=:innerFormat=ShortType,Message,Method:MaxInnerExceptionLevel=1:InnerExceptionSeparator=}" />
<parameter name="ADDITIONALINFO" layout="${Message}" />
<parameter name="USERNAME" layout="${identity}" />
</target>
</targets>
<rules>
<logger name="*" minlevel="Debug" writeTo="database" />
</rules>
</nlog>
Without an error message (see below how to find it), it's a bit guessing. But I think it's an issue you are sending all the parameters as strings. So add the dbType to the parameters, like:
<parameter name="DATE" layout="${date}" dbType="DateTime" />
For the actual error, you should check the internal log. It should be there!
Check Logtxt.txt. See internalLogFile="Logtxt.txt". And you can't find it, change it to an absolute path, e.g. internalLogFile="C:\temp\Logtxt.txt"

How to set current time (-zone) for Logging?

I use for logging Microsoft.Extensions.Logging and NLog. The logs are written to Elasticsearch. When I look up my logs using Kibana all my logs are written at UT what means two houers different to my local time (-zone) MET. I would like to see my logs with my local time at Kibana. How can I adjust the time zone for logging?
I add the NLog.config:
<nlog autoReload="true" throwExceptions="false"
internalLogLevel="Info" internalLogFile="NLogError.log"
xmlns="http://www.nlog-project.org/schemas/NLog.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
<targets>
<target name="Logfile" xsi:type="File"
fileName="${basedir}dap.log"
layout="${longdate} ${level} ${callsite} -> ${message} ${exception:format=Message,StackTrace}"
archiveFileName="${basedir}/archives/log.{#}.log"
archiveEvery="Day"
archiveNumbering="Rolling"
maxArchiveFiles="14"
keepFileOpen="false"
/>
<target xsi:type="Network"
name="CentralLog"
newLine ="false"
maxMessageSize="65000"
connectionCacheSize="5"
encoding="utf-8"
keepConnection="false"
maxQueueSize="100"
address="tcp://myurl.org:5544"
onOverflow="Split">
<layout type="JsonLayout">
<attribute name="machinename" layout="${machinename}" />
<attribute name="level" layout="${level:upperCase=true}" />
<attribute name="processname" layout="${processname}" />
<attribute name="processid" layout="${processid}" />
<attribute name="sendertimestamp" layout="${date:universalTime=true:format=yyyy-MM-ddTHH\:mm\:ss.fff}" />
<attribute name="module" layout="dhp DataPickerApi ${logger}" />
<attribute name="message" layout="${message}" />
<attribute name="exception" layout="${exception}" />
<attribute name="activityId" layout="${activityId}" />
</layout>
</target>
</targets>
<rules>
<logger name="*" minlevel="Trace" writeTo="Logfile" />
<logger name="*" minlevel="Trace" writeTo="CentralLog"/>
</rules>
</nlog>
Don't use universalTime=true?
So instead of this:
<attribute name="sendertimestamp"
layout="${date:universalTime=true:format=yyyy-MM-ddTHH\:mm\:ss.fff}" />
Use this:
<attribute name="sendertimestamp"
layout="${date:format=yyyy-MM-ddTHH\:mm\:ss.fff}" />
You can set the timezone in the Kibana web.
Setting-->Advanced---> DateFormat:tz
to get more info : https://www.elastic.co/guide/en/kibana/current/advanced-options.html

Infinispan/Wildfly: Cannot find a parser for element 'string-keyed-jdbc-store' in namespace

I am trying to configure a JDBC-backed Infinispan cache for storing Hibernate Search indices on Wildfly 10/JBoss EAP 7. I have installed the Infinispan 8.1.x Wildfly/JBoss modules from infinispan.org within my application server, and I've FINALLY gotten a basic infinispan.xml file working with file persistence, and now I am trying to change it over to use JDBC for persistence. My infinispan.xml file looks like this:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:8.1 http://www.infinispan.org/schemas/infinispan-config-8.1.xsd
urn:infinispan:config:jdbc:8.0 http://www.infinispan.org/schemas/infinispan-cachestore-jdbc-config-8.0.xsd"
xmlns="urn:infinispan:config:8.1" xmlns:jdbc="urn:infinispan:config:store:jdbc:8.0">
<jgroups>
<stack-file name="default-jgroups-udp" path="default-configs/default-jgroups-udp.xml" />
</jgroups>
<cache-container name="HibernateSearch" default-cache="default"
statistics="false" shutdown-hook="DONT_REGISTER">
<transport stack="default-jgroups-udp" />
<jmx duplicate-domains="true" />
<!-- *************************************** -->
<!-- Cache to store Lucene's file metadata -->
<!-- *************************************** -->
<replicated-cache name="LuceneIndexesMetadata" mode="SYNC"
remote-timeout="25000">
<locking striping="false" acquire-timeout="10000"
concurrency-level="500" write-skew="false" />
<transaction mode="NONE" />
<eviction max-entries="-1" strategy="NONE" />
<expiration max-idle="-1" />
<persistence passivation="false">
<file-store preload="true" purge="true"
path="${jboss.server.temp.dir}/indexes" />
</persistence>
<indexing index="NONE" />
<state-transfer enabled="true" timeout="480000"
await-initial-transfer="true" />
</replicated-cache>
<!-- **************************** -->
<!-- Cache to store Lucene data -->
<!-- **************************** -->
<replicated-cache name="LuceneIndexesData" mode="SYNC"
remote-timeout="25000">
<locking striping="false" acquire-timeout="10000"
concurrency-level="500" write-skew="false" />
<transaction mode="NONE" />
<eviction max-entries="-1" strategy="NONE" />
<expiration max-idle="-1" />
<persistence passivation="false">
<string-keyed-jdbc-store xmlns="urn:infinispan:config:store:jdbc:8.0"
fetch-state="false" read-only="false" purge="false">
<data-source jndi-url="java:jboss/datasources/InfinispanCacheDS" />
<string-keyed-table drop-on-exit="false"
create-on-start="true" prefix="ISPN_STRING_TABLE">
<id-column name="ID_COLUMN" type="VARCHAR(255)" />
<data-column name="DATA_COLUMN" type="BINARY" />
<timestamp-column name="TIMESTAMP_COLUMN" type="BIGINT" />
</string-keyed-table>
</string-keyed-jdbc-store>
</persistence>
<!-- <persistence passivation="false"> <file-store preload="true" purge="true"
path="${jboss.server.temp.dir}/indexes" /> </persistence> -->
<indexing index="NONE" />
<state-transfer enabled="true" timeout="480000"
await-initial-transfer="true" />
</replicated-cache>
<!-- ***************************** -->
<!-- Cache to store Lucene locks -->
<!-- ***************************** -->
<replicated-cache name="LuceneIndexesLocking" mode="SYNC"
remote-timeout="25000">
<locking striping="false" acquire-timeout="10000"
concurrency-level="500" write-skew="false" />
<transaction mode="NONE" />
<eviction max-entries="-1" strategy="NONE" />
<expiration max-idle="-1" />
<persistence passivation="false">
<file-store preload="true" purge="true"
path="${jboss.server.temp.dir}/indexes" />
</persistence>
<indexing index="NONE" />
<state-transfer enabled="true" timeout="480000"
await-initial-transfer="true" />
</replicated-cache>
</cache-container>
</infinispan>
I am also using the following jboss-deployment-structure.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure xmlns="urn:jboss:deployment-structure:1.2">
<deployment>
<dependencies>
<!-- Core Java/Java EE Dependencies -->
<module name="javax.transaction.api" export="true" services="import" />
<!-- JPA/Hibernate Search Dependencies -->
<module name="javax.persistence.api" export="true" />
<module name="org.hibernate.search.engine" export="true"
services="import" slot="5.5" />
<module name="org.hibernate.search.orm" export="true"
services="import" slot="5.5" />
<module name="org.infinispan.hibernate-search.directory-provider"
slot="for-hibernatesearch-5.5" export="true" services="import" /><!--
<module name="org.apache.lucene" export="true" services="import" /> <module
name="org.infinispan.lucene-directory" slot="ispn-8.1" /> -->
<module name="org.infinispan.core" slot="ispn-8.1" services="import"
export="true" />
<module name="org.infinispan.persistence.jdbc" slot="ispn-8.1"
services="import" export="true" />
<module name="org.infinispan.persistence.remote" slot="ispn-8.1"
services="import" export="true" />
</dependencies>
</deployment>
</jboss-deployment-structure>
However, when I deploy my application I get the following exception:
ISPN000327: Cannot find a parser for element 'string-keyed-jdbc-store' in namespace 'urn:infinispan:config:store:jdbc:8.0'. Check that your configuration is up-to date for this version of Infinispan.
Everything works fine if I change persistence from JDBC to "file". However, once I put in the JDBC parameters I get the parsing error. I'm sure I'm doing something minor and stupid to cause this, but I can't for the life of me figure out what. Any help would be GREATLY appreciated!!
After trying to investigate this problem I ended opening a bug report:
https://issues.jboss.org/browse/ISPN-7572
The problem is that I expected the org.infinispan.core module to depend on its extensions, but this is not the case. I previously mentioned a wokraround, but editing the module XML files is not enough as there's no easy solution which would be able to maintain the isolation of different modules which need to be isolated.
The problem is fixed now, please upgrade to either Infinispan version 9.0.0.Final which will contain the fix, or get Infinispan 8.2.7.Final as I'll backport this fix as well.

Enunciate framework - Not working with Spring Restful project

I have integrated enunciate framework to generate the API document for the Spring RESTful project. I have followed the steps from https://github.com/stoicflame/enunciate/wiki/Executables and deployed the war created from the enunciate configuration in the tomcat server(http://localhost:8080/sample_enunciate) but its displaying the empty document. Here I have provided the configuration details used in the sample project.
NOTE: But the similar configuration is working with Jersey restful project. I really stuck here. Please let me know, is this bug with the enunciate framework integration with Spring project. Thanks in advance.
Project configuration:
java -1.7.0
tomcat -6.0 &7.0
ant -1.9.4
spring -4.0.5
enunciate -1.30
jars:
enunciate-core-1.30-RC1.jar
enunciate-core-annotations-1.30-RC1.j
enunciate-core-rt-1.30-RC1.jar
enunciate-java-client-1.30-RC1.jar
enunciate-docs-1.30-RC1.jar
enunciate-rt-1.30-RC1.jar
enunciate-spring-app-rt-1.30-RC1.jar
enunciate-spring-jaxws-rt-1.30-RC1.ja
spring-aop-4.0.5.RELEASE.jar
spring-beans-4.0.5.RELEASE.jar
spring-context-4.0.5.RELEASE.jar
spring-context-support-2.5.4.jar
spring-core-4.0.5.RELEASE.jar
spring-expression-4.0.5.RELEASE.jar
spring-jdbc-4.0.5.RELEASE.jar
spring-test-4.0.5.RELEASE.jar
spring-tx-4.0.5.RELEASE.jar
spring-web-4.0.5.RELEASE.jar
spring-webmvc-4.0.5.RELEASE.jar
This is my enunciate.xml.
enunciate.xml
<?xml version="1.0"?>
<api-classes>
<include pattern="com.sample.controller.*" />
</api-classes>
<modules>
<!-- Docs -->
<docs title="example" copyright="Example.com"/>
<webapp mergeWebXML="WebContent/WEB-INF/web.xml" />
<spring-app disabled="false" springVersion="4.0.5">
<springImport file="resources/dev/applicationContext.xml" />
<springImport file="WebContent/WEB-INF/rest-servlet.xml" />
</spring-app>
<c disabled="true" />
<csharp disabled="true" />
<java-client disabled="false" />
<cxf disabled="false" />
<gwt disabled="false" />
<jaxws-client disabled="true" />
<jaxws-ri disabled="true" />
<jaxws-support disabled="true" />
<jersey disabled="true" />
<xml disabled="false" />
<obj-c disabled="true" />
<rest disabled="false" />
</modules>
properties file for build.xml
enunciate_build.properties
JAVA_HOME=C:/Java/jdk1.7.0/
tomcat.home=D:/xampp/tomcat
This is my build.xml
build.xml
<?xml version="1.0" encoding="UTF-8"?>
<project default = "enunciate">
<property file ="enunciate_build.properties"/>
<property name="lib.dir" value="../libs" />
<property name="src.dir" value="src"/>
<target name = "enunciate">
<path id= "enunciate.classpath">
<fileset dir = "${lib.dir}">
<include name="*.jar"/>
</fileset>
<fileset dir ="${lib.dir}/modules/spring">
<include name="*.jar"/>
</fileset>
<fileset dir = "${JAVA_HOME}">
<include name = "lib/tools.jar"/>
</fileset>
</path>
<taskdef name="enunciate" classname = "org.codehaus.enunciate.main.EnunciateTask">
<classpath refid = "enunciate.classpath"/>
</taskdef>
<enunciate javacSourceVersion="1.7" javacTargetVersion="1.7" basedir = "${src.dir}" configFile="enunciate.xml">
<include name = "**/*.java"/>
<classpath refid= "enunciate.classpath"/>
<export artifactId="war.file" destination="${tomcat.home}/webapps/sample_enunciate.war"/>
</enunciate>
</target>
</project>
Enunciate only recognizes JAX-RS annotations right now. For further information follow this link.https://github.com/stoicflame/enunciate/issues/60

Nesper adding event types to my winforms app.config

I'm interested in defining multiple event-types my app.config file but it doesn't appear to get loaded by default. Is there something that I'm doing wrong? The event type doesn't exist within com.espertech.esper.client.Configuration.
<configSections>
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler,Log4net" />
<section name="esper-configuration" type="com.espertech.esper.util.EsperSectionHandler,NEsper" />
</configSections>
<esper-configuration>
<engine-settings>
<defaults>
<threading>
<listener-dispatch preserve-order="false" timeout-msec="2000" locking="suspend" />
<insert-into-dispatch preserve-order="false" timeout-msec="3000" locking="suspend" />
<internal-timer enabled="false" msec-resolution="1234567" />
<thread-local style="fast" />
</threading>
<event-meta>
<class-property-resolution style="distinct_case_insensitive" />
</event-meta>
<view-resources>
<share-views enabled="false" />
</view-resources>
<logging>
<execution-path enabled="true" />
</logging>
<variables>
<msec-version-release value="30000" />
</variables>
</defaults>
</engine-settings>
<event-type name="Products" class="ProtoProduct"/>
<event-type name="MarketDepths" class="ProtoDepth"/>
<event-type name="MarketTrades" class="ProtoTrade"/>
<event-type name="Orders" class="ProtoOrder"/>
<event-type name="Positions" class="ProtoPosition"/>
<auto-import import-name="org.mycompany.mypackage.MyUtility"/>
<auto-import import-name="org.mycompany.util.*"/>
</esper-configuration>
The most likely issue is that you haven't used the fully qualified name of the class. In your examples, the classes have no namespace. If your classes are in a namespace add those to the class attribute in your config. If for some reason that isn't the issue, it is most likely that the tips are not visible within the AppDomain. Just make sure they are built into your assembly.

Resources