Using ExtractText for getting log data in nifi - apache-nifi

Question

I have retrieved custom log data in tailFail and then split the data(line by line). Now I want to get useful data from nifi-api.log.
I used this expression like this:
^(.*)$
but processor make flowfiele unmatched.
1. how should i replace my expression?

Answer 1

It depends on what information you are looking for in the log messages. The expression you've posted simply matches the entire content.
Let's say you have the following log output and want to collect the times for flowfile repository checkpointing to do analysis:
2017-08-25 10:36:31,942 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 229 milliseconds
2017-08-25 10:36:35,571 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#6527aa0 checkpointed with 0 Records and 0 Swap Files in 14 milliseconds (Stop-the-world time = 4 milliseconds, Clear Edit Logs time = 7 millis), max Transaction ID -1
2017-08-25 10:38:31,942 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2017-08-25 10:38:32,162 INFO [pool-10-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#6cca70e3 checkpointed with 0 Records and 0 Swap Files in 218 milliseconds (Stop-the-world time = 92 milliseconds, Clear Edit Logs time = 98 millis), max Transaction ID -1
2017-08-25 10:38:32,162 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 218 milliseconds
2017-08-25 10:38:35,584 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#6527aa0 checkpointed with 0 Records and 0 Swap Files in 13 milliseconds (Stop-the-world time = 6 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID -1
2017-08-25 10:40:32,161 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Initiating checkpoint of FlowFile Repository
2017-08-25 10:40:32,341 INFO [pool-10-thread-1] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#6cca70e3 checkpointed with 0 Records and 0 Swap Files in 177 milliseconds (Stop-the-world time = 71 milliseconds, Clear Edit Logs time = 87 millis), max Transaction ID -1
2017-08-25 10:40:32,341 INFO [pool-10-thread-1] o.a.n.c.r.WriteAheadFlowFileRepository Successfully checkpointed FlowFile Repository with 0 records in 178 milliseconds
2017-08-25 10:40:35,592 INFO [Write-Ahead Local State Provider Maintenance] org.wali.MinimalLockingWriteAheadLog org.wali.MinimalLockingWriteAheadLog#6527aa0 checkpointed with 0 Records and 0 Swap Files in 11 milliseconds (Stop-the-world time = 5 milliseconds, Clear Edit Logs time = 4 millis), max Transaction ID -1
Using an expression like ^[\d\-\s\:,]+\s(INFO|WARN|ERROR).*(\d+) milliseconds would allow you to filter those messages and with your capture groups, understand the severity of the message and the timing.

Answer 2

You can use following regex in extractText processor for extracts value.
regex:(.*)
Then use RouteOnAttribute to check that log to be ERROR/WARN/INFO by below expressions.
INFO:${regex:toLower():contains('info')}
ERROR:${regex:toLower():contains('error')}
WARN:${regex:toLower():contains('warn')}
Now route your flowfiles as per attributes and then do whatever you wants.
Hope this helpful for you