logstash to receive log from android? or is this elasticsearch? - elasticsearch

I'm still a bit confused after reading documentation provided by logstash. I'm planning on writing an Android app, and I want to log the activity of the app. Logs will be sent over the network. is logstash not the right solution? because it needs to have an "agent" installed on systems that produces log.
I want a system that can store log from the app activity, but it also needs to be able to export the collected logs into plain text file. I know logstash can output to elasticsearch, but i'm not sure if it can export to plaintext file at the same time. or is this a task that ElasticSearch should do?
thanks a ton for any input you can provide

Logstash forwarder isn't currently available for android/ios unfortunately, nor could I find any existing solution for it from the community. (I asked the same question here but was voted off-topic because it was deemed asking for tool/library suggestions).
Your best bet unfortunately is either to write one yourself (which isn't trivial: you'll need to factor in offline connectivity, batching, scheduling, compressions, file-tracking, and so on), or to use other (usually commercial) logging services such as LogEntries.
By the way, the android/ios clients for LogEntries is open source. I'm not clear on its OSS licensing, but if you're to write an agent for logstash yourself, you could perhaps start by looking at LogEntries' android agent implementation, which already solves all the technical problems mentioned above. https://github.com/logentries/le_android.
And to answer your other question, yes logstash should receive your log (from the mobile-device), usually via lumberjack input (aka logstash forwarder). Logstash can then persist & index these log files to elasticsearch, providing it's configured that way

Related

How do I instrument my code for Splunk metrics?

I'm brand new to Splunk, having worked exclusively with Prometheus before. The one obvious thing I can't see from looking at the Splunk website is how in my code, I create/expose a metric... if I must provide an HTTP endpoint for consumption, or call into some API to push values, etc. Further, I cannot see which languages Splunk provide libraries for, in order to aid instrumentation - I cannot see where all this low level stuff is documented!
Can anyone help me understand how Splunk works, particularly how it compares to Prometheus?
Usually, programs write their normal log files and Splunk ingests those files so they can be searched and data extracted.
There are other ways to get data into Splunk, though. See https://dev.splunk.com/enterprise/reference for the SDKs available in a few languages.
You could write your metrics to collectd and then send them to Splunk. See https://splunkonbigdata.com/2020/05/09/metrics-data-collection-via-collectd-part-2/
You could write your metrics directly to Splunk using their HTTP Event Collector (HEC). See https://dev.splunk.com/enterprise/docs/devtools/httpeventcollector/

What would be the advantages of using ELK for log management over a simple python logging + existing database log table combo?

Assuming I have many Python processes running on an automation server such as Jenkins, let's say I want to use Python's native logging module and, other than writing to the Jenkins console or to a log file, I want to store & centralize the logs somewhere.
I thought of using ELK for that, but then I realized that I can just as well create a dedicated log table in an existing database (I'm using Redshift), use something like Grafana for log dashboards/visualization and save myself the trouble of deploying a new system (most of the people in my team are familiar with Redshift but not with ElasticSearch).
Although it sounds straightforward, I feel like I'm not looking at the big picture and that I would be missing some powerful capabilities that components like Logstash were written for the in the first place. What would these capabilities be and how would it be advantageous to use ELK instead of my solution?
Thank you!
I have implemented a full ELK stack in my company in the past year.
The project was huge and took a lot of time to properly implement. The advantages of using ELK and not implementing our own centralized logging solution would be:
Not needing to re-invent the wheel- There is already a product that is doing just that. (and the installation part is extremely easy)
It is battle tested and can stand huge amount of logs in a short time.
As your business and product grows and shift you will need to parse more logs with different structure which will mean DB changes on self built system. logstash will give you endless possibilities of filtering and parsing those new formatted logs.
It has Cluster and HA capabilities, and you can scale your logging system vertically and horizontally.
Very easy to maintain and change over time.
It can send the needed output to a variety of products including Zabbix, Grafana, elasticsearch and many more.
Kibana will give you ability to view the logs, build graphs and dashboards, alerts and more...
The options with ELK are really endless and the more I work with it, the more I find new ways it can help me. not just from viewing logs on distributed remote server systems, but also security alerts and SLA graphs and many other insights.

CloudWatch single Log Stream for Fargate

I have an ECS Fargate application which log are saved in cloudwatch using awslog driver.
Logging works very well, the only annoying thing is that each container creates a different log stream, which name is, as explained on the documentation here
prefix-name/container-name/ecs-task-id
I make an extensive use of autoscaling, creating a lot of tasks, which in turn produce a lot of log streams.
I was wondering if it's possible to have all the logs into the same log stream, that would help me a lot but it looks like it's not possible off the shelf. How could I achieve my goal ?
This is not a solution to your usecase, but a workaround you could use is to just search on your log group instead of going into your task-id. You can also use range queries on a log group so this ends up providing pretty much the same thing as going into the specific log stream of the each task-id. Each line of the log in the log group also has a link to the task specific logs stream.
Another thing you could try to do is using elastic search to maintain your logs. Querying on elastic search is extremely easy (it comes built in with kibana which is a pretty powerful off the shelf filtering tool.

How to log events from Raspberry Pi running on Windows 10 IoT to HDFS using Flume

I want to log my LED blinking (On/OFF) and different room temperature to HDFS. I found this http://flume.apache.org/FlumeDeveloperGuide.html.
It says "Flume currently supports Avro, log4j, syslog, and Http POST (with a JSON body) as ways to transfer data from a external source".
My question here is what should I use if I want to log my Raspberry Pi generated data to HDFS. Please provide me some tutorial link or guide me. OR Please let me know the best way to do this? Hope I have asked my question simple.
Disclaimer - I am not familiar with Flume.
Just reading your statement about what Flume supports for logging I think your only option is HTTP POST with JSON. I say this because:
log4j is a Java logging utility. There is a .Net version of this but I a) don't know if it will run in a universal app and b) likely does not have an extension to support Flume logging.
syslog is a Linux logging mechanism that does not exist on Win IoT
Avro I am not at all familiar with but likely does not have a Win IoT version.
That leaves you with HTTP POST. .Net on Win IoT has classes that will allow you to accomplish this reasonably easily. Take a look at Windows.Web.HttpClient class's postAsync method. I've not tried it personally.
Mark Radbourne [MSFT]

Collect event logs remotelly

In your opinion, which is the best approach to collect the event logs remotely from several Windows machines in a network?
I need to collect the log events remotely and I have several approach (WMI, EventLog class, etc.) but I don't know what is the best way.
Can you help me?
Thanks
EDIT: Are you programming the remote event log access into an app? Maybe you can elaborate on that. If so, what language are you programming in, etc.
Check out OSSEC, perhaps in concert with Logstash and ElasticSearch.
Or you could look at Wevutil, pull event log data to a management workstation then push it into a database.
There's also PsLogList from the sysinternals guys, which you could also use to pull the event log data, then push it into a database.
In my opinion the best way to do so would be configure redis, rabbitMQ or ZeroMQ (well supported plugins) and send all your logs to a queue server from where your logstash indexer will pickup all the logs and processes it.
In this way all your logs will be on central server, which you can persist as well with messaging systems I mentioned above. All your existing system will be same and do not required additional package except a simple client to push to the messaging queue.
http://logstash.net/docs/1.4.0/

Resources