This question is for asking general advice on what tools should I use for this task, and possibly pointing me to some related tutorials.
I have a Spring Boot web application, which during operation generates log messages into a database. There is a JavaScript management tool in the making for this REST application, and 1 function of it would be to show the log messages real-time. Meaning, when the user is on the log showing page, he should see the new log messages appearing without refresing the page.
My questions:
What should be used to provide this for the javascript client at some endpoint? I'm looking at these spring boot starters right now: websocket, redis, amqp. I have not used any of these before.
How should I "catch" the log messages as they are generated inside the application? So I could send them to the client with the chosen solution.
I'm not really looking for a periodic query type of solution, but rather a server pushing the data as it appears solution.
Any suggestions and code samples are appreciated.
Storing logs in a database is usually not a good option unless you use a database which is capable of handling a lot of write requests, such as Apache Cassandra. Streaming data from a database is not the most intuitive thing to do, however.
A modern alternative is to use a messaging system such as Apache Kafka to stream logs from producing systems to multiple subscribing systems. There are multiple ways how you can achieve that. For example, for streaming logs from your Spring Boot app you could use a special log4j appender (see here and an example here). To be able to present logs in a web browser in real-time, you will need another backend system which will receive the log records from Kafka topics and forward them to JavaScript web clients via websockets, most likely using a publisher/subscriber model.
Also, you could consider using server sent events (SSE) instead of websockets. Because you have only a unidirected message flow (logs are streamed from a backend system to a javascript client in the browser, but not the other way around), SSE can be a good option as a replacement for websockets. Websockets are more difficult to operate than SSE and usually use more resources at the backend. As always, you will need to choose between trade-offs (see here).
Related
I'm starting to work with NestJS, a Javascript framework for building RESTful api's.
The framework encourages you to work with multilayered architecture, separating controllers, services and repositories.
In this project of mine, I have a RESTful api that talks to my website, and I'm currently in the need of using websockets for several reasons.
The general behavior of a module that will use the websocket protocol (in my app) is:
client makes http request with a message -> backend validates -> backend broadcasts the message through websockets.
Much like a chat application.
The problem is that I'm having a hard time figuring out where is the best place to put these broadcast calls. It seems like every layer is the wrong place.
I won't even consider the repository layer. Talking strictly about whether should I choose the controller or the service, they both seem wrong.
I don't want to send websocket messages in the same place where I handle HTTP requests, and I sure don't want to send them in the same place were I deal with business rules.
I've come to an approach, but I'm not sure if it's necessary to do it. I'm using Redis as caching mechanism, and as a store to Socket.io, so that I can horizontally scale my app and consistently send broadcast messages through sockets. Redis also has a Pub/Sub feature within it, and an awesome notification system called "keyspace notifications" that will publish messages to channels depending on the action performed in the cache memory store. Long story short, HTTP requests changes resources in the backend, these changes are reflected in my Redis cache and, with a (to-be) well crafted key design system, I can listen to the modifications in the cache from another, separate, module and fire the necessary broadcasts.
An illustration of the structure:
Actually, using the correct keywords I found this article in which the author is doing something similar to what I'm proposing here.
What I have?
A lot of different microservices managing by different teams. All microservices persist data in Aerospike database.
What I want to achieve?
I'm building new microservice that relies on data handled by another services. I want to listen the changes in entities, but unfortunately that microservices don't put anything in message queue, they have only usual REST APIs, so I cant just subscribe to events.
The idea is to listen a transaction log(event log/commit log/WAL) of database. This approach is also using in different Event Sourcing systems, but I cant found any Aerospike API that would stream this log. So the question - does Aerospike provide any similar functionality, may be with different name?
Aerospike, in its enterprise edition, has a feature called change notification framework which may fit your requirements. It informs an external agent about all the write operations. This is built over the XDR functionality which is meant for replicating across data centers using a digestlog.
If you are not planning for enterprise, you should reconsider having your own message queue in front of Aerospike.
Currently I am developing an application that exposes file contents especially of log files with spring-boot to a web-app driven by react.
I really like the log view of jenkins and asked myself how they are handling this. Unfortunately I couldn't find the log viewer in the source code.
Can someone please give me a hint how they are updating the file on server and client side or just give me their source?
Custom log view implementation might not be required
Spring boot has a special endpoints called actuator api
One of this endpoints is : /actuator/logfile which is used to view the spring boot log :
curl 'http://localhost:8080/actuator/logfile' -i -X GET
how they are updating the file on server and client side
Update log file in server is not our problem.
A cool view in web is our concern.
I used a node.js implementation. Is ready to use and you could take some ideas:
https://github.com/mthenw/frontail
I think a combination of web sockets, css and a correct file stream operation (open, close) and memory management, could be necessary to reach your goal.
I'm developing an application that must both handle events coming from other systems and provide a REST API. I want to split the applications into micro services and I'm trying to figure out which approach I should use. I drew attention to the Spring Cloud Netflix and the Spring Cloud Data Flow toolkit, but it's not clear to me whether they can be integrated and how.
As an example, suppose we have the following functionality in the system:
1. information about users
card of orders
product catalog
sending various notifications
obtaining information about the orders from third-party systems
processing, filtering, and transformation of order events
processing of various rules based on orders and sending notifications
sending information about user orders from third-party systems to other users using websockets (with pre-filtering)
Point 1-4 - there I see the classical micro service architecture. Framework - Spring Netflix Stack.
Point 5-9 - it's best to use an event-driven approach. Toolkit - Spring Data Flow.
The question is how to build communication between these platforms.
In particular - POPULATE ORDER DETAILS SERVICE must transform the incoming orders and save additional information (in case it needed) in the database. ORDER RULE EXECUTOR SERVICE should obtain information about the current saved rules, execute them and send notifications. WEB SOCKET SERVICE should send orders information only if a particular user has set the filters, and ORDER SAVER SERVICE should store the information about the transformed orders in the database.
1.
Communication between the micro-services within the two platforms could be using the API GATEWAY, but in this case, I have the following questions:
Does the Spring Cloud platform allow to work with micro services that way?
Performance - the number of events is very huge, which can significantly slow down the processing of events. Is it possible to use other approaches, for example, communication not through the API Gateway but through in-memory cache?
2.
Since some functionality intersects between these services, I have a question about what is "microservice" in the understanding of the Spring Cloud Stream framework. In particular, does it make sense to have separate services? Can the microservice in the Spring Cloud Stream have a REST API, work with the database and simultaneously process the events? Does such a diagram make sense and is it possible to build such a stack at the moment?
The question is which of these approaches is more correct? What did Spring Data Streams mean by "microservice"?
Given the limited information in the post, it is hard to convince on all the matters pertaining to this type of architecture, but I'll attempt to share some specifics, and point to samples. Also for the same reasons, it is hard to solve for your needs end-to-end. From the surface, it appears you're attempting to build event-driven applications and wondering whether Spring Cloud Stream (SCSt) and Spring Cloud Data Flow (SCDF) could help.
They can, yes.
The Order, User, and Catalog seem like domain objects and it would all come together to solve for a use-case. For instance, querying for a number of orders for a particular product, and group by the user. There are a few samples that articulate the data flow between the entities to solve similar problems. Here's a live code-walkthrough of event-driven systems in action. There's another example of social-graph application, too.
Though these event-driven applications can run standalone as individual services with the help of of message broker (eg: Kafka or RabbitMQ), you could of course also register them in SCDF and use them in the SCDF DSL to build a coherent data pipeline. We are expanding on more direct capabilities in SCDF for these types of use-cases, but there are ways to orchestrate them today with current abilities, too. Follow spring-cloud/spring-cloud-#2331#issuecomment-406444350 for more details.
I hope this gives an idea. Try to build something small using SCSt/SCDF, prove it out, and expand to more complex use-cases.
I'm using Vertx-STOMP over websockets and I have followed the instructions from the documentation with success.
My question is how is it possible to enable session store in order to utilize it in my application? I cannot find any obvious example.
Am I on the right direction if I try to enable Session with instructions from the vertx-web?
Moreover, is it possible to maintain both stomp server and http server to serve normal RESTful requests under different endpoints, for instance:
WEBSOCKET STOMP via /stomp
and
RESTful API via /api/*
If I understood it correctly you're looking into using your STOMP server to store the session data for your application. If that is the case, you're out of luck since there is currently 2 implementations:
Local Storage (in memory)
Clustered Storage (using the underlying cluster manager)
See here: https://github.com/vert-x3/vertx-web/tree/master/vertx-web/src/main/java/io/vertx/ext/web/sstore
If you really need a custom storage and you're willing to contribute to the open source project I'd say provide an implementation of the interface:
https://github.com/vert-x3/vertx-web/blob/master/vertx-web/src/main/java/io/vertx/ext/web/sstore/SessionStore.java
That uses your STOMP backend. If you're a student, this could be a interesting Google Summer of Code project.