We have been thinking recently about integrating our J2EE system with other applications written in Python/Perl. Our application integrates very nice with other Java systems over JMS. Is it possible that non-java systems will receive Serializable messages and do some modification on it (at some level every class property is java primitive type)? Also we would like to do it in the other direction, e.g. python application constructs object which then will be sent over JMS and modified (at least understandable) by our java app. Do you have any experience in this topic / hints for us?
Thanks in advance,
Piotr
You don't want to use Serializeable objects for this. You'll need a more portable format, such as a text based format like XML or JSON or CSV. It's simply not worth the effort to try and read serialized java object on other platforms.
Now you could use another binary format, such as the Google format (protocol buffers I think it's called). You can also change your java classes, specifically the ones that you plan to exchange, and you can implement the Externalizable interface. This let's you have full control over the reading and writing of your java classes. That way you can still use the java serialization protocol and workflow, but write and read a more portable format.
This let's you incrementally add support for the Python system without really disturbing the rest of the system, especially for messaging, as long as there are no legacy messages to be processed in your queues when you switch over.
Related
As with Spring Integration, a lot of the work in writing a Spring Cloud Stream application lies in configuring the sources and targets of messages, as well as the channels that wire them together. That's also a major part of the work of maintaining and troubleshooting such applications, as well.
I'm sure some people are good at looking at YAML or properties files and visualizing The Matrix in their minds, but I am not.
Is there a way to produce a visualization of the sources, targets, and channels? I'm not talking about a transformed JSON or other formatted text view (in other words, not Actuator); I'm talking about a visual graph that makes it easy to see the connections and flow of messages.
It seems like all of the config data is there and available, has anyone integrated it with something like Graphviz or a Java graphing library? Looking for any experience in doing something like this, or a pre-canned solution.
Spring Integration can be configured to expose the runtime wiring as JSON.
See https://docs.spring.io/spring-integration/docs/current/reference/html/system-management.html#integration-graph
See here https://ordina-jworks.github.io/architecture/2018/01/27/Visualizing-your-Spring-Integration-components-and-flows.html for a discussion about visualization.
Let me preface this with the fact that I am still very new to Apache Camel. I'm still trying to understand how it all works, and what needs to be done (and HOW to do it) to achieve a particular effect.
I am trying to develop a Spring Boot application that will use Apache Camel to handle the transmission (and possibly also receipt) of data to/from a number of possible sources and destinations. The purpose of the application is to provide a means to produce/generate network traffic, at the network application level, that will be fed into another Spring Boot application - let's call this the target. We are trying to observe and measure the effects various network loads have on the target.
We would like to be able to transmit data via a number of protocols, including: ftp, http/s, file systems (nfs), various mail protocols (smtp, pop) and data streaming protocols for voice and video. There may be other protocols added at a later time. The data itself is irrelevant, we just need to be able to transmit data via various protocols with various loads.
These applications/services will be running in a containerized environment (Docker) that will be run within our local development and test environment, as well as possibly in a cloud environment, such as AWS. We have used Docker, Ansible, Terraform and are currently working towards using Kubernetes and Istio to manage the configuration, deployment, and operation of these applications.
We need to be able to provide specific configurations of Camel routes for particular deployments.
It would appear that the preferred method to configure Camel routes is via Java DSL, rather than XML DSL. The Camel documentation and nearly every other source of information I've found have a strong bias towards using Java DSL. Examples of XML DSL route configuration are far and few.
My initial impression is that going the Java DSL route (excuse the pun), would not work well with our need to be able to deploy a Camel application with a specific route configuration. It seems like you are required to have Java DSL defined route configurations hardwired into the code.
We think that it will be easier to provide a specific route configuration via an XML file that can be included in a deployment, hence why I've been trying to investigate and experiment with XML DSL. Perhaps we are mistaken in this regard.
My question to the community is: Considering what I've described above, can the Java DSL approach be used to meet the requirements as I've described them? Can we use Java DSL in a way that allows for dynamic route configuration? Keep in mind we would not be attempting to change configuration during operation, just in the course of performing a deployment.
If Java DSL could be used for this purpose, it would be very much appreciated if pointers to documentation, examples, etc. could be provided.
For your use cases you could use XML DSL also. Anyhow below book covers most aspects Camel development with examples. In this book authors describes XML DSL use for most of java DSL examples.
https://www.manning.com/books/camel-in-action-second-edition
In below github repository you can find the source code for all the examples listed in above book.
https://github.com/camelinaction/camelinaction2
Simple tutorial and github repository for Apache Camel using Spring boot.
https://www.baeldung.com/apache-camel-spring-boot
https://github.com/eugenp/tutorials/tree/master/spring-boot-modules/spring-boot-camel
Maven Plugin for build and deployment of spring boot container application into Kubernetes cluster
https://maven.fabric8.io/
In case if your company can afford some funding for your effort look at below link which provides commercial offerings around Camel.
https://camel.apache.org/manual/latest/commercial-camel-offerings.html
Thanks
Madhu Gupta
Our team has a few projects which use the Java DSL for building routes. In order to make them dynamic, there are control structures for iterating and setting endpoints based off configurations. That works for us because the routes are basically all the same, just with different sources and sinks.
If you could dynamically add/change the XML DSL files in a way that doesn't involve redeploying your application, that might be a viable route to follow. One might, for example, change the camel.springboot.xml-routes property to point to a folder which changes as needed.
Spring boot 2.1.5
Project Reactor 3.2.9
I am setting up a bunch of rest reactive APIs using the above-mentioned frameworks and I am running into an annoying problem with MDC (mapped diagnostic context). My applications are in JAVA.
MDC relies on thread locals to store the current query's mapped context to put in the logs. That system, obviously, is not perfect and contradicts the reactive pattern since the different steps of your execution will be executed through different threads.
I have run into the same problem with the Play Reactive framework but found a workaround there by copying the mapped context transparently from one actor to another.
For spring and reactor, I could not find a satisfying solution yet.
Some random examples found on the internet:
First - It works but forces you to use a bunch of utility methods
Same thing
Second - It tries to copy the context during the onNext publisher event but seems to lose some features on the way of doing that. The signal context, for example, is lost.
I am in need of a proper solution to deal with this:
A library which would make the link between MDC and reactor?
A way to tweak reactor/spring to achieve it transparently?
Any advice?
"I could not find a satisfying solution yet."
Working with contexts is the only solution for the moment. Since as you said threadlocals goes against everything that has to do with reactive programming. Using thread local as a storage point during a request is a resource heavy way of solving things and in my opinion poor design. Unless logging frameworks themselves come up with a better solution to the problem we developers must pass the data through the context to accommodate for the logging frameworks blocking nature.
Reactive programming is a paradigm shift in the programming world. Other things like database drivers, that use threadlocal to rollback transactions are also in big trouble. the JDBC database driver spec is defined as blocking in nature, and atm. there has been attempts by spring and the R2DBC project to define a new JDBC driver spec that is inherently non/blocking. This means that all vendors must rewrite ther database driver implementations from scratch.
Reactive program is so new that lots of libraries need to rewrite entire codebases. The logging frameworks as we know it needs to be rewritten from the ground up which is a huge task. And the context in reactive is actually something that should not even be in reactive programming, it was implemented just to accommodate for MDC problems.
It's actually a lot of overhead needing to pass data from thread to thread.
So what can we do?
push on logging frameworks, and/or help logging frameworks to rewrite their codebase
Accept that there is no "tweak" that will magically fix this
use the context and the way suggested in the blogposts
Project reactor context
I'm trying to utilize Spring Batch in one of the projects that I have, as there is another project that is based on Spring Batch.
However the more I read the more I realize that Spring batch is nothing like ApacheBeam or MapReduce, it is only used for transferring the SAME data from one place to another with some type mapping like varchar -> string.
However, the task in hand requires some processing, not only types mapping and converting but also aggregations and data structure.
Can Spring batch used for data processing or it is only an ETL tool ?
well, i disagree on this point that spring batch - is only used for transferring the SAME data from one place to another with some type mapping like varchar -> string.
Worked in 4 years in this technology and have witnessed this framework grow a lot.
Spring batch is well capable of processing data, mapping, required conversion and data aggregations - spring batch can definitely be used for data processing .
being open source technology - you will get lot of material to read about, and the forums like stackoverflow have ton of FAQs around it.
For scaling and paralleling there are various architectures in spring batch, which will help in enhancing your performance.
Further details you can find here
SPRING_BATCH_SCALING_AND_PARALLELING
If you want to monitor your jobs then you cas use - Spring cloud date flow.
Monitoring can also be done - with AppDynamics.
Referrer this blog -
MONITOR_SPRING_BATCH_JOB_WITH_APP_DYNAMICS
Another advantage of using spring batch is you have lot of standerd predefined reader , processor and writer types - which support sources like file , DB , stream etc..
On top of this - as it is a java based framework you can do all stuff that can be done with java.
I hope this helps.
Your below write up is incorrect because its comparing apples to oranges,
However the more I read the more I realize that Spring batch is
nothing like ApacheBeam or MapReduce, it is only used for transferring
the SAME data from one place to another with some type mapping like
varchar -> string.
Unlike ApacheBeam or MapReduce, Spring Batch is not an engine but a programming framework. A programming framework usually consists of two major components - Code Structure Guidelines + APIs
So only restriction on a Java developer is to follow Spring Batch program structure guidelines and usage of Spring Batch APIs is optional.Though the modeling is Read -> Process -> Write, but a Java developer is free to write any kind of logic that he or she wishes to write in these components - only thoughts can limit as what a Java developer could write in these components.Further on, one artifact can be integrated with another artifact.
So I reiterate again that Spring Batch is a programming framework & not an engine or pre configured software like Hadoop so that comparison is like apple to oranges.
See this - Spring Tips: Spring Batch and Apache Kafka
As I have already said, a Java developer can develop any kind of program by being only in program structure limitations but logic being written has no bounds!
Saying one more time - Spring Batch is not an ETL tool like Informatica or Pentaho but a programming framework using Java and Spring. A developer can be as creative as he or she wants to be.
I had developed a real time data matching job that needed free text search capabilities using Apache Lucene by fitting in my programming into Spring Batch model.
Spring Batch (SB) gives us all three - E, T and L.
However, we have to decide whether or not use SB. Its again a quantitative decision whether if an individual/team really needs to learn it, if they dont know it. Need to evaluate ROI (Return on Investment). If its just E or T or L only, there might be another simpler solutions.
If we talk about Java only, AND either of these three, SB is not required. But again, when it comes to simplicity (if you know SB), scalability, monitoring, Transaction Managed Parallel Processing - all these come hand-in-hand with SB out of the box.
Does anybody has an experience with Spring Integration project as embedded ESB?
I'm highly interesting in such use cases as:
Reading files from directory on schedule basis
Getting data from JDBC data source
Modularity and possibility to start/stop/redeploy module on the fly (e.g. one module can scan directory on schedule basis, another call query from jdbc data source etc.)
repeat/retry policy
UPDATE:
I found answers on all my questions except "Getting data from JDBC data source". Is it technically possible?
Remember, "ESB" is just a marketing term designed to sell more expensive software, it's not a magic bullet. You need to consider the specific jobs you need your software to do, and pick accordingly. If Spring Integration seems to fit the bill, I wouldn't be too concerned if it doesn't look much like an uber-expensive server installation.
The Spring Integration JDBC adapters are available in 2.0, and we just released GA last week. Here's the relevant section from the reference manual: http://static.springsource.org/spring-integration/docs/latest-ga/reference/htmlsingle/#jdbc
This link describes the FileSucker with Spring Integration. Read up on your Enterprise Integration patterns for more info I think.
I kinda think you need to do a bit more investigation your self, or do a couple of tries on some of your usecases. Then we can discuss whats good and bad
JDBC Adapters appear to be a work in progress.
Even if there is no specific adapter available, remember that Spring Integration is a thin wrapper around POJOs. You'll be able to access JDBC in any component e.g. your service activators.
See here for a solution based on a polling inbound channel adapter too.