I work devops for a fairly large company that is in process of transitioning to microservices. This is a new area for most people involved and some of the governing requests seem like bad practice to me but I don't have the expertise to convince otherwise.
The request is to generate a report before deploying that would list any new api/events (Kafka is our messaging service) in a microservice.
The path that's being recommended is for devs to follow a style guide and then scrape the source code during CI/CD pipeline to generate a report that can be compared to previous reports and identify any new apis.
This seems backwards and unsustainable but I've been unable to find another solution that would satisfy their requests. I've recommended deploying to dev first, then using a tracing tool to identify any api changes, or event subscriptions, but they insist on having the report before deploying.
I'm hoping for any advice on best practice to accomplish this.
Tracing and detecting version changes is definitely over engineering. Whats simpler like #zenwraight has mentioned, is to version your APIs. While tracing through services to explore the different versions and schema could be a potential solution, it requires a lot more investment upfront and if thats not the bread and butter of the company, I would rather use a vendor product that might support something like this.
If discovery is a mechanism that is needed, I would recommend something that publishes internal API docs using a tool like Swagger so that you can search if there's an API you can consume.
And finally to support moving to different versions, I would recommend having an API onboarding process for the services so that teams can notify other teams that are using specific versions their services are coming to the end of their lifecycle and they will need to migrate to newer ones.
Related
For example,
You have an IT estate where a mix of batch and real-time data sources exists from multiple systems, e.g. ERP, Project management, asset, website, monitoring etc.
The aim is to integrate the datasources into a cloud environment (agnostic).
There is a need for reporting and analytics on combinations of all data sources.
Inevitably, some source systems are not capable of streaming, hence batch loading is required.
Potential use-cases for performing functionality/changes/updates based on the ingested data.
Given a steer for creating a future-proofed platform, architecturally, how would you look to design it?
It's a very open-end question, but there are some good principles you can adopt to help direct you in the right direction:
Avoid point-to-point integration, and get everything going through a few common points - ideally one. Using an API Gateway can be a good place to start, the big players (Azure, AWS, GCP) all have their own options, plus there's lots of decent independent ones like Tyk or Kong.
Batches and event-streams are totally different, but even then you can still potentially route them all through the gateway so that you get the centralised observability (reporting, analytics, alerting, etc).
Use standards-based API specifications where possible. A good REST based API, based off a proper resource model is a non-trivial undertaking, not sure if it fits with what you are doing if you are dealing with lots of disparate legacy integration. If you are going to adopt REST, use OpenAPI to specify the API's. Using this standard not only makes it easier for consumers, but also helps you with better tooling as many design, build and test tools support OpenAPI. There's also AsyncAPI for event/async API's
Do some architecture. Moving sh*t to cloud doesn't remove the sh*t - it just moves it to the cloud. Don't recreate old problems in a new place.
Work out the logical components in your new solution: what does each of them do (what's it's reason to exist)? Don't forget ancillary components like API catalogues, etc.
Think about layering the integration (usually depending on how they will be consumed and what role they need to play, e.g. system interface, orchestration, experience APIs, etc).
Want to handle data in a consistent way regardless of source (your 'agnostic' comment)? You'll need to think through how data is ingested and processed. This might lead you into more data / ETL centric considerations rather than integration ones.
Co-design. Is the integration mainly data coming in or going out? Is the integration with 3rd parties or strictly internal?
If you are designing for external / 3rd party consumers then a co-design process is advised, since you're essentially designing the API for them.
If the API's are for internal use, consider designing them for external use so that when/if you decide to do that later it's not so hard.
Taker a step back:
Continually ask yourselves "what problem are we trying to solve?". Usually, a technology initiate is successful if there's a well understood reason for doing it, which has solid buy-in from the business (non-IT).
Who wants the reporting, and why - what problem are they trying to solve?
As you mentioned its an IT estate aka enterprise level solution mix of batch and real time so first you have to identify what is end goal of this migration. You can think of refactoring applications. If you are trying to make it event driven then assess the refactoring efforts and cost. Separation of responsibility is the key factor for refactoring and migration.
If you are thinking about future proofing your solution then consider Cloud for storing and processing your data. Not necessary it will be cheap but mix of Cloud and on-prem could be a way. There are services available by cloud providers to move your data in minimal cost. Cloud native solutions are there for performing analysis on your data. Database migration service in AWS or Azure can move data and then capture on-going changes. So you can keep using on-prem db & apps and perform analysis for reporting on cloud. It will ease out load on your transactional DB. Most data sync from on-prem to cloud is near real time.
I'm wondering if there are any utilities/patterns/paradigms/standards for monitoring React applications in production.
I've seen a lot of documentation about React performance debugging that recommends the Chrome Dev Tools (which are great, but aren't a passive way to monitor end user performance)
How could I log data to know how long users are waiting for components to mount or render?
The only thing I've thought of so far is creating a Loggable[Pure]Component that extends React.[Pure]Component whose constructor, componentWillMount/Update, and componentDidMount/Update methods log render/mount times to a server. Then, components I want to monitor can extend these components and, if need be, call super() in the lifecycle methods before doing their own work. To specifically know which components these metrics go to, I'd have to expose a method in the Loggable[Pure]Component class that does something silly like setUniqueId and then each derived class would have to call it in the constructor.
This all seems terrible and I'm very much hoping there are some things people out there have implemented, but I haven't found anything thus far.
I would have a look at some APM tools, they handle the frontend monitoring, and the backend monitoring as well. They all support react, and folks use these all the time for that use case. It really depends on your goals in the monitoring, are you doing this for fun? Do you have a startup? Are you working for a large enterprise? There are 3 major players in this market.
AppDynamics - Enterprise APM, handles the most complex apps. Unified product offering delivered SaaS or on-premises. Has deep database, server, and other monitoring.
Dynatrace - Enterprise APM, handles complex apps well. Fragmented portfolio, but the SaaS product is good. The SaaS product has limited depth in some ways. Handles server and cloud infrastructure monitoring well.
New Relic - Easy and cheap(er than others), not as in-depth as some other options. Tends to be popular with small companies. Does a good job monitoring cloud infrastructure services.
These products all do what you are looking for, but it depends on your goals with the data and how you plan to analyze it.
If you want something free and less functional there are ways to do this with open source, but you'll have to stand up and manage a pretty complex stack. Here is one option.
Check out boomerang, which can log/extract the metrics you are looking for, it doesn't "understand" react, but it should work. This data can be posted to many different systems. The best suited is likely the ELK stack (open source log analytics, and more). Here is one of several examples which marries these two together to provide analysis of the browser performance https://github.com/naukri-engineering/NewMonk
I want to create a production management system to be used by a small manufacturing firm. The system will allow to document different stages in manufacturing of equipment. The requirements are as follows:
1.Non browser based interface.Need something like Swing or AWT based.While i understand the convenience of implementing a browser based solution,the business owner insists on a non browser interface
2.Accessed from multiple systems.These systems will allow CRUD operations on the central system (Thin Client?)
3.The application will not have more than 3 concurrent users.
I need some advice regarding what would be a good path for this kind of application.Currently, i'm thinking of using Griffon with RMI. However, i don't have much development experience.Read a bit about Apache River (Jini) too.Would it be a good idea to use Griffon with RMI?
Please provide some advice. Thanks.
EDIT:after some reading, i've decided to use more mainstream frameworks.So, Griffon is not an option. How about Jini(Apache River) or OSGI (Apache Felix)?
Hmm how is that a project which recently moved out of the incubation phase be considered mainstream vs a project that's been used in production for more than 3 years now? Anyway, Apache River gives you access to Jini technology and nothing more; meaning you can't achieve item #1 of your list with River alone. River may use RMI for accessing remote resources, however you can use RMI directly, or try out DRMI, Kryonet, Hessian/Burlap, Spring's HTTP Invoker, Protocol Buffers, Avro/Thrift, REST, SOAP, ZMQ and many more.
Even if you choose one of these options and/or River you still have to define the following things
application structure (file structure and runtime behavior)
build setup
dependency management
testing profiles
packaging
deployment strategies
These things and more are what Griffon brings to the table. As you may have noticed the framework allows you to build up applications by adding plugins, reducing thew amount of time you must allot for hunting down dependencies, setting up bootstrap mechanism and getting things done. On the subject of remoting technologies have a look at the different options Griffon has to offer http://artifacts.griffon-framework.org/tags/plugin/remoting
Even more, you can also combine OpenDolphin (http://open-dolphin.org/dolphin_website/Home.html) with Griffon. There's even an example application found at the opendolhpin repository showing a full client-server application (build with Griffon, Grails and OpenDolphin) https://github.com/canoo/open-dolphin/tree/master/dolphin-griffon-crud
With what seems to be your current understanding of the problem, I would not recommend OSGI, especially for a small manufacturing firm (Possible maintenance issues, depending on the "personel").
The main reason why I wouldn't advocate JINI or OSGI in your case is because of what you said
However, i don't have much development experience.
JINI (Apache River) is a viable option as long as you fully understand the concepts of LookupService and service registrations, etc. There's tons of RMI going on here with possible firewall implications...
OSGI is not difficult but you may have issues deciding how to structure your applications as well as interacting with services, etc.
Try to stick to the simplest approach that you can handle for the implementation (Flexible design in mind): Make it work and then improve it.
There are simple Web Services options such as Spring Remoting (over http/https for example), unless Spring introduces too many concepts and headaches for your app.
I'm in the middle of a project where we will migrate a major software system based on a larger set of custom made technologies to be based on OSGi services. For this we will likely need a some sort of message bus that plays nice with OSGi services.
Sync and ASync delivery
Point-to-point only
Guaranteed delivery - preferable with persistence via files
Strict message ordered from the same client (Async mode), but necessarily from different clients
Support for process-to-process and node-to-node nice but not strictly required
Open source solutions will be preferred, but not required.
I have looked at eventbus (as recommended in https://stackoverflow.com/a/1953453/796559), but that does not seem to work well.
So the question is, which technologies match the above?
Tonny,
Having just come from a very similar, and successful project, please let me share my experience with you to save you some time and your company some money. First and foremost, ESB's were a really good idea 8 years ago when they were proposed. And, they solved an important problem: how do you define a business problem in a way that those pesky coders will understand? The goal was to develop a system that would allow a business person to create a software solution with little or no pesky developer interaction needed that would suck up money better spent on management bonuses.
To answer this the good folks at many organizations came up with JBI, BPMN and a host of other solutions that let business folks model the business processes they wanted to "digitize". But really, they were all flawed at a very critical level: they addressed business issues, but not integration issues. As such, many of these implementations were unsuccessful unless done by some high-priced consultant, and even then your prospects were sketchy.
At the same time, some really smart folks in the very late 90's published a book called "Enterprise Integration Patterns" which identified over 60 design patterns used to solve common integration problems. Many of the folks performing ESB stuff realized that their problem wasn't one of business modelling. Rather the problem was how to integrate thier existing applications. To help solve this Michael Strachan and some really smart guys started the Apache Software Foundation Project "Camel". Camel is a strict implementation of Enterprise Integration Patterns in addition to a huge number of components designed to allow folks like you and I to hook stuff together.
So, if you think of your business process as simply a need to send data from one application to another to another, with the appropriate data transformations between, then Camel is your answer. Now, what if you want to base the "route" (a specified series of application endpoints you want to send data thorugh) off of a set of configurable rules in a database? Well, Camel can do that too! There's an endpoint for that! Anyhow, dont' do the traditional ESB, its old and busted. And Absolutely do the camel thing.
Please let me know if this helps.
The OSGi specification defines a component "Event Admin" which is a lightweight pub-sub event subsystem.
From the RFC0157:
Event Admin specifies a means for an event source to send events to
event listeners. Event sources can create events with a topic and
properties and request Event Admin to deliver the events to event
listeners which have declared interest in specific event topics and/or
property values. The event source can request synchronous (and
unordered) delivery or asynchronous (and ordered) delivery.
Compared to your requirements, it would score as follows:
Sync and ASync delivery: Check
Point-to-point only: No. Pub-Sub
Guaranteed delivery - preferable with persistence via files: NO
Strict message ordered from the same client (Async mode): YES
Support for process-to-process: if (process == OSGi service) -> Yes
Support for node-to-node: Not yet. The guys of
Distributed OSGi have been working on this, but I've not seen
anything concrete.
I like the concept of Camel, but recently decided to go for the (lighter) Event Admin as my requirements are limited. +1 to Mike on the Camel motivation. I'd look into it and then compare options before deciding.
Aren't you looking for an ESB? ServiceMix is a:
flexible, open-source integration container that unifies the features and functionality of Apache ActiveMQ, Camel, CXF, ODE, Karaf into a powerful runtime platform you can use to build your own integrations solutions. It provides a complete, enterprise ready ESB exclusively powered by OSGi.
iPOJO Event Admin Handlers are a nicer-to-use way to access the Event Admin service mentioned by #maasg.
looks like you are talking about an ESB here. If its the case, then you might have look at wso2 ESB. It is powered by apache synapse. it uses OSGi as the modular framework, so that you can add/remove features according to your requirement. There is a whole product stack from wso2 like message brokers, Business process servers (ODE), etc based on the same OSGi core platform.
disclaimer : I work for wso2.
As per title. I don't know if this is the right place or way to ask this, admins feel free to edit/move/close the question if appropriate.
I'd like to get pointers to recent material clarifying the market trends, as well as real life examples. Even pseudo-pundit, Gartner-like stuff is OK. Thanks.
I am curious about the second part of the question. What is the basis of your statement that 'the ESB thing' appears to be fading? I don't believe it is.
The problem with ESBs however is that some vendors call their product an ESB, but it actually is much much more than that. In some companies this happened with their integration product just because Gartner or some other analysts company says that ESB is hot. Marketing strategy is changed: The product is called ESB and maybe somethings are added that are expected in an ESB.
Paul Fremantle of WSO2 wrote a very good article about what an ESB really is [1].
As for OSGi: The first company I saw using it in their middleware was WSO2. I have heard, that TIBCO, another middleware vendor, is also moving or has moved towards using it in their Active Matrix platform.
OSGi may help in various ways. The most important is that it decreases the effort of the installation of the platform. Install a minimum on each system used to deploy the application, and during deployment the components required to run the application will be added. You do not have to worry about having installed the right plug-ins, add-ons and what not. This is what both WSO2 and TIBCO are doing.
With some vendors, you see that you need to install an awful amount of software, of which you in the end may be using just a small part (e.g. IBM WebSphere). Because of this, you may have to use over-dimensioned systems, which adds extra costs.
OSGi may prevent this.
Have a look at the presentation of WSO2 about the WSO2 Carbon platform [2].
The statement at the end of the presentation says it all:
Adapt the middleware to your architecture, not the architecture to the middleware
So yes, I think OSGi has a future in enterprise apps.
[1] http://wso2.org/library/2913
[2] http://www.slideshare.net/wso2.org/the-carbon-story-presentation-855666
Disclaimer:
I am in no way affiliated with WSO2, TIBCO or IBM. I am a certified TIBCO BusinessWorks Developer and have been developing applications for the IBM WebSphere Process Server platform. Above all, I am a WSO2 Enthusiast.
I would say yes..WSO2 has proof for that..Check the following links
http://osgi.dzone.com/articles/carbon-osgi-and-soa
http://www.infoworld.com/d/developer-world/wso2-upgrades-osgi-middleware-695