we are using mass transit and would like to upgrade rabbitmq to the next version.
I couldn't find a list of supported rabbitmq version per mass transit versions
I assume that there aren't breaking changes unless a major version changed so it should be safe.
Also we have CI tests that should let us know if something has been broken.
However,
Is there such a list?
There is no list, but any current version of RabbitMQ should work. MassTransit uses RabbitMQ.Client to communicate with RabbitMQ.
Related
I'm trying to wrap my head around the best practices to manage versioning in microservices based architecture with periodic releases.
Currently our system is decomposed into multiple different repositories:
Frontend
Backend
Database
API gateway
Docker-compose-env
Each of these components must be developed, built, tested, containerized and deployed independently. But the release cycles are synchronized and periodic. Docker-compose-env project contains the environment definition to start all compatible service versions for development and integration testing purpose.
Current versioning strategy is as follows:
Each commit to master branch is tagged with a semantic version and pushed to docker registry (semantic tags are used to track dependencies during development cycle)
Each merge commit to persistent release branch is tagged with a release tag and pushed to docker registry (release tags are used to synchronize project versions together for quarterly release)
master is the trunk, and periodic release build is initiated by PR from master to release.
I'm skeptical if this the best way to manage versions with microservices based architecture on periodic releases. Any feedback or tips are appreciated.
Each of these components must be developed, built, tested, containerized and deployed independently. But the release cycles are synchronized and periodic.
There is a contradiction here. Microservices mostly solve an organizational problem - the main point is that teams should be able to work independently as much as possible.
Synchronization between teams is what make them slow. This can happen in different ways, e.g. waiting for another version to be deployed in a shared test environment, or using the same shared database schema, or making releases at the same time.
I'm skeptical if this the best way to manage versions with microservices based architecture on periodic releases.
Try to avoid "synchronized releases", instead make sure to not break any contracts between the services (e.g. no breaking API changes). Try to release more often, you want to work in small batches to reduce the risk with deployments and changes. Try to not pile of a bunch of changes, deploy continuously - Continuous Delivery.
Release cycles are syncronized
I think the fact that you need to do a synchronized release of all services at the same time could be an indicator that the coupling between your services is higher then it should be and probably the way you are managing it can be improved.
The question is how can you design your development teams working on different micro-services so that when they introduce changes and they do not break each others micro-service?
Versioning and managing changes
There are 2 aspects which are important for this to work and they are:
Versioning and how you implement and work with versioning.
Team Communication. Communication between teams when introducing breaking changes.
What do I mean by this?
First about versioning. Your micro-services are communicating with each other.
Regardless of the fact that the communication is sync or async using Rest(or SOAP or gRPC or other) or Messaging(Queues) they need to rely on some Contracts. Those Contracts will be some API Contracts(in terms of Java/C# classes/interfaces). They need to be stable as they can be used by other micro-services.
Suggestion: I would suggest to do versioning of the micro-service independent from the versioning of the Contracts.
Example:
Micro-service order-micro-service could be at latest version v1.0.0 and Contracts order-micro-service-contracts at version v1.0.0 as well.
Micro-service customer-micro-service could be at latest version v3.0.0 but Contracts customer-micro-service-contracts could be at version v2.2.1.
Micro-service product-micro-service could be at latest version v3.0.0 and Contracts product-micro-service-contracts could be at version v4.0.0.
As you can see from the example above the version from the Micro-service and its corresponding exposed Contracts can be the same but they can also differ. The reason is simply that you can do changes on the micro-service(some internal business logic change) without changing the Contracts. And you can also do changes on the Contracts without changing the micro-service logic. Usually changes happens on both of them in the same time. You update some api business logic for which you adjust the exposed Contract. But sometimes a MAJOR change in the micro-service logic is not necessary a breaking or MAJOR change on the Contracts. As you see this gives you great flexibility. The benefit of this is not only flexibility but also the fact that a micro-service-A will only be dependent on micro-service-B-contracts and not the micro-service-B itself. This is just a suggestion you can also use one version for micro-service and its exposed Contracts.
Now about team communication. By this I mean if you have an organization where you have multiple teams working on different areas of the system and each team is responsible for one or more micro-services from an particular Domain.
If you are using the Semantic versioning like MAJOR.MINOR.PATCH for example v1.3.5 then you can do it in the following way.
There are a couple of things which are important to consider:
Contracts PATCH/MINOR change
A change which is a PATCH or MINOR change version upgrade should not be a breaking change and should always be backwards compatible for the consumers who use those contracts. This means that you should ensure that upgrading from version 1.3.0 to 1.4.0 should not be a problem to the consumer regardless of the fact if he upgraded to 1.4.0 or stayed on 1.3.0 for a little longer. Ideally all consumer should update to latest version but even if they don't for some period they will not be broken by the change. For example a change for which you will do that kind of upgrade would be adding new Contract model or updating existing model with new not mandatory fields, or increasing accepted string length from a field from 20 to 50 or similar.
Contracts MAJOR or breaking change
Is usually a big change which can also be breaking change. If it is a breaking change then we need some team process in place. The teams who use those contracts need to be notified that the change will happen upfront and even when releasing the new version a bridging period of couple weeks(or sprints) should be ensured where both versions of contracts will work(old and new). This will give the affected teams/micro-services enough time to upgrade and adjust their services. After that the backwards compatibility compromise Contracts/code can be deprecated. Sometimes for some cases a solution for a breaking change Contract change is introducing a complete new version of that Model(class) and not do a hard change on the same Model. For example you could have a CustomerModel class and then introduce CustomerModelV2 and remove the old CustomerModel class after some period. This is a common situation where you have a Contract Model for an Event(Message from a queue) like: CustomerCreated. You can have CustomerCreated and CustomerCreatedV2. You can publish both messages for a particular time period until the consumers adopt and deprecate(stop publishing the event and removing the Contract model) the CustomerCreated event. This depends on your particular business logic or case.
Micro-service changes
Regardless of the fact that the change is just a bug fix, small change or a big change in the service if your versioning is separate from the from the Contracts it should not be affecting the other micro-services, at least not from the contract managing prospective. Doing versioning updates on micro-service only gives you the possibility to deploy it independently.
Independent and separate deployments of micro-services
If you apply the above advice's you will come closer to the situation where you can deploy micro-services independently and without synchronized periods where all services have do be deployed at once.
One of the biggest advantages of using micro-services is being able to deploy micro-services independent from other parts of the system so if you have a chance to do that you should go for it.
Each of these components must be developed, built, tested,
containerized and deployed independently. But the release cycles are
synchronized and periodic.
Since you already develop, build and tested independently you could also do the release independently.
I know that all those suggested changes are not only technical but also organizational changes like team communication, team setup and so on. But usually when working with big system using micro-services it a compromise between those 2 worlds and trying to find the best process and solution for your Organization and Business.
I want to communicate my services using events. I gonna publish (internally) all my domain events and allow any other service to subscribe to them. But such approach couples those services togheter. I am not longer allowed to change my events. This is even worse than local coupling because I dont event know my consumers any more. This limits the ability of developing/refactoring to unacceptable dedree. I am thinging about versioning my events which solves most of the issues. But how to subscribe to versioned events? Introducing common interface that groups all event`s versions and then downcast event within listener to accepted one does not sound like a vital solution. I also take into account publishing all supported versions of the event to the bus. By definition each subsriber will handle just one version. I dont want my domain to by involved in this matters so I need to build kind of infrastructure listener that will be translate catched events to other versions. I cant find anything about that topic in the Internet which automatically makes me think if I am not thoroughly wrong :)
UPDATE: After a lot of thought, I no longer want to publish my domain events. I think it is no desirable to expose internal service mechanics to the outer world. It also could violate some domain data access restriction. I think, the way to go is to map my domain events to some more corase integrational events. But I still need way to wersion them probably :)
UPDATE2: After some consultations an idea came up. Assuming we stick to the concept of integration events. Such events may be considered just as type and event id. So outer listener just focus on event type. If event occur then listener will be provided with event id. This enable listener to fetch real event from the stream/bus/wtf in given version. $eventsStore->get($eventGuid, $eventType, 'v27') for example (PHP syntax)
I gonna publish (internally) all my domain events and allow any other service to subscribe to them.
This is a common pattern in Even-Driven Architecture. I assume that you publish the events on an Event Broker, e.g. Apache Kafka and that Consumers subscribe to topics on the Event Broker.
I am not longer allowed to change my events. This is even worse than local coupling because I don't event know my consumers any more. This limits the ability of developing/refactoring to unacceptable degree. I am thinking about versioning my events which solves most of the issues.
Nah, published contracts should be versioned and no backward incompatible changes can be added to them. If you need a change that is not backward compatible, you have to introduce a new version of the published contract - and keep the old one as long as there is consumers. This is no different from REST-based interfaces - you have to fulfill your contracts.
With REST you may do this by using both /v1/orders and /v2/orders at the same time. With an Event-Driven Architecture you use two topics e.g. orders-v1 and orders-v2 - and these two contain data following a schema, e.g. Avro.
With an Event-Driven Architecture, where the services are decoupled (with a broker in between), you can actually phase out the old producer, if you add a smaller transformer, e.g. that consume orders-v2 and transform the events to the old format and publishes them on orders-v1 - so both v1 and v2 is still published.
Building Event-Driven Microservices is a good book about this.
I have:
Microservice-A
Microservice-B
Microservice-C
Microservice-A calls Microservice-B and Microservice-C
When I deploy Microservice-A I want make sure that other microservices it depends on have not changed since I last release it.
Is there a recommended way to do this?
I'm thinking:
when I deploy Microservice-A
Microservice-A makes calls to Microservice-B and Microservice-C
this call would fetch the endpoint specification for the endpoints it depends on and verify whether the endpoints have changed (in a way that would break Microservice-A) since last release.
This should happen before I interrupt the currently running Microservice-A just before deployment procedure commences.
Sure can do testing but that would be too late in my view. I'm looking for an automated way to verify this before deployment.
Has anyone done anything like this before? What tooling can be used for this?
The ideal solution is to never be in a position where you are deploying into an environment where you don't already know the versions of your dependencies. That way madness lies.
Avoiding this is a governance concern and so should be central to any service oriented approach to building software.
For instance, let's say you are developing version 2.0 of your service A. In your target environment, you have service B version 1.0 and service C version 1.0.
So the first step on the path to stress-free releases, as part of your development build, you should be running a set of nearest neighbour automated tests, which stub out B v1.0 and C v1.0 based on the service contracts (more on this later). This can be facilitated using test double tools such as mountebank.
Then, just as you have created your v2.0 release branch, you learn that another team is about to release v1.1 of service C. It should always be possible to work out whether v1.0 to v1.1 constitutes a breaking change to the v1.0 contract for service C (more on this later).
If v1.1 is a breaking change, no problem, you update your tests with v1.1 of the service C contract and fix any failures. You are then good to create a new v2.0.1 patch branch and release. If for whatever reason you are forced to release before service C you can still release from the v2.0 branch.
If v1.1 is not a breaking change, no problem, just release off your existing branch.
There are various strategies for coping with the overhead produced by a centralised release management protocol such as described above.
As stated earlier, contracts for all dependent services should be used when testing your service. (Note: it's very important for the nearest neighbour tests to be driven from contracts, rather than using existing code models, such as DTOs defined in the service's unit tests). Contracts for all the services should be based on a standard (such as swagger) which supplies a complete service description, and be very easy to find - the use of a service repository can simplify this.
Also stated earlier, it should always be possible to know if new versions of dependent services have the potential to break your service. One strategy is to agree on a versioning convention which bestows some kind of meaning when incrementing the version. For instance, you could use a major.minor.patch (eg v1.0.0) where a change the major version number constitutes a change to the service contract and therefore has the potential to break things. In our previous example, service C went from v1.0 to v1.1. With a convention such as the one described above, we could be sure that the change would not break us, as the major version number was unchanged.
While it can be cumbersome to set up and maintain a centralised release management protocol, the benefit is that you always have full confidence that by deploying your service, nothing will break. What's more, this avoids having any complex (and to my mind, contrived) runtime dependency resolution, such as you are proposing in your original question.
Each microservice could provide a versionnumber either via API or by writing to a public file / shared DB. Each microservice would then contain all the expected versionnumbers of its dependencies with and check if the version numbers in the file / database match before startup.
I plan to use Elasticsearch on heroku.
I was looking for the best option of Elasticsearch add-on I can use.
Found was my first choice from the following reasons:
It is now part of elastic.
When using Elasticsearch on heroku it will be opened to the world - a secure wrapper to the transport client was introduced - https://github.com/foundit/elasticsearch-transport-module/
But it looks like this repository is not highly maintained, and Elasticseach 1.5 is the latest version which is supported.
What is the recommended add-on then?
If I want to use the latest version of Elasticsearch I am doomed to use an unsecure connection?
Maybe use the official java client?
Nick with Bonsai here. Based on your question, and my own obvious bias, I'll suggest Bonsai for the following reasons:
All of our clusters have SSL with basic auth to secure the connection. We feel pretty strongly that security comes as a standard feature.
We were the first hosted Elasticsearch provider, ever. (And one of the first addon providers on Heroku, ever, with our first search addon, Websolr.) So we've got plenty of experience hosting search and and thousands of other happy Heroku customers.
One definite tradeoff with using Bonsai is that we're generally always going to lag a bit behind the latest version of ES. As of this posting we're still running ES 1.7, but updates to ES 2.2 are just around the corner.
This is probably going to be true in the future as well. Part of the reason for this is that we're a small, bootstrapped company, and we have to be pragmatic in where we focus our engineering efforts. Plus as an operations company with thousands of businesses, we like to let major new upgrades spend a few months in the wild before we commit to supporting it.
We also work hard on providing managed upgrades, at least for versions that are sufficiently backwards compatible. Everyone has their tools for helping to manage upgrades, but I don't think any of the other providers do actual in-place upgrades.
Unless you have a hard requirement for a specific feature in 2.x (and if you do, please let me know) you may do fine on 1.7 until our 2.x support is fully baked. Drop us a line at info#bonsai.io to get whitelisted for the first release of that in the coming weeks.
We launched an application in the Apple AppStore and Google Play Store, and now we need to launch an update. But this update will change the server-side code (i.e. the API).
Does GeneXus handle multiple API versions? I mean, how to prevent that an app in the 1.0 version break when we launch version 1.1?
There are several considerations when publishing a new version of your application.
If you need both versions to be available at the same time, then the best option is to publish the new version's services to another URL. Say, for instance, you had version 1.0's services at https://example.com/myapp10, then create a new "virtual directory" https://example.com/myapp11 and make the new version point there.
A special consideration is needed if there are also changes in the database. If you only have new tables and/or attributes (and the new attributes are nullable), then you don't need to do anything else.
However, if you remove or change existing attributes, then the "old" services may not work with the new database schema. In that case, you'll also need to keep both versions of the database, and consider some replication mechanism to keep then in sync.
You may find this article interesting, about Pesobook application's deploy process (Spanish only).
Here you will find more detailed information about versioning an SD App with Genexus.
And this article explains how to do it at the Knowledge Base.
You could also create modules in order to manage your services versions. Instead of create a new virtual directory with all the objects, you can move the new (or updated - via save as) services to a new module.
Example:
webapp/wsv1/rest/myservice
webapp/wsv2/rest/myservice
webapp/wsv3/rest/myservice
You must duplicate "myservice", however, the other objects of the KB will not be duplicated.
Then your apps will consume the version of "myservice" as they need.
I use this way to serve some native apps which are not made with GeneXus but they consume GeneXus REST webservices.
Hope it will be usefull :)