I gain my first experience with Microservices and need help on an important descision.
Example: We have
CustomerService (contains CustomerDTO)
InvoiceService (contains CustomerDTO, InvoiceDTO)
PrintService (contains CustomerDTO, InvoiceDTO)
As you see I have a heavy Code Duplication here, each time I modify CustomerDTO, I need to do it in 3 different Microservices.
My potential solution is, to exclude the duplicated classes into a library and share this library between microservices. But in my opinion this will break the microservice approach.
So, what is the right way to handle my problem?
There is no single right way to solve this problem as with most things. There are pros and cons to each approach depending on your circumstances.
Here are probably your main options
Create a new common service:
If your logic is sufficiently complex, you have many services that need this logic, and you have time on your hands then look to creating a common service that will fulfill this need. This may not be a small undertaking as you now have to create one more service you need to manage/deploy/scale
Just accept the duplication:
This may seem counter-intuitive, however if the logic is sufficiently small enough it may be better to just duplicate it rather than couple your micro-services with a library
Create the library
Real life is different than the textbooks. We are often constrained by time and budget among other things. If your micro-services are small enough and you know this logic will not change as much, or you just need to get things out and duplication would take more time. Take this approach. There is nothing to say you can't address this later when you have the time/budget. Countless blog articles will scream at you for doing this however, they aren't you and don't know the circumstances of your project.
Related
I work on a project that leans on an app-level provide for global state management. At first, the provided object was small, but it has grown with the project.
How much does app performance suffer for injecting this object where it's data is needed? Short of implementing a global state management tool, like Pinia, is it worthwhile to decompose the large object we're providing into individual objects to provide "chunks" of data where its needed?
Since it's global, I guess it's quite big memory wise. Take a few hours to setup Pinia, will be faster/easier since it will come with only the parts you need.
We can't give you an exact number to your solution since it depends of a lot of things mostly.
PS: it can also be totally irrelevant and coming from another part of your code tbh.
I tend to write a lot of applications that, say, will expose API to a logic, which will call other APIs within the same logic, and then persist some form of data to a database at the end. So, in other words, I tend to have a method call that encapsulates calling other APIs, processing data and then persisting to the database.
I'm familiar with TDD (test driven development), but I find it difficult to practice, because the last thing I want is persisting useless data in production database while running mandatory tests, and also calling external APIs that I don't control and messing up their database in turn.
I'm probably looking at it the wrong way.
How do I practice efficient TDD and evade this problem?
How do I practice efficient TDD and evade this problem?
Design.
Roughly, we arrange our code so that it is a composition of modules of two flavors
Modules that are easy to test
Modules that simple and stable
So the code that actually needs to talk to the database should be of the second type; no branching, trivial operations, and most importantly it doesn't need to change very often.
The code that we write that feels inherently risky? that goes into an "easy to test" module. These modules are only loosely coupled to their simpler cousins - it should be easy to substitute a "test" implementation for the "real" one. Thus, instead of talking to "the database" during our testing, we talk to some in memory collection that just gives us scripted responses.
In short, we arrange our code so that it is easy to test where the mistakes are.
As for the hard to test bits; well, perhaps we measure the risks in other ways, or we run those tests at a different cadence, or we defer the testing of them to functional tests, or....
I am wondering, if is it possible to apply agent/actor library (Akka, Orbit, Quasar, JADE, Reactors.io) in Function as a Service environment (OpenWhisk, AWS Lambda)?
Does it make sense?
If yes, what is minimal example hat presents added value (that is missing when we are using only FaaS or only actor/agent library)?
If no, then are we able to construct decision graph, that can help us decide, if for our problem should we use actor/agent library or FaaS (or something else)?
This is more opinion based question, but I think, that in the current shape there's no sense in putting actors into FaaS - opposite works actually quite well: OpenWhisk is implemented on top of Akka actually.
There are several reasons:
FaaS current form is inherently stateless, which greatly simplifies things like request routing. Actors are stateful by nature.
From my experience FaaS functions are usually disjointed - ofc you need some external resources, but this is the mental model: generic resources and capabilities. In actor models we tend to think in category of particular entities represented as actors i.e. user Max, rather than table of users. I'm not covering here the scope of using actors solely as unit of concurrency.
FaaS applications have very short lifespan - this is one of the founding stones behind them. Since creation, placement and state recovery for more complex actors may take a while, and you usually need a lot of them to perform a single task, you may end up at point when restoring the state of a system takes more time than actually performing the task, that state is needed for.
That being said, it's possible that in the future, those two approaches will eventually converge, but it needs to be followed with changes in both mental and infrastructural model (i.e. actors live in runtime, which FaaS must be aware of). IMO setting up existing actor frameworks on top of existing FaaS providers is not feasible at this point.
The idea of TDD is great, but i'm trying to wrap my head around how to implement a complex system if a design is not proposed upfront.
For example, let's say I have multiple services for an payment processing application. I'm not sure I understand how development would/can proceed across multiple developers if there is not a somewhat solid design upfront.
It would be great if someone can provide an example and high level steps to putting together a system in this manner. I can see how TDD can lead to simpler and more robust code, I'm just not sure how it can bring together 1) different developers to a common architectural vision and 2) result in a system that can abstract out behavior in order to prevent having to refactor large chunks of code (e.g. accept different payment methods or pricing models based on a long term development roadmap).
I see the refactoring as a huge overhead in a production system where data model changes increase risks for customers and the company.
Clearly i'm probably missing something that TDD gurus have discovered....
IMHO, It depends on the the team's composition and appetite for risk.
If the team consists of several experienced and good designers, you need a less formal 'architecture' phase. It could be just a back of the napkin doodle or a a couple of hours on the whiteboard followed by furious coding to prove the idea. If the team is distributed and/or contains lots of less skilled designers, you'd need to put more time/effort (thinking and documenting) in the design phase before everyone goes off on their own path
The next item that I can think of is to be risk first. Continually assess what are the risks to your project, calculate your exposure/impact and have mitigation plans. Focus on risky and difficult to reverse decisions first. If the decision is easily reversible, spend less time on it.
Skilled designers are able to evolve the architecture in tiny steps... if you have them, you can tone down the rigor in an explicit design phase
TDD can necessitate some upfront design but definitely not big design upfront. because no matter how perfect you think your design is before you start writing code, most of the time it won't pass the reality check TDD forces on it and will blow up to pieces halfway through your TDD session (or your code will blow up if you absolutely want to bend it to your original plan).
The great force of TDD is precisely that it lets your design emerge and refine as you write tests and refactor. Therefore you should start small and simple, making the least assumptions possible about the details beforehand.
Practically, what you can do is sketch out a couple of UML diagrams with your pair (or the whole team if you really need a consensus on the big picture of what you're going to write) and use these diagrams as a starting point for your tests. But get rid of these models as soon as you've written your first few tests, because they would do more harm than good, misleading you to stick to a vision that is no longer true.
First of all, I don't claim to be a TDD guru, but here are some thoughts based on the information in your question.
My thoughts on #1 above: As you have mentioned, you need to have an architectural design up-front - I can't think of a methodology that can be successful without this. The architecture provides your team with the cohesion and vision. You may want to do just-enough-design up front, but that depends on how agile you want to be. The team of developers needs to know how they are going to put together the various components of the system before they start coding, otherwise it will just be one big hackfest.
It would be great if someone can provide an example and high level
steps to putting together a system in this manner
If you are putting together a system that is composed of services, then I would start by defining the service interfaces and any messages that they will exchange. This defines how the various components of your system will interact (this would be an example of your up-front design). Once you have this, you can allocate various development resources to build the services in parallel.
As for #2; one of the advantages of TDD is that it presents you with a "safety net" during refactoring. Since your code is covered with unit tests, when you come to change some code, you will know pretty soon if you have broken something, especially if you are running continuous integration (which most people do with a TDD approach). In this case you either need to adapt your unit tests to cover the new behavior OR fix your code so that your unit tests pass.
result in a system that can abstract out behavior in order to prevent
having to refactor large chunks of code
This is just down to your design, using e.g. a strategy pattern to allow you to abstract and replace behavior. TDD does not prescribe that your design has to suffer. It just asks that you only do what is required to satisfy some functional requirement. If the requirement is that the system must be able to adapt to new payment methods or pricing models, then that is then a point of your design. TDD, if done correctly, will make sure that you are satisfying your requirements and that your design is on the right lines.
I see the refactoring as a huge overhead in a production system where
data model changes increase risks for customers and the company.
One of the problems of software design is that it is a wicked problem which means that refactoring is pretty much inevitable. Yes, refactoring is risky in production systems, but you can mitigate that risk and TDD will help you. You also need to have a supple design and a system with low coupling. TDD will help reduce your coupling since you are designing your code to be testable. And one of the by-products of writing testable code is that you reduce your dependencies on other parts of the system; you tend to code to interfaces which allows you to replace an implementation with a mock or stub. A good example of this is replacing a call to a database with a mock/stub that returns some known data - you don't want to hit a database in your unit tests. I guess I can mention here that a good mocking framework is invaluable with a TDD approach (Rhino mocks and Moq are both open source).
I am sure there are some real TDD gurus out there who can give you some pearls of wisdom...Personally, I wouldn't consider starting a new project with out a TDD approach.
I'm not entirely convinced of the benefits of a 3-tier architecture. Why, then, has LINQ emerged, which is a lighter data access approach? Any input would be appreciated.
One of the main benefits of n-tier applications (there are of course many more than what I mention here) is the separation of concerns that it brings. If you structure your application so, that the responsibility for i.e. data access is held in a data access layer (LINQ2SQL is a perfectly good example of one), validation and other business logic in one or more other layers, presentation in yet another one etc., you can change details in, or even replace, an either layer without having to re-write the rest of your applicaton.
If, on the other hand, you choose not to implement an n-tier approach, you'll quickly notice that for example changing the name of one single database table will require you to go through your entire application - every single line of code - in search for SQL statements that need to be updated. In an n-tier application (if you've done things rigth), you'll only have to change the table name once in your code.
You need to do it the naive way and fail before you realize the problems those frameworks and patterns solve.
Happened to me with many things. SVN branches looked like a disorganized way to do things, until one day I wished I had branched before my last 5 commits. C++ templates seemed useless and confusing, until I got enlightened; now I use them pretty often. And every single J2EE feature will look like useless bloat to anyone, until you actually build an app big enough and have problems; then they may be exactly what you need. (thus it's a flaw to "require" you use them)
Like in most fields of engineering there's never a perfect one-size-fits-all solution for development or architecture. So it is with n-tier architectures.
For example, quite a few applications run perfectly well as a one-tier or two-tier architecture. Microsoft Word, for example, does quite well, thank you, as a single-tier system.
Most business applications have started using layers (as distinct from tiers: layers are virtual, tiers are physical) as it makes life much easier to have presentation logic in one place, business logic in another, and persistence logic somewhere else. It can make sense too depending on the application to have lots more layers: I recently finished up a project with about sixteen layers between the UI client and the SQL database. (We had REST services, co-ordination layers, mixed databases, you name it. Made for quite a deployment challenge.)
The nice thing about all these layers are
testing becomes fairly easy, as each layer does one and only one thing
it's feasible to scale, especially if you design your layers to be stateless: then you can group them together and deploy to separate boxes quite easily
it's feasible to have lots of developers working simultaneously, so long as you keep talkin' to each other
changes are (usually) limited to one layer in the code
LINQ, the Language Integrated Query, really helps too, as can abstracts away much of the harder parts of working with persistence layers. For instance
the SQL-like syntax maps fairly directly to SQL database tables or views
working with more complex non-relational data like XML files is made straightforward
Without LINQ developing persistence layers was invariably repetitive, never a good thing.