I have a Spring Boot REST Microservice which expects an input file and converts this file to other formats using ffmpeg like so
ffmpeg -i <INPUT_FILE> -vf 'scale=320:-2' <OUTPUT_FILE>
I am calling this command at the moment in a Java ProcessBuilder and referencing the container location of the input and output files after pushing the microservice to PCF.
The ffmpeg binary and the input file are packaged in the jarfile
I understand I need to use cloud storage like NFS or S3 to specify the locations, but that is a secondary matter now.
My idea is to make the Microservice a PCF User-provided Service, so that bound Apps will supply the location of the input and converted files.
Also since there are different conversion functions, I have corresponding endpoints for each conversion function.
All examples I have seen with respect to Microservices have to do with Databases whereby you specify information like URL and credentials to access the external database,
which does not shed any light into what I have in mind.
So I have the following questions:
1) Is it possible to simply convert the running Conversion Microservice into a PCF User-provided Service? I know of the CUPS command, but I am not sure what to supply as Parameters, also since I have several endpoints
2) How could bound Apps call the endpoints of this Service and provide the locations of the input and output files.
I will appreciate code snippets if possbile
Thanks
What you're trying to do should be possible, but not quite in the way you describe it.
For starters, you can use a user provided service to pass details about a microservice between apps. Thus if you create a conversion microservice, you can pass information about the microservice to consuming apps via a user provided service.
Where I think you need to change your plans a bit is with the consuming apps. You can't pass the input/output files via a user provided service, at least not without sharing storage between the two and you don't want to go down that road. It's limiting as all apps have to share the storage and it will have scaling limits.
Instead what you'd want to do is to put the location and credentials required by a consuming party in the user provided service (this is what you're seeing with databases). For example, you'd probably need the hostname or route to your microservice and likely a user token/secret or username and password (that way you know who's making requests to your service).
Your consuming app could then read the information out of the bound user provided service and use that to contact your conversion microservice. At that point, it would be up to the consuming app to understand the API that you expose from your microservice so that the consuming app can send the file to be converted to the microservice and retrieve the processed file. You'd likely send/retrieve files over HTTP, and if I may make a recommendation you probably want to look at using the claim check pattern on your service since doing conversions can be time consuming.
Hope that helps!
Related
I have an application made up of several microservices to allow independent pieces to be scaled up/down as needed (running as containers in docker swarm and maybe eventually kubernetes). I'm looking for any suggestions for controlling access to a shared file system from multiple containers as I feel like there is probably some system/software out there that I just don't know how to search for it.
Data in the form of physical files will come in to a single location. I need to have one microservice read each file that comes in and basically upload it to a database.
What solutions exist, if any, that would prevent 2 or more instances of the same microservice from fighting over each file?
This will need to work in a 100% offline environment so cloud storage is not an option. Ideally it would work similar to how a message queue works in that the next available consumer would pick up the next available file from this shared file system.
I'm developing an application that must both handle events coming from other systems and provide a REST API. I want to split the applications into micro services and I'm trying to figure out which approach I should use. I drew attention to the Spring Cloud Netflix and the Spring Cloud Data Flow toolkit, but it's not clear to me whether they can be integrated and how.
As an example, suppose we have the following functionality in the system:
1. information about users
card of orders
product catalog
sending various notifications
obtaining information about the orders from third-party systems
processing, filtering, and transformation of order events
processing of various rules based on orders and sending notifications
sending information about user orders from third-party systems to other users using websockets (with pre-filtering)
Point 1-4 - there I see the classical micro service architecture. Framework - Spring Netflix Stack.
Point 5-9 - it's best to use an event-driven approach. Toolkit - Spring Data Flow.
The question is how to build communication between these platforms.
In particular - POPULATE ORDER DETAILS SERVICE must transform the incoming orders and save additional information (in case it needed) in the database. ORDER RULE EXECUTOR SERVICE should obtain information about the current saved rules, execute them and send notifications. WEB SOCKET SERVICE should send orders information only if a particular user has set the filters, and ORDER SAVER SERVICE should store the information about the transformed orders in the database.
1.
Communication between the micro-services within the two platforms could be using the API GATEWAY, but in this case, I have the following questions:
Does the Spring Cloud platform allow to work with micro services that way?
Performance - the number of events is very huge, which can significantly slow down the processing of events. Is it possible to use other approaches, for example, communication not through the API Gateway but through in-memory cache?
2.
Since some functionality intersects between these services, I have a question about what is "microservice" in the understanding of the Spring Cloud Stream framework. In particular, does it make sense to have separate services? Can the microservice in the Spring Cloud Stream have a REST API, work with the database and simultaneously process the events? Does such a diagram make sense and is it possible to build such a stack at the moment?
The question is which of these approaches is more correct? What did Spring Data Streams mean by "microservice"?
Given the limited information in the post, it is hard to convince on all the matters pertaining to this type of architecture, but I'll attempt to share some specifics, and point to samples. Also for the same reasons, it is hard to solve for your needs end-to-end. From the surface, it appears you're attempting to build event-driven applications and wondering whether Spring Cloud Stream (SCSt) and Spring Cloud Data Flow (SCDF) could help.
They can, yes.
The Order, User, and Catalog seem like domain objects and it would all come together to solve for a use-case. For instance, querying for a number of orders for a particular product, and group by the user. There are a few samples that articulate the data flow between the entities to solve similar problems. Here's a live code-walkthrough of event-driven systems in action. There's another example of social-graph application, too.
Though these event-driven applications can run standalone as individual services with the help of of message broker (eg: Kafka or RabbitMQ), you could of course also register them in SCDF and use them in the SCDF DSL to build a coherent data pipeline. We are expanding on more direct capabilities in SCDF for these types of use-cases, but there are ways to orchestrate them today with current abilities, too. Follow spring-cloud/spring-cloud-#2331#issuecomment-406444350 for more details.
I hope this gives an idea. Try to build something small using SCSt/SCDF, prove it out, and expand to more complex use-cases.
I am a newbie in Microservices, having theoretical knowledge. I want to make a small application in Microservices. Can anyone please help me with the idea of how to implement microservices?
Thanks in Advance!!
You can create something like a currency conversion app with three microservices like these:
Limit service;
Exchange service;
Currency conversion service.
Limit service and currency conversion service can communicate with the database for retrieving the values of the limits and currencies conversion.
For more info check github.com/in28minutes and look after a microservice repository.
No matter how perfect the code of your microservice is, you may face issues with support and development if the microservice architecture doesn’t work according to certain
rules.
The following rules can help you with microservices a lot:
You have to do everything by yourself because you do not have any Rails and architecture out of the box that can be started by one command. Your microservice should load libraries, establish client connections, and be able to release resources if it stops working for any reason.
It means that being in the microservice folder and having made the 'ruby server.rb' command (a file for starting a microservice) we should make the microservice do the following:
Load used gems, vendor libraries (if used), and our own libraries
Use the configuration (depend on the environment) for adapters or classes of client connections
Establish client connections (permanent connections are meant here). As your microservice should be ready for any shutdowns, you should take care of closing these client connections at such moments. EventMachine and its callback mechanism helps a lot with this.
After that your microservice should be loaded and ready for work.
Incapsulate your communication with the services into abstractly named adapters. We name these adapters based on their role (PubSub, SMSMessenger, Mailer, etc.). This way, we can always change the inner implementation of these adapters by replacing the service if the names of our classes are service agnostic.
For example, we almost always use Redis in our application from the very beginning, thus it is also possible to use it as a message bus, so that we don’t have to integrate any other services. However, with the application growth we should think about solutions like RabbitMQ which are more appropriate for cases like ours.
If your code is designed in such a way that your classes are coupled with each other, do it according to the dependency inversion principle. This will help your code to avoid issues with lib booting.
Learn more here
You can try splitting an existing Monolithic application to gain perspective on microservice architecture.
I wrote this article, which talks about splitting a Django App into microservices. Hope it helps.
Let's consider a situation, where multiple services relay on data that can change any time and should be updated in each microservice roughly at the same time - for example there is a list of supported languages or some common policies that could change one day and affect many services at once.
One solution that I could think of is to have another microservice that could hold that data and any service that needs current state can just ask for it. The drawback is that this data is not changing very frequently, asking by HTTP is not that cheap and there is a lot of traffic to this let's say global registry service. As it is not changing very often, many services could just cache the data - in order to not ask for it every time - and not be able to respond to change quick enough when the change is made to the configuration.
The other solution could be to externalize such configuration - in AWS for example there could be some configuration file on S3 that would be available for others. The drawback here is that there is no way (as far as I know) to track changes in such file and there is no way to add some logic for verification if changed value in configuration is correct (there is no typos and so on), etc.
So my question is how to handle global configuration/registry in microservice world so that there is little HTTP overhead, you can audit changes as well as introduce change at the same time in many services?
I will prefer the option 1. Apart from the HTTP overhead, this will also lead your system in an inconsistent state. Service 1 might be working on new values but service 2 will be on old.
Since this is a distributed system that we are talking about, I am willing to take a risk with availability.
Have a configuration service that allows you to plan your config changes. Instead of saying change the value of A from x to y, you say change from x to y at time t. This t allows you to consistently propagate changes to all your system.You need to put in effort to understand what the min value of t should be for you set of services, how will you make all services acknowledge the changes and make them at the right time and how will you manage the new services that come up in between.
Another approach is use Spring Cloud Config (or something similar). It ask the service to register with the centralised config service and make refresh call to all the services to update config. Limitation being not all configs could be refreshed and if you are behind the LB you still need to handle ways to make sure all instances gets updated.
Use Config Server( spring cloud config server) that will maintain centralized configurations, you need to make changes to config server related to configurations, each microservices will come on startup for configurations to config server, even after start up after certain interval of time microservices can come to config server for validating any change in configurations and update accordingly.
There are couple of ways to do it, a better way especially in prod is to use external Configuration Store Pattern.
You can save the configuration in external stores like Azure Key Vault or Azure App configuration
Find more details about Azure key vault here:
Azure key vault
5-Minute quickstarts of Azure key vault integration
If you absolutely must have a shared config, best decoupled architecture I've encountered is as follows:
You have a standalone Config Service, completely private to the outside world and can only be accessed through an internal network for your microservices
ON STARTUP: Microservices do a pull request from the Config Service of what is needed per service and is stored in memory. if it is unable to pull from Config Service do not allow it to start. Have Retry Mechanism on this front.
ON CHANGE of the Config Service: Publish an event to your messaging layer that will force services to update their respective configurations.
Caveats:
do not put time sensitive configurations here, since we are using asynchronous communications here (if you have time critical configs why are they shared in the first place, you might need to revisit)
you need to handle your own plumbing, retry mechanism, memory management etc etc.
I'm new to parse and i've just setup my server and dashboard on my local machine.
For my use, i just not need the simple API from parse, i need to write a server (with NodeJS + Express) to handle users request.
I've just see how to integrate an Express application with parse, so my application instead of the server directly will use my server that will serve:
The standard Parse API (/classes etc)
All my others route, that could not to depend on Parse API
This is correct ?
Reading online i've see that Parse Cloud need to extend Parse functionality with additional "routing" (if i have understand well).
So, in my application i will have
The standard API (ad described up here)
All other routers (that could not depend on Parse)
Other routers (that come from Cloud) and use Parse API
So, Parse Cloud is just a "simple" way to write additional Routing ? (i've see that exists the job function too, but right now i've not studied it).
My question is just because i'm a little confused about the real needed, just would like to have more info on "when to use it"
Thanks
EDIT
I provide here an example (that in part come from Parse Docs).
I have a Video class with an director name field.
In my Application (iOs, Android etc) i setup a view that need to know all the Video provided from a particular director.
I will have three ways:
Get all Videos (/classes/videos) and then filter it directly in APP
Write an NodeJS + Express router endpoint (http://blabla.com/videos/XXX) where XXX is the director and then get the result with Parse JS API and send back it to the app
Write an Clound function (that if i have understand respond to /functions/) that do the same as the router one.
This is just a little example, but is this the usage of Parse Cloud ? (or at least, one on them :))