I currently have a controller that does a bit of heavy lifting processing (bulk csv file processing - cvs files ranging from 150Mb to 400Mb). The CSV files are uploaded to a temporary file location. Processing is done by a service that passes the file location to an APIs from an external jar (basic java API calling - no web service calls or anything). The service method takes about 2-3 times to return and the user has to wait currently for this time for the processing to complete and page to load after submitting a form - not the best user experience.
Grails users who have faced such a problem, what is the best solution to this kind of problem? I am new to Grails and JavaEE and hence this is basically a question on how one would architect such a system and the kind of libraries available for this.
I have googled quite a bit on this. People have responded with JMS, RabbitMQ etc as the solution to similar problems. But these appear to be swapping a fly with a bazooka kind of solution to my noob mind. Your suggestions are very much appreciated.
Thank you.
You can use the Spring #Async annotation on a service method if you want that method to be executed in a different thread. This is the approach I take in my Grails apps, it's dead easy.
There's an example of how to set it up here:
http://tux2323.blogspot.co.uk/2012/05/grails-and-spring-async-annotation.html?m=1
Use the quartz plugin... get the controller to schedule an immediate job (the scheduling is quick and the user will get a response straight away, and the processing will happen in a quartz job which runs in a different thread). Just notify the user when all the work is done (send an email or whatever).
Alternatively, use executor plugin to kick the job off in a new thread. 2.3 will have Async Support which could help here.
Related
Im facing an issue and was wondering if there is some library/framework or something to help me out.
Basically i have method in an API that creates an object for me but the problem is that this is not returned to me right away but is created later in time.
all i get is a guid on method call and have to manually check in the future if my object is created and if it isnt try again.
So my wish is to somehow automise this maybe? My thoughts were for using jobs or mqueues.
Any suggestions are really appreciated. The languages im allowed to use is nestjs or spring boot.
It sounds like you're looking to set up a dynamic cron job after your API kicks off the event, and then have those cron jobs possibly create more cron jobs or send out notifications. Not sure what the Spring Boot alternative would be, but a CRON is definitely what it sounds like you need (at least to me)
I'm currently working on a Spring Batch POC and have got a pretty good handle on most of the actual Spring Batch features. I've currently got a program that uses Spring Integration to receive an HttpRequest and use message channels to eventually send the job executions to the job launcher in a queue. What we'd really like to do is implement some kind of "scheduler/load balancer" (not quite sure what to call it) before the job launcher that will look at the currently running worker nodes and the size of the input file and make a decision on how many worker nodes the job should be allowed. We would probably also want to be able to change the amount of worker nodes a job has while it is running to allow more jobs to run.
The idea is that we'd have a server running that could accept many job requests at any time, and a large cluster of machines that jobs will be partitioned onto. We'd like to be able to scale horizontally, so whenever the server isn't busy it can make full use of the hardware, as well as being able to make sure that small jobs don't get constantly blocked by larger jobs.
From my research it seems like we'd have to implement another framework to do this (do GridGain and Hadoop allow this?), but I figured I'd ask to see what people recommended to do something like this, and if there's a way to do it without implementing another large framework.
Sorry if anything is unclear or confusing, I'm just a lowly intern who started learning Spring and Spring Batch last month and I'm far from completely understanding everything, especially this scaling stuff. Just ask and I'll try to clear things up.
Thanks for any help!
Take a look at the 'spring-batch-integration' project under the spring-batch-admin umbrella project https://github.com/SpringSource/spring-batch-admin
It has a number of examples of using spring-integration to distribute work to other nodes. IN particular see the chunk and partition packages. Just swap out the spring integration channels with jms channel adapters. By distributing work partitions via JMS, you can scale out the number of worker nodes as needed.
There are a number of threads on this subject in the spring integration forum; search for 'PartitionHandler'.
Hope that helps.
I have created apps in the past that would have web pages that would call the persistence layer to get some query results or to insert, delete, etc against a db. However, nothing was left running in the background except for the persistence layer. Now I need to develop an app that has an process that is always running in the background, which is waiting for messages to come thru a zeromq messaging system (cannot change this at this point). I am a little lost as to how to setup the object so that it can always be running and yet I can control or query the results from the object.
Is there any tutorial/examples that covers this configuration?
Thanks,
You could use some kind of timer, to start a method every second to look at a specific ressource and process the input taken from that.
If you use Spring than you could have a look at the #Scheduled annotation.
If your input is some kind of java method invokation, than have a look at the java.util.concurrent Package, and concurrent programming at all. -- But be aware of the fact, that there are some restictions one creating own Threads in an EJB environment.
We Want to Create an online game like this.I think that ,this type of games have a scheduling software on web server. For Example : Player Click to create a resource And resource creation will be take a moment like 20 Minutes.(Every resource creation time will be different). This message will send to web server application but this message will not processed at same time for example must be processed after 20 Minutes. The web server application after getting the message must be put the order in the Queue.
We have Some big problems :
1- The Jobs must be complete by the web server application Even the player Exit the Game. I think that we must create something like Windows service on Web Server. Can we do it? or Is there a better way?
2- The Second problem depended on problem 1 .Because we will have many Jobs (every player can create 20,30 Jobs in every Loggin and we will have thousands of users) , So Our Scheduling System Must be Work On time . it's possible that , there is 100 , 1000 jobs in a same second , if application Can't Done Jobs in him Second will be use the next second Time and the next second jobs will shift to next second and etc. How We can do for this problem ?
Platform : .Net 3.5 On Windows 2003 Web server
Cheers
Asad Safari
Agile Coach , Scrum Master
On unix, use cron to schedule a script to run every minute - the script then handles all the jobs.
http://en.wikipedia.org/wiki/Cron
Well the specific answer would depend on which technology/language/platform you're using. In java, check the scheduling services provided by Quartz to schedule jobs; and check JMS/MDBs to implement the asynchronous event processing you mention in question 2.
I think my answer won't exactly fit your question but it will solve your problem. Both points at once.
First at all, why do you want to automatize the execution of this events? As far as I understand, you need to give the player access to his resource if and only if he created the resource an especific time ago.
Well, this is my solution:
Insert a field in your resource model class named usable_since or something like that and, when the player create his resource, just set it the value of this field to the current time plus the desired waiting time.
In this way you beat the two problems:
the resource will be usable even if the player gets offline and
you just won't need to set up so big job handler, even with a million users.
I hope it helps. I wish you the best luck in developing the game. Make it fun!
I had a somewhat similar question, as I'm also developing an application which relies on server-side scheduling: Job-Scheduling in Play! Framework. I use the Play Framework (Java) and it works like a charm; except I have no experience in how many jobs can be scheduled concurrently without bringing the server to its knees.
This is my setting: I have written a .NET application for local client machines, which implements a feature that could also be used on a webpage. To keep this example simple, assume that the client installs a software into which he can enter some data and gets some data back.
The idea is to create a webpage that holds a form into which the user enters the same data and gets the same results back as above. Due to the company's available web servers, the first idea was to create a mono webservice, but this was dismissed for reasons unknown. The "service" is not to be run as a webservice, but should be called by a PHP script. This is currently realized by calling the mono application via shell_exec from PHP.
So now I am stuck with a mono port of my application, which works fine, but takes way too long to execute. I have already stripped out all unnecessary dlls, methods etc, but calling the application via the command line - submitting the desired data via commandline parameters - takes approximately 700ms. We expect about 10 hits per second, so this could only work when setting up a lot of servers for this task.
I assume the 700m are related to the cost of starting the application every time, because it does not make much difference in terms of time if I handle the request only once or five hundred times (I take the original input, vary it slighty and do 500 iterations with "new" data every time. Starting from the second iteration, the processing time drops down to approximately 1ms per iteration)
My next idea was to setup the mono application as a remoting server, so that it only has to be started once and can then handle incoming requests. I therefore wrote another mono application that serves as the client. Calling the client, letting the client pass the data to the server and retrieving the result now takes 344ms. This is better, but still way slower than I would expect and want it to be.
I have then implemented a new project from scratch based on this blog post and get stuck with the same performance issues.
The question is: am I missing something related to the mono-projects that could improve the speed of the client/server? Although the idea of creating a webservice for this task was dismissed, would a webservice perform better under these circumstances (as I would not need the client application to call the service), although it is said that remoting is faster than webservices?
I could have made that clearer, but implementing a webservice is currently not an option (and please don't ask why, I didn't write the requirements ;))
Meanwhile I have checked that it's indeed the startup of the client, which takes most of the time in the remoting scenario.
I could imagine accessing the server via pipes from the command line, which would be perfectly suitable in my scenario. I guess this would be done using sockets?
You can try to use AOT to reduce the startup time. On .NET you would use ngen for that purpoise, on mono just do a mono --aot on all assemblies used by your application.
AOT'ed code is slower than JIT'ed code, but has the advantage of reducing startup time.
You can even try to AOT framework assemblies such as mscorlib and System.
I believe that remoting is not an ideal thing to use in this scenario. However your idea of having mono on server instead of starting it every time is indeed solid.
Did you consider using SOAP webservices over HTTP? This would also help you with your 'web page' scenario.
Even if it is a little to slow for you in my experience a custom RESTful services implementation would be easier to work with than remoting.