Here is my question:
Is tere any service or technology to run parallel algorythms on more computer without knowing them?
For example: I write a parallel algorythm. My friends install a simple client app, and if they have internet connection, they can help my calculation with their free processor capacity. I would like to see them like an additional core in my CPU.
If there is no technology like that, is there any unsolvable problems with developing one? (I know there must be a lot problems with code trasfering, operation systems, and compatibility)
I believe that you can use BOINC to set up your own volunteer computing project. But I have no experience of this to report.
Related
I'm looking to develop a program that detects calls to certain Windows API functions and simply records the calling process, call count, and hopefully their arguments, to later mark them as benign or malicious.
The GUI program API monitor is a good example of the functionality I'm trying to achieve. Ideally I would like to track each desired API function individually and get the caller PID and parameters when or after it is used, without user input. The program should be able to run on any windows 7 machine, but can be limited to 32bit applications.
I understand there are several methods of hooking a function, and from my understanding Microsoft detours implements one of these, but I don't know if its the one best suited to what I want to do. I've seen detours, easyhook, deviare API hook, and others mentioned on very old posts, but I have a hard time getting my head around the differences and features of each.
So my question is, given what I'm trying to do, what do you recommend and why?
For reference I'm an intermediate level programmer, but a beginner at Windows programming.
Thanks for your help
I'm part of Nektra Deviare API Hook team. Our hooking engine is used by a large number of companies all over the world, in different types of end-user products (e.g.: Anti Virus, Data Loss Prevention, AI, handicapped software, Data Classification, App virtualization).
Deviare-InProc is the MS Detours replacement and Deviare2 has built in all the RPC you need to hook another process and get the calls in your own process.
We continuously fix all reported issues. You can verify it in our GitHub:
https://github.com/nektra/Deviare2
https://github.com/nektra/Deviare-InProc
You can see Deviare2 running in Nektra's SpyStudio API Monitor.
Detours is an excellent software but very expensive (USD 10k). In addition to this, it completely lacks of support. It can be compared to Deviare InProc,
EasyHook used to be a good start point because it was the only free option. But, now Deviare2 family is open source and EasyHook has a lot of stability issues for the real world.
I have a question about a family of softwares, of which one example is INtime, which lets you run a real-time operating system in parallel with Windows.
I have a reasonable grasp on how Windows works, including kernel/driver/application security rings etc. Similarly, I know how a RTOS runs on a dedicated system.
The Simple Question:
How do these go about existing together without fighting over hardware or other similar problems? How is the allocation or resources made, and how is this integrated with Windows?
Slightly more complicated:
What are the steps I would have to take if I wanted to develop something similar myself? Are there some open-source embodiment's of this paradigm I can inspect to glean some more understanding?
I am developing socket reading on an ActiveX EXE (i.e on a seperate thread).
How many sockets i can safely read independently?
I am working on windows XP OS.
I think this might be an operating system limit - I think I remember running up against a limit of 80 sockets on the XP machine I was using several years ago.
I would recommend that you abandon your effort and go with a commercial solution. I remember going down this path back in the 90s and running into a brick wall with ActiveX EXEs as far as threading goes. The thing is that ActiveX EXEs are apartment threaded, not free-threaded, so you don't get completely independent threads.
And doing server side threads properly is hard enough in modern languages, let alone ones that weren't designed for this purpose.
I ended up purchasing Server Sockets from Dart. Easily the best investment for that project. The performance is truly great - you are only limited by the system resources.
MSWINSCK.OCX is a very old way of doing things; it came with Visual Basic 6.0 and i remember using it way back when. i'm not sure the licensing on it... apparently it registers fine under 32-bit win7, but not 64-bit; here's a link to how to get it to register on 64-bit systems: http://angrybyte.com/windows-hacks/mswinsck-ocx-for-64-bit-windows-7-vista/
if you have an MSDN subscription or similar that gives you the ability to download the developer tools (bizSpark, etc. will do it too) then i believe that will also give you a license to redistribute the .ocx.
(btw, i don't actually remember the interface, but i seem to remember it being at least slightly more intuitive than the berkeley socket() interfaces.)
however, personal recommendation given your requirements: learn the APIs, there are lots of examples out there, and just write yourself a class that encapsulates them in a similar way as, say, the .NET Socket class... the APIs aren't that hard and i'm sure there's lots of help to be had here as well, and that's probably better than relying on something that's out-of-date like the control...
My question is one I have pondered around when working on a demanding network application that would explicitly share a task across the network using a server to assign the job to each computer individually and "share the load".
I wondered: could this be done in a more implicit manner?
Question
Is there a possibility of distributing processor intensive tasks around a voluntary and public network of computers to make the job run more efficiently without requiring the job's program or process to be installed on each computer?
Scenario
Lets say we have a ridiculously intensive mathematics scenario where I am trying to get my computer to calculate every prime factorization break down for all numbers from 1 to 10,000,000 and store them in a database (assuming I have the space and that the algorithms are already implemented in their own class, program, dynamic link library or any runnable process.)
Now it would be more efficient to share this burdening process across a network or on a multi-core super computer, however these are both expensive. To my knowledge you would require a specifically designed program to run the specific algorithm and have the program installed across the said cloud/distributed computing network whilst you have a server keep track of what each computer is doing (ie. what number they are currently calculating the primes for).
Conclusion
Overall:
Would it be possible to create a cloud program / OS / suite
where you could share processor time
for an unspecified type of process?
If so how would you implement it, where would you start?
Would you make an OS dedicated to being able to run unspecified non-explicit tasks or would it be possible to do with a cloud enabled program installed on volunteers computers volunteers who were willing to share a percentage of their processor clock to help the general community).
If this was implementable, would you be a voluntary part of the greater cloud?
I would love to hear everyone's thoughts and possible solutions as this would be a wonderful project to start.
I have been dealing with the same challenge for the last few months.
My conclusions thus far:
The main problem with using a public network (internet) for cloud computing is in addressing the computers through NATs and firewalls. Solving this is non-trivial.
Asking all participants to open ports in their firewalls and configure their router for port-forwarding is generally too much to ask for 95% of users and can pose severe security threats.
A solution is to use a registration server where all peers register themselves and can get in contact with others. Connections are kept open by server and communication is routed through server. There are several variations. However, in all scenario's, this requires vast resources to keep everything scalable and is therefore out of reach for anyone but large corporations.
More practical solutions are offered by commercial cloud platforms like Azure (for .Net) or just the .Net ServiceBus. Since everything runs in the cloud, there will be no problems with reaching computers behind NATs and firewalls, and even if you need to do so to reach "on-premise" computers or those of clients, this can be done through the ServiceBus.
Would you trust someone else's code to run on your computer?
Its more practical to not ask their permission: ;)
I once wrote a programming competition solver written with Haxe in a Flash banner on a friend's fairly popular website...
I would not expect a program which allow "share processor time for an unspecified type of process". An prerequisite of sharing processor time is that the task can divided into multiple sub tasks.
To allow automatic sharing of processor time for any dividable task, the key of solution would be an AI program clever enough to know how to divide a task into sub task, which does not seems realistic.
How can i connect two or more machines to form a network grid and how can i distribute work load to the two machines?
What operating systems do i need to run on the machines, and what application should i use to manage the load balancing?
NB: I read somewhere that google uses cheap machines to perform this fete, how do they connect two network cards( 'Teaming' ) and distribute load across the machines?
Good practical examples would serve me good, with actual code samples.
Pointers to some good site i might read this stuff will be highly appreciated.
An excellent place to start is with the Beowulf project. Basically an opensource cluster built on the Linux OS.
There are several software solutions in this expanding market. The term "cloud computing" is certainly gaining traction to describe what you want to do. Are you wanting a service, or do you want to run it in house?
I'm most familiar with Appistry EAF - Runs on commodity based hardware. Its available as a free download. Runs on windows or linux.
Another is GoGrid - I believe this is only available as a service, but I'm not as familiar with it.
There are many different approaches to parallel processing, and many types of system architectures you could use.
For commodity systems, there are clusters and grids, or you can even form a single system image from several pieces of commodity hardware. There is of course also load balancing, high availability, failover, etc.
It's pretty much impossible to answer this question without more detail. What exactly do you want to with these systems? The answer is very highly dependent on the application.
You might want to have a look at some of the stuff to do with Folding At Home and the SETI project, and some of their participants blogs, here is a pretty amazing cluster that a guy built in an IKEA cabinet:
http://helmer.sfe.se/
Might give you some ideas.
The question is too abstract.
One of the (imaginable) ways is to use MPI - a framework for parallel programming, the Wikipedia page includes examples in C++, and there are bindings for other languages.