Is the FMU for Model Exchange code (generated by OpenModelica) parallelized?
If I want parallelization, must I run different FMUs for Co-Simulation in parallel?
Thanks
Is a bit unclear what you want:
Do you want automatic parallelization of the simulation code inside the FMU?
we have some support for this using OpenMP and also some experimental support using TBB.
Do you want to run the FMUs in parallel on different threads or different processes?
we are working on a tool for FMI co-simulation called OMSimulator which will include some parallelization at the master-algorithm level
Related
I have a processor that generates time series data in JSON format. Based on the received data I need to make a forecast using machine learning algorithms on python. Then write the new forecast values to another flow file.
The problem is: when you run such a python script, it must perform many massive preprocessing operations: queries to a database, creating a complex data structure, initializing forecasting models, etc.
If you use ExecuteStreamCommand, then for each flow file the script will be run every time. Is this true?
Can I make in NIFI a python script that starts once and receives the flow files many times, storing the history of previously received data. Or do I need to make an HTTP service that will receive data from NIFI?
You have a few options:
Build a custom processor. This is my suggested approach. The code would need to be in Java (or Groovy, which provides a more Python-like experience) but would not have Python dependencies, etc. However, I have seen examples of this approach for ML model application (see Tim Spann's examples) and this is generally very effective. The initialization and individual flowfile trigger logic is cleanly separated, and performance is good.
Use InvokeScriptedProcessor. This will allow you to write the code in Python and separate the initialization (pre-processing, DB connections, etc., onScheduled in NiFi processor parlance) with the execution phase (onTrigger). Some examples exist but I have not personally pursued this with Python specifically. You can use Python dependencies but not "native modules" (i.e. compiled C code), as the execution engine is still Jython.
Use ExecuteStreamCommand. Not strongly recommended. As you mention, every invocation would require the preprocessing steps to occur, unless you designed your external application in such a way that it ran a long-lived "server" component and each ESC command sent data to it and returned an individual response. I don't know what your existing Python application looks like, but this would likely involve complicated changes. Tim has another example using CDSW to host and deploy the model and NiFi to send it data via HTTP to evaluate.
Make a Custom Processor that can do that. Java is more appropriate. I believe you can do pretty much every with Java you just need to find libraries. Yes, there might be some issues with some initialization and preprocessing that can be handled by all that in the init function of nifi that will allow you preserve the state of certain components.
Link in my use case I had to build a custom processor that could take in images and apply count the number of people in that image. For that, I had to load a deep learning model once in the init method and after through on trigger method, it could be taking the reference of that model every time it processes an image.
I am trying to optimize an airfoil using openMDAO and SU2. I have multiple Designpoints that i want to run in parallel. I managed to do that with a "Parallel Group" and XFoil. But i now want to use SU2 instead of XFoil.
The Big Problem is, SU2 by itself, is started by MPI (mpirun-np 4 SU2_CFD config.cfg). Now i want openMDAO to divide all the available processes evenly to all DesignPoints. And then run one SU2 instance per Designpoint. Every SU2 instance should then use all the processes that openMDAO allocated to that DesginPoint.
How could i do that?
Probably wrong approach:
I played around with the external-code component. But if this component gets 2 processes, it is run twice. I dont want to run SU2 twice. I want to run it once, but using both available processes.
Best Regards
David
I don't think your approach to wrapping SU2 is going to work, if you want to run it in parallel as part of a larger model. ExternalCodeComp is designed for file-wrapping and spawns sub-processes, which doesn't give you any way to share MPI communicators with the parent process (that I know of anyway).
Im not an expert in SU2, so I can't speak to their python interface. But Im quite confident that ExternalCodeComp isn't going to give you what you want here. I suggest you talk to the SU2 developers to discuss their in-memory interface.
I couldn't figure out a simple way. But I discorvered ADflow: https://github.com/mdolab/adflow.
It is a CFD-Solver that comes shipped with an OpenMDAO-Wrapper. So I am going to use that.
I have a question about processing the image while driving a motor. I did some researches, probably I need to use multiprocessing. However, I couldn't find out how to run two processors together.
Let's say I have two functions as imageProcessing() and DrivingMotor(). With coming information from imageProcessing(), I need to update my DrivingMotor() function simultaneously. How can I handle this issue?
In multiprocessing, you must create two process(process means program in execution) and must implement interproccesing communication methods to communicate process each other, this is tedious,hard and inefficient way .Multiproccesing less efficient than multithreading.Therefore I think you should multithread ,it is very efficient way ,communication between thread is very easy, you can use global data for communication.
You shall create two threads, one thread is handle imageProcessing() ,and other thread DrivingMotor().Operating system handled execution of thread,Operating system run synchronous these threads.
there is basic tutorial for multithreading below links
https://www.tutorialspoint.com/python/python_multithreading.htm
I need to write a program which does 2 tasks at the same time for better efficiency & high response. First task is, for example, get vision data from a camera & process it.
Second task is, receive processed data from first task & do sth else with this data (robot control strategy). However, while robot control task is being performed, the camera data receiving should still be working.
Is there a solution for such type of programming in C++/C#?? I'm learning TBB, is it the right choice? However, I'm reading things like "loop parallelization", am I going in the right direction??
This links to a very common style in control programming where the computer is used as a central unit to connect to electronic devices (sensors) & actuators and all these devices are processed concurrently
No, your example of loop paralleling is using parallel programming to speed up the result of a calculation for one set of data.
What you need is multitasking. You didn't mention any target architecture. Assuming this will be an embedded system, like a microprocessor, you have several options. There are embedded micro-OSes like VXworks and uC-OS that allow you to do just what you are asking. These allow you to set up multiple "tasks" that run virtually concurrently. Of course true concurrency is impossible with one CPU, but the scheduler in these OSes is designed to be very deterministic, for quasi-real-time systems like you describe.
Sounds good to me! TBB OK, C# has useful threadpool etc. classes. Just one thing, if you haven't done anything like this before - it's all about the data, not the code. If you design the data flow correctly, the code will write itself, (well OK, not really:).
I need to do some network bound calls (e.g., fetch a website) and I don't want it to block the UI. Should I be using NSThread's or python's threading module if I am working in pyobjc? I can't find any information on how to choose one over the other. Note, I don't really care about Python's GIL since my tasks are not CPU bound at all.
It will make no difference, you will gain the same behavior with slightly different interfaces. Use whichever fits best into your system.
Learn to love the run loop. Use Cocoa's URL-loading system (or, if you need plain sockets, NSFileHandle) and let it call you when the response (or failure) comes back. Then you don't have to deal with threads at all (the URL-loading system will use a thread for you).
Pretty much the only time to create your own threads in Cocoa is when you have a large task (>0.1 sec) that you can't break up.
(Someone might say NSOperation, but NSOperationQueue is broken and RAOperationQueue doesn't support concurrent operations. Fine if you already have a bunch of NSOperationQueue code or really want to prepare for working NSOperationQueue, but if you need concurrency now, run loop or threads.)
I'm more fond of the native python threading solution since I could join and reference threads around. AFAIK, NSThreads don't support thread joining and cancelling, and you could get a variety of things done with python threads.
Also, it's a bummer that NSThreads can't have multiple arguments, and though there are workarounds for this (like using NSDictionarys and NSArrays), it's still not as elegant and as simple as invoking a thread with arguments laid out in order / corresponding parameters.
But yeah, if the situation demands you to use NSThreads, there shouldn't be any problem at all. Otherwise, it's cool to stick with native python threads.
I have a different suggestion, mainly because python threading is just plain awful because of the GIL (Global Interpreter Lock), especially when you have more than one cpu core. There is a video presentation that goes into this in excruciating detail, but I cannot find the video right now - it was done by a Google employee.
Anyway, you may want to think about using the subprocess module instead of threading (have a helper program that you can execute, or use another binary on the system. Or use NSThread, it should give you more performance than what you can get with CPython threads.