I have a kernel distribution object which I can use to generate random numbers in matlab easily by using the random function. However I need to do the same now in a simulink simulation, here I'm at a loss as the different approaches I've tried have all generated an error. What would be the simplest way to achieve this?
These are what my attempts have been:
I first tried to use the from workspace block in simulink, the problem with this was that the from workspace block only imports timeseries object or structs with a specific format. kernel distribution objects are not supported.
Second attempt I've tried to use the the data memory store block but that doesn't seem to support objects either.
Related
I want to describe a Matrix multiplier with Chisel, but there are some things that I do not understand.
First, I found that response giving the code of a 3X5 matrix multiplier. I would like to generalize it for any square matrix up to 128X128. I know that in Chisel, I can parameterize a module by giving a size parameter to the module (so that I'll use n.W instead of a defined size).
But at the end of the day, a Verilog file will be generated right ? So the parameters have to be fixed ? I probably confuse some things. My prupose is to adapt the code to be able to perform any matrix multiplication up to 128x128, and I do not know if it is technically possible.
The advantage of chisel is that everything can be parameterized. That being said at the end of the day when you are making your physical hardware obviously the parameter should be fixed. The advantage of making it parameterized is that if you don't know your exact requirements (like area of the die available etc) you can have a parameterized version ready and when the time comes you plug in the values you need and generate the verilog file for that parameter. And to answer your question, yes it is possible to perform any matrix multiplication up to 128x128 (or beyond if your Laptop RAM is sufficient). You get the verilog only when you create a Hardware driver this tells you how to create verilog from chisel so go ahead and create your parameterized hardware.
I am trying to use lightGBM as a classifier. My data are saved in multiple csv files, but I find there is no way to directly use multiple files as the input.
I have considered to combine all the data into a big one (numpy array), but my computer doesn't have enough memory. How can I use lightGBM to deal with multiple data files when the avaliable memory is poor?
I guess that you are using Python.
What is the size of your data ? (num of rows x num of columns)
Lightgbm will need to load the data in-memory for training.
But if you haven't done it yet, you can wisely choose a suitable datatype for every column of your data.
It can considerably reduce the memory footprint if you use dtypes such as 'uint8' / 'uint16' and help you load everything in memory.
Sample.
You shouldn't ever (Except certain edge cases) need to use your entire dataset if you sample CORRECTLY.
I use a DB that has over 230M records but I usually only select a RANDOM sample of anywhere from 1k-100k to create the model.
Also, you might as well split your data into training, testing and validation. That will help cut down the size per file.
You might want to categorize your features, then to one-hot-encode them. LightGBM works best with sparse features such as one-hot-encoded ones due to its EFB (Effective Feature Bundling) which enhances computation efficiency of LightGBM significantly. Moreover, you will definitely get rid of the floating parts of the numbers.
Think categorization as that; let’s say that values of one of the numerical features vary between 36 to 56, you can digitize it as [36,36.5,37,....,55.5,56] or [40,45,50,55] to make it categorical. Up to your expertise and imagination. You can refer to scikit-learn for one-hot-encoding, it has built-in function for that.
PS: With a numerical feature, always inspect the statistical properties of it, you can use pandas.describe() which summarizes its mean, max, min, std etc.
I'm working with embedded systems. For the sake of explanation, I'm working with a dsPIC33EP and a simple serial EEPROM.
Suppose I'm building a controller that uses a linear control scheme (y=mx+b). If the controller needs may different setting It's easy, store the m and the b in EEPROM and retrieve it for the different settings.
Now suppose I want to have different equations for different settings. I would have to pre program all the equations and then have a method for selecting that equation and pulling the settings from the EEPROM. It's harder because you need to know the equations ahead of time but still doable.
Now suppose that you don't know the equations ahead of time. Maybe you have to do a piece wise approximation for example. How could you store something like that in memory? That all a controller has to do is feed it a sensor reading and it would give back a control variable. Kind of like passing a variable to a function and getting the answer passed back.
How could you store a function like that in memory if only the current state is important?
How could you store a function like that if past states are important (if the control equation is second, third or fourth order for example)?
The dsPICs have limited RAM, but quite a bit of FLASH, enough for a small, but effective text parser. Have you thought of using some form of text based script? These can be translated to a more efficient data format at run-time.
For a project I am undertaking, I will need to calculate the derivative of a given surface brightness profile which is convolved with a pixel response function (as well as PSF etc.)
For various reasons, but mainly for consistency, I wish to do this using the guts of the GALSIM code. However, since in this case the `flux' defined as the sum of the non-parametric model no longer has a physical meaning in terms of the image itself (it will always be considered as noise-free in this case), there are certain situations where I would like to be able to define the interpolated image without a flux normalisation.
The code does not seem to care if the 'flux' is negative, but I am coming across certain situations where the 'flux' is within machine precision of zero, and thus the assertion ``dabs(flux-flux_tot) <= dabs(flux_tot)'' fails.
My question is therefore: Can one specify a non-parametric model to interpolate over without specifying a flux normalisation value?
There is currently no way to do this using the galsim.InterpolatedImage() class; you could open an issue to make this feature request at the GalSim repository on GitHub.
There is a way to do this using the guts of GalSim; an example is illustrated in the lensing power spectrum functionality if you are willing to dig into the source code (lensing_ps.py -- just do a search for SBInterpolatedImage to find the relevant code bits). The basic idea is that instead of using galsim.InterpolatedImage() you use the associated C++ class, galsim._galsim.SBInterpolatedImage(), which is accessible in python. The SBInterpolatedImage can be initialized with an image and choices of interpolants in real- and Fourier-space, as shown in the examples in lensing_ps.py, and then queried using the xValue() method to get the value interpolated to some position.
This trick was necessary in lensing_ps.py because we were interpolating shear fields which tend to have a mean of zero, so we ran into the same problem you did. Use of the SBInterpolatedImage class is not generally recommended for GalSim users (we recommend using only the python classes) but it is definitely a way around your problem for now.
On my test set, the observed variables are not the same for each data point. A given variable can be observed on a data point, and not on the next one. Thus I would like to change the observed flag of those variables without reconstructing the full PyMC model. I read that it wasn't possible (and couldn't manage to do it). Is there any way to do it?
I thus decided to rebuild a PyMC model for each of my test set data point. I instantiate a new PyMC model at each iteration of a for loop.
The problem is that it seems that the memory used by each PyMC model is not deleted. The fact is that my network is huge (1000 binomial/sigmoid nodes) and densely connected. The model takes about 200MB (just the model, without the traces). I am wondering if maybe the python garbage collector wasn't able to delete it because of the numerous circular references between the PyMC nodes of my network.
What do you think? Do you see a proper way to do such a thing?
If you are rebuilding the PyMC model for each data point, then presumably you are not using the built-in samplers (e.g. MCMC). In that case, you can use the set_value() method of the nodes you need to set at each iteration and then call model.draw_from_prior() to get draw a random value for the other nodes.
In other words, instead of using observed=True, you can create your nodes with observed=False and then manually fix the value with set_value().