I am using the prophet model to perform anomaly detection on the metrics of prometheus
However, I found that the yhat of the metrics output by anomaly detection is basically stable at around 0.6, and does not change with the original data.
The figure below is the predicted metrics versus the original metrics in grafana
predicted metrics versus original metrics
Green bar is : node_load1{instance='192.29.3.11:9100'} (origin metrics)
yellow Bar is : yhat.
Orange Bar is :yhat_upper
Sky blue bar is : yhat_lower
Red Bar is : anomaly
It can be seen that the value of yhat does not change with the original value, but is stable at around 0.6
The original training data is like this:
original training data
model.plot_components(forecast)
prophet components
model.plot(forecast)
prophet forecast
How should I adjust the prophet parameters, or is there any other way to make Prophet predict the value accurately？
please help me
Related
I am running a simulation of capacity about different parking lots located according to GIS map tool in AnyLogic. I added a bar chart for Parking1.statsUtilization.max()and histogram for time measure end timeTotal.distribution. But when simulation is running I can not see the bar chart and histogram chart into the simulation. I am not so sure if the problem is because I am using GIS location tool
I would like to density plot of some vectors (some realizations in probabilistic concept) and also their confidence interval in Matlab, enter image description herelike the figure I attached. The grey shadows are the data and the dashed lines are the confidence interval. Any suggestions?
Thanks a lot.
This can be done easily using Gramm toolbox avaialbale at https://github.com/piermorel/gramm/blob/master/README.md
The link also gives a minimal working example as well as information on more advanced control of the figure.
The above image is the H2O GBM classification model lift chart for training and validation data sets. I am confused it with the other lift charts I have seen. Normally the baseline will be 45 degrees and the lift curve used to be somewhat convex shape from the baseline curve. In the above figure if the green line shows the lift curve, why is it constant and coming down and touches the baseline? Also why the baseline is not 45 degree? Can anyone help me to interpret the model using the above graph? Is my model perform well?
The black line is not the baseline, but the cumulative capture rate. The capture rate is the proportion of all the events that fall into the group/bin. E.g. if 90 out of total 100 positive outcomes/events fall into the first bin, then the capture rate for that bin is 0.9.
The green line is the cumulative lift curve, so by definition the two lines converge at 1.
Whether your model performs well or not depends on your goal. According to the validation metrics, you could capture about 80% of the events by targeting only 50% of the population, which means lift of about 1.6.
With some data loaded into ES using LS, is there a way to mention in Kibana a threshold and display an indicator it breaches the threshold. For example number of transaction in an hour across the system lets set to 200. When gets refreshed the dashboard, it should indicate in red or something when it goes beyond 200 in a time line graph or some visualization.
So If we see for whole day some hours will be red or spikes and most will be green or lower lines. This will be useful for us to display capability of anamoly detection. We are still working to understand Prelert but this kind of detection in Kibana will help us to see in graphs.
Regards,
Raj
There's no such direct threshold mechanism which you could apply on the Kibana Visualizations as the issue's still open.
But then a workaround for this could be to go with a scripted field in Kibana so that you could add up a constant value and use it as a margin in your visualization. Note that The scripted field may work for line visualizations, but it isn't compatible with an area visualization using sub-aggregations.
OR you could use a filter and stacked lines with having one color for above threshold and another color for the ones which under the threshold.
You could still follow up on this ticket and this SO could be handy. Hope it helps!
I am working on a software (Ruby - Sketchup) to calculate the radiation (sun, sky and surrounding buildings) within urban development at pedestrian level. The final goal is to be able to create a contour map that shows the level of total radiation. With total radiation I mean shortwave (light) and longwave(heat). (To give you an idea: http://www.iaacblog.com/maa2011-2012-digitaltools/files/2012/01/Insolation-Analysis-All-Year.jpg)
I know there are several existing software that do this, but I need to write my own as this calculation is only part of a more complex workflow.
The (obvious) pseudo code is the following:
Select and mesh surface for analysis
From each point of the mesh
Cast n (see below) rays in the upper hemisphere (precalculated)
For each ray check whether it is in shade
If in shade => Extract properties from intersected surface
If not in shade => Flag it
loop
loop
loop
The approach above is brute force, but it is the only I can think of. The calculation time increases with the fourth power of the accuracy (Dx,Dy,Dazimth, Dtilt). I know that software like radiance use a Montecarlo approach to reduce the number of rays.
As you can imagine, the accuracy of the calculation for a specific point of the mesh is strongly dependent by the accuracy of the skydome subdivision. Similarly the accuracy on the surface depends on the coarseness of the mesh.
I was thinking to a different approach using adaptive refinement based on the results of the calculations. The refinement could work for the surface analyzed and the skydome. If the results between two adjacent points differ more than a threshold value, than a refinement will be performed. This is usually done in fluid simulation, but I could not find anything about light simulation.
Also i wonder whether there are are algorithms, from computer graphics for example, that would allow to minimize the number of calculations. For example: check the maximum height of the surroundings so to exclude certain part of the skydome for certain points.
I don't need extreme accuracy as I am not doing rendering. My priority is speed at this moment.
Any suggestion on the approach?
Thanks
n rays
At the moment I subdivide the sky by constant azimuth and tilt steps; this causes irregular solid angles. There are other subdivisions (e.g. Tregenza) that maintain a constant solid angle.
EDIT: Response to the great questions from Spektre
Time frame. I run one simulation for each hour of the year. The weather data is extracted from an epw weather file. It contains, for each hour, solar altitude and azimuth, direct radiation, diffuse radiation, cloudiness (for atmospheric longwave diffuse). My algorithm calculates the shadow mask separately then it uses this shadow mask to calculate the radiation on the surface (and on a typical pedestrian) for each hour of the year. It is in this second step that I add the actual radiation. In the the first step I just gather information on the geometry and properties of the various surfaces.
Sun paths. No, i don't. See point 1
Include reflection from buildings? Not at the moment, but I plan to include it as an overall diffuse shortwave reflection based on sky view factor. I consider only shortwave reflection from the ground now.
Include heat dissipation from buildings? Absolutely yes. That is the reason why I wrote this code myself. Here in Dubai this is key as building surfaces gets very, very hot.
Surfaces albedo? Yes, I do. In Skethcup I have associated a dictionary to every surface and in this dictionary I include all the surface properties: temperature, emissivity, etc.. At the moment the temperatures are fixed (ambient temperature if not assigned), but I plan, in the future, to combine this with the results from a building dynamic thermal simulation that already calculates all the surfaces temperatures.
Map resolution. The resolution is chosen by the user and the mesh generated by the algorithm. In terms of scale, I use this for masterplans. The scale goes from 100mx100m up to 2000mx2000m. I usually tend to use a minimum resolution of 2m. The limit is the memory and the simulation time. I also have the option to refine specific areas with a much finer mesh: for example areas where there are restaurants or other amenities.
Framerate. I do not need to make an animation. Results are exported in a VTK file and visualized in Paraview and animated there just to show off during presentations :-)
Heat and light. Yes. Shortwave and longwave are handled separately. See point 4. The geolocalization is only used to select the correct weather file. I do not calculate all the radiation components. The weather files I need have measured data. They are not great, but good enough for now.
https://www.lucidchart.com/documents/view/5ca88b92-9a21-40a8-aa3a-0ff7a5968142/0
visible light
for relatively flat global base ground light map I would use projection shadow texture techniques instead of ray tracing angular integration. It is way faster with almost the same result. This will not work on non flat grounds (many bigger bumps which cast bigger shadows and also change active light absorbtion area to anisotropic). Urban areas are usually flat enough (inclination does not matter) so the technique is as follows:
camera and viewport
the ground map is a target screen so set the viewpoint to underground looking towards Sun direction upwards. Resolution is at least your map resolution and there is no perspective projection.
rendering light map 1st pass
first clear map with the full radiation (direct+diffuse) (light blue) then render buildings/objects but with diffuse radiation only (shadow). This will make the base map without reflections and or soft shadows in the Magenta rendering target
rendering light map 2nd pass
now you need to add building faces (walls) reflections for that I would take every outdoor face of the building facing Sun or heated enough and compute reflection points onto light map and render reflection directly to map
in tis parts you can add ray tracing for vertexes only to make it more precise and also for including multiple reflections (bu in that case do not forget to add scattering)
project target screen to destination radiation map
just project the Magenta rendering target image to ground plane (green). It is only simple linear affine transform ...
post processing
you can add soft shadows by blurring/smoothing the light map. To make it more precise you can add info to each pixel if it is shadow or wall. Actual walls are just pixels that are at 0m height above ground so you can use Z-buffer values directly for this. Level of blurring depends on the scattering properties of the air and of coarse pixels at 0m ground height are not blurred at all
IR
this can be done in similar way but temperature behaves a bit differently so I would make several layers of the scene in few altitudes above ground forming a volume render and then post process the energy transfers between pixels and layers. Also do not forget to add the cooling effect of green plants and water vaporisation.
I do not have enough experience in this field to make more suggestions I am more used to temperature maps for very high temperature variances in specific conditions and material not the outdoor conditions.
PS. I forgot albedo for IR and visible light is very different for many materials especially aluminium and some wall paintings