I have a 3 year old Gephi file were I produced a beautiful graph from CSV files where I used multimode network projection. It took all shared bi-nodes between main nodes and removed them to instead create a weighted network where edges had thickness equal to the weight.
I have no memory of settings applied, how to backtrack the process from a saved Gephi file?
Related
When building a large map RTS game, my team are experiencing some performance issues regarding pathfinding.
A* is obviously inefficient due to not only janky path finding, but processing costs for large groups of units moving at the same time.
After research, the obvious solution would be to use FlowField pathfinding, the industry standard for RTS games as it stands.
The issue we are now having after creating the base algorithm is that the map is quite large requiring a grid of around 766 x 485. This creates a noticeable processing freeze or lag when computing the flowfield for the units to follow.
Has anybody experienced this before or have any solutions on how to make the flowfields more efficient? I have tried the following:
Adding flowfields to a list when it is created and referencing later (Works once it has been created, but obviously lags on creation.)
Processing flowfields before the game is started and referencing the list (Due to the sheer amount of cells, this simply doesn't work.)
Creating a grid based upon the distance between the furthest selected unit and the destination point (Works for short distances, not if moving from one end of the map to the other).
I was thinking about maybe splitting up the map into multiple flowfields, but I'm trying to work out how I would make them move from field to field.
Any advice on this?
Thanks in advance!
Maybe this is a bit late answer. Since you have mentioned that this is a large RTS game, then the computation should not be limited to one CPU core. There are a few advice for you to use flowfield more efficiently.
Use multithreads to compute new flow fields for each unit moving command
Group units, so that all units in same command group share the same flowfield
Partition the flowfield grids, so you only have to update any partition that had modification in path (new building/ moving units)
Pre baked flowfields grid slot cost:you prebake basic costs of the grids(based on environments or other static values that won't change during the game).
Divide, e.g. you have 766 x 485 map, set it as 800 * 500, divide it into 100 * 80 * 50 partitions as stated in advice 3.
You have a grid of 10 * 10 = 100 slots, create a directed graph (https://en.wikipedia.org/wiki/Graph_theory) using the a initial flowfield map (without considering any game units), and use A* algorihtm to search the flowfield grid before the game begins, so that you know all the connections between partitions.
For each new flowfield, Build flowfield only with partitions marked by a simple A* search in the graph. And then use alternative route if one node of the route given by A* is totally blocked from reaching the next one (mark the node as blocked and do A* again in this graph)
6.Cache, save flowfield result from step.5 for further usage (same unit spawning from home and going to the enemy base. Same partition routes. Invalidate cache if there is any change in the path, and only invalidate the cache of the changed partition first, check if this partition still connects to other sides, then only minor change will be made within the partition only)
Runtime late updating the units' command. If the map is large enough. Move the units immediately to the next partition without using the flowfield, (use A* first to search the 10*10 graph to get the next partition). And during this time of movement, in the background, build the flowfield using previous step 1-6. (in fact you only need few milliseconds to do the calculation if optimized properly, then the units changes their route accordingly. Most of the time there is no difference and player won't notice a thing. In the worst case, where we finally have to search all patitions to get the only possible route, it is only the first time there will be any delay, and the cache will minimise the time since it is the only way and the cache will be used repeatitively)
Re-do the build process above every once per few seconds for each command group (in the background), just in case anything changes in the middle of the way.
I could get this working with much larger random map (2000*2000) with no fps drop at all.
Hope this helps anyone in the future.
I have a set of water meters for water consumers drawn up as geojson and visualized with ol3. For each consumer house i have their usage of water for the given year, and also the water pipe system is given as linestrings, with metadata for the diameter of each pipe section.
What is the minimum required information I need to be able to visualize/calculate the amount of water that passed each pipe in total of the year when the pipes have inner loops/circles.
is there a library that makes it easy to do the calculations in javascript.
Naive approach, start from each house and move to the first pipe junction and add the used mater measurement for the house as water out of the junction and continue until the water plant is reached. This works if there was no loops within the pipe system.
This sounds more like a physics or civil engineering problem than a programming one.
But as best I can tell, you would need time series data for sources and sinks.
Consider this simple network:
Say, A is a source and B and D are sinks/outlets.
If the flow out of B is given, the flow in |CB| would be dependent on the flow out of D.
So e.g. if B and D were always open at the same time, the total volume that has passed |CB| might be close to 0. Conversely, if B and D were never open at the same time the number might be equal to the volume that flowed through |AB|.
If you can obtain time series data, so you have concurrent values of flow through D and B, I would think there would exist a standard way of determining the flow through |CB|.
Wikipedia's Pipe Network Analysis article mentions one such method: The Hardy Cross method, which:
"assumes that the flow going in and out of the system is known and that the pipe length, diameter, roughness and other key characteristics are also known or can be assumed".
If time series data are not an option, I would pretend it was always average (which might not be so bad given a large network, like in your image) and then do the same thing.
You can use the Ford-Fulkerson algorithm to find the maximum flow in a network. To use this algorithm, you need to represent your network as a graph with nodes that represent your houses and edges to represent your pipes.
You can first simplify the network by consolidating demands on "dead-ends". Next you'll need pressure data at the 3 feeds into this network, which I see as the top feed from the 90 (mm?), centre feed at the 63 and bottom feed near to the 50. These 3 clusters are linked by a 63mm running down, which have the consolidated demand and the pressure readings at the feed would be sufficient to give the flowrate across the inner clusters.
I have a mobility model created by SUMO with area around 2 KM * 2 Km for real map.
I want to compute the results for only part of this model. I read that I can use roiroad or roirect.
Roirect take (x1,y1-x2,y2) as Traci coordination, however, I want to use roiroad to take exactly the cars in specific road.
My question is: if the roiroad function take a string of road name , from where in sumo that I can get this value.
should I construct the map again with Netconvert and using --output-street-names
Edges in SUMO always have an ID. It is stored in the id="..." attribute of the <edge> tag. If you convert a network from some other data format (say, OpenStreetMap) to SUMO's XML representation, you have the option to try and use an ID that closely resembles the road name the edge represents (this is the option you mentioned). The default is to allocate a numeric ID.
Other than by opening the road network XML file in a text editor, you can also find the edge ID by opening the network in the SUMO GUI and right clicking on the edge (or by enabling the rendering of edge IDs in the GUI).
Note that, depending on the application you simulate, you will need to make sure that you have no "gaps" in the Regions Of Interest (ROIs) you specify. When a vehicle is no longer in the ROI its corresponding node is removed from the network simulation. Even if the same vehicle later enters another (or the same) ROI, a brand new node will be created. This is particularly important when specifying edges as ROI (via the roiRoads parameter). Keep in mind that SUMO uses edges not just to represent streets, but also to represent lanes crossing intersections. If you do not specify these internal edges, your ROIs will have small gaps at every intersection.
Note also that up until OMNeT++ 5.0, syntax highlighting of the .ini file in the IDE will (mistakenly) display a string containing a # character as if it were a comment. This is just a problem with the syntax highlighting. The simulation will behave as expected. For example, setting the roiRoads parameter to "-5445204#1 :252726232_7 -5445204#2" in the Veins 4.4 example as follows...
...will result in a Veins simulation where only cars on one of the following three edges are simulated:
on the edge leading to the below intersection; or
on the edge crossing the below intersection; or
on the edge leaving the below intersection.
I have 4,000,000,000 (four billion) edges for an undirected graph. They are represented in a large text file as pairs of node ids. I would like to compute the connected components of this graph. Unfortunately, once you load in the node ids with the edges into memory this takes more than the 128GB of RAM I have available.
Is there an out of core algorithm for finding connected components that is relatively simple to implement? Or even better, can be it cobbled together with Unix command tools and existing (python) libraries?
Based on the description of the problem you've provided and the answers you provided in the comments, I think the easiest way to do this might be to use an approach like the one #dreamzor described. Here's a more fleshed-out version of that answer.
The basic idea is to convert the data to a more compressed format that fits into memory, to run a regular connected components algorithm on that data, then to decompress it. Notice that if you assign each node a 32-bit numeric ID, then the total space required to store all the nodes is at most the space for four billion nodes and eight billion edges (assuming you store two copies of each edge), which is space for twelve billion 32-bit integers, only around 48GB of space, below your memory threshold.
To start off, write a script that reads in the edges file, assigns a numeric ID to each node (perhaps sequentially in the order in which they appear). Have this script write this mapping to a file and, as it goes, write a new edges file that uses the numeric IDs of the nodes rather than the string names. When you're done, you'll have a names file mapping IDs to names and an edges file that takes up much less space than before. You mentioned in the comments that you can fit all the node names into memory, so this step should be very reasonable. Note that you don't need to store all the edges in memory - you can stream them through the program - so that shouldn't be a bottleneck.
Next, write a program that reads the edges file - but not the names file - into memory and finds connected components using any reasonable algorithm (BFS or DFS would be great here). If you're careful with your memory (using something like C or C++ here would be a good call), this should fit comfortably into main memory. When you're done, write out all the clusters to an external file by numeric ID. You now have a list of all the CCs by ID.
Finally, write a program that reads in the ID to node mapping from the names file, then streams in the cluster IDs and writes out the names of all the nodes in each cluster to a final file.
This approach should be relatively straightforward to implement because the key idea is to keep the existing algorithms you're used to but just change the representation of the graph to be more memory efficient. I've used approaches like this before in the past when dealing with huge graphs (Wikipedia) and it's worked beautifully even on systems with less memory than yours.
You can hold only an array of vertices as their "color" (an int value), then run through the file without loading the entire set of links, marking vertices with a color, a new one if neither vertice is colored, the same color if one is colored and the other isn't, and lowest of two colors, together with repainting all the other vertices in the array that are painted with the highest color if both are colored. A pseudocode example:
int nextColor=1;
int merges=0;
int[] vertices;
while (!file.eof()) {
link=file.readLink();
c1=vertices[link.a];
c2=vertices[link.b];
if ((c1==0)&&(c2==0)) {
vertices[link.a]=nextColor;
vertices[link.b]=nextColor;
nextColor++;
} else if ((c1!=0)&&(c2!=0)) {
// both colored, merge
for (i=vertices.length-1;i>=0;i--) if (vertices[i]==c2) vertices[i]=c1;
merges++;
} else if (c1==0) vertices[link.a]=c2; // only c1 is 0
else vertices[link.b]=c1; // only c2 is 0
}
In case you choose the smaller than 32-bit type for storing color of a vertex, you might need to first check if nextColor is maxed, have an array of colors unused (released in merge), and skip coloring a new set of two vertices if no color can be used, then re-run the file reading process if both the colors are all used and any mergings occur.
UPDATE: Since the vertices aren't really ints but strings instead, you should also have a map of string to int while parsing that file. If your strings are limited by length, you can probably fit them all into memory as a hash table, but I'd pre-process the file by creating another file that would have all strings "s1" replaced with "1", "s2" with "2", etc, where "s1", "s2" are whatever names appear as vertices in the file, so that the data will be compacted to a list of pairs of ints. In case you'll be processing similar data later (that is, your graph isn't changing much, and contains largely the same names of vertices, store the "metadata" file with links from names to ints to ease further pre-processings.
I am wondering if it is possible to have data from elasticsearch indices without timestamp attached to them.
I need a list of two columns as a drop down. This list is cross checked against another index to generate maps but if I zoom into the graph breaks cause the drop down list exists from time a to be but not from c to d. (lol)
My macgyver solution to this is to just add the list every few minutes into the index so on the graph, the data is reasonably dense. This allows the user to zoom in pretty well into different parts of the graph. But overtime this is going to make my index unreasonably large.