I'm currently making a tile based mmorpg and am trying to find a good way to store a large 2d game world (at least 1000 squared tiles, but hopefully more like a few thousand squared). The idea is to encourage people to make their own cities on the shared map and users will be able to build houses and shops ingame so the tiles would be able to be edited and have a one to many relationship with some linked tables. I'm thinking of sending them to the client in 64x64 tile chunks.
I currently have it working in php/mysql, my table looks like:
CREATE TABLE `fbmmo`.`tiles` (
`id` int(10) unsigned NOT NULL AUTO_INCREMENT,
`x` int(11) NOT NULL,
`y` int(11) NOT NULL,
`realm` varchar(45) NOT NULL,
`image_id` varchar(45) NOT NULL,
PRIMARY KEY (`id`),
KEY `Index_X` (`x`),
KEY `Index_Y` (`y`)
) ENGINE=InnoDB AUTO_INCREMENT=4327236 DEFAULT CHARSET=latin1;
and queries roughly like
select * from tiles where x<1000 and y<1000 and x>936 and y>936;
When I've only got about half a million records it only takes about half a second, but when i get up to about 4 million records it takes about 5 seconds and I'm sure theres some way to improve that.
I can get around it by cacheing the map requests but I wondered if there is a better method of configuring the database, and also if anyone has tips for effectively storing a large 2d game world?
thanks for your time
As the size of the data set increases, range queries can get out of hand very quickly. You might want to look at some literature on this. Here are some interesting papers.
Data structures for range queries:
http://portal.acm.org/citation.cfm?id=1035798
www.siam.org/proceedings/soda/2010/SODA10_014_yuanh.pdf
Algorithms for quick searching
www.cccg.ca/proceedings/2005/3.pdf
Maybe you could chunk the tiles already at the database level?
Just my small 2 cent. Maybe, depending on how ppl scroll the map, you can prefetch the next N visitable 64x64 tile sets. A small thing about image_id varchar(45) NOT NULL,
are you sure you want to store the realm in each tile? Cant imagine what it is good for.
Related
I will have partitioned tables of on the order of 20M rows on a shared server with a lot of disk space but limited RAM: <8GB. These tables essentially contain a timestamp, an ID, and a numeric performance indicator and I will be aggregating them to 15 minute bins, so the new table would be:
dt DATE NOT NULL,
id char(9) NOT NULL,
bintime int NOT NULL,
avg_score numeric(5,2) NOT NULL
My predecessor had split the date into different numeric components, presumably to optimize for different future aggregation queries so having a table with:
id char(9) NOT NULL,
yyyy smallint NOT NULL,
mm smallint NOT NULL,
dd smallint NOT NULL,
dow smallint NOT NULL,
bintime int NOT NULL,
avg_score numeric(5,2) NOT NULL
I am wondering if it is worthwhile/beneficial to do this. Again, space is not an issue, but RAM is.
From my research, I could even have each grouping column (yyyy,mm,etc.) be an enum type. And I've seen someone ask on the PostgreSQL list about a similar question, but this was storing dates as ints for ordering, not grouping. The answer they got was
Keep in mind what Michael A. Jackson (among others) had to say on
this:
"The First Rule of Program Optimization: Don't do it.
The Second
Rule of Program Optimization (for experts only!): Don't do it yet."
For one thing, adding an extra column to your data would mean more
data you need to cram in the cache as you query, so even if the raw
integer versus date ordering is faster, the "optimization" could still
be a net loss due to the fatter tuples. If you're willing to live with
only integer-based dates, that could help, but that seems
exceptionally painful and not worth considering unless you run into
trouble.
Preliminary results of my investigation.
Table creation
Without ymd:
Time 35:34 minutes (from Python script)
Table size: 538 MB
With ymd:
Time 59:17 minutes to 4:30:14 hrs (from pgadmin query window)
Table size: 562 MB
Aggregation to month-weekday
Without ymd: 1:42
With ymd: 1:36
I have yet to test with indexes.
Preliminary conclusions:
Marginal increase in aggregation query performance not worth substantial creation query time. Table size difference is negligible (surprisingly). Feel free to add more suggestions for testing.
I am designing a web system like reddit's pagination, example
http://example.com/list.html?next=1234
Asume we display 50 items per page, the above URL will retrieve 50 items after the Primary Key 1234.
The problem with this approach is that the total number of items is guessable because PKs are AUTO_INCREMENT, to hide business sensitive data like this, are there any hash/encryption algorithm can
comparable, you know which hash is larger/smaller than another
can not guess growth or total number because it's sparse and randomized.
not very long, can be translated into very short base36.
Following the pointers in an ebay tech blog and a datastax developers blog, I model some event log data in Cassandra 1.2. As a partition key, I use “ddmmyyhh|bucket”, where bucket is any number between 0 and the number of nodes in the cluster.
The Data model
cqlsh:Log> CREATE TABLE transactions (yymmddhh varchar, bucket int,
rId int, created timeuuid, data map, PRIMARY
KEY((yymmddhh, bucket), created) );
(rId identifies the resource that fired the event.)
(map is are key value pairs derived from a JSON; keys change, but not much)
I assume that this translates into a composite primary/row key with X buckets per hours.
My column names are than timeuuids. Querying this data model works as expected (I can query time ranges.)
The problem is the performance: the time to insert a new row increases continuously.
So I am doing s.th. wrong, but can't pinpoint the problem.
When I use the timeuuid as a part of the row key, the performance remains stable on a high level, but this would prevent me from querying it (a query without the row key of course throws an error message about "filtering").
Any help? Thanks!
UPDATE
Switching from the map data-type to a predefined column names alleviates the problem. Insert times now seem to remain at around <0.005s per insert.
The core question remains:
How is my usage of the "map" datatype in efficient? And what would be an efficient way for thousands of inserts with only slight variation in the keys.
My keys I use data into the map mostly remain the same. I understood the datastax documentation (can't post link due to reputation limitations, sorry, but easy to find) to say that each key creates an additional column -- or does it create one new column per "map"?? That would be... hard to believe to me.
I suggest you model your rows a little differently. The collections aren't very good to use in cases where you might end up with too many elements in them. The reason is a limitation in the Cassandra binary protocol which uses two bytes to represent the number of elements in a collection. This means that if your collection has more than 2^16 elements in it the size field will overflow and even though the server sends all of the elements back to the client, the client only sees the N % 2^16 first elements (so if you have 2^16 + 3 elements it will look to the client as if there are only 3 elements).
If there is no risk of getting that many elements into your collections, you can ignore this advice. I would not think that using collections gives you worse performance, I'm not really sure how that would happen.
CQL3 collections are basically just a hack on top of the storage model (and I don't mean hack in any negative sense), you can make a MAP-like row that is not constrained by the above limitation yourself:
CREATE TABLE transactions (
yymmddhh VARCHAR,
bucket INT,
created TIMEUUID,
rId INT,
key VARCHAR,
value VARCHAR,
PRIMARY KEY ((yymmddhh, bucket), created, rId, key)
)
(Notice that I moved rId and the map key into the primary key, I don't know what rId is, but I assume that this would be correct)
This has two drawbacks over using a MAP: it requires you to reassemble the map when you query the data (you would get back a row per map entry), and it uses a litte more space since C* will insert a few extra columns, but the upside is that there is no problem with getting too big collections.
In the end it depends a lot on how you want to query your data. Don't optimize for insertions, optimize for reads. For example: if you don't need to read back the whole map every time, but usually just read one or two keys from it, put the key in the partition/row key instead and have a separate partition/row per key (this assumes that the set of keys will be fixed so you know what to query for, so as I said: it depends a lot on how you want to query your data).
You also mentioned in a comment that the performance improved when you increased the number of buckets from three (0-2) to 300 (0-299). The reason for this is that you spread the load much more evenly thoughout the cluster. When you have a partition/row key that is based on time, like your yymmddhh, there will always be a hot partition where all writes go (it moves throughout the day, but at any given moment it will hit only one node). You correctly added a smoothing factor with the bucket column/cell, but with only three values the likelyhood of at least two ending up on the same physical node are too high. With three hundred you will have a much better spread.
use yymmddhh as rowkey and bucket+timeUUID as column name,where each bucket have 20 or fix no of records,buckets can be managed using counter cloumn family
I've been having some difficulty scaling up the application and decided to ask a question here.
Consider a relational database (say mysql). Let's say it allows users to make posts and these are stored in the post table (has fields: postid, posterid, data, timestamp). So, when you go to retrieve all posts by you sorted by recency, you simply get all posts with posterid = you and order by date. Simple enough.
This process will use timestamp as the index since it has the highest cardinality and correctly so. So, beyond looking into the indexes, it'll take literally 1 row fetch from disk to complete this task. Awesome!
But let's say it's been 1 million more posts (in the system) by other users since you last posted. Then, in order to get your latest post, the database will peg the index on timestamp again, and it's not like we know how many posts have happened since then (or should we at least manually estimate and set preferred key)? Then we wasted looking into a million and one rows just to fetch a single row.
Additionally, a set of posts from multiple arbitrary users would be one of the use cases, so I cannot make fields like userid_timestamp to create a sub-index.
Am I seeing this wrong? Or what must be changed fundamentally from the application to allow such operation to occur at least somewhat efficiently?
Indexing
If you have a query: ... WHERE posterid = you ORDER BY timestamp [DESC], then you need a composite index on {posterid, timestamp}.
Finding all posts of a given user is done by a range scan on the index's leading edge (posterid).
Finding user's oldest/newest post can be done in a single index seek, which is proportional to the B-Tree height, which is proportional to log(N) where N is number of indexed rows.
To understand why, take a look at Anatomy of an SQL Index.
Clustering
The leafs of a "normal" B-Tree index hold "pointers" (physical addresses) to indexed rows, while the rows themselves reside in a separate data structure called "table heap". The heap can be eliminated by storing rows directly in leafs of the B-Tree, which is called clustering. This has its pros and cons, but if you have one predominant kind of query, eliminating the table heap access through clustering is definitely something to consider.
In this particular case, the table could be created like this:
CREATE TABLE T (
posterid int,
`timestamp` DATETIME,
data VARCHAR(50),
PRIMARY KEY (posterid, `timestamp`)
);
The MySQL/InnoDB clusters all its tables and uses primary key as clustering key. We haven't used the surrogate key (postid) since secondary indexes in clustered tables can be expensive and we already have the natural key. If you really need the surrogate key, consider making it alternate key and keeping the clustering established through the natural key.
For queries like
where posterid = 5
order by timestamp
or
where posterid in (4, 578, 222299, ...etc...)
order by timestamp
make an index on (posterid, timestamp) and the database should pick it all by itself.
edit - i just tried this with mysql
CREATE TABLE `posts` (
`id` INT(11) NOT NULL,
`ts` INT NOT NULL,
`data` VARCHAR(100) NULL DEFAULT NULL,
INDEX `id_ts` (`id`, `ts`),
INDEX `id` (`id`),
INDEX `ts` (`ts`),
INDEX `ts_id` (`ts`, `id`)
)
ENGINE=InnoDB
I filled it with a lot of data, and
explain
select * from posts where id = 5 order by ts
picks the id_ts index
Assuming you use hash tables to implement your Data Base - yes. Hash tables are not ordered, and you have no other way but to iterate all elements in order to find the maximal.
However, if you use some ordered DS, such as a B+ tree (which is actually pretty optimized for disks and thus data bases), it is a different story.
You can store elements in your B+ tree ordered by user (primary order/comparator) and date (secondary comparator, descending). Once you have this DS, finding the first element can be achieved in O(log(n)) disk seeks by finding the first element matching the primary criteria (user-id).
I am not familiar with the implementations of data bases, but AFAIK, some of them do allow you to create an index, based on a B+ tree - and by doing so, you can achieve finding the last post of a user more efficiently.
P.S.
To be exact, the concept of "greatest" element or ordering is not well defined in Relational Algebra. There is no max operator. To get the max element of a table R with a single column a one should actually create the Cartesian product of that table and find this entry. There is no max nor sort operator in strict relational algebra (though it does exist in SQL)
(Assuming set, and not multiset semantics):
MAX = R \ Project(Select(R x R, R1.a < R2.a),R1.a)
I have a list of tuples.
[
"Bob": 3,
"Alice": 2,
"Jane": 1,
]
When incrementing the counts
"Alice" += 2
the order should be maintained:
[
"Alice": 4,
"Bob": 3,
"Jane": 1,
]
When all is in memory there rather simple ways (some more or some less) to efficiently implement this. (using an index, insert-sort etc) The question though is: What's the most promising approach when the list does not fit into memory.
Bonus question: What if not even the index fits into memory?
How would you approach this?
B+ trees order a number of items using a key. In this case, the key is the count, and the item is the person's name. The entire B+tree doesn't need to fit into memory - just the current node being searched. You can set the maximum size of the nodes (and indirectly the depth of the tree) so that a node will fit into memory. (In practice nodes are usually far smaller than memory capacity.)
The data items are stored at the leaves of the tree, in so-called blocks. You can either store the items inline in the index, or store pointers to external storage. If the data is regularly sized, this can make for efficient retrieval from files. In the question example, the data items could be single names, but it would be more efficient to store blocks of names, all names in a block having the same count. The names within each block could also be sorted. (The names in the blocks themselves could be organized as a B-tree.)
If the number of names becomes large enough that the B+tree blocks are becoming ecessively large, the key can be made into a composite key, e.g. (count, first-letter). When searching the tree, only the count needs to be compared to find all names with that count. When inserting, or searcing for a specific name with a given count, then the full key can be compared to include filtering by name prefix.
Alternatively, instead of a composite key, the data items can point to offsets/blocks in an external file that contains the blocks of names, which will keep the B+tree itself small.
If the blocks of the btree are linked together, range queries can be efficiently implemented by searching for the start of the range, and then following block pointers to the next block until the end of the range is reached. This would allow you to efficiently implement "find all names with a count between 10 and 20".
As the other answers have noted, an RDBMS is the pre-packaged way of storing lists that don't fit into memory, but I hope this gives an insight into the structures used to solve the problem.
A relational database such as MySQL is specifically designed for storing large amounts of data the sum of which does not fit into memory, querying against this large amount of data, and even updating it in place.
For example:
CREATE TABLE `people` (
`name` VARCHAR(255),
`count` INT
);
INSERT INTO `people` VALUES
('Bob', 3),
('Alice', 2),
('Jane', 1);
UPDATE `people` SET `count` = `count` + 2;
After the UPDATE statement, the query SELECT * FROM people; will show:
+-------+-------+
| name | count |
+-------+-------+
| Bob | 5 |
| Alice | 4 |
| Jane | 3 |
+-------+-------+
You can save the order of people in your table by adding an autoincrementing primary key:
CREATE TABLE `people` (
`id` INT UNSIGNED NOT NULL AUTO_INCREMENT,
`name` VARCHAR(255),
`count` INT,
PRIMARY KEY(`id`)
);
INSERT INTO `people` VALUES
(DEFAULT, 'Bob', 3),
(DEFAULT, 'Alice', 2),
(DEFAULT, 'Jane', 1);
RDMS? Even flat file versions like SQLite. Otherwise a combination utilizing lazy loading. Only keep X records in memory the top Y records & the Z most recent ones that had counts updated. Otherwise a table of Key, Count columns where you run UPDATEs changing the values. The ordered list can be retrieved with a simple SELECT ORDER BY.
Read about B-trees and B+-trees. With these, the index can always be made small enough to fit into memory.
An interesting approach quite unlike BTrees is the Judy Tree
What you seem to be looking for are out of core algorithms for container classes, specifically an out of core list container class. Check out the stxxl library for some great examples of out of core alogorithms and processing.
You may also want to look at this related question
As far as "implementation details tackling this 'by hand'", you could read about how database systems do this by searching for the original papers on database design or finding graduate course notes on database architecture.
I did some searching and found a survey article by G. Graefe titled "Query evaluation techniques for large databases". It somewhat exhaustively covers every aspect of querying large databases, but the entire section 4 addresses how "query evaluation systems ... access base data stored in the database". Also, Graefe's survey was linked to by the course page for CPS 216: Advanced Databases Systems at Duke, Fall 2001. Week 5 was on Physical Data Organization which says that most commerical DBMS's organize data on-disk using blocks in the N-ary Storage Model (NSM): records are stored from the beginning of each block and a "directory" exists at the end.
See also:
Spring 2004 CPS 216 lecture notes
MIT OCW 6.830 Database Systems
Of course I know I could use a database. This question was more about the implementation details tackling this "by hand"
So basically, you are asking "How does a database do this?" To which the answer is, it uses a tree (for both the data and the index), and only stores part of the tree in memory at any one time.
As has already been mentioned, B-Trees are especially useful for this: since hard-drives always read a fixed amount at a time (the "sector size"), you can make each node the size of a sector to maximize efficiency.
You do not specify that you need to add or remove any elements from the list, just keep it sorted.
If so, a straightforward flat-file approach - typically using mmap for convenience - will work and be faster than a more generic database.
You can use a bsearch to locate the item, or maintain a set of the counts of slots with each value.
As you access an item, so the part of the file it is in (think in terms of memory 'pages') gets read into RAM automatically by the OS, and the slot and it's adjacent slots even gets copied into the L1 cache-line.
You can do an immediate comparison on it's adjacent slots to see if the increment or decrement causes the item to be out-of-order; if it is, you can use a linear iteration (perhaps augmented with a bsearch) to locate the first/last item with the appropriate count, and then swap them.
Managing files is what the OS is built to do.