memSQL How do I add partitions to an existing sharded table/database? - cluster-computing

Question

Adding nodes (leaves or aggregators) to a memSQL cluster is straightforward: I edited memsql_cluster.json and reran memsql-cluster setup. The problem is adding partitions to an existing table. The point here is to scale up: need to add more rows, but have exhausted the available memory in the original cluster.
I tried, for example:
mysql> create partition DMP:32 on 'ec2-X-Y-Z.compute-1.amazonaws.com':3306;
ERROR 1773 (HY000): Partition ordinal 32 is out of bounds. It must be in [0, 32).
mysql>
Reading the memsql docs, I could not find any ddl option to change the number of partitions. I would prefer not to drop and recreate these tables. Any ideas on how to do it?
Thanks!

Answer 1

You cannot add more rows to an in-memory database when memory is already exhausted. That said, you can scale out (i.e. add more leaf nodes).
You can add more leaf nodes to your MemSQL cluster, then run rebalance_partitions to distribute your existing partitions evenly across the larger cluster. This will allow each of your partitions to consume more space in your cluster, allowing you to scale out.
If you just wanted to add more partitions, you can use mysqldump to export your MemSQL schema and data, recreate the database with more partitions, then load the schema and data back into your database that now has more partitions.
Learn more about rebalance_partitions here:
http://docs.memsql.com/docs/latest/ref/REBALANCE_PARTITIONS.html

Answer 2

In order to re-partition the data currently you have to (1)export the schema and (2)data from your database (partitions are set on the database level), (3)recreate the database, and (4)reload everything back in.
You can dump database schema and tables to local files using mysqldump. It's best to run mysqldump twice, once to dump the schema and once to dump the data.
This command will create a file in the local directory containing the schemas for all tables in the DB:
mysqldump -h 127.0.0.1 -u root -B <db_name> --no-data > schema.sql
If you have a large database and not enough local disk to dump all data at once you can specify some table(s) per command. For each one use a different filename (eg data1.sql, data2.sql, etc) and put one or more table names in as arguments. I would dump any smaller tables with one statement together and then dump any giant tables separately. Use this command:
mysqldump -h 127.0.0.1 -u root -B <db_name> --no-create-info --tables <table1> <table2> <table3> > data1.sql
Once all tables and the schema have been dumped you can recreate the DB with more partitions. We generally recommend #partitions = #leaves * #cores/leaf. Use this command:
CREATE DATABASE <db_name> PARTITIONS = <#_total_partitions>;
Confirm # of partitions with "SHOW PARTITIONS ON <db_name>;"
Insert the schema and data to the new database with these commands from the shell prompt:
mysql -u root -h 127.0.0.1 < schema.sql
mysql -u root -h 127.0.0.1 < data1.sql
The schema will take a few minutes to load. Time for data to load will depend on the size of the dataset. After each table completes its rows will be committed and queries can run against it.