Does Oracle Nosql Cloud Service have provision to set max read units consumption per second. For e.g. In 40K read units, I want to reserve 20K for 1st operation and rest 20K for 2nd operation. In order to make sure 20K is always reserved for 1st operation, I want to set max read units consumption per second for 2nd operation. Is this something possible to do?
The provision values are for the entire table, so if a table has 40K read units /second, it’s up to the application to apportion them per operation.The SDKs have rate limiting support that can help with this. For example, see https://oracle.github.io/nosql-java-sdk/oracle/nosql/driver/NoSQLHandleConfig.html#setDefaultRateLimitingPercentage(double).
Sets a default percentage of table limits to use. This may be useful
for cases where a client should only use a portion of full table
limits. This only applies if rate limiting is enabled using
setRateLimitingEnabled(boolean).
You could use this method in your case.
Related
Reading the documentation on here I am still not clear of the following points:
Are there any limits on the size of the API request to the Google cloud Storage bucket? (We need to transfer PDFs from a CRM to Google Cloud Storage)
How many files we can send (it mentions a limit of 1000 writes per second) is that the same thing?
Files are the objects you store in Cloud Storage, so the maximum size limit 5 TB should be considered for individual files. There is no limit to write across multiple objects, however buckets support roughly 1000 writes per second and then scale up as needed.
For parallel uploads, please take note that up to 32 objects/files can be composed in a single composition request and a per-project composition rate limit of approximately 1,000 source objects per second.
I also recommend that you take a look at best practices on how to ramp up the request rate
You go with more than 1000 writes per second per bucket - the GCS infrastructure will scale automatically to accommodate it. The only requirement is not to ramp up the load too quickly, namely "double the request rate no faster than every 20 minutes".
https://cloud.google.com/storage/docs/request-rate#ramp-up
I just created a table in HBase and filled it with data. From the 7 regionservers it appears the data was written to region servers 6 and 7.
But I dont understand why the requests per second is zero for servers 6 and 7?
Read request count and Write request count are the total number of read and write requests seen by a particular region server since its restart. These numbers are kept in memory only for performance reasons and exposed via JMX and regionserver load API that the HBase UI uses to expose them. You could fetch them yourself using the API (or JMX) and export to a DB for persistence.
Request per second is the rate of total requests (read+write) that the regionserver in question is seeing right now. The rate is calculated based on the delta of the number of requests seen by that regionserver during a period divided by the length of the period. This particular detail (and this period) can differ based on HBase versions. In HBase 2.x, it is controlled by hbase.regionserver.metrics.period; while in previous versions there was no such setting and the period was fixed (if I remember correctly).
To answer your question, the comparison of rate of total requests and the count of total requests is not apples-to-apples. The rate only reflects current traffic while the count reflects lifetime number of requests since region server's restart. If you really think about it, it does not make sense to have a rate of requests for lifetime, because any real use case is concerned with the current rate only.
If you bulk-filled the tables via put(List<Put>), there would only have been a very small number of requests, as records are sent in batches.
We are experiencing some performance issues or anomalies on a elasticsearch specifically on a system we are currently building.
The requirements:
We need to capture data for multiple of our customers, who will query and report on them on a near real time basis. All the documents received are the same format with the same properties and are in a flat structure (all fields are of primary type and no nested objects). We want to keep each customer’s information separate from each other.
Frequency of data received and queried:
We receive data for each customer at a fluctuating rate of 200 to 700 documents per second – with the peak being in the middle of the day.
Queries will be mostly aggregations over around 12 million documents per customer – histogram/percentiles to show patterns over time and the occasional raw document retrieval to find out what happened a particular point in time. We are aiming to serve 50 to 100 customer at varying rates of documents inserted – the smallest one could be 20 docs/sec to the largest one peaking at 1000 docs/sec for some minutes.
How are we storing the data:
Each customer has one index per day. For example, if we have 5 customers, there will be a total of 35 indexes for the whole week. The reason we break it per day is because it is mostly the latest two that get queried with occasionally the remaining others. We also do it that way so we can delete older indexes independently of customers (some may want to keep 7 days, some 14 days’ worth of data)
How we are inserting:
We are sending data in batches of 10 to 2000 – every second. One document is around 900bytes raw.
Environment
AWS C3-Large – 3 nodes
All indexes are created with 10 shards with 2 replica for the test purposes
Both Elasticsearch 1.3.2 and 1.4.1
What we have noticed:
If I push data to one index only, Response time starts at 80 to 100ms for each batch inserted when the rate of insert is around 100 documents per second. I ramp it up and I can reach 1600 before the rate of insert goes to close to 1sec per batch and when I increase it to close to 1700, it will hit a wall at some point because of concurrent insertions and the time will spiral to 4 or 5 seconds. Saying that, if I reduce the rate of inserts, Elasticsearch recovers nicely. CPU usage increases as rate increases.
If I push to 2 indexes concurrently, I can reach a total of 1100 and CPU goes up to 93% around 900 documents per second.
If I push to 3 indexes concurrently, I can reach a total of 150 and CPU goes up to 95 to 97%. I tried it many times. The interesting thing is that response time is around 109ms at the time. I can increase the load to 900 and response time will still be around 400 to 600 but CPU stays up.
Question:
Looking at our requirements and findings above, is the design convenient for what’s asked? Are there any tests that I can do to find out more? Is there any setting that I need to check (and change)?
I've been hosting thousands of Elasticsearch clusters on AWS over at https://bonsai.io for the last few years, and have had many a capacity planning conversation that sound like this.
First off, it sounds to me like you have a pretty good cluster design and test rig going here. My first intuition here is that you are legitimately approaching the limits of your c3.large instances, and will want to bump up to a c3.xlarge (or bigger) fairly soon.
An index per tenant per day could be reasonable, if you have relatively few tenants. You may consider an index per day for all tenants, using filters to focus your searches on specific tenants. And unless there are obvious cost savings to discarding old data, then filters should suffice to enforce data retention windows as well.
The primary benefit of segmenting your indices per tenant would be to move your tenants between different Elasticsearch clusters. This could help if you have some tenants with wildly larger usage than others. Or to reduce the potential for Elasticsearch's cluster state management to be a single point of failure for all tenants.
A few other things to keep in mind that may help explain the performance variance you're seeing.
Most importantly here, indexing is incredibly CPU bottlenecked. This makes sense, because Elasticsearch and Lucene are fundamentally just really fancy string parsers, and you're sending piles of strings. (Piles are a legitimate unit of measurement here, right?) Your primary bottleneck is going to be the number and speed of your CPU cores.
In order to take the best advantage of your CPU resources while indexing, you should consider the number of primary shards you're using. I'd recommend starting with three primary shards to distribute the CPU load evenly across the three nodes in your cluster.
For production, you'll almost certainly end up on larger servers. The goal is for your total CPU load for your peak indexing requirements ends up under 50%, so you have some additional overhead for processing your searches. Aggregations are also fairly CPU hungry. The extra performance overhead is also helpful for gracefully handling any other unforeseen circumstances.
You mention pushing to multiple indices concurrently. I would avoid concurrency when bulk updating into Elasticsearch, in favor of batch updating with the Bulk API. You can bulk load documents for multiple indices with the cluster-level /_bulk endpoint. Let Elasticsearch manage the concurrency internally without adding to the overhead of parsing more HTTP connections.
That's just a quick introduction to the subject of performance benchmarking. The Elasticsearch docs have a good article on Hardware which may also help you plan your cluster size.
I have 1 master server and 5 region server and each server has 200 GB disk space and 16 GB RAM on each. I created a table in HBase which has 10 million records. I am using hbase-0.96 version on hadoop 2.
Table Name - sh_self_profiles
column family - profile
In this table, we have 30 columns in each row.
When I get a single column value from HBase, it takes around 10 ms. My problem is when I hit 100 or more concurrent requests the time slowly accumulates and increases to more than 400 ms instead of completing in 10ms only. When 100 requests are hit linearly, each one takes 10 ms only.
One thing that you should check is how well distributed your table is.
You can do this by going to the HBase master web console http://:60010, you will be able to see how many regions you have for your table. If you have not done anything special on table creation you could easily have only one or two regions, which means that all the requests are being directed to a single region server.
If this is the case, you can recreate your table with pre-split regions (I would suggest a multiple of 5, such as 15 or 20), and make sure that the concurrent gets that you are doing are equally spread over the row-key space.
Also, pls check how much RAM you have allocated to the region server - you might need to increase it from the default. If you are not running anything else other than HBase Region Sever on those machines, you could probably increase to 8GB ram.
Other than that, you could also adjust the default for hbase.regionserver.handler.count.
I hope this helps.
Which client are you using? Are you using the standard Java client, the Thrift client, the HTTP REST client, or something else? If your use case is a high amount of random reads of single column values, I highly recommend you try asynchbase as it is much faster than the standard synchronous Java client.
Is there any recommendation of how to serve large number of customers with Cassandra?
Imagine that all of them read rows (specific per customer) in Cassandra. If we have one big Column Family (CF) we will correspondingly have single memtable per this CF. So, that customers who read data more frequently will displace cache entries of less frequent-in-read customers. And quality of service (i.e. read speed) will differ for different users. This is not fair (all customers must experience the same performance).
Is it normal to allocate separate CF's per customer (e.g. 5000 CF's or more)? As I understand this will cause creation of 5000 memtables what will lead to fair caching because each customer will be served in separate cache (memtable). Am I correct?
And on the other hand, will creation of large number of CF's decrease performance rather than having single big CF?
Memtables are not caches, they are to ensure writes in Cassandra are sequential on disk. They are read from when doing queries, but are flushed when they are too big rather than using an eviction policy that is appropriate for a cache.
Having separate column families for each customer will be very inefficient - 1000s of CFs is too many. Better would be to make sure your customer CF remains in cache. If you have enough memory then it will (assuming you don't have other CFs on your Cassandra cluster). Or you could use the row cache and set the size to be big enough to hold all the data in your customer CF.