Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 8 years ago.
Improve this question
If in-memory databases are as fast as they claim to be, why aren't they more commonly utilized?
One of the main reasons in-memory databases aren't commonly used is because of cost. As you stated, in-memory databases are usually an order of magnitude faster than disk resident databases for obvious reasons. However, RAM is also significantly more expensive than hard drives and consequently not viable for large databases. With that said, with RAM getting much cheaper in-memory databases today are more than viable for enterprise use.
Another reason is that in-memory databases are often not ACID compliant. This is because memory is volatile, and unforeseen events like power losses may result in complete loss of data. As this is unacceptable for the vast majority of use cases, most in-memory databases do end up utilizing disks to persist data. Of course, this ends up undermining some of the benefits of in-memory databases by re-introducing disk I/O as a performance bottleneck.
In any case, in-memory databases will likely become predominant as RAM becomes cheaper. The performance differences between the two are too drastic to be ignored. Knowing this, multiple vendors have thrown their hats into the in-memory space such as Oracle TimesTen, SAP Hana, and many others. Also, some companies like Altibase are offering “hybrid” DBMS systems, which contain both in-memory and disk resident components.
You may want to read up on these in-memory offerings to get a better understanding of in-memory databases.
http://www.oracle.com/technetwork/database/database-technologies/timesten/overview/index.html
http://www.saphana.com/
http://altibase.com/in-memory-database-hybrid-products/hdbtm-hybrid-dbms/
Possibly because one or more of:
there often a mismatch between the data size and the available RAM size
when the data is small normal disk caching and OS memory/disk management may be as effective
when the data is large, swapping to disk is likely to void any benefit
fast enough to meet performance requirements and service level does not mean as fast as possible - fast enough is good enough.
Related
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 2 years ago.
Improve this question
I have to install an Oracle 18c database server, currently I have 4 960 GB solid state hard drives, 32 2.10 ghz Intel Xeon processors, 255 GB of RAM, the database that Mounted on this server is approximately 750 GB in weight and is increased by 350 GB per year and has a concurrency of 500 simultaneous connections, do you consider that this hardware is the one to use to obtain good performance in Oracle or should it be increased?
The answer is, "it depends":
Is this enough storage capacity to meet your long term needs for data, archived transaction logs, backups, auditing, and whatever else you need to store?
Do you have an existing production system, and what sort of performance do you get out of it? How does the old hardware and new hardware compare?
How many transactions does your database process in an hour? Your overall data might only grow by 350GB, but if there are a lot of updates and not just inserts then you you could be archiving that much in log files and backups every day.
What those 500 concurrent sessions are actually doing at any moment will drive the size of your memory and processor requirements, as will the amount of data that you need to cache to support them.
Do you have any HA requirements, and if so how does that affect the configuration of your storage (i.e. do you lose some percentage of your storage to RAID)?
Which operating system you choose can also affect how efficiently your hardware performs.
My personal preference is to use virtualized servers running Oracle Linux with an SSD SAN , but that might not be right for you. In the end there really isn't enough information in your question to say for sure if the hardware you describe is sufficient for your needs or not.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
What I mean is: memories are becoming larger and larger, and OS and compilers smarter and smarter. Therefore my question, if I have to read data from file, does it make sense to implement a cache? Isn't the operating system already managing data into memory?
edit ok to be more practical, I have 1TB of data sparse in more files, and 180GB of RAM. I need to read some of this data more than once. Does it make sense to implement a cache like LRU, or when I read from file (using c++) the operating system will have been smart enough to have kept these data somewhere so to read them from memory instead of from disk?
Depending on the language and library you are using. It is highly likely that you are actually already caching things into the memory.
In general, you want to cache things that you are currently managing until you are ready to commit the updated data buffer back into the file on the disk simply because disk I/O is a very slow operation.
On files that are very big, you may not want to cache the entire data due to memory constraints, but you would still want to cache the block of data that you are currently managing.
Here's a general diagram of different means of storing data from the fastest (most expensive) to the slowest (least expensive):
CPU data registers -> CPU Cache -> RAM -> SSD -> Hard Disk -> keyboard, etc..
HowStuffWorks.com has a pretty good illustration of this hierarchy and the entire article itself is actually a pretty good read as well: http://computer.howstuffworks.com/computer-memory4.htm
EDIT: There is also another similar discussion here that you may want to check out as well.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
I am looking for rules of thumb for designing algorithms where the data is accessed slowly due to limitations of disk speed, pci speed(gpgpu) or other bottleneck.
Also, how does one manage gpgpu programs where the memory of the application exceeds gpgpu memory?
In general, the GPU memory should not be an arbitrary limitation on the size of data for algorithms. The GPU memory could be considered to be a "cache" of data that the GPU is currently operating on, but many GPU algorithms are designed to operate on more data than can fit in the "cache". This is accomplished by moving data to and from the GPU while computation is going on, and the GPU has specific concurrent execution and copy/compute overlap mechanisms to enable this.
This usually implies that independent work can be completed on sections of the data, which is typically a good indicator for acceleration in a parallelizable application. Conceptually, this is similar to large scale MPI applications (such as high performance linpack) which break the work into pieces and then send the pieces to various machines (MPI ranks) for computation.
If the amount of work to be done on the data is small compared to the cost to transfer the data, then the data transfer speed will still become the bottleneck, unless it is addressed directly via changes to the storage system.
The basic approach to handling out-of-core or algorithms where the data set is too large to fit in GPU memory all at once is to determine a version of the algorithm which can work on separable data, and then craft a "pipelined" algorithm to work on the data in chunks. An example tutorial which covers such a programming technique is here (focus starts around 40 minute mark, but the whole video is relevant).
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 6 years ago.
Improve this question
I have a system which makes use of a remote database and a web application for the front end.
The system is now at the stage where its usage is ready to be scaled up (i.e. from being tested by one person to being used by 100's of people)...
Obviously there will be a stage for testing etc on larger groups of people but I was wondering if anyone with experience could give me the main points for consideration (i.e. common problems and solutions) when scaling a system up in such a way...
One example might be server space...ensuring that there is always enough to accommodate for the growth in system usage...
Would be interesting to hear the relative issues as this is my first big project....
The ultimate answers are unique to each application. Try to find some concrete evidence of what "normal" usage will entail and, if possible, when and what your max usage will be. Determine what acceptable performance looks like for your app, and use a load testing tool like JMeter (free) or WAPT (not free) to simulate user load and see how your app holds up. If you don't get acceptable performance numbers, throw a profiler like VisualVM (free, also bundled with the JDK) or YourKit (not free) into the mix to identify bottlenecks. Fix the most severe bottleneck, rinse, and repeat.
I see three important things to look at:
Unexpected interactions.
Badly tuned parameters and graceless degradation
Unexpected resource consumption
First, you must have clear objectives. #Ryan has already alluded to this. What does acceptable performance look like - your objective is not "as fast as possible" or you will never stop tuning - you must be very clear: with specified workload patterns for specified user populations response times are ...
As you scale up the workload you are likely to hit the problems I alluded to earlier.
Unexpected Interactions: for example some user action results in a lengthy DB activity and during that certain locks are held, other users now experience unacceptable performance. Or, several users all attempt to buy the same product at the same time, an unacceptable number of optimistic lock failures occur. Or the system deadlocks.
Such problems often don't show up until the testing scales. To detect such problems you need to design your test data carefully and your test scripts to cover both normal and "unusual" peaks.
Tuning Parameters: Your infrastructure is likely to have connection pools and thread pools. The default sizes of those pools may well need to be adjusted. There's two considerations here, for your target workload what works? You increase the connection pool size, so now the database server has more open connections, so now you need to increase some database parameter or available memory ... And, what happens in unusual situations when resources run out. Suppose the system times-out waiting for a connection - does the user get a friendly error message and the system administrator gets notified or does something very unpleasant happen?
Unexpected resource consumption: what happens to resource consumption when the workload scales? Are logs now much bigger, so disk space is insufficient? Do you need to increase heap sizes? What's the long term trend over time? Memory growth? Maybe a memory leak? There are often unpleasant surprises.
Suppose, I wanted to develop stack overflow website. How do I estimate the amount of commodity hardware required to support this website assuming 1 million requests per day. Are there any case studies that explains the performance improvements possible in this situation?
I know I/O bottleneck is the major bottleneck in most systems. What are the possible options to improve I/O performance? Few of them I know are
caching
replication
You can improve I/O performance in several ways depending upon what you use for your storage setup:
Increase filesystem block size if your app displays good spatial locality in its I/Os or uses large files.
Use RAID 10 (striping + mirroring) for performance + redundancy (disk failure protection).
Use fast disks (Performance Wise: SSD > FC > SATA).
Segregate workloads at different times of day. e.g. Backup during night, normal app I/O during day.
Turn off atime updates in your filesystem.
Cache NFS file handles a.k.a. Haystack (Facebook), if storing data on NFS server.
Combine small files into larger chunks, a.k.a BigTable, HBase.
Avoid very large directories i.e. lots of files in the same directory (instead divide files between different directories in a hierarchy).
Use a clustered storage system (yeah not exactly commodity hardware).
Optimize/design your application for sequential disk accesses whenever possible.
Use memcached. :)
You may want to look at "Lessons Learned" section of StackOverflow Architecture.
check out this handy tool:
http://www.sizinglounge.com/
and another guide from dell:
http://www.dell.com/content/topics/global.aspx/power/en/ps3q01_graham?c=us&l=en&cs=555
if you want your own stackoverflow-like community, you can sign up with StackExchange.
you can read some case studies here:
High Scalability - How Rackspace Now Uses MapReduce and Hadoop to Query Terabytes of Data
http://highscalability.com/how-rackspace-now-uses-mapreduce-and-hadoop-query-terabytes-data
http://www.gear6.com/gear6-downloads?fid=56&dlt=case-study&ls=Veoh-Case-Study
1 million requests per day is 12/second. Stack overflow is small enough that you could (with interesting normalization and compression tricks) fit it entirely in RAM of a 64 GByte Dell PowerEdge 2970. I'm not sure where caching and replication should play a role.
If you have a problem thinking enough about normalization, a PowerEdge R900 with 256GB is available.
If you don't like a single point of failure, you can connect a few of those and just push updates over a socket (preferably on a separate network card). Even a peak load of 12K/second should not be a problem for a main-memory system.
The best way to avoid the I/O bottleneck is to not do I/O (as much as possible). That means a prevayler-like architecture with batched writes (no problem to lose a few seconds of data), basically a log file, and for replication also write them out to a socket.