When a simple (just sample) Spring Boot Application in Cloud Run, there is no files being written in the application, gets terminated with the following error.
Memory limit of 256M exceeded with 257M used. Consider increasing the
memory limit, see
https://cloud.google.com/run/docs/configuring/memory-limits
When I look at Deleting temporary files, it says that the disk storage is in-memory, so if the code is not writing any files, then how these files are being written. How to find these files using
gsutil ls -h gs://projectName
An interesting question.
Are you confident that your app isn't writing any files? I bet it is.
There's a wrinkle to Google's statement. Anything written to /var/log is not part of the in-memory filesystem|quota and is shipped to Cloud Logging.
I think Google should consider providing metrics that help differentiate between the container's process(es)' use of memory and the in-memory filesystem's use. Currently there is no way to disambiguate this usage (Cloud Monitoring metrics for Cloud Run). Perhaps raise a feature request on Google's public issue tracker?
To answer you question, you may want to consider running the container locally. Then you can grab the container's (process') process id (PID) and then try e.g. ls -la /proc/${PID}/fd to list files that the container is producing.
I considered suggesting Google Cloud Profiler but it requires an agent for Java and so it would be cumbersome to deploy to Cloud Run and would not obviously yield an answer to your question.
It's the problem of Java, and it's worse with Spring. Java, with a standard JVM use a lot of memory by default (at least 128MB). When you run Spring on top of java, tons of beans are loaded in memory, the library loaded in memory and it take easily more than 350Mb for a simple hello world app.
I wrote an article on that. You have 2 solutions to mitigate the cold start and the memory (and container) size:
Use raw java without heavy framework
Use native compilation (graalVM for instance).
I tried to optimize the JVM used (a micro JVM) or Spring directly (limit the beans loaded, use lazy loading, add JVM parameters). It saved a few second at startup (cold start) but not really memory.
I also started to investigate AppCDS with a former Google CLoud Dev Advocate, but he left the company 1 year ago and I stopped my effort (I'm no longer a Java/Spring developer, but I always liked the concept and I worked on it).
Eventually, you can also have a look to new generation framework, like Micronaut, Quarkus or Vert.x
Related
My use case:
A single-node out-of-memory "big dict" (or "big map"). The total size is too large for memory, e.g. 20gb, but is ok for single-node disk. Due to the total size, it's unwieldy with single-file solution like SQLite. Also I want easy close backpacks, so want manageable file sizes. It needs to be a series of size-controllable files managed by the tool in a user transparent way. Further it should be embedded, ie, a simple lib, no client/server.
Long story short, I picked Rocksdb.
Now new requirements or nice-to-haves: I want to use a cloud blobstore as the ultimate storage. For example, a couple levels of hot caches reside in mo.eory or local disk with configurable total size; beyond that, go read/write to a cloud blob store.
After the initial creation of the dataset, the usage is mainly read. I don't care much about "distributed", multiple-machines competing-to-write that kind of complexities.
I don't see Rocksdb has this option.There's rocksdb-cloud that appears to be in "internal dev" mode---no end-user doc whatsoever.
Questions:
Is my use case reasonable? Would a cloud kv store (like GCP Firestore?) plus a naive flat cache in memory going to have similar effect?
How to do this with Rocksdb? Or any alternative?
Thanks.
RocksDB allows you to define your own FileSystem or Env, which you can implement the interaction layer with whatever special filesystem you want. So it's possible, but you need implement or define the integration layer with cloud kv store. (running on HDFS is an example, which defines it's own Env)
Comment to #jayZhuang; too long as comment
This looks like the code is modular in decent ways, but can hardly say it "supports" cloud storage, because that needs forking and hacking the code itself. More reasonable to the end use would be extension or plugin from outside, basically "give me a few auth argents and the location of the storage and I do the rest". The few major blob stores should be a modest effort for this.
For me, I'm using Rocksdb from python via a hardly maintained Rocksdb python client. (There are no active options.) I have nice python utilities for cloud blobstore. I'm sure there's no way to let Rocksdb use that coming from python via an inactive Rocksdb python client package. Although I am able to do c++ extensions for python, that would need digging into both Rocksdb and blobstore in c++. It's not something I'll take on.
Thanks for the pointers. Do you know of any other examples closer to the end user?
I noticed that Spring reference application (Sagan) uses the SimpleCacheManager implementation. See here for source code of Sagan.
I was surprised by this choice because I thought that all but small applications running on a single node would use something like a Redis cache manager and not the simple cache manager.
How can a large application like Sagan -which I assume runs on cloudfoundry- use this simple implementation?
Any comment welcome.
Well, the SimpleCacheManager choice has been made because it was the simplest solution that could possibly work. Note that Sagan is, at least for now, not storing a lot of data in that cache and merely using it to respect various APIs rate-limiting and get better performance on some parts of the application.
Yes, Sagan is running on CloudFoundry (see this presentation) and is using CF marketplace services.
Even if cache consistency between instances is not a constraint for now, we could definitely add another marketplace service, here a Redis Cloud instance, and use this as a central cache repository.
Now that we're considering using that cache for more features, it even makes sense to at least consider that use case, since it could lower our monthly bill (pay a small fee for a redis service and use less memory for our CF instances).
In any case, thanks a lot balteo for this insightful question, we've created a Github issue for that.
I have several background tasks running on my Glassfish server implemented by #TimerService instances. The goal of these services is to extract data from files and insert that data into the database.
I tried initially to do this in JPA but the system stalled far to easily, I have now converted the process to JDBC which is far more responsive. However there are still enormous memory leaks somewhere along the way which I cannot pinpoint.
Each file is extracted in a method which manages its own transactions (1 file = 1 transaction). I would think that once this method finalises all variables loose scope and be GC'ed, but this is not the case. After a very short time I am experiencing OutOfMemoryException.
I am wondering if, how, and why Glassfish would be keeping reference to my variables (which are very heavy objects). What settings or methodologies can I apply to minimize these memory leaks?
For reference I am using the stock Glassfish settings with a couple of modifications :
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
-XX:MaxPermSize=256m
–XmX1024m
You might be dealing with a class loader leak. JAXB can do this when you're unmarshalling. To find out for sure you should use a memory analyzer. I highly recommend using the Eclipse Memory Analyzer Tool. Just follow a few of the tutorials and you should be able to get it figured out.
I use several loadtesting tools (Loadrunner, JMeter, NeoLoad) to performance test different applications. Im wondering if it is possible to monitor all layers of an application stack so for example. Say i have the following data chain.
Loadbalancer <-x-> Application Server <-x-> RMI <-x-> Java Application <-x-> MQ <-x-> Legacy application <-x-> Database
Where i have marked the x in the chain i am interested in monitoring, for example avg responsetimes.
Obviously we could simply create a wrapper on all endpoints which would gather the statistics for us and maybe we could import it into loadrunner or other loadtesting tools and sideline hem with the tools inbuilt performance statistics, but maybe there is tools/applications which already does this?
If not, how should we proceed, in order to gather this kind of statistics?
The standard for this was supposed to be Application Response Measurement (ARM). It was a cross language set of APIs that did just what you were looking for. The issue is that the products that implement this spec all tend to be big, expensive "enterprise" level monitoring tools. Think multi-week installs, consultants, more infrastructure and lots of buzzwords.
Still, if this is a mission critical app with a mission critical budget, this may be what you need. But you may be able to build your own that does just enough without too much effort. A quick search turns up at least one open source ARM implementation if you still want to use that API.
Another option is to simply to have transactions you can run against each tier of the system to check general responsiveness. For example you can have a static web page on the LB, a no-op tx on the app server, a "hello" servlet on the Java app, put a message directly on the queue, etc. During a performance / load test, these could be hit directly by the load testing tool or you could write a wrapper servlet / application call that does this as a single HTTP (RMI?) call. Running these a few times a minute won't add too much load to the system, but it should help you pinpoint which tier is slower. The nice thing about this approach is that it also works in production, just watch out for security issues.
For single user kind of test, where you know you have problem (e.g. this tx is "slow"), I have also had pretty good luck with network tracing. It's very tedious, but when you aren't sure what tier is slow, starting up a network trace on a few machines and running a single tx usually gives a good idea of what the system is doing.
I have handled this decomposition a number of ways in the past. The first is at a very low level using protocol analyzer dumped data to find the time points where a conversation leaves tier X and enters tier Y. The second method is through the use of log examination for the various tiers. Something that can make your examination quite usefule in this case is a common log server for all of your components (syslog, Rsyslog, etc....) and a nice log parsing tool, such as the freely available Microsoft Logparser. The third method utilization of the audit trail for an application stored in the database. You may find this when working on enterprise services bus style applications which have a consumer/producer model and a bus to pass information rather than a direct connection. The audit trails I have seen are typically stored in a database and allow the tracking of an individual transaction through the entire application infrastructure. Your Load balancer, as a network device, may be out of the hunt on this one.
Note, if you go the protocol analyzer or log route, then be sure and synchronize all of your source information devices to a common time server. Having one of your collectors (analyzer, app log) off on a time stamp basis can really be a hair pulling experience when you get into the analysis phase.
As to how you move from your collected data into LoadRunner, that part is very mechanical. The Analysis program supports an interface to import external datapoints. The format is very specific and is documented in both help and the online docs. This import process works very well, as I often have to use it for collection of statistics from hosts which I do not have direct monitoring access to, but which need to be included as a part of the monitored test infrastructure.
James Pulley
Moderator (YahooGroups LoadRunner, Advanced-Loadrunner; GoogleGroups lr-LoadRunner; Linkedin LoadRunner, LoadRunnerByTheHour; SQAForums LoadRunner, WinRunner)
Question is clear as you see in the title, it would be appreciated to hear your ideas about adv./disadv. differences between them.
UPDATE:
I have decided to use Hazelcast because of the advantages like distributed caching/locking mechanism as well as the extremely easy configuration while adapting it to your application.
We tried both of them for one of the largest online classifieds and e-commerce platform. We started with ehcache/terracotta(server array) cause it's well-known, backed by Terracotta and has bigger community support than hazelcast. When we get it on production environment(distributed,beyond one node cluster) things changed, our backend architecture became really expensive so we decided to give hazelcast a chance.
Hazelcast is dead simple, it does what it says and performs really well without any configuration overhead.
Our caching layer is on top of hazelcast for more than a year, we are quite pleased with it.
Even though Ehcache has been popular among Java systems, I find it less flexible than other caching solutions. I played around with Hazelcast and yes it did the job, it was easy to get running etc and it is newer than Ehcache. I can say that Ehcache has much more features than Hazelcast, is more mature, and has big support behind it.
There are several other good cache solutions as well, with all different properties and solutions such as good old Memcache, Membase (now CouchBase), Redis, AppFabric, even several NoSQL solutions which provides key value stores with or without persistence. They all have different characteristics in the sense they implement CAP theorem, or BASE theorem along with transactions.
You should care more about, which one have the functionality you want in your application, again, you should consider CAP theorem or BASE theorem for your application.
This test was done very recently with Cassandra on the cloud by Netflix. They reached to million writes per second with about 300 instances. Cassandra is not a memory cache but you data model is like a cache, which is consist of key value pairs. You can as well use Cassandra as a distributed memory cache.
Hazelcast has been a nightmare to scale and stability is still a major issue.
The dedicated client to grid component choices are
The messy version that cant survive node loss anywhere, negating the point of backups (superclient), or
An incredibly slow native client option that does not allow for any type of load balancing to processing nodes in the grid.
If any host could request records from this data grid it would be a sweet design, but you are stuck with those two lackluster option to get anything out of it.
Also multiple issues with database thread pools locking up on individual members and not writing anything to the databases, causing permanent records loss is a frequent issue and we often have to take the whole thing down for hours to refresh any of the JVM's. Split brain is also still an issue, although in 1.9.6 it seems to have calmed down a little.
Rallying to move to Ehcache and improving the database layer instead of using this as a band-aid.
Hazelcast serializes everything whenever there is a node (standard-one), so the data you will save to Hazelcast must implement serialization.
http://open.bekk.no/efficient-java-serialization/
Hazelcast has been a nightmare for me. I was able to get it "working" in a clustered Websphere environment. I use the term "working" loosely. First, all of Hazelcast's documentation is out of date and only shows examples using deprecated method calls. Trying to use the new code without comments in the Javadocs and no examples in the documentation is very hard. Also, the J2EE container code simply does not work at this point because it does not support XA transactions in Websphere. An error is thrown calling code that follows their only J2EE example explicitly(it does look like Milestone 3.0 is addressing this). I had to forget about joining Hazelcast to a J2EE transaction. It does seem Hazelcast is definitely geared to a non EJB/Non-J2EE container environment. Making calls to Hazelcast.getAllInstances() fails to retain any information about Hazelcast's state when switching from one enterprise java bean to another. That forces me to create a new Hazelcast instance just to run calls that give me access to my data. That causes many Hazelcast Instances to start up on the same JVM. Also,retrieving data from Hazelcast is not fast. I tried retrieving data using both the Native Client and directly as a member of the cluster. I stored 51 lists, each containing only 625 objects in Hazelcast. I could not perform a query directly on a list and did not want to store a map just to get access to that feature (SQL operations can be performed on a map). It took about a half second to retrieve each list of 625 objects because Hazelcast Serializes the entire list and sends it over the wire rather than just giving me the delta (what has changed). Another thing, I had to switch to a TCPIP configuration and explicitly list the ip addresses of the servers I wanted to be in the cluster. The default Multicast configuration did not work and from the group discussions in google, other people are experiencing that difficulty as well. To sum up; I did eventually get 8 machines communicating in a cluster through many hours of torturous programmatic configuration and trial and error (the documentation will be little help) but when I did, I still had no control over the number of instances and partitions being created on each JVM due to the half finished nature of Hazelcast for EJB/J2EE and it was VERY SLOW. I implemented a real use case in the unemployment insurance application I work on and the code was much faster making direct calls to the database. It would have been cool if Hazelcast worked as advertised because I really did not want to use a separate service to implement what I am trying to do. I have used MongoDB extensively so I may skip the whole in memory cache and just serialize my objects as documents in a separate repository.
One advantage of Ehcache is that it is backed by a company (Terracotta) that does extensive performance, failover, and platform testing in a large performance lab. Terracotta provides support, indemnity, etc. For many companies, that sort of thing is important.
I have not used Hazelcast but I've heard that it is easy to use and that it works. I haven't heard anything with respect to scalability or performance of Hazelcast vs Terracotta/Ehcache but given the amount of scalability and failover testing that Terracotta does, it's hard for me to imagine that Hazelcast would be competitive in a production deployment. But I presume it would work fine for smaller uses.
[Bias: I'm a former employee of Terracotta.]
Developers describe Ehcache as "Java's Most Widely-Used Cache". Ehcache is an open-source, standards-based cache for boosting performance, offloading your database, and simplifying scalability. It's the most widely-used Java-based cache because it's robust, proven, and full-featured. Ehcache scales from in-process, with one or more nodes, all the way to mixed in-process/out-of-process configurations with terabyte-sized caches. On the other hand, Hazelcast is detailed as "Clustering and highly scalable data distribution platform for Java". With its various distributed data structures, distributed caching capabilities, elastic nature, memcache support, integration with Spring and Hibernate and more importantly with so many happy users, Hazelcast is feature-rich, enterprise-ready and developer-friendly in-memory data grid solution.
Ehcache and Hazelcast are primarily classified as "Cache" and "In-Memory Databases" tools respectively.