I have a simple Grails application that needs to make a periodic call to an external web service several times during a user's session (while the use the interface).
I'd like to cache this web service response, but the results from the service change about every few days, so I'd like to cache it for a short time (perhaps daily refreshes).
The Grails cache plugin doesn't appear to support "time to live" implementations so I've been exploring a few possible solutions. I'd like to know what plugin or programatic solution would best solve this problem.
Example:
BuildConfig.groovy
plugins{
compile ':cache:1.0.0'
}
MyController.groovy
def getItems(){
def items = MyService.getItems()
[items: items]
}
MyService.groovy
#Cacheable("itemsCache")
class MyService {
def getItems() {
def results
//expensive external web service call
return results
}
}
UPDATE
There were many good options. I decided to go with the plugin approach that Burt suggested. I've included a sample answer with minor changes to above code example to help others out wanting to do something similar. This configuration expires the cache after 24 hours.
BuildConfig.groovy
plugins{
compile ':cache:1.1.7'
compile ':cache-ehcache:1.0.1'
}
Config.groovy
grails.cache.config = {
defaultCache {
maxElementsInMemory 10000
eternal false
timeToIdleSeconds 86400
timeToLiveSeconds 86400
overflowToDisk false
maxElementsOnDisk 0
diskPersistent false
diskExpiryThreadIntervalSeconds 120
memoryStoreEvictionPolicy 'LRU'
}
}
The core plugin doesn't support TTL, but the Ehcache plugin does. See http://grails-plugins.github.com/grails-cache-ehcache/docs/manual/guide/usage.html#dsl
The http://grails.org/plugin/cache-ehcache plugin depends on http://grails.org/plugin/cache but replaces the cache manager with one that uses Ehcache (so you need both installed)
A hack/workaround would be to use a combination of #Cacheable("itemsCache") and #CacheFlush("itemsCache").
Tell the getItems() method to cache the results.
#Cacheable("itemsCache")
def getItems() {
}
and then another service method to flush the cache, which you can call frequently from a Job.
#CacheFlush("itemsCache")
def flushItemsCache() {}
After several hours of fails in battles with SpEL I have won the war in the end!
So as you know Grails cache does not have TTL out of the box. You can stick to ehcache and do some fancy configuration. Or worse add logic flushing it on save/update etc. But my solution is:
#Cacheable(value = 'domainInstance', key="#someId.concat((new java.util.GregorianCalendar().getTimeInMillis()/10000))")
def getSomeStuffOfDb(String someId){
//extract something of db
}
}
and one more thing to point out. You can skip configuration in Config.groovy and it will be created and added automatically itself.
However if your app is under load straight after start it will cause some exceptions.
2017-03-02 14:05:53,159 [http-apr-8080-exec-169] ERROR errors.GrailsExceptionResolver - CacheException occurred when processing request: [GET] /some/get
Failed to get lock for campaignInstance cache creation. Stacktrace follows:
so to avoid that please add config so cache facilities will be ready beforehand.
grails.cache.enabled = true
grails.cache.clearAtStartup = true
grails.cache.config = {
defaults {
maxElementsInMemory 10000
overflowToDisk false
}
cache {
name 'domainInstance'
}
}
GregorianCalendar().getTimeInMillis()/10000 will make TTL ~10sec. /1000 ~1 sec. Pure maths here.
From the grails-cache unit tests(Look for timeToLiveSeconds), I see that you can configure caching at the cache level, not per method call or similar. Using this method, you would configure the settings for grails.cache.config.
You would create a dedicated cache with your time-to-live settings and then reference it in your service.
Related
Most terraform providers demand a predefined flow, Create/Read/Update/Delete/Exists
I am in a weird situation developing a provider against an API where this behavior diverges a bit.
There are two kinds of resources, Host and Scope. A host can have many scopes. Scopes are updated with configurations.
This generally fits well into the terraform flow, it has a full CRUDE flow possible - except for one instance.
When a new Host is made, it automatically has a default scope attached to it. It is always there, cannot be deleted etc.
I can't figure out how to have my provider gracefully handle this, as I would want the tf to treat it like any other resource, but it doesn't have an explicit CREATE/DELETE, only READ/UPDATE/EXISTS - but every other scope attached to the host would have CREATE/DELETE.
Importing is not an option due to density, requiring an import for every host would render the entire thing pointless.
I originally was going to attempt to split Scopes and Configurations into separate resources so one could be full-filled by the Host (the host providing the Scope ID for a configuration, and then other configurations can get their scope IDs from a scope resource)
However this approach falls apart because the API for both are the same, unless I wanted to add the abstraction of creating an empty scope then applying a configuration against it, which may not be fully supported. It would essentially be two resources controlling one resource which could lead to dramatic conflicts.
A paraphrased example of an execution I thought about implementing
resource "host" "test_integrations" {
name = "test.integrations.domain.com"
account_hash = "${local.integrationAccountHash}"
services = [40]
}
resource "configuration" "test_integrations_root_configuration" {
name = "root"
parent_host = "${host.test_integrations.id}"
account_hash = "${local.integrationAccountHash}"
scope_id = "${host.test_integrations.root_scope_id}"
hostnames = ["test.integrations.domain.com"]
}
resource "scope" "test_integrations_other" {
account_hash = "${local.integrationAccountHash}"
host_hash = "${host.test_integrations.id}"
path = "/non/root/path"
name = "Some Other URI Path"
}
resource "configuration" "test_integrations_other_configuration" {
name = "other"
parent_host = "${host.test_integrations.id}"
account_hash = "${local.integrationAccountHash}"
scope_id = "${host.test_integrations_other.id}"
}
In this example flow, a configuration and scope resource unfortunately are pointing to the same resource which I am worried would cause conflicts or confusion on who is responsible for what and dramatically confuses the create/delete lifecycle
But I can't figure out how the TF lifecycle would allow for a resource that would only UPDATE/READ/EXISTS if say a flag was given (and how state would handle that)
An alternative would be to just have a Configuration resource, but then if it was the root configuration it would need to skip create/delete as it is inherently tied to the host
Ideally I'd be able to handle this situation gracefully. I am trying to avoid including the root scope/configuration in the host definition as it would create a split in how they are written and handled.
The documentation for providers implies you can use a resource AS a schema object in a resource, but does not explain how or why. If it works the way I imagine it, it may work to create a resource that is only used to inject into the host perhaps - but I don't know if that is how it works and if it is how to accomplish it.
I believe I tentatively have found a solution after asking some folks on the gopher slack.
Using AWS Provider Default VPC as a reference, I can "clone" the resource into one with a custom Create/Delete lifecycle
Loose Example:
func defaultResourceConfiguration() *schema.Resource {
drc := resourceConfiguration()
drc.Create = resourceDefaultConfigurationCreate
drc.Delete = resourceDefaultConfigurationDelete
return drc
}
func resourceDefaultConfigurationCreate(d *schema.ResourceData, m interface{}) error {
// double check it exists and update the resource instead
return resourceConfigurationUpdate(d, m)
}
func resourceDefaultConfigurationDelete(d *schema.ResourceData, m interface{}) error {
log.Printf("[WARN] Cannot destroy Default Scope Configuration. Terraform will remove this resource from the state file, however resources may remain.")
return nil
}
This should allow me to provide an identical resource that is designed to interact with the already existing one created by its parent host.
We are using icinga2 for monitoring. We have a lot service checks which are applied dynamically through apply rules. Additionally, these are services applied to a hashmap of database instances which are on various hosts. The long and the short of it is that our service names are determined dynamically so one might be, for example HOST!DBNAME-svcvheck.
So the scenario is that most of these services depend on a database is up, e.g., `HOST!DBNAME-tnsping". Unfortunately, the documentation examples are fairly simple and don't include dynamically creating a parent service reference. What I think I want to do is something like this:
apply Dependency "db-connectivity" to Service {
parent_service_name = "$host.name$!$service.vars.envname$-tnsping"
# also tried variants of this, e.g.
# parent_service_name = host.name + "!" + service.vars.envname + "-tnsping"
child_service_name = service.name
child_host_name = host.name
disable_checks = true
assign where "oracle-db-svc" in service.templates
}
The host doesn't really matter in my case because the dependencies are only the services but the child_host_name is a required field.
No matter what I do I can't seem to get it to recognize the parent service. For example:
Error: Dependency 'scan-szepdb041x.myhost.org!UAT2-beqfilelast!db-connectivity' references a parent host/service which doesn't exist.
The rules for referencing other object variables while applying a Dependency seem a bit different from applying a Service.
Does anyone have any ideas or examples of dynamically apply service dependencies to services which were generated dynamically?
you probably have to loop over existing hosts and see if they match. Then you define dependency inside of a loop.
I had a similar example for dynamically generating disk checks. If i find it, i'll post it here in a few days.
Not sure if that is possible with dependencies, but I'll see.
edit: see if somethig like that will be enough to get you started:
for (server in get_objects(Host)) {
if (match("somename*", server.name)) {
apply Dependency "db-connectivity" + server.name to Service use (server) {
parent_service_name = server.name + service.vars.envvname + "-tnsping"
child_service_name = service.name
child_host_name = host.name
disable_checks = true
assign where "oracle-db-svc" in service.templates
}
}
}
I have inherited some GEB tests that are testing logging into a site (and various error cases/validation warnings).
The test runs through some validation failures and then it attempts to re-navigate to the same page (just to refresh the page/dom) and attempts a valid login. Using GEB's to() method, it detects that you are attempting to navigate to the page you are on, it just calls refresh - the problem here is that attempts to refresh the last POST request, and the driver displays the
"To display this page, Firefox must send information that will repeat any action (such as a search or order confirmation) that was performed earlier"
message - as the test is not expecting this popup, it hangs and the tests timeout.
Is there a way to turn off these warnings in Firefox webdriver? or to auto-ignore/accept them via Selenium or GEB?
GEB Version: 0.9.2,
Selenium Version: 2.39.0
(Also tried with minor version above: 0.9.3 & 2.40.0)
Caveats:
I know about the POST/Re-direct/GET pattern - but am not at liberty to change the application code in this case
The warning message only causes an issue intermittently (maybe 1 in 5 times) - I have put this down to speed/race conditions whereby the test completes the next actions before the message appears - I know a possible solution is to update tests to wait for message to appear and then accept, but my question is, is there a global setting that can just avoid these being triggered/displayed?
That refresh() is there to work around an issue with IE driver which ignores calls to driver.get() with the same url as the current one.
Instead of monkey patching Browser class (which might bite you somewhere down the line or might not) I would change the url of your login page class. You might for example add an insignificant query string - I think that simply a ? at the end should suffice. The driver.currentUrl == newUrl condition will evaluate to false and you will not see that popup anymore.
If I understand you issue properly this might help. In Groovy you can modify a class on the fly.
We use Spock with Geb and I placed this in a Super class which all Spock Spec inherit from. Eg: QSpec extends GebSpec.
It is the original method slightly modified with the original code commented out so you know what has been changed. I use this technique in several required places to alter Geb behaviour.
static {
Browser.metaClass.go = { Map params, String url ->
def newUrl = calculateUri(url, params)
// if (driver.currentUrl == newUrl) {
// driver.navigate().refresh()
// } else {
// driver.get(newUrl)
// }
driver.get(newUrl)
if (!page) {
page(Page)
}
}
}
As of newer version of Doctrine2, I know there is Doctrine\ORM\Configuration#setHydrationCacheImpl()
to pass such as MemcacheCache, etc.
But how can it be done in container?
I'm using two entity_manager: named "default" and "other".
I first tried defining hydration_cache into config.yml like
doctrine:
orm:
default_entity_manager: default
...
entity_managers:
default:
...
metadata_cache_driver:
type: service
id: memcache_driver
...
hydration_cache_driver:
type: service
id: memcache_driver
...
other:
...
note: where memcache_driver is defined by me, instanceof Doctrine\Common\Cache\MemcacheCache
then I got Unrecognized options "hydration_cache_driver" under "doctrine.orm.entity_managers.default".
I also tried to directly tweak container in AppKernel#buildContainer,
but there's no instances of \Doctrine\ORM\Configuration defined as service,
so I couldn't retrieve the Configuration instance.
Any suggestions are welcome.
EDIT:
I'm sure that there is feature for caching hydrated object is re-implemented as of Doctrine 2.2.2.
http://www.doctrine-project.org/jira/browse/DDC-1766
https://github.com/doctrine/doctrine2/blob/2.2.2/tests/Doctrine/Tests/ORM/Functional/HydrationCacheTest.php?source=c
For other simple services, I can easily add methods to call by overwriting whole definitions like
service1:
...
calls:
[method calls]
but for the entity_manager, I'm not sure how to add method calls to them.
So my question in other words, how to configure orm at lower level like without using semantic configuration?
In my case, as hydration cache is hardly used,
so I decided this time to call Query#setHydrationCacheProfile just before each query is executed.
...
$query = $queryBuilder->getQuery();
$cache = $this->container->get('...'); //instanceof MemcacheCache
$query->setHydrationCacheProfile(new CacheProfile(null, null $cache));
$query->execute();
...
There is no such option "hydration_cache_driver", you should use "result_cache_driver" to achieve that.
From Doctrine 2.1, Doctrine can cache results of the queries, but it doesn't cache objects after hydration.
Look at doc about doctrine configuration:
http://symfony.com/doc/master/reference/configuration/doctrine.html
I have already got some "Spring-scheduled" tasks up and running successfully.
What I would like now is to post some specific tweets to a known Twitter account (and already configured on Twitter side) based on some event recurrence.
However, all I see in the OAuth process, esp. in order to get an access token, is that it requires some callback URL before being able to do anything.
I might be mistaken but this seems hard to integrate in the context of a scheduled task.
Isn't there any other way to achieve tweeting?
In conjunction with Spring Scheduling features, I would use Twitter4j to post a tweet in a scheduled job.
Here is a sample:
#Componet
public class TwitterSender {
#Scheduled(fixedRate = 10000)
public void sendTweet() {
Twitter twitter = TwitterFactory.getSingleton();
Status status = twitter.updateStatus(latestStatus);
System.out.println("Status updated to: " + status.getText() + ".");
}
}
If you need more information you can check the test case for sending update status with Twitter4j. Or you can just dive and see the source.
It may be a bit of a leap in terms of learning curve, but have you looked at spring-integration's twitter:outbound-channel-adapter ?
<twitter:outbound-channel-adapter twitter-template="twitterTemplate"
channel="twitterChannel"/>
http://static.springsource.org/spring-integration/docs/latest-ga/reference/html/twitter.html