Lets say I have a large object stored in ObjectStorage (in MBs) --
What will happen now if someone (lets say connection A) tries to retrieve it (downloading it) while someone else (lets say connection B) is updating the object by uploading a new version of blob into the same object?
Will connection A still see the old version while the new version of blob has not completely uploaded or will see a corrupted data? At what time A will start seeing the new version of the object?
If you are downloading an object while it is overwritten you will continue to see the old object.
The system does out-of-place overwrites so the old data is available however if the download goes on for a very long time the old data will be garbage collected and the download will fail (this is a very extreme scenario)
Related
I am trying to figure out how GlobalKTable is working and noticed that my in memory key value store is not filled in case a restart. However the documentation sounds that it is filled in case a restart since the whole data is duplicated on clients.
When I debug my application see that there is a file on /tmp/kafka-streams/category-client-1/global/.checkpoint and it is including an offset about my topic. This might be maybe necessary for stores which are persisting their data and improve restarts however since there is an offset in this file, my application skips restoring its state.
How can I be sure that each restart or fresh start includes whole data of my topic?
Because you are using in-memory store I assume that you are hitting this bug: https://issues.apache.org/jira/browse/KAFKA-6711
As a workaround, you can delete the local checkpoint file for the global store -- this will trigger the bootstrapping on restart. Or you switch back to default RocksDB store.
I saw Neo4j can run as Impermanent DB for unit testing porpouses, I'm not sure if this fits my needs. I have my data stored in neo4j the usual way (persistent) but, starts from my data, I want to let each user start an "experimental session": the users add/delete nodes and relationships, but NOT in permanent way, just experimenting with the data (after that session the edits should be lost). The edits shouldn't be saved and obiouvsly they shouldn't be visibile to the others. What's the best way to accomplish that?
Using impermanent database should work. You would
need to import the data to each new database
spring-data-neo4j is not able to connect to multiple databases (in current release), you would need to start multiple instances of your application, e.g. in a tomcat container
when your application stops (or crashes) you would obviously lose data
Or you could potentially use only 1 database with the base data being public (= visible to everyone) and then for all new nodes/relationships you can add owner property.
When querying the data you would check the property is either public or the current user.
At the end of the session you would just delete all nodes and relationships with given owner.
If you also want to edit existing data then it gets more complicated, you could create a copy of the node/relationship and somehow handle that, or if it's not too large copy whole dataset.
You can build a docker image from the neo4j base image (or build your own) and copy your graph.db into it.
Then you can have every user start a docker container from said image.
If that doesn't answer your question, more info is needed.
So playing around with iMessages and thinking of ways to back them up and various things.
I found their location at ~/Library/Messages.
There are three files
1. chat.db
2. chat.db-wal
3. chat.db-shm
If I run a node script that watches for file changes while sending a iMessage to someone I see chat.db-wal is changed instantly but chat.db takes awhile to update.
I would like to get the messages as soon as possible, but I am not sure I can read the .db-wal file. Anyone know if I can read that file? Or why the .db file seems to take longer to update?
Thanks.
Everything is fine. Your data is there. This is just how SQLite works.
In order to support ACID transactions, where your data is guaranteed to be stored properly in the case of crashes or power-offs, SQLite first writes your data into a "write-ahead log" (the *-wal file). When the database is properly closed, or the write-ahead log gets too full, SQLite will update the database file with the contents of the log.
SQLite, when reading, will consult the write-ahead log first, even if multiple connections are using the same database. Data in the log is still "in the database".
SQLite should apply the log to the database as part of closing the database. If it does not, you can run PRAGMA wal_checkpoint; to manually checkpoint the log file.
Corollary to this: do not delete the -wal file, especially if you have not cleanly closed the database last time you used it.
More information about write-ahead logging in SQLite can be found in the SQLite documentation.
Normally when I backed up the core data file for my app, I would just copy the .sqlite file to another location while the app was running. But now that journaling (wal) is enabled, this does not work anymore. I cannot see a way for NSPersistentStoreCordinator or NSManagedObjectContext to write a new file. I'm guessing maybe I have 2 methods:
Close the persistent store and opening it again with #{#"journal_mode" : #"DELETE"} and then copy the .sqlite file.
Add another persistent store and maybe copy from the original ps to the new one ?
Any better ideas ?
Thank you.
Changing the journal mode will eliminate the journal files, so it's simple. I don't know that I'd trust it for your use, though-- because there's no guarantee that Core Data has actually flushed all new changes to the SQLite file. It might be OK, but there might be some in-memory changes that Core Data hasn't written out yet. This is almost certainly safe, but there's a small chance that it won't work right once in a while.
Option 2 would be safer, though more work. I'd create the second persistent store using NSPersistentStoreCoordinator's migratePersistentStore:toURL:options:withType:error: method (which the docs specifically mention as being useful for "save as" operations). Telling Core Data to create the copy for you should ensure that everything necessary is actually copied. Just don't do this on your main persistent store coordinator, because after migration, the PSC drops the reference to the original store object (the file's still there, but it's no longer used by that PSC). The steps would be
Create a new migrate-only NSPersistentStoreCoordinator and add your original persistent store file.
Use this new PSC to migrate to a new file URL.
Drop all reference to this new PSC, don't use it for anything else.
The past days I've been working with Azure Caching. Good practice is to use the local cache functionality to prevent roundtrips to the distributed cache.
As can be read in the documentation; when you make a call to dataCache.Get(...), the application first checks if a version in the local cache is available and, if not, the object is retrieved from the distributed cache.
The problem is that the local version can be older than the distributed version of the object. To solve this problem, the method 'dataCache.GetIfNewer(...)' is available that can be used to check if the version of the local object differs from the distributed version and, if it does, it returns the new object.
So far, so good...now my questions; I've created two seperate applications (app A en app B) to test the Azure Caching mechanism. Both applications run on two different (physical) locations so they both have their own local-cache but they both use the same distributed cache.
Is it true that something has changed in the process of invalidating the local cache? I've tested the following scenario and found out that the local cache is update automatically:
App A stores the value "123" in the distributed cache using the key "CacheTest"
App B uses the dataCache.Get(...) method to retrieve the object for the key "CacheTest" which cannot be found in the local cache so it is retrieved from the distributed cache en returns the object with value "123".
App A changes the object with key "CacheTest" to the value "456"
App B uses the datacache.Get(...) method to (again) retrieve the object. Because the object should be in the local cache, I would expect the value "123" but it returns the new value "456"!
How strange is that? Is something changed in Azure Caching lately? And yes...I'm sure that I've turned on local caching and yes, I've set the time-out on the local cache to 3600 seconds (1 hour).
Can somebody confirm that Azure Caching has been changed?
Edit for Nick:
So what you're saying is that the next lines of code that I've found on a Dutch Microsoft site are nonsense? When the local-cache is updated automatically, there's no need to call the 'GetIfNewer' method: http://www.dotnetmag.nl/Artikel/1478/Windows-Azure-AppFabric-Caching
///
/// Ensures that the newest version of a cached object is returned
/// from either the local cache or the distrbuted cache.
///
public TCachedObjectType GetNewest<TCachedObjectType>(string key) :
where TCachedObjectType : class
{
DataCacheItemVersion version;
// Gets cached object from local cache if it exists.
// Otherwise gets cached object from distributed cache and
// adds it to local cache.
object cachedObject = cache.Get(key, out version);
// Gets cached object from distributed cached if it is newer
// than given version.
// If newer it adds it to local cache.
object possiblyNewerCachedObject =
cache.GetIfNewer(key, ref version);
if (possiblyNewerCachedObject != null)
{
// Cached object from distributed cache is newer
// than cached object from local cache.
cachedObject = possiblyNewerCachedObject;
}
return cachedObject as TCachedObjectType;
}
If the described behaviour is the same as appfabric velocity, the behaviour described is as expected. When local caching is enabled it means that when a given node requests a cache item from the distributed cache, it asks the distributed cache what the current version is.
If the locally cached version matches the distributed version, it returns the data from the local cache. If not, it retrieves the latest value from the distributed cache, caches it locally and then returns it.
The idea is that if any node updates the key, all nodes will always be ensured to get the latest version even if appfabric had already cached them locally. The distributed cache keeps track of the latest versions and where their data is stored.