I am using the fantastic StackExchange.Redis library to implement ObjectCache. One of the interface methods to implement in ObjectCache is long GetCount(...) which returns the number of keys in the database. It looks like this can be satisfied by the IServer.DatabaseSize(...) method in StackExchange.Redis.
I plan on fetching the server endpoints from ConnectionMultiplexer.GetEndPoints(), getting an IServer for each endpoint, and then querying the database size for each database I am interested in on each server (ignore size discrepancies for the moment).
Now, ConnectionMultiplexer.GetEndPoints() has an optional parameter called "configuredOnly". What is the consequence of not providing it, versus true, versus false?
In the ConnectionMultiplexer.GetEndPoints() implementation, I see that it returns the EndPoints from the multiplexer configuration if configuredOnly is true, or else returns EndPoints from an array called "serverSnapshot".
As best as I can tell, "serverSnapshot" is populated here, which seems to be populated as servers are connected, or at least are attempted to be connected to.
Does GetEndPoints(true) return all EndPoints that were configured on the ConnectionMultiplexer? Does GetEndPoints() and GetEndPoints(false) return EndPoints that actually are connected/valid? The documentation for the GetEndPoints method with respect to the configuredOnly parameter is sparse, and my subsequent use of the returned EndPoints needs one behavior and not the other.
When configuredOnly is set to true, GetEndPoints() only returns endpoints for the Redis servers explicitly specified in the call to ConnectionMultiplexer.Connect(). Alternately when configuredOnly is false, endpoints are returned for every Redis servers in the cluster, whether or not they were specified in the initial ConnectionMultiplexer.Connect() call.
Somewhat strangly, if you use DNS names in the ConnectionMultiplexer.Connect() call, GetEndPoints(false) will return rows for both the DNS name and also the resolved IP address. For example, with a six-node Redis cluster the following code:
ConnectionMultiplexer redis = ConnectionMultiplexer("localhost:6379,localhost:6380");
foreach (var endpoint in redis.GetEndPoints(false))
{
Console.WriteLine(endpoint.ToString());
}
will output
$127.0.0.1:6379
Unspecified/localhost:6379
Unspecified/localhost:6380
127.0.0.1:6380
127.0.0.1:6381
127.0.0.1:6382
127.0.0.1:6383
127.0.0.1:6384
If I had called redis.GetEndPoints(true), only Unspecified/localhost:6379 and Unspecified/localhost:6380 would be returned.
Related
I want to hit a service which has a paginated response using Web Client. ie. I hit a service, check if it returns hasMoreElements as TRUE, then call the service again with updated request parameters like START_ROW, END_ROW, PAGE_NUMBER. What is the best approach to achieve this? Currently am just looping through the results and hitting the service again. But their should be a better approach to this. PFB my pseudocode. Any libraries I can use?
boolean hasMoreElements=true;
while(!hasMoreElements==false)
{
response=webClient.post().header(HEADERS).bodyValue(REQUEST).block();
Get the NEW START ROW, END ROW, AND PAGE NUMBER and SET in the REQUEST
Get the hasMoreElements value
}
Use JPARepository with paging for this.
You can return a list of objects and check if its length is less then the limit passed, if yes then you can stop fetching.
You could also return a Page or a Slice instead which gives you a little bit more information about the current and next fetch cycle
I want to store all the blockchain data in offchain database.
rpc has a function called EXPERIMENTAL_changes, I was told that I can do that by http polling of this method but I am unable to find out how to use it.
http post https://rpc.testnet.near.org jsonrpc=2.0 id=dontcare method=EXPERIMENTAL_changes \ params:='{ "changes_type": "data_changes", "account_ids": ["guest-book.testnet"], "key_prefix_base64": "", "block_id": 19450732 }'
For example here the results give:
"change": { "account_id": "guest-book.testnet", "key_base64": "bTo6Mzk=", "value_base64": "eyJwcmVtaXVtIjpmYWxzZSwic2VuZGVyIjoiZmhyLnRlc3RuZXQiLCJ0ZXh0IjoiSGkifQ==" }
What is key_base64?
Decoding it to string gives m::39
What is m::39?
For example, I have the following state data in the rust structure.
pub struct Demo {
user_profile_map: TreeMap<u128, User>,
user_products_map: TreeMap<u128, UnorderedSet<u128>>, // (user_id, set<product_id>)
product_reviews_map: TreeMap<u128, UnorderedSet<u128>>, // (product_id, set<review_id>)
product_check_bounty: LookupMap<u128, Vector<u64>>
}
How to know anything gets changed in these variables?
Will I have to check every block id for the point the contract is deployed, to know where there is the change?
I want to store all the blockchain data in offchain database.
If so, I recommend you take a look at the Indexer Framework, which allows you to get a stream of blocks and handle them. We use it to build Indexer for Wallet (keeps track of every added and deleted access key, and stores those into Postgres) and Indexer for Explorer (keeps track of every block, chunk, transaction, receipt, execution outcome, state changes, accounts, and access keys, and stores all of that in Postgres)
What is m::39?
Contracts in NEAR Protocol have access to the key-value storage (state), so at the lowest-level, you operate with key-value operations (NEAR SDK for AssemblyScript defines Storage class with get and set operations, and NEAR SDK for Rust has storage_read and storage_write calls to preserve data).
Guest Book example uses a high-level abstraction called PersistentVector, which automatically reads and writes its records from/to NEAR key-value storage (state). As you can see:
export const messages = new PersistentVector<PostedMessage>("m");
Guest Book defines the messages to be stored in the storage with m prefix, hense you see m::39, which basically means it is messages[39] stored in the key-value storage.
What is key_base64?
As key-value storage implies, the data is stored and accessed by keys, and the key can be binary, so base64 encoding is used to enable JSON-RPC API users with a way to query those binary keys as well (there is no way you can pass a raw binary blob in JSON).
How to know anything gets changed in these variables? Will I have to check every block id for the point the contract is deployed, to know where there is the change?
Correct, you need to follow every block, and check the changes. That is why we have built the Indexer Framework in order to enable community building services on top of that (we chose to build applications Indexer for Wallet and Indexer for Explorer, but others may decide to build GraphQL service like TheGraph)
I'm bothering with situation when I want to emit query update via queryUpdateEmitter but in different module (microservice). I have application built upon microservices and both are connected to the same Axon Server. First service creates subscriptionQuery, and sends some commands. After a while (through few commands and events) second service handles some event, and emits update for firstly subscribed query. Unfortunately it seems like this emit doesn't get to subscriber. Queries are exactly the same and sits in the same packages.
Subscription:
#GetMapping("/refresh")
public Mono<MovieDTO> refreshMovies() {
commandGateway.send(
new CreateRefreshMoviesCommand(UUID.randomUUID().toString()));
SubscriptionQueryResult<MovieDTO, MovieDTO> refreshedMoviesSubscription =
queryGateway.subscriptionQuery(
new GetRefreshedMoviesQuery(),
ResponseTypes.instanceOf(MovieDTO.class),
ResponseTypes.instanceOf(MovieDTO.class)
);
return refreshedMoviesSubscription.updates().next();
}
Emitter:
#EventHandler
public void handle(DataRefreshedEvent event) {
log.info("[event-handler] Handling {}, movieId={}",
event.getClass().getSimpleName(),
event.getMovieId());
queryUpdateEmitter.emit(GetRefreshedMoviesQuery.class, query -> true,
Arrays.asList(
MovieDTO.builder().aggregateId("as").build(),
MovieDTO.builder().aggregateId("be").build()));
}
This situation is even possible in the newest version of Axon? Similar configuration but within one service is working as expected.
#Edit
I have found a workardound for this situation:
Second service instead of emitting query via queryUpdateEmitter, publishes event with list of movies
First service handles this event and then emits update via queryUpdateEmitter
But still I'd like to know if there is a way to do this using queries only, because it seems natural to me (commandGateways/eventGateways works as expected, queryUpdateEmitter is the exception).
This follows from the implementation of the QueryUpdateEmitter (regardless of using Axon Server yes/no).
The QueryUpdateEmitter stores a set of update handlers, referencing the issued subscription queries. It however only maintains the issued subscription queries handled by the given JVM (as the QueryUpdateEmitter implementation is not distributed).
It's intent is to be paired in the component (typically a Query Model "projector") which answers queries about a given model, updates the model and emits those updates.
Hence, placing the QueryUpdateEmitter operations in a different (micro)service as where the query is handled will not work.
I have a publisher that executes a long-running and large query on MongoDB and returns the data in a Flux. Entities that are marked in the database as "processed" will be filtered out and the entities are then buffered and passed to a concatMap operator (so that all buffered ≤elements are processed before elements in the next buffer are processed). It looks something like this:
Flux<Entity> entitiesFromMongoDb = myMongoRepository.executeLargeQuery();
entitiesFromMongoDb.filter(entity -> !entity.isProcessed())
.buffer(10)
.concatMap(bufferedEntityList ->
Flux.fromIterable(bufferedEntityList)
.flatMap(makeExternalCall)
.then()));
Where makeExternalCall calls a third-party remote server and sets the entity to processed after the call has been made. In most cases this works fine, but when the remote server is really slow or has an error then makeExternalCall will retry (with exponential backoff) the operation to the remote server. In some cases it can take quite a while before all 10 external calls have been processed. In fact it can take so long that the myMongoRepository.executeLargeQuery() publisher is restarted and the query is executed again. Now we run into a problem that I'll try to describe here:
Entity A is read from the database (i.e. it's returned in the flux produced by myMongoRepository.executeLargeQuery()). It's not yet marked as "processed" which means that entity.isProcessed() will return false and it'll be retained in the stream.
The external server is really slow or down so that makeExternalCall is forced to retry the operation before entity A has been marked as "processed" in the DB.
myMongoRepository.executeLargeQuery() is restarted and the query is executed again.
Entity A is read from the database once more. But the problem is that there's already another instance of entity A in-flight since it has not yet been marked as "processed" by the previous call to myMongoRepository.executeLargeQuery().
This means that makeExternalCall will be called twice for entity A, which is not optimal!
I could make an additional request to the DB and check the status of processed for each entity in the makeExternalCall method, but this will cause additional load (since an extra request is necessary for each entity) to the DB which is not optimal.
So my question is:
Is there a way to somehow "restart" the entire stream, and thus clear intermediary buffers (i.e. remove entity A that is in-flight from the ongoing stream) when the MongoDB query triggered by myMongoRepository.executeLargeQuery() is restarted/re-executed? Or is there a better way to handle this?
I'm using Spring Boot 2.2.4.RELEASE, project reactor 3.3.2.RELEASE and spring-boot-starter-data-mongodb-reactive 2.2.4.RELEASE.
Not sure If I understood the problem completely. But trying to answer as it sounds interesting.
As you need to be aware of the requests which are already being processed by the makeExternalCall, can you maintain a set / local cache which contains the entities which are being processed?
Set<Entity> inProgress = new HashSet<>();
Flux<Entity> entitiesFromMongoDb = myMongoRepository.executeLargeQuery();
entitiesFromMongoDb.filter(entity -> !entity.isProcessed())
.buffer(10)
.map(bufferedEntityList -> { // remove the inprogress requests to avoid redundant processing
bufferedEntityList.removeIf(inProgress::contains);
return bufferedEntityList;
})
.concatMap(bufferedEntityList ->
inProgress.addAll(bufferedEntityList);
Flux.fromIterable(bufferedEntityList)
.flatMap(makeExternalCall) //assuming once processed, it emits the entity object
.map(entity -> { //clear growing set
inProgress.remove(entity);
return entity;
})
.then()));
This approach is not a good solution when you need to scale your application horizontally. In that case instead of maintaining a local cache, you could go for an external cache server like redis.
I have written a REST API (javax.ws.rs) that uses the high-level Elasticsearch API. This is with ES 7.2.
The client needs to index a record, then execute a search that includes that record and there is some delay after the index operation before the document will actually appear in searches.
Is there any way to block the index operation until the newly index record appears in search results?
Failing that, is there any way to get an asynchronous notification that the document is now searchable?
To give an idea of my use case, here is the code from the client side:
const cr = await this.client.dNodeCreate(fixedNode).toPromise();
const fr = await this.client.dNodeGetById(cr._id).toPromise();
await this.client.dNodeCreate(replyRoot).toPromise();
The first line causes a Index request to ES, and returns the status object. That object includes the ID of the new document.
The second line fetches the record by ID. This always works.
The third line fails. The document it attempts to index is dependent on the first document, which the REST middleware attempts to look up by a search (not by the ID). This is the equivalent of an SQL relation enforced by the REST layer.
I can always make the code work by introducing a delay (say 1500ms) before the third call but this is really a non-robust solution. It might always work in development mode (all the servers are on my laptop and no other users) but there is no way to predict how long the delay needs to be in actual production.
UPDATE: Solved.
The marked answer below seems to do the trick. For reference, the needed call in the Java API looks like this:
IndexRequest req = new IndexRequest(DNode.INDEX);
req.setRefreshPolicy(WriteRequest.RefreshPolicy.WAIT_UNTIL);
The 'refresh' parameter is what you are looking for. From the Elasticsearch documentation:
Refresh (Index API): ) If true, Elasticsearch refreshes the affected shards to make this operation visible to search, if wait_for then wait for a refresh to make this operation visible to search, if false do nothing with refreshes. Valid values: true, false, wait_for. Default: false
So your index request should look something like this:
PUT /<index>/_doc/<_id>?refresh=wait_for
I do not believe there is a built-in way to get an asynchronous notification that the document is now searchable. That being said, if you already have access to the document ID it might make more sense to use that in the code instead of a search.