azure logic app with table storage get last rowKey - sorting

How can I use the "Get Entity for Azure table storage" connector in a Logic App to return the last rowKey.
This would be used in situation where the rowkey is say an integer incremented each time a new entity is added. I recognize the flaw in design of this but this question is about how some sort of where clause or last condition could be used in the Logic app.
Currently the Logic App code view snippet looks like this:
"actions": {
"Get_entity": {
"inputs": {
"host": {
"connection": {
"name": "#parameters('$connections')['azuretables']['connectionId']"
}
},
"method": "get",
"path": "/Tables/#{encodeURIComponent('contactInfo')}/entities(PartitionKey='#{encodeURIComponent('a')}',RowKey='#{encodeURIComponent('b')}')"
},
"runAfter": {},
"type": "ApiConnection"
}
Where I have the hard coded:
RowKey='#{encodeURIComponent('b')}'
This is fine if I always want this rowKey. What I want though is the last rowKey so something sort of like:
RowKey= last(RowKey)
Any idea on how this can be achieved?

This is fine if I always want this rowKey. What I want though is the last rowKey so something sort of like: RowKey= last(RowKey)
AFAIK, there is no build-in functions for you to achieve this purpose. I assumed that you could use the Azure Functions connector to retrieve the new RowKey value. Here are the detailed steps, you could refer to them:
For test, I created a C# Http Trigger function, then add a Azure Table Storage Input, then retrieve all the items under the specific PartitionKey, then order by the RowKey and calculate the new Row Key.
function.json:
{
"bindings": [
{
"authLevel": "function",
"name": "req",
"type": "httpTrigger",
"direction": "in"
},
{
"name": "$return",
"type": "http",
"direction": "out"
},
{
"type": "table",
"name": "inputTable",
"tableName": "SampleTable",
"take": 50,
"connection": "AzureWebJobsDashboard",
"direction": "in"
}
],
"disabled": false
}
run.csx:
#r "Microsoft.WindowsAzure.Storage"
using Microsoft.WindowsAzure.Storage.Table;
using System.Net;
public static async Task<HttpResponseMessage> Run(HttpRequestMessage req, IQueryable<SampleTable> inputTable,TraceWriter log)
{
log.Info("C# HTTP trigger function processed a request.");
// parse query parameter
string pk = req.GetQueryNameValuePairs()
.FirstOrDefault(q => string.Compare(q.Key, "pk", true) == 0)
.Value;
// Get request body
dynamic data = await req.Content.ReadAsAsync<object>();
// Set name to query string or body data
pk = pk ?? data?.pk;
if(pk==null)
return req.CreateResponse(HttpStatusCode.BadRequest, "Please pass a pk on the query string or in the request body");
else
{
var latestItem=inputTable.Where(p => p.PartitionKey == pk).ToList().OrderByDescending(i=>Convert.ToInt32(i.RowKey)).FirstOrDefault();
if(latestItem==null)
return req.CreateResponse(HttpStatusCode.OK,new{newRowKey=1});
else
return req.CreateResponse(HttpStatusCode.OK,new{newRowKey=int.Parse(latestItem.RowKey)+1});
}
}
public class SampleTable : TableEntity
{
public long P1 { get; set; }
public long P2 { get; set; }
}
Test:
For more details about Azure Functions Storage table bindings, you could refer to here.

azure table storage entities are sorted lexicographically. So choose a row key that actually decrements every time you add a new entity, ie. if your row key is an integer that gets incremented when new entity is created than choose your row key as Int.Max - entity.RowKey. The latest entity for that partition key will always be on the top since it is going to have the lowest row key, so all you need to do then to retrieve it is query only with partition key and Take(1). This is called Log Tail pattern, if you want to read more about it.

Related

Prevent a repeatable component on Strapi from being deleted by passing an empty array in the Repeater Field mutation?

What would be the best way to prevent a repeatable component on Strapi from being inadvertently deleted by passing an empty array in the mutation?
As an example, I have the following Collection Type:
"kind": "collectionType",
"collectionName": "books",
"info": {
"name": "Books"
},
"options": {
"increments": true,
"timestamps": true
},
"attributes": {
"pages": {
"type": "component",
"repeatable": true,
"component": "page.pages"
}
}
}
If I have passed in an empty array for the repeatable fields (pages) then it appears to remove the entire repeatable component. For context, I want users to be able to update the elements in the repeatable component but restrict wiping everything out. Per this thread, it appears I have to send in the entire array in the mutation or put request. What I don't want is for someone to send an empty array and delete everything.
mutation {
updateBook(input: {
where: { id: "602ea4add988a28e57fb355a" },
data: {
pages: []
}
}){
book {
id
pages {
text
}
}
}
}
Aparently you have to query the data on the component and then add it again with the new entry
"in strapi groups i can't just add the new entry, i have to include all previous ones or otherwise they will be deleted !
currently to update an entry in a group you have to use the parent update(), or there is an other way ?"
Originally posted by #bouhendfaycal in https://github.com/strapi/strapi/issues/4590#issuecomment-558741136

How to create a HashMap with custom object as a key?

In Elasticsearch, I have an object that contains an array of objects. Each object in the array have type, id, updateTime, value fields.
My input parameter is an array that contains objects of the same type but different values and update times. Id like to update the objects with new value when they exist and create new ones when they aren't.
I'd like to use Painless script to update those but keep them distinct, as some of them may overlap. Issue is that I need to use both type and id to keep them unique. So far I've done it with bruteforce approach, nested for loop and comparing elements of both arrays, but I'm not too happy about that.
One of the ideas is to take array from source, build temporary HashMap for fast lookup, process input and later store all objects back into source.
Can I create HashMap with custom object (a class with type and id) as a key? If so, how to do it? I can't add class definition to the script.
Here's the mapping. All fields are 'disabled' as I use them only as intermidiate state and query using other fields.
{
"properties": {
"arrayOfObjects": {
"properties": {
"typ": {
"enabled": false
},
"id": {
"enabled": false
},
"value": {
"enabled": false
},
"updated": {
"enabled": false
}
}
}
}
}
Example doc.
{
"arrayOfObjects": [
{
"typ": "a",
"id": "1",
"updated": "2020-01-02T10:10:10Z",
"value": "yes"
},
{
"typ": "a",
"id": "2",
"updated": "2020-01-02T11:11:11Z",
"value": "no"
},
{
"typ": "b",
"id": "1",
"updated": "2020-01-02T11:11:11Z"
}
]
}
And finally part of the script in it's current form. The script does some other things, too, so I've stripped them out for brevity.
if (ctx._source.arrayOfObjects == null) {
ctx._source.arrayOfObjects = new ArrayList();
}
for (obj in params.inputObjects) {
def found = false;
for (existingObj in ctx._source.arrayOfObjects) {
if (obj.typ == existingObj.typ && obj.id == existingObj.id && isAfter(obj.updated, existingObj.updated)) {
existingObj.updated = obj.updated;
existingObj.value = obj.value;
found = true;
break;
}
}
if (!found) {
ctx._source.arrayOfObjects.add([
"typ": obj.typ,
"id": obj.id,
"value": params.inputValue,
"updated": obj.updated
]);
}
}
There's technically nothing suboptimal about your approach.
A HashMap could potentially save some time but since you're scripting, you're already bound to its innate inefficiencies... Btw here's how you initialize & work with HashMaps.
Another approach would be to rethink your data structure -- instead of arrays of objects use keyed objects or similar. Arrays of objects aren't great for frequent updates.
Finally a tip: you said that these fields are only used to store some intermediate state. If that weren't the case (or won't be in the future), I'd recommend using nested arrays to enable querying independently of other objects in the array.

How do I update MongoDB query results using inner query?

BACKGROUND
I have a collection of json documents that represent chemical compounds. A compound will have an id and a name. An external process generates new compound documents at intervals, and ids may change across iterative generations. Compound documents whose compound ids have changed need to be updated to point to the most recent iterations ids, and as such, a "lastUpdated" field and "relatedCompoundIds" field will be added. To demonstrate, consider the following compounds across 3 steps:
Step 1: initial compound document for 'acetone' is generated with id="001".
{
"id": "001",
"name": "acetone",
"lastUpdated": "2000-01-01",
}
Step 2: another iteration generates acetone, but with a different id.
{
"id": "001",
"name": "acetone",
"lastUpdated": "2000-01-01"
}
{
"id": "002",
"name": "acetone",
"lastUpdated": "2000-01-02"
}
Step 3: compound with id of "001" will append a "relatedCompoundIds" array pointing to any other compounds with the same name.
{
"id": "001",
"name": "acetone",
"lastUpdated": "2000-01-02",
"relatedCompoundIds": ["002"]
}
{
"id": "002",
"name": "acetone",
"lastUpdated": "2000-01-02"
}
I'm using MongoDB to house these records, and to resolve relatedCompoundId "pointers". I'm accessing Mongo using Spring ReactiveMongoTemplate. My process is as follows:
Upsert newly generated compounds into MongoDB.
For each record where "lastUpdated" is before now:
Get all related compounds (searching by name), and set "relatedCompoundIds".
CODE
public class App {
public static void main(String[] args) {
public static ReactiveMongoTemplate mongoOps = new ReactiveMongoTemplate(MongoClients.create(),
"CompoundStore");
Date updatedDate = new Date();
upsertAll(updatedDate, readPath);
setRelatedCompounds(updatedDate);
}
private static void upsertAll(Date updatedDate, String readPath) {
// [upsertion code here] <- this is working fine
}
private static void setRelatedCompounds(Date updatedDate) {
mongoOps.find(//
Query.query(Criteria.where("lastUpdated").lt(updatedDate)), Compound.class, "compound")//
.doOnNext(compound -> {
findRelatedCompounds(updatedDate, compound)//
.doOnSuccess(rc -> {
if (rc.size() > 0) {
compound.setRelatedCompoundIDs(rc);
mongoOps.save(Mono.just(compound)).subscribe();
}
})//
.subscribe();
}).blockLast();
}
private static Mono<List<String>> findRelatedCompounds(Date updatedDate, Compound compound) {
Query query = new Query().addCriteria(new Criteria().andOperator(//
Criteria.where("lastUpdated").gte(updatedDate), //
Criteria.where("name").is(compound.getName)));
query.fields().include("id");
return mongoOps.find(query, Compound.class)//
.map(c -> c.getId())//
.filter(cid -> !StringUtils.isEmpty(cid))//
.distinct().collectSortedList();
}
}
ERROR
Upon running, I get the following error:
17:08:35.957 [Thread-12] ERROR org.mongodb.driver.client - Callback onResult call produced an error
com.mongodb.MongoException: org.springframework.data.mongodb.UncategorizedMongoDbException: Too many operations are already waiting for a connection. Max number of operations (maxWaitQueueSize) of 500 has been exceeded.; nested exception is com.mongodb.MongoWaitQueueFullException: Too many operations are already waiting for a connection. Max number of operations (maxWaitQueueSize) of 500 has been exceeded.
at com.mongodb.MongoException.fromThrowableNonNull(MongoException.java:79)
Is there a better way to accomplish what I'm trying to do?
How do I adjust backpressure so as not to overload the mongo?
Other advice?
EDIT
The above error can be resolved by adding a limitRate modifier after the find method inside setRelatedCompounds.
private static void setRelatedCompounds(Date updatedDate) {
mongoOps.find(//
Query.query(Criteria.where("lastUpdated").lt(updatedDate)), Compound.class, "compound")//
.limitRate(500)//
.doOnNext(compound -> {
// do work here
}).subscribe();
}).blockLast();
}
Still open to suggestions for alternative solutions.

GraphQL: Explore API without a wildcard (*)?

I am new to GraphQL and I wonder how I can explore an API without a possible wildcard (*) (https://github.com/graphql/graphql-spec/issues/127).
I am currently setting up a headless Craft CMS with GraphQL and I don't really know how my data is nested.
Event with the REST API I have no chance of just getting all the data, because I have to setup all the endpoints and therefore I have to know all field names as well.
So how could I easily explore my CraftCMS data structure?
Thanks for any hints on this.
Cheers
merc
------ Edit -------
If I use #simonpedro s suggestion:
{
__schema {
types {
name
kind
fields {
name
}
}
}
}
I can see a lot of types (?)/fields (?)...
For example I see:
{
"name": "FlexibleContentTeaser",
"kind": "OBJECT",
"fields": [
{
"name": "id"
},
{
"name": "enabled"
},
{
"name": "teaserTitle"
},
{
"name": "text"
},
{
"name": "teaserLink"
},
{
"name": "teaserLinkConnection"
}
]
But now I would like to know how a teaserLink ist structured.
I somehow found out that the teaserLink (it is a field with the type Entries, where I can link to another page) has the properties url & title.
But how would I set up query to explore the properties available within teaserLink?
I tried all sorts of queries, but I am always confrontend with messages like this:
I would be really glad if somebody could give me another pointer how I can find out which properties I can actually query...
Thank you
As far as I'm concerned currently there is no graphql implementation with that capability. However, if what you want to do is to explore the "data structure", i.e, the schema, you should use schema instrospection, which was thought for that (explore the graphql schema). For example, a simple graphql instrospection query would be like this:
{
__schema {
types {
name
kind
fields {
name
}
}
}
}
References:
- https://graphql.org/learn/introspection/
UPDATE for edit:
What you want to do I think is the following:
Make a query like this
{
__schema {
types {
name
kind
fields {
name
type {
fields {
name
}
}
}
}
}
}
And then find the wished type field to grab more information (the fields) from it. Something like this (I don't know if this works, just an idea):
const typeFlexibleContentTeaser = data.__schema.types.find(t => t === "FlexibleContentTeaser")
const teaserLinkField = typeFlexibleContentTeaser.fields.find(f => f.name === "teaserLink")
const teaserLinkField = teaserLinkField.type.fields;
i.e, you have to transverse recursively through the type field.

couchDB- complex query on a view

I am using cloudantDB and want to query a view which looks like this
function (doc) {
if(doc.name !== undefined){
emit([doc.name, doc.age], doc);
}
what should be the correct way to get a result if I have a list of names(I will be using option 'keys=[]' for it) and a range of age(for which startkey and endkey should be used)
example: I want to get persons having name "john" or "mark" or "joseph" or "santosh" and lie between age limit 20 to 30.
If i go for list of names, query should be keys=["john", ....]
and if I go for age query should use startkey and endkey
I want to do both :)
Thanks
Unfortunately, you can't do so. Using the keys parameter query the documents with the specified key. For example, you can't only send keys=["John","Mark"]&startkey=[null,20]&endkey=[{},30]. This query would only and ONLY return the document having the name John and Mark with a null age.
In your question you specified CouchDB but if you are using Cloudant, index query might be interesting for you.
You could have something like that :
{
"selector": {
"$and": [
{
"name": {
"$in":["Mark","John"]
}
},
{
"year": {
"$gt": 20,
"$lt": 30
}
}
]
},
"fields": [
"name",
"age"
]
}
As for CouchDB, you need to either separate your request (1 request for the age and 1 for the people) or you do the filtering locally.

Resources