Parse Query by subfield/dot notation - parse-platform

tl;dr
Can ParseCloud/MongoDB filter by Pointer<class>.filed ? By
Pointer<class>.Pointer<class> ? By existence of data in that filed?
Long question:
Round is object which will be played automatically when time will come.
Payment object which indicates that user made payment. When payment being spent we set field round to it.
Player which links online User with Payment
I need to query player for few conditions:
Player
online
has valid(no round and valid equal to 'valid') payment
Player
user equal to specific user
has no payment
Player
user equal to specific user
has valid(no round and valid equal to 'valid') payment
And I made everything to work except validating Payment inside Player query.
Here is condition 1 from the list.
var query = new Parse.Query(keys.Player);
query.skip(0);
query.limit(oneRoundMaxPlayers);
query.greaterThanOrEqualTo(keys.last_online_date, lastAllowedOnline);
// looks like no filter applied here
query.doesNotExist("payment.round");
query.exists(keys.payment);
// This line will make query return 0 elements
// query.equalTo("payment.valid", "valid");
query.include(keys.user);
query.include(keys.payment);
Here is 2 OR 3
var queryPaymentExists = new Parse.Query(keys.Player);
queryPaymentExists.skip(0);
queryPaymentExists.limit(1);
queryPaymentExists.exists(keys.payment);
//This line not filtering
queryPaymentExists.doesNotExist(keys.payment + "." + keys.round);
queryPaymentExists.equalTo(keys.user, user);
// This line makes query always return 0 elements
// queryPaymentExists.equalTo(keys.payment + "." + keys.valid, keys.payment_valid);
var queryPaymentDoesNotExist = new Parse.Query(keys.Player);
queryPaymentDoesNotExist.skip(0);
queryPaymentDoesNotExist.limit(1);
queryPaymentDoesNotExist.doesNotExist(keys.payment);
queryPaymentDoesNotExist.equalTo(keys.user, user);
var compoundQuery = Parse.Query.or(queryPaymentExists, queryPaymentDoesNotExist);
compoundQuery.include(keys.user);
compoundQuery.include(keys.payment);
compoundQuery.include(keys.payment + "." + keys.round);
I've checked logs from Mongo and they looks following
verbose: REQUEST for [GET] /classes/Player: {
"include": "user,payment,payment.round",
"where": {
"$or": [
{
"payment": {
"$exists": true
},
"payment.round": {
"$exists": false
},
"user": {
"__type": "Pointer",
"className": "_User",
"objectId": "ASPKs6UVwb"
}
},
{
"payment": {
"$exists": false
},
"user": {
"__type": "Pointer",
"className": "_User",
"objectId": "ASPKs6UVwb"
}
}
]
}
}
Here is response:
verbose: RESPONSE from [GET] /classes/Player: {
"response": {
"results": [
{
"objectId": "VHU9uwmLA7",
"last_online_date": {
"__type": "Date",
"iso": "2017-10-28T15:15:23.547Z"
},
"user": {
"objectId": "ASPKs6UVwb",
"username": "cn92Ekv5WPJcuHjkmTajmZMDW",
},
"createdAt": "2017-10-22T11:43:16.804Z",
"updatedAt": "2017-10-25T09:23:20.035Z",
"ACL": {
"*": {
"read": true
},
"ASPKs6UVwb": {
"read": true,
"write": true
}
},
"__type": "Object",
"className": "_User"
},
"createdAt": "2017-10-27T21:03:35.442Z",
"updatedAt": "2017-10-28T15:15:23.556Z",
"payment": {
"objectId": "nr7ln7U3eJ",
"payment_date": {
"__type": "Date",
"iso": "2017-10-27T23:42:50.614Z"
},
"user": {
"__type": "Pointer",
"className": "_User",
"objectId": "ASPKs6UVwb"
},
"createdAt": "2017-10-27T23:42:50.624Z",
"updatedAt": "2017-10-28T15:12:30.131Z",
"valid": "valid",
"round": {
"objectId": "jF9gqG4ndh",
"round_date": {
"__type": "Date",
"iso": "2017-10-28T15:12:00.027Z"
},
"createdAt": "2017-10-28T15:11:00.036Z",
"updatedAt": "2017-10-28T15:12:30.108Z",
,
"ACL": {
"*": {
"read": true
}
},
"__type": "Object",
"className": "Round"
},
"ACL": {
"ASPKs6UVwb": {
"read": true
}
},
"__type": "Object",
"className": "Payment"
},
"ACL": {
"ASPKs6UVwb": {
"read": true
}
}
}
]
}
}
You can see that response contains payment.round.
My question is following:
Can ParseCloud/MongoDB filter by Pointer<class>.filed ? By Pointer<class>.Pointer<class> ? By existence of data in that filed?
How can I workaround in situation when I need to check field presence if User can have may Players, User can have many Payments.
UPD
As far as I found mongo should support filtering by "dot notation"
mongodb query by sub-field
So what am I doing wrong?

Short answer:
No
Simplify your data structure
Long answer:
Dot notation can be used to
include documents of pointers, as you already did in your code, e.g. include(keys.user)
filter for properties of fields, e.g. {properyA: 1, propertyB: 2}. All the data is in the field, not in another document in another collection that is referenced by a Parse pointer.
Dot notation cannot be used as filter parameter for referenced pointers in a Parse query. MongoDB also does not support such a filtering, the concept of pointer is one by Parse and not by MongoDB. In a NoSQL environment like MongoDB there are no relations between tables to be used in the query language, as it is not a "relational database" like an SQL database. However Parse provides some comfort of an SQL for simple queries with its concepts of pointer, compoundQuery and matchesKeyInQuery.
If that is not sufficient in your case, simply add the fields to the collection. To the expense that you may have the same fields and data in multiple collections but with the advantage of faster query execution time.
Finding the right data structure is one of the big topics for NoSQL as there is no general right structure. The collections and document structures are basically designed as a trade off between:
execution performance
query necessity / frequency
security (access level)
and data storage size
And they are liquid and can change over time. As your app and its queries mutate you'd also change the data structure if the long term gain is greater than the one time effort.

Related

Cursor Based Pagination Naming Convention in GraphQL

In the GraphQL API, I often see naming conventions such as NQ and MQ as parameters used in cursor. This is an example, shown below,
"data": {
"items": {
"totalCount": 351,
"pageInfo": {
"hasNextPage": true,
"hasPreviousPage": false,
"endCursor": "Mw",
"startCursor": "MQ"
},
"edges": [
{
"cursor": "MQ",
"node": {
"id": "UGxhY2UtMzUy",
"displayName": "Redbeard"
}
},
{
"cursor": "Mg",
"node": {
"id": "UGxhY2UtMzUx",
"displayName": "Frey of Riverrun"
}
},
{
"cursor": "Mw",
"node": {
"id": "QmlsbGVyLTI=",
"displayName": "Something Else"
}
}
]
}
}
}
Source:
https://dev.to/tymate/first-dive-into-graphql-ruby-nak
Other examples include this rails example: https://www.2n.pl/blog/graphql-pagination-in-rails
What are these naming conventions and how would you for example paginate?
The Relay Server Specification defines how pagination should be done in order to be compatible with the Relay GraphQL Client. While it is not the only way how pagination can be done, it has evolved as a standard - at least in examples, since it can be easily referenced.
The section on connections gives more info about how cursors work:
Each edge gets a cursor value. This value is - what they call - an opaque value, meaning it should not be interpreted by the server. It is a reference/a pointer that only the server can interpret. So, if you have a query that gets a bunch of values:
edges: [
{ cursor: "abc", node: {...} },
{ cursor: "def", node: {...} },
{ cursor: "ghi", node: {...} },
{ cursor: "jkl", node: {...} },
{ cursor: "mno", node: {...} }
]
You can request the next page by looking at the cursor of the last element mno and pass it into the query.
query {
manyQuery(first: 5, after: "mno") {
edges {
cursor
node {...}
}
}
}
This will give you the next 5 nodes. See also this section on graphql.org.
So to answer your question: The string can potentially contain anything that the server can use to reference one of your nodes. E.g. an id in the database. To remove the temptation to pass in an arbitrary value from the API user this string is often encoded into the base64 format. The value should be meaningless to the client and only be used to be passed around back to the server.

Parse server - get related classes - rest api

Assuming i have a class User and a class Profile
The profile class has a field called "sex" and a field called "user" which is a pointer to user class.
If i get the profile endpoint with : https://myapi.back4app.io/classes/Profile i can get the Profile object:
{
"results": [
{
"objectId": "sIE6lOZP7R",
"user": {
"__type": "Pointer",
"className": "_User",
"objectId": "asP3EFYSR4"
},
"sex": "male",
"createdAt": "2020-05-25T17:15:49.324Z",
"updatedAt": "2020-05-25T17:15:49.324Z"
}
]
}
and if i want to include the user of this profile, i can include with: https://myapi.back4app.io/classes/Perfil?include=user so i get:
{
"results": [
{
"objectId": "sIE6lOZP7R",
"user": {
"objectId": "asP3EFYSR4",
"username": "fabiojansen",
"createdAt": "2020-05-25T17:15:16.273Z",
"updatedAt": "2020-05-25T17:15:16.273Z",
"ACL": {
"*": {
"read": true
},
"asP3EFYSR4": {
"read": true,
"write": true
}
},
"__type": "Object",
"className": "_User"
},
"sex": "male",
"createdAt": "2020-05-25T17:15:49.324Z",
"updatedAt": "2020-05-25T17:15:49.324Z"
}
]
}
Its ok, but if i want to get all the users, with the profile information in one query? Its possible? In my User class, i dont have any pointer to Profile class, only in profile class.
Is there any way?
Thanks
You have several options:
1) You can use an aggregate pipeline and $lookup the user in the Perfil class which performs a LEFT JOIN. However, this will not return an array of Parse.Object, you'd have to parse the results manually. From the docs:
{
$lookup:
{
from: <collection to join>,
localField: <field from the input documents>,
foreignField: <field from the documents of the "from" collection>,
as: <output array field>
}
}
2) You can do 2 requests by first getting all the users and then getting all their profiles by user IDs.
3) You can change your data model and add a pointer to Perfil in your User class. If you are running this query at scale it may be beneficial.

DynamoDB DocumentClient returns Set of strings (SS) attribute as an object

I'm new to DynamoDB.
When I read data from the table with AWS.DynamoDB.DocumentClient class, the query works but I get the result in the wrong format.
Query:
{
TableName: "users",
ExpressionAttributeValues: {
":param": event.pathParameters.cityId,
":date": moment().tz("Europe/London").format()
},
FilterExpression: ":date <= endDate",
KeyConditionExpression: "cityId = :param"
}
Expected:
{
"user": "boris",
"phones": ["+23xxxxx999", "+23xxxxx777"]
}
Actual:
{
"user": "boris",
"phones": {
"type": "String",
"values": ["+23xxxxx999", "+23xxxxx777"],
"wrapperName": "Set"
}
}
Thanks!
The [unmarshall] function from the [AWS.DynamoDB.Converter] is one solution if your data comes as e.g:
{
"Attributes": {
"last_names": {
"S": "UPDATED last name"
},
"names": {
"S": "I am the name"
},
"vehicles": {
"NS": [
"877",
"9801",
"104"
]
},
"updatedAt": {
"S": "2018-10-19T01:55:15.240Z"
},
"createdAt": {
"S": "2018-10-17T11:49:34.822Z"
}
}
}
Please notice the object/map {} spec per attribute, holding the attr type.
Means you are using the [dynamodb]class and not the [DynamoDB.DocumentClient].
The [unmarshall] will Convert a DynamoDB record into a JavaScript object.
Stated and backed by AWS. Ref. https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/DynamoDB/Converter.html#unmarshall-property
Nonetheless, I faced the exact same use case, as yours. Having one only attribute, TYPE SET (NS) in my case, and I had to manually do it. Next a snippet:
// Please notice the <setName>, which represents your set attribute name
ddbTransHandler.update(params).promise().then((value) =>{
value.Attributes[<setName>] = value.Attributes[<setName>].values;
return value; // or value.Attributes
});
Cheers,
Hamlet

How to filter GraphQL results by a descendent without edges?

I just started looking at GraphQL and I am wondering if there is a way to filter results that don't have any nodes. Here is a relatively simple example query:
query {
organization(login:"GitHub") {
repositories(first: 20) {
edges {
node {
name
pullRequests(first: 5, states: OPEN){
edges {
node {
title
author{
login
}
updatedAt
}
}
}
}
}
}
}
}
and here is a subset of the results that query returns:
{
"data": {
"organization": {
"repositories": {
"edges": [
{
"node": {
"name": "gitignore",
"pullRequests": {
"edges": [
{
"node": {
"title": "Create new CodeComposerStudio.gitignore",
"author": {
"login": "wolf99"
},
"updatedAt": "2017-07-26T20:31:53Z"
}
},
{
"node": {
"title": "Create PVS.gitignore",
"author": {
"login": "cesaramh"
},
"updatedAt": "2017-05-01T19:42:07Z"
}
},
{
"node": {
"title": "gitignore for Magic Software Enterprises product xpa ",
"author": {
"login": "tommes"
},
"updatedAt": "2017-05-01T19:41:53Z"
}
},
{
"node": {
"title": "Create PSoC.gitignore",
"author": {
"login": "dbrwn"
},
"updatedAt": "2017-05-01T19:41:39Z"
}
},
{
"node": {
"title": "add ThinkPHP gitignore file",
"author": {
"login": "swumao"
},
"updatedAt": "2017-05-01T19:40:53Z"
}
}
]
}
}
},
{
"node": {
"name": "dmca",
"pullRequests": {
"edges": []
}
}
}
]
}
}
}
}
So I'd like to know if there is a way to modify my query so that it would not return the node named dmca since there are no edges on pullRequests.
If you are using githubs graphql api than it seems that there is no way to filter those edges,
But if you're implementing the graphql server then it's possible to know what the edges nodes are and thus filter it in the edge resolver
According to GitHub repositories documentation does not allow that kind of filtering.
first: Int
Returns the first n elements from the list.
after: String
Returns the elements in the list that come after the specified cursor.
last: Int
Returns the last n elements from the list.
before: String
Returns the elements in the list that come before the specified cursor.
privacy: RepositoryPrivacy
If non-null, filters repositories according to privacy
orderBy: RepositoryOrder
Ordering options for repositories returned from the connection
affiliations: [RepositoryAffiliation]
Affiliation options for repositories returned from the connection
isLocked: Boolean
If non-null, filters repositories according to whether they have been locked
isFork: Boolean
If non-null, filters repositories according to whether they are forks of another repository
So I don't think that can be done.

mongo slow query

My mongodb is currently loaded with 105,000 documents, and I still have to insert 500,000 more, and it is taking more than 4hours just to insert 1000 documents, due to querying for references:
Insert DocA, and DocA have many citations (about 30)
Find documents in the database which are cited by DocA. [ie: findBy-Doi-Or-Pmid-Or-Pmc(...)]
-so for each of the query for DocA's citation, it is taking about 400ms to complete.
following is one of the profile:
Query { $or [ {$or [ {doi: ""}, {pmid: "10508155"} ] }, {pmc: "" } ]}
{
"ts": ISODate("2012-12-22T11: 55: 39.796Z"),
"op": "query",
"ns": "fyparticles.mArticle",
"query": {
"$or": {
"0": {
"$or": {
"0": {
"doi": ""
},
"1": {
"pmid": "10508155"
}
}
},
"1": {
"pmc": ""
}
}
},
"ntoreturn": NumberInt(1),
"nscanned": NumberInt(105707),
"responseLength": NumberInt(20),
"millis": NumberInt(477),
"client": "192.168.0.15",
"user": ""
}
And the index I have created:
{
"v": NumberInt(1),
"key": {
"doi": NumberInt(1),
"pmid": NumberInt(1),
"pmc": NumberInt(1)
},
"ns": "fyparticles.system.indexes",
"background": NumberInt(1),
"name": "params"
}
Please help me out here! Am I missing something or doing something wrong?
First off you are using an $or which in itself is not the fastest operator in the world due to its need to run multiple queries and then merge duplicates to return a result.
Second you are using an $or with one index. Since an $or is basically one or more queries you may need one or more indexes to cover the unique fields you have in each clause.
Third you are using nested $ors it is good to note that nested $ors do not use indexes: https://jira.mongodb.org/browse/SERVER-3327
So already you have like 3 or more performance problems with your query.
first off, take out that nested $or:
{ $or: [ {doi: ""}, {pmid: "10508155"}, {pmc: ""} ] }
And then you will probably need to create three indexes on this (you might be able to get one to fit all I haven't tested):
db.col.ensureIndex({ doi: 1 });
db.col.ensureIndex({ pmdi: 1 });
db.col.ensureIndex({ pmc: 1 });
That should be the first place to start to make your query faster.

Resources