My app uses AppSync resolvers to fetch data from DDB and return it to our front-end. One table we have is for Notifications. A Notification can be either pending or default (non-pending). The table itself has a primary key of notification_id and we have a GSI called userIndex to grab the notifications for a user, with a sort key of timestamp.
In the app, I show all notifications in a list, pending first and then default. Given that a user may have many notifications, I'd like to implement pagination to fetch a batch at a time. The only way I've been able to do this is to
change the query to include a isPending parameter, which I use as a filter expression for the query to only return notifications that are isPending or isNotPending.
Store two "nextTokens", one for each isPending and isNotPending, along with corresponding lists.
Make separate queries for pending/non-pending, and use the filter to return to the appropriate list.
This is obviously inefficient and I am re-reading data from DynamoDB. My question is, given my DynamoDB table/requirements, is there a way I can paginate so that I can get all the pending notifications first (sorted by timestamp) and then all the default notifications next (sorted by timestamp) by using one query and one nextToken
I've seen the use of #model and #key, but I haven't been able to make it work in my app.
Thanks!
No, not really. There is a hard limit on returns for a Dynamodb query - and that cannot be bypassed. the only way to make use of nextToken is another query.
However, it is also worth noting that the FilterExpression happens after the data has already been retrieved and is filtered client side. It does not reduce the documents pulled from the query - only whats displayed. So the next token is still going to be (relatively) the same for each query. You can instead filter it yourself after the call before the next pagination query and save yourself a little bit in terms of multiple calls.
Related
I want to work with Elasticsearch to process some Whatsapp chats. So I am initially planning the data load.
The problem is that the data exported from Whatsapp, doesn't contain a real unique id per user but it only contains the name of the user taken from the contact directory of the device where the chat is exported (ie. a user can change the number or have two numbers in the same group).
Because of that, I need to create a custom explicit mapping table between the user names and a self-generated unique id, that gets populated in an additional column.
Then, my question is: "How can I implement such kind of explicit mapping in Elasticsearch to generate an additional unique column?". Alternatively, a valid answer could be a totally different approach to the problem.
PS. As I write, I think the solution could be in the ingestion process, like in a python script, but I still want to post the question to understand if this is something that Elasticsearch can do by itself.
yes, do it during the index process
if you had the data that maps the name and the id stored in a separate index you could do this with an enrich processor when you index the data to add whichever value you want to the document via a pipeline
also - Elasticsearch doesn't have columns, only fields
Description:
I have a seeProfile resolver, which gives back a profile of a particular user.
This user has a photos array and I want to paginate these photos and I can't figure out how. I have a workaround but I'd like to know a way - shall it exist - to paginate a field of this one user.
Workaround:
I can simply make a resolver for fetching pictures for a particular user with .findMany() and paginate them with take: and skip: . But then I make two queries instead of one. So, if anyone knows a way to paginate the photos inside findUnique query, please let me know, thanks!
Unfortunately, there's no way as of now to implement pagination inside the include relation in Prisma.
You will have to make two separate queries with Prisma, as you described in your workaround:
One findUnique query to fetch the user records.
One findMany query to fetch the approrpriate photo records.
I can simply make a resolver for fetching pictures for a particular user.
Perhaps it just makes sense to do both queries in the same resolver, to avoid two requests to the API.
I want to query all instances of a model by the most recently created.
Reading the official docs, they suggest a way of querying by the default timestamps (updatedAt/createdAt) but only when also querying by another key. So I know I could query a hypothetical User model by name and createdAt, but I can't query all instances of User by createdAt.
Is there an established way of doing this?
I have tried adding a #key directive to sort by updatedAt, but that results in an error because updatedAt is automatically added and not described in my schema. If I then add the timestamps to my schema this creates problems when mutating clients because it expects the timestamps to be added by me, which I obviously don't do because it's automatically added by DynamoDB.
Thanks
You could try using a Global Secondary Index on the field you want to query. In your AppSync resolver, you need to specify the index you want to use for the query.
Another way would be to run a scan operation against your DB (you don't need to specify a key in this case), although that would be way more inefficient than a GSI.
I implementing RESTful API service and i have a question about saving related records.
For example i have users table and related user_emails table. User emails should be unique.
On client side i have a form with user data fields and a number of user_email fields (user can add any number of fields independently). When the user saves the form i must first make query to create record in users table to get her ID, and only then i can make query to save user emails (because in now i have id of record which come with response after saving user data). But if user enters not unique email in any field then the request will fail. So I create a record in the users table but not create record in user_emails table.
What are the approaches to implement validation of all this data before saving?
This is nor related restful api but transactional processing on the backend. If you are using Java, with JPA you can persist both element in the same transaction then you can notice if there is a problem and rollback the entire transaction returning a response.
I would condense it down to a single request, if you could. Just for performance's sake, if nothing else. Use the user_email as your key, and have the request return some sort of status result: if the user_email is unique, it'll respond with a success message. Otherwise, it'd indicate failure.
It's much better to implement that check solely on the server side and not both with the ID value unless you need to. It'll offer better performance to do that, and it'll let you change your implementation later more easily.
As for the actual code you use, since I'm not one hundred percent on what you're actually asking, you could use a MERGE if you're using SQL Server. That'd make it a bit easier to import the user's email and let the database worry about duplicates.
I'm trying to implement a dashboard similar to facebook in cakephp (getting posts and post them to timeline and while you press see more it keeps retrieving posts from previous offsets) , but im still confused about the logic and tools , should i use the cakephp pagination class in my implementations.
$this->paginate();
it somehow should be called through ajax accourding to some performance wise
Any helps or suggestions where to start from ?
Thanks All
Don't use paginate
If you paginate something that you are prepending data to - you're going to get data overlapping such that you ask for page 2 - and get the end of, as far as the current user is concerned, the previous page.
Use a timestamp
The normal technique for an endless stream of data is to use a query like:
SELECT *
FROM foos
WHERE created >= $previousLastTimestamp
ORDER BY created DESC
LIMIT 20
Note that while I'm using created in this example - it can be any field that is pseudo unique.
When you first render the page, store the timestamp of the last entry in a javascript variable, then your "get more posts" logic should be:
Make an ajax (get) request, passing the last timestamp
Perform the above sql query (as a $this->Foo->find call)
in your js update the last timestamp so that you know where you are up to for the next time the user clicks "get more posts"
The reason to use a >= condition is that, unless the field you are testing against has unique values, it's possible for there to be multiple rows with the value you're testing for. If you have a naturally-unique field that you are sorting by (id) then you don't need to use greater-or-equal, you can simply use greater-than, and avoid needing to think about duplicate rows.
Here's a reference which explains in more detail why you should handle systems like this avoiding traditional pagination.