I am working on a search filter atm where people can specify things like (example) size, color, fabric and so on.
Obviously I have models for each i.e. size, color and fabric. But since this "search" should return every result i.e. size + color + fabric and not just one of the three I would need to make a new struct which contains all the (size, color and fabric) to be able to consume the result that Gorm would return.
Since we have a LOT of filters, this could get very messy. Does anyone know if there is a better way to do it? or what would be a best practice to do this?
type Result struct {
ID int
Name string
Age int
}
var result Result
db.Raw("SELECT id, name, age FROM users WHERE name = ?", 3).Scan(&result)
The above example illustrates how I think it should be done, but as you can expect with the amount of data I need to return this struct would become huge.
The results are quite large in terms of data. I mean what I would want in return in the final version for example (this is about sales):
What products were bought, the amount, color and size. Payment data i.e. price, tax, payment method. Customer information, et cetera.
So in total there is a lot of information to store.
All this information should be returned as a JSON format so we can call it through an API call.
Each call should give back 100 up to 15.000 results, to give you an idea of the side of the data.
Hope someone can explain a bit about a best practice method and -or how I should solve this problem as I am unsure on how to code this effectively.
Related
My situation is this: I have multiple components in my view that ultimately depend on the same data, but in some cases the view state is derived from the data. How do I make sure my whole view stays in sync when the underlying data changes? I'll illustrate with an example using everyone's favorite Star Wars API.
First, I show a list of all the films, with a query like this:
# ALL_FILMS
query {
allFilms {
id
title
releaseDate
}
}
Next, I want a separate component in the UI to highlight the most recent film. There's no query for that, so I'll implement it with a client-side resolver. The query would be:
# MOST_RECENT_FILM
query {
mostRecentFilm #client {
id
title
}
}
And the resolver:
function mostRecentFilmResolver(parent, variables, context) {
return context.client.query({ query: ALL_FILMS }).then(result => {
// Omitting the implementation here since it's not relevant
return deriveMostRecentFilm(result.data);
})
}
Now, where it gets interesting is when SWAPI gets around to adding The Last Jedi and The Rise of Skywalker to its film list. We can suppose I'm polling on the list so that it gets periodically refetched. That's great, now my list UI is up to date. But my "most recent film" UI isn't aware that anything has changed — it's still stuck in 2015 showing The Force Awakens, even though the user can clearly see there are newer films.
Maybe I'm spoiled; I come from the world of MobX where stuff like this Just Works™. But this doesn't feel like an uncommon problem. Is there a best practice in the realm of Apollo/GraphQL for keeping things in sync? Am I approaching this problem in entirely the wrong way?
A few ideas I've had:
My "most recent film" query could also poll periodically. But you don't want to poll too often; after all, Star Wars films only come out every other year or so. (Thanks, Disney!) And depending on how the polling intervals overlap there will still be a big window where things are out of sync.
Instead putting the deriveMostRecentFilm logic in a resolver, just put it in the component and share the ALL_FILMS query between components. That would work, but that's basically answering "How do I get this to work in Apollo?" with "Don't use Apollo."
Some complicated system of keeping track of the dependencies between queries and chaining refreshes based on that. (I'm not keen to invent this if I can avoid it!)
In Apollo observables are (in components) over queried values (cached data 'slots') but your mostRecentFilm is not an observable, is not based on cached values (they are cached) but on one time fired query result (updated on demand).
You're only missing an 'updating connection', f.e. like this:
# ALL_FILMS
query {
allFilms {
id
title
releaseDate
isMostRecentFilm #client
}
}
Use isMostRecentFilm local resolver to update mostRecentFilm value in cache.
Any query (useQuery) related to mostRecentFilm #client will be updated automatically. All without additional queries, polling etc. - Just Works? (not tested, it should work) ;)
This is the page I am trying to make it dynamic by enabling cross-filtering.
So the thing is they are having multiple API.
For the top first two: TOTAL CASES & DAILY CASES
They are using this API and the third one in the top is based on this API.
The bottom three AGE, GENDER, and NATIONALITY are from this API.
In all the API one thing is common that is a date but there are some API in which some data are missing for few dates like there is a gap( Not available for some of the dates).
So I thought of combining all the JSON API in terms of dates and then allow cross filter because I believe I can enable cross-filtering between them. Correct me If I am wrong.
Like If I click on gender female since it gives info about total cases where the patient was female so only confirmed cases from the Total cases will change not the recovered, deaths as data is not available. SO I guess I should combine the top 3 charts together and gender, age and nationality charts, together. Then Dc js would be able to handle nicely filtering between each segments (cases related to landmark, cases related to person info).
Line 123:
var log = console.log;
var q = queue()
.defer(d3.json, "https://api.covid19india.org/data.json")
.defer(d3.json, "https://api.rootnet.in/covid19-in/unofficial/covid19india.org/statewise/history");
q.await(function(error, data1, data2) {
log("==========>");
log("data1:", error,data1);
log("data2:", data2);
});
This is not working because I can't see console.log() output.
https://blockbuilder.org/ninjakx/8c48ab6481311aa0452046d66c4d8701
So my questions are:
1) Why d3.queue is not working?
2) Suggestion whether combining all the datas together and allowing a filltering is a good idea or not as there is limited data. Should I go for cross filtering between the same api charts. So in this case I will have 2 segments (cases related to landmark, cases related to person info)..
Using DC js I want to make it more interactive and display more info.
d3.queue is obsolete
The answer to your first question is cut-and-dried: you don't need d3.queue, and it was deprecated and removed in d3#5.
As of d3#5, D3's data loading APIs use ES6 Promises instead of asynchronous callbacks, so you can use Promise.all([...]) instead of d3.queue. Apparently no way to make the new API emit errors when called in the old way, so it just fails silently. :-/
The new way to write your code is
Promise.all([
d3.json("https://api.covid19india.org/data.json"),
d3.json("https://api.rootnet.in/covid19-in/unofficial/covid19india.org/statewise/history")
]).then(([data1,data2]) => {
log("==========>");
log("data1:", data1);
log("data2:", data2);
})
.catch(error => log('error', error))
I find this much easier to read and understand. A nice side effect is that if you neglect to do error handling (like most people), you'll automatically get a clear message in the log.
Working fork of your block.
Combining multiple data sets
Your second question is pretty open-ended, maybe it would be better to bring that to the dc.js users group?
In general, it's difficult to cross-filter more than one data set. You would have more than one chart group that redraws together, and you'd have to manually add handlers on some chart to initiate, clear filters, and redraw the other chart group.
I haven't seen too many dashboards that do this. You'd have to make it clear to users what is going on.
I am working on a service that provides information about a few related entities, somewhat like a database. Suppose that there's calls to retrieve information about a school:
service MySchool {
rpc GetClassRoom (ClassRoomRequest) returns (ClassRoom);
rpc GetStudent (StudentRequest) returns (Student);
}
Now, suppose that I want to find out a class room's information, I'd receive a proto that looks like so:
message ClassRoom {
string id = 1;
string address = 2;
string teacher = 3;
}
Sometimes I also want to know all of the students of the classroom. I am struggling to think which is the better design pattern.
Option A) Add an extra rpc like so: rpc GetClassRoomStudents (ClassRoomRequest) returns (ClassRoomStudents), where ClassRoomStudents has a single field repeated Student students. This technique requires more than one call to get all the information that we want (and many if we wanted to know information for more than one classroom).
Option B) Add an extra repeated Student students field to the ClassRoom proto, and B') Fill it up only when necessary, or B") Fill it up whenever the server receives a GetClassRoom call. This may sometimes fetch extra information, or lead to ambiguity according to what fields are filled up.
I am not sure what's the best / most conventional way of dealing with this. How have some of you dealt with this?
There is no simple answer. It's a tradeoff between simplicity (option A) and performance (option B), and it depends on the situation which solution is best.
In general, I'd recommend to go with the simple solution first, unless your measurements demonstrate that it leads to performance issues. At that point, it's easy to add repeated Student students to ClassRoom and a field bool fetch_students [default=false] to ClassRoomRequest. Then clients are free to continue using the simple API, or choose to upgrade to the more performant API if they need to.
Note that this isn't specific to gRPC; the same issue is seen in REST APIs, and basically almost any request/response model.
I would like to add a calculated attribute (property) to Products. It's value is to be calculated using a PHP function eg:
function CalculateCustomAttribute() {
...
//Do some calculations based on other Product attributes, date, etc
...
return $calculatedValue; // type float
}
This calculated attribute needs to be:
displayed in the Product page,
filterable through the "Layered Navigation", and
sortable in the "Product Listing".
Could this be done? And how?
What you want to do might be possible, but I am not sure that the approach you have described would be doable, I think it is too simplistic to work with the very complex Magento platform.
I had a similar project where the actual price of the products was constantly changing based on a few inputs and I was able to solve the problem fairly well, but it was definitely more complicated thank what you seem to be hoping for. I am not sure this scenario is helpful to you or not, but here it goes...
The basic idea was that I created new product attributes (eav attributes). These served as the inputs to determine what the price really should be. Note that in my case, these attributes were being updated fairly regularly by an outside process.
Then I created an observer on the "catalog_product_save_before" event that would simply do something like this:
//some calculations to get the $newPrice
$product->setPrice($newPrice);
So basically that will make it so that the price field will always be current whenever you save a product in the administrative screens.
Then also, since several of the attributes that were used as inputs were constantly changing (updated by an outside process), so we also had to add a magento cron job to run every so often, and it would recalculate the price for all the affected products with something like this...
//some calculations to get the $newPrice
$product->addAttributeUpdate("price", $newPrice, Mage::app()->getStore()->getStoreId());
So it all boils down to the fact that you should have the attribute saved in the db. And of course you need to find the specific spots of where to update that derived attribute. Maybe your requirements will vary slightly from what I have described, but it might get you on the right path at least.
Something I'm curious about.. What would be "most efficient" to cache the generation of, say, an RSS feed? Or an API response (like the response to /api/films/info/a12345).
For example, should I cache the entire feed, and try and return that, as psuedo code:
id = GET_PARAMS['id']
cached = memcache.get("feed_%s" % id)
if cached is not None:
return cached
else:
feed = generate_feed(id)
memcache.put("feed_%s" % id, feed)
return feed
Or cache the queries result, and generate the document each time?
id = sanitise(GET_PARMS['id'])
query = query("SELECT title, body FROM posts WHERE id=%%", id)
cached_query_result = memcache.get(query.hash())
if cached_query_result:
feed = generate_feed(cached_query_result)
return feed
else:
query_result = query.execute()
memcache.put("feed_%s" % id, query_result)
feed = generate_feed(query_result)
(Or, some other way I'm missing?)
As for my experience, You should use multiple levels of cache. Implement both of Your solutions (provided that it's not the only code that uses "SELECT title, body FROM posts WHERE id=%%". If it is use only the first one).
In the second version of code, You memcache.get(query.hash()), but memcache.put("feed_%s" % id, query_result). This might not work as You want it to (unless You have an unusual version of hash() ;) ).
I would avoid query.hash(). It's better to use something like posts-title-body-%id. Try deleting a video when it's stored in cache as query.hash(). It can hang there for months as a zombie-video.
By the way:
id = GET_PARMS['id']
query = query("SELECT title, body FROM posts WHERE id=%%", id)
You take something from GET and put it right into the sql query? That's bad (will result in SQL injection attacks).
Depends on the usage pattern, but all things equal I'd vote for the first way because you'll only do the work of generating the feed 1 time.
It really depends on what your app does... The only way to answer this is to get some performance numbers from your existing app. Then you can find the code that takes the largest amount of time and work on improving that one.
As others have suggested here I'd profile your code and work out what is the slowest or most expensive part of the operation.