I'm building an API which handles purely the storage of data.
Let's imagine inside Redis I've remembered the key foo:123 for 20 minutes, which holds an Eloquent Collection since I'm using the collection later on rather than returning the raw json.
As example the foo collection could look like
[
{
"name":"Doe",
"first_name":"John",
"age":42,
"favorite_color":"red"
},
{
"name":"Example",
"first_name":"Eric",
"age":37,
"favorite_color":"black"
},
....
]
How would I store a new collection, which has the same structure but entries having black as favorite_color? Would I have to store something like foo:123:black? Do I store the full collection and filter it down manually? Or is this done completely different when using Redis?
Q: How would I store a new collection, which has the same structure but entries having black as favorite_color?
Why not just map over the collection and update the favorite_color to black? https://laravel.com/docs/5.4/collections#method-map. Then store it however you want, either overwrite the old data or create new data.
Q: Would I have to store something like foo:123:black?
It's not clear to me what you're asking for here. I'm not sure we can give you any answers on how you choose to store it or the naming convention.
Q: Do I store the full collection and filter it down manually?
This is something you would need to decide based on your app requirements. It should be possible to serialize a collection and store it, but it may be a huge waste of resources if you're doing this with a bunch of different collections that have minor differences.
Related
I have a Core Data entity that is used to store earthquake reports. I would like to store a bit of global data to keep track of when the data was last updated.
The date should be a single date common to all entity objects in the store.
Can I use the Entity userInfo for this? I can't determine if this is read-only or if it can be altered at runtime.
Is there a better way? I don't want to store it in UserDefaults because if the sql files are deleted I want the "date updated" data to go away as well.
Persistent stores can have metadata about the store in addition to the actual data. That would be a good place for this. Metadata is part of the NSPersistentStore, so you have to reach down into the NSPersistentContainer to get it and change it. Something like this
guard let persistentStore = persistentContainer.persistentStoreCoordinator.persistentStores.first else {
return
}
var metadata = persistentContainer.persistentStoreCoordinator.metadata(for: persistentStore)
metadata["updated"] = Date()
persistentContainer.persistentStoreCoordinator.setMetadata(metadata, for: persistentStore)
self.saveContext()
The last line is there because metadata doesn't get saved until you save changes, even though it's metadata instead of actual data.
It's best to read the existing metadata first because Core Data puts some of its own metadata in there. You probably wouldn't be able to corrupt that, but make sure by reading it first and leaving the existing values alone.
I'm trying to export a large number of records from my database, but I need relationship data in order to build the export correctly. Ideally I would be able to use cursor() to get a Lazy Collection, but that won't load the relationships. I can't load the relationship within a loop, because that will create N+1 queries, and this could be hundreds of thousands of additional queries, which is unacceptable.
Here's what "works" (but runs out of memory):
Record::with('projects')->get()->map(function ($record) {
dd($record); // Shows the `projects` relationship
});
But when I use cursor()...
Record::with('projects')->cursor()->map(function ($record) {
dd($record); // Does NOT show the `projects` relationship
});
Is there a way to get a lazy collection that includes a record's relationship? I have looked in the documentation and it's not clear. Other suggestions have been to use chunk() which is unfortunately not a possibility in this situation.
EDIT: I shouldn't say chunk isn't a possibility, but it's a very expensive re-write. Currently, the data is structured with a lot of variability. So in order to construct the CSV for export, I need (for example) a header for the file. I currently grab that header by looping through all the records (the fields are stored in a JSONB field) and building out an array based on the fields present on those records.
I am also normalizing the data against those headers. So if one record has the field "address-1" but another record doesn't have that, the one that doesn't have it instead shows a blank value in the appropriate column. Otherwise, when inserting the row into the CSV, it doesn't respect the header.
These operations currently grab the entire data set and use a LazyCollection to map the header and normalize the records, and then feed it into the CSV one at a time. It would be ideal if I could grab relationships in a LazyCollection as well rather than having to rewrite the workflow.
according to this doc
cursor work in db stage, while loading relations come after method 'get' or 'first' ...
so: the code in cursor will work in db row represented as Model instance before the overall result, means that this code will run into db, without loading the relation, again db row (iterate through your database records...)
if you can't use chunk... then i think that you can use mySql to manage your data using raw-expressions
In our application we will give titles to users based on their points. So, if a user has 10-99 points, that user might get the "Novice" title, but a user with 100-199 points might get the "Regular User" title. I plan on eager loading a user's points using an attribute and relationship, and once I have those points I will use an attribute method to assign the title.
But how do I get the list of possible titles?
I could make a model, a migration, and a seed file, but I feel like these titles won't change much and certainly would never need to be updated in an API call. I could also hardcode an array of points and titles and do a quick lookup to see which title belongs to a user, but then I need to somehow deliver those titles to the user in an Attribute method. Or I could store them in a repository or the cache.
Can I access a repository from within a model? Is it better to store this sort of data in a DB anyways, regardless of how often it's updated or queried?
You could use entries in your .env file to store the entries and then use some logic in your php to select the correct .env entry.
LEVEL_1_TITLE=Novice
LEVEL_2_TITLE=Regular
...
if($user->points < 99){
$title = env('LEVEL_1_TITLE');
}
...
Or do the same thing from an array in a class that you create and just select the correct array entry based on the points.
I am trying to wrap my head around Bleve and I understand everything that is going on in the tutorials, videos and documentation. I however get very confused when I am using it on BoltDB and don't know how to start.
Say I have an existing BoltDB database called data.db populated with values of struct type Person
type Person struct {
ID int `json:"id"`
Name string `json:"name"`
Age int `json:"age"`
Sex string `json:"sex"`
}
How do I index this data so that I can do a search? How do I handle the indexing of data that will be stored in the database in the future?
Any help will be highly appreciated.
Bleve uses BoltDB as one of several backend stores and is separate from where you store your application data. To index your data in Bleve, simply add your Index:
index.Index(person.ID, person)
That index exists separately from your application data (whether it's in Bolt, Postgres, etc).
To retrieve your data, you'll need to construct a search request using bleve.NewSearchRequest(), then call Index.Search(). This will return a SearchResult which includes a Hits field where you can retrieve the ID for your object. You can use this to look up the object in your application data store.
Disclaimer: I am the author of BoltDB.
How you index your data depends on how you want to query for it.
If you want to query by any arbitrary fields, like {Age:15, Name:"Bob"} then BoltDB isn't an awesome fit for your problem.
BoltDB is simply a key value store with fast access to sequential keys and efficient prefix seeking. It's not really a replacement for general use databases.
You likely want something more like a document store (ie: MongoDB) or RDBMS (ie: PostgreSQL).
If you just wanted something that uses simple files and is embedded, you could also use SQlite with the Go module
If you want to search by only a single field, like ID or Name, then use that as the key.
If lookup speed doesn't matter at all, I guess you can use Bolt to just iterate over the entire db, parse the json and check the fields. But that's probably the worst approach you could take.
I can do bit of coding in ruby. I just touched objects and I am not so object literate, I mean I do not think in objects yet :-)
I have data that I scrape from the forum on regular basis. I need fields like
author, date posted, title, category, number of views, etc etc = array in my point of view.
Then I want to be able to these in ruby
save the whole lot (quick solution is csv or xml - later probably some sql database)
sort it by field
load/read my file to update fields and do some statistics, extract some data
add new fields easily in case I need to
edit, modify my "file/database" outside ruby.
I believe that I can do every operation like change the number of views of post, change the date of the last reply in the post etc etc either using array or object.
so my Question is: would you use
...................................... custom class/object or array?
could you tell why?
It would seem logical to me, at least, to make an object for storing and working with the data that you're scraping. Typically, you'd have instance variables for each of the fields that you have mentioned (author, title, category, views, date_posted) and probably some methods to populate them from the scraped data as well as read/write them.
In terms of storing the data for these objects, using an ORM such as ActiveRecord or DataMapper makes this very easy. An ORM let's you map the data in a data store, such as MySQL, to the corresponding Ruby objects. It will also provide a bunch of convenience methods for saving, updating and querying those objects.
However, it might be a good learning experience to try writing your own methods to map the data to XML files.
Do you mean "would you use an array or a custom-made class" do process this data.
What I would probably do is create a class that stores the data you want internally as an array or hash. You would then have methods of that class you could call to perform the tasks that you describe.
An object encapsulates data with behaviour i.e. functions or operations that can be performed on data. However, array is just a data structure that has a collection of element. Basically data structures expose data and have no meaningful functions.
Since you want to perform save, sort, update, stat, etc operations on your collected data so it makes sense to have a Post object with data/attributes (like author, date posted, title, category, etc.) and the operations/methods you would like to perform on your data. Abstracting the data and behaviour of your object into a class will make your code easy to maintain and understand where you can easily see the responsibility of the class by the methods defined in that class and how those methods change the state of your object by manipulating the object attributes/data.