I need to use redis in golang as cache and store a array of structs in it. Since redis allows only storing array of strings(correct me if I'm wrong), I'll marshal the items in my array. Was wondering shall I use redis list where I'm storing marshaled structs in string format OR I can just marshal the whole array and store as key<>value in redis and not use list. One pro of using list is I can fetch ranged items from the list but scale is not the problem here since I'll be storing less than 100 items in the list. What else should I consider here.
Thankyou!!
The answer depends of how you want to use redis
For instance, store one struct using json (or any kind of serialization) and store it in a single position is easy to read / write.
But if you need to efficiently retrieve/ update one field, you can save it in a different way. However this scenario is pretty rare and complex to handle.
For instance you need to be sure you write always in the same order, to calculate the right offset. If you need to add a new field, will be really difficult to be 100% backward compatible. You probably need to create a new type (like a version 2).
Related
I'm intending to rewrite a game server I already have working in PHP-cli.
As I cannot touch the client, I have to use the on-wire protocol as-is. This is a pure binary format, with multiple fields per packet. I use a modified version of PHP's pack/unpack commands to convert to and from this. A packet is generally in the form:
header:
unpack('nitem/ncmdNum/NdataLen', $buf);
data:
any of several dozen subsequent unpack strings, as identified by cmdNum. e.g.:
a32first_name/a32second_name/a32third_name/nfirst_itemid/nfirst_flags/nfirst_level/nunused/Nfirst_perm_flags/Ncvsize/a{$cvsize}contentsVector
nsuccess/ndummyfielda/a*xerror_msg/a*xtext/NcurrentVersion/NtimeLimit/a*xlimitValue/a*xlimit
[Where a{$cvsize} means a fixed length string determined by the named (usually immediately previous) value, and a* means a zero-terminated variable-length string.. ]
The current PHP implementation unpacks this and calls the a function that deals with command 'cmdNum' passing an associative array containing the unpacked data. This in turn calls the sending code with a similar array for the return values.
Whilst I'm sure I could map these to structures, reading from the input (and writing to it) wouldn't simply be a matter of dropping the buffer over the struct. Plus, most packet types are only used once, in a function dedicated to dealing with that message, so coding up several dozen structures, and the code to deal with loading each field individually, seems like a lot of work.
Is there any method or package that I can use as the basis for dealing with this sort of thing? My searching for "php unpack in go" only seems to return results based on people unpacking a single numerical value, which is obviously easy enough to replace with encoding/binary!
The unpack/pack strings are auto-generated by some other PHP based on a specification grabbed from the client, so I could change that to create a different format fairly easily, if there is something I can use. I'd normally have no issues with writing my own functions to do this sort of thing, but being totally new to Go, this might be too much off-the-bat.
So, I have this list of attributes that I would like to represent in my Go code. That is, I'm looking for some data structure that can convey all these.
Mind like a diamond
Knows what's best
Shoes that cut
Eyes that burn like cigarettes
The right allocations
Fast
Thorough
Sharp as a tack
Playing with her jewellry
Putting up her hair
Touring the facilities
Picking up slack
Short skirt
Long jacket
Gets up early
Stays up late
Uninterrupted prosperity
Uses a machete to cut through red tape
Fingernails that shine like justice
Voice that is dark like tinted glass
Smooth liquidation
Right dividends
Wants a car with a cupholder arm rest
Wants a car that will get her there
Changing her name from Kitty to Karen
Trading her MG for a white Chrysler LeBaron
They're not going to change any time soon, but I will want to do the following things with them:
Read them from and write them to a JSON file.
Associate a boolean with each attribute (bonus points for a way to associate any one arbitrary type).
Easily display these strings to a human.
Easily count how many are "set" (That is, a function that, when called on someone who gets up early, has the right dividends, and is sharp as a tack, but nothing else, returns 3).
Should be as strongly typed as possible (It should be impossible, or at least difficult, to ask for an attribute that doesn't exist. You should be able to see all the attributes by looking at the code).
Preserving this order would be a plus, but not strictly necessary.
The approaches I've thought of so far are:
A struct with all of these as fields, but the count function is inelegant to write, and the 'nice' string name of the fields has to be stored elsewhere. Even if that can be solved with tags, the count problem remains.
A map with constant keys (or an array with enumerated values). Counting is a simple loop, and the keys can be the nice string names. However, this creates the problem of asking for keys that don't exist, and you have to hide the dictionary behind a function to stop users from trying to add new keys.
Is there something I'm missing here, or will I have to compromise?
I'd use a uint64 as a bitset (or an actual bitset type, of which none are built in but some are open source). It meets your requirements in the following ways:
JSON storage as "010101" string or base64 (11 chars).
Easy display: just define an array of the strings in the same order they are stored in the bitset.
Easily count how many are "set" - use "popcount" (open source implementations are easy to come by).
Order is naturally preserved--the first attribute will always be bit 0, etc.
If you have more than 64 attributes, you can use math/big.
You can use map[int]interface{} to store the value of any length inside the interface with integer as key or you can also use slice of interface for looping inside the data like this []interface{}. Later on you can easily type assert the interface to get underlying value which is string in your case.
Since it will be slice of interface or map of interface you can easily marshal it to convert it into json and save into file which can be read or write easily.
I'm using Go-Yaml to serialize some maps to YAML. Is there a way to ensure the serialized YAML is writen with the keys in the order they were inserted into the Go Map? Or will it be necessary to reimplement the Marshal interface myself?
Go maps do not keep track of order of insertion. In order to do this you'd have to implement your own mechanism for reading in the keys and storing the order.
I keep running into a certain kind of data structure, and wonder if there is a name for it. It maps very closely to JSON, but not exactly. The rules are:
It is composed entirely of maps, arrays, and primitives.
It is hierarchical. Maps contain name/value pairs, where a value can
be another map, an array, or a primitive. Arrays contain values with the same rules.
The top level is always a map.
The primitives are strings, integers, floats, booleans, and possibly
dates.
Sometimes the map is just an unordered hash, and sometimes the order
of the name/value pairs matter.
This is a really, really useful structure. You can use it to represent documents, database records, various messages, http requests, lots of stuff. I've run into it in Freemarker (as the 'data model'), Mongo, and anything that uses JSON.
It's not really JSON, because that's a file format, not a specification for a particular data structure. It's not an "object", because object trees can point to other things, like streams and functions. It's not a DOM.
What is it?
Around the office, we've started to call it a "garg", for "generalized argument".
It's not really JSON, because that's a file format, not a specification for a particular data structure.
It might not be JSON (since the specs include syntax rules), but your structure definition defines the same data structure as JSON does.
I don't think it's useful to name this structure. When you are talking about data, just call it data. When you need to interchange data you need a data-interchange format. Now JSON proves to be one damn good one.
JSON isn't just a file format. JSON is also a data structure.
From JSON.org
JSON is built on two structures:
A collection of name/value pairs. In various languages, this is
realized as an object, record, struct, dictionary, hash table, keyed
list, or associative array.
An ordered list of values. In most
languages, this is realized as an array, vector, list, or sequence.
These are universal data structures.
It is a generic data storage structure that carries around hierarchical data. I don't have a generic name for it, but if I were to implement such a beast in, say, C++, I'd probably call the abstract base class a Variant, and name the concrete types by their names: Integer, Array, Map, etc. I'd chuck them in a namespace that would relate to where I'd use them - or maybe I'd prefix the types themselves. I've seen such structures used as well, but I don't know if there is a name that I'd recognize. A DataStore, Environment, StorageBin, or anything that is generic and implies storage of data would do.
I don't see myself calling such a class hierarchy JSON, though. I would provide a JsonSerializer or some such to map this data to JSON, if I needed it.
It sounds like you're describing an associative array, with optional ordering.
That's what JSON represents, except that (I believe) JSON doesn't impose an ordering requirement. Naturally, many other representations also describe associative arrays, which is why JSON is a popular text serialization.
Update 1: JSON isn't properly an associative array. It is a description of object properties. Because it is very often construed as an associative array, many people make the same mistake I did. In fact, "object notation" is the proper name for it - surprise, surprise. :) In addition, JSON isn't a file format - it's a text serialization or markup language, which is different from a file format.
The structure is a tree with different kinds of values stored at its leafs.
In Boost, a similar structure is called Property Tree.
I'm trying to make an algorithm that easily simplifies and groups synonyms (with mismatches, capitals, acronims, etc) into only one. I supose there should exist a standard way to build such a structure that, looking for a string with possible mismatches, if the string exists in the structure, it returns a normalized string key. In short, sometimes the same concept could be written in several ways, but I only want to keep the concept.
For instance: Supose I want to normalize or simplify the appearances of
"General Director", "General Manager", "G, Dtor", "Gen Dir", ...
into
"GEN_DIR"
and keep only this result for further reference.
By the way, I suppose that building a Hash with key/value pairs like
hash["General Director"]="GEN_DIR"
hash["General Manager"]="GEN_DIR"
hash["G, Dtor"]="GEN_DIR"
hash["G, Dir"]="GEN_DIR"
could be a solution, but I suspect that there are more elegant or adequate solutions to that.
I would also need the way to persist this associative structure easily without any database because it should grow as I find more mismatches of the same word or sentence. A possible approach I think is to define this structure by means of a DSL, but I'm open to suggestions.
Well, there is no rule, at least a clear one.
My aim is to scrap from web some "structured" data that sometimes is incorrectly or incompletely typed. Some fields are descriptions and can be left as is. But some fields are suposedly to be "sets" but aren's correctly typed (as in my example). As a human can read that, he immediatelly knows what it means and can associate that with its meaning.
But I would like to automate as much as possible the process of reducing those possible mismatches to only one "string" (or symbol) before, for instance, saving it into a database. So, what I would need is a kindof hash or dictionary, as sawa correctly stated, that I can use to lookup any of such dirty strings to get the normalized string or symbol.
Also, of course, it would be desirable a way to make this hash (or whatelse it could be) to learn from new mismatches in some way and add a new association automatically (possibly it could be based on a distance measure between mismatched string and normalized string that, if lower than X, a new association is built). The whole association (i.e, hash) should grow as new mismatches and concepts arise and, though, it should be kept anywhere (possibly in an xml file, or something like what Mori answered below) for future uses.
Any new Idea?