I want to parse a huge json with more accuracy and less code implementation. Is there any possibility to implement.i am able to parse a huge file into the buffer but when I try to access values of json I need to do individually for all the keys and values. Will there any possibility to retrieve values of objects only when needed. Please provide my example for parsing huge json file and using values
Related
This is my first time using Ruby. I'm writing an application that parses data and performs some calculations based on it, the source of which is a JSON file. I'm aware I can use JSON.parse() here but I'm trying to write my program so that it will work with other sources of data. Is there a clear cut way of doing this? Thank you.
When your source file is JSON then use JSON.parse. Do not implement a JSON parser on your own. If the source file is a CSV, then use the CSV class.
When your application should be able to read multiple different formats then just add one Reader class for each data type, like JSONReader, CSVReader, etc. And then decide depending on the file extension which reader to use to read the file.
I am currently playing with libnftables and json-c (C libraries)
my aim is to parse nftables rules in json format with json-c library
The thing is there, nftables elements (a table for example) look like this:
{ "table":{
"family":"ip",
"name":"filter",
"handle":6
}
}
A nftables table is basically a json_object structure with the key "table" and another json_object structure as value storing all other informations. Chains and rules are similar but with different key
{"chain":{
"family":"ip",
"table":"filter",
"name":"INPUT",
"handle":1,
"type":"filter",
"hook":"input",
"prio":0,
"policy":"accept"
}
},
I need a way to retrieve the key (like "table", "rule" , "chain") from a json_object since the whole nftables rules are a mixture of many things making it difficult to guess the kind of nftables elements we are dealing with when browsing through the array of nftables rules.
Thank you!
use libnftables in json mode to retrieve ruleset in json object
get a good json parsin library (I recommend libjansson) to parse json object in order to retrieve informations
NB: you might have to compile nftables to be able to use json output format.
I'm new to protocol buffers and I was wondering whether it was possible to search a protocol buffers binary file and read the data in a structured format. For example if a message in my .proto file has 4 fields I would like to serialize the message and write multiple messages into a file and then search for a particular field in the file. If I find the field I would like to read back the message in the same structured format as it was written. Is this possible with protocol buffers ? If possible any sample code or examples would be very helpful. Thank you
You should treat protobuf library as one serialization protocol, not an all-in-one library which supports complex operations (such as querying, indexing, picking up particular data). Google has various libraries on top of open-sourced portion of protobuf to do so, but they are not released as open source, as they are tied with their unique infrastructure. That being said, what you want is certainly possible, yet you need to write some code.
Anyhow, some of your requirements are:
one file contains various serialized binaries.
search a particular field in each serialized binary and extract that chunk.
There are several ways to achieve them.
The most popular way for serial read/write is that the file contains a series of [size, type, serialization output]. That is, one serialized output is always prefixed by size and type (either 4/8 byte or variable-length) to help reading and parsing. So you just repeat this procedure: 1) read size and type, 2) read binary with given size, 3) parse with given type 4) goto 1). If you use union type or one file shares same type, you may skip type. You cannot drop size, as there is no way know the end of output by itself. If you want random read/write, other type of data structure is necessary.
'search field' in binary file is more tricky. One way is to read/parse output one by one and to check the existance of field by HasField(). It is most obvious and slow yet straightforward way to do so. If you want to search field by number (say, you want to search 'optional string email = 3;'), thus search by binary blob (like 0x1A, field number 3, wire type 2), it is not possible. In a serialized binary stream, field information is saved merely a number. Without an exact context (.proto scheme or binary file's structure), the number alone doesn't mean anything. There is no guarantee that 0x1A is from field information, or field information from other message type, or actually number 26, or part of other number, etc. That is, you need to maintain the information by yourself. You may create another file or database with necessary information to fetch particular message (like the location of serialization output with given field).
Long story short, what you ask is beyond what open-sourced protobuf library itself does, yet you can write them with your requirements.
I hope, this is what you are looking for:
http://temk.github.io/protobuf-utils/
This is a command line utility for searching within protobuf file.
I want to store large amount of data in a protobuf format in which include time-stamp parameter. And I want to retrieve the data based on the time-stamp value.
Thanks.
Protobuf is a sequential-access format. There's no way to jump into the middle of a message looking for data; you have to parse through the whole thing.
Some options:
Devise a framing format that allows you to break up your datastore into many small chunks, each of which is a separate protobuf message. This is a fairly large project.
Use SQLite or even an actual database.
Use a random-access-fieldly format like Cap'n Proto instead. (Disclosure: I'm the author of Cap'n Proto, and also of Protobufs v2 (Google's open source release).)
How would I create a list of elements in VB.NET, save it to a .dat file, and make Ruby re-create such list (as an array) with such elements (they will be strings, booleans and integers)?
You can do it, but you'd need to find some representation for it. The easiest is probably JSON, so you would
make the data structure in VB
write it to JSON as a file
read the JSON file using Ruby.
Here's a JSON serializer for .Net:
A .dat file is just a binary blob, 'tis it not? If there's any particular format you use you could easily translate that to equivalent Ruby code. Just as long as the knowledge is duplicated on both ends, though that leads to a violation of the DRY principle. JSON might be a good intermediate representation (as noted by #Charlie Martin) because it's a plain text format and you can always add compression.