I am new to web development, and I've seen many sites preaching the benefits of using protocol buffers -- for example: https://codeclimate.com/blog/choose-protocol-buffers/.
I'm not sure if some of the benefits apply to my use case:
Having a unified schema out of the .proto file: If I validate my data in the front and back-end, which I should do anyway, a unified schema is enforced explicitly. I don't see any added benefit in this regard from using protocol buffers.
Auto generating the setters and getters from the .proto file: This looks like a nice selling point. But, I wouldn't need any setters and getters if I don't use protocol buffers in the first place. I found them really cumbersome to work with:
They remove capitalization, which alters the original variable names
They are unnatural to work with. For example, in c++ I would want work with just a plain old data structure, but instead I have to do something like ptr_message->shouldBeStruct1().shouldBeStructArray(20).shouldBeInt();
Easy Language Interoperability: I really doubt it is good practice to design my data consuming code so that it works for a protobuf message rather than a struct. So, I would need to parse the protobuf into a plain data struct first.
The only potential benefit I see is the reduced data size when transmitting on the wire. But, does this really justify the overhead of additional middleware to work with protocol buffers? What am I missing?
I guess traditionally, one would for a RESTful web service use one type of DTO objects for POJO/JSON conversion, and a separate DTO object for database entity/POJO conversion?
Spring Boot should be more opinionated and easier to use, but would you still use different DTO object types for JSON and database entity representation, or do you convert entity objects directly to JSON?
Let me share my opinion.
At first, I think your question has nothing to do with spring boot. Spring boot just provides a fancy and lightweight way to start the application and allows to build the app in an easier manner.
But still you have your rest controller there and from that point it doesn't differ much for any other type of application.
So what you're actually asking is whether it makes sense to maintain an abstraction of JSON objects and converting them to the Business Logic Entity objects and later on converting them once again to Database objects or its enough to maintain only 2 levels and ditch the Json level.
I think the answer is "it depends".
First of all, In general currently the trend is a simplification. So maybe its enough to maintain only 1 level of objects.
There are a lot of advantages of such an approach:
Obviously less code to maintain
Speed of development and testing (POJOs should be checked, converters should be tested and so forth)
Speed of execution - you don't need to waste the CPU time on conversion. A kind of obvious implication.
Less obvious: Memory consumption. Lets say you work with a big bulk of data returned by your DAO. Let's say it occupies 10MB of memory (just for the sake of example). Now if you start to convert, to Business Entities, you'll spend yet another 10MB and now if its A JSon objects, well its again 10MB. The point is that all these objects may co-exist in memory simultaneously. Of course GC will probably take care of them if you implemented everything right, but this is a different story.
However there is one drawback of such a simplification.
In one word I would call it a Commitment
There are three Types of APIs in the application.
The API you're committed to at the level of Web Service - The JSon structure.
The chances are that various clients (not necessary using the JVM at all) are running against your Web service and consume the data. So they really expect you to provide a JSon objects of the given structure.
The API of your business. If your Business logic layer is pretty complicated, you probably have an entire team that develops that logic. So you usually work at the level of APIs between the teams.
The level of DAO - the same story as Business Logic actually.
So now, what happens if you, say, change that API at one level. Does it mean that all the levels will be broken?
Example
Lets say, we don't maintain "JSon" level. In this case, if we change the API at the level of Business Logic, the JSON will also change automatically. All the rest frameworks will happily convert the object for us, and the chances are that the user will get another data.
Another example
Lets say, your BL layer provides a Person entity that looks like this:
class Person {
String firstName;
String lastName;
List<Language> languages;
}
class Language {
...
}
Now, let's say you have a UI that consumes your REST service that provides a list of Persons upon request. What if there are 2 different pages in UI. One that shows only the Persons (in this case it doesn't make sense to provide a list of language, spoken by a person).
In the second page however you want to get the full information.
So, you'll end up exposing 2 web services or complicating the existing one by some parameters (the more params like this you have, the less it resembles the rest :) )
Maybe separation would help a little here? I don't know.
Bottom line.
I would say that as long as you can live without such a separation - do it. It can work even for quite big projects. And of course it can work for small or middle-sized projects.
If you find yourself struggling around fixes and you feel like such a separation would solve the issues - do the separation.
Hope this helps to understand the implications and chose what works for you
GraphQL consists of a type system, query language and execution
semantics, static validation, and type introspection, each outlined
below. To guide you through each of these components, we've written an
example designed to illustrate the various pieces of GraphQL.
- https://github.com/facebook/graphql
Falcor lets you represent all your remote data sources as a single
domain model via a virtual JSON graph. You code the same way no matter
where the data is, whether in memory on the client or over the network
on the server.
- http://netflix.github.io/falcor/
What is the difference between Falcor and GraphQL (in the context of Relay)?
I have viewed the Angular Air Episode 26: FalcorJS and Angular 2 where Jafar Husain answers how GraphQL compares to FalcorJS. This is the summary (paraphrasing):
FalcorJS and GraphQL are tackling the same problem (querying data, managing data).
The important distinction is that GraphQL is a query language and FalcorJS is not.
When you are asking FalcorJS for resources, you are very explicitly asking for finite series of values. FalcorJS does support things like ranges, e.g. genres[0..10]. But it does not support open-ended queries, e.g. genres[0..*].
GraphQL is set based: give me all records where true, order by this, etc. In this sense, GraphQL query language is more powerful than FalcorJS.
With GraphQL you have a powerful query language, but you have to interpret that query language on the server.
Jafar argues that in most applications, the types of the queries that go from client to server share the same shape. Therefore, having a specific and predictable operations like get and set exposes more opportunities to leverage cache. Furthermore, a lot of the developers are familiar with mapping the requests using a simple router in REST architecture.
The end discussion resolves around whether the power that comes with GraphQL outweighs the complexity.
I have now written apps with both libraries and I can agree with everything in Gajus' post, but found some different things most important in my own use of the frameworks.
Probably the biggest practical difference is that most of the examples and presumably work done up to this point on GraphQL has been concentrated on integrating GraphQL with Relay - Facebook's system for integrating ReactJS widgets with their data requirements. FalcorJS on the other hand tends to act separately from the widget system which means both that it may be easier to integrate into a non-React/Relay client and that it will do less for you automatically in terms of matching widget data dependencies with widgets.
The flip side of FalcorJS being flexible in client side integrations is that it can be very opinionated about how the server needs to act. FalcorJS actually does have a straight up "Call this Query over HTTP" capability - although Jafar Husain doesn't seem to talk about it very much - and once you include those, the way the client libraries react to server information is quite similar except that GraphQL/Relay adds a layer of configuration. In FalcorJS, if you are returning a value for movie, your return value better say 'movie', whereas in GraphQL, you can describe that even though the query returns 'film', you should put that in the client side datastore as 'movie'. - this is part of the power vs complexity tradeoff that Gajus mentioned.
On a practical basis, GraphQL and Relay seems to be more developed. Jafar Husain has mentioned that the next version of the Netflix frontend will be running at least in part on FalcorJS whereas the Facebook team has mentioned that they've been using some version of the GraphQL/Relay stack in production for over 3 years.
The open source developer community around GraphQL and Relay seems to be thriving. There are a large number of well-attended supporting projects around GraphQL and Relay whereas I have personally found very few around FalcorJS. Also the base github repository for Relay (https://github.com/facebook/relay/pulse) is significantly more active than the github repository for FalcorJS (https://github.com/netflix/falcor/pulse). When I first pulled the Facebook repo, the examples were broken. I opened a github issue and it was fixed within hours. On the other hand, the github issue I opened on FalcorJS has had no official response in two weeks.
Lee Byron one of the engineer behind GraphQL did an AMA on hashnode, here is his answer when asked this question:
Falcor returns Observables, GraphQL just values. For how Netflix wanted to use Falcor, this makes a lot of sense for them. They make multiple requests and present data as it's ready, but it also means that the client developer has to work with the Observables directly. GraphQL is a request/response model, and returns back JSON, which is trivially easy to then use. Relay adds back in some of the dynamicism that Falcor presents while maintaining only using plain values.
Type system. GraphQL is defined in terms of a type system, and that's allowed us to built lots of interesting tools like GraphiQL, code generators, error detection, etc. Falcor is much more dynamic, which is valuable in its own right but limits the ability to do this kind of thing.
Network usage. GraphQL was originally designed for operating Facebook's news feed on low end devices on even lower end networks, so it goes to great lengths to allow you to declare everything you need in a single network request in order to minimize latency. Falcor, on the other hand, often performs multiple round trips to collect additional data. This is really just a tradeoff between the simplicity of the system and the control of the network. For Netflix, they also deal with very low end devices (e.g. Roku stick) but the assumption is the network will be good enough to stream video.
Edit: Falcor can indeed batch requests, making the comment about the network usage inaccurate. Thanks to #PrzeoR
UPDATE: I've found the very useful comment under my post that I want to share with you as a complementary thing to the main content:
Regarding lack of examples, you can find the awesome-falcorjs repo userful, there are different examples of a Falcor's CRUD usage:
https://github.com/przeor/awesome-falcorjs ... Second thing, there is a book called "Mastering Full Stack React Development" which includes Falcor as well (good way to learn how to use it):
ORGINAL POST BELOW:
FalcorJS (https://www.facebook.com/groups/falcorjs/) is much more simpler to be efficient in comparison to Relay/GraphQL.
The learning curve for GraphQL+Relay is HUGE:
In my short summary: Go for Falcor. Use Falcor in your next project until YOU have a large budget and a lot of learning time for your team then use RELAY+GRAPHQL.
GraphQL+Relay has huge API that you must be efficient in. Falcor has small API and is very easy to grasp to any front-end developer who is familiar with JSON.
If you have an AGILE project with limited resources -> then go for FalcorJS!
MY SUBJECTIVE opinion: FalcorJS is 500%+ easier to be efficient in full-stack javascript.
I have also published some FalcorJS starter kits on my project (+more full-stack falcor's example projects): https://www.github.com/przeor
To be more in technical details:
1) When you are using Falcor, then you can use both on front-end and backend:
import falcor from 'falcor';
and then build your model based upon.
... you need also two libraries which are simple to use on backend:
a) falcor-express - you use it once (ex. app.use('/model.json', FalcorServer.dataSourceRoute(() => new NamesRouter()))). Source: https://github.com/przeor/falcor-netflix-shopping-cart-example/blob/master/server/index.js
b) falcor-router - there you define SIMPLE routes (ex. route: '_view.length'). Source:
https://github.com/przeor/falcor-netflix-shopping-cart-example/blob/master/server/router.js
Falcor is piece of cake in terms of learning curve.
You can also see documentation which is much simpler than FB's lib and check also the article "why you should care about falcorjs (netflix falcor)".
2) Relay/GraphQL is more likely like a huge enterprise tool.
For example, you have two different documentations that separately are talking about:
a) Relay: https://facebook.github.io/relay/docs/tutorial.html
- Containers
- Routes
- Root Container
- Ready State
- Mutations
- Network Layer
- Babel Relay Plugin
- GRAPHQL
GraphQL Relay Specification
Object Identification
Connection
Mutations
Further Reading
API REFERENCE
Relay
RelayContainer
Relay.Route
Relay.RootContainer
Relay.QL
Relay.Mutation
Relay.PropTypes
Relay.Store
INTERFACES
RelayNetworkLayer
RelayMutationRequest
RelayQueryRequest
b) GrapQL: https://facebook.github.io/graphql/
2Language
2.1Source Text
2.1.1Unicode
2.1.2White Space
2.1.3Line Terminators
2.1.4Comments
2.1.5Insignificant Commas
2.1.6Lexical Tokens
2.1.7Ignored Tokens
2.1.8Punctuators
2.1.9Names
2.2Query Document
2.2.1Operations
2.2.2Selection Sets
2.2.3Fields
2.2.4Arguments
2.2.5Field Alias
2.2.6Fragments
2.2.6.1Type Conditions
2.2.6.2Inline Fragments
2.2.7Input Values
2.2.7.1Int Value
2.2.7.2Float Value
2.2.7.3Boolean Value
2.2.7.4String Value
2.2.7.5Enum Value
2.2.7.6List Value
2.2.7.7Input Object Values
2.2.8Variables
2.2.8.1Variable use within Fragments
2.2.9Input Types
2.2.10Directives
2.2.10.1Fragment Directives
3Type System
3.1Types
3.1.1Scalars
3.1.1.1Built-in Scalars
3.1.1.1.1Int
3.1.1.1.2Float
3.1.1.1.3String
3.1.1.1.4Boolean
3.1.1.1.5ID
3.1.2Objects
3.1.2.1Object Field Arguments
3.1.2.2Object Field deprecation
3.1.2.3Object type validation
3.1.3Interfaces
3.1.3.1Interface type validation
3.1.4Unions
3.1.4.1Union type validation
3.1.5Enums
3.1.6Input Objects
3.1.7Lists
3.1.8Non-Null
3.2Directives
3.2.1#skip
3.2.2#include
3.3Starting types
4Introspection
4.1General Principles
4.1.1Naming conventions
4.1.2Documentation
4.1.3Deprecation
4.1.4Type Name Introspection
4.2Schema Introspection
4.2.1The "__Type" Type
4.2.2Type Kinds
4.2.2.1Scalar
4.2.2.2Object
4.2.2.3Union
4.2.2.4Interface
4.2.2.5Enum
4.2.2.6Input Object
4.2.2.7List
4.2.2.8Non-null
4.2.2.9Combining List and Non-Null
4.2.3The __Field Type
4.2.4The __InputValue Type
5Validation
5.1Operations
5.1.1Named Operation Definitions
5.1.1.1Operation Name Uniqueness
5.1.2Anonymous Operation Definitions
5.1.2.1Lone Anonymous Operation
5.2Fields
5.2.1Field Selections on Objects, Interfaces, and Unions Types
5.2.2Field Selection Merging
5.2.3Leaf Field Selections
5.3Arguments
5.3.1Argument Names
5.3.2Argument Uniqueness
5.3.3Argument Values Type Correctness
5.3.3.1Compatible Values
5.3.3.2Required Arguments
5.4Fragments
5.4.1Fragment Declarations
5.4.1.1Fragment Name Uniqueness
5.4.1.2Fragment Spread Type Existence
5.4.1.3Fragments On Composite Types
5.4.1.4Fragments Must Be Used
5.4.2Fragment Spreads
5.4.2.1Fragment spread target defined
5.4.2.2Fragment spreads must not form cycles
5.4.2.3Fragment spread is possible
5.4.2.3.1Object Spreads In Object Scope
5.4.2.3.2Abstract Spreads in Object Scope
5.4.2.3.3Object Spreads In Abstract Scope
5.4.2.3.4Abstract Spreads in Abstract Scope
5.5Values
5.5.1Input Object Field Uniqueness
5.6Directives
5.6.1Directives Are Defined
5.7Variables
5.7.1Variable Uniqueness
5.7.2Variable Default Values Are Correctly Typed
5.7.3Variables Are Input Types
5.7.4All Variable Uses Defined
5.7.5All Variables Used
5.7.6All Variable Usages are Allowed
6Execution
6.1Evaluating requests
6.2Coercing Variables
6.3Evaluating operations
6.4Evaluating selection sets
6.5Evaluating a grouped field set
6.5.1Field entries
6.5.2Normal evaluation
6.5.3Serial execution
6.5.4Error handling
6.5.5Nullability
7Response
7.1Serialization Format
7.1.1JSON Serialization
7.2Response Format
7.2.1Data
7.2.2Errors
AAppendix: Notation Conventions
A.1Context-Free Grammar
A.2Lexical and Syntactical Grammar
A.3Grammar Notation
A.4Grammar Semantics
A.5Algorithms
BAppendix: Grammar Summary
B.1Ignored Tokens
B.2Lexical Tokens
B.3Query Document
It's your choice:
Simple sweet and short documented Falcor JS VERSUS Huge-enterprise-grade tool with long and advanced documentation as GraphQL&Relay
As I said before, if you are a front-end dev who grasp idea of using JSON, then JSON graph implementation from Falcor's team is best way to do your full-stack dev project.
In short, Falcor or GraphQL or Restful solve the same problem - provide a tool to query/manipulate data effectively.
How they differ is in how they present their data:
Falcor wants you to think their data as a very big virtual JSON tree, and uses get, set and call to read, write data.
GraphQL wants you to think their data as a group of predefined typed objects, and uses queries and mutations to read, write data.
Restful wants you to think their data as a group of resources, and uses HTTP verbs to read, write data.
Whenever we need to provide data for user, we end up with something liked: client -> query -> {a layer translate query into data ops} -> data.
After struggling with GraphQL, Falcor and JSON API (and even ODdata), I wrote my own data query layer. It's simpler, easier to learn, and more equivalent with GraphQL.
Check it out at:
https://github.com/giapnguyen74/nextql
It also integrates with featherjs for real time query/mutation.
https://github.com/giapnguyen74/nextql-feathers
OK, just start from a simple but important difference, GraphQL is a query based while Falcor is not!
But how they help u?
Basically, they both helping us to manage and querying data, but GraphQL has a req/res Model and return the data as JSON, basically the idea in GraphQL is having a single request to get all your data in one goal... Also, have exact response by having an exact request, So something to run on low-speed internet and mobile devices, like 3G networks... So if you have many mobile users or for some reasons you'd like to have less requests and faster response, use GraphQL... While Faclor is not too far from this, so read on...
On the other hand, Falcor by Netflix, usually have extra request (usually more than once) to retrieve all your data, eventhough they trying to improving it to a single req... Falcor is more limited for queries and doesn't have pre-defined query helpers like range and etc...
But for more clarification, let's see how each of them introduce itself:
GraphQL, A query language for your API
GraphQL is a query language for APIs and a runtime for fulfilling
those queries with your existing data. GraphQL provides a complete and
understandable description of the data in your API, gives clients the
power to ask for exactly what they need and nothing more, makes it
easier to evolve APIs over time, and enables powerful developer tools.
Send a GraphQL query to your API and get exactly what you need,
nothing more and nothing less. GraphQL queries always return
predictable results. Apps using GraphQL are fast and stable because
they control the data they get, not the server.
GraphQL queries access not just the properties of one resource but
also smoothly follow references between them. While typical REST APIs
require loading from multiple URLs, GraphQL APIs get all the data your
app needs in a single request. Apps using GraphQL can be quick even on
slow mobile network connections.
GraphQL APIs are organized in terms of types and fields, not
endpoints. Access the full capabilities of your data from a single
endpoint. GraphQL uses types to ensure Apps only ask for what’s
possible and provide clear and helpful errors. Apps can use types to
avoid writing manual parsing code.
Falcor, a JavaScript library for efficient data fetching
Falcor lets you represent all your remote data sources as a single
domain model via a virtual JSON graph. You code the same way no matter
where the data is, whether in memory on the client or over the network
on the server.
A JavaScript-like path syntax makes it easy to access as much or as
little data as you want, when you want it. You retrieve your data
using familiar JavaScript operations like get, set, and call. If you
know your data, you know your API.
Falcor automatically traverses references in your graph and makes
requests as needed. Falcor transparently handles all network
communications, opportunistically batching and de-duping requests.
I am currently designing system which should allow access to database. Assumptions are as follows:
Database should has access layer. The access layer should provide objects that represents database tables. (This would be done using some ORM framework).
Client which want to get data from database, should get object from access layer first, and then get data using those objects.
Clients could use Python, Java or C++.
Access layer is based on Java.
There won't be to many clients, but they will be opearating on large amounts of data.
The question which is hard for me is what technology should be used for passing object between acces layer and clients. I consider using ZeroC ICE, Apache Thrift or Google Protocol Buffers.
Does anyone have opinion which one is worth using?
This is my research for Protocol Buffers:
Advantages:
simple to use and easy to start
well documented
highly optimized
defining object data structure in java-like language
automatically generating implementation of setters and getters and build methods for Python, Java and C++
open-source bidnings for other languages
object could be extended without affecting old version of an applications
there are many of open-source RpcChanel and RpcController implementation (not tested)
Disadvantages:
need to implement object transfer
objects structure have to be defined before use, so we can't add some fields on the fly (Updated: there are posibilities to do that, see the comments)
if there is a need for reading one object's filed, we have to parse whole file (in contrast, in XML we could ignore chosen tags)
if we want to use RPC for invoke object methods, we need to define services and deliver RpcChanel and RpcController implementation
This is my research for Apache Thrift:
Advantages:
provide compiler that generates source code for supported languages (classes, all things that are important)
allow defining optional fields in the structures ( when we do not set value on a field, the size of transfered data is lower)
enable point out some methods that are "one way" (returning nothing and client after invokation do not wait for answer from server about completion processing of query)
support collections (maps, lists, sets), objects, primitives serialization (deserialization), constants, enumerations, exceptions
most of problems, errors are solved and explained
provide different methods of serialization: (TBinaryProtocol...) and different ways of exchanging data: (TBufferedTransport, TZlibTransport... )
compiler produces classes (structures) for languages thaw we can extend by adding some new methods.
possible to add fields to protocol(server as well as client) and remove other- old code and new one can properly interact(some rules in update)
enable asynchronous calls
easy to use
Disadvantages:
documentation - contains some errors that sometimes it is really hard to get to know what is the source of the problem
not allways problems are well taged (when we look for solution in the Internet).
not support overloading for service methods
tutorials cover only simple examples of thrift usage
hard to start
ICE ZeroC:
Is better than Protocol Buffers, because I wouldn't need to implement object passing by myself via e.g. sockets. ICE also gives ServantLocators which can provide management of connections.
The question is: whether ICE is much slower and less efficient than the PB?
I am working on one real estate website which is Using RETS service to get the data to my local server.
but I have one little bit problem here,I can fetch data from RETS which is having about 3lacks record in RETS Database but I didn't find the way,How can I fetch that all records in bunch of 50k at a time ?
I didn't find any 'LIMIT' keyword on RETS.so how can I fetch without 'LIMIT' 50k records at a time?
Please help me.
RETS is not really much of a standard. It's more closely resembles a pseudo standard. It loosely defines an XML schema that describes real estate listings.
In version 1.x, the "standard" was composed of DTD documents. In 2.x, the "standard" uses XSD documents to describe the list.
http://www.rets.org/documentation
However, in practice, there is almost no consistency amongst implementers. Having connected to hundreds of "RETS Compliant" service providers, I'm convinced that not one of them is like any other one.
Furthermore, the 2.x "standard" has not changed in 3 years. It's an unmaintained, sloppy attempt at a standard. It (RETS) is often used as a business buzz word by non-technical people. In reality, it's just an arbitrary attempt at modeling real estate listing in XML.
Try asking the specific implementer for their documentation. Often, they don't have any. So, emailing the lead developer has frequently been helpful. Sometimes they'll provide a WSDL which will outline the supported calls. Often, the WSDL doesn't coincide with the actual service, so beware.
As for your specific question, try caching the results. Usually, the use of a limit on a RETS call is a sign of a direct dependency. As requests for your service increase, the load that your service puts on theirs will break (and not be appreciated). Also, if their service goes down (even temporarily), yours will be interrupted as well. Most importantly, it will make the live requests to your pages really, really slow (especially if their system is slow at the time). The listings usually don't change frequently enough for worries about stale data, so caching up to and hour is pretty acceptable.
Best of luck!
libRets provides support for generating a query with fetch limits:
http://www.crt.realtors.org/projects/rets/librets/documentation/api/classlibrets_1_1_search_request.html
But last I knew: I remember the company Intereality either ignored or outright didn't provide complete compatibility to RETS. Quickest way to know your dealing with them is that also thought making all "System" name's for table fields numeric.
If you're lucky, you're using a Rapattoni backed server and they do provide spec. compatible servers.
Last point, I can't for the life of me remember it's name, but I used to use a free Java based RETS tool to build valid queries ( included offset/limit clauses ) and that made it a tad easier to build automated fetchers for a client's batch processing system.
IN RETS if Count More Than limit then We can download using Batch form or we can remove that Limit using regex while downloading
Best way to solve Problem divide Data Count in small unit of download and while we have to consider download limit in mind Field for Divide that one in MLS/IDX I Suggest Modification Date and ListingDate