I am looking for some high level advice on the following:
I want to use elasticsearch in an ember.js single page application.
I have a sinatra application which is the API layer for this application.
What would be the best way of utilising elasticsearch?
Should I create a proxy layer in the sinatra application that creates elasticsearch queries and transforms the results into the appropriate ember.js models?
I don't think that querying the elasticsearch API directly is a good idea.
Both might be valid approaches but I would say you better stick with the sinatra "proxy".
Here are some reasons:
Easier to change implementation of the search engine. Maybe you want to use Solr later on?
Control usage
ElasticSearch is like a database. Do you want to give direct access to the end user?
What if you want to monitor the search your users make? With sinatra in the middle this is really easy.
Related
I'm new to using elastic search, and I'm trying to find a datastore for our application where we can also add a front end for analytics, in this case Kibana. I'm planning to use them as a datastore for dr/cr transactions on our billing system.
Most use case I read is towards data analytics and searching related. I don't see a use case wherein it is used as a regular datastore for an application. So I'm worried I might use it on a wrong use case.
I was hoping if anyone can add their insights on this. Like why or why not use Elastic Search as authoritative/primary datastore for applications.
You should read a official blog of elasticsearch, where they clearly mentioned that databases must be robust and should not stop working unless you tell to do it.
From the robustness section of same blog
A database should be robust, especially if it is your authoritative
system of record. Ideally, a costly query should be possible to
cancel, and you certainly don't want the database to stop working
unless you tell it to.
Unfortunately, Elasticsearch (and the components it's made of) does
not currently handle OutOfMemory-errors very well. We cover this in
more depth in Elasticsearch in Production, OutOfMemory-Caused Crashes.
It is very important to provide Elasticsearch with enough memory and
be careful before running searches with unknown memory requirements on
a production cluster.
In short, you shouldn't use Elasticsearch as a primary data-store where you can't afford to loose the data.
I was trying to understand how Elastic-search compares with GraphQL when they try to solve similar purpose, or does GraphQL uses Elastic-search as a datasource? If anyone done further research share your understanding here ? Thanks in advance.
GraphQL is as the name suggests a query language (mostly for Web APIs). Elastic Search is a data store that exposes a "RESTful" interface. This interface also has some kind of a query language. In that sense they solve different problems:
GraphQL is for exposing data to web clients or apps. It is build to solve challenges faced in client server communication and app development. GraphQL tries to reduce the amount requests and the size of data sent between client and server. Furthermore it give you the ability to extend your API without versioning to keep old clients (e.g. old versions of your mobile app) working.
Elastic search is built to query large amounts of data effectively. Some of their prime use cases are advertised on their website. Usually you would not want to expose the elastic API directly to your client. GraphQL could act as a layer in between that restricts the operations allowed for clients and uses - as you said - elastic as a data source. Or maybe elastic search at some point likes GraphQL so much that they offer an API to write queries in GraphQL that would replace the REST API.
So now that we know that they solve different problems and can be used together, comparing them doesn't really make much sense.
I am learning elasticsearch. I wanted to know how safe (in terms of access control & validating user access) it is to access ES server directly from JavaScript API rather than accessing it through your backend ? Will it be a safe to access ES directly from Javascript API ?
Depends on what you mean by "safe".
If you mean "safe to expose to the internet", then no, definitely not, as there isn't any access control and anyone will be able to insert data or even drop all the indexes.
This discussion gives a good overview of the issue. Relevant section:
Just as you would not expose a database directly to the Internet and let users send arbitrary SQL, you should not expose Elasticsearch to the world of untrusted users without sanitizing the input. Specifically, these are the problems we want to prevent:
Exposing private data. This entails limiting the searches to certain indexes, and/or applying filters to the searches.
Restricting who can update what.
Preventing expensive requests that can overwhelm or crash nodes and/or the entire cluster.
Preventing arbitrary code execution through dynamic scripts.
Its most certainly possible, and it can be "safe" if say you're using it as an internal tool behind some kind of authentication. In general, no, its not secure. Elasticsearch API can create, delete, update, and search, which is not something you want to give a client access to or they could essentially do any or all of these things.
You could, in theory, create role based auth with ElasticSearch Shield, but it's far from standard practice. Its not really anymore difficult to implement search on your backend then just have a simple call to return search results.
I am trying to see if I can create a simple website, like a blog, using only Ruby. No Rails or a database or outside web servers. I plan to store the data in a file for persistence.
I wanted to use TCPServer, CGI, and Net::HTTP.
Is there an easier way I can use?
There are a lot of moving parts when designing a website.
Depending on the purpose of the exercise, you might want to consider using a very simple web framework like Camping, Sinatra, or Ramaze. This is probably the best solution if you're trying to get a top level understanding of web programming because it only has exactly what you need (Camping is less than 4k!) and handles stuff like routing.
Building a web server is more an exercise in HTTP parsing. You might want to omit the framework and try to build something on top of Rake (an API for lots of popular web servers) and a simple web server like Webrick or Thin.
You could also try Espresso
It is easy to learn and fast to run.
And offers all the liberty you need for creation process.
Also it has no hidden "fees", everything is transparent.
Hy,
I'm implementing an IMAP client as a Mac OSX application using MacRuby.
For the sake of offline availability, I wanted to allow fulltext indexing and attribute based indexing of all messages. Attributes include common E-Mail stuff like from:, to:, etc...
This would allow for advanced results sprinkled with faceting, analytic calculations and such.
Now I'm unsure about the choices and good practices when it comes to integrating such a search feature. I have a strong web development background, therefore my intuitive action would be to setup a Solr server and start feeding it with data. This might just work theoretically, as I could write an Agent that manages the solr instance for my application in the background. But to me, this approach seems like an infrastructure hassle.
On the other side, I've read about people using the FTS3 functionality from SQLite. This approach is easily accessible by CoreData. I haven't used SQLite's FTS3 but I don't think it is as powerful as Solr can be.
What is your weapon of choice for a use case like mine?
I'm mainly interested in solutions that are actually in use by Objective-C/Cocoa/MacRuby developers.
In you're going to develop the app with Ruby give a try to picky. It is very simple to use.
There is an Objective-C Lucene port
http://svn.gna.org/viewcvs/etoile/trunk/Etoile/Frameworks/LuceneKit/
I have not used it, but in your situation, I'd at least check it out. In my experience, SQL based full text search can't compete with Lucene, but haven't tried SQLite for this.
EDIT: just noticed the ruby tag -- this started out as port of Lucene
https://github.com/dbalmain/ferret