I am using logstash+elasticsearch to index server logs. The Elasticsearch server is running at localhost:9200 with millions of server log docs.
I also have a Rails app running at localhost:3000. I need to connect this rails app to the ES server.
I have read about the "elasticsearch-rails" gem but everywhere i found them using ActiveRecords/ Models. However, i don't think ActiveRecords are required for this. I just need a way to query the ES server index and fetch the documents inside my Rails app.
Is there a way to do this? Can anyone please help me with this situation? please comment if I am not clear with my question.
Thanks in advance.
There are many solutions out there, but one that stands out is (e.g.) Stretcher which provides a very simple API that closely matches the Elasticsearch API.
For instance:
server = Stretcher::Server.new('http://localhost:9200')
res = server.index(:myindex).search(size: 10, query: {match_all: {}})
# res is of type Stretcher::SearchResults
res.total # => 10
res.documents # => [#<Hashie::Mash _id="4" text="Hello 4">, ...]
...
Related
Am trying to use elasticsearch with my neo4j database for fast querying.I tried many sites but they are all old articles so i didn't get any clear idea. Steps I followed until now,
Installed neo4j
Installed elasticsearch
Copy pasted elastic search plugins into neo4j plugins folder
added this line into neo4j. properties file
elasticsearch.host_name=http://localhost:9200
elasticsearch.index_spec=people:Person(first_name,last_name), places:Place(name)
Here my question is,
How elasticsearch and neo4j are integrated. Please clarify me on this.
I followed this ,
Link
You have to install Apoc procedures plugin (https://github.com/neo4j-contrib/neo4j-apoc-procedures). The documentation about ES integration is here : ES Integration with Apoc procedures
[edit]
download and drop apoc.jar in plugins's Neo4j directory, regarding the targetted Neo4j version
restart Neo4j
in Neo4j Web browser, launch the following Cypher query to show all ES procedures:
CALL apoc.help("apoc.es")
Sample query for logs:
CALL apoc.es.getRaw("localhost","_search?q=level:ERROR",null)
YIELD value
UNWIND value.hits.hits as hits
RETURN hits LIMIT 100
The recommanded way is to store the ES host in neo4j.conf by adding a key (after restart of Neo4j):
apoc.es.myKey.url=localhost
Then the query looks like:
CALL apoc.es.getRaw("myKey","_search?q=level:ERROR",null)
YIELD value
UNWIND value.hits.hits as hits
RETURN hits LIMIT 100
For those of you who already have APOC plugin installed and accessible, but don't have access to the neo4j.properties file (or are more comfortable working with ES through curl) you can do this without using apoc.es.getRaw and can instead use the JSON returned with apoc.load.json:
WITH "http://myelasticurl:9200/my_index/_search?q=level:ERROR" as search_url
CALL apoc.load.json(search_url) YIELD value
UNWIND value.hits.hits as hit
WITH hit._source as source
...
# do work
...
I have elasticsearch 5.6 and using log4j2 to config it now,
I save data in elasticsearch, And now i want to push data to sentry 8.22.
If elasticsearch reviceve a data then push the data to sentry automatically.
Can someone tell me how to do this?
PS:
I found some links like this Using sentry logging with elasticsearch
But the solution there is too old.
IMO that's not what Sentry is for: You want to find errors in your application, but it isn't a general log collector. You're also not trying to get your operating system, webserver, database,... hooked into Sentry, right?
If anything in Elasticsearch is going wrong, Sentry should collect the error in your application and you can dig deeper from there. No need to connect Elasticsearch directly.
PS: Adding logging libraries is definitely untested and you might run into various issues (at the very least every upgrade will be more complicated) — I'd be pretty careful with this.
Hi am new in elastic search, I installed the elastic search in my windows 7 machine but I can't know, how to run and use elastic search queries in windows where should I type the elastic search queries and where should I run this queries?..
Any one know about it help me. Thanks in advance...
There are multiple ways to do that.
via HTTP interface, which means that you can run GET queries via your browser (Firefox, Chrome etc.) by accesing the proper url like:
http://localhost:9200/_search?q=tag:wow
Elasticsearch's HEAD plugin. You can execute any query with it. It also has multiple additional functionalities.
Install cUrl for Windows and then run queries just like every tutorial suggests.
use any programming language like PHP that supports curl library.
Personally I prefer HEAD plugin since it has other functionalities that I use anyway.
you can also check sense plugin for chrome. It will also help you in syntax for queries.
you can get it from here
https://github.com/bleskes/sense
i am a beginner in elasticsearch and hadoop. i am having a problem with moving data from hdfs into elasticsearch server using es.net.proxy.http.host with credentials. Server secured with credentials using nginx proxy configuration. But when i am trying to move data using pig script it shows null pointer exception.
My pig script is
REGISTER elasticsearch-hadoop-1.3.0.M3/dist/elasticsearch-hadoop-1.3.0.M3.jar
A = load 'date' using PigStorage() as (date:datetime);
store A into 'doc/id' using org.elasticsearch.hadoop.pig.EsStorage('es.net.proxy.http.host=ipaddress','es.net.proxy.http.port=portnumber','es.net.proxy.http.user=username','es.net.proxy.http.pass=password');
I don't understand where is the problem with my script. Can anyone please help me?
Thanks in Advance.
I faced this type of problem.elasticsearch-hadoop-1.3.0.M3.jar should not support the Proxy and authentication setting.You will try to elasticsearch-hadoop-1.3.0.BUILDSHAPSHOT.jar file. But I couldn't move object data like Totuble to Production server with authentication
Thank U
I am new to Cassandra and was trying to achieve some simple
operations like inserting data into cassandra. I am using cassandra gem
to achieve this.
client = Cassandra.new('tags_logs', 'ec2-xxx-xxx-xxx.com:9160')
client.disable_node_auto_discovery!
client.get('tag_data','red')
And I get the following error:
ThriftClient::NoServersAvailable - No live servers in ...
I'm running this code from my local machine. And while I've no problem connecting using cassandra-cli (so it is not a firewall issue), the code refuses to work. It works perfectly when accessing Cassandra on my own local machine.
Any ideas?
Thanks,
Eden.
I recommend you to use this gem I'm developing: https://github.com/hsgubert/cassandra_migrations
It gives access to Cassandra through CQL3 and manages schema with migrations.
Note: it requires Rails.
For future generations: simply change the timeout ...
client = Cassandra.new('tags_logs', 'ec2-example-example-example.com:9160',:connect_timeout => 10000)