Integration of elasticsearch with neo4j database - elasticsearch

Am trying to use elasticsearch with my neo4j database for fast querying.I tried many sites but they are all old articles so i didn't get any clear idea. Steps I followed until now,
Installed neo4j
Installed elasticsearch
Copy pasted elastic search plugins into neo4j plugins folder
added this line into neo4j. properties file
elasticsearch.host_name=http://localhost:9200
elasticsearch.index_spec=people:Person(first_name,last_name), places:Place(name)
Here my question is,
How elasticsearch and neo4j are integrated. Please clarify me on this.
I followed this ,
Link

You have to install Apoc procedures plugin (https://github.com/neo4j-contrib/neo4j-apoc-procedures). The documentation about ES integration is here : ES Integration with Apoc procedures
[edit]
download and drop apoc.jar in plugins's Neo4j directory, regarding the targetted Neo4j version
restart Neo4j
in Neo4j Web browser, launch the following Cypher query to show all ES procedures:
CALL apoc.help("apoc.es")
Sample query for logs:
CALL apoc.es.getRaw("localhost","_search?q=level:ERROR",null)
YIELD value
UNWIND value.hits.hits as hits
RETURN hits LIMIT 100
The recommanded way is to store the ES host in neo4j.conf by adding a key (after restart of Neo4j):
apoc.es.myKey.url=localhost
Then the query looks like:
CALL apoc.es.getRaw("myKey","_search?q=level:ERROR",null)
YIELD value
UNWIND value.hits.hits as hits
RETURN hits LIMIT 100

For those of you who already have APOC plugin installed and accessible, but don't have access to the neo4j.properties file (or are more comfortable working with ES through curl) you can do this without using apoc.es.getRaw and can instead use the JSON returned with apoc.load.json:
WITH "http://myelasticurl:9200/my_index/_search?q=level:ERROR" as search_url
CALL apoc.load.json(search_url) YIELD value
UNWIND value.hits.hits as hit
WITH hit._source as source
...
# do work
...

Related

ELK - Removing old logs viewable in Kibana

I have managed to process log files using the ELK kit and I can now see my logs on Kibana.
I have scoured the internet and can't seem to find a way to remove all the old logs, viewable in Kibana, from months ago. (Well an explaination that I understand). I just want to clear my Kibana and start a fresh by loading new logs and them being the only ones displayed. Does anyone know how I would do that?
Note: Even if I remove all the Index Patterns (in Management section), the processed logs are still there.
Context: I have been looking at using ELK to analyse testing logs in my work. For that reason, I am using ElasticSearch, Kibana and Logstatsh v5.4, and I am unable to download a newer version due to company restrictions.
Any help would be much appreciated!
Kibana screenshot displaying logs
Update:
I've typed "GET /_cat/indices/*?v&s=index" into the Dev Tools>Console and got a list of indices.
I initially used the "DELETE" function, and it didn't appear to be working. However, after restarting everything, it worked the seond time and I was able to remove all the existing indices which subsiquently removed all logs being displayed in Kibana.
SUCCESS!
Kibana is just the visualization part of the elastic stack, your data is stored in elasticsearch, to get rid of it you need to delete your index.
The 5.4 version is very old and already passed the EOL date, it does not have any UI to delete the index, you will need to use the elasticsearch REST API to delete it.
You can do it from kibana, just click in Dev Tools, first you will need to list your index using the cat indices endpoint.
GET "/_cat/indices?v&s=index&pretty"
After that you will need to use the delete api endpoint to delete your index.
DELETE /name-of-your-index
On the newer versions you can do it using the Index Management UI, you should try to talk with your company to get the new version.

To perform search operation through UI in Jhipster using elastic search for existing database MySQL

I started Jhipster for PoC purpose and I need to perform search through UI for already existing data present in MySQL database.I have the following doubts:
Do we need to install/have elastic search in order to run it first and check for the results?
Or choosing elastic search while creating the Jhipster application and configuring it is enough to use it further?
I have tried using generator-jhipster-elasticsearch-reindexer module by installing it. but it did not worked as expected.After installing it I ended up with the following error.
java.lang.IllegalStateException: handshake failed, mismatched cluster name [Cluster [internal-test-cluster-name2843e241-29cc-4bc0-82db-600eb78ed261]] - {127.0.0.1:9300}{pbkSwq2SQ-CTopOjTqsVcg}{127.0.0.1}{127.0.0.1:9300}
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:404)
at org.elasticsearch.transport.TransportService.handshake(TransportService.java:367)
at org.elasticsearch.discovery.zen.UnicastZenPing$PingingRound.getOrConnect(UnicastZenPing.java:366)
at org.elasticsearch.discovery.zen.UnicastZenPing$3.doRun(UnicastZenPing.java:471)
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723)
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
The above error was resolved by adding sprig.data.jest.uri in application-dev.yml, but still the search mechanism is not working. i.e., It is not able to query on existing database.
Actual result: able to perform search from UI when i create an entity from the UI. Reason being when an entity is made through API then it is updating the elastic search database and producing the result.
Expected result: To able to perform the same search for already existing data when I connect to the MySQL database.
Jhipster already explained how to work with elasticsearch on their website:
https://www.jhipster.tech/using-elasticsearch/
Here is short answer for your question:
1. You do not need to install elasticsearch if you run your app in dev profile, because it uses embedded Elasticsearch instance.
2. You must select Elasticsearch open while creating Jhipster app so that the generator add search capabilities to your code.
The generator-jhipster-elasticsearch-reindexer module only works if you have enabled Elasticsearch in your app.

Elastic search next steps

I'm new to elasticsearch and am still trying to set it up. I have installed elasticsearch 5.5.1 using default values I have also installed Kibana 5.5.1 using the default values. I've also installed the ingest-attachment plugin with the latest x-pack plugin. I have elasticsearch running as a service and I have Kibana open in my browser. On the Kibana dashboardI have an error stating that it is unable to fetch mappings. I guess this is because I havn't set up any indices or pipelines yet. This is where I need some steer, all the documentation I've found so far on-line isn't particularly clear. I have a directory with a mixture of document types such as pdf and doc files. My ultimate goal is to be able to search these documents with values that a user will enter via an app. I'm guessing I need to use the Dev Tools/console window in Kibana using the 'PUT' command to create a pipeline next, but I'm unsure of how I should do this so that it points to my directory with the documents. Can anybody provide me an example of this for this version please.
If I understand you correctly, let's first set some basic understanding about elasticsearch:
Elasticsearch in it's simple definition is a "Search engine". so you need to store some data, and then elastic will help you to search using a search criteria, and it will retrieve relevant data back
You need a "Container" to save your data to, and elastic has this thing like any database engine to store your data, but the terms are somehow different. for example a "Database" in sql-like systems is called "Index", and what you know as "table" is called "Type" in elastic.
from my understanding, you will need to create your index (with or without mappings) to have a starting point, and I recommend you to start without mappings just to "start" and get things working, but later on it's highly recommend to work with "mappings" if applicable, because elastic is smart, but it cannot know more about your data than you do
Because Kibana has failed to find a proper index to start with, it has complained and asked you to either provide a syntax for index names, or a specific index name so it can infer the inline mappings and give you the nice features of querying, displaying charts, etc of your data, so once you create your index, you will provide that to the starting page of Kibana, and you will be ready to go.
Let me know if you need something more specific to your needs :)

Spring Data Couchbase - Search without having admin rights on the cluster

I'm currently working on a POC with Couchbase, using Spring Data to put & get documents on/off a bucket on a cluster.
As I'm working in a big company, I'm lucky they gave me a bucket, but still I don't have the admin rights on the cluster, so I only have access to the bucket.
But as I'm digging into the Spring Data documentation, I'm not able to find a way to retrieve documents without creating views on the server. (I'm getting errors like "Unknown query param" ). Nevertheless with couchbase java sdk i'm able to, through n1ql queries, but the use of the Spring data layer is mandatory.
The answers I found always point me to the server-side function direction
ex : https://stackoverflow.com/a/30928169/3744307
What I would like to find, is a way to add a repository method like
List findReceiptByAccount(String Account)
without having to specificly declare the function server-side.
Is this possible, or have I to send a request to the administrators to create functions for me everytime I have to add a findByX method?
Thanks for your time,
What version of CB is it ?
I think that prior to 4.5, a n1ql access (which you seems to have) is enough to build your index yourself !
With Spring Data Couchbase 2.x that would use a N1QL index in the background, and it would work with a single primary index (although having 1 index per repository entity class would be best for performance). Maybe you can ask your admin to create that index once?

Elastic Search JDBC River Plugin SQL Server Integrated Security

So I've been working on implementing elastic search using the JDBC River plugin to get data from our SQL Server DB into elastic search.
I've got it working fine using the SQL Server credentials, but trying to use integrated security doesn't work. It will create the index, but it doesn't have data in it.
The parameters I've been using are:
PUT /_river/test_river/_meta
{
"type":"jdbc",
"jdbc":
{
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://testServer:1433;databaseName=TestDb;
integratedSecurity=true;",
"user":"",
"password":"",
"sql": "select * from users",
"poll":"30s",
"index":"testindex",
"type":"testusers"
}
}
I've tried quite a few things, including removing the user and password fields completely, removing integratedSecurity=true, but it gave the same result.
I've checked on their github for the river plugin and it says this issue was fixed back in January, but it still doesn't seem to be working.
Also I'm using elastic search version: 1.5.1
and jdbc river plugin version : 1.4.0.10
Any help would be much appreciated
Get rid of the user and password options. You're not gonna need them.
Check the console when running elasticserch.bat, you should see an error message when it tries to update the river. I'm going to go out on a limb and assume you're probably seeing an error stating that the file sqljdbc_auth.dll can't be found. If this is the case, you can download this file from here and copy the x64 version of sqljdbc_auth.dll to your java lib folder. For me, this folder is C:\ProgramData\Oracle\Java\javapath but you can type echo %path% in a console window to find yours.
Once you have followed these steps, restart elasticsearch.bat, and it should start processing your river. If not, post back with the output you're seeing when running elasticsearch.bat.

Resources