I have upgraded sonarqube server from 6.2 to 6.3.1 and since then I see a weird behaviour regarding the quality profile (it might have occurred before, it is only now I see it).
When I click on the Quality Profile SonarWay (Java) I see
so it seems, that all rules are inactive.
When I click Activate More I see the following
so it looks, that there are rules are active (I assume due to the "Deactivate" option").
But when switching in the left bar to "active" under Quality Profile results in this
so clearly, no rules are active.
What is the second image then showing, what does the "Deactivate" mean, although it is inactive ?
How could this happen that suddenly all rules seem to be inactivated ?
This specific behaviour is a common symptom of a corrupted Elastic Search index (no longer in sync with SonarQube database).
Solution
Rebuild the SonarQube ElasticSearch index:
stop your SonarQube server
delete the ElasticSearch index # sonar_install_dir/data/es
start your SonarQube server
(reminder: ElasticSearch is a search engine used by SonarQube to index issues, rules etc. so that it can access this data rapidly without having to query the database all the time, see SonarQube Architecture)
Root-cause
Why did that happen ? A common case is an ElasticSearch index not being properly rebuilt after upgrading and/or changing database. Here's a typical scenario: you first start SonarQube on embedded H2 database, experiment a bit with it, then plug it to a full-fledged database. If the ElasticSearch index does not get scratched/rebuilt in between, then the index gets corrupted as the database/dataset it used to be in synch with just changed all of the sudden.
FYI there's an improvement planned to handle this more gracefully: SONAR-5681 .
Note: independently from above solution, do not take ElasticSearch index rebuild as a lightweight operation that should be performed regularly. SonarQube does self-manage its ElasticSearch index, so any issue must be investigated first.
Related
Now I am using gradle mlReloadSchemas tasks to reload TDE templates.
I guess even if the change is for one tde file only, the reload schemas task may delete all in DB and load all TDE templates to ML DB.
I wonder whether it will cause a performance issue for ML. Will that trigger indexing even for the TDE files that have not yet changed?
I am using DevOps pipeline to trigger the schema reload from GIT repository. As such, I could not load only the change TDE file. I have to reload everything. If there is performance issue, how to load only changed file with the pipeline?
Redeploying TDE can cause reindexing. How many records to be reindexed depends upon the context matching for those TDE.
A properly resourced cluster should be able to handle the load of reindexing.
That being said, the merging activities can compete with online traffic and query demands. You can help minimize the impact by setting the reindex throttle to a lower level (1-5 with 1 being the lowest), and you can set a background-io limit to restrict the amount of IO any node will use for background activities such as merges and backups.
You can also choose when to enable/disable reindexing, and adjust the reindexing level to a higher/lower level at different periods.
https://help.marklogic.com/Knowledgebase/Article/View/how-reindexing-works-and-its-impact-on-performance
https://help.marklogic.com/Knowledgebase/Article/View/indexing-best-practices
I am using Elasticsearch, and I am building a client (using the Java Client API) to export logs indexed via Logstash.
I would like to be able to be notified (by adding a listener somewhere) when a new document is index (= a new log line have been added) instead of querying the last X documents.
Is it possible ?
This is what you're looking for: https://github.com/ForgeRock/es-change-feed-plugin
Using this plugin, you can register to a websocket channel to receive indexation/deletion events as they happen. It has some limitations, though.
Back in the days, it was possible to install river plugins to stream documents to ES. The river feature has been removed, but this plugin above is like a "reverse river", where outside clients are notified by ES as documents get indexed.
Very useful and seemingly up-to-date with ES 6.x
UPDATE (April 14th, 2019):
According to what was said at Elastic{ON} Zurich 2019, at some point in the 7.x series, there will be a Changes API that will provide index changes notifications (document creation, update, deletion and more).
UPDATE (July 22nd, 2022):
ES 8.x is out and the Changes API is still nowhere in sight ... Good to know, though, that's it's still open at least.
I have managed to process log files using the ELK kit and I can now see my logs on Kibana.
I have scoured the internet and can't seem to find a way to remove all the old logs, viewable in Kibana, from months ago. (Well an explaination that I understand). I just want to clear my Kibana and start a fresh by loading new logs and them being the only ones displayed. Does anyone know how I would do that?
Note: Even if I remove all the Index Patterns (in Management section), the processed logs are still there.
Context: I have been looking at using ELK to analyse testing logs in my work. For that reason, I am using ElasticSearch, Kibana and Logstatsh v5.4, and I am unable to download a newer version due to company restrictions.
Any help would be much appreciated!
Kibana screenshot displaying logs
Update:
I've typed "GET /_cat/indices/*?v&s=index" into the Dev Tools>Console and got a list of indices.
I initially used the "DELETE" function, and it didn't appear to be working. However, after restarting everything, it worked the seond time and I was able to remove all the existing indices which subsiquently removed all logs being displayed in Kibana.
SUCCESS!
Kibana is just the visualization part of the elastic stack, your data is stored in elasticsearch, to get rid of it you need to delete your index.
The 5.4 version is very old and already passed the EOL date, it does not have any UI to delete the index, you will need to use the elasticsearch REST API to delete it.
You can do it from kibana, just click in Dev Tools, first you will need to list your index using the cat indices endpoint.
GET "/_cat/indices?v&s=index&pretty"
After that you will need to use the delete api endpoint to delete your index.
DELETE /name-of-your-index
On the newer versions you can do it using the Index Management UI, you should try to talk with your company to get the new version.
I'd like to use Kibana + Elasticsearch as a tool for application failures investigation.
This is how I see that: application logs all failures, some responsible engineer (me) see that new unknown failures appeared on Kibana's dashboard, investigates why failures happened and than marks related documents with specific attribute.
Unfortunately, the only way to update documents from Kibana UI I found so far are raw queries from "Dev tools".
Maybe there are some plugins that allow documents editing via fancy UI?
I've a "Failed" message displayed when I declare a "false-positive" issue from the Web UI of Sonarqube (5.0.1).
A 400 error is returned with this response : "Transition from state RESOLVED does not exist: falsepositive"
But the issue is declared "unresolved" in the web ui (both on global issues and project page) and it broke the build (since I'm using Build Breaker plugin).
I have a doubt on the shown issue because it should be logically solved by my previous commit (cache problem ?).
How can I resolve this problem ?
This can happen when the ElasticSearch index of issues is out of sync with the database. You should stop your server, remove the $SQ_HOME/data/es directory and restart the server. It might take some time, depending on the number of issues in your database, but it should solve the problem.