Reporting Kibana - elasticsearch

I am using kibana 6.5.4 which means that I should be able to see the reporting tab in my kibana as Xpack should've already been installed. However, the reporting tab is not shown.
I've followed these instructions to enable reporting
https://www.elastic.co/guide/en/kibana/current/reporting-settings-kb.html#reporting-kibana-server-settings
When I set the property in my kibana.yml like below
xpack.reporting.enabled: true
I get the following error when attempting to start kibana.
error [11:30:40.296] [fatal][root] Error: "xpack.reporting.enabled"
setting was not applied. Check for spelling errors and ensure that
expected plugins are installed.
at KbnServer.exports.default
(/usr/local/Cellar/kibana/6.5.4/src/server/config/complete.js:21:17)
at KbnServer.mixin
(/usr/local/Cellar/kibana/6.5.4/src/server/kbn_server.js:170:16)
at <anonymous>
FATAL Error: "xpack.reporting.enabled" setting was not applied. Check for spelling errors and ensure that expected plugins are installed.
I don't understand, as per docs Xpack should be automatically installed with kibana since 6.3+?

Related

Suitecrm Elasticsearch Integration

I'm trying to integrate elsaicsearch with suitecrm.
I've followed the document as per https://docs.suitecrm.com/admin/administration-panel/search/elasticsearch/
Then I tried to run full indexing from suitecrm, I got "index_not_found_exception" hence created indexes manually in the elasticsearch.
After that also when I am trying to run the indexing, no logs showing in suitecrm or elasticsearch and search in elasticsearch is not working.
Suitecrm version
Version 7.12.5
Sugar Version 6.5.25 (Build 344)
Elasticsearch version
"number" : "7.17.5",
Please advise. Thanks.
I didn't get any suspicious logs in suitecrm.log so had enabled debug mode for the logs
https://docs.suitecrm.com/developer/logging/
Then I clicked on full indexing I found below log line
Elasticsearch trying to re-indexing a bean but this module is blacklisted: SchedulersJobs
I followed this document then https://docs.suitecrm.com/blog/scheduler-jobs/
And lastly this step Admin / Repairs / Quick Repair and Rebuild
After that it started working

FATAL Error: [config validation of [monitoring].enabled]: definition for this key is missing

I upgraded the ELK stack to version 8.0.0 and this is the error I am getting in the logs of my Kibana pod: "FATAL Error: [config validation of [monitoring].enabled]: definition for this key is missing". How should I fix this? kibana.yml
From 8.x.x, the general monitoring settings (monitoring.enabled) is no longer supported. Take a look at the settings that are at different levels in Kibana's monitoring settings page. For you, removing the settings will run the kibana.
I had similar issue with elk 8.x.x "config validation of xpack.fleet.enabled: definition for this key is missing". after I removed xpack.fleet.enabled from kibana.yml issue was fixed. Take a look at this fleet settings in kibana

Unable to upgrade elastic search from 5 to 6

I am trying to run the upgrade assistant in order to upgrade ElasticSearch from 5 to 6, however I am hitting the following error:
Deprecated script security settings changes
This issue must be resolved to upgrade. Read Documentation
Details: [node[instance-0000000093] used these script-security settings:[script.inline, script.stored, script.file, script.engine.painless.stored, script.engine.painless.file, script.engine.expression.stored, script.engine.expression.file, script.engine.mustache.inline, script.engine.mustache.stored, script.engine.mustache.file]]
I have looked over the settings and our Kibana config and cannot find any references to any scrips being used.
Has anyone else run into a similar issue and how did you solve it?
Thanks.

Search fails after upgrade to TFS 2018 Update 2

After performing an upgrade of a TFS server to 2018 Update 2 the search and indexing seems to be broken on one of our environments.
Any search gives "We encountered an unexpected error when processing your request" and I have work through all the troubleshooting docs to clean and reindex all collections. Also completely reinstalled the search package to the separate server we run for search and indexing to make sure we got the correct version running.
In the event logs on the TFS App Server a large number of these exceptions are logged:
Events (81277) completed with status FailedAndRetry. Event 81277
completed with message 'BeginBulkIndex-PushEventNotification: The
operation did not complete successfully because of exception
Microsoft.VisualStudio.Services.Search.Common.FeederException: Lots of
files rejected by Elasticsearch, failing this job. Failure Reason :
Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException:
ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ index
returned 404 _index: codesearchshared_1_0 _type:
SourceNoDedupeFileContractV3 _version: 0 error: Type:
type_missing_exception Reason: "type[SourceNoDedupeFileContractV3]
missing"
And another exception type also logged a lot of times indicating failure to index work items:
Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException:
ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ update
returned 404 _index: workitemsearchshared_0_2 _type: workItemContract
_version: 0 error: Type: type_missing_exception Reason: "type[workItemContract] missing" update returned 404 _index:
workitemsearchshared_0_2 _type: workItemContract _version: 0 error:
Type: type_missing_exception Reason: "type[workItemContract] missing"
The exceptions seems to indicate that some type registrations are missing like the workItemContract and SourceNoDedupeFileContractV3 but I can not find any errors on the search server installation logs.
Anyone got some suggestions on how to solve this and get the Elastic Search back into a working state?
We resolved to situation by completely uninstalling and then reinstalling everything related to search.
Uninstalled all Code/Work/Wiki extensions from all collections from the extension management in web admin
Removed the TFS Search Service feature from the TFS Admin Console.
Uninstalled the elasticsearch service from the separate search server, using the PowerShell script .\Configure-TFSSearch.ps1 -operation remove
Restart TFS Job Agent service
Deleted old Search related databas content from ALL collection databases using
DELETE FROM [Search].[tbl_IndexingUnit]
DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]
DELETE FROM [Search].[tbl_IndexingUnitChangeEventArchive]
DELETE FROM [Search].[tbl_JobYield]
DELETE FROM [Search].[tbl_TreeStore]
DELETE FROM [Search].[tbl_DisabledFiles]
DELETE FROM [Search].[tbl_ItemLevelFailures]
DELETE FROM [Search].[tbl_ResourceLockTable]
Restart TFS Job Agent service
Reboot Search server
Run Configure Search feature wizard from TFS Admin Console, using existing search server
Install search package according to instructions by PowerShell script
.\Configure-TFSSearch.ps1 -Operation install -TFSSearchInstallPath
D:\ES -TFSSearchIndexPath D:\ESDATA -Port 9200 -Verbose
Completed Search configuration wizard from TFS Admin Console enabling code search for all existing collections
Checked that services were running and tested searching from the web application, it works!
According to your error info and TFS version, this issue similar to Unable to start search after upgrade to TFS 2018 Update 2
Try the solution in the question below:
It seemed that I had an invalid/problematic setting in following reg
key that update/install did not fix.
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software
Foundation\Procrun
2.0\elasticsearch-service-x64\Parameters\Java\Options
Value contained:
-Xms0m
-Xmx0m
Changing both from '0m' to '1g' fixed the issue. As far as I
understand '0m' defaults to 4GB, which might collide with my server
only having 2,8 GB of RAM. I will upgrade the server to follow the
minimum requirements.
Maybe the configuration tool could warn about this problem or set the
value to something that is possible.
Also take a look at this article, which maybe helpful: Elasticsearch 6.2.2 fails to run as a windows service
[This post refers to azure devops server 2019 to 2020 upgrade]
I also got
"We encountered an unexpected error when processing your request"
for any search - this was after migrating azure devops server 2019 to 2020 on a new box (was unable to perform the upgrade on the same box)
http://localhost:9200/_cat/health?v showed the only cluster TFS_Search_AZURE-DEVOPS in red status
In my particular case everything on the same box - windows 2019 + sql server 2019 + search service
Followed Per Salmi instructions and unfortunately that did not resolve the issue.
The solution in my case was rebuilding elastic search indexes (details and scripts):
Reindex repo
To reindex a Git or TFVC repository, execute the appropriate version of the script Re-IndexingCodeRepository.ps1 for your Azure DevOps Server or TFS version with administrative privileges. You're prompted to enter: [my values in brackets]
The SQL server instance name where the Azure DevOps Server or TFS configuration database is [azure-devops]
The name of the Azure DevOps Server or TFS collection database [Tfs_DefaultCollection]
The name of the Azure DevOps Server or TFS configuration database [Tfs_Configuration]
The type of reindexing to execute. Can be one of the following values: Git_Repository TFVC_Repository [Git_Repository]
The name of the collection [DefaultCollection]
The name of the repository to reindex. [your repo name here]
Reindex collection
To reindex a collection, execute the script TriggerCollectionIndexing.ps1 with administrative privileges. You're prompted to enter: [my values in brackets]
The SQL server instance name where the Azure DevOps Server or TFS configuration database is [azure-devops]
The name of the Azure DevOps Server or TFS collection database [Tfs_DefaultCollection]
The name of the Azure DevOps Server or TFS configuration database [Tfs_Configuration]
The name of the collection [DefaultCollection]
The entities to reindex. Can be one of the following values: All Code WorkItem Wiki [All]
The scripts above place reindexing jobs that typically take a few minutes to complete.
Right after execution I was getting an encouraging "We are not able to show results because one or more projects in your organization are still being indexed"
And after a few minutes results started to come in, http://localhost:9200/_cat/health?v showing the only cluster TFS_Search_AZURE-DEVOPS in green status

Sonarqube Error when activating rules in new Quality Profile

I'm using SonarQube server version 5.6.1 in docker. I have correctly configured the default Quality Profiles but now when trying to activate existing rules in a new Quality profile, the application throws the following error in the log:
ERROR web[o.s.s.es.BulkIndexer] Fail to execute bulk index request: Bulk[1 index request(s) on index rules and type activeRule] org.elasticsearch.transport.RemoteTransportException: [sonar-470917037901][inet[/127.0.0.1:9001]][indices:data/write/bulk] Caused by: org.elasticsearch.index.mapper.MapperParsingException: The provided routing value [squid] doesn't match the routing key stored in the document: [squid:S2204]
Do any of you have an idea of what could be happening? The rule appears as active in that profile but the profile says 0 active rules.
Thank you for your help.
I manage to find another question that contains a lookalike problem like mine in: Elasticsearch bulk index error on Sonar startup.
Tried the same answer and it worked.
TL;DR Delete the directory $SONAR_HOME/data/es/ and restart server.

Resources