After performing an upgrade of a TFS server to 2018 Update 2 the search and indexing seems to be broken on one of our environments.
Any search gives "We encountered an unexpected error when processing your request" and I have work through all the troubleshooting docs to clean and reindex all collections. Also completely reinstalled the search package to the separate server we run for search and indexing to make sure we got the correct version running.
In the event logs on the TFS App Server a large number of these exceptions are logged:
Events (81277) completed with status FailedAndRetry. Event 81277
completed with message 'BeginBulkIndex-PushEventNotification: The
operation did not complete successfully because of exception
Microsoft.VisualStudio.Services.Search.Common.FeederException: Lots of
files rejected by Elasticsearch, failing this job. Failure Reason :
Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException:
ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ index
returned 404 _index: codesearchshared_1_0 _type:
SourceNoDedupeFileContractV3 _version: 0 error: Type:
type_missing_exception Reason: "type[SourceNoDedupeFileContractV3]
missing"
And another exception type also logged a lot of times indicating failure to index work items:
Microsoft.VisualStudio.Services.Search.Common.SearchPlatformException:
ES Exception: [HTTP Status Code: [200] BULK_API_ERROR: [ update
returned 404 _index: workitemsearchshared_0_2 _type: workItemContract
_version: 0 error: Type: type_missing_exception Reason: "type[workItemContract] missing" update returned 404 _index:
workitemsearchshared_0_2 _type: workItemContract _version: 0 error:
Type: type_missing_exception Reason: "type[workItemContract] missing"
The exceptions seems to indicate that some type registrations are missing like the workItemContract and SourceNoDedupeFileContractV3 but I can not find any errors on the search server installation logs.
Anyone got some suggestions on how to solve this and get the Elastic Search back into a working state?
We resolved to situation by completely uninstalling and then reinstalling everything related to search.
Uninstalled all Code/Work/Wiki extensions from all collections from the extension management in web admin
Removed the TFS Search Service feature from the TFS Admin Console.
Uninstalled the elasticsearch service from the separate search server, using the PowerShell script .\Configure-TFSSearch.ps1 -operation remove
Restart TFS Job Agent service
Deleted old Search related databas content from ALL collection databases using
DELETE FROM [Search].[tbl_IndexingUnit]
DELETE FROM [Search].[tbl_IndexingUnitChangeEvent]
DELETE FROM [Search].[tbl_IndexingUnitChangeEventArchive]
DELETE FROM [Search].[tbl_JobYield]
DELETE FROM [Search].[tbl_TreeStore]
DELETE FROM [Search].[tbl_DisabledFiles]
DELETE FROM [Search].[tbl_ItemLevelFailures]
DELETE FROM [Search].[tbl_ResourceLockTable]
Restart TFS Job Agent service
Reboot Search server
Run Configure Search feature wizard from TFS Admin Console, using existing search server
Install search package according to instructions by PowerShell script
.\Configure-TFSSearch.ps1 -Operation install -TFSSearchInstallPath
D:\ES -TFSSearchIndexPath D:\ESDATA -Port 9200 -Verbose
Completed Search configuration wizard from TFS Admin Console enabling code search for all existing collections
Checked that services were running and tested searching from the web application, it works!
According to your error info and TFS version, this issue similar to Unable to start search after upgrade to TFS 2018 Update 2
Try the solution in the question below:
It seemed that I had an invalid/problematic setting in following reg
key that update/install did not fix.
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Apache Software
Foundation\Procrun
2.0\elasticsearch-service-x64\Parameters\Java\Options
Value contained:
-Xms0m
-Xmx0m
Changing both from '0m' to '1g' fixed the issue. As far as I
understand '0m' defaults to 4GB, which might collide with my server
only having 2,8 GB of RAM. I will upgrade the server to follow the
minimum requirements.
Maybe the configuration tool could warn about this problem or set the
value to something that is possible.
Also take a look at this article, which maybe helpful: Elasticsearch 6.2.2 fails to run as a windows service
[This post refers to azure devops server 2019 to 2020 upgrade]
I also got
"We encountered an unexpected error when processing your request"
for any search - this was after migrating azure devops server 2019 to 2020 on a new box (was unable to perform the upgrade on the same box)
http://localhost:9200/_cat/health?v showed the only cluster TFS_Search_AZURE-DEVOPS in red status
In my particular case everything on the same box - windows 2019 + sql server 2019 + search service
Followed Per Salmi instructions and unfortunately that did not resolve the issue.
The solution in my case was rebuilding elastic search indexes (details and scripts):
Reindex repo
To reindex a Git or TFVC repository, execute the appropriate version of the script Re-IndexingCodeRepository.ps1 for your Azure DevOps Server or TFS version with administrative privileges. You're prompted to enter: [my values in brackets]
The SQL server instance name where the Azure DevOps Server or TFS configuration database is [azure-devops]
The name of the Azure DevOps Server or TFS collection database [Tfs_DefaultCollection]
The name of the Azure DevOps Server or TFS configuration database [Tfs_Configuration]
The type of reindexing to execute. Can be one of the following values: Git_Repository TFVC_Repository [Git_Repository]
The name of the collection [DefaultCollection]
The name of the repository to reindex. [your repo name here]
Reindex collection
To reindex a collection, execute the script TriggerCollectionIndexing.ps1 with administrative privileges. You're prompted to enter: [my values in brackets]
The SQL server instance name where the Azure DevOps Server or TFS configuration database is [azure-devops]
The name of the Azure DevOps Server or TFS collection database [Tfs_DefaultCollection]
The name of the Azure DevOps Server or TFS configuration database [Tfs_Configuration]
The name of the collection [DefaultCollection]
The entities to reindex. Can be one of the following values: All Code WorkItem Wiki [All]
The scripts above place reindexing jobs that typically take a few minutes to complete.
Right after execution I was getting an encouraging "We are not able to show results because one or more projects in your organization are still being indexed"
And after a few minutes results started to come in, http://localhost:9200/_cat/health?v showing the only cluster TFS_Search_AZURE-DEVOPS in green status
Related
Completely new install of MS Exchange 2019, installed all the prerequisites for the Mailbox role successfully and schema created across the 4 domain controllers in the environment.
Had to manually create the schema on the root domain as the Exchange server was on the child domain, however schema replicated successfully across the domains in the environment.
During the installation on the Exchange server, the installation stages will always stop at stage 8/12 (Installation of Mailbox role). Checked the logs and it returns the following error messages:
***[ERROR] Active Directory operation failed on dc.domain. The error is not retriable. Additional information: Directory object not found.
Active Directory response: 0000208D: NameErr: DSID-03100288, problem 2001 (NO_OBJECT), data 0, best match of: 'DC= dc, Dc= dc, Dc= dc'
[ERROR] The object does not exist
[ERROR-REFERENCE] Id=MailboxServiceControlLast_05b3bbd421504e0c93fefa6d5d1ae590
Component=EXCHANGE14:\Current\Release\Shared\Datacenter\Setup
Setup is stopping now because of one or more critical errors
Finished executing components tasks
Ending process Install-MailboxRole***
P.S In the right security groups to execute the installation (Schema and Enterprise Admins).
Any help or advice will be highly appreciated!
Have you changed the Users Container CN=Users,DC=domain,DC=com or the Object in question from Type Container to an Organizational Unit? The setup expects some Objects as Container. Users is one of them.
I keep getting the same error when starting the Akeneo Community Edition! It seems to be an error caused by Elastictsearch, but I cannot figure out what is wrong.
The Error message:
[OK] Database schema created successfully!
Updating database schema...
37 queries were executed
[OK] Database schema updated successfully!
Reset elasticsearch indexes
In StaticNoPingConnectionPool.php line 50:
No alive nodes found in your cluster
Im running on a uberspace server without docker and i'm trying to start it like mentioned here:
https://docs.akeneo.com/4.0/install_pim/manual/installation_ee_archive.html but with the community Edition instead.
Does anyone had the same error and knows how to help me out?
Maybe it a problem with the .env file for the entry point of elastic search. My .env: APP_INDEX_HOSTS=localhost:9200
Can you verify that the Elasticsearch search server is available on localhost:9200 when accessing it via curl/Postman/Sense or something else?
That error usually means the node is either not running, or not running on the configured port.
Pay also attention that your server follow the system requirements - https://docs.akeneo.com/4.0/install_pim/manual/system_requirements/system_requirements.html
We have: multi tier TFS 2017 update 3.
recently i moved elasticsearch from one of application tiers to another server.
So What did i do:
deleted elasticsearch service using cmd: sc delete servicename.
deleted IndexStore folder
deleted this tables
[Search].[tbl_IndexingUnit]
[Search].[tbl_IndexingUnitChangeEvent]
[Search].[tbl_IndexingUnitChangeEventArchive]
[Search].[tbl_JobYield]
[Search].[tbl_TreeStore]
[Search].[tbl_DisabledFiles]
[Search].[tbl_ResourceLockTable]
from all collection DB's
Installed elasticsearch at another server (actually it is oone of sql servers of our TFS instance)
Checked that service is avaliable at url : http://SearchUrl:9200
Set up both AT's search feature at http://SearchUrl:9200
Tried To search something.
Result:
1.Search returns "There was a problem processing your request.
Internal Server Error".
Index folder is only 36.5 Kbytes
In windows logs there is unhandled exception:
Assembly: Microsoft.TeamFoundation.Framework.Server, Version=15.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a; v4.0.30319
Service Host: 9025d248-2b1b-48a7-bb43-8abf779eeeaa (Development)
Process Details:
Process Name: w3wp
Process Id: 2580
Thread Id: 4688
Account name: Username
Detailed Message: TF30065: An unhandled exception occurred.
how can i fix it?
According to the error message "TF30065: An unhandled exception occurred." Seems it's similar with this issue : Search Code and Work Items does not work after upgrade to 2017 Update 2
Add based on David Jansen [MSFT]'s comments, you can try calling the Re-index scripts manually to fix that. The solution is also mentioned in the link Resetting Search Index in Team Foundation Server provided by Daniel's comment above.
There is a dedicated "code search server". I used the latest Code
Search Tools scripts (TFS2017.2) and did a WipeAndRestart, followed by
manually calling the Re-index scripts. This did the trick.
More related information you can reference below articles:
Manage Search in Team Foundation Server
Re-index a repository or collection
Well. Now it works
What did i do:
1) Restored all deleted tables :)
2) Set up Elasticsearch on a separate server
3) Removed search feature from AT servers
4) Deleted all data from tables
[Search].[tbl_IndexingUnit]
[Search].[tbl_IndexingUnitChangeEvent]
[Search].[tbl_IndexingUnitChangeEventArchive]
[Search].[tbl_JobYield]
[Search].[tbl_TreeStore]
[Search].[tbl_DisabledFiles]
[Search].[tbl_ResourceLockTable]
5) removed all search extensions
6) instaled search feature. Configured it to the elastic server
7) waited for indexing
Actually, i've noticed that "unhalndled exception" in search disappeared after indexation finished
So I've been working on implementing elastic search using the JDBC River plugin to get data from our SQL Server DB into elastic search.
I've got it working fine using the SQL Server credentials, but trying to use integrated security doesn't work. It will create the index, but it doesn't have data in it.
The parameters I've been using are:
PUT /_river/test_river/_meta
{
"type":"jdbc",
"jdbc":
{
"driver":"com.microsoft.sqlserver.jdbc.SQLServerDriver",
"url":"jdbc:sqlserver://testServer:1433;databaseName=TestDb;
integratedSecurity=true;",
"user":"",
"password":"",
"sql": "select * from users",
"poll":"30s",
"index":"testindex",
"type":"testusers"
}
}
I've tried quite a few things, including removing the user and password fields completely, removing integratedSecurity=true, but it gave the same result.
I've checked on their github for the river plugin and it says this issue was fixed back in January, but it still doesn't seem to be working.
Also I'm using elastic search version: 1.5.1
and jdbc river plugin version : 1.4.0.10
Any help would be much appreciated
Get rid of the user and password options. You're not gonna need them.
Check the console when running elasticserch.bat, you should see an error message when it tries to update the river. I'm going to go out on a limb and assume you're probably seeing an error stating that the file sqljdbc_auth.dll can't be found. If this is the case, you can download this file from here and copy the x64 version of sqljdbc_auth.dll to your java lib folder. For me, this folder is C:\ProgramData\Oracle\Java\javapath but you can type echo %path% in a console window to find yours.
Once you have followed these steps, restart elasticsearch.bat, and it should start processing your river. If not, post back with the output you're seeing when running elasticsearch.bat.
Background:
I am using the deployment tools in Visual Studio 2010.
I right clicked my project and selected Package/Publish settings. Put all my settings in there ...
I am then using "web deploy" to tranfer the files to my remote server running a remote agent service and this is working fine. The transforms i have on my Web.Release.config do their thing and the server can access the database I created manually.
Problem:
My next step was to get the Database Deployment working too.
I went into the Package / Publish SQL tab and entered my Connection string for the destination database.
(Data Source=MyDBServer;Initial Catalog=Database2;User ID=User;Password=pass)
This database is empty ready to accept the import.
I also enter in the connection string for the source database. This lives on the same server.
(Data Source=MyDBServer;Initial Catalog=Database;User ID=User;Password=pass)
Database Scripting options are set to Schema and Data (changing this makes no difference) and the database scripts are set to [Auto Generated Schema and Data]
When i deploy this now, i get the error:
Error 4 Web deployment task
failed.((09/06/2010 16:41:51) An error
occurred when the request was
processed on the remote computer.)
(09/06/2010 16:41:51) An error
occurred when the request was
processed on the remote computer. The
entry type 'Unknown' was not expected
at this time. The serialization stream
may be corrupted.
Additional Info:
I can successfully create a package with no problems. I looked at the contents in the zip and can see the SQL is generated fine (so no problems connecting to the database). I can then copy this SQL and run it as a new query on the new database and the tables and data are created fine.
I can not seem to work out where this is going wrong, i googled the error and there are no entries on the whole internet. Anyone have any ideas?
Addendum:
To get some further idea of what might be going on, i sent the package across to the server and imported it using IIS. It told me i needed SQL Server Management Objects. So I installed that.
Next attempt it told me my user did not have permission to create the database, I thought excellent this must be the problem. :Granted access - Re-run. Passed!
So i deleted all the tables and went back to VS2010 clicked publish and i get the same error. :(
Sorted it!
Thank goodness, i was totally out of ideas when i went back to a video by hanselman. He mentioned that the Web Deployment Agent can have permissions. I went in had a look and there was a tab in it's properties called log on.
I entered the detials of an account with a decent level of access and clicked okay.
I then restarted the service as requested to enable the changes.
I then went back to VS 2010 and clicked Publish Web.
Music to my eyes, i see the words "Publish succeeded", I check the database and the tables are there. Excellent!
I think i scared the office by getting a little over excited, if you get this problem and this solution fixes it for you, try to hold in the temptation to shout out "YES!, yes, get in!" while laughing maniacally or people will think you're weird like me.