We have a lab Exchange Server and an O365 tenant, and have Azure AD Connector working to sync the directories. However, the HCW fails, and it seems to reach the point where it is running this cmdlet to test whether mailbox migration is available.
Please help!
PS > Test-MigrationServerAvailability -ExchangeRemoteMove: $true -RemoteServer '4042d42c-144f-4329-914c-1d0a94ec27eb.resource.mailboxmigration.his.msappproxy.net' -Credentials (Get-Credential -UserName *****)
PowerShell credential request
Enter your credentials.
Password for user *****: **********
RunspaceId : 7cb350b0-463e-4d34-9a06-2c50f64993f3
Result : Failed
Message : The connection to the server '4042d42c-144f-4329-914c-1d0a94e
c27eb.resource.mailboxmigration.his.msappproxy.net' could
not be completed.
SupportsCutover : False
ErrorDetail : Microsoft.Exchange.Migration.MigrationServerConnectionFailedE
xception: The connection to the server '4042d42c-144f-4329-91
4c-1d0a94ec27eb.resource.mailboxmigration.his.msappproxy.net'
could not be completed. ---> Microsoft.Exchange.MailboxRepli
cationService.MRSRemoteTransientException: The call to 'https
://4042d42c-144f-4329-914c-1d0a94ec27eb.resource.mailboxmigra
tion.his.msappproxy.net/EWS/mrsproxy.svc' timed out. Error
details: The open operation did not complete within the
allotted timeout of 00:00:50. The time allotted to this
operation may have been a portion of a longer timeout. ---> M
icrosoft.Exchange.MailboxReplicationService.MRSRemotePermanen
tException: The open operation did not complete within the
allotted timeout of 00:00:50. The time allotted to this
operation may have been a portion of a longer timeout.
--- End of inner exception stack trace ---
at Microsoft.Exchange.MailboxReplicationService.MailboxRep
licationServiceFault.ReconstructAndThrow(String serverName,
VersionInformation serverVersion)
at Microsoft.Exchange.Connections.Common.WcfClientWithFaul
tHandling`2.<>c__DisplayClass4_0.<CallService>b__0()
at
Microsoft.Exchange.Net.WcfClientBase`1.CallService(Action
serviceCall, String context)
at Microsoft.Exchange.Connections.Common.WcfClientWithFaul
tHandling`2.CallService(Action serviceCall, String context)
at Microsoft.Exchange.MailboxReplicationService.WcfClientW
ithVersion`2.CallService(Action serviceCall, String context)
at Microsoft.Exchange.Migration.MigrationExchangeProxyRpcC
lient.CanConnectToMrsProxy(Fqdn serverName, Guid mbxGuid,
NetworkCredential credentials, LocalizedException& error)
--- End of inner exception stack trace ---
at Microsoft.Exchange.Migration.MigrationEndpointVerifier.
VerifyConnectivity(MigrationEndpointBase endpoint)
at Microsoft.Exchange.Management.Migration.MigrationServic
e.Endpoint.TestMigrationServerAvailability.InternalProcessEnd
point(Boolean fromAutoDiscover)
TestedEndpoint :
IsValid : True
Identity :
ObjectState : New
Related
I have a springboot application which uses azure sdk. I want to set the retry count to just once for authenticating since currently it uses the default value of 3 as I want the exception to be thrown without much delay for incorrect credentials.
com.azure.core.http.policy.RetryPolicy : Retry attempts have been exhausted after 3 attempts.
I tried debugging and found this, https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/resourcemanager/docs/AUTH.md but the Retry Policy only specifies after how long we can retry, not how many times. Further checking, RetryPolicy creates a new ExponentialBackOff instance - and here I see this comment:
Creates an instance of ExponentialBackoff with a maximum number of retry attempts configured by the environment property Configuration.PROPERTY_AZURE_REQUEST_RETRY_COUNT, or three if it isn't configured or is less than or equal to 0. This strategy starts with a delay of 800 milliseconds and exponentially increases with each additional retry attempt to a maximum of 8 seconds.
At this point, not sure how to proceed. Can someone point me how we can set the retries only for this particular method?
public AzureResourceManager getAzureResourceManagerClient(String clientId, String clientSecret, String tenantId,
String subscriptionId) {
AzureProfile profile = new AzureProfile(tenantId, subscriptionId, AzureEnvironment.AZURE);
TokenCredential clientSecretCredential = new ClientSecretCredentialBuilder()
.clientId(clientId)
.clientSecret(clientSecret)
.tenantId(tenantId)
.authorityHost(profile.getEnvironment().getActiveDirectoryEndpoint())
.build();
return AzureResourceManager.configure()
.authenticate(clientSecretCredential, profile)
.withSubscription(subscriptionId);
}
I'm trying to debug my lambda_function.py in AWS.
It writes the logs to CloudWatch always but..
In some case (cannot understand which) of 'Internal Server Error' it doesnt write anything but only START and END records to CloudWatch, which makes impossible to understand the root cause of the failure.
Here is my code:
import json
import psycopg2
def lambda_handler(event, context):
try:
print('started')
s = psycopg2.__version__
print(s)
conn = psycopg2.connect(
user='pg_user',
password='*********',
host='pg_host',
port='5432',
database='dev_db'
)
cur = conn.cursor()
cur.execute("select count(1) q from keywords_to_scrape")
for q in cur:
print(f'q = {q}')
except Exception as e:
print(f'exception: {e} ')
finally:
print('returning result')
return {
'statusCode' : 200,
'body' : json.dumps(f'{s}')
}
and if to comment this part
.............
#conn = psycopg2.connect(
# user='pg_user',
# password='*********',
# host='pg_host',
# port='5432',
# database='dev_db'
#)
.............
then it perfectly writes to CloudWatch the lines "started", "exception" with clear exception message and finally returns 200 OK
But with the lines of connection to DB it just dies with 'Internal server error' and with no messages in CloudWatch.
Could you please advice how to track such failures?
You are hitting timeout error as according to your comment.
Task timed out after 3.01 seconds
A few things for you to try and check:
Make your Lambda Timeout longer. E.g. 10 seconds.
If your Lambda is still hitting timeout error after you longer your Lambda Timeout, then you might want to check your database connections to the database. E.g. Make sure your Lambda is placed in the same VPC as your database and your database security group enables traffic from your Lambda.
LOGGER.debug("Connection Status Disb isClosed = " + conn.isClosed());
// returns true.
crsDisbDetailstmp = DataAccess.getData("select 1 cnt from dual", conn, new String[] {});
crsDisbDetailstmp.first();
LOGGER.debug("crsDisbDetailstmp"+ crsDisbDetailstmp.getString("cnt"));
DataAccess.executeProc("PRC_MCLR_TRNPRCDTL", new String[]{strOrgId, strAccountid,strFixedRate ,strModifiers }, conn);
Exception occured while executing last statement i.e execute procedure.
Exception=com.ibm.websphere.ce.cm.ObjectClosedException: DSRA9110E: Connection is closed.
I search a lot on google it shows this exception is occurred because connection is closed also i checked with conn.isclosed() which return true..
But If connection is closed then how i am able to fire select queries???
Please help me to figure it out as i worked on JBOSS only and first time on Websphere
I'm trying to get all the users from my LDAP server, doing the search from the base, this is my code:
public LdapTemplate ldapTemplate() {
LdapContextSource ctxSrc = new LdapContextSource();
ctxSrc.setUrl("ldap://127.0.0.1:389/");
ctxSrc.setBase("dc=test,dc=com");
ctxSrc.setUserDn("admin");
ctxSrc.setPassword("password");
ctxSrc.afterPropertiesSet();
LdapTemplate lt = new LdapTemplate(ctxSrc);
return lt;
}
private LdapTemplate ldapTemplate = ldapTemplate();
public List<User> getAllUsers() {
LdapQuery query= query().base("").where("objectclass").is("user");
return ldapTemplate.search(query, new UserAttributesMapper());
}
This is the error:
10:07:09.406 [main] DEBUG o.s.l.c.s.AbstractContextSource - AuthenticationSource not set - using default implementation
10:07:09.413 [main] DEBUG o.s.l.c.s.AbstractContextSource - Not using LDAP pooling
10:07:09.416 [main] DEBUG o.s.l.c.s.AbstractContextSource - Trying provider Urls: ldap://127.0.0.1:389/dc=test,dc=com
10:07:09.548 [main] DEBUG o.s.l.c.s.AbstractContextSource - Got Ldap context on server 'ldap://127.0.0.1:389/dc=test,dc=com'
Exception in thread "main" org.springframework.ldap.PartialResultException: Unprocessed Continuation Reference(s); nested exception is javax.naming.PartialResultException: Unprocessed Continuation Reference(s); remaining name '/'
at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:216)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:385)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:309)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:616)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:586)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:1651)
at ldap.example.UserRepositoryImpl.getAllUsers(UserRepositoryImpl.java:81)
at ldap.example.test.LdapApp.main(LdapApp.java:23)
Caused by: javax.naming.PartialResultException: Unprocessed Continuation Reference(s); remaining name '/'
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2914)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2888)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.getNextBatch(AbstractLdapNamingEnumeration.java:148)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMoreImpl(AbstractLdapNamingEnumeration.java:217)
at com.sun.jndi.ldap.AbstractLdapNamingEnumeration.hasMore(AbstractLdapNamingEnumeration.java:189)
at org.springframework.ldap.core.LdapTemplate.search(LdapTemplate.java:365)
... 6 more
BUILD FAILED (total time: 1 second)
When I filter by ou it works, but I need to filter from the root.
You write in question's comment that changing port helps.
But changing port doesn't solve this problem.
Port 3268 points to Active Directory special place - Global Catalog. There is set of all object - but each of them has only small subset of attributes (for example distinguishedName, cn, sAMAccountName...).
So - it works until you don't need more specific attributes.
Problem analysis
The exception occurs because AD, as the result of your query, returns referral objects:
[Active Directory] (...) generate referrals in response to queries that request data about objects that exist in the forest, but not contained on the directory server handling the request. These are called internal cross references, because they refer to domains, schema, and configuration containers within the forest.
And if referral chasing is disabled:
If referral chasing is not enabled and a subtree search is performed, the search will return all objects within the specified domain that meet the search criteria. The search will also return referrals to any subordinate domains that are direct descendants of the directory server domain. The client must resolve the referrals by binding to the path specified by the referral and submitting another query.
You can enable referral chasing, but it cost - it slow down application - you can read about this here. And I think it is not necessary in most cases.
Solution 1:
Sometimes the sufficient solution is to assign more specific baseDN - ctxSrc.setBase() method in your question. Maybe all your users are inside inner path e.g "ou=user,dc=department,dc=test,dc=com".
Read more in this answer.
Solution 2:
In Spring LdapTemplate you can also ignore this exception with method setIgnorePartialResultException():
ldapTemplate.setIgnorePartialResultException(true);
Read more in this answer.
I have a 4 node elasticsearch cluster. I have a .net console application that is designed to fill the cluster with data which comes from sql. Everything works fine as long as I keep the rate of records being added (or deleted) fairly low. If I increase the number of threads eventually I will see timeout errors from my console app. The cluster has a total of 48 cores and the average time it takes to index a record is about .1 seconds.
I have been able to get it to do about 7000 records (documents) per second. I never see any exceptions thrown from elasticsearch.net that indicate low resources. I never see any of the indexing queues overloaded. The servers never peak to more than about 10% cpu. It looks like the issue is not the cluster or it's configuration but something in the nest connection. Here is my code for the connection:
//set up the es client
Uri node = new Uri(ConfigurationManager.AppSettings["ESConnectionString"]);
var connectionPool = new SniffingConnectionPool(new[] { node });
ConnectionSettings settings = new ConnectionSettings(connectionPool);
settings.SetDefaultPropertyNameInferrer(p => p); //ditch the camelcase
settings.SniffOnConnectionFault(true);
settings.SniffOnStartup(true);
settings.SniffLifeSpan(TimeSpan.FromMinutes(1));
settings.SetPingTimeout(3000);
settings.SetTimeout(5000);
settings.MaximumRetries(5);
//settings.SetMaximumAsyncConnections(20);
settings.SetDefaultIndex("dummyindex");
settings.SetBasicAuthentication(ConfigurationManager.AppSettings["ESUser"], ConfigurationManager.AppSettings["ESPass"]);
ElasticClient client = new ElasticClient(settings);
I have the cluster set up with http.basic authentication, but I have tried with it turned on and off and there is no difference.
Here are some of the pertinent settings from the ES nodes:
discovery.zen.minimum_master_nodes: 2
discovery.zen.fd.ping_timeout: 30s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["CACHE01","CACHE02","CACHE03","CACHE04"]
cluster.routing.allocation.node_concurrent_recoveries: 5
indices.recovery.max_bytes_per_sec: 50mb
http.basic.enabled: true
http.basic.user: "admin"
http.basic.password: "XXXXXXX"
At this point I can't seem to figure out if it's the .Net client that is the issue or the servers? Everything points to the client but I'm at a loss for what to try next.
I don't think I can use the BulkAPI because I'm essentially just replicating changes from a SQL server and in order to keep them in sync I execute the change as soon as it's received.
It seems when I'm inserting new documents I can go at a much faster pace then when updating. I have read the updating docs and it almost reads like partial updates are better than full updates, but the there is the whole get-update-delete-reindex things that seems to happen with every update.
According to the es docs I'm not supposed to tweak the thread pools or the performance settings. I don't think I'm hitting any of those limits anyway. The ES error logs don't indicate any issue either.
Anyone have advice on what I can do to track down the connection errors?
UPDATE:
This is the actual error:
Error: Unexpected result (SaveToES). Elasticsearch.Net.Exceptions.MaxRetryException: Sniffing known nodes in the cluster caused a maxretry exception of its own ---> Elasticsearch.Net.Exceptions.SniffException: Sniffing known nodes in the cluster caused a maxretry exception of its own ---> Elasticsearch.Net.Exceptions.MaxRetryException: Retry timeout 00:00:05 was hit after retrying 1 times: 'GET _nodes/_all/clear?timeout=3000'.
InnerException: WebException, InnerMessage: The operation has timed out, InnerStackTrace: at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(HttpWebRequest request, Byte[] data, IRequestConfiguration requestSpecificConfig)
InnerException: WebException, InnerMessage: The operation has timed out, InnerStackTrace: at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(HttpWebRequest request, Byte[] data, IRequestConfiguration requestSpecificConfig) ---> System.AggregateException: One or more errors occurred. ---> System.Net.WebException: The operation has timed out
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.Connection.HttpConnection.DoSynchronousRequest(HttpWebRequest request, Byte[] data, IRequestConfiguration requestSpecificConfig)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandlerBase.ThrowMaxRetryExceptionWhenNeeded[T](TransportRequestState1 requestState, Int32 maxRetries)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.Request[T](TransportRequestState1 requestState, Object data)
at Elasticsearch.Net.Connection.Transport.Elasticsearch.Net.Connection.ITransportDelegator.Sniff(ITransportRequestState ownerState)
--- End of inner exception stack trace ---
--- End of inner exception stack trace ---
at Elasticsearch.Net.Connection.Transport.Elasticsearch.Net.Connection.ITransportDelegator.Sniff(ITransportRequestState ownerState)
at Elasticsearch.Net.Connection.Transport.Elasticsearch.Net.Connection.ITransportDelegator.SniffClusterState(ITransportRequestState requestState)
at Elasticsearch.Net.Connection.Transport.Elasticsearch.Net.Connection.ITransportDelegator.SniffOnConnectionFailure(ITransportRequestState requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.RetryRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.DoRequest[T](TransportRequestState1 requestState)
at Elasticsearch.Net.Connection.RequestHandlers.RequestHandler.Request[T](TransportRequestState1 requestState, Object data)
at Elasticsearch.Net.Connection.Transport.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticsearchClient.DoRequest[T](String method, String path, Object data, IRequestParameters requestParameters)
at Elasticsearch.Net.ElasticsearchClient.IndicesCreatePost[T](String index, Object body, Func2 requestParameters)
at Nest.RawDispatch.IndicesCreateDispatch[T](ElasticsearchPathInfo1 pathInfo, Object body)
at Nest.ElasticClient.<CreateIndex>b__281_0(ElasticsearchPathInfo1 p, ICreateIndexRequest d)
at Nest.ElasticClient.Nest.IHighLevelToLowLevelDispatcher.Dispatch[D,Q,R](D descriptor, Func3 dispatch)
at Nest.ElasticClient.CreateIndex(Func2 createIndexSelector)
at DCSCache.esvRepository.CreateIndex(String IndexName, String IndexVersion)
at DCSCache.esvRepository.Save(esv ItemToSave, String IndexName, String IndexVersion)