Using AWSMibileHub to set up my backend. Working on an admin page that shows all the users from AWS Cognito.
I’m using ListUsers to get user list and noticed 502 error on the request with a PagenationToken and it seems to be happening randomly.
In CloudWatch shows this error below.
InvalidParameterException: 1 validation error detected: Value 'CAISlAIIARLtAQgDEugBADSdLB5dZXQEaQjoL8y8CE1RGGT3PZ4FpCqxFwJkNuhIeyJAbiI6IlBhZ2luYXRpb25Db250aW51YXRpb25EVE8iLCJuZXh0S2V5IjoiQUFBQUFBQUFCZHM4QVFFQmJBZmU5OFgzUnJTM1BHcnYzVmRiVWNScVdIck82VXFZaTdlZklleVRCSEZsYm1ZN01UVTRPV1k0TVRNdE9UYzVZUzAwWTJKakxUazROekl0TkdRek9UYzROMlpoWmpVM093PT0iLCJwcmV2aW91c1JlcXVlc3RUaW1lIjoxNTQzMDkzODIyNDk5fRogNk7FThSKBOuwQGi DoZBnmNN85UY5oFiSAbHWfOzreY=' at 'paginationToken' failed to satisfy constraint: Member must satisfy regular expression pattern: [\S]+
Does anybody have the same issue or any idea how to fix this issue?
Store the pagination Token in a variable and then add this .replace(/\ /g,'+') to the variable to change the space back to +.
e.g:
const pagination = 'TOKEN FROM COGNITO'.replace(/\ /g,'+')
The pagination_token consists of non alphanumeric characters, so its better to encrypt and decrypt when required.
Encrypt?decrypt utility file
const crypto = require("crypto");
const algorithm = "aes-256-ctr";
const password = "sassed";
class Util {
static encrypt_aes(text) {
let cipher = crypto.createCipher(algorithm, password);
let crypted = cipher.update(text, "utf8", "hex");
crypted += cipher.final("hex");
return crypted;
}
static decrypt_aes(text) {
let decipher = crypto.createDecipher(algorithm, password);
let dec = decipher.update(text, "hex", "utf8");
dec += decipher.final("utf8");
return dec;
}
}
module.exports = Util;
PaginationToken encryption/decription
const Util = require("./utils");
class Users {
static async getUsersAttributes(params) {
let listparams = {
UserPoolId: userPoolID /* required */,
Limit: params.limit || 10,
};
if (params.paginationToken) {
listparams.PaginationToken = Util.decrypt_aes(params.paginationToken);
}
let response = await cognitoISPClient.listUsers(listparams).promise();
response.PaginationToken = Util.encrypt_aes(response.PaginationToken);
return response;
}
}
Related
I am in the need of listing the users data belonging to a specific group within the organization. The documentation does not specify if this is possible. I was really hoping there could be some kind of query that would allow this. For example email in (1#domain.com,2#domain.com). However, I don't see that being possible. The only way I could think to accomplish this would be:
Get a list of all the members in the group (https://developers.google.com/admin-sdk/directory/reference/rest/v1/members/list)
Get each user data by email (https://developers.google.com/admin-sdk/directory/reference/rest/v1/users/get)
The problem with the above approach is that if a group contains 50+ members, this means that I have to make all that amount of requests, which is counter productive. Imagine how long that would take.
Any ideas? Greatly appreciate it.
Unfortunately I don’t think you can skip this two step process, but you can speed it up using batch requests. This
allows you to request up to 1000 calls in a single request. The steps would be:
Make a batch request to get all the members of all the groups you want (using members.list).
Make a batch request to get all the user info that you need using their id (using user.get).
Notice that the data in the result won’t be sorted, but they will be tagged by Content-ID.
References
Sending Batch Requests (Directory API)
Method: members.list (Directory API)
Method: users.get (Directory API)
I thought about the batching request a couple of hours after I posted the question. The problem with Node JS is that it does not has built in support for batch requests, unlike the php client library for example; Therefore, I had to spent some time implementing support for it on my own since I was not able to find any example. I'll share the solution in case it helps someone else or for my future reference.
async function getGroupMembersData(){
const groupEmail = "group#domain.com"; //google group email
const groupMembers = await getGroupMembers(groupEmail).catch(error=>{
console.error(`Error querying group members: ${error.toString()}`);
});
if(!groupMembers){ return; }
const url = "https://www.googleapis.com/batch/admin/directory_v1";
const scopes = ["https://www.googleapis.com/auth/admin.directory.user.readonly"];
const requests = [];
for(let i=0; i<groupMembers.length; ++i){
const user = groupMembers[i];
const request = {
email: user,
endpoint: `GET directory_v1/admin/directory/v1/users/${user}?fields=*`
};
requests.push(request);
}
const batchRequestData = await batchProcess(url, scopes, requests).catch(error=>{
console.error(`Error processing batch request: ${error.toString()}`);
});
if(!batchRequestData){ return; }
const usersList = batchRequestData.map(i=>{
return i.responseBody;
});
console.log(usersList);
}
//get group members using group email address
async function getGroupMembers(groupKey){
const client = await getClient(scopes); //function to get an authorized client, you have to implement on your own
const service = google.admin({version: "directory_v1", auth: client});
const request = await service.members.list({
groupKey,
fields: "members(email)",
maxResults: 200
});
const members = !!request.data.members ? request.data.members.map(i=>i.email) : [];
return members;
}
//batch request processing in groups of 100
async function batchProcess(batchUrl, scopes, requests){
const client = await getClient(scopes); //function to get an authorized client, you have to implement on your own
let results = [];
const boundary = "foobar99998888"; //boundary line definition
let batchBody = ""; const nl = "\n";
const batchLimit = 100; //define batch limit (max supported = 100)
const totalRounds = Math.ceil(requests.length / batchLimit);
let batchRound = 1;
let batchItem = 0;
let roundLimit = batchLimit;
do{
roundLimit = roundLimit < requests.length ? roundLimit : requests.length;
//build the batch request body
for(batchItem; batchItem<roundLimit; batchItem++){
const requestData = requests[batchItem];
batchBody += `--${boundary}${nl}`;
batchBody += `Content-Type: application/http${nl}`;
batchBody += `Content-Id: <myapprequest-${requestData.email}>${nl}${nl}`;
batchBody += `${requestData.endpoint}${nl}`;
}
batchBody += `--${boundary}--`;
//send the batch request
const batchRequest = await client.request({
url: batchUrl,
method: "POST",
headers: {
"Content-Type": `multipart/mixed; boundary=${boundary}`
},
body: batchBody
}).catch(error=>{
console.log("Error processing batch request: " + error);
});
//parse the batch request response
if(!!batchRequest){
const batchResponseData = batchRequest.data;
const responseBoundary = batchRequest.headers["content-type"].split("; ")[1].replace("boundary=", "");
const httpResponses = batchResponseParser(batchResponseData, responseBoundary);
results.push(...httpResponses);
}
batchRound++;
roundLimit += batchLimit;
} while(batchRound <= totalRounds);
return results;
};
//batch response parser
function batchResponseParser(data, boundary){
const nl = "\r\n";
data = data.replace(`--${boundary}--`,"");
const responses = data.split(`--${boundary}`);
responses.shift();
const formattedResponses = responses.map(i=>{
const parts = i.split(`${nl}${nl}`);
const responseMetaParts = (parts[0].replace(nl, "")).split(nl);
let responseMeta = {};
responseMetaParts.forEach(part=>{
const objectParts = part.split(":");
responseMeta[objectParts[0].trim()] = objectParts[1].trim();
});
const responseHeadersParts = parts[1].split(nl);
let responseHeaders = {};
responseHeadersParts.forEach(part=>{
if(part.indexOf("HTTP/1.1") > -1){
responseHeaders.status = part;
} else {
const objectParts = part.split(":");
responseHeaders[objectParts[0].trim()] = objectParts[1].trim();
}
});
const reg = new RegExp(`${nl}`, "g");
const responseBody = JSON.parse(parts[2].replace(reg, ""));
const formatted = {
responseMeta: responseMeta,
responseHeaders: responseHeaders,
responseBody: responseBody
};
return formatted;
});
return formattedResponses;
}
I am hoping there is someone here who is familiar with a Real Estate data standard known as RETS. The National Association of Realtors provides a dll for interfacing with their services called libRETS, but it is not being supported like it once was and recent events have prompted us to create our own as a replacement. For logistics reasons, we can't do this in Core and are using the current C#.Net 4.7.2.
There are 2 or 3 different "security levels" for connecting to a RETS Server, with the method being a per case basis from one MLS to the next. We can successfully connect to those who only require a login and password, but are hitting a wall on those who also require what is called a UserAgent and UserAgentPassword, which must passed somehow using Md5 encryption. The server is returning:
The remote server returned an error: (401) Unauthorized.
private WebResponse GetLoginBasicResponse()//*** THIS ONE WORKS ***
{
try
{
ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var request = (HttpWebRequest)WebRequest.Create(new Uri(_cred.loginUri));
request.Method = "GET";
request.Headers.Add("RETS-Version", _retsVersion);
request.Credentials = new NetworkCredential(_login, _password);
return request.GetResponse();
}
catch (Exception ex)
{
string ignore = ex.Message;
return null;
}
}
private WebResponse GetLoginWithUserAgentResponse()//*** THIS ONE DOES NOT WORK ***
{
try
{
// ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
var request = (HttpWebRequest)WebRequest.Create(new Uri(_cred.loginUri));
request.Method = "GET";
request.Headers.Add("RETS-Version", _retsVersion);
if (!string.IsNullOrEmpty(_cred.userAgent))
{
request.UserAgent = Md5(_cred.userAgent + ":" + _cred.userAgentPassword);
//request.Headers.Add("RETS-UA-Authorization", "Digest " + Md5(_cred.userAgent + ":" + _cred.userAgentPassword));
}
request.Credentials = new NetworkCredential(_login, _password);
return request.GetResponse();
}
catch (Exception ex)
{
string ignore = ex.Message;
return null;
}
}
public string Md5(string input) //*** Borrowed this from from .NET Core Project and presume it works
{
// Use input string to calculate MD5 hash
using (System.Security.Cryptography.MD5 md5 = System.Security.Cryptography.MD5.Create())
{
byte[] inputBytes = Encoding.ASCII.GetBytes(input);
byte[] hashBytes = md5.ComputeHash(inputBytes);
// Convert the byte array to hexadecimal string
StringBuilder sb = new StringBuilder();
for (int i = 0; i < hashBytes.Length; i++)
{
sb.Append(hashBytes[i].ToString("X2"));
}
return sb.ToString();
}
}
Page 20 of this document describes how to build the UA header: https://www.ranww.org/documents/resources/rets_1_8.pdf
There’s a few other fields you need to include.
We were not able to solve the issue in .NET but found a .NET Core project in GitHub that we are using instead. https://github.com/CrestApps/RetsConnector
This case can be closed
Not seeing an option to "Mark as Answer". Have tried both MS Edge and Google Chrome
Does anybody know how to get the serial number of a device?
I know I can do this in a NativeScript-5 application (TypeScript):
import { Page } from "tns-core-modules/ui/page";
import * as platformModule from "tns-core-modules/platform";
export function onNavigatingTo(args: EventData) {
let page = <Page>args.object;
console.log("Manufacturer:" + platformModule.device.manufacturer);
console.log("Model:" + platformModule.device.model);
console.log("UUID:" + platformModule.device.uuid);
}
But I couldnt find any property for the device's serial number.
Anybody an idea? It's alright if your solution only covers Android (since my project is targeting Android tablets only).
Thanks!
Update:
Manoj pointed me to some Java code that probably solves my problem. However, I wasn't able to marshal the following code to TypeScript.
public static String getManufacturerSerialNumber() {
String serial = null;
try {
Class<?> c = Class.forName("android.os.SystemProperties");
Method get = c.getMethod("get", String.class, String.class);
serial = (String) get.invoke(c, "ril.serialnumber", "unknown");
} catch (Exception ignored) {}
return serial;
}
Maybe anybody could help me? That would be awesome!
iOS never allows any confidential information to be accessed by apps, that applies to serial number too. With Android, you may read the serial number with
android.os.Build.SERIAL
I had it tested with Playground
Edit 1:
Manufactures like Samsung seems to have a different serial number which is not same as android.os.Build.SERIAL. Here is another SO thread that seems very relevant for Samsung, the sample code is written in Java.
Edit 2:
Here is how you may get serial number on Samsung devices
let serialNumber = '';
try {
const cl = application.android.context.getClassLoader();
const SystemProperties = cl.loadClass('android.os.SystemProperties');
const paramTypes = (<any>Array).create(java.lang.Class, 2);
paramTypes[0] = java.lang.String.class;
paramTypes[1] = java.lang.String.class;
const getMethod = SystemProperties.getMethod('get', paramTypes);
const params = (<any>Array).create(java.lang.Object, 2);
params[0] = new java.lang.String('ril.serialnumber');
params[1] = new java.lang.String('unknown');
serialNumber = getMethod.invoke(SystemProperties, params);
} catch (err) {
console.error(err);
}
Updated Playground
You can check properties 'ro.serialno', 'ril.serialnumber', 'gsm.sn1', 'sys.serialnumber'
Android: How to programmatically access the device serial number shown in the AVD manager (API Version 8)
In my case Samsung TAB S2 has 2 serials. Property 'ro.serialno' more accurate then 'ril.serialnumber'. So i check it first.
getSerial() {
if (isAndroid) {
let serialNumber = null;
try {
const cl = application.android.context.getClassLoader();
const SystemProperties = cl.loadClass('android.os.SystemProperties');
const paramTypes = (<any>Array).create(java.lang.Class, 2);
paramTypes[0] = java.lang.String.class;
paramTypes[1] = java.lang.String.class;
const getMethod = SystemProperties.getMethod('get', paramTypes);
const params = (<any>Array).create(java.lang.Object, 2);
// Reorder if needed
const props = ['ro.serialno', 'ril.serialnumber', 'gsm.sn1', 'sys.serialnumber'];
for(let i = 0, len = props.length; i < len; i++) {
params[0] = new java.lang.String(props[i]);
serialNumber = getMethod.invoke(SystemProperties, params);
if (serialNumber !== '') {
return serialNumber;
}
}
} catch (err) {
console.error(err);
}
return serialNumber;
}
}
Full code: https://play.nativescript.org/?template=play-tsc&id=ViNo6g&v=6
I've created a method to add members in a Batch Request to a google group using .NET core and google's .NET client library. The code looks like this:
private void InitializeGSuiteDirectoryService()
{
_directoryServiceCredential = GoogleCredential
.FromJson(GlobalSettings.Instance.GSuiteSettings.Credentials)
.CreateScoped(_scopes)
.CreateWithUser(GlobalSettings.Instance.GSuiteSettings.User);
_directoryService = new DirectoryService(new BaseClientService.Initializer()
{
HttpClientInitializer = _directoryServiceCredential,
ApplicationName = _applicationName
});
}
public OperationResult<int> AddGroupMembers(Group group, IEnumerable<Member> members)
{
var result = new OperationResult<int>();
var memberList = members.ToList();
var batchRequestCount = 0;
if (memberList.Any())
{
var request = new BatchRequest(_directoryService);
foreach (var member in memberList)
{
batchRequestCount++;
request.Queue<Members>(_directoryService.Members.Insert(member, group.Id), (content, error, i, message) =>
{
if (message.IsSuccessStatusCode)
{
//log OK
}
else
{
// Implement Exponential backoff only on the request that failed.
}
});
if (batchRequestCount == 30|| member.Equals(memberList.Last()))
{
request.ExecuteAsync().Wait();
request = new BatchRequest(_directoryService); //Clear queue
}
}
}
return result;
}
The logic works fine if the amount of members is small; however, when the members count is let's say 100( this is the max amount of users in my google's test instance), I get an Error from Google that reads: "quotaExceeded". According to Google's documentation, the limit for a batch request on their Admin SDK is 1000 and I've set my logic to Execute when we reach a limit of 30.
The QUESTION is: How do I implement error handling to retry whenever I get this error? Their documentation suggests implementing 'Exponential Backoff' with a response that contains a 'retry-able error'(I don't see this when I inspect my response).
So here's what I ended up doing to implement Exponential Backoff on my call to add members to a Gsuite group. Since I'm using dotnet core, I was able to use 'Polly', which is a resilience and transient-fault-handling library that offers this functionality out of the box. There may be some need for refactoring, but here's what the code looks like for now:
public OperationResult<int> AddGroupMembers(Group group, IEnumerable<Member> members)
{
var result = new OperationResult<int>();
var memberList = members.ToList();
var batchRequestCount = 0;
if (memberList.Any())
{
var request = new BatchRequest(_directoryService);
foreach (var member in memberList)
{
retryRequest = false; // This variable needs to be declared at the class level to guarantee the value is available to the original thread running the process.
batchRequestCount++;
request.Queue<Members>(_directoryService.Members.Insert(member, group.Id), (content, error, i, message) =>
{
// If error code is 'quotaExceeded' retry the request ( You can add as many error codes as you'd like to retry here)
if (error.Code == 403)
{
retryRequest = true;
}
});
// Execute batch request to add members in batches of 30 member max
if (batchRequestCount == 30|| member.Equals(memberList.Last()))
{
// Below is what the code to retry using polly looks like
var response = Policy
.HandleResult<HttpResponseMessage>(message => message.StatusCode == HttpStatusCode.Conflict)
.WaitAndRetry(new[]
{
TimeSpan.FromSeconds(1),
TimeSpan.FromSeconds(2),
TimeSpan.FromSeconds(4)
}, (results, timeSpan, retryCount, context) =>
{
// Log Warn saying a retry was required.
})
.Execute(() =>
{
var httpResponseMsg = new HttpResponseMessage();
// Execute batch request Synchronously
request.ExecuteAsync().Wait();
if (retryRequest)
{
httpResponseMsg.StatusCode = HttpStatusCode.Conflict;
retryRequest = false;
}
else
{
httpResponseMsg.StatusCode = HttpStatusCode.OK;
}
return httpResponseMsg;
});
if (response.IsSuccessStatusCode)
{
// Log info
}
else
{
// Log warn
}
requestCount = 0;
request = new BatchRequest(_directoryService);
batchCompletedCount++;
}
}
}
return result;
}
I'm looking at the Microsoft-provided sample "Process Tasks as they Finish" and adapting that TPL sample for Azure Storage.
The problem I have is marked below where the variable domainData reports the errors in the compiler: Unknown method Select(?) of TableQuerySegment<DynamicTableEntity> (fully qualified namespace removed)
I also get the following error DynamicTableEntity domainData \n\r Unknown type of variable domainData
/// if you have the necessary references the following most likely should compile and give you same error
CloudStorageAccount acct = CloudStorageAccount.DevelopmentStorageAccount;
CloudTableClient client = acct.CreateCloudTableClient();
CloudTable tableSymmetricKeys = client.GetTableReference("SymmetricKeys5");
TableContinuationToken token = new TableContinuationToken() { };
TableRequestOptions opt = new TableRequestOptions() { };
OperationContext ctx = new OperationContext() { ClientRequestID = "ID" };
CancellationToken cancelToken = new CancellationToken();
List<Task> taskList = new List<Task>();
var task2 = tableSymmetricKeys.CreateIfNotExistsAsync(cancelToken);
task2.Wait(cancelToken);
int depth = 3;
while (true)
{
Task<TableQuerySegment<DynamicTableEntity>> task3 = tableSymmetricKeys.ExecuteQuerySegmentedAsync(query, token, opt, ctx, cancelToken);
// Run the method
task3.Wait();
Console.WriteLine("Records retrieved in this attempt = " + task3.Result.Count());// + " | Total records retrieved = " + state.TotalEntitiesRetrieved);
// HELP! This is where I'm doing something the compiler doesn't like
//
IEnumerable<Task<int>> getTrustDataQuery =
from domainData in task3.Result select QueryPartnerForData(domainData, "yea, search for this.", client, cancelToken);
// Prepare for next iteration or quit
if (token == null)
{
break;
}
else
{
token = task3.Result.ContinuationToken;
// todo: persist token token.WriteXml()
}
}
//....
private static object QueryPartnerForData(DynamicTableEntity domainData, string p, CloudTableClient client, CancellationToken cancelToken)
{
throw new NotImplementedException();
}
Your code is missing a query. In order to test the code I created the following query:
TableQuery<DynamicTableEntity> query = new TableQuery<DynamicTableEntity>()
.Where(TableQuery.GenerateFilterCondition("PartitionKey", QueryComparisons.Equal, "temp"));
I also added the method QueryPartnerForData which doesn't do anything (simply returns null) and everything works fine. So maybe it's an issue with the QueryPartnerForData method? The best way to find the actual error is by setting a breakpoint here and there.
A StackOverflowException often means you are stuck in an endless loop. Run through the breakpoints a few times and see where your code is stuck. Could it be that QueryPartnerForData calls the other method and that the other method calls QueryPartnerForData again?