So, I have been struggling with a problem in the last couple of days for which I could really use come guidance.
Short version, I am trying to create a AVRO Reader Controller service in NiFi, using the nipyapi library.
Now, based on the documentation, we have:
nipyapi.canvas.create_controller(parent_pg, controller, name=None)
parent_pg = Target Parent PG
controller = Type of Controller to create, found via the list_all_controller_types method
name = Name for the new Controller as a String (Optional)
First of all I have tried to retrieve the list of all controller types and the following is the output for the controller I am trying to create:
{'bundle': {'artifact': 'nifi-record-serialization-services-nar',
'group': 'org.apache.nifi',
'version': '1.15.3'},
'controller_service_apis': [{'bundle': {'artifact': 'nifi-standard-services-api-nar',
'group': 'org.apache.nifi',
'version': '1.15.3'},
'type': 'org.apache.nifi.serialization.RecordReaderFactory'}],
'deprecation_reason': None,
'description': 'Parses Avro data and returns each Avro record as an separate '
'Record object. The Avro data may contain the schema itself, '
'or the schema can be externalized and accessed by one of the '
"methods offered by the 'Schema Access Strategy' property.",
'explicit_restrictions': None,
'restricted': False,
'tags': ['comma',
'reader',
'record',
'values',
'delimited',
'separated',
'parse',
'row',
'avro'],
'type': 'org.apache.nifi.avro.AvroReader',
'usage_restriction': None}
With this information, based on the documentation, I have tried to create this controller service:
root_id = canvas.get_root_pg_id() # retrieve the parent process group where I want to create the controller service
nipyapi.canvas.create_controller(root_id, 'org.apache.nifi.avro.AvroReader' , name="[TEST][PROJECT1] AVRO Reader - Table 1")
Unfortunately, I keep on getting an AssertionError within the create_controller function for "assert isinstance(controller, nipyapi.nifi.DocumentedTypeDTO) at line 1087"
Has anyone else encountered a similar issue?
And if so, how did you manage to fix it?
Or am I calling the create_controller function with invalid parameters?
Thank you.
Related
I work at the moment on a personal project. I'm using React.js for the front-end and Express.js and PostgreSQL for the back-end.
When I'm trying to register an user in the database, I want the details to be written in 2 tables: 'users'(for generic user details) and 'login'(for keeping user's login details). On the server side I'm trying to achieve this with EXPRESS.js and KNEX query builder npm package to connect to PostgreSQL database. The following code should give a better picture of things:
db.transaction(trx => {
trx.insert({
email: email,
hash: hash
})
.into('login')
.returning('email')
.then(loginEmail => {
return trx('users')
.returning('*')
.insert({
email: loginEmail[0].email,
name: name,
joined: new Date()
})
.then(user => {
res.json(user[0]);
})
})
.then(trx.commit)
.catch(trx.rollback)
})
.catch(err => res.status(400).json(err));
});
After running the app and trying to register an user I recieve the following error back:
{"length":104,"name":"error","severity":"ERROR","code":"42P01","position":"13","file":"parse_relation.c","line":"1384","routine":"parserOpenTable"}
I've checked this error online and got that the following issue is coming up only when a relation doesn't exist in the database at all it could be a mixed-case spelling issue when I created the 'login' table. I dropped the database and created a new one considering the solutions found on the internet and the app is returning the same error.
If anyone encountered with such error or similar case, the help would be very much appreciated!
I'm trying to set up a simple Batch Compute Environment using a LaunchTemplate, so that I can specify a larger-than-default volume size:
const templateName = 'my-template'
const jobLaunchTemplate = new ec2.LaunchTemplate(stack, 'Template', {
launchTemplateName: templateName,
blockDevices: [ ..vol config .. ]
})
const computeEnv = new batch.CfnComputeEnvironment(stack, 'CompEnvironment', {
type: 'managed',
computeResources: {
instanceRole: jobRole.roleName,
instanceTypes: [
InstanceType.of(InstanceClass.C4, InstanceSize.LARGE).toString()
],
maxvCpus: 64,
minvCpus: 0,
desiredvCpus: 0,
subnets: vpc.publicSubnets.map(sn => sn.subnetId),
securityGroupIds: [vpc.vpcDefaultSecurityGroup],
type: 'EC2',
launchTemplate: {
launchTemplateName: templateName,
}
},
})
They both initialize fine when not linked, however as soon as the launchTemplate block is added to the compute environment, I get the following error:
Error: Resource handler returned message: "Resource of type 'AWS::Batch::ComputeEnvironment' with identifier 'compute-env-arn' did not stabilize." (RequestToken: token, HandlerErrorCode: NotStabilized)
Any suggestions are greatly appreciated, thanks in advance!
For anyone running into this - check the resource that is being created in the AWS Console - i.e go to aws.amazon.com and refresh the page over and over until you see it created by CF. This gave me a different error message regarding the instance profile not existing (A bit more helpful than the terminal error...)
A simple CfnInstanceProfile did the trick:
new iam.CfnInstanceProfile(stack, "batchInstanceProfile", {
instanceProfileName: jobRole.roleName,
roles: [jobRole.roleName],
});
I faced similar error.
But in my case cdk had created subnetGroups list in cdk.context.json and was trying to use the same in the CfnComputeEnvironment definition.
The problem was; I was using the default vpc and had manually modified few subnets. and cdk.context.json was not updated.
Solved by deleting the cdk.context.json
This file was recreated with correct values in next synth.
Tip for others facing similar problem:
Don't just rely on the error message; watch closely the Cloud-formation Script that's generated from CDK for the resource.
I am using PyDrive to create a Google Sheets file on a Google Shared Drive, the below code snippet successfully creates the file in my shared drive folder:
f = gd.CreateFile(
{'title': name, 'mimeType': 'application/vnd.google-apps.spreadsheet', 'parents': [{'teamDriveId': '1234', 'id': '1234'}]})
f.Upload(param={'supportsTeamDrives': True})
On adding the file, I am also trying to set permissions for an email address to gain write access to the file as below:
f.InsertPermission({'type': 'user', 'value': 'myemail#email.com', 'role': 'writer'})
On attempting to add the permissions, I receive the below error:
<HttpError 404 when requesting https://www.googleapis.com/drive/v2/files/file_id_here/permissions?alt=json returned "File not found: file_id_here">
I have checked and the file id appears to be the same as the one that has been created.
I assumed that it wouldn't be any authorisation issue if I am able to create the file? Any suggestions would be very much appreciated.
Thanks
I have found it appears to be an issue with no entry for 'supportsTeamDrives' in the PyDrive's files.py file in the InsertPermission function.
I added the 'supportsTeamDrives' parameter into the code at line 325 as follows and it now appears to work:
permission = self.auth.service.permissions().insert(
fileId=file_id, body=new_permission, supportsTeamDrives=True).execute(http=self.http)
The best way I found using pyDrive was doing this:
file = gd.CreateFile({'title': 'filename', 'parents': [{'teamDriveId': '<your teamDriveId>', 'id': '<file id>'}]})
file.Upload(param={'supportsTeamDrives': True})
# here we can use the pydrive object to update.
new_permission = {
'id': 'anyoneWithLink',
'type': 'anyone',
'value': 'anyoneWithLink',
'withLink': True,
'role': 'reader'
}
permission = file.auth.service.permissions().insert(
fileId=file['id'], body=new_permission, supportsTeamDrives=True).execute(http=file.http)
I am following Google's tutorial on setting up Google Cloud endpoint (not AWS API Gateway) in front of my Cloud Function. I am triggering my Cloud Function to trigger an AWS lambda function, AND I am trying to pass a path parameter from my Endpoint as defined by OpenAPI spec.
Path parameters are variable parts of a URL path. They are typically used to point to a specific resource within a collection, such as a user identified by ID. A URL can have several path parameters, each denoted with curly braces { }.
paths: /users/{id}:
get:
parameters:
- in: path
name: id # Note the name is the same as in the path
required: true
schema:
type: integer
GET /users/{id}
My openapi.yaml
swagger: '2.0'
info:
title: Cloud Endpoints + GCF
description: Sample API on Cloud Endpoints with a Google Cloud Functions backend
version: 1.0.0
host: HOST
x-google-endpoints:
- name: "HOST"
allowCors: "true
schemes:
- https
produces:
- application/json
paths:
/function1/{pathParameters}:
get:
operationId: function1
parameters:
- in: path
name: pathParameters
required: true
type: string
x-google-backend:
address: https://REGION-FUNCTIONS_PROJECT_ID.cloudfunctions.net/function1
responses:
'200':
description: A successful response
schema:
type: string
The error I get when I use Endpoint URL https://REGION-FUNCTIONS_PROJECT_ID.cloudfunctions.net/function1/conversations is a TypeError from my AWS lambda function
StatusCode:200, FunctionError: "Unhandled", ExecutedVersion: "$LATEST". Payload: "errorType":"TypeError", errorMessage:"Cannot read property 'startsWith' of undefined..."
It is saying that on line
var path = event.pathParameters;
...
...
if (path.startsWith('conversations/'){...};
my path var is undefined.
I initially thought my Google Function was not correctly passing pathParameters but when I tested my Google function using triggering event {"pathParameters":"conversations"}, my Lambda returns the payload successfully.
My Google Cloud Function:
let AWS = require('aws-sdk');
AWS.config.update({
accessKeyId: 'key',
secretAccessKey: 'secret',
region: 'region'
})
let lambda = new AWS.Lambda();
exports.helloWorld = async(req,res) => {
let params = {
FunctionName:'lambdafunction',
InvocationType: "RequestRespone",
Payload: JSON.stringify(req.body)
};
res.status(200).send(await lambda.invoke(params, function(err,data){
if(err){throw err}
else{
return data.Payload
}
}).promise());
}
EDIT 1:
Seeing this Google Group post, I tried adding to my openapi.yaml file path_translation: APPEND_PATH_TO_ADDRESS, yet still I'm having no luck.
...
paths:
/{pathParameters}:
get:
...
x-google-backend:
address: https://tomy.cloudfunctions.net/function-Name
path_translation: APPEND_PATH_TO_ADDRESS
#Arunmainthan Kamalanathan mentioned in the comments that testing in AWS and Google Cloud directly with trigger event {"pathParameters":"conversations"} is not equivalent to passing req.body from my Google function to AWS lambda. I think this is where my error is occurring -- I'm not correctly passing my path parameter in the payload.
EDIT 2:
There is this Stackoverflow post concerning passing route parameters to Cloud Functions using req.path. When I console.log(req.path) I get / and console.log(req.params) I get {'0': '' }, so for some reason my path parameter is not getting passed correctly from Cloud Endpoint URL to my Google function.
I was running into the same issue when specifying multiple paths/routes within my openapi.yaml. It all depends on where you place the x-google-backend (top-level versus operation-level). This has implications on the behaviour of the path_translation. You could also overwrite path_translation yourself, as the documentation clearly describes:
path_translation: [ APPEND_PATH_TO_ADDRESS | CONSTANT_ADDRESS ]
Optional. Sets the path translation strategy used by ESP when making
target backend requests.
Note: When x-google-backend is used at the top level of the OpenAPI
specification, path_translation defaults to APPEND_PATH_TO_ADDRESS,
and when x-google-backend is used at the operation level of the
OpenAPI specification, path_translation defaults to CONSTANT_ADDRESS.
For more details on path translation, please see the Understanding
path translation section.
This means that if you want the path to be appended as a path parameter instead of a query parameter in your backend, you should adhere to the following scenario's:
Scenario 1: Do you have one cloud function in the x-google-backend.address that handles all of your paths in the openapi specification? Put x-google-backend at the top-level.
Scenario 2: Do you have multiple cloud functions corresponding to different paths? Put x-google-backend at the operation-level and set x-google-backend.path_translation to APPEND_PATH_TO_ADDRESS.
When your invocation type is RequestRespone, you can access the payload directly from the event parameter of lambda.
Look at the `Payload' parameter of the Google function:
let params = {
FunctionName:'lambdafunction',
InvocationType: "RequestRespone",
Payload: JSON.stringify({ name: 'Arun'})
};
res.status(200).send(await lambda.invoke(params)...)
Also the Lambda part:
exports.handler = function(event, context) {
context.succeed('Hello ' + event.name);
};
I hope this helps.
AWS-loaded question, but does anyone know what the proper way to access an Athena Query String (in CloudFormation) in Lambda?
I have set up both the Athena NamedQuery and the Lambda in CloudFormation. Abstracting out some of the more project-specific details, the general form I have is:
MyQuery:
Type: AWS::Athena::NamedQuery
Properties:
Database: "mydatabase"
Name: "DataQuery"
QueryString: SELECT * FROM mydatabase
MyLambda:
Type: AWS::Serverless::Function
Properties:
Handler: 'handlers.migration_handler'
Runtime: python3.6
CodeUri:
Bucket: BATS::SAM::CodeS3Bucket
Key: BATS::SAM::CodeS3Key
Description: Migrates data from output of Athena query to S3
Policies:
- AmazonS3FullAccess
- AWSLambdaExecute
- AmazonAthenaFullAccess
Environment:
Variables:
MY_QUERY:
Ref: MyQuery
When I'm writing the handler for the lambda, I want to call:
athena_client = boto3.client('athena')
response = athena_client.start_query_execution(
QueryString = os.environ['MY_QUERY']
ResultConfiguration = {'OutputLocation: 's3://my-bucket'}
)
However, QueryString needs to be a string, so this currently isn't working. I want to access the QueryString property in MY_QUERY, and I feel like I'm so close but I'm not quite sure how to get that last step. Any help here would be greatly appreciated.
Figured it out yesterday (or more specifically, my teammate figured it out).
Boto3 happens to have another method called get_named_query(NamedQueryId), and that returns a dictionary in the form of:
{
'NamedQuery': {
'Name': 'string',
'Description': 'string',
'Database': 'string',
'QueryString': 'string',
'NamedQueryId': 'string'
}
Thus, my code worked when I modified my lambda handler to:
athena_client = boto3.client('athena')
query_info = athena_client.get_named_query(
NamedQueryId = os.environ['MY_QUERY']
)
response = athena_client.start_query_execution(
QueryString = query_info['NamedQuery']['QueryString']
ResultConfiguration = {'OutputLocation: 's3://my-bucket'}
)