I am working with Apache Kafka to send messages to Kafka topics. I am trying to use unions in Avro Schemas including enum types for message validation. But I am facing an issue with the usage of enum types within union. I am using Kafka REST API through POSTMAN tool to post a record/message to a topic with schema validation. Below is the request payload including schema and records inline -
{
"key_schema": "{\"type\": \"record\", \"name\": \"key\", \"fields\": [{\"name\": \"keyInput\", \"type\": \"string\"}]}",
"value_schema": "{\"type\": \"record\", \"name\": \"value\", \"fields\": [{\"name\": \"valueInput1\", \"type\": \"string\"},{\"name\": \"valueInput2\",\"type\":[{\"type\":\"enum\",\"name\":\"actorobjType\",\"symbols\":[\"Agent\",\"Group\"]},\"null\"],\"default\":null}]}",
"records": [
{
"key": {
"keyInput": "testUser-key"
},
"value": {
"valueInput1": "testUser-value",
"valueInput2": "Agent"
}
}
]
}
I am getting the following error when I am trying to insert a record using above request payload -
{
"error_code": 42203,
"message": "Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Expected start-union. Got VALUE_STRING"
}
After searching in different sites including stackoverflow, I came through a suggestion
asking to explicitly specify the type while passing the record as below -
{
"key_schema": "{\"type\": \"record\", \"name\": \"key\", \"fields\": [{\"name\": \"keyInput\", \"type\": \"string\"}]}",
"value_schema": "{\"type\": \"record\", \"name\": \"value\", \"fields\": [{\"name\": \"valueInput1\", \"type\": \"string\"},{\"name\": \"valueInput2\",\"type\":[{\"type\":\"enum\",\"name\":\"actorobjType\",\"symbols\":[\"Agent\",\"Group\"]},\"null\"],\"default\":null}]}",
"records": [
{
"key": {
"keyInput": "testUser-key"
},
"value": {
"valueInput1": "testUser-value",
"valueInput2": {
"enum": "Agent"
}
}
}
]
}
But then I face the below error -
{
"error_code": 42203,
"message": "Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Unknown union branch enum"
}
The same suggestion worked fine for unions with other types like string and map, but with unions including enum, that does not seem to work.
I also thought there may be some other type which needs to be used for enum specification, Hence I tried some other words like below -
"valueInput2": {
"string": "Agent"
}
and
"valueInput2": {
"enumeration": "Agent"
}
But none of them seem to work. Please help me resolve this.
I ended up here, and davis michael's answer gave a hint, which helped me eventually figure it out.
Within the context of the question,
"valueInput2": {
"actorobjType": "Agent"
}
As ENUM type is not exist in JSON format, value type should be changed to correct one:
namespace + type name
In your case, it will be namespace + actorobjtype : "agent"
Related
Problem you have encountered:
Following steps at link below for transferJobs.patch API
https://cloud.google.com/storage-transfer/docs/reference/rest/v1/transferJobs/patch
Patch API works as expected if want to update description. Sample Below
Request:
{
"projectId": "<MY_PROJECT>",
"transferJob": {
"transferSpec": {
"objectConditions": {
"lastModifiedSince": "2022-01-24T18:30:00Z"
}
},
"description": "updated description"
},
"updateTransferJobFieldMask": "description"
}
Response: Success 200
Patch API do not work if want to update nested object field. Sample Below
{
"projectId": "<MY_PROJECT>",
"transferJob": {
"transferSpec": {
"objectConditions": {
"lastModifiedSince": "2022-01-22T18:30:00Z"
}
},
"description": "updated description"
},
"updateTransferJobFieldMask": "transferSpec.objectConditions.lastModifiedSince"
}
Response: 400
{"error": {
"code": 400,
"message": "Invalid path in the field mask.",
"status": "INVALID_ARGUMENT"}}
Tried other combinations following documentation/sample code reference but none of them work. Tried options as
transferSpec.objectConditions.lastModifiedSince
transferJob.transferSpec.objectConditions.lastModifiedSince
objectConditions.lastModifiedSince lastModifiedSince Snake case
combination referring to FieldMaskUtil as transfer_spec.object_conditions.last_modified_since
What I expected to happen:
Patch API to work successfully for nested object as per documentation I.e. "updateTransferJobFieldMask": "transferSpec.objectConditions.lastModifiedSince"
updateTransferJobFieldMask works on the top level object, in this case transferSpec.
Changing that line to updateTransferJobFieldMask: transferSpec should work.
From the documentation:
The field mask of the fields in transferJob that are to be updated in this request. Fields in transferJob that can be updated are: description, transfer_spec, notification_config, and status. To update the transfer_spec of the job, a complete transfer specification must be provided. An incomplete specification missing any required fields will be rejected with the error INVALID_ARGUMENT.
Providing complete object having required child field worked. Sample example for future reference to other dev.
Below job transfer dat from Azure to GCP bucket and during patch updating last modified time. Both transfer_spec and transferSpec works as updateTransferJobFieldMask.
{
"projectId": "<MY_PROJECT>",
"updateTransferJobFieldMask": "transfer_spec",
"transferJob": {
"transferSpec": {
"gcsDataSink": {
"bucketName": "<BUCKET_NAME>"
},
"objectConditions": {
"lastModifiedSince": "2021-12-30T18:30:00Z"
},
"transferOptions": {},
"azureBlobStorageDataSource": {
"storageAccount": "<ACCOUNT_NAME>",
"container": "<CONTAINER>",
"azureCredentials": {
"sasToken": "<SAS TOKEN>"
}
}
}
}
}
I have been able to successfully execute Appsync GraphQL queries and mutations from Postman. However, i'm struggling to connect to subscriptions which are websocket urls.
How can I achieve the same ?
Since Postman supports WebSockets testing GraphQL subscriptions is achievable as well. Such a testing requires two steps:
connection to a server,
sending a start message.
Establishing a connection:
Create a new WebSocket request.
Put your server URL ws:// or wss://.
Add custom header parameter Sec-WebSocket-Protocol: graphql-ws. Other headers may depend on your server configuration.
Press the "Connect" button.
When the connection is established we may start a subscription.
In the "New message" field put the command.
Press the "Send" button.
The start message should look like this:
{
"id":"1",
"payload": {
"operationName": "MySubscription",
"query": "subscription MySubscription {
someSubscription {
__typename
someField1
someField2 {
__typename
someField21
someField22
}
}
}",
"variables": null
},
"type": "start"
}
operationName is just the name of your subscription, I guess it's optional. And someSubscription must be a subscription type from your schema.
query reminds regular GraphQL syntax with one difference:
__typename keyword precedes every field list.
For example, the query from the payload in regular syntax looks like the following:
subscription MySubscription {
someSubscription {
someField1
someField2 {
someField21
someField22
}
}
}
Example message with parameters (variables):
{
"id":"1",
"payload": {
"operationName": "MySubscription",
"query": "subscription MySubscription($param1: String!) {
someSubscription((param1: $param1)) {
__typename
someField
}
}",
"variables": {
"param1": "MyValue"
}
},
"type": "start"
}
It also reminds regular GraphQL syntax as described above.
variables is an object with your parameters.
#Vladimir's answer is spot on. Adding a few notes for folks still having trouble.
Full document here # https://docs.aws.amazon.com/appsync/latest/devguide/real-time-websocket-client.html
Step 1 - establish connection:
make sure to base64 encode values in "header" and "payload" querystrings
header example:
{
"host":"example1234567890000.appsync-api.us-east-1.amazonaws.com",
"x-api-key":"da2-12345678901234567890123456"
}
payload: You can pass in empty payload
{}
Step 2 - register subscription:
Include the authorization in the message. Escape line feeds properly "\n" throws an error but "\\n" works. it throws the following error - misleading.
Don't forget to stringify value in "data" field.
{
"type": "error",
"payload": {
"errors": [
{
"errorType": "UnsupportedOperation",
"message": "unknown not supported through the realtime channel"
}
]
}
}
{
"id": "2",
"payload": {
"data": "{\"query\":\"subscription onCreateMessage { changeNotification{ __typename changeType from } }\",\"variables\":{}}",
"extensions":{
"authorization":{
"host":"example1234567890000.appsync-api.us-east-1.amazonaws.com",
"x-api-key":"da2-12345678901234567890123456"
}
}
},
"type": "start"
}
In a project where I'm sourcing data from NetlifyCMS I need to extend the GraphQL schema of Gatsby because there are optional fields in my CMS, which would cause errors trying to query non-existing data from GraphQL.
The below code extends the GraphQL types to always include the optional fields
// gatsby-node.js
// ADD OPTIONAL TYPES
// note: I have also tried exports.sourceNodes
exports.createSchemaCustomization = ({ actions }) => {
const { createTypes } = actions
const typeDefs = `
type SettingsJson implements Node {
tags: String
name: String
phone: String
email: String
additional_info: [SettingsJsonAdditional_info]
}
type SettingsJsonAdditional_info {
type: String
text: String
}
`
createTypes(typeDefs)
}
Sourcing and transforming the tags, name, phone and email work as expected. Queries return null if an optional field has not been filled out, and the query returns the correct value after being set. However, querying for additional_info always returns null even when containing data.
This is the json-file generated by NetlifyCMS:
{
"name": "Name Nameson",
"phone": "+41 1234 5678",
"email": "mail#example.com",
"additional-info": [
{
"type": "Booking",
"text": "Booker McBookerson <book#book.com>"
}
]
}
The following GraphQL query shows that the data is not being transformed properly when extending the GraphQL schema myself.
Query
query {
file(relativePath: {eq: "settings/contacts.json"}) {
internal {
content
}
childSettingsJson {
name
phone
email
additional_info {
type
text
}
}
}
}
Response
{
"data": {
"file": {
"internal": {
"content": "{\n \"name\": \"Name Nameson\",\n \"phone\": \"+41 1234 5678\",\n \"email\": \"mail#example.com\",\n \"additional-info\": [\n {\n \"type\": \"Booking\",\n \"text\": \"Booker McBookerson <book#book.com>\"\n },\n {\n \"type\": \"Booking2\",\n \"text\": \"Booker2 McBookerson <book#book.com>\"\n }\n ]\n}"
},
"childSettingsJson": {
"name": "Name Nameson",
"phone": "+41 1234 5678",
"email": "mail#example.com",
"additional_info": null
}
}
},
"extensions": {}
}
When the types are inferred by the transformer plugin itself I get the expected data when querying
// ...
"additional_info": [
{
"type": "Booking",
"text": "Booker McBookerson <book#book.com>"
}
]
// ...
This example uses json files with gatsby-transformer-json. I have tried with gatsby-transformer-yaml too with the same results.
Is it possible to add my array of SettingsJsonAdditional_info to the schema to get the "optional field" functionality I'm looking for?
One kind of hacky solution I have found is to make the transformer plugin infer the types by adding a dummy-file that will be sourced and transformed along with "real" files.
// dummy.json
{
"name": "dummy",
"phone": "dummy",
"email": "dummy",
"tags": "dummy",
"additional-info": [
{
"type": "dummy",
"text": "dummy"
}
]
}
This file can be hidden from NetlifyCMS (by simply not including a UI entry for it in the config.yml file of NetlifyCMS. It will guarantee that you can always query for the fields included in this file without getting GraphQL "field doesn't exist" errors.
I am trying to publish a message which has a union for one field as
{
"name": "somefield",
"type": [
"null",
{
"type": "array",
"items": {
"type": "record",
Publishing the message using the Kafka REST Proxy keeps throwing me the following error when somefield has an array populated.
{
"error_code": 42203,
"message": "Conversion of JSON to Avro failed: Failed to convert JSON to Avro: Expected start-union. Got START_ARRAY"
}
Same schema with somefield: null is working fine.
The Java classes are built in the Spring Boot project using the gradle plugin from the Avro schemas. When I use the generated Java classes and publish a message, with the array populated using the Spring KafkaTemplate, the message is getting published correctly with the correct schema. (The schema is taken from the generated Avro Specific Record) I copy the same json value and schema and publish via REST proxy, it fails with the above error.
I have these content types in the API call
accept:application/vnd.kafka.v2+json, application/vnd.kafka+json, application/json
content-type:application/vnd.kafka.avro.v2+json
What am I missing here? Any pointers to troubleshoot the issue is appreciated.
The messages I tested for were,
{
"somefield" : null
}
and
{
"somefield" : [
{"field1": "hello"}
]
}
However, it should be instead passed as,
{
"somefield" : {
"array": [
{"field1": "hello"}
]}
}
I'm trying to test the GraphQL server I built, by sending GraphQL queries to the server using Postman.
It works when I'm using raw radio button, but when I'm trying to use GraphQL radio button, it returns "message": "Syntax Error: Expected Name, found String \"query\"".
I have tried to change the syntax: mainly add or delete curly braces but nothing happened.
The query I sent in raw mode (working):
{
person(id:"123456789") {
personal_info {
address
}
}
}
The query I sent in GraphQL mode:
QUERY:
query getPerson ($id: String){
person(id: $id){
personal_info {
address
}
}
}
GRAPHQL VARIABLES:
{
"id": "123456789"
}
I expect to get the data I asked for, but I get the error message:
{
"errors": [
{
"message": "Syntax Error: Expected Name, found String \"query\"",
"locations": [
{
"line": 1,
"column": 2
}
]
}
]
}
I had the same problem. During researching I have found the next answer on stackoverflow, thanks #gbenga_ps.
Resolved by adding the correct header to Postman request:
Body of request should be something like next:
{
courses {
title
}
}
If incorrect content-type set, error like next happened:
{
"errors": [
{
"message": "Syntax Error: Expected Name, found String \"query\"",
"locations": [
{
"line": 1,
"column": 2
}
]
}
]
}