I learn App Connect Enterprise v11 and try to make a DFDL Schema for JSON data and I do not know how. I successfully made a Schema for Record-Oriented-Text like the below but do not know how to do it for JSON.
Record-Oriented-Text example:
Delivery+++XYZ123ABC+++My order was delivered in time, but the package was torn|C01-COM684a2da-384+++Your complaint has been received
JSON example:
{
"YourComplaint": {
"Type": "Delivery",
"Reference": "XYZ123ABC",
"Text": "My order was delivered in time, but the package was torn"
},
"Reply": {
"OurReference": "C01-COM684a2da-384",
"Text": "Your complaint has been received"
}
}
You should not create a DFDL schema for JSON data. ACE can parse JSON without any help from a schema. You should use the JSON parser/domain for JSON (just as you use the XMLNSC domain for XML).
If you need to output your record-oriented data as JSON then you need to map from InputRoot.DFDL to OutputRoot.JSON. You may also need to set some field types to ensure that the JSON data looks exactly how you need it.
Related
I was wondering if an object type that has an id property has to have the same content given the same id. At the moment the same id can have different content.
The following query:
const query = gql`
query products(
$priceSelector: PriceSelectorInput!
) {
productProjectionSearch(
priceSelector: $priceSelector
) {
total
results {
masterVariant {
# If you do the following it will work
# anythingButId: id
id
scopedPrice {
country
}
}
}
}
}
`;
If the PriceSelectorInput is {currency: "USD", country: "US"} then the result is:
{
"productProjectionSearch": {
"total": 2702,
"results": [
{
"name": "Sweater Pinko white",
"masterVariant": {
"id": 1,
"scopedPrice": {
"country": "US",
"__typename": "ScopedPrice"
},
"__typename": "ProductSearchVariant"
},
"__typename": "ProductProjection"
}
],
"__typename": "ProductProjectionSearchResult"
}
}
If the PriceSelectorInput is {currency: "EUR", country: "DE"} then the result is:
{
"productProjectionSearch": {
"total": 2702,
"results": [
{
"name": "Sweater Pinko white",
"masterVariant": {
"id": 1,
"scopedPrice": {
"country": "DE",
"__typename": "ScopedPrice"
},
"__typename": "ProductSearchVariant"
},
"__typename": "ProductProjection"
}
],
"__typename": "ProductProjectionSearchResult"
}
}
My question is that masterVariant of type ProductSearchVariant has id of 1 in both cases but different values for scopedPrice. This breaks apollo cache defaultDataIdFromObject function as demonstrated in this repo. My question is; is this a bug in apollo or would this be a violation of a graphql standard in the type definition of ProductSearchVariant?
TLDR
No it does not break the spec. The spec forces absolutely nothing in regards caching.
Literature for people that may be interested
From the end of the overview section
Because of these principles [... one] can quickly become productive without reading extensive documentation and with little or no formal training. To enable that experience, there must be those that build those servers and tools.
The following formal specification serves as a reference for those builders. It describes the language and its grammar, the type system and the introspection system used to query it, and the execution and validation engines with the algorithms to power them. The goal of this specification is to provide a foundation and framework for an ecosystem of GraphQL tools, client libraries, and server implementations -- spanning both organizations and platforms -- that has yet to be built. We look forward to working with the community in order to do that.
As we just saw the spec says nothing about caching or implementation details, that's left out to the community. The rest of the paper proceeds to give details on how the type-system, the language, requests and responses should be handled.
Also note that the document does not mention which underlying protocol is being used (although commonly it's HTTP). You could effectively run GraphQL communication over a USB device or over infra-red light.
We hosted an interesting talk at our tech conferences which you might find interesting. Here's a link:
GraphQL Anywhere - Our Journey With GraphQL Mesh & Schema Stitching • Uri Goldshtein • GOTO 2021
If we "Ctrl+F" ourselves to look for things as "Cache" or "ID" we can find the following section which I think would help get to a conclusion here:
ID
The ID scalar type represents a unique identifier, often used to refetch an object or as the key for a cache. The ID type is serialized in the same way as a String; however, it is not intended to be human‐readable. While it is often numeric, it should always serialize as a String.
Result Coercion
GraphQL is agnostic to ID format, and serializes to string to ensure consistency across many formats ID could represent, from small auto‐increment numbers, to large 128‐bit random numbers, to base64 encoded values, or string values of a format like GUID.
GraphQL servers should coerce as appropriate given the ID formats they expect. When coercion is not possible they must raise a field error.
Input Coercion
When expected as an input type, any string (such as "4") or integer (such as 4) input value should be coerced to ID as appropriate for the ID formats a given GraphQL server expects. Any other input value, including float input values (such as 4.0), must raise a query error indicating an incorrect type.
It mentions that such field it is commonly used as a cache key (and that's the default cache key for the Apollo collection of GraphQL implementations) but it doesn't tell us anything about "consistency of the returned data".
Here's the link for the full specification document for GraphQL
Warning! Opinionated - My take on ID's
Of course all I am about to say has nothing to do with the GraphQL specification
Sometimes an ID is not enough of a piece of information to decide whether to cache something. Let's think about user searches:
If I have a FavouriteSearch entity that has an ID on my database and a field called textSearch. I'd commonly like to expose a property results: [Result!]! on my GraphQL specification referencing all the results that this specific text search yielded.
These results are very likely to be different from the moment I make the search or five minutes later when I revisit my favourite search. (Thinking about a text-search on a platform such as TikTok where users may massively upload content).
So based on this definition of the entity FavouriteSearch it makes sense that the caching behavior is rather unexpected.
If we think of the problem from a different angle we might want a SearchResults entity which could have an ID and a timestamp and have a join-table where we reference all those posts that were related to the initial text-search and in that case it would make sense to return a consistent content for the property results on our GraphQL schema.
Thing is that it depends on how we define our entities and it's ultimately not related to the GraphQL spec
A solution for your problem
You can specify how Apollo generates the key for later use as key on the cache as #Matt already pointed in the comments. You may want to tap into that and override that behavior for those entitites that have a __type equal to your masterVariant property type and return NO_KEY for all of them (or similar) in order to avoid caching from your ApolloClient on those specific fields.
I hope this was helpful!
I'm trying to write logs to an Elasticsearch index from a Kubernetes cluster. Fluent-bit is being used to read stdout and it enriches the logs with metadata including pod labels. A simplified example log object is
{
"log": "This is a log message.",
"kubernetes": {
"labels": {
"app": "application-1"
}
}
}
The problem is that a few other applications deployed to the cluster have labels of the following format:
{
"log": "This is another log message.",
"kubernetes": {
"labels": {
"app.kubernetes.io/name": "application-2"
}
}
}
These applications are installed via Helm charts and the newer ones are following the label and selector conventions as laid out here. The naming convention for labels and selectors was updated in Dec 2018, seen here, and not all charts have been updated to reflect this.
The end result of this is that depending on which type of label format makes it into an Elastic index first, trying to send the other type in will throw a mapping exception. If I create a new empty index and send in the namespaced label first, attempting to log the simple app label will throw this exception:
object mapping for [kubernetes.labels.app] tried to parse field [kubernetes.labels.app] as object, but found a concrete value
The opposite situation, posting the namespaced label second, results in this exception:
Could not dynamically add mapping for field [kubernetes.labels.app.kubernetes.io/name]. Existing mapping for [kubernetes.labels.app] must be of type object but found [text].
What I suspect is happening is that Elasticsearch sees the periods in the field name as JSON dot notation and is trying to flesh it out as an object. I was able to find this PR from 2015 which explicitly disallows periods in field names however it seems to have been reversed in 2016 with this PR. There is also this multi-year thread from 2015-2017 discussing this issue but I was unable to find anything recent involving the latest versions.
My current thoughts on moving forward is to standardize the Helm charts we are using to have all of the labels use the same convention. This seems like a band-aid on the underlying issue though which is that I feel like I'm missing something obvious in the configuration of Elasticsearch and dynamic field mappings.
Any help here would be appreciated.
I opted to use the Logstash mutate filter with the rename option as described here:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-mutate.html#plugins-filters-mutate-rename
The end result looked something like this:
filter {
mutate {
'[kubernetes][labels][app]' => '[kubernetes][labels][app.kubernetes.io/name]'
'[kubernetes][labels][chart]' => '[kubernetes][labels][helm.sh/chart]'
}
}
Although personally I've never encountered the exact same issue, I had similar problems when I indexed some test data and afterwards changed the structure of the document that should have been indexed (especially when "unflattening" data structures).
Your interpretation of the error message is correct. When you first index the document
{
"log": "This is another log message.",
"kubernetes": {
"labels": {
"app.kubernetes.io/name": "application-2"
}
}
}
Elasticsearch will recognize the app as an object/structure due to dynamic mapping.
When you then try to index the document
{
"log": "This is a log message.",
"kubernetes": {
"labels": {
"app": "application-1"
}
}
}
the previously, dynamically created mapping defined the field app as an object with sub-fields but elasticsearch encounters a concrete value, namely "application-1".
I suggest that you setup an index template to define the correct mappings. For the 'outdated' logging-versions I suggest to pre-process the particular documents either through an elasticsearch ingest-pipeline or with e.g. Logstash to get the documents in the correct format.
Hope that helps.
I've been tasked with updating a legacy Notes application - I'd rather do basically anything else, but such is life. As per the API, I should be able to update a rich text field if the data is in format
FieldName: { contentType: 'text/html', data: newData, type: 'richtext' }
(serialized to JSON, of course)
But what happens is that the original RT field gets replaced by three MIME Part fields (with the same name, containing what you'd expect, "Content-Type: multipart/mixed", "boundary" and so on), and "newData" gets stored in a $FILE attachment. And also few MIME-specific fields get added to the document ($MIMETrack, $NoteHasNativeMIME, MIME_Version).
Now this certainly wouldn't be the first time that Notes documentation doesn't match actual functionality, but I was wondering if anyone has been successfully able to do this? Alternatively, any other way to update an RT field via AJAX (preferably with HTTP PATCH)?
EDIT: upon further inspection, this seems to be a configuration issue. I tried doing a GET from a document with an RT field (that contains the text "testing rt field", submitted via a regular web form), the expected result would be according to the API
"FieldName": {
"contentType":"text/html",
"data":"testing rt field",
"type":"richtext"
}
but instead what is returned is
"FieldName": {
"type":"multipart",
"content": [
{
"contentType":"multipart\/alternative; Boundary=\"0__=4DBB0A82DFA47A268f9e8a93df938690918c4DBB0A82DFA47A26\"",
"contentDisposition":"inline"
},
{
"contentType":"text\/plain; charset=US-ASCII",
"data":"testing rt field",
"boundary":"--0__=4DBB0A82DFA47A268f9e8a93df938690918c4DBB0A82DFA47A26"
},
{
"contentType":"text\/html; charset=US-ASCII",
"contentDisposition":"inline",
"data":"<html><body><font size=\"2\" face=\"sans-serif\">testing rt field<\/font><\/body><\/html>",
"boundary":"--0__=4DBB0A82DFA47A268f9e8a93df938690918c4DBB0A82DFA47A26"
}
]
}
(sorry for the formatting)
So I'm guessing there is a problem with our Domino configuration somewhere. Where, I have no idea, any tips would be greatly appreciated.
In Domino 8.5.3 UP1 the data service represented rich text fields as one HTML part ("type": "richtext" as described in the 8.5.3 doc). This had some severe limitations. For example, you couldn't create a rich text field with embedded images and attachments.
Since Domino 9.0 the data service represents rich text fields as multiple parts ("type": "multipart" as described in the 9.0.1 doc). However, you can still PUT and POST the old rich text format, and you can GET the old format by specifying multipart=false in the URL. In other words, the original poster seems to be using Domino 9.0 and it is working as expected.
We have an internal database ID for each course we have. When using the batch function to create courses, the returned information doesn’t not give us any way to get this ID back so we cannot update our database and let it know that the course has been created.
Also updating the deletion response for the same reason would also be handy.
The response for a batch of deletes looks like:
--batch_123456
{}
--batch_123456
--batch_123456
{}
--batch_123456
--batch_123456
{}
--batch_123456
If they are all successful then it's not really a problem, but if something goes wrong then it's impossible to tell which ones were deleted and which ones were not deleted, because according to the API:
" The server may perform your calls in any order. Don't count on their being executed in the order in which you specified them."
A simple message that said "course #123 has been deleted" would be very helpful. That way I could program the callback code to update the database using the ID
For the course creation the response looks like:
--batch_123456
{
"id": "208571459",
"name": "***",
"section": "***",
"ownerId": "123456",
"creationTime": "2015-09-14T13:13:59.622Z",
"updateTime": "2015-09-14T13:14:02.820Z",
"enrollmentCode": "***",
"courseState": "PROVISIONED",
"alternateLink": "***"
}
--batch_123456
I only put in one response to save space. The ID is the new GoogleID of the course, the ID we actually send in which corresponds to the records in our DB does not get returned. So we have to use a combination of other data (ownerID + name) to get the record and update it, but this is not ideal and may not work fro everyone. Again adding in all of the information that get's sent into the response would be very helpful so we can use the callback properly.
I also want to add the more useful specific information should also be added to all error messages. In fact every batch response should have some sort of ID, that way the callback functions can be more useful.
As mentioned briefly in the documentation, you can pass an alias into Courses.id when creating a course to assign an alias to a course as you create it. The alias can include your internal identifier for the course and can be used in future requests.
Alternatively, each request in a batch can be assigned a unique Content-ID header, which will be returned in the response. Many client libraries provide built-in support for setting and getting these IDs.
I'm using MongoLab add-on on Heroku.
My app use Mongoose and According to the docs the document id type is ObjectID (by default).
This is why my json looks something like that:
{
"__v" : 0,
"_id" : ObjectId("53c824d6f26327e00f9ae117"),
"company" : "53c824d6f26327e00f9ae118",
...
}
The problem: MongoLab addon does not khow how to parse the keyword "ObjectID", displaying an error message.
Am i missing something here? What can be done?
If you're referring to the JSON editor in the MongoLab web UI, it only accepts strict JSON formatting. For special types like ObjectId's and dates, you need to use their associated extended JSON format. For an ObjectId, that would look like:
{
"__v": 0,
"_id": {"$oid": "53c824d6f26327e00f9ae117"},
"company": "53c824d6f26327e00f9ae118",
...
}
Hopefully that helps! You can always feel free to write us at support#mongolab.com for any questions or issues.
Kind regards,
Sean#MongoLab