Related
I am new to the mongoDB aggregation and I have this situation. I have this Json and I need to convert by "select" this object:
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"adress": [
{
"localization": "DEATH STAR",
"createDate": 1627990848665
},
{
"localization": "TATOOINE",
"createDate": 1627990555665
},
]
}
this way:
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"localization": "DEATH STAR",
"createDate": 1627990848665
},
{
"type": "PF",
"code": 12345
"Name": Darth Vader,
"currency": "BRL",
"status": "SINGLE",
"localization": "TATOOINE",
"createDate": 1627990555665
}
So, after my query is complete, I will have 02 objects instead of 01. How can I do this?
I would like to do this via select because after converting I will sort by createDate and limit the number of records to return to the API. I'm using Criteria em my project.
The way to do this is $unwind, this will make 1 copy of the document, for each member of the array.
Test code here
db.collection.aggregate([
{
"$unwind": {
"path": "$user.adress"
}
},
{
"$set": {
"user": {
"$mergeObjects": [
"$user",
"$user.adress"
]
}
}
},
{
"$unset": [
"user.adress"
]
},
{
"$sort": {
"createDate": 1
}
},
{
"$limit": 10
}
])
Edit1 (the above is if user is a field, if it was the name of the collection)
$$ROOT is a system variable that has as value all the document
Test code here
Query
db.collection.aggregate([
{
"$unwind": {
"path": "$adress"
}
},
{
"$replaceRoot": {
"newRoot": {
"$mergeObjects": [
"$$ROOT",
"$adress"
]
}
}
},
{
"$unset": [
"adress"
]
},
{
"$sort": {
"createDate": 1
}
},
{
"$limit": 10
}
])
I am trying to convert a JSON File into CSV but I don't seem to have any luck in doing so. My JSON looks something like that:
...
{
{"meta": {
"contentType": "Response"
},
"content": {
"data": {
"_type": "ObjectList",
"erpDataObjects": [
{
"meta": {
"lastModified": "2020-08-10T08:37:21.000+0000",
},
"head": {
"fields": {
"number": {
"value": "1",
},
"id": {
"value": "10000"
},
}
}
{
"meta": {
"lastModified": "2020-08-10T08:37:21.000+0000",
},
"head": {
"fields": {
"number": {
"value": "2",
},
"id": {
"value": "10001"
},
}
}
{
"meta": {
"lastModified": "2020-08-10T08:37:21.000+0000",
},
"head": {
.. much more data
I basically want my csv to look like this:
number,id
1,10000
2,10001
My flow looks like this:
GetFile -> Set the output-file name -> ConvertRecord -> UpdateAttribute -> PutFile
ConvertRecord uses the JsonTreeReader and a CSVRecordSetWriter
JsonTreeReader
CsvRecordSetWriter.
They both call on an AvroSchemaRegistry which looks like this:
AvroSchemaRegistry
The AvroSchema itself looks like this:
{
"type": "record",
"name": "head",
"fields":
[
{"name": "number", "type": ["string"]},
{"name": "id", "type": ["string"]},
]
}
But I only get this output:
number,id
,
Which makes sense because I'm not specifically indicating where those values are located. I used the JsonPathReader instead before but it only looked like this:
JsonPathReader
Which obvioulsy only gave me one record. I'm not really sure how I can configure either of the two to output exactly what I want. Help would be much appreciated!
Using ConvertRecord for JSON -> CSV is mostly intended for "flat" JSON files where each field in the object becomes a column in the outgoing CSV file. For nested/complex structures, consider JoltConvertRecord, it allows you to do more complex transformations. Your example doesn't appear to be valid JSON as-is, but assuming you have something like this as input:
{
"meta": {
"contentType": "Response"
},
"content": {
"data": {
"_type": "ObjectList",
"erpDataObjects": [
{
"meta": {
"lastModified": "2020-08-10T08:37:21.000+0000"
},
"head": {
"fields": {
"number": {
"value": "1"
},
"id": {
"value": "10000"
}
}
}
},
{
"meta": {
"lastModified": "2020-08-10T08:37:21.000+0000"
},
"head": {
"fields": {
"number": {
"value": "2"
},
"id": {
"value": "10001"
}
}
}
}
]
}
}
}
The following JOLT spec should give you what you want for output:
[
{
"operation": "shift",
"spec": {
"content": {
"data": {
"erpDataObjects": {
"*": {
"head": {
"fields": {
"number": {
"value": "[&4].number"
},
"id": {
"value": "[&4].id"
}
}
}
}
}
}
}
}
}
]
When the other person declined the meeting and proposed a new time. In Outlook, you can see the proposed time.
Now I am trying to use Microsoft Graph API to get that proposed time.
For example, the original meeting date is 2018-03-08, and the other person declined and proposed a new date 2018-03-12.
I tried
GET /beta/me/messages/{messageId}=?$expand=microsoft.graph.eventMessage/event
However, I cannot find the proposed time from the result returned. How can I get it? Thanks
{
"#odata.context": "https://graph.microsoft.com/beta/$metadata#users('576552d5-3bc0-42a6-a53d-bfceb405db23')/messages/$entity",
"#odata.type": "#microsoft.graph.eventMessage",
"#odata.etag": "W/\"DAAAABYAAACpTc/InBsuTYwTUBb+VIb4AADoRAyI\"",
"id": "AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk-iwQZRbXMgkfKtmYhBwCpTc-InBsuTYwTUBb_VIb4AAAAAAEMAACpTc-InBsuTYwTUBb_VIb4AADnwc8mAAA=",
"createdDateTime": "2018-03-06T22:29:10Z",
"lastModifiedDateTime": "2018-03-06T22:29:11Z",
"changeKey": "DAAAABYAAACpTc/InBsuTYwTUBb+VIb4AADoRAyI",
"categories": [],
"receivedDateTime": "2018-03-06T22:29:11Z",
"sentDateTime": "2018-03-06T22:29:05Z",
"hasAttachments": false,
"internetMessageId": "<MWHPR15MB18399806CC97C61817C9A2B18BD90#MWHPR15MB1839.namprd15.prod.outlook.com>",
"subject": "New Time Proposed: Test",
"bodyPreview": "",
"importance": "normal",
"parentFolderId": "AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgAuAAAAAACUbnk-iwQZRbXMgkfKtmYhAQCpTc-InBsuTYwTUBb_VIb4AAAAAAEMAAA=",
"conversationId": "AAQkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgAQAOnuCMgoRLdGs-1scw6i7EU=",
"conversationIndex": "AdO1mnWL6e4IyChEt0az/WxzDqLsRQAABO5D",
"isDeliveryReceiptRequested": null,
"isReadReceiptRequested": false,
"isRead": false,
"isDraft": false,
"webLink": "https://outlook.office365.com/owa/?ItemID=AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk%2FiwQZRbXMgkfKtmYhBwCpTc%2FInBsuTYwTUBb%2BVIb4AAAAAAEMAACpTc%2FInBsuTYwTUBb%2BVIb4AADnwc8mAAA%3D&exvsurl=1&viewmodel=ReadMessageItem",
"inferenceClassification": "focused",
"unsubscribeData": [],
"unsubscribeEnabled": false,
"meetingMessageType": "meetingDeclined",
"type": "singleInstance",
"isOutOfDate": false,
"isAllDay": false,
"isDelegated": false,
"body": {
"contentType": "html",
"content": "Hi"
},
"sender": {
"emailAddress": {
"name": "Rose",
"address": "rose#example.com"
}
},
"from": {
"emailAddress": {
"name": "Rose",
"address": "rose#example.com"
}
},
"toRecipients": [
{
"emailAddress": {
"name": "Jack",
"address": "jack#example.com"
}
}
],
"ccRecipients": [],
"bccRecipients": [],
"replyTo": [],
"mentionsPreview": null,
"flag": {
"flagStatus": "notFlagged"
},
"startDateTime": {
"dateTime": "2018-03-08T04:00:00.0000000",
"timeZone": "UTC"
},
"endDateTime": {
"dateTime": "2018-03-08T05:00:00.0000000",
"timeZone": "UTC"
},
"location": {
"displayName": "Test",
"locationType": "default",
"uniqueIdType": "unknown"
},
"recurrence": null,
"event#odata.context": "https://graph.microsoft.com/beta/$metadata#users('576552d5-3bc0-42a6-a53d-bfceb405db23')/messages('AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk-iwQZRbXMgkfKtmYhBwCpTc-InBsuTYwTUBb_VIb4AAAAAAEMAACpTc-InBsuTYwTUBb_VIb4AADnwc8mAAA%3D')/microsoft.graph.eventMessage/event/$entity",
"event": {
"#odata.etag": "W/\"qU3PyJwbLk2ME1AW/lSG+AAA6EQMeA==\"",
"id": "AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk-iwQZRbXMgkfKtmYhBwCpTc-InBsuTYwTUBb_VIb4AAAAAAENAACpTc-InBsuTYwTUBb_VIb4AADnwkmiAAA=",
"createdDateTime": "2018-03-06T22:28:32.3852279Z",
"lastModifiedDateTime": "2018-03-06T22:29:11.4604154Z",
"changeKey": "qU3PyJwbLk2ME1AW/lSG+AAA6EQMeA==",
"categories": [],
"originalStartTimeZone": "Pacific Standard Time",
"originalEndTimeZone": "Pacific Standard Time",
"iCalUId": "040000008200E00074C5B7101A82E0080000000000000000000000000000000000000000310000007643616C2D5569640100000033324633333433392D433744452D344338362D393046452D44424639314131363444323900",
"reminderMinutesBeforeStart": 15,
"isReminderOn": true,
"hasAttachments": false,
"subject": "Test",
"bodyPreview": "Test",
"importance": "normal",
"sensitivity": "normal",
"isAllDay": false,
"isCancelled": false,
"isOrganizer": true,
"responseRequested": true,
"seriesMasterId": null,
"showAs": "busy",
"type": "singleInstance",
"webLink": "https://outlook.office365.com/owa/?itemid=AAMkADBlZTUwNTkxLWVmODgtNDVhNC1iZjhlLTdjNjA1ODZlMDI5MgBGAAAAAACUbnk%2FiwQZRbXMgkfKtmYhBwCpTc%2FInBsuTYwTUBb%2BVIb4AAAAAAENAACpTc%2FInBsuTYwTUBb%2BVIb4AADnwkmiAAA%3D&exvsurl=1&path=/calendar/item",
"onlineMeetingUrl": null,
"responseStatus": {
"response": "organizer",
"time": "0001-01-01T00:00:00Z"
},
"body": {
"contentType": "html",
"content": "Hi"
},
"start": {
"dateTime": "2018-03-08T04:00:00.0000000",
"timeZone": "UTC"
},
"end": {
"dateTime": "2018-03-08T05:00:00.0000000",
"timeZone": "UTC"
},
"location": {
"displayName": "Test",
"locationType": "default",
"uniqueId": "Test",
"uniqueIdType": "private"
},
"locations": [
{
"displayName": "Test",
"locationType": "default",
"uniqueId": "Test",
"uniqueIdType": "private"
}
],
"recurrence": null,
"attendees": [
{
"type": "required",
"status": {
"response": "declined",
"time": "2018-03-06T22:29:05Z"
},
"emailAddress": {
"name": "Rose",
"address": "rose#example.com"
}
}
],
"organizer": {
"emailAddress": {
"name": "Jack",
"address": "jack#example.com"
}
}
}
}
The API doesn't expose this information directly. The related entities just don't have the fields defined to show this information. However, the data is obviously there, you just need to know how to get to it. The answer lies in extended properties.
Basically since Graph doesn't expose these values, we need to dig into the MAPI properties that store this data. After a little digging through the Exchange server protocol docs I found PidLidAppointmentProposedStartWhole and PidLidAppointmentProposedEndWhole. We just need to translate those MAPI property definitions into the Graph extended property syntax.
From the doc, those are both in the PSETID_Appointment namespace, which uses the GUID {00062002-0000-0000-C000-000000000046}. PidLidAppointmentProposedStartWhole uses id 0x8250, and PidLidAppointmentProposedEndWhole uses id 0x8251. So those should translate to:
PidLidAppointmentProposedStartWhole: 'SystemTime {00062002-0000-0000-C000-000000000046} Id 0x8250'
PidLidAppointmentProposedEndWhole: 'SystemTime {00062002-0000-0000-C000-000000000046} Id 0x8251'
If we use those in an $expand clause as per Get singleValueLegacyExtendedProperty, we get something like:
GET /me/mailfolders/inbox/messages?$expand=singleValueExtendedProperties($filter=id eq 'SystemTime {00062002-0000-0000-C000-000000000046} Id 0x8250' or id eq 'SystemTime {00062002-0000-0000-C000-000000000046} Id 0x8251')
And the response looks something like this (other properties omitted):
{
"subject": "New Time Proposed: Let's meet",
"singleValueExtendedProperties": [
{
"id": "SystemTime {00062002-0000-0000-c000-000000000046} Id 0x8250",
"value": "2018-03-20T20:00:00Z"
},
{
"id": "SystemTime {00062002-0000-0000-c000-000000000046} Id 0x8251",
"value": "2018-03-20T21:00:00Z"
}
]
}
Let's say I have two JSON objects (call them 'websites' and 'links'). I need to end up with both objects in a single CSV (ideally in separate columns).
I'm working with something like this:
File.open("file.json", "w") do |f|
combined = [websites, links]
f.write(JSON.pretty_generate(combined))
end
And then I'm using the Ruby gem json2csv to convert this file to a CSV. But when I do, I get the following error:
error: undefined method 'keys' for #<Array:0x007fea8a8e33f8>
I can't figure out what's wrong. When I look in file.json, it appears to be structured like this: [{websites}, {links}]. From my limited knowledge of JSON, I think that's right, but I could easily be wrong.
Also, I know this won't get me separate columns in the CSV. If anyone has an answer for that part, major bonus points.
EDIT: JSON examples included below; error message changed after minor fix on my end.
websites:
{
"uri": "https://v1/websites",
"id": 28235674,
"background": null,
"createdDate": 1399585684000,
"lastActivityDate": 1430682494000,
"lastCommunicationDate": 1430682494000,
"lastNonCommunicationChronicleDate": 1430330886000,
"lastModifiedDate": 1449263116000,
"lastViewedDate": 1421429034000,
"preferredContactType": null,
"rss": "",
"emailAddresses": [
{
"email": "",
"type": "Work"
},
{
"email": "",
"type": "Work"
},
{
"email": "",
"type": "Work"
},
{
"email": "not found",
"type": "Work"
}
],
"phoneNumbers": [
],
"streetAddresses": [
],
"socialNetworks": [
{
"profileUrl": "http://twitter.com",
"name": "Twitter"
},
{
"profileUrl": "http://www.facebook.com",
"name": "Facebook"
},
{
"profileUrl": "http://plus.google.com",
"name": "GooglePlus"
},
{
"profileUrl": "http://www.linkedin.com",
"name": "LinkedIn"
},
{
"profileUrl": "http://twitter.com",
"name": "Twitter"
}
],
"contactUrls": [
],
"tags": [
"tag1",
"tag2"
],
"mostRecentActivity": "https://v1/history",
"mostRecentChronicle": "https://v1/history",
"mostRecentCommunication": "https://v1/history",
"mostRecentNonCommunicationChronicle": "https://v1/history",
"projectStates": "https://v1/websites",
"history": "https://v1/history",
"customFieldValues": [
],
"name": "",
"primaryDomain": "",
"domains": [
""
],
"associatedPeople": "https://v1/people",
"payments": "https://payments",
"links": "https://v1/links",
"type": "https://v1/websites"
}
links:
{
"uri": "https://v1/links/custom_fields",
"id": 15529329,
"value": "Name",
"backgroundColor": null,
"customField": "https://v1/links/custom_fields"
}
combined output:
[
{
"uri": "https://v1/websites",
"id": 28235674,
"background": null,
"createdDate": 1399585684000,
"lastActivityDate": 1430682494000,
"lastCommunicationDate": 1430682494000,
"lastNonCommunicationChronicleDate": 1430330886000,
"lastModifiedDate": 1449263116000,
"lastViewedDate": 1421429034000,
"preferredContactType": null,
"rss": "",
"emailAddresses": [
{
"email": "",
"type": "Work"
},
{
"email": "",
"type": "Work"
},
{
"email": "",
"type": "Work"
},
{
"email": "not found",
"type": "Work"
}
],
"phoneNumbers": [
],
"streetAddresses": [
],
"socialNetworks": [
{
"profileUrl": "http://twitter.com/",
"name": "Twitter"
},
{
"profileUrl": "http://www.facebook.com",
"name": "Facebook"
},
{
"profileUrl": "http://plus.google.com",
"name": "GooglePlus"
},
{
"profileUrl": "http://www.linkedin.com/",
"name": "LinkedIn"
},
{
"profileUrl": "http://twitter.com/",
"name": "Twitter"
}
],
"contactUrls": [
],
"tags": [
"tag1",
"tag2"
],
"mostRecentActivity": "https://v1/history/",
"mostRecentChronicle": "https://v1/history/",
"mostRecentCommunication": "https://v1/history/",
"mostRecentNonCommunicationChronicle": "https://v1/history/",
"projectStates": "https://v1/websites/",
"history": "https://v1/history",
"customFieldValues": [
],
"name": "",
"primaryDomain": "",
"domains": [
""
],
"associatedPeople": "https://v1/people",
"links": "https://v1/links",
"type": "https://v1/websites"
},
{
"uri": "https://links/custom_fields",
"id": 15529329,
"value": "Name",
"backgroundColor": null,
"customField": "https://links/custom_fields"
}
]
JSON.pretty_generate() is expecting a hash and you are passing an array of 2 hashes. Start with: combined.map { |c| f.write(JSON.pretty_generate(c)) } and then mapping them into your CSV should be as easy as following the CSV documentation.
when I upload a photo using the arrowdb dashboard on https://platform.appcelerator.com
Cloud.Users.query only shows the photo_id
but whne I created a new user using the dashboard and attached a phot it is showing in the Cloud.Users.query
eg . photo uploaded after user created
{
"id": "563019f18cb04aede69e2111",
"first_name": "store1",
"last_name": "123",
"created_at": "2015-10-28T00:42:25+0000",
"updated_at": "2016-01-22T08:59:44+0000",
"external_accounts": [],
"confirmed_at": "2015-10-28T00:42:25+0000",
"username": "user",
"admin": "false",
"stats": {
"photos": {
"total_count": 0
},
"storage": {
"used": 0
}
},
"photo_id": "56a1dc083a654d090d126792",
"friend_counts": {
"requests": 0,
"friends": 0
}
}
eg. photo uploaded while creating user
{
"id": "56a1f0333a65234234390d7",
"first_name": "qqqq",
"last_name": "wwwe",
"created_at": "2016-01-22T09:02:43+0000",
"updated_at": "2016-01-22T09:07:18+0000",
"external_accounts": [],
"confirmed_at": "2016-01-22T09:02:43+0000",
"username": "qwe",
"admin": "false",
"stats": {
"photos": {
"total_count": 0
},
"storage": {
"used": 0
}
},
"photo": {
"id": "56a1f0333a654d090d0390d8",
"filename": "userPhoto.jpg",
"size": 25394,
"md5": "e20f4fcadf6cde9fccfb458dd11951d4",
"created_at": "2016-01-22T09:02:43+0000",
"updated_at": "2016-01-22T09:02:43+0000",
"processed": true,
"urls": {
"original": "https://s3-us-west-1.amazonaws.com/storage-platform.cloud.appcelerator.com/xmqh1djNEIChtQFP6d37HNH5DQNCXQoX/photos/51/d4/56a1f0333a654d090d0390d9/userPhoto_original.jpg"
},
"content_type": "image/jpeg",
"user": {
"id": "56a1f0333a65234234390d7",
"first_name": "qqqq",
"last_name": "wwwe",
"created_at": "2016-01-22T09:02:43+0000",
"updated_at": "2016-01-22T09:07:18+0000",
"external_accounts": [],
"confirmed_at": "2016-01-22T09:02:43+0000",
"username": "qwe",
"admin": "false",
"stats": {
"photos": {
"total_count": 0
},
"storage": {
"used": 0
}
},
"photo_id": "56a1f0333a654d090d0390d8",
"friend_counts": {
"requests": 0,
"friends": 0
}
}
},
"friend_counts": {
"requests": 0,
"friends": 0
}
}
basically the user which had photo uploaded during creation shows this extra info
"photo": {
"id": "56a1f0333a654d090d0390d8",
"filename": "userPhoto.jpg",
"size": 25394,
"md5": "e20f4fcadf6cde9fccfb458dd11951d4",
"created_at": "2016-01-22T09:02:43+0000",
"updated_at": "2016-01-22T09:02:43+0000",
"processed": true,
"urls": {
"original": "https://s3-us-west-1.amazonaws.com/storage-platform.cloud.appcelerator.com/xmqh1djNEIChtQFP6d37HNH5DQNCXQoX/photos/51/d4/56a1f0333a654d090d0390d9/userPhoto_original.jpg"
},
"content_type": "image/jpeg",
"user": {
"id": "56a1f0333a65234234390d7",
"first_name": "qqqq",
"last_name": "wwwe",
"created_at": "2016-01-22T09:02:43+0000",
"updated_at": "2016-01-22T09:07:18+0000",
"external_accounts": [],
"confirmed_at": "2016-01-22T09:02:43+0000",
"username": "qwe",
"admin": "false",
"stats": {
"photos": {
"total_count": 0
},
"storage": {
"used": 0
}
},
I've been experiencing also that kind of problem, plus I've noticed that the images won't show up from the ArrowDB user interface (not sure if related).
The API returns only an empty object when you query the model:
photo: {}
I've been creating a ticket for the ArrowDB user interface https://jira.appcelerator.org/browse/API-1277.
sachinmw did you create a ticket for the initial problem yet?
A workaround could be to use the photo_id and run another separate query to retrieve the photo model but this is not good for network optimisation purposes.
EDIT
Ok after dealing with Appcelerator directly about having an empty photo: {} Object return the answer is pretty simple:
Whenever you use the query() function like Cloud.Objects.query() from ti.cloud on any ArrowDB object, there is a parameter called response_json_depth which is set to 1 by default and will only return one level of the JSON Object returned by the API.
Without touching that parameter I was seeing:
{
"Vehicle": [
{
"name": "foo",
"photo: {}
}
]
}
By setting response_json_depth to 3 I managed to have:
{
"Vehicle": [
{
"name": "foo",
"photo: {
"urls": {
"original": "http://bar.com"
}
}
}
]
}
Hope that will help someone. The also applies for the Cloud.Objects.show() method for any ArrowDB object.