I would appreciate your help on importing and expanding an XML file, with Power Query.
The XML link is here:
UN Consolidated Sanctions List
After expanding the table named ”Individual” I get a lot of tables nested within cells.
Nested Table
I tried to apply the solution offered here:
table-within-a-cell-unable-to-expand
... but nothing happens, although three steps are recorded in query settings:
Empty steps
The code in the Advanced Editor is this:
Source Query:
let
Source = Xml.Tables(Web.Contents("https://scsanctions.un.org/resources/xml/en/consolidated.xml")),
#"Expanded ENTITIES" = Table.ExpandTableColumn(Source, "ENTITIES", {"ENTITY"}, {"ENTITY"}),
#"Expanded INDIVIDUALS" = Table.ExpandTableColumn(#"Expanded ENTITIES", "INDIVIDUALS", {"INDIVIDUAL"}, {"INDIVIDUAL"})
in
#"Expanded INDIVIDUALS"
And in the referenced query:
let
Source = #"UN Source",
#"Expanded ENTITY" = Table.ExpandTableColumn(Source, "ENTITY", {"DATAID", "VERSIONNUM", "FIRST_NAME", "UN_LIST_TYPE", "REFERENCE_NUMBER", "LISTED_ON", "COMMENTS1", "LIST_TYPE", "LAST_DAY_UPDATED", "ENTITY_ALIAS", "ENTITY_ADDRESS", "SORT_KEY", "SORT_KEY_LAST_MOD", "NAME_ORIGINAL_SCRIPT", "SUBMITTED_ON"}, {"ENTITY.DATAID", "ENTITY.VERSIONNUM", "ENTITY.FIRST_NAME", "ENTITY.UN_LIST_TYPE", "ENTITY.REFERENCE_NUMBER", "ENTITY.LISTED_ON", "ENTITY.COMMENTS1", "ENTITY.LIST_TYPE", "ENTITY.LAST_DAY_UPDATED", "ENTITY.ENTITY_ALIAS", "ENTITY.ENTITY_ADDRESS", "ENTITY.SORT_KEY", "ENTITY.SORT_KEY_LAST_MOD", "ENTITY.NAME_ORIGINAL_SCRIPT", "ENTITY.SUBMITTED_ON"}),
#"Expanded INDIVIDUAL" = Table.ExpandTableColumn(#"Expanded ENTITY", "INDIVIDUAL", {"DATAID", "VERSIONNUM", "FIRST_NAME", "SECOND_NAME", "THIRD_NAME", "UN_LIST_TYPE", "REFERENCE_NUMBER", "LISTED_ON", "COMMENTS1", "DESIGNATION", "NATIONALITY", "LIST_TYPE", "LAST_DAY_UPDATED", "INDIVIDUAL_ALIAS", "INDIVIDUAL_ADDRESS", "INDIVIDUAL_DATE_OF_BIRTH", "INDIVIDUAL_PLACE_OF_BIRTH", "INDIVIDUAL_DOCUMENT", "SORT_KEY", "SORT_KEY_LAST_MOD", "NAME_ORIGINAL_SCRIPT", "FOURTH_NAME", "GENDER", "TITLE", "SUBMITTED_BY"}, {"INDIVIDUAL.DATAID", "INDIVIDUAL.VERSIONNUM", "INDIVIDUAL.FIRST_NAME", "INDIVIDUAL.SECOND_NAME", "INDIVIDUAL.THIRD_NAME", "INDIVIDUAL.UN_LIST_TYPE", "INDIVIDUAL.REFERENCE_NUMBER", "INDIVIDUAL.LISTED_ON", "INDIVIDUAL.COMMENTS1", "INDIVIDUAL.DESIGNATION", "INDIVIDUAL.NATIONALITY", "INDIVIDUAL.LIST_TYPE", "INDIVIDUAL.LAST_DAY_UPDATED", "INDIVIDUAL.INDIVIDUAL_ALIAS", "INDIVIDUAL.INDIVIDUAL_ADDRESS", "INDIVIDUAL.INDIVIDUAL_DATE_OF_BIRTH", "INDIVIDUAL.INDIVIDUAL_PLACE_OF_BIRTH", "INDIVIDUAL.INDIVIDUAL_DOCUMENT", "INDIVIDUAL.SORT_KEY", "INDIVIDUAL.SORT_KEY_LAST_MOD", "INDIVIDUAL.NAME_ORIGINAL_SCRIPT", "INDIVIDUAL.FOURTH_NAME", "INDIVIDUAL.GENDER", "INDIVIDUAL.TITLE", "INDIVIDUAL.SUBMITTED_BY"}),
#"Expanded INDIVIDUAL.DESIGNATION" = Table.ExpandTableColumn(#"Expanded INDIVIDUAL", "INDIVIDUAL.DESIGNATION", {"VALUE"}, {"INDIVIDUAL.DESIGNATION.VALUE"}),
#"Expanded INDIVIDUAL.NATIONALITY" = Table.ExpandTableColumn(#"Expanded INDIVIDUAL.DESIGNATION", "INDIVIDUAL.NATIONALITY", {"VALUE"}, {"INDIVIDUAL.NATIONALITY.VALUE"}),
#"Expanded INDIVIDUAL.LAST_DAY_UPDATED" = Table.ExpandTableColumn(#"Expanded INDIVIDUAL.NATIONALITY", "INDIVIDUAL.LAST_DAY_UPDATED", {"VALUE"}, {"INDIVIDUAL.LAST_DAY_UPDATED.VALUE"}),
#"Removed Columns" = Table.RemoveColumns(#"Expanded INDIVIDUAL.LAST_DAY_UPDATED",{"INDIVIDUAL.DATAID", "INDIVIDUAL.VERSIONNUM", "INDIVIDUAL.FIRST_NAME", "INDIVIDUAL.SECOND_NAME", "INDIVIDUAL.THIRD_NAME", "INDIVIDUAL.UN_LIST_TYPE", "INDIVIDUAL.REFERENCE_NUMBER", "INDIVIDUAL.LISTED_ON", "INDIVIDUAL.COMMENTS1", "INDIVIDUAL.DESIGNATION.VALUE"}),
Transform=Table.TransformColumns(#"Expanded INDIVIDUAL.LAST_DAY_UPDATED", {{"INDIVIDUAL.NATIONALITY.VALUE", each if Value.Is(_, type table) then _ else #table({"Item"}, {{_}} ) }} ),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(Transform, "INDIVIDUAL.NATIONALITY.VALUE"), each if _ is table then Table.ColumnNames(_) else {}))),
Expanded = Table.ExpandTableColumn(Transform, "INDIVIDUAL.NATIONALITY.VALUE", ColumnsToExpand, ColumnsToExpand)
in Expanded
Thank you.
Its hard to figure out even what the output should look like with all the different nested tables.
Does this work? If not provide sample desired output
let Source = Xml.Tables(Web.Contents("https://scsanctions.un.org/resources/xml/en/consolidated.xml")),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Attribute:dateGenerated", type datetimezone}}),
INDIVIDUALS = #"Changed Type"{0}[INDIVIDUALS],
INDIVIDUAL = INDIVIDUALS{0}[INDIVIDUAL],
#"Changed Type1" = Table.TransformColumnTypes(INDIVIDUAL,{{"DATAID", Int64.Type}, {"VERSIONNUM", Int64.Type}, {"FIRST_NAME", type text}, {"SECOND_NAME", type text}, {"THIRD_NAME", type text}, {"UN_LIST_TYPE", type text}, {"REFERENCE_NUMBER", type text}, {"LISTED_ON", type text}, {"COMMENTS1", type text}, {"SORT_KEY", type text}, {"SORT_KEY_LAST_MOD", type text}, {"FOURTH_NAME", type text}, {"GENDER", type text}, {"SUBMITTED_BY", type text}}),
#"Unpivoted Other Columns" = Table.UnpivotOtherColumns(#"Changed Type1", {"DATAID", "VERSIONNUM", "FIRST_NAME", "SECOND_NAME", "THIRD_NAME", "UN_LIST_TYPE", "REFERENCE_NUMBER", "LISTED_ON", "COMMENTS1", "SORT_KEY", "SORT_KEY_LAST_MOD", "FOURTH_NAME", "GENDER", "SUBMITTED_BY"}, "Attribute", "Value"),
Transform=Table.TransformColumns(#"Unpivoted Other Columns", {{"Value", each if Value.Is(_, type table) then _ else #table({"Item"}, {{_}} ) }} ),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(Transform, "Value"), each if _ is table then Table.ColumnNames(_) else {}))),
Expanded = Table.ExpandTableColumn(Transform, "Value", ColumnsToExpand, ColumnsToExpand)
in Expanded
For the other one
let Source = Xml.Tables(Web.Contents("https://scsanctions.un.org/resources/xml/en/consolidated.xml")),
#"Changed Type" = Table.TransformColumnTypes(Source,{{"Attribute:dateGenerated", type datetimezone}}),
ENTITIES = #"Changed Type"{0}[ENTITIES],
ENTITY = ENTITIES{0}[ENTITY],
#"Changed Type1" = Table.TransformColumnTypes(ENTITY,{{"DATAID", Int64.Type}, {"VERSIONNUM", Int64.Type}, {"FIRST_NAME", type text}, {"UN_LIST_TYPE", type text}, {"REFERENCE_NUMBER", type text}, {"LISTED_ON", type text}, {"COMMENTS1", type text}, {"SORT_KEY", type text}, {"SORT_KEY_LAST_MOD", type text}, {"NAME_ORIGINAL_SCRIPT", type text}, {"SUBMITTED_ON", type text}}),
#"Unpivoted Columns" = Table.UnpivotOtherColumns(#"Changed Type1", {"DATAID", "VERSIONNUM", "FIRST_NAME", "UN_LIST_TYPE", "REFERENCE_NUMBER", "LISTED_ON", "COMMENTS1", "SORT_KEY", "SORT_KEY_LAST_MOD", "NAME_ORIGINAL_SCRIPT", "SUBMITTED_ON"}, "Attribute", "Value"),
Transform=Table.TransformColumns( #"Unpivoted Columns", {{"Value", each if Value.Is(_, type table) then _ else #table({"Item"}, {{_}} ) }} ),
ColumnsToExpand = List.Distinct(List.Combine(List.Transform(Table.Column(Transform, "Value"), each if _ is table then Table.ColumnNames(_) else {}))),
Expanded = Table.ExpandTableColumn(Transform, "Value", ColumnsToExpand, ColumnsToExpand)
in Expanded
Related
I have 2 lambdas that do the exact same thing, however, they are both written using different langages.
1st lambda - runs on a node.js environment, when I create my arguments to putItem, as follows:
const args = {
id: "my id",
__typename: "a type name",
_version: 1,
_lastChangedAt: now.toISOString(),
createdAt: now.toISOString(),
updatedAt: fields.LastModifiedDate
}
var recParams = {
TableName: dynamoTable,
Key: {
"id": Id
},
Item: args,
ReturnValues: "ALL_OLD"
};
and then I use the docClient to insert the row. Everything works fine, all the properties are populated in my dynamo row.
I have the exact same written in Golang:
item := RecentItem{
Id: "some Id",
_version: 1,
__typename: "a type name",
_lastChangedAt: currentTime.UTC().Format("2006-01-02T15:04:05-0700"),
createdAt: currentTime.UTC().Format("2006-01-02T15:04:05-0700"),
updatedAt: currentTime.UTC().Format("2006-01-02T15:04:05-0700"),
}
av, err := dynamodbattribute.MarshalMap(item)
input := &dynamodb.PutItemInput{
Item: av,
TableName: aws.String(tableName),
}
Everything ALMOST works, the item is inserted, but I am missing all the properties except for the id.
Structure declaration :
type RecentItem struct {
Id string `json:"id"`
_version int `json:"_version"`
_lastChangedAt string `json:"_lastChangedAt"`
createdAt string `json:"createdAt"`
updatedAt string `json:"updatedAt"`
}
Not sure why in Go my dynamoDb row is missing properties. Am I missing something?
Properties other than Id must be exported, i.e, started with an Upper case:
type RecentItem struct {
ID string `dynamodbav:"id"`
Version int `dynamodbav:"_version"`
LastChangedAt string `dynamodbav:"_lastChangedAt"`
CreatedAt string `dynamodbav:"createdAt"`
UpdatedAt string `dynamodbav:"updatedAt"`
}
I am trying to update my link to pull from my own One Drive account. I tried opening the sheet online and copying the source, and I tried copying the share link and posting it. I am still unable to make it work without getting the HTML error shown.
= Excel.Workbook(Web.Contents("https://onedrive.live.com/download?&resid=7BA5E7B92829DE18!1263"), null, true)
This works fine for me
let Source = Excel.Workbook(Web.Contents("https://onedrive.live.com/download?&resid=7BA5E7B92829DE18!1263"), null, true),
Sheet1_Sheet = Source{[Item="Sheet1",Kind="Sheet"]}[Data],
#"Promoted Headers" = Table.PromoteHeaders(Sheet1_Sheet, [PromoteAllScalars=true]),
#"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{{"ABREV", type text}, {"NICKNAME", type text}, {"Column3", type text}, {"Name", type text}, {"Standard Name", type text}, {"B", type text}, {"T", type text}, {"TEAM", type text}, {"Batting POS", type text}, {"POS", type text}, {"Standard Name_1", type text}, {"T2", type any}, {"Column13", type any}, {"Column14", type any}, {"Column15", type any}, {"Column16", type any}, {"SAVANT Table", type text}, {"Column18", type text}, {"Column19", type any}, {"Column20", type any}, {"Column21", type any}, {"Column22", type text}, {"Column23", type text}, {"Column24", type any}, {"Column25", type any}})
in #"Changed Type"
how to recall this stryc?
struct User {
var name: String
var street: String
var city: String
var postalCode: String
func printAddress() -> String {
return """
\(name)
\(street)
\(city)
\(postalCode)
"""
}
}
I expect to have an address in different lines like the method, for example
Well Smith
streetnumber this one
lalaland
19890
but result comes back in struct form
Not sure what you mean with "recall struct" but if you want to print it in the expected format simply use printAddress()
let user = User(name: "name", street: "street", city: "city", postalCode: "postalCode")
print(user.printAddress())
Output:
name
street
city
postalCode
So I have an Struct that holds data that has a AddedByUser which links to my User Struct.
What I want to be able to do it remove the UserLevel from the AddedByUser
Now I want to be able to do it from this function only, so using the json:"-" is not an option. That would remove it from all json output. I only want to remove it form this one function.
I should also say that these are Gorm models and when I have been trying to remove the 10 option (UserLevels) it only removes the outer data set not the UserLevel from all of the data.
{
"ID": 1,
"CreatedAt": "2019-01-08T16:33:09.514711Z",
"UpdatedAt": "2019-01-08T16:33:09.514711Z",
"DeletedAt": null,
"UUID": "00000000-0000-0000-0000-000000000000",
"Title": "title000",
"Information": "info999",
"EventDate": "2006-01-02T15:04:05Z",
"AddedByUser": {
"ID": 2,
"CreatedAt": "2019-01-08T15:27:52.435397Z",
"UpdatedAt": "2019-01-08T15:27:52.435397Z",
"DeletedAt": null,
"UUID": "b019df80-a7e4-4397-814a-795e7e84b4ca",
"Firstname": "Me",
"Surname": "admin",
"Password": "....",
"Email": "admin#email.co.uk",
"UserLevel": {
"ID": 0,
"CreatedAt": "0001-01-01T00:00:00Z",
"UpdatedAt": "0001-01-01T00:00:00Z",
"DeletedAt": null,
"LevelTitle": "",
"UserLevel": null
},
So this is what I have tried,
data := []models.MyData{}
data = append(data[0:2])
I have about 14 results, with out the append it loads all the results but with this is only loads two results. The idea was to remove either UpdateAt or Title. As I am not sure if the gorm model information is all 0 or if the slice sees them as 0,1,2,3,4 etc.
I have also tried to range over the slice of models, while I can access each of the sections, I can not seem to find a simple method to remove data by name from a struct? Maps seem to have that but not structs which I am not sure why?
Thanks.
UPDATE
This is the model I am using:
//Model
type MyData struct {
gorm.Model
UUID uuid.UUID
Title string
Information string
EventDate time.Time
AddedByUser Users `gorm:"ForeignKey:added_by_user_fk"`
AddedByUserFK uint
}
//Users Model
type Users struct {
gorm.Model
UUID uuid.UUID
Firstname string
Surname string
Password string
Email string
UserLevel UserLevels `gorm:"ForeignKey:user_level_fk" json:",omitempty"`
UserLevelFK uint
}
As mentioned in the comments, you cannot remove fields from a struct value, because that would yield a value of a different type.
However, you can set fields to their zero value. Combined with the omitempty JSON tag, you can exclude fields from the JSON encoding. To make this work properly, you have to change the UserLevel field to a pointer type (otherwise you end up with empty objects in the JSON document).
Types shortened for brevity:
package main
import (
"encoding/json"
"fmt"
)
type MyData struct {
Title string
AddedByUser Users
}
type Users struct {
ID int
UserLevel *UserLevels `json:",omitempty"` // pointer type with omitempty
}
type UserLevels struct {
LevelTitle string
}
func main() {
var x MyData
x.Title = "foo"
x.AddedByUser.ID = 2
x.AddedByUser.UserLevel = &UserLevels{}
f(x)
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("main:\n" + string(b))
}
func f(x MyData) {
// "unset" UserLevel. Since we are receiving a copy of MyData, this is
// invisible to the caller.
x.AddedByUser.UserLevel = nil
b, _ := json.MarshalIndent(x, "", " ")
fmt.Println("f:\n" + string(b))
}
// Output:
// f:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2
// }
// }
// main:
// {
// "Title": "foo",
// "AddedByUser": {
// "ID": 2,
// "UserLevel": {
// "LevelTitle": ""
// }
// }
// }
Try it on the playground: https://play.golang.org/p/trUgnYamVOA
Alternatively, you can define new types that exclude the AddedByUser field. However, since this field isn't at the top level, this is a lot of work, and it's easy to forget to update those types when new fields are added to the original types.
If the field were at the top level, the compiler would do most of the work for you, because types that only differ in their field tags can be directly converted to one another:
type MyData struct {
ID int
Title string
}
func main() {
var x MyData
x.ID = 1
x.Title = "foo"
f(x)
}
func f(x MyData) {
type data struct { // same as MyData, except the field tags
ID int
Title string `json:"-"`
}
b, _ := json.MarshalIndent(data(x), "", " ")
fmt.Println("main:\n" + string(b))
}
I'm receiving this error in my //Fill Objects section. I cannot seem to find a solution for "Expected Element Type". I've only seen solutions for "Expected Argument Type". Not sure what the difference is or how to fix this. Never came across this error before.
//User Profile Info
struct userObject {
var profilePicture: UIImage
var name: String
var description: String
var cityState: String
}
//Fill Objects
var userArray: [userObject] = ["profilePicture", "name", "description", "cityState"]
You need to do this for declaring an array.
let userArray: NSArray = ["profilePicture", "name", "description", "cityState"]
Try this.