Phantom messages in Exchange Calendar folder - exchange-server

I'm developing an IMAP client, and testing it against a Microsoft Exchange server. When I select the Calendar mailbox, UID SEARCH suggests that there are messages in the mailbox, but UID FETCH cannot fetch them:
S: * OK The Microsoft Exchange IMAP4 service is ready.
...
C: 38 SELECT "Calendar"
S: * 13 EXISTS
S: * 0 RECENT
S: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
S: * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
S: * OK [UIDVALIDITY 37122] UIDVALIDITY value
S: * OK [UIDNEXT 190] The next unique identifier value
S: 38 OK [READ-WRITE] SELECT completed.
C: 39 UID SEARCH OR UNSEEN FLAGGED
C: 40 UID SEARCH SEEN UNFLAGGED SINCE 19-Oct-2014
S: * SEARCH
S: 39 OK SEARCH completed.
S: * SEARCH 90 183
S: 40 OK SEARCH completed.
C: 41 UID FETCH 90,183 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: 41 OK FETCH completed.
I presume that Calendar is some kind of virtual mailbox, but this behaviour doesn't make sense to me — why would the server return the message uids in UID SEARCH if it's not possible to FETCH them? Is there any way to fetch these messages, or somehow manipulate them using IMAP?
Update: After being upgraded to 2010 SP3, it seems like the Exchange server had a change of heart. Now it at least returns something in response to the FETCH command:
C: 120 UID FETCH 773,948 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: * 16 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 773" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "05-Mar-2015 17:22:02 +0000" FLAGS (\Seen) BODY[] {783}
S: <783 bytes omitted>
S: UID 773)
S: * 17 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 948" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "11-Mar-2015 17:56:04 +0000" FLAGS (\Seen) BODY[] {747}
S: <747 bytes omitted>
S: UID 948)
S: 120 OK FETCH completed.
The messages look like this:
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
From: Microsoft Exchange Server 2010
To: ███████████ <████████#██████████.com>
Subject: Retrieval using the IMAP4 protocol failed for the following
message: 773
Content-Transfer-Encoding: quoted-printable
The server couldn't retrieve the following message:
Subject: "████ █████ ███████ ██ ████ ███ █████ ███████.. ███ ███ ██████ ███=
██!"
From: "███████████ ██████████" ("/O=3D████████/OU=3D████████ █████████████=
█ █████ (███████████████)/CN=3D██████████/CN=3D███████████ █████████████")
Sent date: 3/5/2015 5:22:02 PM
The message hasn't been deleted. You might be able to view it using either =
Outlook or Outlook Web App. You can also contact the sender to find out wha=
t the message says.=

Sounds like a server bug. It is not a secret that Exchange "implaments IMAP" in order to be able to tick a required checkbox when evaluating tenders. You will see other bugs in there as well.
It might be interesting to get a list of all UIDs which that mailbox allegedly contains, and to try fetching them one-by-one to see if at least some of them return some usable value. However, considering that this is a mailbox called "Calendar", chances are that it's just some random Exchange export of data which only make sense via MAPI. Random googling results in https://technet.microsoft.com/en-us/library/aa998606%28v=exchg.150%29.aspx which suggests that Exchange can be configured to offer these calendar data in various formats. Maybe your server is "misconfigured".
TL;DR: it's Exchange. Don't waste time on it.

Related

How I encode the length 32 []byte nonce from golang to hit rust's jormungandr rpc server?

If you look at how this go-cardano-client is making it's handshake request payload:
https://github.com/gocardano/go-cardano-client/blob/master/shelley/handshake.go#L64
versionTable.Add(cbor.NewPositiveInteger8(1), cbor.NewPositiveInteger(764824073))
versionTable.Add(cbor.NewPositiveInteger16(32770), cbor.NewPositiveInteger(764824073))
versionTable.Add(cbor.NewPositiveInteger16(32771), cbor.NewPositiveInteger(764824073))
But the grpc generated struct is:
type HandshakeRequest struct {
// Nonce for the server to authenticate its node ID with.
Nonce []byte `protobuf:"bytes,1,opt,name=nonce,proto3" json:"nonce,omitempty"`
}
And this []byte needs to come through for the nonce referenced:
https://github.com/input-output-hk/jormungandr/blob/master/jormungandr/src/network/service.rs#L60
It's length 32:
https://github.com/input-output-hk/jormungandr/blob/master/jormungandr/src/network/client/connect.rs#L58
https://github.com/input-output-hk/jormungandr/blob/6f324b706a13273afb6a0808e589735020bb59da/jormungandr/src/network/mod.rs#L73
So this line in the golang code:
versionTable.Add(cbor.NewPositiveInteger8(1), cbor.NewPositiveInteger(764824073))
can't be length 32 []byte right? How do I encode this:
req := HandshakeRequest{}
req.Nonce = []byte{}
for i := 0; i < 32; i++ {
req.Nonce = append(req.Nonce, byte(rand.Intn(256)))
}
into this versionTable "params"?
see also and proto
Edit: You seem to assume that the the handshake from gocardano/go-cardano-client and the one described in node.proto somehow are related to the same implementation. Actually, I don't think they do.
The TCP-based handshake follows the Shelley protocol specs and sends a payload with the encoded versionTable. The gRPC-based HandshakeRequest instead is, as you also considered, just a nonce. There's nothing in the proto schema that hints to the Shelley protocol. The comments on the Nonce field also say that quite explicitly: "Nonce for the server to authenticate its node ID with."
So it would be a bit strange to assume that this nonce and the versionTable payload have anything in common at all.
Edit 2: In addition, it seems the "Jormungandr" rust node implementation does not support Shelley at all, so when you say you can't connect to the nodes in the relay topology, I think you shouldn't look for answers in the Jormungandr repository. Instead, I think the relays run the Haskell implementations of the Ouroboros network.
Now as for why you can't connect, the go-cardano client panics on some unchecked type assertions, because after the QueryTip Shelley message chainSyncBlocks.RequestNext, the relay servers respond with a different mini-protocol altogether, transactionSubmission.msgRequestTxIds as shown by running the client with TCP and tracing the messages:
MiniProtocol: 4 / MessageMode: 1 / f1bb7f80800400058400f50003
Array: [4]
PositiveInteger8(0)
False
PositiveInteger8(0)
PositiveInteger8(3)
You'll also have the same result when sending a sync chain request with MiniProtocol 2 (ChainSyncHeaders). I checked the Shelley protocol specs but couldn't find an explicit indication about why the server would switch protocols... Unfortunately I'm not familiar enough with Haskell to gain further insight from the Ouroboros sources.
In the unexpected case that the nonce in the proto HandshakeRequest is indeed related to the Shelley protocol, its content might be the CBOR array in your linked Cardano client (speculations follows):
arr := cbor.NewArray()
arr.Add(cbor.NewPositiveInteger8(handshakeMessagePropose))
versionTable := cbor.NewMap()
arr.Add(versionTable)
versionTable.Add(...)
versionTable.Add(...)
versionTable.Add(...)
return []cbor.DataItem{arr}
By inspecting the client where the handshake request is used, we can see:
messageResponse, err := c.queryNode(multiplex.MiniProtocolIDMuxControl, handshakeRequest())
and then in queryNode:
sdu := multiplex.NewServiceDataUnit(miniProtocol, multiplex.MessageModeInitiator, dataItems)
...
c.socket.Write(sdu.Bytes())
The sdu.Bytes() method serializes the entire payload, in particular:
// EncodeList return CBOR representation for each item in the list
func EncodeList(list []DataItem) []byte {
result := []byte{}
for _, item := range list {
result = append(result, item.EncodeCBOR()...)
}
return result
}
The EncodeCBOR() method is implemented by both the Array and the Map used in the handshakeRequest() []cbor.DataItem function. Watch out that the handshake function returns a slice []cbor.DataItem which contains one Array item which contains (as documented) the handshakeMessagePropose and the versionTable map.
If you carefully follow how the serialization proceeds, you'll eventually obtain the breakdown of the byte array — hereafter in decimal:
[130 0 163 1 26 45 150 74 9 25 128 2 26 45 150 74 9 25 128 3 26 45 150 74 9]
Where:
130 is the array data item prefix
0 is the handshakeMessagePropose
163 is the map data item prefix
and the subsequent bytes are the versionTable map
It is 25 bytes in total. At this point, I don't know if the multiplex wrapper built in queryNode function is part of the nonce or not. With the full wrapper, the length of the serialized byte array goes up to 33. So excluding some control bits or whatnot, this might be what you're supposed to write into the HandshakeRequest.Nonce.

Outlook SMTP Oauth Send - Authentication Unsuccessful

I have been trying to implement email functionality in a legacy app using the Legacy Mail API to add OAuth support after getting a token through the Device Code Flow starting with the Microsoft Sample Project.
Along that route I have added SMTP.Send and many other API permissions to find the missing piece. (Including { "User.Read", "User.ReadBasic.All", "SMTP.Send", "offline_access", "Mail.Send" } in fear of missing one)
I have been testing with the MailKit library to build a proof of concept.
So far I have the following Code Snippet that fails after trying to authenticate.
public void SendSmtpMessageAsync(string id, string accessToken)
{
var message = new MimeMessage();
message.From.Add(new MailboxAddress("From Name", "From Address#Example.com"));
message.To.Add(new MailboxAddress("To Name", "To Address#Example.com"));
message.Subject = "How you doin'?";
message.Body = new TextPart("plain")
{
Text = #"Test Email Content"
};
using (var client = new SmtpClient(new ProtocolLogger(Console.OpenStandardOutput())))
{
try
{
client.Connect("smtp.office365.com", 587, SecureSocketOptions.StartTls);
var oauth2 = new SaslMechanismOAuth2(id, accessToken);
var temp = client.AuthenticationMechanisms;
client.Authenticate(oauth2);
client.Send(message);
client.Disconnect(true);
}
catch (Exception ex)
{
Console.WriteLine(ex);
}
}
}
MailKit Log
I have enabled logging and gotten a log that shows the client connecting than sending the token to authenticate but the authentication failing.
Connected to smtp://smtp.office365.com:587/?starttls=always
S: 220 MW3PR05CA0003.outlook.office365.com Microsoft ESMTP MAIL Service ready at Mon, 25 May 2020 21:31:07 +0000
C: EHLO [192.168.0.7]
S: 250-MW3PR05CA0003.outlook.office365.com Hello [<<My IP>>]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-STARTTLS
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 SMTPUTF8
C: STARTTLS
S: 220 2.0.0 SMTP server ready
C: EHLO [192.168.0.7]
S: 250-MW3PR05CA0003.outlook.office365.com Hello [<<My IP>>]
S: 250-SIZE 157286400
S: 250-PIPELINING
S: 250-DSN
S: 250-ENHANCEDSTATUSCODES
S: 250-AUTH LOGIN XOAUTH2
S: 250-8BITMIME
S: 250-BINARYMIME
S: 250-CHUNKING
S: 250 SMTPUTF8
C: AUTH XOAUTH2 <<Token omitted but I have confirmed that it is Base64 encoded and
in the format of base64("user=" + userName + "^Aauth=Bearer " + accessToken + "^A^A")>>
S: 535 5.7.3 Authentication unsuccessful [MW3PR05CA0003.namprd05.prod.outlook.com]
MailKit.Security.AuthenticationException: 535: 5.7.3 Authentication unsuccessful
[MW3PR05CA0003.namprd05.prod.outlook.com]
Any direction or resources would be appreciated since most existing posts are from pre-2020 when Legacy SMTP support was added. Additionally, if you see any misunderstanding let me know so I can do some additional reading.
After lots of searching and trying to talk to Microsoft I was pointed in the direction of this answer of a different post. (I had adding a POP3 call that was in the same format as the SMTP call in the OP.) The answer said to include the scope https://outlook.office.com/POP.AccessAsUser.All so after replacing the POP.AccessAsUser.All I had before the SMTP and POP3 calls worked using modern authentication.
Additionally, now that I know the answer I see it is documented in Microsoft's Docs Authenticate an IMAP, POP or SMTP connection using OAuth verifying that this is the proper solution.
Make sure to specify the full scopes, including Outlook resource URLs, when authorizing your application and requesting an access token.
| Protocol | Permission scope string
|-----------|-------------------------------------
| IMAP | https://outlook.office.com/IMAP.AccessAsUser.All
| POP | https://outlook.office.com/POP.AccessAsUser.All
| SMTP AUTH | https://outlook.office.com/SMTP.Send
Hope this helps anyone else facing this same issue
After spending hours and days trying to solve the Authentication unsuccessful error here's an important thing I discovered:
When requesting "scopes" during oAuth2 flow make sure you don't request any scopes, other than offline_access and https://outlook.office.com/SMTP.Send
In my case the problem was - I was also requesting the User.Read scope
I had a similar problem, which was cause by my misconfiguration of Microsoft 365. Please see MailKit unsuccessful SMTP OAuth with Microsoft 365 server for details. Hope this helps.

How do i know, which Peer did the Transaction in Hyperledger Fabric Go?

I am working on getting a transaction id info, which will give the peer details for the transaction. Currently, I am able to retrieve the History for a key, which gives me the list of transaction committed to that key.
MyCode:
historyRes, err := stub.GetHistoryForKey(userNameIndexKey)
if err != nil {
return shim.Error(fmt.Sprintf("Unable to get History key from the ledger: %v", err))
}
for historyRes.HasNext() {
history, errIt := historyRes.Next()
if errIt != nil {
return shim.Error(fmt.Sprintf("Unable to retrieve history in the ledger: %v", errIt))
}
deleted := history.GetIsDelete()
ds := strconv.FormatBool(deleted)
fmt.Println("History TxId = "+history.GetTxId()+" -- Delete = "+ds)
}
Output
History TxId = 78c8d17c668d7a9df8373fd85df4fc398388976a1c642753bbf73abc5c648dd8 -- Deleted = false
History TxId = 102bbb64a7ca93367334a8c98f1f7be17e6a8d5277f0167c73da47072d302fa3 -- Deleted = true
But I don't know, which peer did this transaction. Is there any API available in fabric-sdk-go to retrieve peer info for a transaction id.
please suggest me some solution.
The call stub.GetHistoryForKey(userNameIndexKey) will query the state database and not the ledger (channel). The information about the identity who made the transaction is stored in the block.
I have implemented the same thing with NodeJS SDK. However, Go SDK too contains similar API calls. The following steps worked for me:
Using your SDK, get the transactionId
Use the SDK function for querying block by transactionId. References here.
At this step, you'll get the block. Now the identity of the submitter is located within this block. Hints: Payload -> Header -> Signature Header -> Creater -> IdBytes.
These identity bytes are the X509 certs of the submitter. You can read the certificate info to find out which member did submit this transaction. The subject and OUs will indicate the Organization of the peer that did the transaction.

Delphi and Walmart Bulk API Upload - Internal server error

I am using Delphi XE5 and I am trying to send Items to Walmart using the Bulk API Upload but I am struggling with making proper connection with the server of Walmart.
I have already checked some similar problems here like Walmart API - payload mime type issue
The strangest thing of all is that the feed is actually accepted by Walmart (I see it in the list of feeds) but I receive Internal server error and then meaningless characters (probably type of encoding or just kind of error) like
(#$D#$A#$1F#$008B#8#0#0#0#0#0#0#3#$D'A±'#$D#$0080'0'#$C#4'AU~'#2'v ?d'#$0087#4'?'#1#$0091'OE'#$E#$008A'O'#$1E'®?O?'#$18#$0091#$A#$0091'M^'#$C'?;'#$1D'G'#$A'dR±U?oip?m'#$1A'AVS'#4#$16'R?iR'#$1B#$0095'RO?}u'#0'enIjM'#0#0#0)
So I use Indy IdHTTP and I have tried using TIdMultipartFormDataStream and also creating the payload on my own both with receiving the internal server error.
I presume I am breaking their parser somehow because if I miss one CR LF (#13#10) at the beginning I get a specific error
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ns2:errors xmlns:ns2="http://walmart.com/"><ns2:error><ns2:code>SYSTEM_ERROR.GMP_GATEWAY_API</ns2:code><ns2:field>PDR-0012</ns2:field><ns2:description></ns2:description><ns2:info>System encountered some internal error.</ns2:info><ns2:severity>ERROR</ns2:severity><ns2:category>SYSTEM</ns2:category><ns2:causes/><ns2:errorIdentifiers/></ns2:error></ns2:errors>
If I add an extra CRLF at the end then I get another error stating that there is more than one feed in the request.
So I really presume that there is something which makes that server / parser return the Internal error but I don't get what and again I repeat that the feed actually gets accepted and then processed by the server in the case I receive this error.
Here is an example request with a payload created on my own:
Headers
POST /v3/feeds?feedType=item HTTP/1.1
Content-Type: multipart/form-data; boundary=qwerty
Content-Length: 5077
WM_SVC.NAME: Walmart Marketplace
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.TIMESTAMP: 1528364885909
WM_SEC.AUTH_SIGNATURE: nSHgqzPOtzSR4wJU+U/vQJk+rk6Ke2QwodTHjzkjau2BonXZxiU9e+3NFPzaat2OUyc+vr0jqRk0H0QWTSC21PrI87mvqei5UJCJwNiIx0zVjAGpxsnIuvtIKkQsBpuUAa8C6SjTiTpDRsNt4IOxrk+tLWxlwQubWVCV+009a6o=
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE: 0f3e4dd4-0514-4346-b39d-af0e00ea066d
Host: marketplace.walmartapis.com
Accept: application/xml, */*
Accept-Encoding: identity
Accept-Language: en-US
User-Agent: Mozilla/3.0 (compatible; Indy Library)
Request
--qwerty
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{694A5564-7E8E-46C7-B772-C944A8C9CF99}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
<SkuUpdate>Yes</SkuUpdate>
<productName>Caramel Cantina 4 Pack Sleeping Nursing Cross Front Maternity Bra (Large, White)</productName>
<ProductIdUpdate>Yes</ProductIdUpdate>
<category>
<ClothingCategory>
<Clothing>
<shortDescription>Stretchy comfort cross front sleep bra by Caramel Cantina. Whether for nursing / maternity or for wearing as a sleep bra (or both nursing and sleep bras) - you&apos;ll experience a wire free regular back strap sleep bra. This mid-weight fabric but light support wireless bra is great for all day wear and comfort. The straps are wider than a standard bra - offering a longer range of wear time. Great for wearing indoors (ie when you&apos;re kicking around the home or going to bed). These are padless - most moms use their own favorite nursing pads with these (though no pads are included). The front cups have two layers of fabric to add a bit more support (though we still call them our light support comfort bra). The criss cross front style allows for easy access while nursing your baby. 92% Nylon 8% Spandex fabric washes up nicely. Cool wash and cool dry. Hang dry for longer life. The straps narrowest point at the shoulders are 1.25-1.5 inches (depending on the size). The bras are super stretchy - if you&apos;re looking for something super supportive (ie - they pull you in really tight - more like a sports bra) these probably aren&apos;t what you&apos;re looking for. Not recommended for heavy sports activity. Best worn for activities such as walking, sleeping, lounging, light activity, running errands, etc.</shortDescription>
<keyFeatures>
<keyFeaturesValue>4 Pack Includes 4 Bras of the Same Design - Color Option Above</keyFeaturesValue>
<keyFeaturesValue>Soft Cup - Front Double Layered - No padding - Wireless</keyFeaturesValue>
<keyFeaturesValue>Super Soft Sleep In Style Bra - Great for Nursing or Sleeping</keyFeaturesValue>
<keyFeaturesValue>Criss Cross Front Design Gives Easy Nursing Access</keyFeaturesValue>
<keyFeaturesValue>Regular Straps - Scoop Back Design - Strap Width 1.25-1.5 Inches Depending on Size</keyFeaturesValue>
</keyFeatures>
<brand>Caramel Cantina</brand>
<mainImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</mainImageUrl>
<productSecondaryImageURL>
<productSecondaryImageURLValue>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-wht-m1.jpg</productSecondaryImageURLValue>
</productSecondaryImageURL>
<color>White</color>
<clothingSize>Large</clothingSize>
<variantGroupId>cc-a1-4pk-parent</variantGroupId>
<variantAttributeNames>
<variantAttributeName>color</variantAttributeName>
<variantAttributeName>clothingSize</variantAttributeName>
</variantAttributeNames>
<countryOfOriginTextiles>Imported</countryOfOriginTextiles>
<fabricCareInstructions>
<fabricCareInstruction>Machine wash cold gentle cycle with like colors. Hang Dry. Do Not Bleach. Do Not Iron.</fabricCareInstruction>
</fabricCareInstructions>
<multipackQuantity>1</multipackQuantity>
<countPerPack>4</countPerPack>
<count>4</count>
<pattern>Solid</pattern>
<material>Nylon; Spandex</material>
<gender>Female</gender>
<ageGroup>
<ageGroupValue>Adult</ageGroupValue>
</ageGroup>
<clothingSizeGroup>Maternity</clothingSizeGroup>
<isSet>No</isSet>
<requiresTextileActLabeling>Yes</requiresTextileActLabeling>
<clothingTopStyle>Sleep Bra</clothingTopStyle>
<upperBodyStrapConfiguration>Regular</upperBodyStrapConfiguration>
<braStyle>Nursing</braStyle>
<clothingStyle>Maternity</clothingStyle>
<clothingFit>Maternity</clothingFit>
<isMaternity>Yes</isMaternity>
<swatchImages>
<swatchImage>
<swatchVariantAttribute>color</swatchVariantAttribute>
<swatchImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</swatchImageUrl>
</swatchImage>
</swatchImages>
</Clothing>
</ClothingCategory>
</category>
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
--qwerty--
The response which I get is as following
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$c88958a9
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528364905439
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528364905567
WM_SVC.SERVER_IP: 10.65.34.219
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-4-96141283
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Expires: Thu, 07 Jun 2018 09:48:25 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 09:48:25 GMT
Connection: close
And then those characters from above. Note the Cteonnt-Length: 77 header, it is not my TYPO it just comes in this way.
According to the support they have success using the following headers and ask me to use them, but I don't see any difference to what I make
WM_SVC.NAME:Walmart Marketplace
WM_QOS.CORRELATION_ID:123456abcdef
WM_SEC.AUTH_SIGNATURE:TnjevCf+voP9dmnafuCTFruwq6leBuAihSvag89WLieDRBsz7aULxgEqV71ZjIp572wVYPI07y6tdMutNLklDGwxvNdlJ2Q2xGvUIqjVPtqlhdcWvsmgqdpio7puQ4G03q1lReWzTquKecDEbB1ztH6ukj9F5rMe7d7PH8QkFsY=
WM_SEC.TIMESTAMP:1528152581896
WM_CONSUMER.ID:16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE:0f3e4dd4-0514-4346-b39d-af0e00ea066d
Accept:application/xml
Host:marketplace.walmartapis.com
Content-Type:multipart/form-data
I already make successful communication with eBay, Amazon, PayPal, Magento and many more, but here obviously I am missing something and really don't know how to solve it.
Thank you for reading this, and hopefully someone can give a clue what is going on here.
EDIT I am adding some Delphi code as required
Version 1 without TIdMultipartFormDataStream
function TdmSyncWallmart.CreateItemFeed(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime; var aReqID: String): TStringStream;
var
aXML: TNativeXML;
aNode: TXmlNode;
aItemNode: TXmlNode;
aCatNode: TXmlNode;
aTmpNode: TXmlNode;
aShortDesc: String;
i: Integer;
aIDItems: String;
begin
Result := nil;
dstListPlatformsSyncItemsWallmart.Close;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#IDPlatforms').Value := aIDPlatforms;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#LastChanged').Value := aLastChanged;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#CurrDate').Value := aCurrDate;
dstListPlatformsSyncItemsWallmart.Open;
if dstListPlatformsSyncItemsWallmart.IsEmpty then
Exit;
aReqID := GenGuid;
aXML := TNativeXml.CreateName('MPItemFeed');
try
aXML.Declaration.AttributeAdd('standalone', 'yes');
aXML.Root.AttributeAdd('xmlns:ns2', 'http://walmart.com/');
//Header
aNode := aXML.Root.NodeNew('MPItemFeedHeader');
with aNode.NodeNew('version') do
Value := '3.1';
with aNode.NodeNew('requestId') do
Value := aReqID;
with aNode.NodeNew('mart') do
Value := 'WALMART_US';
//... some code for generating the XML
Result := TStringStream.Create;
aXML.XmlFormat := xfReadable;
Result.WriteString('--qwerty'#13#10#13#10{'Content-Disposition: form-data; filename="wallmartreq.xml"'#13#10}+aXML.WriteToString +'--qwerty--'#13#10);
Result.Position := 0;
finally
aXML.Free;
end;
end;
function TdmSyncWallmart.CreateHTTP(aContentType: String): TIdHTTP;
begin
Result := TIdHTTP.Create(nil);
Result.ConnectTimeout:=5000;
Result.ReadTimeout:=20000;
Result.ProtocolVersion:=pv1_1;
Result.HTTPOptions := [hoForceEncodeParams, hoKeepOrigProtocol];
Result.HandleRedirects:=True;
Result.IOHandler := SSLHandler;
Result.Request.Accept:='application/xml, */*';
Result.Request.AcceptLanguage:='en-US';
Result.Request.ContentType:=aContentType;
// Result.Request.CharSet:='utf-8';
Result.Intercept := IdLogEvent1;
end;
procedure TdmSyncWallmart.AddCustomHeaders(aHTTP: TIdHTTP; aSign,
aTimeStamp: String);
begin
aHTTP.Request.CustomHeaders.AddValue('WM_SVC.NAME', 'Walmart Marketplace');
aHTTP.Request.CustomHeaders.AddValue('WM_QOS.CORRELATION_ID', GenGuid);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.TIMESTAMP', aTimeStamp);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.AUTH_SIGNATURE', aSign);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.ID', dstPlatformsUserName.AsString);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.CHANNEL.TYPE', dstPlatformsPassword.AsString);
end;
//some code for debugging purposes
procedure TdmSyncWallmart.IdLogEvent1Send(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_req_txt.txt');
finally
aList.Free;
end;
end;
procedure TdmSyncWallmart.IdLogEvent1Receive(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_Resp.txt');
finally
aList.Free;
end;
end;
Actual call
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data;boundary=qwerty');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := CreateItemFeed(aIDPlatforms, aLastChanged, aCurrDate, aReqID);
if aStrm <> nil then
begin
try
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
EDIT2
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
aIdStream: TIdMultipartFormDataStream;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := TStringStream.Create();
begin
try
aIdStream := TIdMultiPartFormDataStream.Create;
aIdStream.AddFile('file', 'e:\wm_req.xml', 'multipart/form-data');
aIdStream.Position := 0;
astrm.LoadFromStream(aIdStream);
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
This is the sent data
----------060718213723831
Content-Disposition: form-data; name="file"; filename="wm_req.xml"
Content-Type: multipart/form-data
Content-Transfer-Encoding: binary
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{164726BF-288E-4953-A1DA-1EC7595C5725}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
..... the other part of the XML
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
----------060718213723831--
Returned data captured by the intercepting part is the same as the one already posted with again the Cteonnt-Length: 77 header, but I guess if I succeed in reading the gzip data, I can reach an error which can be forwarded to the support.
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {CB0208AB-2135-4A12-B45F-DD2889273971}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$2ca3f3b6
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528396669253
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528396669365
WM_SVC.SERVER_IP: 10.65.34.2
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-1-96141274
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Vary: Accept-Encoding
Expires: Thu, 07 Jun 2018 18:37:49 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 18:37:49 GMT
Connection: close
EDIT3
For my Edit1, I am not adding Content-Disposition at all (it is commented) - with so many tests I have left it in my post here, so sorry for the confusion.
For Edit2 - thanks for pointing this out, actually I made my first tests without transforming into TStringStream, but I wanted to see what exactly goes to them.
Anyway now I directly used the TIdMultipartFormDataStream and changed the code in this way
aIdStream := TIdMultiPartFormDataStream.Create;
with aIdStream.AddFile('xml', 'e:\wm_req.xml', 'application/xml') do
begin
FileName := '';
//ContentType := '';
FileName := 'wmreq.xml';//somewhere I think I saw that only letters are allowed here
end;
aIdStream.Position := 0;
aStrm := TMemoryStream.Create;
//aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aIdStream));
aHTTP.Post(aURL, aIdStream, aStrm);
and thanks to Remy I added TIdCompressorZLib and now I get meaningful error:
HTTP/1.1 500 Internal Server Error
No message body writer has been found for response class FeedAcknowledgement.
I guess this is from them, not from me, will need to check with them.
EDIT4 Solution
OK, crazy or not, maybe my problem, but once Remy said that I am not asking for compressed answer and they send me such, I decided to change // Result.Request.Accept:='application/xml, */*'; to only // Result.Request.Accept:='application/xml'; and all started working properly! In fact having */* means that I am accepting everything, isn't it? It doesn't matter if you are using TIdMultipartFormDataStream or the body is manually prepared, it just works. Thank you Remy for the different point of view, another pair of eyes always helps. Crazy ... one week of stupid tests!
In your first Edit, you are not populating the TStringStream correctly. Use this instead:
Result.WriteString('--qwerty'#13#10{'Content-Disposition: form-data; name="file"; filename="wallmartreq.xml"'#13#10}#13#10 + aXML.WriteToString + #13#10'--qwerty--'#13#10);
In your second Edit, you are providing the wrong ContentType to TIdMultipartFormDataStream.AddFile(). You need to use application/xml instead of multipart/form-data:
aIdStream.AddFile('file', 'e:\wm_req.xml', 'application/xml');
Or, leave the ContentType blank (in which case, application/octet-stream will be used instead):
aIdStream.AddFile('file', 'e:\wm_req.xml');
If you don't want to send a ContentType at all, you can set the TIdFormDataField.ContentType property to a blank string:
with aIdStream.AddFile('file', 'e:\wm_req.xml') do
begin
FileName := ''; // needed so ContentType setter will not use a default value...
ContentType := '';
FileName := 'wm_req.xml'; // or whatever you want, or leave blank...
end;
But, more importantly, you are NOT specifying a boundary attribute for the TIdHTTP.Request.ContentType header anymore. In fact, you SHOULD NOT be saving the TIdMultipartFormDataStream to a TStringStream and then posting the TStringStream at all. You SHOULD be posting the TIdMultipartFormDataStream itself instead. That will automatically set the TIdHTTP.Request.ContentType to multipart/form-data, as well as give it a correct boundary attribute (whose value is generated by TIdMultipartFormDataStream):
aHTTP.Post(aURL, aIdStream)
Now, that being said, in both cases, you need to remove the call to TIdURI.URLDecode() on the return value of TIdHTTP.Post(). The server should be sending you raw XML, not URL-encoded XML, so you need to process the XML as-is and not URL-decode it at all.
Also, since XML is technically binary data that could be encoded in any charset that may or may not be present in the HTTP response headers, you SHOULD have your XML parser process the XML exactly as it is sent by the server. To do that, use the overloaded version of TIdHTTP.Post() that populates a TStream instead of returns a String. Have it receive the server's XML into a TMemoryStream, and then load that stream into your parser (TNativeXml has a LoadFromStream() method).
Lastly, you show that the server's response has a Content-Encoding: gzip response header, but I do not see you assigning any object to the TIdHTTP.Compressor property, or manually setting the TIdHTTP.Request.AcceptEncoding property, so the server SHOULD NOT be sending the response body in a compressed format since you are not asking for that format. TIdHTTP will not automatically decompress the data without a Compressor assigned.
If the server really is sending compressed data without being asked, that is a bug on the server side that you should report to Walmart.

Fetching and deleting a message over IMAP using cURL

I want to fetch and delete the top message (with UID=1) through IMAP using curl. I can do it in multiple commands, however when I send the delete command, I have no guarantee that the uid has not changed. Is there a way to persist the imap (or pop3) session in curl?
You actually do have a guarantee that the UID has not changed: this is what the IMAP UIDVALIDITY field is all about. As long as the UIDVALIDITY value for a mailbox has not changed, any message UIDs you have will continue to point to the same message. Read about this in more detail in the IMAP RFC.
Using curl, you can assert a UIDVALIDITY value in a request by including that in the IMAP URL, like this:
curl 'imaps://imap.example.com/inbox;UIDVALIDITY=6' ...
If the UIDVALIDITY has changed, curl will report:
curl: (78) Mailbox UIDVALIDITY has changed
You can see the UIDVALIDITY value for a mailbox by passing the -v option to curl when doing, e.g., an EXAMINE:
$ curl -v imaps://imap.example.com/ -X 'examine inbox'
[...]
> A003 examine inbox
< * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $NotPhishing $Phishing Old SEEN)
< * OK [PERMANENTFLAGS ()] Flags permitted.
< * OK [UIDVALIDITY 7] UIDs valid.
< * 9 EXISTS
< * 0 RECENT
< * OK [UIDNEXT 7422] Predicted next UID.
< * OK [HIGHESTMODSEQ 14264365]
< A003 OK [READ-ONLY] inbox selected. (Success)
A necessary prerequisite to using the UIDVALIDITY value is to request message uids. You can ask for the UID attribute when fetching information about messages:
$ curl -sv --netrc 'imaps://imap.example.com/inbox' -X 'fetch 1:* (UID FLAGS)'
[...]
< * 1 FETCH (UID 7186 FLAGS (\Seen))
< * 2 FETCH (UID 7188 FLAGS (\Seen))
< * 3 FETCH (UID 7278 FLAGS (\Seen))
< * 4 FETCH (UID 7288 FLAGS (Old \Seen))
< * 5 FETCH (UID 7364 FLAGS (\Seen))
< * 6 FETCH (UID 7413 FLAGS (\Seen))
< * 7 FETCH (UID 7417 FLAGS (\Seen))
< * 8 FETCH (UID 7419 FLAGS (\Seen))
< * 9 FETCH (UID 7421 FLAGS (\Seen))
Once you have a UID, you can use that to refer to a message by prefixing commands with the UID keyword. For example, instead of asking for message 9 like this:
FETCH 9 (ENVELOPE)
We would instead need to send the following IMAP command:
UID FETCH 7421 (ENVELOPE)
Curl doesn't appear to have native support for UID commands, but we can pass custom commands using the -X option. For example, to fetch message with UID 7421, we would run:
curl 'imaps://imap.example.com/inbox;uidvalidity=7' -X 'UID FETCH 7421 (ENVELOPE)'
This would assert that the mailbox UIDVALIDITY must still be 7 (because if it has changed, our UID is no longer valid), and retrieve the message envelope.
To delete that message:
curl 'imaps://imap.example.com/inbox;uidvalidity=7' -X 'UID STORE 7421 +Flags \Deleted'

Resources