Delphi and Walmart Bulk API Upload - Internal server error - indy

I am using Delphi XE5 and I am trying to send Items to Walmart using the Bulk API Upload but I am struggling with making proper connection with the server of Walmart.
I have already checked some similar problems here like Walmart API - payload mime type issue
The strangest thing of all is that the feed is actually accepted by Walmart (I see it in the list of feeds) but I receive Internal server error and then meaningless characters (probably type of encoding or just kind of error) like
(#$D#$A#$1F#$008B#8#0#0#0#0#0#0#3#$D'A±'#$D#$0080'0'#$C#4'AU~'#2'v ?d'#$0087#4'?'#1#$0091'OE'#$E#$008A'O'#$1E'®?O?'#$18#$0091#$A#$0091'M^'#$C'?;'#$1D'G'#$A'dR±U?oip?m'#$1A'AVS'#4#$16'R?iR'#$1B#$0095'RO?}u'#0'enIjM'#0#0#0)
So I use Indy IdHTTP and I have tried using TIdMultipartFormDataStream and also creating the payload on my own both with receiving the internal server error.
I presume I am breaking their parser somehow because if I miss one CR LF (#13#10) at the beginning I get a specific error
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ns2:errors xmlns:ns2="http://walmart.com/"><ns2:error><ns2:code>SYSTEM_ERROR.GMP_GATEWAY_API</ns2:code><ns2:field>PDR-0012</ns2:field><ns2:description></ns2:description><ns2:info>System encountered some internal error.</ns2:info><ns2:severity>ERROR</ns2:severity><ns2:category>SYSTEM</ns2:category><ns2:causes/><ns2:errorIdentifiers/></ns2:error></ns2:errors>
If I add an extra CRLF at the end then I get another error stating that there is more than one feed in the request.
So I really presume that there is something which makes that server / parser return the Internal error but I don't get what and again I repeat that the feed actually gets accepted and then processed by the server in the case I receive this error.
Here is an example request with a payload created on my own:
Headers
POST /v3/feeds?feedType=item HTTP/1.1
Content-Type: multipart/form-data; boundary=qwerty
Content-Length: 5077
WM_SVC.NAME: Walmart Marketplace
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.TIMESTAMP: 1528364885909
WM_SEC.AUTH_SIGNATURE: nSHgqzPOtzSR4wJU+U/vQJk+rk6Ke2QwodTHjzkjau2BonXZxiU9e+3NFPzaat2OUyc+vr0jqRk0H0QWTSC21PrI87mvqei5UJCJwNiIx0zVjAGpxsnIuvtIKkQsBpuUAa8C6SjTiTpDRsNt4IOxrk+tLWxlwQubWVCV+009a6o=
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE: 0f3e4dd4-0514-4346-b39d-af0e00ea066d
Host: marketplace.walmartapis.com
Accept: application/xml, */*
Accept-Encoding: identity
Accept-Language: en-US
User-Agent: Mozilla/3.0 (compatible; Indy Library)
Request
--qwerty
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{694A5564-7E8E-46C7-B772-C944A8C9CF99}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
<SkuUpdate>Yes</SkuUpdate>
<productName>Caramel Cantina 4 Pack Sleeping Nursing Cross Front Maternity Bra (Large, White)</productName>
<ProductIdUpdate>Yes</ProductIdUpdate>
<category>
<ClothingCategory>
<Clothing>
<shortDescription>Stretchy comfort cross front sleep bra by Caramel Cantina. Whether for nursing / maternity or for wearing as a sleep bra (or both nursing and sleep bras) - you&apos;ll experience a wire free regular back strap sleep bra. This mid-weight fabric but light support wireless bra is great for all day wear and comfort. The straps are wider than a standard bra - offering a longer range of wear time. Great for wearing indoors (ie when you&apos;re kicking around the home or going to bed). These are padless - most moms use their own favorite nursing pads with these (though no pads are included). The front cups have two layers of fabric to add a bit more support (though we still call them our light support comfort bra). The criss cross front style allows for easy access while nursing your baby. 92% Nylon 8% Spandex fabric washes up nicely. Cool wash and cool dry. Hang dry for longer life. The straps narrowest point at the shoulders are 1.25-1.5 inches (depending on the size). The bras are super stretchy - if you&apos;re looking for something super supportive (ie - they pull you in really tight - more like a sports bra) these probably aren&apos;t what you&apos;re looking for. Not recommended for heavy sports activity. Best worn for activities such as walking, sleeping, lounging, light activity, running errands, etc.</shortDescription>
<keyFeatures>
<keyFeaturesValue>4 Pack Includes 4 Bras of the Same Design - Color Option Above</keyFeaturesValue>
<keyFeaturesValue>Soft Cup - Front Double Layered - No padding - Wireless</keyFeaturesValue>
<keyFeaturesValue>Super Soft Sleep In Style Bra - Great for Nursing or Sleeping</keyFeaturesValue>
<keyFeaturesValue>Criss Cross Front Design Gives Easy Nursing Access</keyFeaturesValue>
<keyFeaturesValue>Regular Straps - Scoop Back Design - Strap Width 1.25-1.5 Inches Depending on Size</keyFeaturesValue>
</keyFeatures>
<brand>Caramel Cantina</brand>
<mainImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</mainImageUrl>
<productSecondaryImageURL>
<productSecondaryImageURLValue>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-wht-m1.jpg</productSecondaryImageURLValue>
</productSecondaryImageURL>
<color>White</color>
<clothingSize>Large</clothingSize>
<variantGroupId>cc-a1-4pk-parent</variantGroupId>
<variantAttributeNames>
<variantAttributeName>color</variantAttributeName>
<variantAttributeName>clothingSize</variantAttributeName>
</variantAttributeNames>
<countryOfOriginTextiles>Imported</countryOfOriginTextiles>
<fabricCareInstructions>
<fabricCareInstruction>Machine wash cold gentle cycle with like colors. Hang Dry. Do Not Bleach. Do Not Iron.</fabricCareInstruction>
</fabricCareInstructions>
<multipackQuantity>1</multipackQuantity>
<countPerPack>4</countPerPack>
<count>4</count>
<pattern>Solid</pattern>
<material>Nylon; Spandex</material>
<gender>Female</gender>
<ageGroup>
<ageGroupValue>Adult</ageGroupValue>
</ageGroup>
<clothingSizeGroup>Maternity</clothingSizeGroup>
<isSet>No</isSet>
<requiresTextileActLabeling>Yes</requiresTextileActLabeling>
<clothingTopStyle>Sleep Bra</clothingTopStyle>
<upperBodyStrapConfiguration>Regular</upperBodyStrapConfiguration>
<braStyle>Nursing</braStyle>
<clothingStyle>Maternity</clothingStyle>
<clothingFit>Maternity</clothingFit>
<isMaternity>Yes</isMaternity>
<swatchImages>
<swatchImage>
<swatchVariantAttribute>color</swatchVariantAttribute>
<swatchImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</swatchImageUrl>
</swatchImage>
</swatchImages>
</Clothing>
</ClothingCategory>
</category>
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
--qwerty--
The response which I get is as following
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$c88958a9
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528364905439
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528364905567
WM_SVC.SERVER_IP: 10.65.34.219
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-4-96141283
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Expires: Thu, 07 Jun 2018 09:48:25 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 09:48:25 GMT
Connection: close
And then those characters from above. Note the Cteonnt-Length: 77 header, it is not my TYPO it just comes in this way.
According to the support they have success using the following headers and ask me to use them, but I don't see any difference to what I make
WM_SVC.NAME:Walmart Marketplace
WM_QOS.CORRELATION_ID:123456abcdef
WM_SEC.AUTH_SIGNATURE:TnjevCf+voP9dmnafuCTFruwq6leBuAihSvag89WLieDRBsz7aULxgEqV71ZjIp572wVYPI07y6tdMutNLklDGwxvNdlJ2Q2xGvUIqjVPtqlhdcWvsmgqdpio7puQ4G03q1lReWzTquKecDEbB1ztH6ukj9F5rMe7d7PH8QkFsY=
WM_SEC.TIMESTAMP:1528152581896
WM_CONSUMER.ID:16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE:0f3e4dd4-0514-4346-b39d-af0e00ea066d
Accept:application/xml
Host:marketplace.walmartapis.com
Content-Type:multipart/form-data
I already make successful communication with eBay, Amazon, PayPal, Magento and many more, but here obviously I am missing something and really don't know how to solve it.
Thank you for reading this, and hopefully someone can give a clue what is going on here.
EDIT I am adding some Delphi code as required
Version 1 without TIdMultipartFormDataStream
function TdmSyncWallmart.CreateItemFeed(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime; var aReqID: String): TStringStream;
var
aXML: TNativeXML;
aNode: TXmlNode;
aItemNode: TXmlNode;
aCatNode: TXmlNode;
aTmpNode: TXmlNode;
aShortDesc: String;
i: Integer;
aIDItems: String;
begin
Result := nil;
dstListPlatformsSyncItemsWallmart.Close;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#IDPlatforms').Value := aIDPlatforms;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#LastChanged').Value := aLastChanged;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#CurrDate').Value := aCurrDate;
dstListPlatformsSyncItemsWallmart.Open;
if dstListPlatformsSyncItemsWallmart.IsEmpty then
Exit;
aReqID := GenGuid;
aXML := TNativeXml.CreateName('MPItemFeed');
try
aXML.Declaration.AttributeAdd('standalone', 'yes');
aXML.Root.AttributeAdd('xmlns:ns2', 'http://walmart.com/');
//Header
aNode := aXML.Root.NodeNew('MPItemFeedHeader');
with aNode.NodeNew('version') do
Value := '3.1';
with aNode.NodeNew('requestId') do
Value := aReqID;
with aNode.NodeNew('mart') do
Value := 'WALMART_US';
//... some code for generating the XML
Result := TStringStream.Create;
aXML.XmlFormat := xfReadable;
Result.WriteString('--qwerty'#13#10#13#10{'Content-Disposition: form-data; filename="wallmartreq.xml"'#13#10}+aXML.WriteToString +'--qwerty--'#13#10);
Result.Position := 0;
finally
aXML.Free;
end;
end;
function TdmSyncWallmart.CreateHTTP(aContentType: String): TIdHTTP;
begin
Result := TIdHTTP.Create(nil);
Result.ConnectTimeout:=5000;
Result.ReadTimeout:=20000;
Result.ProtocolVersion:=pv1_1;
Result.HTTPOptions := [hoForceEncodeParams, hoKeepOrigProtocol];
Result.HandleRedirects:=True;
Result.IOHandler := SSLHandler;
Result.Request.Accept:='application/xml, */*';
Result.Request.AcceptLanguage:='en-US';
Result.Request.ContentType:=aContentType;
// Result.Request.CharSet:='utf-8';
Result.Intercept := IdLogEvent1;
end;
procedure TdmSyncWallmart.AddCustomHeaders(aHTTP: TIdHTTP; aSign,
aTimeStamp: String);
begin
aHTTP.Request.CustomHeaders.AddValue('WM_SVC.NAME', 'Walmart Marketplace');
aHTTP.Request.CustomHeaders.AddValue('WM_QOS.CORRELATION_ID', GenGuid);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.TIMESTAMP', aTimeStamp);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.AUTH_SIGNATURE', aSign);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.ID', dstPlatformsUserName.AsString);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.CHANNEL.TYPE', dstPlatformsPassword.AsString);
end;
//some code for debugging purposes
procedure TdmSyncWallmart.IdLogEvent1Send(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_req_txt.txt');
finally
aList.Free;
end;
end;
procedure TdmSyncWallmart.IdLogEvent1Receive(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_Resp.txt');
finally
aList.Free;
end;
end;
Actual call
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data;boundary=qwerty');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := CreateItemFeed(aIDPlatforms, aLastChanged, aCurrDate, aReqID);
if aStrm <> nil then
begin
try
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
EDIT2
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
aIdStream: TIdMultipartFormDataStream;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := TStringStream.Create();
begin
try
aIdStream := TIdMultiPartFormDataStream.Create;
aIdStream.AddFile('file', 'e:\wm_req.xml', 'multipart/form-data');
aIdStream.Position := 0;
astrm.LoadFromStream(aIdStream);
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
This is the sent data
----------060718213723831
Content-Disposition: form-data; name="file"; filename="wm_req.xml"
Content-Type: multipart/form-data
Content-Transfer-Encoding: binary
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{164726BF-288E-4953-A1DA-1EC7595C5725}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
..... the other part of the XML
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
----------060718213723831--
Returned data captured by the intercepting part is the same as the one already posted with again the Cteonnt-Length: 77 header, but I guess if I succeed in reading the gzip data, I can reach an error which can be forwarded to the support.
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {CB0208AB-2135-4A12-B45F-DD2889273971}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$2ca3f3b6
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528396669253
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528396669365
WM_SVC.SERVER_IP: 10.65.34.2
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-1-96141274
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Vary: Accept-Encoding
Expires: Thu, 07 Jun 2018 18:37:49 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 18:37:49 GMT
Connection: close
EDIT3
For my Edit1, I am not adding Content-Disposition at all (it is commented) - with so many tests I have left it in my post here, so sorry for the confusion.
For Edit2 - thanks for pointing this out, actually I made my first tests without transforming into TStringStream, but I wanted to see what exactly goes to them.
Anyway now I directly used the TIdMultipartFormDataStream and changed the code in this way
aIdStream := TIdMultiPartFormDataStream.Create;
with aIdStream.AddFile('xml', 'e:\wm_req.xml', 'application/xml') do
begin
FileName := '';
//ContentType := '';
FileName := 'wmreq.xml';//somewhere I think I saw that only letters are allowed here
end;
aIdStream.Position := 0;
aStrm := TMemoryStream.Create;
//aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aIdStream));
aHTTP.Post(aURL, aIdStream, aStrm);
and thanks to Remy I added TIdCompressorZLib and now I get meaningful error:
HTTP/1.1 500 Internal Server Error
No message body writer has been found for response class FeedAcknowledgement.
I guess this is from them, not from me, will need to check with them.
EDIT4 Solution
OK, crazy or not, maybe my problem, but once Remy said that I am not asking for compressed answer and they send me such, I decided to change // Result.Request.Accept:='application/xml, */*'; to only // Result.Request.Accept:='application/xml'; and all started working properly! In fact having */* means that I am accepting everything, isn't it? It doesn't matter if you are using TIdMultipartFormDataStream or the body is manually prepared, it just works. Thank you Remy for the different point of view, another pair of eyes always helps. Crazy ... one week of stupid tests!

In your first Edit, you are not populating the TStringStream correctly. Use this instead:
Result.WriteString('--qwerty'#13#10{'Content-Disposition: form-data; name="file"; filename="wallmartreq.xml"'#13#10}#13#10 + aXML.WriteToString + #13#10'--qwerty--'#13#10);
In your second Edit, you are providing the wrong ContentType to TIdMultipartFormDataStream.AddFile(). You need to use application/xml instead of multipart/form-data:
aIdStream.AddFile('file', 'e:\wm_req.xml', 'application/xml');
Or, leave the ContentType blank (in which case, application/octet-stream will be used instead):
aIdStream.AddFile('file', 'e:\wm_req.xml');
If you don't want to send a ContentType at all, you can set the TIdFormDataField.ContentType property to a blank string:
with aIdStream.AddFile('file', 'e:\wm_req.xml') do
begin
FileName := ''; // needed so ContentType setter will not use a default value...
ContentType := '';
FileName := 'wm_req.xml'; // or whatever you want, or leave blank...
end;
But, more importantly, you are NOT specifying a boundary attribute for the TIdHTTP.Request.ContentType header anymore. In fact, you SHOULD NOT be saving the TIdMultipartFormDataStream to a TStringStream and then posting the TStringStream at all. You SHOULD be posting the TIdMultipartFormDataStream itself instead. That will automatically set the TIdHTTP.Request.ContentType to multipart/form-data, as well as give it a correct boundary attribute (whose value is generated by TIdMultipartFormDataStream):
aHTTP.Post(aURL, aIdStream)
Now, that being said, in both cases, you need to remove the call to TIdURI.URLDecode() on the return value of TIdHTTP.Post(). The server should be sending you raw XML, not URL-encoded XML, so you need to process the XML as-is and not URL-decode it at all.
Also, since XML is technically binary data that could be encoded in any charset that may or may not be present in the HTTP response headers, you SHOULD have your XML parser process the XML exactly as it is sent by the server. To do that, use the overloaded version of TIdHTTP.Post() that populates a TStream instead of returns a String. Have it receive the server's XML into a TMemoryStream, and then load that stream into your parser (TNativeXml has a LoadFromStream() method).
Lastly, you show that the server's response has a Content-Encoding: gzip response header, but I do not see you assigning any object to the TIdHTTP.Compressor property, or manually setting the TIdHTTP.Request.AcceptEncoding property, so the server SHOULD NOT be sending the response body in a compressed format since you are not asking for that format. TIdHTTP will not automatically decompress the data without a Compressor assigned.
If the server really is sending compressed data without being asked, that is a bug on the server side that you should report to Walmart.

Related

gomail.v2 quoted-printable encoding issue on html email body

I'm using gomail.v2 to send html email messages. There is a problem with the encoding / decoding of the html. Here are the relevant the lines of code:
m := gomail.NewMessage()
// ... a bunch of lines removed, for clarity...
s, err := GenerateBodyHTML(si.MsgFName, si.Hostname, &p, t)
if err != nil {
return err
}
m.SetBody("text/html", s)
err = d.DialAndSend(m)
I generate the HTML as a string and I verified that it is correct -- I can write the contents of s to a file and open that file in a browser and it all looks fine. So, for example, here are a few lines from the middle of s:
<body>
<img src="http://ec2-54-152-108-202.compute-1.amazonaws.com:8275/iblogosm.png">
<p>Hello Tom,</p>
But in the 3 email clients I tested, here's what the raw message body looks like:
<body>
<img src=3D"http://ec2-54-152-108-202.compute-1.amazonaws.com:8275/iblogosm.png">
<p>Hello Tom,</p>
So the link and image tags don't work when the message renders. The equal sign is still in quoted-printable form. Here are the relevant headers received by the email client (I removed most of them for brevity, plus nothing else looked relevant):
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
What do I need to do to get the HTML decoded correctly on the mail client?
After digging into this problem further, the data inside the mail client should look the way it is shown above, and the HTML does render properly. It turns out that the first link I was looking at didn't work for a completely unrelated reason.

Having issue with UTL_mail.send for subject length more than 47

I am using utl_mail.send for automate email in my Pl-SQL package.
When i use the subject charater length more than 47 then the body of the email is getting corrupted(Attached the screenshot). Shorter subject length e-mail is coming perfect.
declare
begin
utl_mail.send(sender => 'Test#supersoft.com',
subject => 'test10 Test10 more than 47 test 1234567890912312453254325345',
message => 'Test',
mime_type => 'text/plain; charset=us-ascii',
recipients => 'interanet#supersoft.com');
end;
Could anyone please help me what might be the issue with this.
I have read this article, Does UTL_MAIL have an attachment limit of 32k, on AskTom earlier today, and this seems to be a bug. As he advised, you should check for patch support in MOSC:
The question:
I am on 10g release 2 on windows 64 bit.
When I tried to send an email with subject more than 44( or so ) characters . it gets truncated.
Is there any limitation in using this package ?
I could not see any limitations specified in the documentation or in metalink.
His Anonymous Block:
begin
utl_mail.send ( sender => 'sender#ru.com" ,
recipients => "powerusers#ru.com" ,
subject => "Process is completed for both Sales Order Data."
test => "Test");
end;
The asker's output (similar to yours):
Subject: Process is completed for both Sales Order D
=?WINDOWS-1252?Q?ata.?=
X-Priority: 3
Content-Type: multipart/mixed;
boundary="------------4D8C24=_23F7E4A13B2357B3"
This is a multi-part message in MIME format.
--------------4D8C24=_23F7E4A13B2357B3
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Test
Tom's Answer:
I see some filed issues regarding this with support.
Your options would be
a) file an issue with support and see if a patch exists for your system.
b) use utl_smtp to send the mail yourself.

Get data with restful webservices from Oracle PL/SQL

I want to get data from webservices from oracle database.
But I'm looking for it unfortunately I can't anything .
Please give me some hints.I want to accesses it with username and password in pl/sql code.
How can I do it?
Thank you..
I solve it like following code.I think it will be useful .
declare
t_http_req utl_http.req;
t_http_resp utl_http.resp;
t_request_body varchar2(30000);
t_respond varchar2(30000);
t_start_pos integer := 1;
t_output varchar2(2000);
begin
/*Construct the information you want to send to the webservice.
Normally this would be in a xml structure. But for a REST-
webservice this is not mandatory. The webservice i needed to
call excepts plain test.*/
t_request_body := 'the data you want to send to the webservice';
/*Telling Oracle where the webservice can be found, what kind of request is made
and the version of the HTTP*/
t_http_req:= utl_http.begin_request( 'http://urlofwebservice'
, 'GET'
, 'HTTP/1.1');
/*In my case the webservice used authentication with a username an password
that was provided to me. You can skip this line if it's a public webservice.*/
utl_http.set_authentication(t_http_req,'username','password');
/*Describe in the request-header what kind of data is send*/
utl_http.set_header(t_http_req, 'Content-Type', 'text/xml charset=UTF-8');
/*Describe in the request-header the lengt of the data*/
utl_http.set_header(t_http_req, 'Content-Length', length(t_request_body));
/*Put the data in de body of the request*/
utl_http.write_text(t_http_req, t_request_body);
/*make the actual request to the webservice en catch the responce in a
variable*/
t_http_resp:= utl_http.get_response(t_http_req);
/*Read the body of the response, so you can find out if the information was
received ok by the webservice.
Go to the documentation of the webservice for what kind of responce you
should expect. In my case it was:
<responce>
<status>ok</status>
</responce>
*/
utl_http.read_text(t_http_resp, t_respond);
dbms_output.put_line(t_respond);
/*Some closing?1 Releasing some memory, i think....*/
utl_http.end_response(t_http_resp);
end;
You might want to look into the UTL_HTTP package

Phantom messages in Exchange Calendar folder

I'm developing an IMAP client, and testing it against a Microsoft Exchange server. When I select the Calendar mailbox, UID SEARCH suggests that there are messages in the mailbox, but UID FETCH cannot fetch them:
S: * OK The Microsoft Exchange IMAP4 service is ready.
...
C: 38 SELECT "Calendar"
S: * 13 EXISTS
S: * 0 RECENT
S: * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
S: * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
S: * OK [UIDVALIDITY 37122] UIDVALIDITY value
S: * OK [UIDNEXT 190] The next unique identifier value
S: 38 OK [READ-WRITE] SELECT completed.
C: 39 UID SEARCH OR UNSEEN FLAGGED
C: 40 UID SEARCH SEEN UNFLAGGED SINCE 19-Oct-2014
S: * SEARCH
S: 39 OK SEARCH completed.
S: * SEARCH 90 183
S: 40 OK SEARCH completed.
C: 41 UID FETCH 90,183 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: 41 OK FETCH completed.
I presume that Calendar is some kind of virtual mailbox, but this behaviour doesn't make sense to me — why would the server return the message uids in UID SEARCH if it's not possible to FETCH them? Is there any way to fetch these messages, or somehow manipulate them using IMAP?
Update: After being upgraded to 2010 SP3, it seems like the Exchange server had a change of heart. Now it at least returns something in response to the FETCH command:
C: 120 UID FETCH 773,948 (ENVELOPE INTERNALDATE FLAGS BODY.PEEK[])
S: * 16 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 773" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "05-Mar-2015 17:22:02 +0000" FLAGS (\Seen) BODY[] {783}
S: <783 bytes omitted>
S: UID 773)
S: * 17 FETCH (ENVELOPE (NIL "Retrieval using the IMAP4 protocol failed for the following message: 948" (("Microsoft Exchange Server 2010" NIL NIL ".MISSING-HOST-NAME.")) NIL NIL (("██████████" NIL "██████████" "██████████")) NIL NIL NIL NIL) INTERNALDATE "11-Mar-2015 17:56:04 +0000" FLAGS (\Seen) BODY[] {747}
S: <747 bytes omitted>
S: UID 948)
S: 120 OK FETCH completed.
The messages look like this:
MIME-Version: 1.0
Content-Type: text/plain; charset="us-ascii"
From: Microsoft Exchange Server 2010
To: ███████████ <████████#██████████.com>
Subject: Retrieval using the IMAP4 protocol failed for the following
message: 773
Content-Transfer-Encoding: quoted-printable
The server couldn't retrieve the following message:
Subject: "████ █████ ███████ ██ ████ ███ █████ ███████.. ███ ███ ██████ ███=
██!"
From: "███████████ ██████████" ("/O=3D████████/OU=3D████████ █████████████=
█ █████ (███████████████)/CN=3D██████████/CN=3D███████████ █████████████")
Sent date: 3/5/2015 5:22:02 PM
The message hasn't been deleted. You might be able to view it using either =
Outlook or Outlook Web App. You can also contact the sender to find out wha=
t the message says.=
Sounds like a server bug. It is not a secret that Exchange "implaments IMAP" in order to be able to tick a required checkbox when evaluating tenders. You will see other bugs in there as well.
It might be interesting to get a list of all UIDs which that mailbox allegedly contains, and to try fetching them one-by-one to see if at least some of them return some usable value. However, considering that this is a mailbox called "Calendar", chances are that it's just some random Exchange export of data which only make sense via MAPI. Random googling results in https://technet.microsoft.com/en-us/library/aa998606%28v=exchg.150%29.aspx which suggests that Exchange can be configured to offer these calendar data in various formats. Maybe your server is "misconfigured".
TL;DR: it's Exchange. Don't waste time on it.

Alternative to UrlDownloadToFile

UrlDownloadToFile is a nice command in AutoHotkey and works just fine, most of the time, but sometimes a download mechanism is too complex for it. For example if the download requires a specific user-agent to be set or if the download requires a cookie or maybe even a password.
So the question is:
Is there a more advanced download function, which could handle all of the above said?
I wrote this quite some time ago and thought it would be a nice idea to wrap it up in a function and post it here:
Download(UrlToFile,SaveFileAs:="",Overwrite:=True,headers:="",method:="GET",postData:="") {
WinHttpObj := ComObjCreate("WinHttp.WinHttpRequest.5.1")
WinHttpObj.Open(method, UrlToFile)
For header, value in headers
WinHttpObj.SetRequestHeader(header, value)
WinHttpObj.Send(postData)
ADODBObj := ComObjCreate("ADODB.Stream")
ADODBObj.Type := 1
ADODBObj.Open()
ADODBObj.Write(WinHttpObj.ResponseBody)
If !SaveFileAs {
urlSplitArray := StrSplit(UrlToFile, "/")
SaveFileAs := urlSplitArray[urlSplitArray.MaxIndex()]
}
ADODBObj.SaveToFile(SaveFileAs, Overwrite ? 2:1)
ADODBObj.Close()
}
Example 1
Download("http://ahkscript.org/download/1.1/AutoHotkey111402_Install.exe")
Example 2
customHeaders := {"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0"
,"Cache-Control": "max-age=0"
,"Cookie": "downloadtoken=b82416fdb23e421fb5a"}
Download("http://download.piriform.com/ccsetup410.exe","",True,customHeaders)
Example 3
Download("http://foo.bar/example.exe","example.exe",True,{"Cookie":"sessionid=abc123"},"POST","username=foo_bar&password=qwerty")

Resources