Alternative to UrlDownloadToFile - download

UrlDownloadToFile is a nice command in AutoHotkey and works just fine, most of the time, but sometimes a download mechanism is too complex for it. For example if the download requires a specific user-agent to be set or if the download requires a cookie or maybe even a password.
So the question is:
Is there a more advanced download function, which could handle all of the above said?

I wrote this quite some time ago and thought it would be a nice idea to wrap it up in a function and post it here:
Download(UrlToFile,SaveFileAs:="",Overwrite:=True,headers:="",method:="GET",postData:="") {
WinHttpObj := ComObjCreate("WinHttp.WinHttpRequest.5.1")
WinHttpObj.Open(method, UrlToFile)
For header, value in headers
WinHttpObj.SetRequestHeader(header, value)
WinHttpObj.Send(postData)
ADODBObj := ComObjCreate("ADODB.Stream")
ADODBObj.Type := 1
ADODBObj.Open()
ADODBObj.Write(WinHttpObj.ResponseBody)
If !SaveFileAs {
urlSplitArray := StrSplit(UrlToFile, "/")
SaveFileAs := urlSplitArray[urlSplitArray.MaxIndex()]
}
ADODBObj.SaveToFile(SaveFileAs, Overwrite ? 2:1)
ADODBObj.Close()
}
Example 1
Download("http://ahkscript.org/download/1.1/AutoHotkey111402_Install.exe")
Example 2
customHeaders := {"User-Agent": "Mozilla/5.0 (Windows NT 6.3; WOW64; rv:37.0) Gecko/20100101 Firefox/37.0"
,"Cache-Control": "max-age=0"
,"Cookie": "downloadtoken=b82416fdb23e421fb5a"}
Download("http://download.piriform.com/ccsetup410.exe","",True,customHeaders)
Example 3
Download("http://foo.bar/example.exe","example.exe",True,{"Cookie":"sessionid=abc123"},"POST","username=foo_bar&password=qwerty")

Related

cURL got different page source than what Chrome browser did

In short: I'm trying to get that page source of https://www.etoro.com/app/sv-iframe using curl in Bash.
I understand this ask is quite "simple". I have read thru 10+ similar questions here. Unfortunately, none of them could solve my problem.
When you open the URL above in Chrome browser, it's blank. You can either right click -> View Page Source, or sniff network using Chrome Developer Tool. Both will give you the correct page source. The page contains javascripts, in which there is a long hex string - what I need ultimately. I tried disabling javascript and reloading the page. I still got the right page source. So javascript doesn't play trick here. It sounds getting such page source via curl should be just straight forward, right?
When I right click the request in Chrome Developer Tool -> Copy as cURL, and execute it in terminal, things turned nasty - I got a CloudFlare security check page. I reopened the page several times in Chrome Incognito mode. I swear never saw a CloudFlare security check in browser. I double checked the cURL command. It has user-agent set as well.
Here is what I tried so far:
Manually compose curl command and fill headers from Chrome Developer Tool
Sniff packages on an Android device, and use headers set on mobile browser
Post request online from Postman Web
All gave me the same CloudFlare security check page.
The CloudFlare page says "Please enable cookies". I suspect if server in this way determined I was not calling from a browser. Following some threads, I tried to set -b/-c/-j flag with curl. Also no luck.
Here's more detailed steps what I've done:
Open Chrome Incognito mode
Open Developer Tool
Use Command+Shift+P (Mac) to open command menu
Type "disable javascript" and hit enter
Switch to Network tab
Open https://www.etoro.com/app/sv-iframe
Observe the request list - there should be only 1 request (request screenshot 1 / request screenshot 2 / response body / response cookie)
Right click on the request -> Copy as cURL
Here's my curl command:
curl 'https://www.etoro.com/app/sv-iframe' \
-H 'authority: www.etoro.com' \
-H 'pragma: no-cache' \
-H 'cache-control: no-cache' \
-H 'sec-ch-ua: "Google Chrome";v="89", "Chromium";v="89", ";Not A Brand";v="99"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'upgrade-insecure-requests: 1' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36' \
-H 'accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9' \
-H 'sec-fetch-site: none' \
-H 'sec-fetch-mode: navigate' \
-H 'sec-fetch-user: ?1' \
-H 'sec-fetch-dest: document' \
-H 'accept-language: en-US,en;q=0.9' \
--compressed
The request itself I don't think it requires cookie, as page was able to be opened in Incognito mode. I anyways tried to set the response cookie together with the request. It doesn't help either.
-H 'cookie: __cfduid=d2edf...; TS01047baf=01d53...; __cf_bm=a3803...; __cflb=02Di3...'
Already spent whole evening on it but couldn't get it resolved. I appreciate any suggestions or help to get me thru it. I have a feeling that the actual fix would be fairly simple. The request has no cookie. Only thing to update is header. Maybe I didn't have correct header specified? Or some extra curl flag would help?
There is some obfuscated js eval code on that page, that is basically setting cookies or sending logs, digging a bit deeper, this is what ended up with:
(function() {
var s = '9a7xxx......';
function setCookie(cname, cvalue, domain, exdays) {
var d = new Date();
d.setTime(d.getTime() + (exdays * 1000 * 60 * 60 * 24));
var expires = "expires=" + d.toUTCString();
var cookie = cname + "=" + cvalue;
if (domain) {
cookie += ";" + "domain=" + domain;
}
cookie += ";" + expires + ";path=/";
document.cookie = cookie;
}
function deleteCookie(cname, domain) {
setCookie(cname, "", domain, 0);
}
var ta = ["window.callPhantom", "window.__nightmare", "window._phantom", "window.__webdriver_script_fn", "navigator.webdriver", "document.$cdc_asdjflasutopfhvcZLmcfl_"];
var re;
try {
re = [!!window.callPhantom, !!window.__nightmare, !!window._phantom, !!window.__webdriver_script_fn, !!navigator.webdriver, !!document.$cdc_asdjflasutopfhvcZLmcfl_];
} catch (err) {}
if (re && re.indexOf(true) == -1) {
setCookie("TMIS2", s, ".etoro.com", 14);
} else {
var resultsObj = {};
for (var i = 0; i < ta.length; i++) {
resultsObj[ta[i]] = re[i];
}
var img = new Image();
img.src = 'https://etorologsapi.etoro.com/api/v2/monitoring?applicationIdentifier=JSCClient&LogEvents=' + encodeURIComponent(JSON.stringify([{
ApplicationIdentifier: 'JSCClient',
ApplicationVersion: '0.0.11',
Level: "error",
Message: "ClientSel",
Results: resultsObj,
Type: 'log'
}]));
}
})();

Delphi and Walmart Bulk API Upload - Internal server error

I am using Delphi XE5 and I am trying to send Items to Walmart using the Bulk API Upload but I am struggling with making proper connection with the server of Walmart.
I have already checked some similar problems here like Walmart API - payload mime type issue
The strangest thing of all is that the feed is actually accepted by Walmart (I see it in the list of feeds) but I receive Internal server error and then meaningless characters (probably type of encoding or just kind of error) like
(#$D#$A#$1F#$008B#8#0#0#0#0#0#0#3#$D'A±'#$D#$0080'0'#$C#4'AU~'#2'v ?d'#$0087#4'?'#1#$0091'OE'#$E#$008A'O'#$1E'®?O?'#$18#$0091#$A#$0091'M^'#$C'?;'#$1D'G'#$A'dR±U?oip?m'#$1A'AVS'#4#$16'R?iR'#$1B#$0095'RO?}u'#0'enIjM'#0#0#0)
So I use Indy IdHTTP and I have tried using TIdMultipartFormDataStream and also creating the payload on my own both with receiving the internal server error.
I presume I am breaking their parser somehow because if I miss one CR LF (#13#10) at the beginning I get a specific error
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><ns2:errors xmlns:ns2="http://walmart.com/"><ns2:error><ns2:code>SYSTEM_ERROR.GMP_GATEWAY_API</ns2:code><ns2:field>PDR-0012</ns2:field><ns2:description></ns2:description><ns2:info>System encountered some internal error.</ns2:info><ns2:severity>ERROR</ns2:severity><ns2:category>SYSTEM</ns2:category><ns2:causes/><ns2:errorIdentifiers/></ns2:error></ns2:errors>
If I add an extra CRLF at the end then I get another error stating that there is more than one feed in the request.
So I really presume that there is something which makes that server / parser return the Internal error but I don't get what and again I repeat that the feed actually gets accepted and then processed by the server in the case I receive this error.
Here is an example request with a payload created on my own:
Headers
POST /v3/feeds?feedType=item HTTP/1.1
Content-Type: multipart/form-data; boundary=qwerty
Content-Length: 5077
WM_SVC.NAME: Walmart Marketplace
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.TIMESTAMP: 1528364885909
WM_SEC.AUTH_SIGNATURE: nSHgqzPOtzSR4wJU+U/vQJk+rk6Ke2QwodTHjzkjau2BonXZxiU9e+3NFPzaat2OUyc+vr0jqRk0H0QWTSC21PrI87mvqei5UJCJwNiIx0zVjAGpxsnIuvtIKkQsBpuUAa8C6SjTiTpDRsNt4IOxrk+tLWxlwQubWVCV+009a6o=
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE: 0f3e4dd4-0514-4346-b39d-af0e00ea066d
Host: marketplace.walmartapis.com
Accept: application/xml, */*
Accept-Encoding: identity
Accept-Language: en-US
User-Agent: Mozilla/3.0 (compatible; Indy Library)
Request
--qwerty
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{694A5564-7E8E-46C7-B772-C944A8C9CF99}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
<SkuUpdate>Yes</SkuUpdate>
<productName>Caramel Cantina 4 Pack Sleeping Nursing Cross Front Maternity Bra (Large, White)</productName>
<ProductIdUpdate>Yes</ProductIdUpdate>
<category>
<ClothingCategory>
<Clothing>
<shortDescription>Stretchy comfort cross front sleep bra by Caramel Cantina. Whether for nursing / maternity or for wearing as a sleep bra (or both nursing and sleep bras) - you&apos;ll experience a wire free regular back strap sleep bra. This mid-weight fabric but light support wireless bra is great for all day wear and comfort. The straps are wider than a standard bra - offering a longer range of wear time. Great for wearing indoors (ie when you&apos;re kicking around the home or going to bed). These are padless - most moms use their own favorite nursing pads with these (though no pads are included). The front cups have two layers of fabric to add a bit more support (though we still call them our light support comfort bra). The criss cross front style allows for easy access while nursing your baby. 92% Nylon 8% Spandex fabric washes up nicely. Cool wash and cool dry. Hang dry for longer life. The straps narrowest point at the shoulders are 1.25-1.5 inches (depending on the size). The bras are super stretchy - if you&apos;re looking for something super supportive (ie - they pull you in really tight - more like a sports bra) these probably aren&apos;t what you&apos;re looking for. Not recommended for heavy sports activity. Best worn for activities such as walking, sleeping, lounging, light activity, running errands, etc.</shortDescription>
<keyFeatures>
<keyFeaturesValue>4 Pack Includes 4 Bras of the Same Design - Color Option Above</keyFeaturesValue>
<keyFeaturesValue>Soft Cup - Front Double Layered - No padding - Wireless</keyFeaturesValue>
<keyFeaturesValue>Super Soft Sleep In Style Bra - Great for Nursing or Sleeping</keyFeaturesValue>
<keyFeaturesValue>Criss Cross Front Design Gives Easy Nursing Access</keyFeaturesValue>
<keyFeaturesValue>Regular Straps - Scoop Back Design - Strap Width 1.25-1.5 Inches Depending on Size</keyFeaturesValue>
</keyFeatures>
<brand>Caramel Cantina</brand>
<mainImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</mainImageUrl>
<productSecondaryImageURL>
<productSecondaryImageURLValue>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-wht-m1.jpg</productSecondaryImageURLValue>
</productSecondaryImageURL>
<color>White</color>
<clothingSize>Large</clothingSize>
<variantGroupId>cc-a1-4pk-parent</variantGroupId>
<variantAttributeNames>
<variantAttributeName>color</variantAttributeName>
<variantAttributeName>clothingSize</variantAttributeName>
</variantAttributeNames>
<countryOfOriginTextiles>Imported</countryOfOriginTextiles>
<fabricCareInstructions>
<fabricCareInstruction>Machine wash cold gentle cycle with like colors. Hang Dry. Do Not Bleach. Do Not Iron.</fabricCareInstruction>
</fabricCareInstructions>
<multipackQuantity>1</multipackQuantity>
<countPerPack>4</countPerPack>
<count>4</count>
<pattern>Solid</pattern>
<material>Nylon; Spandex</material>
<gender>Female</gender>
<ageGroup>
<ageGroupValue>Adult</ageGroupValue>
</ageGroup>
<clothingSizeGroup>Maternity</clothingSizeGroup>
<isSet>No</isSet>
<requiresTextileActLabeling>Yes</requiresTextileActLabeling>
<clothingTopStyle>Sleep Bra</clothingTopStyle>
<upperBodyStrapConfiguration>Regular</upperBodyStrapConfiguration>
<braStyle>Nursing</braStyle>
<clothingStyle>Maternity</clothingStyle>
<clothingFit>Maternity</clothingFit>
<isMaternity>Yes</isMaternity>
<swatchImages>
<swatchImage>
<swatchVariantAttribute>color</swatchVariantAttribute>
<swatchImageUrl>https://www.dancecheerskate.com/_inv/D54173DB8F934DB2A4C68D24FAC087F5/wm-cc-a1-4pk-white.jpg</swatchImageUrl>
</swatchImage>
</swatchImages>
</Clothing>
</ClothingCategory>
</category>
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
--qwerty--
The response which I get is as following
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {67F0E2F9-5EAC-4E8C-9C90-650D8F7B3B7A}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$c88958a9
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528364905439
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528364905567
WM_SVC.SERVER_IP: 10.65.34.219
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-4-96141283
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Expires: Thu, 07 Jun 2018 09:48:25 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 09:48:25 GMT
Connection: close
And then those characters from above. Note the Cteonnt-Length: 77 header, it is not my TYPO it just comes in this way.
According to the support they have success using the following headers and ask me to use them, but I don't see any difference to what I make
WM_SVC.NAME:Walmart Marketplace
WM_QOS.CORRELATION_ID:123456abcdef
WM_SEC.AUTH_SIGNATURE:TnjevCf+voP9dmnafuCTFruwq6leBuAihSvag89WLieDRBsz7aULxgEqV71ZjIp572wVYPI07y6tdMutNLklDGwxvNdlJ2Q2xGvUIqjVPtqlhdcWvsmgqdpio7puQ4G03q1lReWzTquKecDEbB1ztH6ukj9F5rMe7d7PH8QkFsY=
WM_SEC.TIMESTAMP:1528152581896
WM_CONSUMER.ID:16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.CHANNEL.TYPE:0f3e4dd4-0514-4346-b39d-af0e00ea066d
Accept:application/xml
Host:marketplace.walmartapis.com
Content-Type:multipart/form-data
I already make successful communication with eBay, Amazon, PayPal, Magento and many more, but here obviously I am missing something and really don't know how to solve it.
Thank you for reading this, and hopefully someone can give a clue what is going on here.
EDIT I am adding some Delphi code as required
Version 1 without TIdMultipartFormDataStream
function TdmSyncWallmart.CreateItemFeed(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime; var aReqID: String): TStringStream;
var
aXML: TNativeXML;
aNode: TXmlNode;
aItemNode: TXmlNode;
aCatNode: TXmlNode;
aTmpNode: TXmlNode;
aShortDesc: String;
i: Integer;
aIDItems: String;
begin
Result := nil;
dstListPlatformsSyncItemsWallmart.Close;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#IDPlatforms').Value := aIDPlatforms;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#LastChanged').Value := aLastChanged;
dstListPlatformsSyncItemsWallmart.Parameters.ParamByName('#CurrDate').Value := aCurrDate;
dstListPlatformsSyncItemsWallmart.Open;
if dstListPlatformsSyncItemsWallmart.IsEmpty then
Exit;
aReqID := GenGuid;
aXML := TNativeXml.CreateName('MPItemFeed');
try
aXML.Declaration.AttributeAdd('standalone', 'yes');
aXML.Root.AttributeAdd('xmlns:ns2', 'http://walmart.com/');
//Header
aNode := aXML.Root.NodeNew('MPItemFeedHeader');
with aNode.NodeNew('version') do
Value := '3.1';
with aNode.NodeNew('requestId') do
Value := aReqID;
with aNode.NodeNew('mart') do
Value := 'WALMART_US';
//... some code for generating the XML
Result := TStringStream.Create;
aXML.XmlFormat := xfReadable;
Result.WriteString('--qwerty'#13#10#13#10{'Content-Disposition: form-data; filename="wallmartreq.xml"'#13#10}+aXML.WriteToString +'--qwerty--'#13#10);
Result.Position := 0;
finally
aXML.Free;
end;
end;
function TdmSyncWallmart.CreateHTTP(aContentType: String): TIdHTTP;
begin
Result := TIdHTTP.Create(nil);
Result.ConnectTimeout:=5000;
Result.ReadTimeout:=20000;
Result.ProtocolVersion:=pv1_1;
Result.HTTPOptions := [hoForceEncodeParams, hoKeepOrigProtocol];
Result.HandleRedirects:=True;
Result.IOHandler := SSLHandler;
Result.Request.Accept:='application/xml, */*';
Result.Request.AcceptLanguage:='en-US';
Result.Request.ContentType:=aContentType;
// Result.Request.CharSet:='utf-8';
Result.Intercept := IdLogEvent1;
end;
procedure TdmSyncWallmart.AddCustomHeaders(aHTTP: TIdHTTP; aSign,
aTimeStamp: String);
begin
aHTTP.Request.CustomHeaders.AddValue('WM_SVC.NAME', 'Walmart Marketplace');
aHTTP.Request.CustomHeaders.AddValue('WM_QOS.CORRELATION_ID', GenGuid);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.TIMESTAMP', aTimeStamp);
aHTTP.Request.CustomHeaders.AddValue('WM_SEC.AUTH_SIGNATURE', aSign);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.ID', dstPlatformsUserName.AsString);
aHTTP.Request.CustomHeaders.AddValue('WM_CONSUMER.CHANNEL.TYPE', dstPlatformsPassword.AsString);
end;
//some code for debugging purposes
procedure TdmSyncWallmart.IdLogEvent1Send(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_req_txt.txt');
finally
aList.Free;
end;
end;
procedure TdmSyncWallmart.IdLogEvent1Receive(ASender: TIdConnectionIntercept;
var ABuffer: TIdBytes);
var
aList: TStringList;
begin
aList := TStringList.Create;
try
aList.Text := BytesToString(ABuffer);
aList.SaveToFile('e:\wm_Resp.txt');
finally
aList.Free;
end;
end;
Actual call
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data;boundary=qwerty');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := CreateItemFeed(aIDPlatforms, aLastChanged, aCurrDate, aReqID);
if aStrm <> nil then
begin
try
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
EDIT2
procedure TdmSyncWallmart.SendItems(aIDPlatforms: String; aLastChanged,
aCurrDate: TDateTime);
var
aSign, aTimeStamp: String;
aURL: String;
aHTTP: TIdHTTP;
aRes: String;
aStrm: TMemoryStream;
aReqID: String;
aXML: TNativeXml;
aNode: TXmlNode;
aErrMsg: String;
aIdStream: TIdMultipartFormDataStream;
begin
aURL := dstPlatformsURLAddress.AsString+'/v3/feeds?feedType=item';
GenrateSignature(aSign, aTimeStamp, aURL, 'POST');
if aSign = '' then
Exit;
aHTTP := CreateHTTP('multipart/form-data');
AddCustomHeaders(aHTTP, aSign, aTimeStamp);
try
aStrm := TStringStream.Create();
begin
try
aIdStream := TIdMultiPartFormDataStream.Create;
aIdStream.AddFile('file', 'e:\wm_req.xml', 'multipart/form-data');
aIdStream.Position := 0;
astrm.LoadFromStream(aIdStream);
aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aStrm));
aXML := TNativeXml.Create(nil);
try
aXML.ReadFromString(aRes);
aNode := aXML.Root.FindNode('feedId');
cmdUpdFeedID.Parameters.ParamByName('ID').Value := aRes;
cmdUpdFeedID.Parameters.ParamByName('FeedID').Value := aNode.Value;
cmdUpdFeedID.Execute;
finally
aXML.Free;
end;
except
on E:Exception do
begin
if E is EIdHTTPProtocolException then
aErrMsg := E.Message + #13#10 + (E as EIdHTTPProtocolException).ErrorMessage
else
aErrMsg := E.Message;
dmMain.InsLog(cLTError, aErrMsg, aIDPlatforms);
end;
end;
aStrm.Free;
end;
finally
aHTTP.Free;
end;
end;
This is the sent data
----------060718213723831
Content-Disposition: form-data; name="file"; filename="wm_req.xml"
Content-Type: multipart/form-data
Content-Transfer-Encoding: binary
<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<MPItemFeed xmlns:ns2="http://walmart.com/">
<MPItemFeedHeader>
<version>3.1</version>
<requestId>{164726BF-288E-4953-A1DA-1EC7595C5725}</requestId>
<mart>WALMART_US</mart>
</MPItemFeedHeader>
<MPItem>
<processMode>REPLACE_ALL</processMode>
<sku>w-cc-a1-4pk-white-l</sku>
<productIdentifiers>
<productIdentifier>
<productIdType>UPC</productIdType>
<productId>192082276845</productId>
</productIdentifier>
</productIdentifiers>
<MPProduct>
..... the other part of the XML
</MPProduct>
<MPOffer>
<price>25.00</price>
<ShippingWeight>
<measure>11.80</measure>
<unit>lb</unit>
</ShippingWeight>
<ProductTaxCode>2038895</ProductTaxCode>
</MPOffer>
</MPItem>
</MPItemFeed>
----------060718213723831--
Returned data captured by the intercepting part is the same as the one already posted with again the Cteonnt-Length: 77 header, but I guess if I succeed in reading the gzip data, I can reach an error which can be forwarded to the support.
HTTP/1.1 500 Internal Server Error
Method: null
URI: null
WM_CONSUMER.ID: 16248274-1f53-4e6d-880e-b5417698b878
WM_CONSUMER.INTIMESTAMP:
WM_QOS.CORRELATION_ID: {CB0208AB-2135-4A12-B45F-DD2889273971}
WM_SEC.AUTH_TOKEN:
WM_SEC.REFRESH_AUTH_TOKEN:
WM_SVC.CLASS_NAME: com.walmart.services.impl.GMPGatewayService$$EnhancerBySpringCGLIB$$2ca3f3b6
WM_SVC.ENV: prod
WM_SVC.INTIMESTAMP: 1528396669253
WM_SVC.METHOD_NAME: doPostMultiPart
WM_SVC.NAME: Walmart Marketplace
WM_SVC.OUTTIMESTAMP: 1528396669365
WM_SVC.SERVER_IP: 10.65.34.2
WM_SVC.SERVER_NAME: PartnerGMPService-5480119-1-96141274
WM_SVC.VERSION:
X-Powered-By: soari-interceptors-4.4.4
Content-Type: text/plain
Cteonnt-Length: 77
Server: web
Content-Encoding: gzip
Content-Length: 89
Vary: Accept-Encoding
Expires: Thu, 07 Jun 2018 18:37:49 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Thu, 07 Jun 2018 18:37:49 GMT
Connection: close
EDIT3
For my Edit1, I am not adding Content-Disposition at all (it is commented) - with so many tests I have left it in my post here, so sorry for the confusion.
For Edit2 - thanks for pointing this out, actually I made my first tests without transforming into TStringStream, but I wanted to see what exactly goes to them.
Anyway now I directly used the TIdMultipartFormDataStream and changed the code in this way
aIdStream := TIdMultiPartFormDataStream.Create;
with aIdStream.AddFile('xml', 'e:\wm_req.xml', 'application/xml') do
begin
FileName := '';
//ContentType := '';
FileName := 'wmreq.xml';//somewhere I think I saw that only letters are allowed here
end;
aIdStream.Position := 0;
aStrm := TMemoryStream.Create;
//aRes := TIdURI.URLDecode(aHTTP.Post(aURL, aIdStream));
aHTTP.Post(aURL, aIdStream, aStrm);
and thanks to Remy I added TIdCompressorZLib and now I get meaningful error:
HTTP/1.1 500 Internal Server Error
No message body writer has been found for response class FeedAcknowledgement.
I guess this is from them, not from me, will need to check with them.
EDIT4 Solution
OK, crazy or not, maybe my problem, but once Remy said that I am not asking for compressed answer and they send me such, I decided to change // Result.Request.Accept:='application/xml, */*'; to only // Result.Request.Accept:='application/xml'; and all started working properly! In fact having */* means that I am accepting everything, isn't it? It doesn't matter if you are using TIdMultipartFormDataStream or the body is manually prepared, it just works. Thank you Remy for the different point of view, another pair of eyes always helps. Crazy ... one week of stupid tests!
In your first Edit, you are not populating the TStringStream correctly. Use this instead:
Result.WriteString('--qwerty'#13#10{'Content-Disposition: form-data; name="file"; filename="wallmartreq.xml"'#13#10}#13#10 + aXML.WriteToString + #13#10'--qwerty--'#13#10);
In your second Edit, you are providing the wrong ContentType to TIdMultipartFormDataStream.AddFile(). You need to use application/xml instead of multipart/form-data:
aIdStream.AddFile('file', 'e:\wm_req.xml', 'application/xml');
Or, leave the ContentType blank (in which case, application/octet-stream will be used instead):
aIdStream.AddFile('file', 'e:\wm_req.xml');
If you don't want to send a ContentType at all, you can set the TIdFormDataField.ContentType property to a blank string:
with aIdStream.AddFile('file', 'e:\wm_req.xml') do
begin
FileName := ''; // needed so ContentType setter will not use a default value...
ContentType := '';
FileName := 'wm_req.xml'; // or whatever you want, or leave blank...
end;
But, more importantly, you are NOT specifying a boundary attribute for the TIdHTTP.Request.ContentType header anymore. In fact, you SHOULD NOT be saving the TIdMultipartFormDataStream to a TStringStream and then posting the TStringStream at all. You SHOULD be posting the TIdMultipartFormDataStream itself instead. That will automatically set the TIdHTTP.Request.ContentType to multipart/form-data, as well as give it a correct boundary attribute (whose value is generated by TIdMultipartFormDataStream):
aHTTP.Post(aURL, aIdStream)
Now, that being said, in both cases, you need to remove the call to TIdURI.URLDecode() on the return value of TIdHTTP.Post(). The server should be sending you raw XML, not URL-encoded XML, so you need to process the XML as-is and not URL-decode it at all.
Also, since XML is technically binary data that could be encoded in any charset that may or may not be present in the HTTP response headers, you SHOULD have your XML parser process the XML exactly as it is sent by the server. To do that, use the overloaded version of TIdHTTP.Post() that populates a TStream instead of returns a String. Have it receive the server's XML into a TMemoryStream, and then load that stream into your parser (TNativeXml has a LoadFromStream() method).
Lastly, you show that the server's response has a Content-Encoding: gzip response header, but I do not see you assigning any object to the TIdHTTP.Compressor property, or manually setting the TIdHTTP.Request.AcceptEncoding property, so the server SHOULD NOT be sending the response body in a compressed format since you are not asking for that format. TIdHTTP will not automatically decompress the data without a Compressor assigned.
If the server really is sending compressed data without being asked, that is a bug on the server side that you should report to Walmart.

Trying to http get "www.shopyourway.com" Fail

I tried to do a http get for the website http://www.shopyourway.com using ruby Net::HTTP.get, but I got an error with code 512. And I tried to do a get with ssl for url "https://www.shopyourway.com". It just followed a redirection to the url without ssl.
code is as below:
uri = URI('https ://www.shopyourway.com') #space between https and : does not exist
body = Net::HTTP.get(uri)
I can browse the url using browser. But why I can't do a http get for that url?
Thanks
finally get this to works, need to add couple headers to the Get request.
uri = URI('http://www.shopyourway.com/today')
req = Net::HTTP::Get.new(uri)
req['Upgrade-Insecure-Requests'] = '1'
req['Connection'] = 'keep-alive'
req['User-Agent'] = 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_2) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/55.0.2883.95 Safari/537.36'

F# error FS0193 (VS 2010) : The type 'WebRequest' is not compatible with the type 'HttpWebRequest'

Recently I work in a CBIR student project. And we need lots of photos of world landmarks. I've found this Flickr Crawler project (blog, sourcecode) written in F#. But I encounter an error when building it in my Visual Studio 2010:
error FS0193:
Type constraint mismatch. The type WebRequest is not compatible with type HttpWebRequest . The type 'WebRequest' is not compatible with the type 'HttpWebRequest'
Line 2 (let req=...) is where the error happens:
let downloadUrl(url:string) =
let req = HttpWebRequest.Create(url) :> HttpWebRequest
req.UserAgent <- "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322)";
req.Method <- "GET";
req.AllowAutoRedirect <- true;
req.MaximumAutomaticRedirections <- 4;
req.MaximumAutomaticRedirections <- 4;
let resp = req.GetResponse()
let stream = resp.GetResponseStream()
let reader = new StreamReader(stream)
reader.ReadToEnd()
I did not learn F# before so I am not familiar with F#. What does this error message suggest and how do I solve it?
The error was quite small. The :> operator will only perform casts which are known to succeed at compile time and so won't cast from HttpWebRequest.Create(url) to HttpWebRequest as this actually returns a WebRequest. As a reult you need to use
let req = HttpWebRequest.Create(url) :?> HttpWebRequest
instead to allow for casting in the other direction in the inheritance tree. The difference is that casts made by :?> may fail at run time (but shouldn't in this case as all urls should be http)

Forming Xpath Query for total Google results

I've used this formula in Google Spreadsheets in the past to input the number of search results into a cell. The formula's not working anymore and I know I need to fix the xpath query.
any ideas?
Current formula:
=importXML("http://www.google.com/search?num=100&q="&A2,"//p[#id='resultStats']/b[3]")
Spreadsheet for public testing:
https://spreadsheets8.google.com/ccc?key=tJzmFllp7Sk1lt23cXSVXFw&authkey=CM625OUO#gid=0
=ImportXML(A2,"//*[#id='resultStats']")
It doesn't produce the number only, but it's a start
Set C2 =ImportXML(A2,"//*[#id='resultStats']")
Then use =REGEXREPLACE(C2 ;"^About(.*) results";"$1")
import requests
from bs4 import BeautifulSoup
import re
def google_results(query):
url = 'https://www.google.com/search?q=' + query
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:57.0) Gecko/20100101 Firefox/57.0'
}
html = requests.get(url, headers=headers).text
soup = BeautifulSoup(html, 'html.parser')
div = soup.find('div', id='resultStats')
return int(''.join(re.findall(r'\d+', div.text.split()[1])))
print(google_results('test'))

Resources