UrlUnescape() and unicode chars - windows

Question

I am trying to write the program on C++ that will decode the URL-encoded string that contains some URL-encoded unicode characters.
#include <windows.h>
#include <string>
#include <shlwapi.h>
#pragma comment(lib, "Shlwapi.lib")
int _tmain(int argc, _TCHAR* argv[])
{
std::wstring test = L"bla+%D0%B1%D0%BB%D0%BE%D1%84+%E6%97%A5%E6%9C%AC%E8%AA%9E";
PWSTR urlencodedStr = const_cast<WCHAR*>(test.c_str());
WCHAR decodedStr[1025];
DWORD size = 1024;
HRESULT hres = UrlUnescape(urlencodedStr, decodedStr, &size, NULL);
if (hres == S_OK)
MessageBox(NULL, decodedStr, L"decoded string", MB_OK);
return 0;
}
I'm expecting to get L"bla блоф 日本語" in decodedStr. But i'm getting L"bla+блоф+日本語" instead.
I am using unicode charset in my build.
What am i doing wrong?

Answer 1

UrlUnescape converts URL-decoded %xx bytes into characters using the default (ANSI) code page by default. This is almost never what you want.
From Windows 8 onwards, you can pass the UNESCAPE_AS_UTF8 flag to make it behave. If you can't depend on Win8, you'll have to use/write a different URL-decoding library call that doesn't suffer from this problem.
Also there is the issue of the +: in plain-URL-encoding (for example for use in a path part), this means a plus, but in form-url-encoding (for example in a query parameter), which is what you seem to have here, it means a space. A good URL decoder will give you the option to say which one you mean; UrlUnescape does not. An alternative is to manually replace the + with space on input before URL-decoding; this is one special case and no other characters are similarly affected.

Answer 2

Ok. So i wrote my own function to decode URL-encoded strings with unicode characters. Here it is:
#include <windows.h>
#include <string>
#include <shlwapi.h>
#include <sstream>
#include <iostream>
#include <wininet.h> // For INTERNET_MAX_URL_LENGTH
#pragma comment(lib, "Shlwapi.lib")
bool IsHexChar(const WCHAR _char)
{
return ((_char == L'A') ||
(_char == L'B') ||
(_char == L'C') ||
(_char == L'D') ||
(_char == L'E') ||
(_char == L'F') ||
iswalnum(_char));
}
std::wstring UrlDecode(const std::wstring& _encodedStr)
{
std::string charStr;
for (size_t i = 0; i < _encodedStr.length(); ++i)
{
if ((_encodedStr[i] == L'%') && (IsHexChar(_encodedStr[i+1])) && (IsHexChar(_encodedStr[i+2])))
{
std::wstring hexCodeStr = L"0x";
hexCodeStr += _encodedStr[i+1];
hexCodeStr += _encodedStr[i+2];
unsigned int hexCharCode;
std::wstringstream ss;
ss << std::hex << hexCodeStr;
ss >> hexCharCode;
charStr += static_cast<char>(hexCharCode);
i += 2;
}
else if (_encodedStr[i] == L'+')
charStr += L' ';
else
charStr += _encodedStr[i];
}
WCHAR decodedStr[INTERNET_MAX_URL_LENGTH];
MultiByteToWideChar(CP_UTF8, 0, charStr.c_str(), -1, decodedStr, sizeof(decodedStr));
return decodedStr;
}
Use like this:
std::wstring encodedStr = L"bla+%D0%B1%D0%BB%D0%BE%D1%84+%E6%97%A5%E6%9C%AC%E8%AA%9E";
std::wstring decodedStr = UrlDecode(encodedStr);