How to widen a standard string while maintaining the characters? - c++11

Question

I have a function to take a std::string and change it into a wchar_t*. My current widen function looks like this
wchar_t* widen(const std::string& str){
wchar_t * dest = new wchar_t[str.size()+1];
char * temp = new char[str.size()];
for(int i=0;i<str.size();i++)
dest[i] = str[i];
dest[str.size()] = '\0';
return dest;
}
This works just fine for standard characters, however (and I cannot believe this hasn't been an issue before now) when I have characters like á, é, í, ó, ú, ñ, or ü it breaks and the results are vastly different.
Ex: my str comes in as "Database Function: áFákéFúnctíóñü"
But dest ends up as: "Database Function: ￡F￡k￩Fnct￭￳￱"
How can I change from a std::string to a wchar_t* while maintaining international characters?

Answer 1

Short answer: You can't.
Longer answer: std::string contains char elements which typically contain ASCII in the first 127 values, while everything else ("international characters") is in the values above (or the negative ones, if char is signed). In order to determine the according representation in a wchar_t string, you first need to know the encoding in the source string (could be ISO-8859-15 or even UTF-8) and the one in the target string (often UTF-16, UCS2 or UTF-32) and then transcode accordingly.

Answer 2

It depends if the source is using old ANSI code page or UTF8. For ANSI code page, you have to know the locale, and use mbstowcs. For UTF8 you can make a conversion to UTF16 using codecvt_utf8_utf16. However codecvt_utf8_utf16 is deprecated and it has no replacement as of yet. In Windows you can use WinAPI function to make the conversions more reliably.
#include <iostream>
#include <string>
#include <codecvt>
std::wstring widen(const std::string& src)
{
int len = src.size();
std::wstring dst(len + 1, 0);
mbstowcs(&dst[0], src.c_str(), len);
return dst;
}
int main()
{
//ANSI code page?
std::string src = "áFákéFúnctíóñü";
setlocale(LC_ALL, "en"); //English assumed
std::wstring dst = widen(src);
std::wcout << dst << "\n";
//UTF8?
src = u8"áFákéFúnctíóñü";
std::wstring_convert<std::codecvt_utf8_utf16<wchar_t>, wchar_t> convert;
dst = convert.from_bytes(src);
std::wcout << dst << "\n";
return 0;
}

Answer 3

For a Windows solution, here's some utility functions I use based on the wisdom of http://utf8everywhere.org/
/// Convert a windows UTF-16 string to a UTF-8 string
///
/// #param s[in] the UTF-16 string
/// #return std::string UTF-8 string
inline std::string Narrow(std::wstring_view wstr) {
if (wstr.empty()) return {};
int len = ::WideCharToMultiByte(CP_UTF8, 0, &wstr[0], wstr.size(), nullptr, 0,
nullptr, nullptr);
std::string out(len, 0);
::WideCharToMultiByte(CP_UTF8, 0, &wstr[0], wstr.size(), &out[0], len,
nullptr, nullptr);
return out;
}
/// Convert a UTF-8 string to a windows UTF-16 string
///
/// #param s[in] the UTF-8 string
/// #param n[in] the UTF-8 string's length, or -1 if string is null-terminated
/// #return std::wstring UTF-16 string
inline std::wstring Widen(std::string_view str) {
if (str.empty()) return {};
int len = ::MultiByteToWideChar(CP_UTF8, 0, &str[0], str.size(), NULL, 0);
std::wstring out(len, 0);
::MultiByteToWideChar(CP_UTF8, 0, &str[0], str.size(), &out[0], len);
return out;
}
Usually used inline in windows API calls like:
std::string message = "Hello world!";
::MessageBoxW(NULL, Widen(message).c_str(), L"Title", MB_OK);
A cross-platform and possibly faster solution could be found by exploring Boost.Nowide's conversion functions: https://github.com/boostorg/nowide/blob/develop/include/boost/nowide/utf/convert.hpp