I know nothing about Arabic writing, but we need to add support for it.
I'm getting confused about the letter order. As you can see in the screenshot, the order of the characters is different depending on the display method.
In Xcode I also noticed a different letter order in the preview and in the description.
Your screenshots have same letter order, but different layout directions, i.e. the ordering of fragments of text (e.g. words). If you remove the left-to-right fragment 12345 and the letter z, the string will look the same in both cases.
You can learn about the tricky bidirectional text layout in Wikipedia.
In your case, I believe that U+200F right-to-left mark as the first character of your string will fix the problem. Be careful with the editor, though. XCode editor does not support bidirectional text good enough.
Related
I have not done any MACOS programming in several years. However, I now need to test out some algorithms for formatting text as paragraphs. For these tests, I need to be able to display characters with absolute position control.
These tests need to determine where line breaks are to go within a paragraph and the amount of work spacing, while taking into account hyphenation, ligatures, page breaks, kerning and the like.
When I last did such things, I was using the text functions of the CGContext class. Upon returning from my hiatus, I find all of these function are marked "deprecated" in the documentation—which conveniently leaves out what the current acceptable replacement is.
What is the current, approved process for drawing single glyphs on Mac OS?
Functions that deal with formatting strings would defeat the purpose of the academic experiment here.
Correct display of Unicode in a terminal would appear to benefit from the displaying app knowing the number of character cells used to display text. Functions like wcwidth() are a reasonable start, but there can be a lot of variation, for example what a terminal displays for invalid characters, ambiguous width Asian characters, combining characters out of context, etc.
Would it be reasonable to extend terminal apps with a new control sequence to measure with display width of a string, which display apps could use to characterize the terminal? If so, what details are worth considering, e.g. what sequence to use, whether to specify UTF-8, also how to handle terminals that do not know this hypothetical new control sequence? Would it have any likelihood of wide adoption?
If not, what is the flaw in the idea? Is perhaps reading the cursor position after display a better (and already supported) option? Or is there a good different approach?
There's no need, because the existing cursor position report (which can be used to get the position before and after printing a string) gives the length.
Adding a new control sequence to get the attributes of a character (width, combining, controls such as tab) wouldn't help much because the application still has to work with the system's locale information for performance reasons: it would drastically slow down an application if it had to ask after each character where the cursor really was.
Background
The Arabic letter noon ghunna (ں) is displayed incorrectly on my Windows 10 PC (in Chrome, Edge, Notepad and Word). The sequence ALEF, NOON GHUNNA, ALEF is displayed as:
The same sequence is displayed correctly on my Android phone without the dot:
For completeness, the actual unicode string (for copy/paste purposes) is:
اںا
There has been some controversy regarding this letter (L2-12/381) which has settled by now as seen from the Unicode Standard which states (since version 7 and up to the current 11):
Rendering systems should display U+06BA as a dual-joining letter, with all four contextual forms shown dotless, regardless of the language of the text.
But the dot appears in word-initial (ںا) and mid-word (اںا) positions. Final (اں) and isolated (ں) forms are fine.
Question
Now my question is, how can this be fixed, other than by waiting for Microsoft to fix it? I want to understand where the problem lies. Is it in the Uniscribe library, or is it down to the font being used? Can it be fixed by using a specifically crafted TrueType/OpenType font?
This turned out to be a font problem. Quite a few fonts on fonts.google.com show this letter correctly:
https://fonts.google.com/?subset=arabic&selection.family=Amiri|Aref+Ruqaa|Cairo|El+Messiri|Harmattan|Lemonada|Mada|Reem+Kufi|Scheherazade
I have found this letter / character in facebook, but how can this have a color? is just insane for me, look this: ✔️
Added image (From Firefox on windows)
It's not an ASCII character, it's likely an emoji. Emoji are part of Unicode and the actual glyph displayed to the user is open to interpretation by the platform displaying it. The spec suggests a name/description, but the implementation varies.
So while you may see a colored check mark, I see black & white. Other times, a single glyph will have multiple styles made available on a particular platform; for example, I can select multiple "skin" tones when I use a smiley face on my iPhone, but your Android device may only show a generic one.
Edit: The image edited into the original post is a perfect example. Using Chrome on Windows, I see a black check mark. The screenshot from Firefox shows green.
The symbols used here aren't ascii-encoded. They use the much more vast range of Unicode encoding. Ascii(extended) is restricted to a 256 symbol set.
The unicode interpretation for symbols/glyphs(small pictorial representation)(these ticks aren't characters), can vary for different platforms as some the range of unicode is open for usage and isn't set as global.
Which is why, while the unicode encryption remains the same for every device irrespective, the decryption is differently interpreted by different devices/online-platforms, allowing us to perceive either a coloured or a black symbol.
This picture illustrates my predicament:
All of the characters appear to be the same size, but the space between them is different when presented in a RichEdit control compared with when I use ExtTextOut.
I would like to present the characters the same as in the RichEdit control (ideally), in order to preserve wrap positions.
Can anyone tell me:
a) Which is the more correct representation?
b) Why the RichEdit control displays the text with no gaps between the Asian Characters?
c) Is there any way to make ExtTextOut reproduce the behaviour of the RichEdit control when drawing these characters?
d) Would this be any different if I was working on an Asian version of Windows?
Perhaps I'm being optimistic, but if anyone has any hints to offer, I'd be very interested to hear.
In case it helps:
Here's my text:
快的棕色狐狸跳在懶惰狗1 2 3 4 5 6 7 8 9 0
apologies to Asian readers, this is merely for testing our Unicode implemetation and I don't even know what language the characters are taken from, let alone whether they mean anything
In order to view the effect by pasting these characters into a RichEdit control (eg. Wordpad), you may find you have to swipe them and set the font to 'Arial'.
The rich text that I obtain is:
{\rtf1\ansi\ansicpg1252\deff0\deflang2057{\fonttbl{\f0\fnil\fcharset0 Arial;}}{\colortbl ;\red0\green0\blue0;}\viewkind4\uc1\pard\sa200\sl276\slmult1\lang9\fs22\u24555?\u30340?\u26837?\u33394?\u29392?\u29432?\u36339?\u22312?\u25078?\u24816?\u29399?1 2 3 4 5 6 7 8 9 0\par\pard\'a3 $$ \'80\'80\cf1\lang2057\fs16\par}
It doesn't appear to contain a value for character 'pitch' which was my first thought.
I don't know the answer, but there are several things to suspect:
There are several versions of the rich edit control. Perhaps you're using an older one that doesn't have all the latest typographic improvements.
There are many styles and flags that affect the behavior of a rich editcontrol, so you might want to explore which ones are set and what they do. For example, look at EM_GETEDITSTYLE.
Many Asian fonts come in two versions on Windows. One is optimized for horizontal layout, and the other for vertical layout. That latter usually has the same name, but has # prepended to it. Perhaps you are using the wrong one in the rich edit control.
UPDATE: By messing around with Wordpad, I was able to reproduce the problem with the crowded text in the rich edit control.
Open a new document in Wordpad on Windows 7. Note that the selected font is Calibri.
Paste the sample text into the document.
Text appears correct, but Wordpad changed the font to SimSun.
Select the text and change the font back to Calibri or Arial.
The text will now be overcrowded, very similar to your example. Thus it appears the fundamental problem is with font linking and fallback. ExtTextOut is probably selecting an appropriate font for the script automatically. Your challenge is to figure out how to identify the right font for the script and set that font in the rich edit control.
This will only help with part of your problem, but there is a way to draw text to a DC that will look exactly the same as it does with RichEdit: what's called the windowless RichEdit control. It not exactly easy to use: I wrote a CodeProject article on it a few years back. I used this to solve the problem of a scrollable display of blocks of text, each one of which can be edited by clicking on it: the normal drawing is done with the windowless RichEdit, and the editing by showing a "real" RichEdit control on the top of it.
That would at least get you the text looking the same in both cases, though unfortunately both cases would show too little character spacing.
One further thought: if you could rely on Microsoft Office being installed, you could also try later versions of RichEdit that come with office. There's more about these on Murray Sargent's blog, as well as some interesting articles on font binding that might also help.
ExtTextOut allows you to specify the logical spacing between records. It has the parameter lpDx which is a const pointer to an array of values that indicate the distance between origins of adjacent character cells. The Microsoft API documentation notes that if you don't set it, then it sets it's own default spacing. I would have to say that's why ExtTextOut is working fine.
In particular, when you construct a EMR_EXTTEXTOUTW record in EMF, it populates an EMR_TEXT structure with this DX array - which looking at one of your comments, allowed the RichEdit to insert the EMF with the information contained in the record, whereby if you didn't set a font binding then the RTF record does some matching to work out what font to use.
In terms of the RichEdit control, the following article might be useful:
Use Font Binding in a Rich Edit Control
After character sets are assigned, Rich Edit scans the text around the
insertion point forward and backward to find the nearest fonts that
have been used for the character sets. If no font is found for a
character set, Rich Edit uses the font chosen by the client for that
character set. If the client hasn't specified a font for the character
set, Rich Edit uses the default font for that character set. If the
client wants some other font, the client can always change it, but
this approach will work most of the time. The current default font
choices are based on the following table. Note that the default fonts
are set per-process, and there are separate lists for UI usage and for
non-UI usage.
If you haven't set the characterset, then it further explains that it falls back to ANSI_CHARSET. However, it's most definitely a lot more complicated than that, as that blog article by Murray Sargent (a programmer at Microsoft) shows.