mirror of https://github.com/koreader/koreader
You cannot select more than 25 topics
Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
6f1b70e5eb
Previously the CJK character detection defined only characters in the range U+4000..U+AFFF as "CJK characters". This excludes an incredibly large number of CJK characters within the BMP, let alone the whole two planes dedicated to rarer CJK characters (the SIP and TIP). As a result, a very large number of Chinese, Japanese, and Korean characters were not detected as being CJK characters. While slightly less elegant-looking, it is far more accurate to compute the codepoint from the utf8 character and then see if it falls within one of the defined CJK blocks. This is not future-proof against future CJK ideograph extensions in future Unicode versions, but there is no real way to accurately predict such changes so this is the best we can do without accidentally treating characters explicitily defined as being non-CJK in Unicode as CJK. While we're at it, copy Lua 5.3's utf8.charpattern constant definition so that we can more easily write utf8 iterators with string.gmatch (at least in the interim until there is a rework of utf8 handling in KOReader and everything is rebuilt on top of utf8proc). Some unit tests are added for Korean and Japanese text, and the existing unit tests needed a minor adjustment to handle the fact that isSplittable now correctly detects CJK punctuation as a character to compare against the forbidden split rules. Signed-off-by: Aleksa Sarai <cyphar@cyphar.com> |
3 years ago | |
|---|---|---|
| .. | ||
| unit | 3 years ago | |