[FFe] Demilight (OS/2 weight=350) confuses fontconfig

Hi again, Mingye!

I think that fontconfig .conf files are effective whether the fonts were installed via .deb packages or in some other way, as long as the font files are placed in a directory which fontconfig looks at. With that said, if it's possible to make fontconfig pick the most sensible default font weight without extra configuration, it's indeed a better way.

Hi Mingye,

I tried to build fontconfig in a PPA with your patch. The patch applies, but unfortunately the build failed.

"error: could not locate FcRangeCreateDouble in src/*.c"

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test/+packages

I may have missed something. You know, naive cherry-picking can go wrong...

Found the commit which added FcRangeCreateDouble and cherry-picked it to the result of applying my current patch. Note that this commit (3cd573f) is way before the ones in the patch (be6506c, bf9df5a, ffda7c0, 01bb697, 80edacc) and you may want to do the cherry-pick yourself in the correct order instead of using my shuffled crap....

So here is 3cd573f.

Thanks for your attempt to fix it. Haven't tried the new patch yet, but before I saw it, I created a new PPA for trying the Arch Linux way: A possible upgrade to fc 2.11.94:

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test2

Much better result. :) (The below is without the influence of the language-selector .conf files.)

$ fc-match -V
fontconfig version 2.11.1
$ fc-match -a | grep 'Noto Sans CJK SC'
NotoSansCJK.ttc: "Noto Sans CJK SC" "DemiLight"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Regular"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Medium"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Thin"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Light"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Bold"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Black"
$ fc-match -V
fontconfig version 2.11.94
$ fc-match -a | grep 'Noto Sans CJK SC'
NotoSansCJK.ttc: "Noto Sans CJK SC" "Regular"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Medium"
NotoSansCJK.ttc: "Noto Sans CJK SC" "DemiLight"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Light"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Thin"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Bold"
NotoSansCJK.ttc: "Noto Sans CJK SC" "Black"

I can't tell if it would be possible to have this accepted for 16.04. It would be very interesting to know if fc 2.11.94 also fixes the Firefox issue, which has been reported at bug #1468027 (see my latest comment there).

A potential problem with 2.11.94 is that it is actually a devel snapshot for 2.12 (https://www.freedesktop.org/wiki/Software/fontconfig/Devel/), so there may be some new bugs (e.g. fd.o 94427) lying around, and similar efforts may be needed to bump fc@16.04 to 2.12.0 when 2.12 is out...

Even if 2.11.94 is a development release, the answer you got from a fontconfig developer (<https://bugs.freedesktop.org/94505#c5>) is striking. The alternative, i.e. patching the "ancient" 2.11.1 with multiple selected commits, is also risky business.

As regards <https://bugs.freedesktop.org/94427>, I'm not able to tell how big problem it is in practice. Wouldn't that one happen with your proposed patches?

I'll talk to someone on IRC (#ubuntu-desktop) tomorrow, and get their view.

https://launchpadlibrarian.net/251449771/upstream_git-log.txt has a lot of changes which seem desirable, and not actually much in terms of new features (aside from documentation updates). However, the sheer number/size of the changes makes this risky, particularly as testing this is extremely difficult (e. g. how reliably can we test Ubuntu/Kubuntu/Xubuntu in Bengali, Thai, various CJK, or e. g. Georgian and are confident that fonts look right?).

I don't know much about fontconfig, but is there any better/simpler way of mass-testing with our fonts than visual inspection? I. e. can we compare which glyphs are being used from which font (for our default font selection) with the current and the new version, to ensure that we don't suddenly swap fonts around, or if that happens, that this is expected? Is there some way to iterate over the unicode glyphs for fonts that we have available, to ensure that the new fontconfig does not crash on those? Any other ideas for a test plan?

Does this need any other dependencies (freetype etc.)? I. e. if 2.11.94 causes trouble, how easily can we go back to 2.11.1?

Thanks for your comments, Martin.

I for one haven't considered extensive testing yet. The reason for the proposal is to prevent a regression wrt Chinese rendering (due to new default font), and we have a positive report from a Chinese user at bug #1468027 (comment #105).

As regards your last question, no changes were made to debian/control, so AFAICT going back to 2.11.1 if necessary would be easy.

The reporter of this bug report, Mingye Wang, has identified a bunch of upstream commits (attached as patches to this bug report), and patching 2.11.1 with them *might* be an alternative to upgrading. We haven't tested that approach successfully yet. However, given the number of commits that would need to be cherry picked, I'm not sure it would be a safer solution.

> Is there some way to iterate over the unicode glyphs for fonts that we have available, to ensure that the new fontconfig does not crash on those?

fc-match doesn't seem to provide glyph-level lists, but anyway `diff`-ing over some fallback order dump should help:

#!/bin/bash
FC_MATCH=/usr/bin/fc-match
# use dictionary keys as a string-set
declare -A locales

# grab a list-of-locales
for i in /var/lib/locales/supported.d/*; do
  mapfile -t arr < "$i"
  # kill the longest trailing [. ]*, so what we got are basically the lang_LOC part
  for l in "${arr[@]%%[. ]*}"; do
    locales["$l"]='' # make it defined, that's all we need to snap the key in
  done
done

for loc in "${!locales[@]}"; do
  echo "# $loc:"
  export LC_ALL=${loc}.UTF-8
  for font in Sans Serif Monospace Ubuntu; do
    for weight in regular bold; do
      # more inner loops for styles.......
      echo "* $font:weight=$weight:"
      fc-match -a "$font:weight=$weight:"
      echo
    done
  done
  echo
done

> Does this need any other dependencies (freetype etc.)? if 2.11.94 causes trouble, how easily can we go back to 2.11.1?

No.

2.11.94 seems to be 100% ABI and API compatible with 2.11.1.
Archlinux and AOSC OS survived the update, and Fedora has been using 2.11.94 for quite a while.

Oops, oops, I meant to use "$FC_MATCH" -a in the script posted.

Considering Martin's hesitation, I completed the attempt to fix it using the other approach: Patching a bunch of cherry picked upstream commits:

https://launchpad.net/~gunnarhj/+archive/ubuntu/fontconfig-test

Seems to work. I'll ask at the other bug report too.

Building right now. Thanks for your persistence on this matter, Mingye. ;)

This bug was fixed in the package fontconfig - 2.11.1-0ubuntu9

---------------
fontconfig (2.11.1-0ubuntu9) xenial; urgency=medium

  * debian/control:
    - Add gperf to build dependencies
    - Bump FreeType build dependency version to 2.5.1

  [ Mingye Wang ]
  * debian/patches/0002-demilight.patch:
    - Handle Demilight sensibly (LP: #1556457)

 -- Gunnar Hjalmarsson <email address hidden> Mon, 04 Apr 2016 21:45:00 +0200

Gunnar, could you please check https://bugs.launchpad.net/ubuntu/+source/fontconfig/+bug/1566651

I believe it's related to this fontconfig update.

Reopening due to regression report.

I discussed that with Gunnar and Sèbastien on IRC. Given the known regressions with 2.11.1 and the backported patch and that .94 has been used for a while in other distros and that most changes are bug fixes, we agreed to go ahead with the update. Still risky, but Gunnar will send out a call for testing, and if we find regressions we can revert back to 2.11.1 and revisit that.

Version 2.11.95 (2.12 RC5) was just released 6 hours ago.
https://cgit.freedesktop.org/fontconfig/tree/README

2.11.95 (2.12 RC5)

Akira TAGOH (22):
      Add one more debugging option to see transformation on font-matching
      Fix a crash when no objects are available after filtering
      No need to be public
      mark as private at this moment
      Don't return FcFalse even when no fonts dirs is configured
      Add a warning for blank in fonts.conf
      Fix a memory leak in FcFreeTypeQueryFace
      Update CaseFolding.txt to Unicode 8.0
      Bug 90867 - Memory Leak during error case in fccharset
      Fix the broken cache more.
      Fail on make runtime as needed instead of configure if no python installed
      Use long long to see the same size between LP64 and LLP64
      Fix build issue on MinGW
      Use int64_t instead of long long
      Fix compiler warnings on MinGW
      Fix assertion on 32bit arch
      remomve unnecessary code
      Bug 93075 - Possible fix for make check failure on msys/MinGW...
      Avoid an error message on testing when no fonts.conf installed
      Add hintstyle templates and make hintslight default
      Revert "Workaround another race condition issue"
      Update libtool revision

Behdad Esfahbod (6):
      Revert changes made to FcConfigAppFontAddDir() recently
      Call FcFreeTypeQueryFace() from fcdir.c, instead of FcFreeTypeQuery()
      [GX] Support instance weight, width, and style name
      [GX] Enumerate all named-instances in TrueType GX fonts
      Improve OpenType to Fontconfig weight mapping
      [GX] Improve weight mapping

Patrick Haller (1):
      Optimizations in FcStrSet

On 2016-04-06 19:53, Amr Ibrahim wrote:
> Version 2.11.95 (2.12 RC5) was just released 6 hours ago.

Right. The reasons why 2.11.94 was still uploaded for now is that we are very late in the 16.04 development cycle, and 2.11.94 has been in use for a while by some distros.

Since fontconfig 2.11.94 is on its way into the archive, I'd like to mention this "call for testing" message I posted to the ubuntu-devel mailing list:

https://lists.ubuntu.com/archives/ubuntu-devel/2016-April/039303.html

> Version 2.11.95 (2.12 RC5) was just released 6 hours ago.

Right, we started with .94 as we already had a tested package with that. But we indeed should track the .9x upstream releases now, changelog looks fine.

This bug was fixed in the package fontconfig - 2.11.94-0ubuntu1

---------------
fontconfig (2.11.94-0ubuntu1) xenial; urgency=medium

  * New upstream release (LP: #1556457)
    - Fixes blurry fonts regression from previous upload (LP: #1566651)
  * d/p/0001-Revert-Bug-73291-poppler-does-not-show-fl-ligature.patch,
    d/p/0002-demilight.patch:
    - Dropped, applied in new release
  * Bump freetype build dep to 2.5.1 as per configure.ac.
  * Drop gperf build dep again, not necessary any more.

 -- Gunnar Hjalmarsson <email address hidden> Sun, 03 Apr 2016 02:54:00 +0200

Thanks Gunnar,

2.11.9x are release candidates on the road to 2.12. I just wanted to mention that.