Qt applications can't open files which have special characters in their name

bsduck

For example I create a text file called "caffè". I can open it with Leafpad (GTK2) but FeatherPad (Qt5) tells me "The file does not exist". If I rename it to "caffe", it works.

Same with audio files in Strawberry.

This reminds me of https://www.unitedbsd.com/d/606-keyboard-dead-keys-dont-work-in-qt-applications

What am I missing this time?

(NetBSD current, LXQt)

rvp

bsduck For Featherpad, enable the Native file dialog in Options > Preferences > Window

nia

It's working for me with these locale settings:

$ locale
LANG=""
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="C"
LC_TIME="en_GB.UTF-8"
LC_NUMERIC="C"
LC_MONETARY="C"
LC_MESSAGES="C"
LC_ALL=""

(What weird locale settings. Maybe they should all be "C.UTF-8" except TIME...)

Whether "Native file dialog" is ticked, or not, doesn't seem to change anything.

I'm wondering if Qt thinks you have a non-UTF-8 locale and is doing strange stuff.

rvp

nia Whether "Native file dialog" is ticked, or not, doesn't seem to change anything.

Without that checked (in an openbox WM session), I got the behaviour described by OP (and the file dialog widget is also different). The same when I ran Featherpad standalone (ie. without the LXQT DE). I only use these locale settings everywhere:

export LANG=en_GB.UTF-8
export LC_CTYPE=$LANG
export LC_ALL=""

nia

rvp What does it look like? I'm curious what a "native" file dialog is on Unix...

Hmm... I have,

~ > locale
LANG="en_US.UTF-8"
LC_CTYPE="en_US.UTF-8"
LC_COLLATE="C"
LC_TIME="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_ALL=""

If I create a file named caffé it doesn't show in the open dialogue of FeatherPad, with or without Native file dialog enabled.
I don't really care since I never use special characters in file names but, I'm wondering what type of settings are required for it to work. I don't have an IME enabled or, even installed.

rvp

This is the "native" file chooser where all files are listed (forget the box-chars. in the filenames--the standard fonts that come with native X in 9.2_STABLE and LXQT (2022Q2) don't cover these code-points):

This is the built-in one where filenames containing Hangul are not shown:

pin If I create a file named caffé it doesn't show in the open dialogue of FeatherPad, with or without Native file dialog enabled.

Did you run Featherpad inside LXQT or standalone? If standalone, then you always get the built-in file chooser.

rvp Did you run Featherpad inside LXQT or standalone?

Standalone, no DE just a WM.

rvp

pin Standalone, no DE just a WM.

That doesn't work. And neither does running Featherpad from an xterm started before exec startlxqt. You need the whole LXQT kit and caboodle running for this.

Bug-report time? :D

rvp Now I get it :)

rvp Bug-report time? :D

Maybe.

So, I didn't open an issue but, I did ask the featherpad dev. Quoting,

That can't be related to FeatherPad, its standalone mode, or LXQt file dialog. It should be about something much more basic.

I don't know about *BSD, but in Linux, all file names are in UTF-8, which supports all possible characters in the world.

Of course, the font you use may not support all characters. For Korean, Japanese, Chinese and some other languages, you need appropriate fonts. Since I installed them a long time ago, I don't remember their names; sorry!

However, something fundamental may be missing or wrong in that system, because a simple text like "caffè" shouldn't make any trouble with any well-known font like DejaVu Sans. Actually, if the name was in Korean, Chinese, Japanese,..., it shouldn't make any trouble in file opening either; at most, in the absence of appropriate fonts, file names wouldn't be human-readable.

https://www.unitedbsd.com/d/606-keyboard-dead-keys-dont-work-in-qt-applications/ suggests that there's a general problem with Qt. Perhaps it's related to how Qt is compiled there (just a wild guess). Ping me if you find the cause; I got curious.

Because Qt has never had that problem here, and supposing that it happens anywhere Qt apps are used, logically, I think there are three possibilities: (1) Something may be wrong in Qt for BSD (if it exits at all); (2) It may be related to how Qt is compiled/used; or (3) The cause may be outside Qt but related to it.

If it doesn't happen everywhere Qt apps are used, then the problem is definitely outside Qt.

But my knowledge of BSD is next to zero; so, please don't be misled by my words!

bsduck

"Native file dialog" is enabled by default in FeatherPad. This file dialog can see my caffè but refuses to drink it (the Open button is greyed out when this file is selected). With that option disabled, the built-in file chooser just doesn't show my file.

I tried using LC_ALL=C as well as LC_ALL=C.UTF-8, and also tried with the IME disabled, but that didn't solve the problem.

It can't be a bug in FeatherPad specifically because I get the same with Strawberry. However it isn't a general issue with Qt programs either: PCManFM-Qt and qpdfview aren't affected.

bsduck It can't be a bug in FeatherPad specifically because I get the same with Strawberry. However it isn't a general issue with Qt programs either: PCManFM-Qt and ...

I can confirm this with pcmanfm-qt and xpdf-4, puzzling.

rvp

bsduck This is the result of wrongly using the result of a char. conversion instead of the original filename. A ktrace shows what's happening:

$ ktrace fpad
$ kdump > k.txt
[...]
  1051     31 fpad     CALL  __lstat50(0x7aef1933be00,0x7aeef6fffcc0)
  1051     31 fpad     NAMI  "/mnt/snd/Korean/??? ?? ??-dNyDsJtY2sI.webm"
  1051     31 fpad     RET   __lstat50 -1 errno 2 No such file or directory
[...]

Something is converting UTF-8 strings into (probably) US-ASCII using the standard iconv() function and then passing this string (instead of the original filename) when opening the file. (iconv() replaces any valid multibyte sequence it can't convert with a ? char.)

In the "native" file chooser, the filenames are passed correctly when listing the directory, however:

[...]
  1051     28 fpad     CALL  access(0x7aef187c6180,2)
  1051     28 fpad     NAMI  "/mnt/snd/Korean/아이유 너의 의미-dNyDsJtY2sI.webm"
  1051     28 fpad     RET   access -1 errno 30 Read-only file system
[...]

You can do this same conversion yourself:

$ ls /mnt/snd/Korean/ | iconv -s -f utf8 -t us-ascii
Shin Joong Hyun & The Men - Beautiful Rivers and Mountains-d_3Qe7KfBJA.webm
playlist.m3u
???? ???? The Meaning of You-zXkhG2rSkAo.webm
??? ?? ??-dNyDsJtY2sI.webm
$

BTW, Featherpad 1.2.0 on FreeBSD shows exactly the same behaviour. I can't install the program on my Ubuntu 19.04 'cuz it's too old and the repository has disappeared.

Someone who knows LXQT/QT ought to investigate...

rvp Now, I'm lost.

rvp In the "native" file chooser, the filenames are passed correctly when listing the directory, however:

Which native file-chooser? FeatherPad's or LXQt's? Could you be clearer, I would like to understand what is working correctly and what is not.

bsduck

rvp BTW, Featherpad 1.2.0 on FreeBSD shows exactly the same behaviour.

I don't have problems on FreeBSD with the same programs. Both file dialogs show the file and can open it.

pin EDIT 2: If you disable uim do you get the same behaviour.

Yes.

rvp

pin Which native file-chooser?

Whatever you get when Native file dialog is ticked--prolly LXQT.

pin I would like to understand what is working correctly and what is not.

The situation is unchanged: Only the LXQT (ticked) file dialog shows these UTF-8 encoded filenames in the directory listing, but, when you double-click on these files, you get a can't open ???.ext error. The other (unticked) file chooser doesn't list the files at all.

pcmanfm-qt doesn't have any problems as OP reported. I can open any of these files just fine by double-clicking 'em.

rvp Ok! I don't have LXQt, I'm talking standalone apps.

featherpad file-chooser doesn't see the file, with or without the ticked box.
pcmanfm-qt does see the file.
xpdf also sees the file when the .pdf extension is appended to the file name.

LXQt and featherpad-1.2.0 on Void Linux. All file-choosers see the file, no difference between the applications.
If this is not a general Qt issue I would like to understand what is causing it.

EDIT: https://github.com/tsujan/FeatherPad/discussions/670#discussioncomment-3400803

EDIT 2: If you disable uim do you get the same behaviour.

rvp

pin featherpad file-chooser doesn't see the file, with or without the ticked box.

Yes. (this would be the nice-looking Fusion QT5 chooser if you don't use the LXQT DE: see pic above)

pin pcmanfm-qt does see the file.

Yes, and opens the files too.

pin LXQt and featherpad-1.2.0 on Void Linux.

Can files be open by double-clicking?

rvp Can files be open by double-clicking?

Yes! At least in pcmanfm-qt. I need to go back a see if I can do it from the featherpad file-chooser.

EDIT: Again, if you disable uim do you get the same behaviour.

bsduck Yes.

Yes where? Does pcmanfm-qt stops seeing the file? Does LXQt stops seeing it?
As you have noticed, not all Qt-apps are affected and the issue is not a general Qt issue. Finding out why certain applications are affected while others are not, is the puzzle we need to solve.

rvp

bsduck I don't have problems on FreeBSD with the same programs. Both file dialogs show the file and can open it.

Bizarre. Featherpad behaves identically for me on both FreeBSD and NetBSD.

pin Can you please test this settings and at the same time disable uim on LXQt?

No change (and, didn't expect any) with:

$ env LANG=C LC_TIME=C LC_NUMERIC=C LC_MONETARY=C LC_MESSAGES=C featherpad

Native (ticked) chooser shows the file, but, tries to open caff?.txt instead of caffè.txt
Unticked chooser doesn't show the file at all.

Can you guys try this patch?

diff -urN FeatherPad-1.3.1.orig/featherpad/main.cpp FeatherPad-1.3.1/featherpad/main.cpp
--- FeatherPad-1.3.1.orig/featherpad/main.cpp	2022-07-31 14:37:30.000000000 +0000
+++ FeatherPad-1.3.1/featherpad/main.cpp	2022-08-17 23:02:38.864572116 +0000
@@ -43,9 +43,17 @@
 
 int main (int argc, char **argv)
 {
+    std::locale::global(std::locale(""));
+    QLocale::setDefault(QLocale::system());
+
     const QString name = "FeatherPad";
     const QString version = "1.3.1";
     const QString firstArg = QString::fromUtf8 (argv[1]);
+{
+    QTextStream out (stdout);
+    out << "File: " << firstArg << Qt::endl;
+}
+
     if (firstArg == "--help" || firstArg == "-h")
     {
         QTextStream out (stdout);

It should be using the user-configured locale (LANG= and LC_ALL=, etc) now, but, doesn't seem to for some C++ reason I can't fathom.

Run Featherpad as featherpad caffè.txt (or any other UTF-8 encoded filename) and see if the program prints caff?.txt or caffè.txt when LANG or LC_ALL is set to a UTF-8 locale (any will do: en_US.UTF-8, en_GB.UTF-8, even C.UTF-8).

If they are set to the C locale, then it should (obviously) always print caff?.txt.

However, even with the patch Featherpad always shows caff?.txt, when this code prints caffè.txt in a UTF-8 locale:

//
// c++ -fpic q.cpp $(pkg-config --cflags Qt5Core) $(pkg-config --libs Qt5Core)
//
#include <locale>
#include <QTextStream>
int main(int argc, char* argv[]) {
	if (argc != 2)
		return 1;
        // replace the C++ global locale as well as the C locale
        //  with the user-preferred locale
	std::locale::global(std::locale(""));
	const QString filename = QString::fromUtf8(argv[1]);
	QTextStream out (stdout);
	out << filename << Qt::endl;
	return 0;
}

C++ devs please have a look...

PS. I made a cmake project out of this just to test it out. PM me if you want the tarball.