If I wanted to create a string which is guaranteed not to represent a filename, I could put one of the following characters in it on Windows:
\ / : * ? | < >
e.g.
this-is-a-filename.png
?this-is-not.png
Is there any way to identify a string as 'not possibly a file' on Linux?
There are almost no restrictions - apart from '/' and '\0', you're allowed to use anything. However, some people think it's not a good idea to allow this much flexibility.
An empty string is the only truly invalid path name on Linux, which may work for you if you need only one invalid name. You could also use a string like "///foo", which would not be a canonical path name, although it could refer to a file ("/foo"). Another possibility would be something like "/dev/null/foo", since /dev/null has a POSIX-defined non-directory meaning. If you only need strings that could not refer to a regular file you could use "/" or ".", since those are always directories.
Technically it's not invalid but files with dash(-) at the beginning of their name will put you in a lot of troubles. It's because it has conflicts with command arguments.
I personally find that a lot of the time the problem is not Linux but the applications one is using on Linux.
Take for example Amarok. Recently I noticed that certain artists I had copied from my Windows machine where not appearing in the library. I check and confirmed that the files were there and then I noticed that certain characters in the folder names (Named for the artist) were represented with a weird-looking square rather than an actual character.
In a shell terminal the filenames look even stranger: /Music/Albums/Einst$'\374'rzende\ Neubauten is an example of how strange.
While these files were definitely there, Amarok could not see them for some reason. I was able to use some shell trickery to rename them to sane versions which I could then re-name with ASCII-only characters using Musicbrainz Picard. Unfortunately, Picard was also unable to open the files until I renamed them, hence the need for a shell script.
Overall this a a tricky area and it seems to get very thorny if you are trying to synchronise a music collection between Windows and Linux wherein certain folder or file names contain funky characters.
The safest thing to do is stick to ASCII-only filenames.
Related
As soon as I try to access a folder/file containing an emoji in its name from my Lua 5.2 script, for example like this:
os.execute('start "" "' .. path .. "\\scripts\\menu\\📄 My Scripts" .. '"')
The Windows' Command Prompt simply refuses to open it with the following error message:
I'm aware Windows' Command Prompt doesn't support emojis and therefore is not possible to make it work just like that, but my doubt is if won't exist some workaround or whatever I can do to ensure any Windows/Unix user is going to able to get the folder open by my Lua script without any problem.
I have tried i.e. things like use the codes instead (1246 and U+1F4F0 in this page facing up case) without success. Couldn't I for example simply use some kind of "wildcard" instead? I mean, knowing it's always going to be the very first character in the name. Or, well, any other ideas will be welcomed, cause nothing I'm trying really seems to work...
Of course if it's going to represent any problem I'll simply refuse to use them, but it came in handy for some "first sight" folder distinction and, if possible, I'd like to can count this little visual resource 🙄
This is a Problem about how the string is constructed.
I found only one solution with [[command "path"]] (on Windows 11 and Lua 5.3)...
os.execute([[start ]] .. path .. [["\scripts\menu\📄 My Scripts"]])
-- My Testpath is/was: os.execute([[dir "%localappdata%\nvim\📄 Lua"]])
...the long string ([[]]) will not be interpreted (coercionated) by Lua.
That also have the side effect that you can use single backslashs with that kind of string.
Environment variable expansion (e.g. Windows: %localappdata%) only works inside doublequotes.
Single quotes instead ([[command '%localappdate%\path\']]) will not work (expanded).
os.execute accepts only ANSI-encoded strings (win-1252 in European Windows), but it is unable to encode an emoji.
Hint: you can create .bat-file to do the task for you and invoke it from Lua with os.execute.
I have pattern to search. Say "*.txt".
Now I have some files I do not want to list there. I believe they do not match this pattern.
But on windows, they do.
I know tilde character is used to make short form of legacy 8.3 filename. That is LongFilename.json might be LONGFI~1.JSO. But I did not know they are handled somehow on windows in file search patterns. They are. I cannot find any documentation about what they mean and how to match files my way.
My problem is NOT with short forms. Or I think it is not directly related to it.
I have file "A.txt". Now I wanted temporary file and used "A.txt~". It is unix backup files that is not usually visible. But on windows, they should not have special meaning by itself. Only for my application.
Now I want list of "*.txt" files. Command
dir *.txt
returns to my surprise also all .txt~ files in the same directory. And I do not want them. I use FindFileFirst from Win32 API. I did not find anything about tilde character in documentation. FindFileFirst(".txt", handle) returns also files "A.txt~". Can I use some flag to exclude them? I know I can make special condition, like I have for "." and "..". How does ~ operator work? A.txt~1 is also matched. Is everything after tilde ignored? Is that feature or bug?
I am testing that on Windows 7 Professional, 64 edition, if that changes anything.
FindFirstFile also includes short names for legacy reasons so the pattern *.txt will include anything with an 8.3 representation ending in *.txt which includes *.txtANYTHING , not just the ~ character (see dir /xfor what's being matched against).
You will need to filter in your FindNext enumeration.
If you are searching for .txt files for example, you can use "kind:text" option in windows to exclude txt~ and similar files since they are not a recognized type anymore.
That's something that works on regular windows search. I'm not 100% sure about the API, but it should also be there.
Does anyone know a pure Win32 solution for renaming a file and only changing its capitalization, that does not involve intermediate renaming to a different name or special privileges (e.g. backup, restore).
Since the Win32 subsystem generally regards two file names differing only in capitalization as the same, I haven't been able to find any solution to the problem.
A test program I made with the MoveFile API seems to work. So does the rename command in cmd.exe. What have you tried, and what error are you getting?
This isn't relevant, but further testing shows that renaming a long filename in this way works but will change the short filename (alternating between ~1 and ~2 for example), incidentally.
Just use the normal MoveFile API. That call probably just turns into ZwSetInformationFile(..., FileRenameInformation,...) The docs for FILE_RENAME_INFORMATION states that you need DELETE access and the file can't be locked etc, but those restrictions will probably apply to other solutions also.
I do not believe there is a way to expose two files with identical names that differ only in spelling to the Win32 subsystem. Even if some how you were able to create these files, the most likely result would be that only one file would be accessible - defeating the purpose of staying soley in Win32.
If you want to go into the Native layer, you can create a file with NtCreateFile and InitializeObjectAttributes w/o OBJ_CASE_INSENSITIVE or you can pad the end with extra spaces (if you pad with extra spaces, the file will not be accessible from Win32 dos paths). See here: http://www.osronline.com/ddkx/kmarch/k109_66uq.htm . I'm pretty sure you were already aware but I included it incase you did not know.
So long as your file is not immediately needed by another program, you can use my solution.
When you rename the file, capitalize, and delete the last letter. Then rename again and return the letter.
:)
I'm working on maintenance of an application that transfers a file to another system and uses a structured filename to include meta data including a language code. The current app uses a two character language code and a dash/hyphen for a delimiter.
Ex. Canada-EN-ProdName-ProdCode.txt
I'm converting it to use IETF language code and so the dash delimiter won't do and need a replacement. I'm trying to determine a delimiter to avoid future errors and am considering the tilde ~.
Ex. Canada~en-GB~ProdName~ProdCode.txt
This will be use only on Windows Sever 2003 + systems. I certainly didn't come up with this system of parsing a filename to get meta data. Unfortunately, I can't include this in the file itself and the destination system is expecting the language code to be in IETF format with the dash.
Any thoughts on potential issues with using the tilde in the filename, or perhaps a better character to use? I'm just looking for a second opinion in case I'm overlooking a possible failure. I believe windows will use the tilde when shortening a long filename to 8.3 format, but I don't see that as an issue here as the OSs can handle lang filenames.
The tilde is probably fine, but what's wrong with the good old underscore _ ? It has no special meaning on either windows or unix, and makes names that are relatively easy to read. If there are no other special considerations, I would avoid the tilde solely out of paranoia, since windows does use it as a special character sometimes, as you mentioned.
For anyone readiong this question I would strongly recommend anything but the tilde in the file name or at least be careful in testing for any speed problems with any .NET path work where one exists.
I used this as a file name delimiter some time ago. I couldn't understand why simply getting a list of files from the folders was taking so long. It was a number of years later (having written a lot of speed up code that had marginal advantage) that I discovered there is a problem with the (DirectoryInfo(path).name in .NET at least) where simple existience of the tilde was forcing underlying code to through a lot of hoops.
The slow down was substantial (it was over a network so I had thought it was bandwidth/Network issues for a fair while)
I understand this is a legacy overhang for when alternative short versions of filenames could be used for Windows files.
I am now stuck with the tilde in these file names but, given that the problem lay in some of the .NET path functions (I don't actually know if it still does), I could work around it by spotting a tilde and creating my own answers when it existed rather than passing it through.
If in any doubt just run speed tests with and without the tilde in filenames for say just 500-1,000 files.
PLEASE don't tell me why you think its a bad idea. Just tell me if its a workable idea.
I want to create files in a folder with names like the following:
asdf#qwerty.com.eml
abc+def#asdf.net.eml
abc_def#sasdf.at.eml
Is there some fundamental incompatibility in the characters allowed in email addresses and those allowed by a unix system?
I will be having a bash script reading the file names, subtracting the ".eml" ending, converting it into the "correct" usable format and sending an email to the address.
A simple test showed that it saved the above as files called
asdf\#qwerty.com.eml
abc+def\#asdf.net.eml
abc_def\#sasdf.at.eml
The only characters not allowed in a *nix filename are \0 and /, neither of which is allowed in an email address anyways. How your shell may handle symbols is another matter.
There are no characters disallowed in UNIX file names except / (directory separator) and ASCII 0 (string terminator), so there is no problem at a fundamental level.
Handling those file names in shell scripts is a different matter; it requires at least quoting every variable reference as "$FILENAME", and forgetting even one quotatino will create a very rare, insidious bug. (Also, many other utilities will fail on strange characters such as | or newline unless you consistently use the -0 option.)
So yes, technically your bad idea is workable :-)
Short answer:
przemek#linux-634b:~/tmp/email> touch john.smith#example.com
przemek#linux-634b:~/tmp/email> ls
john.smith#example.com
Works perfectly;)
Long answer:
It depends on filesystem you're using. See Wikipedia entry which lists allowed characters in file names. Most UNIX file systems support all characters that can be included in e-mail addresses. Non-UNIX filesystems, such as FAT, however, may cause problems.
Note that your problems may come from improper escaping. Check how are you creating your files.
What was your "simple test"?
Typing abc and hitting tab?
The bash autocompletion will add a \ before every special character.
But this does not mean, it is stored with a \ in its name.
Use ls to see the true name.
There is no problem with such file names on systems which treat file names as blobs and allow all byte sequences, i.e. Linux. But they are not portable to systems which treat file names as Unicode strings and disallow certain characters (Windows) or transform file names (Mac OS X, canonical decomposition).