I am trying to understand flags and modes of file descriptors.
The man page for
fcntl - manipulate file descriptor
int fcntl(int fd, int cmd);
states:
File descriptor flags
The following commands manipulate the flags associated with a file
descriptor. Currently, only one such flag is defined: FD_CLOEXEC,...
File status flags
Each open file description has certain associated status flags,
initialized by open(2)...
The file status flags and their semantics are described in open(2).
Given that fcntl refers entirely to file descriptors (no dealing with streams), I guess the second title should be "File descriptor status flags".
So now we have for a FD "flags" and "status flags".
This man page also mentions that when cmd=F_GETFL, the return value of fcntl is "the file access mode and the file status flags".
So now we have also a file access mode.
Now in the man page for open
there are flags and modes, as if they were two different items.
There is even a prototype that makes explicit the difference
int open(const char *pathname, int flags, mode_t mode);
So now we have, for each file descriptor, "flags", "status flags", "file access modes", and "modes" (I would identify the latter two as the same).
To begin with,
1. I don't know the difference between these three categories.
Traversing both quoted man pages, I collected a list of "entities" (below, in order of appearance, some are repeated).
2. I don't know which category each belongs to.
FD_CLOEXEC, O_RDONLY, O_WRONLY, O_RDWR, O_CREAT, O_EXCL, O_NOCTTY, O_TRUNC, O_APPEND, O_ASYNC, O_DIRECT, O_NOATIME, O_NONBLOCK, O_DSYNC, O_SYNC, O_CLOEXEC
O_CREAT, O_DIRECTORY, O_EXCL, O_NOCTTY, O_NOFOLLOW, O_TMPFILE, O_TRUNC, O_LARGEFILE, O_NDELAY, O_PATH
I couldn't find a simple list telling "X, Y, Z are flags, W, V are modes, etc."
Perhaps they are terms that are used interchangeably, or the mode is a subset of the flags, or...
Related:
Difference between "file pointer", "stream", "file descriptor" and ... "file"? (answers there may be a guide in the present OP, even if not the same).
How to make sense of O_RDONLY = 0?
Difference between "file pointer", "stream", "file descriptor" and ... "file"?
How to get the mode of a file descriptor?
https://www.gnu.org/software/libc/manual/html_node/Access-Modes.html
https://www.gnu.org/software/libc/manual/html_node/File-Status-Flags.html#File-Status-Flags
File descriptors can be duplicated. For example, when a process forks, it gets its own set of FDs that the parent doesn't affect, and the dup syscall can be used to explicitly duplicate individual FDs.
When file descriptors get duplicated, every descriptor has its own set of file descriptor flags, but they'll all share the same file status flags. For example, consider this code:
int fdA = open('/tmp/somefile', O_WRONLY);
int fdB = dup(fdA);
fcntl(fdA, F_SETFD, FD_CLOEXEC);
fcntl(fdA, F_SETFL, O_APPEND);
After running it, fdA will be close-on-exec and in append mode, and fdB will be in append mode but not close-on-exec. This is because close-on-exec is a file descriptor flag and append mode is a file status flag.
The file access mode and file creation flags are passed along with the file status flags when they're supported.
The third parameter to open, also confusingly called mode, is unrelated to everything else discussed so far. If the file is created by the call to open, then that mode is used as the permissions for the new file. Otherwise, it has no effect.
FD_CLOEXEC - file descriptor flag
O_RDONLY - file access mode
O_WRONLY - file access mode
O_RDWR - file access mode
O_CLOEXEC - file creation flag
O_CREAT - file creation flag
O_DIRECTORY - file creation flag
O_EXCL - file creation flag
O_NOCTTY - file creation flag
O_NOFOLLOW - file creation flag
O_TMPFILE - file creation flag
O_TRUNC - file creation flag
The rest of the flags you listed are file status flags.
And one final note: O_CLOEXEC is only relevant for a new FD. For existing FDs, you'll only ever use FD_CLOEXEC.
I will summarize the description by Joseph Sible-Reinstate Monica and add a few remarks on possibly confusing wording across man pages, likely the cause of the OP.
As per headings in http://man7.org/linux/man-pages/man2/fcntl.2.html (as cited in the OP) Flags = File descriptor flags + File status flags.
Remark 1: this usage of File status flags is not consistent with the rest of available information, so it should rather be called something like
Flags = File descriptor flags + Non-FD flags.
The distinction between these two groups of flags is given by Joseph Sible-Reinstate Monica.
As per http://man7.org/linux/man-pages/man2/open.2.html,
Non-FD Flags = Access mode + File creation flags + File status flags
Note that:
The man page does not use the name Non-FD Flags. It simply calls this flags, as the name of the argument in the prototypes listed. But this should not be taken as if, conceptually, these flags encompass all flags since File descriptor flags are not included.
"The distinction between these two groups of flags is that the file creation flags affect the semantics of the open operation itself, while the file status flags affect the semantics of subsequent I/O operations." [ref]
This is the most common usage of File status flags.
This is the basic classification of "entities".
Remark 2: I use quotes since the general use of flags is quite misleading.
Access mode are not flags in the usual sense, and this is clarified in
How to make sense of O_RDONLY = 0?.
Remark 3:
GNU uses a different naming, adding to the confusion.
The translation POSIX.1 <-> GNU is shown below.
Usage of File Status Flags in GNU may be especially confusing.
POSIX.1 GNU
Non-FD Flags* File Status Flags
Access modes Access mode
File creation flags Open-time Flags
File status flags Operating Modes
As for the listings enumerating each category, they are given by Joseph Sible-Reinstate Monica.
GNU also has its own Access modes, File creation flags (Open-time Flags) and File status flags (Operating Modes).
Related
In /usr/lib/modules/$(uname -r)/modules.devname, the first line is a comment:
# Device nodes to trigger on-demand module loading.
Assuming what it means is that the first time a device file under /dev is accessed, a module which populates that file will be automatically loaded.
But I don't see the code that does module loading when a file lookup failed, in /drivers/base/devtmpfs.c or /mm/shmem.c(tmpfs). Where does that logic live then?
The modules.devname file has nothing to do with module auto-loading. It contains information that can be used during system initialization to create files in the /dev directory. The file is read by the kmod static-nodes command. By default, kmod static-nodes produces human-readable output, but during system initialization it is run as kmod static-nodes --format=tmpfiles to generate output in a more machine-parseable form. Each line contains information that can be used to create a single directory or a single special file (see the tmpfiles.d man page for details of the format). It does not contain the module name.
On systems using Systemd init, the kmod command is run from the kmod-static-nodes.service service. The output file in tmpfiles.d format is placed in "/run/tmpfiles.d/static-nodes.conf", which will be read later by the systemd-tmpfiles --prefix=/dev --create --boot command run from the systemd-tmpfiles-setup-dev.service service to create the actual files in "/dev".
On systems using Sysv init, the kmod command may be run by the /etc/init.d/udev init script (on Debian type systems) or from somewhere else. The same init script creates the actual files in "/dev" based on the output from kmod.
When a character special file for an unregistered character device number is being opened, the kernel will request the module with alias char-major-MAJOR-MINOR or char-major-MAJOR where MAJOR and MINOR are the major and minor device numbers of the special file. (See base_probe() in "fs/char_dev.c".) If the kernel is configured with CONFIG_BLOCK_LEGACY_AUTOLOAD=y, there is similar functionality when opening block special files for unregistered block device numbers, the kernel will request the module with alias block-major-MAJOR-MINOR or block-major-MAJOR. (See blk_request_module() in "block/genhd.c" and blkdev_get_no_open() in "block/bdev.c".)
The source code for a module uses the MODULE_ALIAS_CHARDEV(), MODULE_ALIAS_CHARDEV_MAJOR(), MODULE_ALIAS_BLOCKDEV(), or MODULE_ALIAS_BLOCKDEV_MAJOR() macros (which wrap the MODULE_ALIAS() macro) to put these aliases into the module's .modinfo section where the depmod command can find them.
We can specify both of flag and perm at os.OpenFile.
They have really similar options, O_APPEND and ModeAppend. What's the difference between them?
f, _ := os.OpenFile("access.log", os.O_APPEND|os.O_CREATE, os.ModeAppend|0644)
The flag specify the flags used on the system call to open the file while perm sets the File mode on the file. The file mode includes the permissions and type of file eg. symlink, directory, etc...
os.O_APPEND tells the underlying OS that all the write calls you do on that file handler should always append to the file so you don't need to set the offset to write on the correct part of the file.
ModeAppend sets the file mode to be append. This means that the this file can only be modified by appending to it, not by rewriting the file contents. The specifics of this depends on the OS and file system you are using. I believe Plan 9, implements it by ignoring the offset on any write call to the file and always appending to it, while in linux it means that the file can only be open for writing in append mode. I think that on most linux distros you need to be root to set the file mode to append.
In 99.99% of cases you just want to use perm to set the file permissions rwx. In your case if you want to open a file and append to it you should use:
// os.O_WRONLY tells the computer you are only going to writo to the file, not read
// os.O_CREATE tells the computer to create the file if it doesn't exist
// os.O_APPEND tells the computer to append to the end of the file instead of overwritting or truncating it
f, err := os.OpenFile("access.log", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0644)
You might have only ignore the return error on os.OpenFile to put the example online, but you should get used to always checking for errors. You have no idea how many users ran into trouble when starting with go because they ignore the errors. Sometimes is something stupid and easy to fix like a typo, but if you ignore the error you don't know what the issue is.
You can read more about the append file mode here.
Oracle's sqlldr defaults to a .dat extension. That I want to override. I don't like to rename the file. When googled get to know few answers to use . like data='fileName.' which is not working. Share your ideas, please.
Error message is fileName.dat is not found.
Sqlloder has default extension for all input files data,log,control...
data= .dat
log= .log
control = .ctl
bad =.bad
PARFILE = .par
But you have to pass filename without apostrophe and dot
sqlloder pass/user#db control=control data=data
sqloader will add extension. control.ctl data.dat
Nevertheless i do not understand why you do not want to specify extension?
You can't, at least in Unix/Linux environments. In Windows you can use the trailing period trick, specifying either INFILE 'filename.' in the control file or DATA=filename. on the command line. WIndows file name handling allows that; you can for instance do DIR filename. at a command prompt and it will list the file with no extension (as will DIR filename). But you can't do that with *nix, from a shell prompt or anywhere else.
You said you don't want to copy or rename the file. Temporarily renaming it might be the simplest solution, but as you may have a reason not to do that even briefly you could instead create a hard or soft link to the file which does have an extension, and use that link as the target instead. You could wrap that in a shell script that takes the file name argument:
# set variable from correct positional parameter; if you pass in the control
# file name or other options, this might not be $1 so adjust as needed
# if the tmeproary file won't be int he same directory, need to be full path
filename=$1
# optionally check file exists, is readable, etc. but overkill for demo
# can also check temporary file does not already exist - stop or remove
# create soft link somewhere it won't impact any other processes
ln -s ${filename} /tmp/${filename##*/}.dat
# run SQL*Loader with soft link as target
sqlldr user/password#db control=file.ctl data=/tmp/${filename##*/}.dat
# clean up
rm -f /tmp/${filename##*/}.dat
You can then call that as:
./scriptfile.sh /path/to/filename
If you can create the link in the same directory then you only need to pass the file, but if it's somewhere else - which may be necessary depending on why renaming isn't an option, and desirable either way - then you need to pass the full path of the data file so the link works. (If the temporary file will be int he same filesystem you could use a hard link, and you wouldn't have to pass the full path then either, but it's still cleaner to do so).
As you haven't shown your current command line options you may have to adjust that to take into account anything else you currently specify there rather than in the control file, particularly which positional argument is actually the data file path.
I have the same issue. I get a monthly download of reference data used in medical application and the 485 downloaded files don't have file extensions (#2gb). Unless I can load without file extensions I have to copy the files with .dat and load from there.
I wonder if there a standard way to include a postscript file into another.
For example, say I have got one file of data generated by a 3rd party program:
%!PS
\mydata [ 1 2 3 4 5 6
(...)
1098098
1098099
] def
and I would like to include it into a main PS document
%PS
\processData
{
mydata { (..) } foreach
}
(...)
(data.ps) include %<=== ???
Thanks
The operator you want is run.
string run -
execute contents of named file
Unfortunately, run is not allowed if the interpreter has the SAFER option set.
Edit: Bill Casselman, author of *Mathematical Illustrations" has a Perl script called psinc you can use to "preprocess" yor postscript files, inlining all (...) run files.
The standard way to include PostScript is to make the code to be included an EPS (Encapsulated PostScript) file. There are rules on how encapsulated PostScript must be created, and how to include it. See Adobe Tech Note 5002 'Encapsulated PostScript File Format Specification'
Simply executing 'run' on a PostScript file may well work, but it might also cause problems. Many PostScript files (especially those produced by 3rd parties) will include procedure definitions which may clash with your own names, and also the included program may leave the interpreter in a state different from the one it was in when the included file was executed. At the very least you should execute a save/restore pair around the code included via 'run'.
I would suggest meta-solution: use C preprocessor or M4 preprocessor. They are powerful tools and their power may find use in other ways as well, not only file inclusion. Though this was not asked, but use of Makefile will be wise to automate whole workflow. By using a preprocessor and Makefile in combination you can elegantly automate complex inclusions processing and beyond.
C Preprocessor
Including a file:
#include "other.ps"
Commandline for preprocessing:
cpp -P main.pps main.ps
M4 Preprocessor
Including a file:
include(other.ps)
Commandline for preprocessing:
m4 main.pps > main.ps
Build Rules are documented in the Xcode Build System Guide
They are well adapted to the common case where one input file is transformed into a fixed number (usually one) of output files.
The output files must be described in the "Output Files" area of the build rule definition; one line per output file. Typically the output files have the same name as the input file but have different extensions.
In my case, one single input file is transformed into a variable number of files with the same extensions. The number and the names of the output files depend on the content of the input file and are not known in advance.
The output files will have to be further processed later on (they are in this case C files to be compiled).
How can I set up a build rule for such a case?
Any suggestions welcome.
(I asked the same question on the Apple developer forum, but I figured it'd be a good idea to ask here too).
I dealt with this by, instead of generating multiple C files, just concatenating them all together into one file (e.g. "AUTOGENERATED.c"), and specifying that as the output file.
So long as your output files don't contain anything that will conflict (static functions with the same name, conflicting #defines etc.) this works well.
See this article on Cocoa With Love:
http://cocoawithlove.com/2010/02/custom-build-rules-generated-tables-and.html
This has an example of generating custom C code and using that as input to the normal build process. He's using ${} variable syntax in the output
The best way I found to add any number of files to my xcode project (and make some processing) is to write a little php script. The script can simply copy files into the bundle. The tricky part is the integration with xcode. It took me some time to find a clean way. (You can use the script language you like with this method).
First, use "Add Run Script" instead of "Add Copy File"
Shell parameter:
/bin/sh
Command parameter:
${SRCROOT}/your_script.php -s ${SRCROOT} -o ${CONFIGURATION_BUILD_DIR}/${UNLOCALIZED_RESOURCES_FOLDER_PATH}
exit $?
(screenshot in xcode)
${SRCROOT} is your project directory.
${CONFIGURATION(...) is the bundle directory. Exactly what you need :)
This way, your script return code can stop xcode build (use die(0) for success and die(1) for failures) and the output of script will be visible in xcode's build log.
Your script will look like that: (don't forget chmod +x on it)
#!/usr/bin/php
<?php
error_reporting(E_ALL);
$options = getopt("s:o:");
$src_dir = $options["s"]."/";
$output_dir = $options["o"]."/";
// process_files (...)
die(0);
?>
BONUS: here my 'add_file' function.
Note the special treatment for PNG (use apple's png compression)
Note the filemtime/touch usage to prevent copy files each times.
l
define("COPY_PNG", "/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/copypng -compress");
function add_file_to_bundle($output_dir, $filepath) {
// split path
$path_info = pathinfo($filepath);
$output_filepath = $output_dir.$path_info['basename'];
// get file's dates of input and output
$input_date = filemtime($filepath);
$output_date = #filemtime($output_filepath);
if ($input_date === FALSE) { echo "can't get input file's modification date"; die(1); }
// skip unchanged files
if ($output_date === $input_date) {
//message("skip ".$path_info['basename']);
return 0;
}
// special copy for png with apple's png compression tool
if (strcasecmp($path_info['extension'], "png") == 0) {
//message($path_info['basename']." is a png");
passthru(COPY_PNG." ".escapeshellarg($filepath)." ".escapeshellarg($output_filepath), $return_var);
if ($return_var != 0) die($return_var);
}
// classic copy
else {
//message("copy ".$path_info['basename']);
passthru("cp ".escapeshellarg($filepath)." ".escapeshellarg($output_filepath), $return_var);
if ($return_var != 0) die($return_var);
}
// important: set output file date with input file date
touch($output_filepath, $input_date, $input_date);
return 1;
}