set pipefail for commands run by tup - bash

In a large amount of Tupfiles I use extensive pipelining, e.g.
: input |> < %f command1 | command2 > %o |> output
The problem with this is that Tup calls system which executes these :-rules in sh, which doesn't support set -o pipefail. As a result, if only command1 fails, tup will still mark this as a success because it had a 0 exit code. This is highly problematic.
I know of two solutions to this, neither of which is ideal.
a. I could abandon pipelining and instead do:
: input |> < %f command1 > %o |> intermediate
: intermediate |> < %f command2 > %o |> output
This will work, but would require rewriting a bunch of rules tediously, and more importantly will use significantly more disk space and disk writes every time there is an update.
b) I can wrap every command in bash like:
: input |> bash -c 'set -o pipefail && < %f command1 | command2 > %o' |> output
This seems slightly better as it involves fewer rewrites, and avoids the io, but is still very cumbersome. It also requires escaping any ' in my :-rules.
Ideally there would be Tup configs that could just specify what shell / interpreter to use to read :-rules. Ideally, there would also be a configuration for a common prefix, so all scripts could be run with set -o pipefail && or anything else I want. As far as I know this is not immediately possible. A wrapper around system would need to be written whenever tup invokes a rule. However, maybe I've missed some aspect of Tup that would allow something more elegant than the two solutions proposed.
Edit:
While the call to system did allow me to "inject" pipefail into calls to system. I miss-stated the fact that programs are run using system. With some help from the mailing list it turns out that they are actually run using execle. Below is the code I used to do the interposition in case anyone wants to accomplish the same thing.
Solution
#include <stdlib.h>
#include <string.h>
#include <stdarg.h>
#include <unistd.h>
int execle(const char* path, const char* arg0, ...) {
/* We're going to interpose this function, modify the arguments if we need
* to, and then convert it into a call to execve. Due to a weirdness in the
* consts of the api, we need to discard a const qualifier on the
* characters in the arguments. The call is `int execve(const char*
* filename, char* const argv[], char* const envp[]);` but it should
* probably be `int execve(const char* filename, const char* const argv[],
* char* const envp[]);` at the very least, e.g. arguments shouldn't be
* modified. These aren't actually modified by the call, so in order to
* avoid the inefficiency of copying the strings into memory we don't need,
* we just do this unsafely and compile with `-Wno-discarded-qualifiers`.
* */
// Count the number of variable arguments for malloc
unsigned int num_args;
va_list ap;
va_start(ap, arg0);
if (arg0) {
num_args = 1;
while(va_arg(ap, const char*)) {
num_args++;
}
} else {
num_args = 0;
}
char* const* env = va_arg(ap, char* const*); // Also grab env
va_end(ap);
// Test for specific tup execle call
va_start(ap, arg0);
int intercept = num_args == 4
&& strcmp(path, "/bin/sh") == 0
&& strcmp(arg0, "/bin/sh") == 0
&& strcmp(va_arg(ap, const char*), "-e") == 0
&& strcmp(va_arg(ap, const char*), "-c") == 0;
va_end(ap);
// Switch on whether to intercept the call, or pass it on
/*const*/ char** args;
if (intercept) { // We want to switch to bash with pipefail enabled
args = malloc(7 * sizeof(args));
path = "/bin/bash";
args[0] = "/bin/bash";
args[1] = "-e";
args[2] = "-o";
args[3] = "pipefail";
args[4] = "-c";
va_start(ap, arg0);
va_arg(ap, const char*);
va_arg(ap, const char*);
args[5] = va_arg(ap, const char*); // command
va_end(ap);
args[6] = NULL;
} else { // Just copy args into a null terminated array for execve
args = malloc((num_args + 1) * sizeof(*args));
char** ref = args;
if (arg0) {
*ref++ = arg0;
const char* arg;
va_start(ap, arg0);
while ((arg = va_arg(ap, const char*))) {
*ref++ = arg;
}
va_end(ap);
}
*ref = NULL;
}
int error_code = execve(path, args, env);
free(args);
return error_code;
}

You could implement your own system as
switch(pid = fork()) {
case 0:
// Modify command to prepend "set -o pipefail &&" to it.
execl("/bin/bash", "bash", "-c", command, (char *) 0);
case -1: // handle fork error
default:
waitpid(pid, ...);
}
and LD_PRELOAD that system implementation into your tup process.
If you don't feel like doing low-level process management, you can interpose system to just wrap the command in bash -c "set -o pipefail && " and escape quotes, then invoke the original system. See this article on library interposition.

Related

Passing a temporary stream object to a lambda function as part of an extraction expression

I have a function which needs to parse some arguments and several if clauses inside it need to perform similar actions. In order to reduce typing and help keep the code readable, I thought I'd use a lambda to encapsulate the recurring actions, but I'm having trouble finding sufficient info to determine whether I'm mistakenly invoking undefined behavior or what I need to do to actualize my approach.
Below is a simplified code snippet of what I have currently:
int foo(int argc, char* argv[])
{
Using ss = std::istringstream;
auto sf = [&](ss&& stream) -> ss& {
stream.exceptions(ss::failbit);
return stream;
};
int retVal = 0;
bool valA = false;
bool valB = false;
try
{
for(int i=1; i < argc; i++)
{
std::string arg( argv[i] );
if( !valA )
{
valA = true;
sf( ss(arg) ) >> myInt;
}
else
if( !valB )
{
valB = true;
sf( ss(arg) ) >> std::hex >> myOtherInt;
}
}
}
catch( std::exception& err )
{
retVal = -1;
std::cerr << err.what() << std::endl;
}
return retVal;
}
First, based on what I've read, I don't think that specifying the lambda argument as an rvalue reference (ss&&) is doing quite what I want it to do, however, trying to compile with it declared as a normal reference (ss&) failed with the error cannot bind non-const lvalue reference of type 'ss&'. Changing ss& to ss&& got rid of the error and did not produce any warnings, but I'm not convinced that I'm using that construct correctly.
I've tried reading up on the various definitions for each, but the wording is a bit confusing.
I guess ultimately my questions are:
Can I expect the lifetime of my temporary ss(arg) object to extend through the entire extraction expression?
What is the correct way to define a lambda such that I can use the lambda in the way I demonstrate above, assuming that such a thing is actually possible?

Changing tab-completion for read builtin in bash

The current tab-completion while "read -e" is active in bash seems to be only matching filenames:
read -e
[[TabTab]]
abc.txt bcd.txt cde.txt
I want the completion to be a set of strings defined by me, while file/dir/hostname-completion etc. should be deactivated for the duration of "read -e".
Outside of a script
complete -W 'string1 string2 string3' -E
works well, but i cant get this kind of completion to work inside a script while using "read -e".
Although it seems like a reasonable request, I don't believe that is possible.
The existing implementation of the read builtin sets the readline completion environment to a fairly basic configuration before calling readline to handle -e input.
You can see the code in builtins/read.def, in the edit_line function: it sets rl_attempted_completion_function to NULL for the duration of the call to readline. readline has several completion overrides, so it's not 100% obvious that this resets the entire completion environment, but as far as I know this is the function which is used to implement programmable completion as per the complete command.
With some work, you could probably modify the definition of the read command to allow a specific completion function instead of or in addition to the readline standard filename completion function. That would require a non-trivial understanding of bash internals, but it would be a reasonable project if you wanted to gain familiarity with those internals.
As a simpler but less efficient alternative, you could write your own little utility which just accepts one line of keyboard input with readline and echoes it to stdout. Then invoke read redirecting its stdin to your utility:
read -r < <(my_reader string1 string2 string3)
(That assumes that my_reader uses its command-line arguments to construct the potential completion list for the readline library. You'd probably want the option to present a prompt as well.)
The readline documentation includes an example of an application which does simple custom completion; once you translate it from the K&R function prototype syntax, it might be pretty easy to adapt to your needs.
Edit: After I looked at that example again, I thought it had a lot of unnecessary details, so I wrote the following example with fewer unnecessary details. I might upload it to github, but for now it's here even though it's nearly 100 lines:
#include <stdbool.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <readline/readline.h>
static void version(const char* progname) {
fprintf(stderr, "%s 0.1\n", progname);
}
static void usage(const char* progname) {
fprintf(stderr, "Usage: %s [-fhv] [-p PROMPT] [-n PROGNAME] [COMPLETION...]\n", progname);
fprintf(stderr,
"Reads one line using readline, and prints it to stdout.\n"
"Returns success if a line was read.\n"
" -p PROMPT Output PROMPT before requesting input.\n"
" -n PROGNAME Set application name to PROGNAME for readline config file\n"
" (Default: %s).\n"
" -f Use filename completion as well as specified completions.\n"
" -h Print this help text and exit.\n"
" -v Print version number and exit.\n"
" COMPLETION word to add to the list of possible completions.\n",
progname);
}
/* Readline really likes globals, so none of its hooks take a context parameter. */
static char** completions = NULL;
static char* generate_next_completion(const char* text, int state) {
static int index = 0;
if (state == 0) index = 0; /* reset index if we're starting */
size_t textlen = strlen(text);
while (completions[index++])
if (strncmp(completions[index - 1], text, textlen) == 0)
return strdup(completions[index - 1]);
return NULL;
}
/* We use this if we will fall back to filename completion */
static char** generate_completions(const char* text, int start, int end) {
return rl_completion_matches(text, generate_next_completion);
}
int main (int argc, char **argv) {
const char* prompt = "";
const char* progname = strrchr(argv[0], '/');
progname = progname ? progname + 1 : argv[0];
rl_readline_name = progname;
bool use_file_completion = false;
for (;;) {
int opt = getopt(argc, argv, "+fp:n:hv");
switch (opt) {
case -1: break;
case 'f': use_file_completion = true; continue;
case 'p': prompt = optarg; continue;
case 'n': rl_readline_name = optarg; continue;
case 'h': usage(progname); return 0;
case 'v': version(progname); return 0;
default: usage(progname); return 2;
}
break;
}
/* The default is stdout, which would interfere with capturing output. */
rl_outstream = stderr;
completions = argv + optind;
rl_completion_entry_function = rl_filename_completion_function;
if (*completions) {
if (use_file_completion)
rl_attempted_completion_function = generate_completions;
else
rl_completion_entry_function = generate_next_completion;
} else {
/* No specified strings */
if (!use_file_completion)
rl_inhibit_completion = true;
}
char* line = readline(prompt);
if (line) {
puts(line);
free(line);
return 0;
} else {
fputc('\n', rl_outstream);
return 1;
}
}

WinAPI C++ client detect write on anonymous pipe before reading

I am writing a C++ (Windows) client console application which reads from an anonymous pipe on STDIN. I would like to be able to use my program as follows:
echo input text here | my_app.exe
and do something in the app with the text that is piped in
OR
my_app.exe
and then use some default text inside of the app instead of the input from the pipe.
I currently have code that successfully reads from the pipe on STDIN given the first situation:
#include <Windows.h>
#include <iostream>
#include <string>
#define BUFSIZE 4096
int main(int argc, const char *argv[]) {
char char_buffer[BUFSIZE];
DWORD bytes_read;
HANDLE stdin_handle;
BOOL continue_reading;
unsigned int required_size;
bool read_successful = true;
stdin_handle = GetStdHandle(STD_INPUT_HANDLE);
if (stdin_handle == INVALID_HANDLE_VALUE) {
std::cout << "Error: invalid handle value!\n\n";
} else {
continue_reading = true;
while (continue_reading) {
continue_reading = ReadFile(stdin_handle, char_buffer, BUFSIZE,
&bytes_read, NULL);
if (continue_reading) {
if (bytes_read != 0) {
// Output what we have read so far
for (unsigned int i = 0; i < bytes_read; i++) {
std::cout << char_buffer[i];
}
} else {
continue_reading = false;
}
}
}
}
return 0;
}
I know that my only option with anonymous pipes is to do a blocking read with ReadFile. If I understand correctly, in regard to how I am invoking it, ReadFile will continue to read from the buffer on STDIN until it detects an end of write operation on the other end of the pipe (perhapse reads some sort of "end of write" token??). I would like to know if there is some sort of "beginning write" token that will be in the buffer if something is being piped in which I can check on STDIN BEFORE I call ReadFile. If this were the case I could just skip calling ReadFile and use some default text.
If there is not a way to do this, I can always pass in a command line argument that denotes that I should not check the pipe and just use the default text (or the other way around), but I would much prefer to do it the way that I specified.
Look at PeekNamedPipe(). Despite its name, it works for both named and anonymous pipes.
int main(int argc, const char *argv[])
{
char char_buffer[BUFSIZE];
DWORD bytes_read;
DWORD bytes_avail;
DWORD dw;
HANDLE stdin_handle;
bool is_pipe;
stdin_handle = GetStdHandle(STD_INPUT_HANDLE);
is_pipe = !GetConsoleMode(stdin_handle, &dw);
if (stdin_handle == INVALID_HANDLE_VALUE) {
std::cout << "Error: invalid handle value!\n\n";
} else {
while (1) {
if (is_pipe) {
if (PeekNamedPipe(stdin_handle, NULL, 0, NULL, &bytes_avail, NULL)) {
if (bytes_avail == 0) {
Sleep(100);
continue;
}
}
}
if (!ReadFile(stdin_handle, char_buffer, min(bytes_avail, BUFSIZE), &bytes_read, NULL)) {
break;
}
if (bytes_read == 0) {
break;
}
// Output what we have read so far
for (unsigned int i = 0; i < bytes_read; i++) {
std::cout << char_buffer[i];
}
}
}
return 0;
}
It looks like what you're really trying to do here is to determine whether you've got console input (where you use default value) vs pipe input (where you use input from the pipe).
Suggest testing that directly instead of trying to check if there's input ready: the catch with trying to sniff whether there's data in the pipe is that if the source app is slow in generating output, your app might make an incorrect assumption just because there isn't input yet available. (It might also be possible that, due to typeahead, there's a user could have typed in characters that area ready to be read from console STDIN before your app gets around to checking if input is available.)
Also, keep in mind that it might be useful to allow your app to be used with file redirection, not just pipes - eg:
myapp.exe < some_input_file
The classic way to do this "interactive mode, vs used with redirected input" test on unix is using isatty(); and luckily there's an equivalent in the Windows CRT - see function _isatty(); or use GetFileType() checking for FILE_TYPE_CHAR on GetStdHandle(STD_INPUT_HANDLE) - or use say GetConsoleMode as Remy does, which will only succeed on a real console handle.
This also works without overlapped I/O while using a second thread, that does the synchronous ReadFile-call. Then the main thread waits an arbitrary amount of time and acts like above...
Hope this helps...

AuthorizationExecuteWithPrivileges() as root Error

I am a beginner in cocoa...
I just want to start Apache and other process in my Cocoa App.
Here is my code :
OSStatus myStatus;
AuthorizationFlags myFlags = kAuthorizationFlagDefaults;
AuthorizationRef myAuthorizationRef;
FILE *pipe = NULL;
myStatus = AuthorizationCreate(NULL, kAuthorizationEmptyEnvironment, myFlags, &myAuthorizationRef);
AuthorizationItem myItems = {kAuthorizationRightExecute, 0, NULL, 0};
AuthorizationRights myRights = {1, &myItems};
myFlags = kAuthorizationFlagDefaults |
kAuthorizationFlagInteractionAllowed |
kAuthorizationFlagPreAuthorize |
kAuthorizationFlagExtendRights;
myStatus = AuthorizationCopyPrivilegedReference (&myAuthorizationRef,kAuthorizationFlagDefaults);
myStatus = AuthorizationCopyRights (myAuthorizationRef,&myRights, NULL, myFlags, NULL );
char *tool = "/usr/sbin/apachectl";
char *args[] = { "start",NULL} ;
myStatus = AuthorizationExecuteWithPrivileges(myAuthorizationRef, tool, kAuthorizationFlagDefaults, args, &pipe);
char c[100];
int n=fread(c,1,100,pipe);
c[n] = '\0';
NSLog(#"%s\n",c);
theResult : This operation requires rootWhen I run a 'whoami', I'm 'root' but when I run a getuid(), I'm '501'...
I try to use setuid(0); But it doesn't set !!
Can you help me? Thanks
I had this exact same problem. AuthorizationExecuteWithPrivileges allows you escalate your permission to root, but does not do it automatically (I guess to preserve the user session or whatever).
I ended up making a generic executable that would be run via AuthorizationExecuteWithPrivileges, and then that executable would setuid to root, and then exec the process you actually want to run as root.
Here's the source for the setuid wrapper executable:
#include <stdio.h>
int main(int argc, char** argv) {
if (argc < 2) {
printf("not enough arguments\n");
return -1;
}
if (0 != setuid(0)) {
printf("setuid failed.\n");
return -3;
}
int i;
char** argvz = (char**)malloc(sizeof(char*) * (argc - 1));
for (i = 1; i < argc; i++) {
argvz[i - 1] = argv[i];
}
execv(argv[1], argvz);
printf("execv returned?\n");
return -2;
}
Then, basically run (calling it via AuthorizationExecuteWithPrivileges) it as:
setuid my-program-to-run and arguments to pass
It will setuid as root, and then run the program in question with the args given.
Note that you must call setuid from AuthorizationExecuteWithPrivileges As only the pid that was created by AuthorizationExecuteWithPrivileges will have escalated privileges (and setuid will exec and replace the process with your own).
I would define access for user with uid=501 access to /usr/sbin/apachectl in /etc/sudoers, and execute "sudo /usr/sbin/apachectl" instead of /usr/sbin/apachectl in the code.
I'm facing exactly the same problem !
This is insane that a whoami returns "root" and a root command returns "root required" !!
The sudoers solution may works but I think we would have to search around the setuid bit, as written here https://github.com/notbrien/OSXSlightlyBetterAuth/blob/master/OSXSlightlyBetterAuth.m#L94

difficulty in returning string from dll function call using Win32::API import in Perl

The function is like this, string MyFuntion(long,long*)
so I have tried with lots of things but failed to get the string return.
Please help me out.
Win32::API->Import('My.dll','DWORD MyFunction(long a,long* b)')or die $^E;
my $var = MyFunction(1,0);
printf "%d : '%s'\n", length($var),$var;
DWORD is just a "long" type, and Win32::API will not do any conversion on such a return value. If your function returns a char *, just declare its prototype as char* MyFunction(...).
Or use one of the many many aliases for pointer to char that are already defined in Win32::API::Type.
Edit: It really is as simple as setting a prototype that returns char *. The complicated part is creating a DLL that exports the functions that you (and Win32::API) expects it to. This code, for example, which creates its own DLL and then imports and invokes its function through Win32::API, works on my system (Strawberry Perl 5.12.0):
$STRAWBERRY = "C:/strawberry512"; # YMMV
unlink "my_func.dll";
open DLL_SRC, '>', 'my_func.c';
print DLL_SRC q!
#include <stdlib.h>
#include <stdio.h>
#include <windows.h>
char* WINAPI MyFunc(int a, int b)
{
char *s = (char *) malloc(32);
if (a==0 && b==0) {
strcpy(s, "JAPH");
} else {
s[0] = 32 + (a % 64);
s[1] = 32 + (b % 64);
s[2] = '\0';
}
return(s);
}
!;
close DLL_SRC;
open DLL_DEF, '>', 'my_func.def';
print DLL_DEF "EXPORTS\nMyFunc\#8\n";
close DLL_DEF;
system("$STRAWBERRY/c/bin/gcc.exe", "-c", "my_func.c") ||
system("$STRAWBERRY/c/bin/gcc.exe",
"-mdll",
"-o", "junk.tmp",
"-Wl,--base-file,my_func.tmp", "my_func.o") ||
system("$STRAWBERRY/c/bin/dlltool",
"--dllname", "my_func.dll",
"--base-file", "my_func.tmp",
"--output-exp", "my_func.exp",
"--def", "my_func.def", "-k") ||
system("$STRAWBERRY/c/bin/gcc",
"-mdll",
"-o", "my_func.dll",
"my_func.o",
"-Wl,my_func.exp") ||
print "my_func.dll seems to have created successfully.\n\n";
use Win32::API;
Win32::API->Import('my_func',
'char* MyFunc(int a, int b)') or die $!,$^E;
$val = MyFunc(0,0);
print $val;
print MyFunc(1,65);
unlink "my_func.dll", "libmy_func.a", "my_func.def",
"my_func.o", "my_func.exp", "my_func.tmp", "my_func.c";
If you are having trouble replicating this example, start with something simple -- a trivial function that takes no args and returns an integer, for example -- and get that working first. Check $! and $^E at every step.

Resources