We just upgraded one of our database servers from Oracle 12.2.0.1 to 19.15.0 and one our perl programs is behaving differently. We are running with RedHat 8.4 using Perl v5.26.3 and the latest DBD::Oracle and DBI from cpan.
Under 19c, a forked child hangs on exit if the parent has (or had) an oracle connection where the connection handle is a global (using 'our'). This does not happen if the connection handle is a local (declared with 'my'). When running against a 12c database, with DBD::Oracle using 12c libraries, the child process exits normally regardless of how the variable is 'declared'.
Note also that this issue does not happen if the child exec's another program.
The block of code below exhibits the issue in our environment. This is just a simplification. The real code uses packages and so the connection handle needs to be a global.
Any help appreciated.
use strict;
use warnings;
use DBI;
# If $dbh declared with my, this works witn 19c
# otherwise, exit in child below never returns
our $dbh = DBI->connect('dbi:Oracle:ORCL', 'scott', 'tiger',
{ RaiseError => 1, AutoCommit => 0, PrintError => 0, InactiveDestroy => 1 });
print "Connected to db\n";
$dbh->disconnect;
#undef $dbh; # Child process exits normally if this uncommented
my $pid;
$pid = fork;
if( $pid == 0 ) {
print "This is child process\n";
print "Child process exiting now\n";
exit 0; # Never returns if $dbh is a global
}
print "This is parent process and child ID is $pid\n";
print "Parent Waiting on child\n";
my $chldPid = wait;
print "Parent done. Child pid $chldPid has completed\n";
exit 0;
Related
I'm trying to write a DTrace script which does the following:
Whenever a new thread is started, increment a count.
Whenever one of these threads exits, decrement the count, and exit the script if the count is now zero.
I have something like this:
BEGIN {
threads_alive = 0;
}
proc:::lwp-start /execname == $$1/ {
self->started = timestamp;
threads_alive += 1;
}
proc:::lwp-exit /self->started/ {
threads_alive -= 1;
if (threads_alive == 0) {
exit(0);
}
}
However, this doesn't work, because threads_alive is a scalar variable and thus it is not multi-cpu safe. As a result, multiple threads will overwrite each other's changes to the variable.
I have also tried using an aggregate variable instead:
#thread_count = sum(1)
//or
#threads_entered = count();
#threads_exitted = count();
Unfortunately, I haven't found syntax to be able to do something like #thread_count == 0 or #threads_started == #threads_stopped.
DTrace doesn't have facilities for doing the kind of thread-safe data sharing you're proposing, but you have a few options depending on precisely what you're trying to do.
If the executable name is unique, you can use the proc:::start and proc:::exit probes for the start of the first thread and the exit of the last thread respectively:
proc:::start
/execname == $$1/
{
my_pid = pid;
}
proc:::exit
/pid == my_pid/
{
exit(0);
}
If you're using the -c option to dtrace, the BEGIN probe fires very shortly after the corresponding proc:::start. Internally, dtrace -c starts the specified forks the specified command and then starts tracing at one of four points: exec (before the first instruction of the new program), preinit (after ld has loaded all libraries), postinit (after each library's _init has run), or main (right before the first instruction of the program's main function, though this is not supported in macOS).
If you use dtrace -x evaltime=exec -c <program> BEGIN will fire right before the first instruction of the program executes:
# dtrace -xevaltime=exec -c /usr/bin/true -n 'BEGIN{ts = timestamp}' -n 'pid$target:::entry{printf("%dus", (timestamp - ts)/1000); exit(0); }'
dtrace: description 'BEGIN' matched 1 probe
dtrace: description 'pid$target:::entry' matched 1767 probes
dtrace: pid 1848 has exited
CPU ID FUNCTION:NAME
10 16757 _dyld_start:entry 285us
The 285us is due to the time it takes dtrace to resume the process via /proc or ptrace(2) on macOS. Rather than proc:::start or proc:::lwp-start you may be able to use BEGIN, pid$target::_dyld_start:entry, or pid$target::main:entry.
I had a simple perl scrip that runs some DB opertions on using DBD::Oracle package and also forks some processes. Child processes also connects with DB but creates there own DBH object.
This was working fine until I migrated from Linux RH4 to Linux RH6. With OS migration perl version also changed from 5.8 to 5.10.
On migration I face error as foll following:
DBD::Oracle::db DESTROY failed: ORA-03135: connection lost contact
Process ID: 1984105
Session ID: 1511 Serial number: 32085 (DBD ERROR: OCITransRollback) during global destruction.
My perl script:
$dbh = DBI->connect("DBI:Oracle:$server", "$username", "$password") or die "Couldn't connect to database: " . DBI->errstr;
$dbh->{AutoCommit} = 0;
load_metrics(); # here I do some DB operations using $dbh
##Create ForkManager instance with number of child processes equal to batches
my $pm = new Parallel::ForkManager(15);
for ($proc_num = 100; $proc_num >= 1; $proc_num--)
{
$pm->start($proc_num) and next;
printf localtime(). " [$PID] Start sampling for Metric\n";
system("./sample_impl.pl");
printf localtime()." [$PID] Finished Resampling Metric\n";
$pm->finish;
}
printf localtime(). " [$PID] Parent process waiting for Child processes\n";
$pm->wait_all_children;
print localtime()." "."sampling Finished\n";
The above error gets repeated 100 times.
If I remove ForkManager scrpts works just fine. I am not sharing DBH in Parent with Child, hence I am bit surprised to see this error.
Please advice on reason of this error and how to get this fixed.
I have a tcl script which runs multiple shell commands serially.
Something like this:
abc.tcl
command 1
command 2
command 3
...
command n
This script prints the outputs of these commands into a text file in the following format:
### ### ### ### ### ###
Command name
### ### ### ### ### ###
Command Output
### ### ### ### ### ##
I was trying to get the script to run faster but making the shell commands run in parallel instead of serially. By pushing them to the background (command a &). But I'm at a loss how to retain the formatting of my output text file as was the case before.
When I push the commands in to background I'm forced to append their outputs into a temporary file, but these files just have the output of the commands in a dump together. It's difficult to differentiate between the different outputs.
Is there someway I can redirect the output of each command running in the background to an individual temp file (maybe the name of the temp file can have the process id of the background running process). And once all commands have run, I can cat the outputs together in to the proper format? Any ideas/suggestions on how I can accomplish this.
If the commands don't have state that depends on each other, you can parallelize them. There are many ways to do this, but one of the easier is to use the thread package's thread pooling (which requires a threaded Tcl, the norm on many platform nowadays):
package require Thread
set pool [tpool::create -maxworkers 4]
# The list of *scripts* to evaluate
set tasks {
{command 1}
{command 2}
...
{command n}
}
# Post the work items (scripts to run)
foreach task $tasks {
lappend jobs [tpool::post $pool $task]
}
# Wait for all the jobs to finish
for {set running $jobs} {[llength $running]} {} {
tpool::wait $pool $running running
}
# Get the results; you might want a different way to print the results...
foreach task $tasks job $jobs {
set jobResult [tpool::get $pool $job]
puts "TASK: $task"
puts "RESULT: $jobResult"
}
The main tweakable is the size of the thread pool, which defaults to a limit of 4. (Set it via the -maxworkers option to tpool::create which I've listed explicitly above.) The best value to choose depends on how many CPU cores you've got and how much CPU load each task generates on average; you'll need to measure and tune…
You can also use the -initcmd option to pre-load each worker thread in the pool with a script of your choice. That's a good place to put your package require calls. The workers are all completely independent of each other and of the master thread; they do not share state. You'd get the same model if you ran each piece of code in a separate process (but then you'd end up writing more code to do the coordinating).
[EDIT]: Here's a version that will work with Tcl 8.4 and which uses subprocesses instead.
namespace eval background {}
proc background::task {script callback} {
set f [open |[list [info nameofexecutable]] "r+"]
fconfigure $f -buffering line
puts $f [list set script $script]
puts $f {fconfigure stdout -buffering line}
puts $f {puts [list [catch $script msg] $msg]; exit}
fileevent $f readable [list background::handle $f $script $callback]
}
proc background::handle {f script callback} {
foreach {code msg} [read $f] break
catch {close $f}
uplevel "#0" $callback [list $script $code $msg]
}
proc accumulate {script code msg} {
puts "#### COMMANDS\n$script"
puts "#### CODE\n$code"
puts "#### RESULT\n$msg"
# Some simple code to collect the results
if {[llength [lappend ::accumulator $msg]] == 3} {
set ::done yes
}
}
foreach task {
{after 1000;subst hi1}
{after 2000;subst hi2}
{after 3000;subst hi3}
} {
background::task $task accumulate
}
puts "WAITING FOR TASKS..."
vwait done
Notes: the tasks are Tcl commands that produce a result, but they must not print the result out; the fabric code (in background::task) handles that. These are subprocesses; they share nothing with one another, so anything you want them to do or be configured with must be sent as part of the task. A more sophisticated version could keep a hot pool of subprocesses around and in general work very much like a thread pool (subject to the subtle differences due to being in a subprocess and not a thread) but that was more code than I wanted to write here.
Result codes (i.e., exception codes) are 0 for “ok”, 1 for “error”, and other values in less common cases. They're exactly the values documented on the Tcl 8.6 catch manual page; it's up to you to interpret them correctly. (I suppose I should also add code to make the ::errorInfo and ::errorCode variable contents be reported back in the case of an error, but that makes the code rather more complex…)
I always become crazy with bash, i don't understand it.
I basically want to do this (i'm not using some specific syntax, it's just to explain my problem):
processes_count = 20;
for (i = 0; i < processes_count; i++)
{
php -f file.php "{$i}-{$processes_count}" &
proc_id[i] = $!
}
The above cycle start the processes. The next one should keep the processes "alive for ever"!
while(true)
{
foreach(proc_id as id)
{
if(!exist(proc_id[id]))
{
php -f file.php "{$id}-{$processes_count}" &
proc_id[id] = $!
}
}
sleep 5
}
If someone can help translating this into bash, python or something, thank you :)
I don't think you can do that because bash doesn't provide a method to 'wait for any one child process to die and let me know which one it was that died'. The nearest approach is wait:
wait
wait [jobspec or pid ...]
Wait until the child process specified by each process id pid or job specification
jobspec exits and return the exit status of the last command waited for. If a
job spec is given, all processes in the job are waited for. If no arguments are
given, all currently active child processes are waited for, and the return status
is zero. If neither jobspec nor pid specifies an active child process of the shell,
the return status is 127.
This means you can wait for a specific child to die, or you can wait for all children to die, but you can't do what you want.
If you drop into Perl or Python, you can do it, using the wait system call.
I have a program which is calling another program and processing the child's output, ie:
my $pid = open($handle, "$commandPath $options |");
Now I've tried a couple different ways to read from the handle without blocking with little or no success.
I found related questions:
perl-win32-how-to-do-a-non-blocking-read-of-a-filehandle-from-another-process
why-does-my-perl-sysread-block-when-reading-from-a-socket
But they suffer from the problems:
ioctl consistently crashes perl
sysread blocks on 0 bytes (a common occurrence)
I'm not sure how to go about solving this problem.
Pipes are not as functional on Windows as they are on Unix-y systems. You can't use the 4-argument select on them and the default capacity is miniscule.
You are better off trying a socket or file based workaround.
$pid = fork();
if (defined($pid) && $pid == 0) {
exit system("$commandPath $options > $someTemporaryFile");
}
open($handle, "<$someTemporaryFile");
Now you have a couple more cans of worms to deal with -- running waitpid periodically to check when the background process has stopped creating output, calling seek $handle,0,1 to clear the eof condition after you read from $handle, cleaning up the temporary file, but it works.
I have written the Forks::Super module to deal with issues like this (and many others). For this problem you would use it like
use Forks::Super;
my $pid = fork { cmd => "$commandPath $options", child_fh => "out" };
my $job = Forks::Super::Job::get($pid);
while (!$job->is_complete) {
#someInputToProcess = $job->read_stdout();
... process input ...
... optional sleep here so you don't consume CPU waiting for input ...
}
waitpid $pid, 0;
#theLastInputToProcess = $job->read_stdout();