I'm trying to get a list of running domains from an application server. It takes a few seconds for the query to respond; so, it would be nice to run it in the background. However, it hangs, apparently waiting on something even though the command takes no input. When I bring it to the foreground, it immediately displays the results and quits. I also tried disconnecting stdin with 0<&-.
java -jar appserver-cli.jar list-domains &
How can I diagnose the issue? Or better yet, what's the problem?
I can see some open pipes and sockets.
ps --forest
PID TTY TIME CMD
16876 pts/1 00:00:00 bash
2478 pts/1 00:00:00 \_ java
2499 pts/1 00:00:00 | \_ stty
ls -l /proc/2478/fd
lrwx------. 1 vagrant vagrant 64 Mar 23 09:08 0 -> /dev/pts/1
lrwx------. 1 vagrant vagrant 64 Mar 23 09:08 1 -> /dev/pts/1
lrwx------. 1 vagrant vagrant 64 Mar 23 09:08 10 -> socket:[148228]
lrwx------. 1 vagrant vagrant 64 Mar 23 09:08 2 -> /dev/pts/1
lrwx------. 1 vagrant vagrant 64 Mar 23 09:08 24 -> socket:[148389]
lr-x------. 1 vagrant vagrant 64 Mar 23 09:08 73 -> pipe:[18170535]
lr-x------. 1 vagrant vagrant 64 Mar 23 09:08 75 -> pipe:[18170536]
I also see the following signal which does not show up when I run the process in the foreground.
futex(0x7fda7e0309d0, FUTEX_WAIT, 9670, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
--- SIGTTOU {si_signo=SIGTTOU, si_code=SI_KERNEL} ---
--- stopped by SIGTTOU ---
The following resources were helpful in figuring this out.
sigttin-sigttou-deep-dive-linux
cannot-rewrite-trap-command-for-sigtstp-sigttin-and-sigttou
From my post, it's clear the process is receiving SIGTTOU. It may be trying to configure terminal settings. I noticed that when I run from a script instead of interactive, there is no empty handler defined for SIGTTOU. The following resolves the issue.
bash -c "java -jar appserver-cli.jar list-domains &"
Related
All of a sudden, my Git-Bash is not starting up the ssh-agent and it hangs in there. So trouble this further, I have manually executed the below command and it hangs in there.
$ eval ssh-agent -s
And in the Windows Machine, I could see the process is alive
> Get-Process *bash, *ssh-add, *ssh-agent
Handles NPM(K) PM(K) WS(K) CPU(s) Id SI ProcessName
------- ------ ----- ----- ------ -- -- -----------
229 20 8484 12980 0.17 2516 1 bash
149 18 2884 8544 0.06 37152 1 git-bash
231 19 6628 9416 133.69 9844 1 ssh-agent
> Get-Service ssh-agent | Select StartType
StartType
---------
Manual
Additional Note:
If I type ssh-agent in command-prompt or in powershell it seems to execute fine. Only in Git Bash it seems to not work and this happened very recently.
Trying to understand, why ssh-agent is not getting started and hangs in ?
I am having a scenario where I need to create a script file that runs a couple of commands.
node index.js which runs continuously
./gradlew run which also run continuously
some commands which will run and finish automatically
I want to write one script file that does all the jobs like running the node server, Gradle run, and also other commands.
I'm thinking of one of the approaches to creating a new terminal tab and executing the commands in it. But I'm unable to find the exact code to create a new tab irrespective of the operating system.
You can use one terminal for this. Just run a command in background. You can do this by adding & at the end of the line. The commands stdout will still log to the terminal session.
# Sorry for the Polish date
while true; do echo $(date); sleep 2; done &
[1] 8873
czw, 28 paź 2021, 09:41:52 CEST
user#user-pc:~$ czw, 28 paź 2021, 09:41:54 CEST
czw, 28 paź 2021, 09:41:56 CEST
czw, 28 paź 2021, 09:41:58 CEST
If you want to bring back the job to the foreground fg.
user#user-pc:~$
user#user-pc:~$ fg
while true; do
echo $(date); sleep 2;
done
czw, 28 paź 2021, 09:48:07 CEST
czw, 28 paź 2021, 09:48:09 CEST
^C
So the script finally could look like this:
#!/bin/bash
./task1 &
./task2 &
./task3
I have a systemd unit script which looks something like this
cat /usr/lib/systemd/system/hello.service
[Unit]
Description=Simple Hello World service
After=syslog.target network.target
[Service]
Type=forking
EnvironmentFile=/root/hello.env
ExecStart=/bin/gdb /root/hello
StandardInput=tty-force
StandardOutput=inherit
TTYPath=/dev/pts/0
TTYReset=yes
TimeoutStartSec=infinty
[Install]
WantedBy=multi-user.target
The whole point is, i want to start the service with gdb on start up.[Since the process involves lot of environmental variables i cannot use the gdb directly on the process.]
systemctl start hello (which is actually working).
But once i exit out of gdb tty is completely messed up.None of the control key work, ^Z, ^C.
This are the observation till now.
As describer by systemd man pages with "StandardInput=tty-force", will actually force the executing process to take control of tty.
Before i launch the process
# tty
/dev/pts/0
# ps -aef | grep bash
root 2805 2803 0 10:42 pts/0 00:00:00 -bash
root 2860 2805 0 10:45 pts/0 00:00:00 grep --color=auto bash
After i launch
# tty
/dev/pts/0
# ps -aef | grep bash
root 2805 2803 0 10:42 ? 00:00:00 -bash
root 2884 2805 0 10:47 ? 00:00:00 grep --color=auto bash
Tried reset the terminal, still doesn't work.
subsequent systemctl command dsplay the below error
systemctl stop hello
Error creating textual authentication agent: Error opening current controlling terminal for the process (`/dev/tty'): No such device or address (polkit-error-quark, 0)
So the question is is there a way to reset the tty back to bash ?
I have a ruby 2.0 sinatra 'faceless' app that serves up json by calling an external service. It works fine.
The main app is run on port 80 in a ubuntu machine.
I also start an instance using 'foreman start' - so it runs on port 5000 on the same ubuntu virtual machine.
On the port 80 instance, the process 'foreman master' soaks up CPU time, while with the same load, the one on port 5000 uses essentially 0 CPU.
$ ps -a l
F UID PID PPID PRI NI VSZ RSS WCHAN STAT TTY TIME COMMAND
4 0 1615 1614 20 0 26140 17236 wait Sl+ tty1 0:28 foreman: master
0 1000 1899 1659 20 0 25036 16612 wait Sl+ pts/1 0:00 foreman: master
The apps were started at the same time and both had the same load (very light for 20 mins).
The only difference I can see is that the problem one is started on port 80 using a sudo command, and the other one is just started as a user process.
Is there a difference in how foreman needs to output log entries in a tty terminal vs a pts/1 terminal?
Note that with 40 people banging away on the app, the foreman master process is using 90% cpu while all the other ruby processes that are supposed to be doing the work are at 1% (9 unicorns).
I think its something to do with terminal output handled differently but I'm not sure.
Thanks for any help.
Is there a way to tell foreman or ruby to not write log stuff out at all?
EDIT
I now think that it is related to terminal logging, since i turned 95% of it off for the deployment app, and loads are better, but still higher than the normal non rvmsudo command.
I read this article somewhere:
"With a plain Apache server, it doesn’t matter much if you run many child processes—the processes are about 1 MB each (most of it shared), so they don’t eat a lot of RAM. The situation is different with mod_perl, where the processes can easily grow to 10 MB and more. For example, if you have MaxClients set to 50, the memory usage becomes 50 × 10 MB = 500 MB.Do you have 500 MB of RAM dedicated to the mod_perl server?"
I'm not using mod_perl on my server. I am using phusion passenger and ruby on rails with apache2. I am using prefork MPM and the MaxClients is set to the default 256. That means I can have 256 processes running concurrently at any given time. The article piqued my interest because I never have 256 apache2 processes running concurrently, usually I only have 80 apache2 processes running at any given time. But sometimes even just 80 bogs down my server to the point where the site just hangs when you try to load it.
When I run the following command, it sometimes shows 80 apache2 processes, for example:
ps aux | grep apache2
USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
root 1130 0.0 0.1 149080 10600 ? Ss 12:36 0:00 /usr/sbin/apache2 -k start
www-data 2051 0.0 0.3 163608 23592 ? S 16:46 0:00 /usr/sbin/apache2 -k start
www-data 2506 0.0 0.1 149376 7952 ? S 16:47 0:00 /usr/sbin/apache2 -k start
www-data 5149 0.0 0.1 149416 7980 ? S 16:49 0:00 /usr/sbin/apache2 -k start
www-data 5175 0.0 0.1 149368 7876 ? S 16:49 0:00 /usr/sbin/apache2 -k start
www-data 10212 0.0 0.1 149368 7848 ? S 16:53 0:00 /usr/sbin/apache2 -k start
www-data 19114 0.0 0.1 149368 7904 ? S 17:01 0:00 /usr/sbin/apache2 -k start
www-data 19138 0.0 0.1 150768 11856 ? S 17:01 0:00 /usr/sbin/apache2 -k start
www-data 20592 0.0 0.1 149428 8092 ? S 16:35 0:00 /usr/sbin/apache2 -k start
www-data 21336 0.0 0.1 149368 7808 ? S 17:03 0:00 /usr/sbin/apache2 -k start
www-data 21375 0.0 0.1 149432 7916 ? S 17:03 0:00 /usr/sbin/apache2 -k start
1000 26458 0.0 0.0 8112 896 pts/6 S+ 17:07 0:00 grep apache2
www-data 30848 0.0 0.1 149396 8044 ? S 16:43 0:00 /usr/sbin/apache2 -k start
But under memory, they range from 0.1 to 0.4, which doesn't seem like a lot of memory. So my question is when you send a request to the site from the browser, in addition to spawning a new apache2 process as a child process to the parent apache2 process, does passenger also create another process, something that could possibly be bogging down memory? When I run the top command, I notice sometimes it shows a ruby process at %100 CPU. I am wondering is that ruby process somehow linked to the apache2 processes via passenger. Something must be causing these processes to grow to big memory consumers, like that article stated. There must be something I am not looking at.
By the way, I have more than 5 gigs of memory on the machine:
$ cat meminfo
MemTotal: 6113156 kB
Phusion Passenger spawns your Ruby application processes separately. View them with passenger-memory-stats or passenger-status.