Nginx occupies all the available disk space. How to set limit for log files on Mac OS?
Rotate the log files. On OS X, newsyslog is the preferred utility to do that. Set up a file like this in /etc/newsyslog.d/nginx.conf:
# logfilename [owner:group] mode count size when flags [/pid_file] [sig_num]
/var/log/nginx.log deceze:wheel 644 2 1024 * J
Read https://www.newsyslog.org/manual.html for more information.
Building on #deceze's answer, here's an adapted version of Apple's Apache configuration for a Homebrew-installed nginx:
#logfilename [owner:group] mode count size when [flags] [/pid_file] [sig_num]
/opt/homebrew/var/log/nginx/access.log _nginx:_nginx 644 10 20480 * Z /opt/homebrew/var/run/nginx.pid 30
This is assuming you're running as user:group _nginx:_nginx. It sends the correct SIGUSR1 signal (30 on macOS) to nginx. Also changed to using gzip instead of bzip2.
Related
I am trying to install Oracle 11gR2 on oracle linux 7.
As per the documentation given in
https://oracle-base.com/articles/11g/oracle-db-11gr2-installation-on-oracle-linux-7 there i think i have properly configured it.
but unfortunately i get the following error.
but when i see parameters with sysctl -p i get
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 536870912
kernel.shmmni = 4096
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default = 262144
net.core.rmem_max = 4194304
net.core.wmem_default = 262144
net.core.wmem_max = 1048586
or
sysctl -a | grep "kernel.sem"
gives
kernel.sem = 250 32000 100 128
kernel.sem_next_id = -1
sysctl: reading key "net.ipv6.conf.all.stable_secret"
sysctl: reading key "net.ipv6.conf.default.stable_secret"
sysctl: reading key "net.ipv6.conf.ens192.stable_secret"
sysctl: reading key "net.ipv6.conf.lo.stable_secret"
what am i missing ?
Nothing wrong with your installation style,
Firstly, log as root
$ su root
and then check whether the resource limits are in the recomended range for the file /etc/security/limits.conf as below ( if not, you need to update the values in that file ) :
--# the soft limit for the stack setting
$ ulimit -Ss
10240
$ ulimit -Hs
32768
--# the soft and the hard limits for the number of processes available to a user
$ ulimit -Su
2047
$ ulimit -Hu
16384
--# he soft and hard limits for the file descriptor setting
$ ulimit -Sn
4096
$ ulimit -Hn
65536
if the problem still persists, and you use version before 11.2.0.4(there might be a bug),
you might meet such situation, re-install at least with version 11.2.0.4.
According to this answer
there is a bug in RHEL 7 with 11.2.0.4, if you really need to install 11.2.0.4 on RHEL 7, you should really think about it again, otherwise you can go ahead with 12c.
I have plain vanilla CouchDB from Apache, which runs as an App running on a Mac OS X 10.9. If I try to attach an attachment to a document that is above 1 Meg in size, it just hangs and does nothing.
I have tried to use couchdbs on Linux, and there the sky is the limit.
I first thought it had to do with low limits on the mac but it doesn't seem so :
➜ ~ ulimit -a
-t: cpu time (seconds) unlimited
-f: file size (blocks) unlimited
-d: data seg size (kbytes) unlimited
-s: stack size (kbytes) 8192
-c: core file size (blocks) 0
-v: address space (kbytes) unlimited
-l: locked-in-memory size (kbytes) unlimited
-u: processes 709
-n: file descriptors 256
What is causing this ? Why ? And how to fix this ?
Check the config files given by couchdb -c. You probably have this somewhere in them (for some unknown reason):
[couchdb]
max_attachment_size = 1048576 ; bytes
Remove or comment the line and you should be fine.
Or maybe it was compiled with this hardcoded so you could add this line to one of the config file and increase the value.
Update
max_attachment_size is undocumented so probably not safe to use. I leave the original answer as it seems to have solved the problem of the OP but according to the docs, the attachment size should be unlimited. Also attachment_stream_buffer_size is the config key controlling the chunk size of the attachments which might relevant.
I have a job running on production which process xml files.
xml files counts around 4k and of size 8 to 9 GB all together.
After processing we get CSV files as output. I've a cat command which will merge all CSV files to a single file I'm getting:
Errno::ENOMEM: Cannot allocate memory
on cat (Backtick) command.
Below are few details:
System Memory - 4 GB
Swap - 2 GB
Ruby : 1.9.3p286
Files are processed using nokogiri and saxbuilder-0.0.8.
Here, there is a block of code which will process 4,000 XML files and output is saved in CSV (1 per xml) (sorry, I'm not suppose to share it b'coz of company policy).
Below is the code which will merge the output files to a single file
Dir["#{processing_directory}/*.csv"].sort_by {|file| [file.count("/"), file]}.each {|file|
`cat #{file} >> #{final_output_file}`
}
I've taken memory consumption snapshots during processing.It consumes almost all part of the memory, but, it won't fail.
It always fails on cat command.
I guess, on backtick it tries to fork a new process which doesn't get enough memory so it fails.
Please let me know your opinion and alternative to this.
So it seems that your system is running pretty low on memory and spawning a shell + calling cat is too much for the few memory left.
If you don't mind loosing some speed, you can merge the files in ruby, with small buffers.
This avoids spawning a shell, and you can control the buffer size.
This is untested but you get the idea :
buffer_size = 4096
output_file = File.open(final_output_file, 'w')
Dir["#{processing_directory}/*.csv"].sort_by {|file| [file.count("/"), file]}.each do |file|
f = File.open(file)
while buffer = f.read(buffer_size)
output_file.write(buffer)
end
f.close
end
You are probably out of physical memory, so double check that and verify your swap (free -m). In case you don't have a swap space, create one.
Otherwise if your memory is fine, the error is most likely caused by shell resource limits. You may check them by ulimit -a.
They can be changed by ulimit which can modify shell resource limits (see: help ulimit), e.g.
ulimit -Sn unlimited && ulimit -Sl unlimited
To make these limit persistent, you can configure it by creating the ulimit setting file by the following shell command:
cat | sudo tee /etc/security/limits.d/01-${USER}.conf <<EOF
${USER} soft core unlimited
${USER} soft fsize unlimited
${USER} soft nofile 4096
${USER} soft nproc 30654
EOF
Or use /etc/sysctl.conf to change the limit globally (man sysctl.conf), e.g.
kern.maxprocperuid=1000
kern.maxproc=2000
kern.maxfilesperproc=20000
kern.maxfiles=50000
I have the same problem, but instead of cat it was sendmail (gem mail).
I found problem & solution here by installing posix-spawn gem, e.g.
gem install posix-spawn
and here is the example:
a = (1..500_000_000).to_a
require 'posix/spawn'
POSIX::Spawn::spawn('ls')
This time creating child process should succeed.
See also: Minimizing Memory Usage for Creating Application Subprocesses at Oracle.
Why result of php memory_get_peak_usage differs so much from memory size that is shown as allocated to process when using 'top' or 'ps' commands in Linux?
I've set 2 Mb of memory_limit in php.ini
My single-string php-script with
echo memory_get_peak_usage(true);
says that it is using 786432 bytes (768 Kb)
If I try to ask system about current php process
echo shell_exec('ps -p '.getmypid().' -Fl');
it gives me
F S UID PID PPID C PRI NI ADDR SZ WCHAN RSS PSR STIME TTY TIME CMD
5 S www-data 14599 14593 0 80 0 - 51322 pipe_w 6976 2 18:53 ? 00:00:00 php-fpm: pool www
RSS param is 6976, so memory usage is 6976 * 4096 = 28573696 = ~28 Mb
Where that 28 Mb come from and is there any way to decrease memory size that is being used by php-fpm process?
The memory size is mostly used by the PHP process itself. memory_get_peak_usage() returns the memory used by your specific script. Ways to reduce the memory overhead is to remove the number of extensions, statically compile PHP, etc.. But don't forget that php-fpm (should) fork and that a lot of the memory usage that's not different between PHP process is in fact shared (until it changes).
PHP itself may only be set to a 2meg limit, but it's running WITHIN a Apache child process, and that process will have a much higher memory footprint.
If you were running the script from the command line, you'd get memory usage of PHP by itself, as it's not wrapped within Apache and is running on its own.
The peak memory usage is for the current script only.
I have the following code that tries to tie arrays to files. Except, when I run this code, it only creates 2045 files. What is the issue here?
#!/usr/bin/perl
use Tie::File;
for (my $i = 0; $i < 10000; $i++) {
#files{$i} = ();
tie #{$files{$i}}, 'Tie::File', "files//tiefile$i";
}
Edit: I am on windows
You are accumulating open file handles (see ulimit -n, setrlimit RLIMIT_NOFILE/RLIMIT_OFILE), and you ultimately hit a 2048 open file descriptors limit (2045 + stdin + stdout + stderr.)
Under Windows you will have to rewrite your application so that it has at most 2048 open file handles at any one time, since the 2048 limit is hard limit (cannot be modified) in MSVC's stdio.
On Linux machines go to /etc/security/limits.conf and add or modify these lines
* soft nofile 10003
* hard nofile 10003
This will increase the number of files each process can have open to 10003 (remember that you always start with three open: stdin, stdout, and stderr).
Based on you comments it sounds like you are using a Win32 machine. I can't find a way to increase the number of open files per process, but you might, and I stress might, be able to handle this through fork'ing (which is really threading on Win32).