I upgraded my pc from 2.1gh and 2gb ram to dual coure 2.6gh processor and 4GB RAM, magento runs faster but I am still not happy with it (takes 4-6 seconds to open a page).
My memory usage is around 40% alltogether.
Would upgrating to 8GB RAM speed up my magento locally?
I would say, by itself, No. The fact that you are sharing resources on a local machine between MySQL and PHP with Magento is inherently slow within itself. Will you get more throughput? Probably, but not enough to notice.
You will get more of a performance gain by installing Varnish, and enabling Magento full page caching AFTER you install more RAM. Magento cache stores itself in the RAM and so does Varnish. Also make sure you have APC cache installed. Those three COMBINED with more RAM will make all the difference in the world.
For Varnish .. Give it about 1GB RAM in the VCL settings .. Sounds like a lot, but it'll save your life.
For APC, give it at least 256MB of room in the APC settings ... It would probably behove you to do 512MB if you can afford it.
I am also going to include my PHP.INI magento optimized settings as well as my MySQL settings:
PHP.INI
max_execution_time = 18000
max_input_time = 60
memory_limit = 1024M
max_input_vars = 10000
post_max_size = 102M
upload_max_filesize =100 M
max_file_uploads = 20
default_socket_timeout = 60
pdo_mysql.cache_size = 2000
mysql.cache_size = 2000
mysqli.cache_size = 2000
apc.enabled = 1
apc.shm_segments = 1
apc.shm_size = 1024M
apc.num_files_hint = 10000
apc.user_entries_hint = 10000
apc.ttl = 0
apc.user_ttl = 0
apc.gc_ttl = 3600
apc.cache_by_default = 1
apc.filters = "apc\.php$"
apc.mmap_file_mask = "/tmp/apc.XXXXXX"
apc.slam_defense = 0
apc.file_update_protection = 2
apc.enable_cli = 0
apc.max_file_size = 10M
apc.use_request_time = 1
apc.stat = 1
apc.write_lock = 1
apc.report_autofilter = 0
apc.include_once_override = 0
apc.localcache = 0
apc.localcache.size = 256M
apc.coredump_unmap = 0
apc.stat_ctime = 0
apc.canonicalize = 1
apc.lazy_functions = 1
apc.lazy_classes = 1
And MySQL
MY.CNF
key_buffer = 256M
max_allowed_packet = 16M
thread_stack = 192K
thread_cache_size = 32
myisam-recover = BACKUP
max_connections = 2500
query_cache_limit = 2M
query_cache_size = 64M
expire_logs_days = 10
max_binlog_size = 100M
[mysqldump]
quick
quote-names
max_allowed_packet = 16M
[isamchk]
key_buffer = 16M
I hope that helps you
If your memory usage is 40% now, then no. Sufficient RAM does make a difference, but in this case the extra 4 wouldn't make much of a difference.
Magento is quite slow due do its complexity and the fact that it uses thousands of files.
To increase Magento's load speed, try to disable stuff you don't need in the admin section, and perhaps Google for other tips. Also, the load speed will differ between different browsers.
Related
This is my first ever post so would appreciate some guidance. Situation is as below,
Replicating from VM A (Main Production Serer) to VM B (Slave used mainly for DB Replica and generating a few reports and also replicating down the stream to 2 other testing servers)
VM A Cofig:
32GB vRAM
8 vCPU (some Xeon CPU running at 2.4GHz)
1.5TB SSD
Windows Server 2012 R2
Production Applicaton running on XAMPP using Apache & MariaDB 10.1.38
VM B Config:
16GB vRAM
4 vCPU (same variant as VM A)
300 GB SSD
Same Windows Server 2012 R2 as VM A
Using the same platform as VM A- XAMPP with Apache & MariaDB 10.1.38
Problem:
No matter what settings I use, parallel replication just doesn't work like it should. Seconds behond master is continuously increasing. I've tried various values for binlog_commit_wait_count & binlog_commit_wait_usec as well but to no avail. At 1 or 2 combos, the group commit ratio increases a little but then eventually reduces again to <1.05. Settings for my.ini for the slave VM B is pasted below,
my.ini:
key_buffer = 8M
max_allowed_packet = 32M
#was 256M
sort_buffer_size = 16M
#was 2M
net_buffer_length = 8K
read_buffer_size = 8M
#was 256K then 4M then 8M
read_rnd_buffer_size = 16M
myisam_sort_buffer_size = 4M
query_cache_size=512M
query_cache_limit = 128M
#didn't exist, was 64M
join_buffer_size=128M
#didn't exist
thread_cache_size = 32
table_open_cache = 20000
#didn't exist
tmp_table_size = 256M
#didn't exist
max_heap_table_size = 256M
#didn't exist
log_error = "mysql_error.log"
expire_logs_days=30
log_bin_trust_function_creators = 1
slave-skip-errors = 1062,1060,1032
connect_timeout=100
max_connections = 2000
skip-host-cache
#skip-name-resolve
#innodb_force_recovery = 1
slave_parallel_threads = 4
#slave_parallel_workers = 4
slave_parallel_mode=optimistic
slave_domain_parallel_threads=4
slave_compressed_protocol = 1
#slave_parallel_max_queued = 131072
#slave_parallel_max_queued=16777216
server-id = 101
binlog-format = mixed
log-bin=mysql-bin
relay-log = mysql-relay-bin
log-slave-updates = 1
report_host=slave-main-replication-server
sync_binlog=1
#binlog_commit_wait_count = 30
binlog_commit_wait_count = 25
binlog_commit_wait_usec = 100000
innodb_buffer_pool_size = 8G
#was 2560M
#innodb_additional_mem_pool_size = 32M
innodb-buffer-pool-instances=4
#didn't exist
innodb_log_file_size = 256M
innodb_log_buffer_size = 64M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
innodb_io_capacity=1000
#default 200
innodb_io_capacity_max=3000
Any ideas would be appreciated. I've gone through dozens of posts and wikis and everything else inbetween of whatever I could find on the subject but none of them helped. I've tried setting "sync_binlong=0" as well, nothing happened. Is it just the CPU cores causing a bottleneck? I'm all out of ideas. Been trying to solve this for almost 2 months now. Please help.
I have seen many tutorial on how to tune Linux to scale Node.js or Erlang server to 600K+ concurrent connections.
But I have not found similar tutorial for windows, can someone help with what are the similar knobs that exist for Windows.
/etc/security/limits.d/custom.conf
In root soft nofile 1000000
root hard nofile 1000000
* soft nofile 1000000
* hard nofile 1000000
* List item
/etc/sysctl.conf
fs.file-max = 1000000
fs.nr_open = 1000000
net.ipv4.netfilter.ip_conntrack_max = 1048576
net.nf_conntrack_max = 1048576
“fs.file-max”
The maximum file handles that can be allocated
“fs.nr_open”
Max amount of file handles that can be opened
“net.ipv4.netfilter.ip_conntrack_max”
how many connections the NAT can keep track of in the “tracking” table Default is: 65536
// Increase total number of connection allowed
[HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
TcpnumConnections = 16,777,214
// increase MaxFreeTcbs to > than number concurrent connection
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \Services \Tcpip
\Parameters] MaxFreeTcbs = 2000 (Default = RAM dependent, but usual
Pro = 1000, Srv=2000)
// Increase hash table size for the hashtable helping TCP lookup
[HKEY_LOCAL_MACHINE \System \CurrentControlSet \services \Tcpip
\Parameters] MaxHashTableSize = 512 (Default = 512, Range = 64-65536)
// reduce tcp delay to limit half-open connection
[HKEY_LOCAL_MACHINE\System \CurrentControlSet \services \Tcpip \Parameters]
TcpTimedWaitDelay = 120 (Default = 240 secs, Range = 30-300)
After a crash of NN-A(active) because of memory not enough (too much blocks/files), we upgrade the NN-A with much more memory, but do not upgrade NN-B(not active) immediately.
With difference HeapSize, we deleted some files(80million to 70million), and then NN-B crashed. NN-A became active.
Then we upgraded NN-B, and start it. It stucks in safemode with logs like:
The reported blocks 4620668 needs additional 62048327 blocks to reach the threshold 0.9990 of total blocks 66735729.
The reported blocks X needs.. X increases slowly, and I checked the Heap usage:
Attaching to process ID 11598, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.79-b02
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 107374182400 (102400.0MB)
NewSize = 2006515712 (1913.5625MB)
MaxNewSize = 2006515712 (1913.5625MB)
OldSize = 4013096960 (3827.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 1805910016 (1722.25MB)
used = 1805910016 (1722.25MB)
free = 0 (0.0MB)
100.0% used
Eden Space:
capacity = 1605304320 (1530.9375MB)
used = 1605304320 (1530.9375MB)
free = 0 (0.0MB)
100.0% used
From Space:
capacity = 200605696 (191.3125MB)
used = 200605696 (191.3125MB)
free = 0 (0.0MB)
100.0% used
To Space:
capacity = 200605696 (191.3125MB)
used = 0 (0.0MB)
free = 200605696 (191.3125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 105367666688 (100486.4375MB)
used = 105192740832 (100319.61520385742MB)
free = 174925856 (166.82229614257812MB)
99.83398526179955% used
Perm Generation:
capacity = 68755456 (65.5703125MB)
used = 41562968 (39.637535095214844MB)
free = 27192488 (25.932777404785156MB)
60.45042883578577% used
14501 interned Strings occupying 1597840 bytes.
at the same time, NN-A's Heap:
Attaching to process ID 6061, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.79-b02
using parallel threads in the new generation.
using thread-local object allocation.
Concurrent Mark-Sweep GC
Heap Configuration:
MinHeapFreeRatio = 40
MaxHeapFreeRatio = 70
MaxHeapSize = 107374182400 (102400.0MB)
NewSize = 1134100480 (1081.5625MB)
MaxNewSize = 1134100480 (1081.5625MB)
OldSize = 2268266496 (2163.1875MB)
NewRatio = 2
SurvivorRatio = 8
PermSize = 21757952 (20.75MB)
MaxPermSize = 85983232 (82.0MB)
G1HeapRegionSize = 0 (0.0MB)
Heap Usage:
New Generation (Eden + 1 Survivor Space):
capacity = 1020723200 (973.4375MB)
used = 643184144 (613.3881988525391MB)
free = 377539056 (360.04930114746094MB)
63.01259185644061% used
Eden Space:
capacity = 907345920 (865.3125MB)
used = 639407504 (609.7865142822266MB)
free = 267938416 (255.52598571777344MB)
70.47009193582973% used
From Space:
capacity = 113377280 (108.125MB)
used = 3776640 (3.6016845703125MB)
free = 109600640 (104.5233154296875MB)
3.3310377528901736% used
To Space:
capacity = 113377280 (108.125MB)
used = 0 (0.0MB)
free = 113377280 (108.125MB)
0.0% used
concurrent mark-sweep generation:
capacity = 106240081920 (101318.4375MB)
used = 42025146320 (40078.30268859863MB)
free = 64214935600 (61240.13481140137MB)
39.55677138092327% used
Perm Generation:
capacity = 51249152 (48.875MB)
used = 51131744 (48.763031005859375MB)
free = 117408 (0.111968994140625MB)
99.77090742886828% used
16632 interned Strings occupying 1867136 bytes.
We tried to restart both, NN-A startup and became active in 10 minutes, but NN-B stuck forever.
Finally I dumped the heap usage:
num #instances #bytes class name
----------------------------------------------
1: 185594071 13362773112 org.apache.hadoop.hdfs.protocol.proto.HdfsProtos$BlockProto
2: 185594071 13362773112 org.apache.hadoop.hdfs.protocol.proto.DatanodeProtocolProtos$ReceivedDeletedBlockInfoProto
3: 101141030 10550504248 [Ljava.lang.Object;
4: 185594072 7423762880 org.apache.hadoop.hdfs.protocol.Block
5: 185594070 7423762800 org.apache.hadoop.hdfs.server.protocol.ReceivedDeletedBlockInfo
6: 63149803 6062381088 org.apache.hadoop.hdfs.server.namenode.INodeFile
7: 23241035 5705267888 [B
It shows very very large count of ReceivedDeletedBlock, but Why?
I solved this by change the dfs.blockreport.initialDelay to 300, the reason of failure is Block Report Storm.
These are the statistics for my Ehcache.
I have it configured to use only memory (no persistence, no overflow to disk).
cacheHits = 50
onDiskHits = 0
offHeapHits = 0
inMemoryHits = 50
misses = 1194
onDiskMisses = 0
offHeapMisses = 0
inMemoryMisses = 1138
size = 69
averageGetTime = 0.061597
evictionCount = 0
As you can see, misses is higher than onDiskMisses + offHeapMisses + inMemoryMisses. I do have statistics strategy set to best effort:
cache.setStatisticsAccuracy(Statistics.STATISTICS_ACCURACY_BEST_EFFORT)
But the hits add up, and the difference between misses is rather large. Is there a reason why the misses do not add up correctly?
This quesiton is similar to Ehcache misses count and hitrate statistics, but the answer attributes the differences to the multiple tiers. There is only one tier here.
I'm almost certain that you're seeing this because inMemoryMisses does not include misses due to expired elements. On a get if the value is stored, but expired then you will not see an inMemoryMiss recorded, but you will see a cache miss.
I have an issue on running MySql on Windows server 2008 which has 32 GB Ram and Intel Xeon 2.70 Quad Core processor.
The database is only of 258 MB.
While running a php script which export the data from database and dumps into dbf files. It takes 50 mins.
CPU Usage goes around ~50%. mostly take by mysqld.exe.
CPU Usage goes around ~30% even if a single user logs in. I tested with 3 users and it went upto 63%
Below is my current configuration for my.ini:
[client]
port = 3306
socket = "C:/xampp/mysql/mysql.sock"
[mysqld]
port= 3306
socket = "C:/xampp/mysql/mysql.sock"
basedir = "C:/xampp/mysql"
tmpdir = "C:/xampp/tmp"
datadir = "D:/ADAT_System/Database/data"
pid_file = "mysql.pid"
key_buffer = 16M
max_allowed_packet = 1M
sort_buffer_size = 512K
net_buffer_length = 8K
read_buffer_size = 256K
read_rnd_buffer_size = 512K
myisam_sort_buffer_size = 8M
log_error = "mysql_error.log"
log-output=FILE
slow_query_log = ON
long_query_time = 5
query_cache_size = 128MB
query_cache_type = ON
query_cache_limit = 10MB
plugin_dir = "C:/xampp/mysql/lib/plugin/"
skip-federated
server-id
innodb_data_home_dir = "D:/ADAT_System/Database/data"
innodb_data_file_path = ibdata1:10M:autoextend
innodb_log_group_home_dir = "D:/ADAT_System/Database/data"
innodb_buffer_pool_size = 16M
innodb_additional_mem_pool_size = 2M
innodb_log_file_size = 5M
innodb_log_buffer_size = 8M
innodb_flush_log_at_trx_commit = 1
innodb_lock_wait_timeout = 50
[mysqldump]
quick
max_allowed_packet = 16M
[mysql]
no-auto-rehash
[isamchk]
key_buffer = 20M
sort_buffer_size = 20M
read_buffer = 2M
write_buffer = 2M
[myisamchk]
[mysqlhotcopy]
interactive-timeout
Could someone please help me in optimizing the MySQL.I am pretty new to this.
Thanks,
Himanshu
Generally a SQL request (Given that the DB is local) should be quicker than that. *granted.
Try looking here for a great list of performance Tweaking.
http://www.askapache.com/mysql/performance-tuning-mysql.html
That aside, I am presuming its the structure of your requests.
Try keep your call into very few requests (1-10) and handle the data as a lump sum.
Doing Many requests can hinder the system and drastically reduce that performance.
Your server has more than enough ram and processing power.
Thats the best i can offer for now.
Look at MySQLTuner-0.6. It is on codeplex. It gave me a huge amount of hints in optimizing our environment.