Solr is printing a lot of logs with empty params - performance

Question

Solr is printing lot of lines like below in slow_query logs.
solr_slow_requests.log.1:2023-02-03 17:30:27.084 WARN (qtp1961945640-50747) [c:products s:shard4 r:core_node47 x:products_shard4_replica_p44] o.a.s.c.S.SlowRequest slow: [products_shard4_replica_p44] webapp=/solr path=/select params={} rid=10.0.61.80-5704218 hits=9309 status=0 QTime=1280
solr_slow_requests.log.1:2023-02-03 17:30:27.157 WARN (qtp1961945640-50744) [c:products s:shard4 r:core_node47 x:products_shard4_replica_p44] o.a.s.c.S.SlowRequest slow: [products_shard4_replica_p44] webapp=/solr path=/select params={} rid=10.0.61.80-5704223 hits=9730 status=0 QTime=1508
solr_slow_requests.log.1:2023-02-03 17:30:27.325 WARN (qtp1961945640-50742) [c:products s:shard5 r:core_node59 x:products_shard5_replica_p56] o.a.s.c.S.SlowRequest slow: [products_shard5_replica_p56] webapp=/solr path=/select params={} rid=10.0.61.80-5704234 hits=9309 status=0 QTime=1993
solr_slow_requests.log.1:2023-02-03 17:30:27.326 WARN (qtp1961945640-50746) [c:products s:shard4 r:core_node47 x:products_shard4_replica_p44] o.a.s.c.S.SlowRequest slow: [products_shard4_replica_p44] webapp=/solr path=/select params={} rid=10.0.61.80-5704235 hits=9309 status=0 QTime=1994
solr_slow_requests.log.1:2023-02-03 17:30:27.657 WARN (qtp1961945640-50668) [c:products s:shard2 r:core_node23 x:products_shard2_replica_p20] o.a.s.c.S.SlowRequest slow: [products_shard2_replica_p20] webapp=/solr path=/select params={} rid=10.0.61.80-5704247 hits=9730 status=0 QTime=1140
solr_slow_requests.log.1:2023-02-03 17:30:27.700 WARN (qtp1961945640-50757) [c:products s:shard3 r:core_node35 x:products_shard3_replica_p32] o.a.s.c.S.SlowRequest slow: [products_shard3_replica_p32] webapp=/solr path=/select params={} rid=10.0.61.80-5704249 hits=9730 status=0 QTime=1068
solr_slow_requests.log.1:2023-02-03 17:30:27.720 WARN (qtp1961945640-50661) [c:products s:shard3 r:core_node35 x:products_shard3_replica_p32] o.a.s.c.S.SlowRequest slow: [products_shard3_replica_p32] webapp=/solr path=/select params={} rid=10.0.61.80-5704254 hits=9309 status=0 QTime=1023
solr_slow_requests.log.1:2023-02-03 17:30:27.816 WARN (qtp1961945640-49782) [c:products s:shard6 r:core_node71 x:products_shard6_replica_p68] o.a.s.c.S.SlowRequest slow: [products_shard6_replica_p68] webapp=/solr path=/select params={} rid=10.0.61.80-5704262 hits=9730 status=0 QTime=1246
solr_slow_requests.log.1:2023-02-03 17:30:27.825 WARN (qtp1961945640-50750) [c:products s:shard5 r:core_node59 x:products_shard5_replica_p56] o.a.s.c.S.SlowRequest slow: [products_shard5_replica_p56] webapp=/solr path=/select params={} rid=10.0.61.80-5704263 hits=9730 status=0 QTime=1847
solr_slow_requests.log.1:2023-02-03 17:30:27.888 WARN (qtp1961945640-50711) [c:products s:shard3 r:core_node35 x:products_shard3_replica_p32] o.a.s.c.S.SlowRequest slow: [products_shard3_replica_p32] webapp=/solr path=/select params={} rid=10.0.61.80-5704266 hits=9309 status=0 QTime=1150
solr_slow_requests.log.1:2023-02-03 17:30:27.995 WARN (qtp1961945640-50734) [c:products s:shard5 r:core_node59 x:products_shard5_replica_p56] o.a.s.c.S.SlowRequest slow: [products_shard5_replica_p56] webapp=/solr path=/select params={} rid=10.0.61.80-5704277 hits=9730 status=0 QTime=1481
I have two queries:
Why it doesn't print the params? Why params is always empty? params={} even though our requests send a lot of params.
Why is it taking too much time while No CPU spike?
We have 6 Node cluster with 6 shards of product collection. We are using 16 core machines with 32 GB Ram each.
A sample of our query looks like this:
solr.log.1:2023-02-06 12:02:54.090 INFO (qtp1961945640-4677) [c:products s:shard1 r:core_node11 x:products_shard1_replica_p8] o.a.s.c.S.Request [products_shard1_replica_p8] webapp=/solr path=/select params={df=_text_&distrib=false&fl=id&fl=score&shards.purpose=16388&start=0&fsv=true&fq=channel_identifier:632aff00940b4e27c80986f3&fq=zone_identifier:"_all_"&fq=is_available:True&fq=image_nature:("standard"+OR+"substandard"+OR+"default")&fq=product_online_date:[*+TO+NOW]&fq={!tag%3Dbrand_id}brand_id:("74"+OR+"235")&sort=popularity+desc+,id+asc&shard.url=http://IP:8983/solr/products_shard1_replica_p8/|http://IP:8983/solr/products_shard1_replica_n2/|http://IP:8983/solr/products_shard1_replica_p6/|http://IP:8983/solr/products_shard1_replica_t4/|http://IP:8983/solr/products_shard1_replica_p10/|http://IP:8983/solr/products_shard1_replica_n1/&rows=11550&rid=IP-225159&version=2&q=*:*&omitHeader=false&NOW=1675684973896&json={"query":+"*:*",+"params":+{"df":+"_text_",+"_route_":+"632aff00940b4e27c80986f3/2!",+"start":+11500,+"rows":+50},+"fields":+["*+score"],+"filter":+["channel_identifier:632aff00940b4e27c80986f3",+"zone_identifier:\"_all_\"",+"is_available:True",+"image_nature:(\"standard\"+OR+\"substandard\"+OR+\"default\")",+"product_online_date:[*+TO+NOW]",+{"#brand_id":+"brand_id:(\"74\"+OR+\"235\")"}],+"sort":+"popularity+desc+,id+asc"}&isShard=true&wt=javabin&_route_=632aff00940b4e27c80986f3/2!} hits=42585 status=0 QTime=193
It is happening only during a Load Test. Some of the params of Solr are as below:
-Djetty.home=/opt/solr/server
-Djetty.port=8983
-Dlog4j2.formatMsgNoLookups=true
-Dnewrelic.environment=[]
-Dsolr.data.home=
-Dsolr.default.confdir=/opt/solr/server/solr/configsets/_default/conf
-Dsolr.documentCache.initialSize=8339
-Dsolr.documentCache.size=8339
-Dsolr.filterCache.initialSize=8339
-Dsolr.filterCache.size=8339
-Dsolr.install.dir=/opt/solr
-Dsolr.jetty.inetaccess.excludes=
-Dsolr.jetty.inetaccess.includes=
-Dsolr.log.dir=/var/solr/data/logs
-Dsolr.log.muteconsole
-Dsolr.queryResultCache.initialSize=6671
-Dsolr.queryResultCache.size=6671
-Dsolr.solr.home=/var/solr/data/data
-Duser.timezone=UTC
-DzkClientTimeout=30000
-XX:+AggressiveOpts
-XX:+AlwaysPreTouch
-XX:+ParallelRefProcEnabled
-XX:+PrintGCApplicationStoppedTime
-XX:+PrintGCDateStamps
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintHeapAtGC
-XX:+PrintTenuringDistribution
-XX:+UseG1GC
-XX:+UseGCLogFileRotation
-XX:+UseLargePages
-XX:-OmitStackTraceInFastThrow
-XX:-OmitStackTraceInFastThrow
-XX:ConcGCThreads=4
-XX:G1ReservePercent=10
-XX:GCLogFileSize=20M
-XX:InitiatingHeapOccupancyPercent=80
-XX:MaxGCPauseMillis=100
-XX:MaxTenuringThreshold=8
-XX:NewRatio=3
-XX:NumberOfGCLogFiles=9
-XX:OnOutOfMemoryError=/opt/solr/bin/oom_solr.sh 8983 /var/solr/data/logs
-XX:ParallelGCThreads=4
-XX:PretenureSizeThreshold=64m
-XX:SurvivorRatio=4
-Xloggc:/var/solr/data/logs/solr_gc.log
-Xms20000m
-Xmx20000m
-Xss256k
-javaagent:/opt/solr/contrib/newrelic/newrelic.jar
-verbose:gc