Below is the question, I was asked in an interview,
Datacenter has 10000 servers.We have a single syslog driver which collates all the logs from all the servers in the datacenter and writes it in a single file called syslog.log
Let's say the datacenter has 1000 Admins.At any point of time any admin can login to syslog server and invoke a command say
getlog --serverid --severity
And the command should continuously tail the logs matching the conditions provided by the user untill he interupts.
Any number of users can concurrently login to this server and run this command. His request should be honoured, but with one condition, at any given point in time there can be only one file descriptor open for syslog.log file.
Implement getlog such that it satisfies the above condition.
I told my approach as Critical Section problem, we can use mutex/semaphore to lock the file until a user finishes. But he is expecting something like Client-Server Model.
How to serve this functionality using client and server architecture?
What is the best approach to solve this?
Related
I have a nifi flow that sends more than 50 files per minute using the putFTP processor. The server has limited resources, but I need to send in a faster pace. I looked at the ftp server logs (not nifi), and my conclusion:
A new ftp connection (session) is created for every file. Is there an option to configure many files on one session? (connect to port 21, authenticate once, and then send many files on different ports)
When sending one file, many CWD (Change Working Directory) commands are sent. For example, sending file to /myfiles/test/dest/file.txt:
CWD /
CWD /myfiles
CWD /
CWD /myfiles/test
CWD /
CWD /myfiles/test/dest
This is not efficient. Is there any way to improve the putFTP? Is this a bug?
First question: use run duration
A new ftp connection (session) is created for every file. Is there an
option to configure many files on one session? (connect to port 21,
authenticate once, and then send many files on different ports)
First, (if it fits your use case) you can use the MergeContent processor to merge multiple (smaller) flow files into one (bigger) flow file and feed it to PutFTP.
Second, the PutFTP processor has the SupportsBatching annotation:
Marker annotation a Processor implementation can use to indicate that
users should be able to supply a Batch Duration for the Processor. If
a Processor uses this annotation, it is allowing the Framework to
batch ProcessSessions' commits, as well as allowing the Framework to
return the same ProcessSession multiple times...
Source: https://github.com/apache/nifi/blob/master/nifi-api/src/main/java/org/apache/nifi/annotation/behavior/SupportsBatching.java
Increase the run duration of your PutFTP processor towards more throughput to use the same task for many flow files. You might want to adjust the Maximum Batch Size in the properties tab to accommodate to that change.
Read more about it here:
Dcoumentation: Run duration
Understanding NiFi processor's "Run Duration" functionality.
What should be Ideal Run-duration and Run schedule configuration in nifi processors
Second question: inspect source code
When sending one file, many CWD (Change Working Directory) commands
are sent. For example, sending file to /myfiles/test/dest/file.txt
By inspecting FTPTransfer.java you can see, that the put method does the following:
put -> get client
put -> get client -> resetWorkingDirectory -> changeWorkingDirectory(homeDirectory)
put -> setAndGetWorkingDirectory
This might be the behavior you discovered.
My Situation
I am using the SSH Command tool to retrieve log info from a remote Linux server using the tail command. A separate thread group logs in users which causes the log file to update with the login information. To get the data I need, I use a regex extractor on the response data. For every thread jmeter creates a new ssh connection to retrieve the new messages in that log file.
Here's my current setup:
Thread Group
- Get random user
- Login User
- SSH into server using tail command
- Extract relevant data
My Question
Instead of sshing into the Linux server every thread, causing extra load and log messages, I want to connect once at the start of the test and continuously extract from the log file using the tail -f command, for example, combined with the regex. Is this possible?
I would say that it is not possible or at least not easy.
In order to be able to extract the data from the response using Regular Expression Extractor you need to have SampleResult, it means that the SSH Command request must be completed. If you're going to use tail -f the request will never end.
If you want to minimize the number of connections you can consider using JSch library to establish the connection once and execute the commands within the bounds of a single Session.
If the above solution makes sense - consider migrating to JSR223 Sampler and Groovy language, you can find example code to execute remote command over SSH in i.e. Exec.java class
I am quite new for log4j2 logger and my requirement to write a log from application server and web server.
I am having two different environment on which J BOSS server is deployed.
Now I am having a log file on web server environment which is writing logs for errors and I want to write logs from application server also in same file.
Please suggest.
If you want the logs to be integrated together you should use a solution like Splunk or Elastic Search/Logstash/Kibana (ELK).
When you try to write to a file from 2 different processes your file will get corrupted unless you use file locking. However, your throughput will decrease significantly and it isn't supported for rolling files. So the best approach is to send the logs to a single process where they can be aggregated.
The task is to have an Oracle report generated daily, automatically, and e-mailed to a user.
So I've sort of got this working (it works if I hardcode one of the reports server names below).
I created a job on the database that will generate the report. I'm able to get the report to email as a PDF to the destination with this command:
UTL_HTTP.REQUEST('http://server/reports/rwservlet?server=specific_report_server &report='||p_report_name||'&userid='||p_connstring||'&destype=mail'||p_parameters||'&desname='||p_to_recipientlist||' &cc='||p_cc_recipientlist||'&bcc='||p_bcc_recipientlist||'&subject=%22' || REPLACE(p_subject,' ','%20') || '%22¶mform=no&DESformat=pdf&ENVID='||p_envid);
That works great...
The problem however is that my organization has two report servers that are load balanced. Our server team could take down one of the servers without really any warning, so I can't just hardcode the report server name (the ?server= parameter above) with one of the report server names because it will work for a while, then when that server goes down, it will stop working.
My server team asked me to look for a way to pull the server from the formsweb.cfg file or from default.env value within the job (there are parameters in there that hold the server name). The idea there is that the "http://server" piece will direct the report to be run on the appropriate server, and the first part of the job could get the reports server name from the config file that the report is run on. I'm not sure if this is possible from the database level, or how to do this. Any ideas?
Is there a better way that this can be done, perhaps?
If there are two load-balanced servers, that strongly implies that the network folks must have configured some sort of virtual IP (VIP) for the service. You (and everyone else) should be using that VIP rather than a specific server name.
For example, if you have two servers reportA.yourdomain.com and reportB.yourdomain.com, you would almost certainly create a VIP for reports.yourdomain.com that load balances between the two servers (and knows whether one of the servers is down or whether a new reportC server has been added). This VIP would either do the load balancing itself or would point to an actual physical load balancer that distributes the traffic. All applications would reference the reports.yourdomain.com VIP rather than any hard-coded server names.
I have a web service that calls some stored procedure on a AS400 via JTOpen.
What I would like to do is that the connections used to call the stored procedures was opened in a specific subsystem with a specific user, instead of qusrwrk/quser as now (default).
I think I can be able to clone the qusrwrk subsystem to make it start with a specific user, but what I cannot figure out is the mechanism to open the connection in the specific subsystem.
I guess there should be a property at connection level to say subsystem=MySubsystem.
But unfortunatly I haven't found that property.
Any hint would be appreciated.
Flavio
Let the system take care of the subsystem the job database server job is started in.
You should just focus on the application (which is what IBM i excels in).
If need be, you can tweak subsystem parameters for QUSRWRK to improve performance by allocating memory, etc.
The system uses a pool of prestarted jobs as described in the FAQ: When I do WRKACTJOB, why is the host server job running under QUSER instead of the profile specified on the AS400 object?
To improve performance, the host server jobs are prestarted jobs running under QUSER. When the Toolbox connects to a host server job in order to perform an API call, run a command, etc, a request is sent from the Toolbox to an available prestarted job. This request includes the user profile specified on the AS400 object that represents the connection. The host server job receives the request and swaps to the specified user profile before it runs the request. The host server itself originally runs under the QUSER profile, so output from the WRKACTJOB command will show the job as being owned by QUSER. However, the job is in fact running under the profile specified on the request. To determine what profile is being used for any given host server job, you can do one of three things:
1. Display the job log for that job and find the message indicating which user profile is used as a result of the swap.
2. Work with the job and display job status attributes to view the current user profile.
3. Use Navigator for i to view all of the server jobs, which will list the current user of each job. You can also use Navigator for i to look at the server jobs being used by a particular user.