I am running some kismet captures, and I need to continuously parse the outputted PCAP files, so in order to do this I need Kismet to save the file, and start a new one periodically (I use inotify-tools to detect newly created files).
But the problem is I do not find a way for kismet to do it. In man pages i found that -m option allows to limit the file size by packet size, so I ran it like this:
sudo kismet -c wlan0 -m 10
But that did not create multiple files, it carried on just putting all traffic to one file.
Any other ways to somehow make kismet break output to different files? I don't really care about what criteria is used (time, packet count, file size.. I'll take anything)
Thanks!
I think that you can modify it in the kismet.conf file.
There is an option that says
writeinterval=300
It means that every 300 seconds the pcap file will be saved. It will make a new file every 300 seconds.If you want a shorter time you can change it.
Hope it helps
Related
I am trying to find a way to make rsyslog write logs into multiple files simultaneous, that is received from the same host. For example write to two files, and switch every log line.
I have one host that sends insane amounts of data (600gb a day), and i want write this logs to multiple files instead of one.
This is because of Splunk that read this file for indexing purposes, cant utilize multiple pipelines (aka. multiple CPU threads) on the same file, and this causes an bottleneck.
I was thinking about making rsyslog writing a new file every second, but this does not sound optimal.
Any suggestion is much appreciated :)
I have not been able to find any documentation that describes this ability in rsyslog.
The solution for this is to use part of the subseconds to split the file randomly (using last 2 digits, nano seconds) to 100 files at the same time.
template(name="template_nano_file_format" type="list") {
property(name="$.path")
property(name="fromhost-ip")
constant(value="/")
property(name="$year")
property(name="$month")
property(name="$day")
constant(value="-")
property(name="$hour")
constant(value="-")
property(name="timegenerated" dateFormat="subseconds" regex.expression="..$" regex.type="ERE")
constant(value=".log")
}
I am new to Praat and wondering, if someone can help me to find out, how I can cut all my sound files with a script or anything.
I have like 100 sound files I need for my research. They all have a different length, some are 1 min and others are 3 min long.
I would like to have only the first 22 sec from each sound file.
Thanks in advance!
Kind regards
Olga
The first step is to construct a script that extracts the initial 22 seconds of some specific sound object that is already open. In general, the easiest way to at least start a script is to do a thing manually once, and after you've done that, in a Praat script window, copy the command history (with ctrl-h) to see what the underlying commands are. The manual approach is to look for "Extract part" under "Convert", which corresponds to the command
Extract part: 0, 22, "rectangular", 1, "no"
There is also a command to save a file as a wav file, so you would add that to the core of the script.
Then you need to add a loop that does this a number of times, to different files. You will (probably) need a file with wav file names, and some system for naming the output files, for example if you have "input1.wav", you might want to call the cut-down version "output1.wav". This implies some computation of the output file name based on the input file name, so you need to get familiar with how string manipulation works in Praat.
If you have that much sorted out, then the basic logic is
get next input file name file
compute output name
open the input file
extract from that file
save the extracted file
remove the extract
remove the original
loop until no more files
I would plan on spending a lot of time trying to understand simple things like string variables, or object selection. I left out explicitly selecting objects since it is not necessarily required, but every command works on "the selected object" and it's easy to lose track of what is selected.
Another common approach is to beg a colleague to write it for you.
I'm currently running a ruby script which logs its HTTP traffic to stdout. Since I wanted the logs to be persistent, I redirected the output to a log file with ruby ruby_script.rb >> /var/log/ruby_script.log. However, the logs are now getting very large so I wanted to implement logrotate using the following:
"/var/log/ruby_script.log" {
missingok
daily
rotate 10
dateext
}
However, after running logrotate --force -v ruby_script where "ruby_script" is the name of the logrotate.d configuration file, no new file is created for the script to write to, and it writes to the rotated file instead. I'm guessing this behavior happens because the file descriptor that is passed by >> sticks to the file regardless of moving it, and is unrelated to the filename after the first call. Thus, my question is, what is the correct way to achieve the functionality I'm looking for?
Take a look at option copytruncate.
From man logrotate:
copytruncate: Truncate the original log file to zero size in place after creating a copy, instead of moving the old log file and
optionally creating a new one. It can be used when some program cannot be told to close its logfile and thus might
continue writing (appending) to the previous log file forever. Note that there is a very small time slice between
copying the file and truncating it, so some logging data might be lost. When this option is used, the create option
will have no effect, as the old log file stays in place.
I am trying to write a video ruby transformer script (using ffmpeg) that depends on mov files being ftped to a server.
The problem I've run into is that when a large file is uploaded by a user, the watch script (using rb-inotify) attempts to execute (and run the transcoder) before the mov is completely uploaded.
I'm a complete noob. But I'm trying to discover if there is a way for me to be able to ensure my watch script doesn't run until the file(s) is/are completely uploaded.
My watch script is here:
watch_me = INotify::Notifier.new
watch_me.watch("/directory_to_my/videos", :close_write) do |directories|
load '/directory_to_my/videos/.transcoder.rb'
end
watch_me.run
Thank you for any help you can provide.
Just relying on inotify(7) to tell you when a file has been updated isn't a great fit for telling when an upload is 'complete' -- an FTP session might time out and be re-started, for example, allowing a user to upload a file in chunks over several days as connectivity is cheap or reliable or available. inotify(7) only ever sees file open, close, rename, and access, but never the higher-level event "I'm done modifying this file", as the user would understand it.
There are two mechanisms I can think of: one is to have uploads go initially into one directory and ask the user to move the file into another directory when the upload is complete. The other creates some file meta-data on the client and uses that to "know" when the upload is complete.
Move completed files manually
If your users upload into the directory ftp/incoming/temporary/, they can upload the file in as many connections is required. Once the file is "complete", they can rename the file (rename ftp/incoming/temporary/hello.mov ftp/incoming/complete/hello.mov) and your rb-inotify interface looks for file renames in the ftp/incoming/complete/ directory, and starts the ffmpeg(1) command.
Generate metadata
For a transfer to be "complete", you're really looking for two things:
The file is the same size on both systems.
The file is identical on both systems.
Since "identical" is otherwise difficult to check, most people content themselves with checking if the contents of the file, when run through a cryptographic hash function such as MD5 or SHA-1 (or better, SHA-224, SHA-256, SHA-384, or SHA-512) functions. MD5 is quite fine if you're guarding against incomplete transmission but if you intend on using the output of the function for other means, using a stronger function would be wise.
MD5 is really tempting though, since tools to create and validate MD5 hashes are very widespread: md5sum(1) on most Linux systems, md5(1) on most BSD systems (including OS X).
$ md5sum /etc/passwd
c271aa0e11f560af419557ef49a27ac8 /etc/passwd
$ md5sum /etc/passwd > /tmp/sums
$ md5sum -c /tmp/sums
/etc/passwd: OK
The md5sum -c command asks the md5sum(1) program to check the file of hashes and filenames for correctness. It looks a little silly when used on just a single file, but when you've got dozens or hundreds of files, it's nice to let the software do the checking for you. For example: http://releases.mozilla.org/pub/mozilla.org/firefox/releases/3.0.19-real-real/MD5SUMS -- Mozilla has published such files with 860 entries -- checking them by hand would get tiring.
Because checking hashes can take a long time (five minutes on my system to check a high-definition hour-long video that wasn't recently used), it'd be a good idea to only check the hashes when the filesizes match. Modify your upload tool to send along some metadata about how long the file is and what its cryptographic hash is. When your rb-inotify script sees file close requests, check the file size, and if the sizes match, check the cryptographic hash. If the hashes match, then start your ffmpeg(1) command.
It seems easier to upload the file to a temporal directory on the server and move it to the location your script is watching once the transfer is completed.
I got an application which is polling on a folder continuously. Once any file is ftp to the folder, the application has to move this file to some other folder for processing.
Here, we don't have any option to verify whether ftp is complete or not.
One command "lsof" is suggested in the technical forums. It got a file description column which gives the file status.
Since, this is a free bsd command and not present in old versions of linux, I want to clarify the usage of this command.
Can you guys tell us your experience in file verification and is there any other alternative solution available?
Also, is there any risk in using this utility?
Appreciate your help in advance.
Thanks,
Mathew Liju
We've done this before in a number of different ways.
Method one:
If you can control the process sending the files, have it send the file itself followed by a sentinel file. For example, send the real file "contracts.doc" followed by a one-byte "contracts.doc.sentinel".
Then have your listener process watch out for the sentinel files. When one of them is created, you should process the equivalent data file, then delete both.
Any data file that's more than a day old and doesn't have a corresponding sentinel file, get rid of it - it was a failed transmission.
Method two:
Keep an eye on the files themselves (specifically the last modification date/time). Only process files whose modification time is more than N minutes in the past. That increases the latency of processing the files but you can usually be certain that, if a file hasn't been written to in five minutes (for example), it's done.
Conclusion:
Both those methods have been used by us successfully in the past. I prefer the first but we had to use the second one once when we were not allowed to change the process sending the files.
The advantage of the first one is that you know the file is ready when the sentinel file appears. With both lsof (I'm assuming you're treating files that aren't open by any process as ready for processing) and the timestamps, it's possible that the FTP crashed in the middle and you may be processing half a file.
There are normally three approaches to this sort of problem.
providing a signal file so that when your file is transferred, an additional file is sent to mark that transfer is complete
add an entry to a log file within that directory to indicate a transfer is complete (this really only works if you have a single peer updating the directory, to avoid concurrency issues)
parsing the file to determine completeness. e.g. does the file start with a length field, or is it obviously incomplete ? e.g. parsing an incomplete XML file will result in a parse error due to the lack of an end element. Depending on your file's size and format, this can be trivial, or can be very time-consuming.
lsof would possibly be an option, although you've identified your Linux portability issue. If you use this, note the -F option, which formats the output suitable for processing by other programs, rather than being human-readable.
EDIT: Pax identified a fourth (!) method I'd forgotten - using the fact that the timestamp of the file hasn't updated in some time.
There is a fifth method. You can also check if the FTP Session is still active. This will work if every peer has it's own ftp user account. As long as the user is not logged off from FTP, assume the files are not complete.