I'm trying to tail a log file and format the output into columns. This gives me what I want without tail:
cat /var/log/test.log | column -t -s "|"
How can I pipe the output of tail -f var/log/test.log into column?
EDIT: Here's an excerpt from the file. I'm manually adding the first line of the file so it could be used as the column headers, but I could format it differently if necessary.
timestamp|type|uri|referer|user_id|link|message
Feb 5 23:58:29 181d5d6339bd drupal_overlake: 1612569509|geocoder|https://overlake.lando/admin/config/development/configuration/config-split/add|https://overlake.lando/admin/config/development/configuration/config-split/add|0||Could not execute query "https://maps.googleapis.com/maps/api/geocode/json?address=L-054%2C%20US&language=®ion=US".
Feb 5 23:58:29 181d5d6339bd drupal_overlake: 1612569509|geocoder|https://overlake.lando/admin/config/development/configuration/config-split/add|https://overlake.lando/admin/config/development/configuration/config-split/add|0||Unable to geocode 'L-054, US'.
You can't do it with the -f option to tail. column can't produce any output until it receives all its input, since it needs to calculate the number of rows and columns by examining all the input. tail -f never stops writing, so column doesn't know when it's done.
You can use
tail -n 100 test.log | column -t -s "|"
to format the last 100 lines of the log.
Related
I know how to tail a text file with a specific number of lines,
tail -n50 /this/is/my.log
However, how do I make that line count a variable?
Let's say I have a large log file which is appended to daily by some program, all lines in the log file start with a datetime in this format:
Day Mon YY HH:MM:SS
Every day I want to output the tail of the log file but only for the previous days records. Let's say this output runs just after midnight, I'm not worried about the tail spilling over into the next day.
I just want to be able to work out how many rows to tail, based on the first occurrence of yesterdays date...
Is that possible?
Answering the question of the title, for anyone who comes here that way, head and tail can both accept a code for how much of the file to exclude.
For tail, use -n +num for the line number num to start at
For head, use -n -num for the number of lines not to print
This is relevant to the actual question if you have remembered the number of lines from the previous time you did the command, and then used that number for tail -n +$prevlines to get the next portion of the partial log, regardless of how often the log is checked.
Answering the actual question, one way to print everything after a certain line that you can grep is to use the -A option with a ridiculous count. This may be more useful than the other answers here as you can get a number of days of results. So to get everything from yesterday and so-far today:
grep "^`date -d yesterday '+%d %b %y'`" -A1000000 log_file.txt
You can combine 2 greps to print between 2 date ranges.
Note that this relies on the date actually occurring in the log file. It has the weakness that if no events were logged on a particular day used as the range marker, then it will fail to find anything.
To resolve that you could inject dummy records for the start and end dates and sort the file before grepping. This is probably overkill, though, and the sort may be expensive, so I won't example it.
I don't think tail has any functionality like this.
You could work out the beginning and ending line numbers using awk, but if you just want to exact those lines from the log file, the simplest way is probably to use grep combined with date to do it. Matching yesterday's date at beginning of line should work:
grep "^`date -d yesterday '+%d %b %y'`" < log_file.txt
You may need to adjust the date format to match exactly what you've got in the log file.
You can do it without tail, just grep rows with previous date:
cat my.log | grep "$( date -d "yesterday 13:00" '+%d %m %Y')"
And if you need line count you can add
| wc -l
I worked this out through trial and error by getting the line numbers for the first line containing the date and the total lines, as follows:
lines=$(wc -l < myfile.log)
start=$(cat myfile.log | grep -no $datestring | head -n1 | cut -f1 -d:)
n=$((lines-start))
and then a tail, based on that:
tail -n$n myfile.log
I have a (very) large csv file almost around 70GB which I am trying to sort using the sort command. As much as I am trying, the output is not being written to file. Here is what I tried
sort -T /data/data/.tmp -t "," -k 38 /data/data/raw/KKR.csv > /data/data/raw/KKR_38.csv
sort -T /data/data/.tmp -t "," -k 38 /data/data/raw/KKR.csv -o /data/data/raw/KKR-38.csv
What happens is that the KKR_38.csv file is created and its size is the same as the KKR.csv file but there is nothing inside it. When I do
head -n 100 /data/data/raw/KKR_38.csv
It prints out 100 empty lines.
If you sort, it is quite normal the empty lines come first. Try this:
tail -100 /data/data/raw/KKR_38.csv
You can use the following commands if you want to not take into account the empty lines:
cat -s /data/data/raw/KKR_38.csv | less #to squeeze the successive empty lines to only one
or if you want to remove them:
sed '/^$/d' /data/data/raw/KKR_38.csv | less
You can redirect the output of those commands to create another file without the empty line (watch out for the space on your file system).
I have two .txt files both 42 lines each (42nd line is just a blank space).
One file is called date.txt, and each line has a format like:
2017-03-16 10:45:32.175 UTC
The second file is called version, and each line has a format like:
1.2.3.10
Is there a way to merge the two files together so that the date is appended to the version number (separated by a space). So the 1st line of each file is merged together, then the second line, third line, etc...
So it would look like:
1.2.3.10 2017-03-16 10:45:32.175 UTC
After that, is it possible to reorder the new file by the date and time? (Going from the oldest date to the latest/current one).
The end file should still be 42 lines long.
Thanks!
Use paste:
paste file1.txt file2.txt -d' ' > result.txt
-d is used to set the delimiter.
You can then attempt to sort by second and third column using sort:
sort -k2,3 result.txt > sorted.txt
-k is used to select columns to sort by.
But note that this doesn't parse the date and time, only sorts them as strings.
In general:
paste file1.txt file2.txt | sort -k2,3 > result.txt
How to get the first few lines from a gziped file ?
I tried zcat, but its throwing an error
zcat CONN.20111109.0057.gz|head
CONN.20111109.0057.gz.Z: A file or directory in the path name does not exist.
zcat(1) can be supplied by either compress(1) or by gzip(1). On your system, it appears to be compress(1) -- it is looking for a file with a .Z extension.
Switch to gzip -cd in place of zcat and your command should work fine:
gzip -cd CONN.20111109.0057.gz | head
Explanation
-c --stdout --to-stdout
Write output on standard output; keep original files unchanged. If there are several input files, the output consists of a sequence of independently compressed members. To obtain better compression, concatenate all input files before compressing
them.
-d --decompress --uncompress
Decompress.
On some systems (e.g., Mac), you need to use gzcat.
On a mac you need to use the < with zcat:
zcat < CONN.20111109.0057.gz|head
If a continuous range of lines needs be, one option might be:
gunzip -c file.gz | sed -n '5,10p;11q' > subFile
where the lines between 5th and 10th lines (both inclusive) of file.gz are extracted into a new subFile. For sed options, refer to the manual.
If every, say, 5th line is required:
gunzip -c file.gz | sed -n '1~5p;6q' > subFile
which extracts the 1st line and jumps over 4 lines and picks the 5th line and so on.
If you want to use zcat, this will show the first 10 rows
zcat your_filename.gz | head
Let's say you want the 16 first row
zcat your_filename.gz | head -n 16
This awk snippet will let you show not only the first few lines - but a range you can specify. It will also add line numbers which i needed for debugging an error message pointing to a certain line way down in a gzipped file.
gunzip -c file.gz | awk -v from=10 -v to=20 'NR>=from { print NR,$0; if (NR>=to) exit 1}'
Here is the awk snippet used in the one liner above. In awk NR is a built-in variable (Number of records found so far) which usually is equivalent to a line number. the from and to variable are picked up from the command line via the -v options.
NR>=from {
print NR,$0;
if (NR>=to)
exit 1
}
I have a question regarding using the head and tail commands in UNIX to truncate a specific transaction from a huge transaction log.
head -X <<<filename>>> | tail -Y > <<<Truncatedfile>>>
where X is the number of lines I want from the beginning of the file and Y is the number of lines I want from the bottom of the file.
How can I modify this code to have the truncated file with just the transactions for a unique transaction ID? For example:-
The file contains transaction logs for n number of transaction IDs in a sequence. So, if I only need logs extracted for just 1 single transaction ID how to modify the above code?
You wouldn't modify the above code, instead you'd
grep -w transactionid filename
Assuming that the transactionid appears as a separate word (-w)
Edit You can include some context lines (this includes 10 lines after the match:)
grep -w -A 10 transactionid filename
Alternatively,
grep -vw transactionid filename
Simple hides all lines NOT containing the transaction id. This close to equivalent to doing sed -e '/transactionid/!d'.
To print lines 5-12
sed -n '5,12p' filename