I have a below table
1 Data 2021-02-04
2 Data Two 2021-02-05
If I column -t -s '' I get
1 Data 2021-02-04
2 Data Two 2021-02-05
Any way I can format this as:
1 Data 2021-02-04
2 Data Two 2021-02-05
Or format by column number 3
You have to first convert the two spaces separating columns in the original into a single character before feeding it to column. Something like:
$ sed 's/ /|/g' input.txt | column -t -s '|'
1 Data 2021-02-04
2 Data Two 2021-02-05
Use a different character that doesn't appear in the input if | is present in it.
Related
I have a file containing three TAB-separated columns. The 1st column is a number, the second is a sequence of 8 characters followed by 1-3 digits, the 3rd is the same as the 2nd column. Here's a minimum reproducible example:
1 abceefgh10 abceefgh22
1 abceefgh10 abceefgh9
1 abceefgh11 abceefgh10
1 abceefgh13 abceefgh11
1 abceefgh14 abceefgh13
1 abceefgh15 abceefgh14
1 abceefgh17 abceefgh16
-1 abceefgh18 abceefgh17
1 abceefgh19 abceefgh18
-1 abceefgh1 abceefgh2
-1 abceefgh20 abceefgh12
1 abceefgh21 abceefgh19
1 abceefgh22 abceefgh20
-1 abceefgh23 abceefgh21
1 abceefgh24 abceefgh24
1 abceefgh2 abceefgh1
1 abceefgh3 abceefgh3
1 abceefgh5 abceefgh5
1 abceefgh6 abceefgh25
1 abceefgh6 abceefgh6
1 abceefgh7 abceefgh7
-1 abceefgh8 abceefgh3
1 abceefgh9 abceefgh8
This example is what I get when I try to sort the columns with sort -gk2.9.
To the best of my knowledge I should expect to see the second column sorted from 1 to 24, and with increasing numerical value (i.e. 1,2,3,4,... and not 1,10,2,20,..., which would result if using -n).
If I cut the 2nd column and sort it with the same command (cut -f 2 ${file} | sort -gk1.9), I actually get the sorting that I want. Am I getting something obvious wrong?
Using --debug option you can see column selection does not work as expected:
1>abceefgh10>abceefgh9
^ no match for key
specifying separator in accordance with Nahuel's comment works (sort -t $'\t' --debug -gk2.9):
1>abceefgh10>abceefgh9
__
I'm working with fairly large tsv zip files whereby each file has 3 columns only. I would like to count the number of unique occurrences for a particular regex (which is contained in column 3) across all files.
How do I make sure the count number in the output removes any duplicates based on values contained in column 1?
Tried both of these, but not sure if they are correct:
zgrep -c ",80447," AU_AAID_201812*.tsv.gz | uniq -c
zgrep -c ",80447," AU_AAID_201812*.tsv.gz
I want to get the unique count number so that if:
Column 1/Row 1 = "xyz123" and Column 3/Row 1 = ",80447,"
Column 1/Row 2 = "xyz123" and Column 3/Row 2 = ",80447,"
Then my output would be still be "1".
Use cut to get just column1 and column3, use sort -u to remove duplicates, and then use wc -l to get the count:
zgrep ',80447,' AU_AAID_201812*.tsv.gz | cut -d, -f1,3 | sort -u | wc -l
I have 2 text files as below
A.txt (with 2 rows):
abc-1234
tik-3456
B.txt (with 4 rows)
123456
234567
987
12
I want to combine these 2 to get the below file in CSV format:
column-1 column-2
abc-1234 123456
tik-3456 234567
987
12
I am trying below command. However, not achieving the above result.
paste -d "," A.txt B.txt > C.csv
It is giving below output:
abc-1234
,123456
tik-3456,234567
,987
,12
Can anyone please let me know, what I am missing here?
In linux we have one utility that does one think very good. So:
paste merges files
column with -t creates tables
The following:
paste -d',' /tmp/1 /tmp/2 | column -t -N 'column-1,column-2' -s',' -o' '
outputs the desired result.
I'm working with a very large text file (4GB) and I want to make a smaller file with only the data I need in it. It is a tab deliminated file and there are row and column headers. I basically want to select a subset of the data that has a given column and/or row name.
colname_1 colname_2 colname_3 colname_4
row_1 1 2 3 5
row_2 4 6 9 1
row_3 2 3 4 2
I'm planning to have a file with a list of the columns I want.
colname_1 colname_3
I'm a newbie to bash scripting and I really don't know how to do this. I saw other examples, but they all new what column number they wanted in advance and I don't. Sorry if this is a repeat question, I tried to search.
I would want the result to be
colname_1 colname_3
row_1 1 3
row_2 2 9
row_3 2 4
Bash works best as "glue" between standard command-line utilities. You can write loops which read each line in a massive file, but it's painfully slow because bash is not optimized for speed. So let's see how to use a few standard utilities -- grep, tr, cut and paste -- to achieve this goal.
For simplicity, let's put the desired column headings into a file, one per line. (You can always convert a tab-separated line of column headings to this format; we're going to do just that with the data file's column headings. But one thing at a time.)
$ printf '%s\n' colname_{1,3} > columns
$ cat columns
colname_1
colname_2
An important feature of the printf command-line utility is that it repeats its format until it runs out of arguments.
Now, we want to know which column in the data file each of these column headings corresponds to. We could try to write this as a loop in awk or even in bash, but if we convert the header line of the data file into a file with one header per line, we can use grep to tell us, by using the -n option (which prefixes the output with the line number of the match).
Since the column headers are tab-separated, we can get turn them into separate lines just by converting tabs to newlines using tr:
$ head -n1 giga.dat | tr '\t' '\n'
colname_1
colname_2
colname_3
colname_4
Note the blank line at the beginning. That's important, because colname_1 actually corresponds to column 2, since the row headers are in column 1.
So let's look up the column names. Here, we will use several grep options:
-F The pattern argument consists of several patterns, one per line, which are interpreted as ordinary strings instead of regexes.
-x The pattern must match the complete line.
-n The output should be prefixed by the line number of the match.
If we have Gnu grep, we could also use -f columns to read the patterns from the file named columns. Or if we're using bash, we could use the bashism "$(<columns)" to insert the contents of the file as a single argument to grep. But for now, we'll stay Posix compliant:
$ head -n1 giga.dat | tr '\t' '\n' | grep -Fxn "$(cat columns)"
2:colname_1
4:colname_3
OK, that's pretty close. We just need to get rid of everything other than the line number; comma-separate the numbers, and put a 1 at the beginning.
$ { echo 1
> grep -Fxn "$(<columns)" < <(head -n1 giga.dat | tr '\t' '\n')
> } | cut -f1 -d: | paste -sd,
1,2,4
cut -f1 Select field 1. The argument could be a comma-separated list, as in cut -f1,2,4.
cut -d: Use : instead of tab as a field separator ("delimiter")
paste -s Concatenate the lines of a single file instead of corresponding lines of several files
paste -d, Use a comma instead of tab as a field separator.
So now we have the argument we need to pass to cut in order to select the desired columns:
$ cut -f"$({ echo 1
> head -n1 giga.dat | tr '\t' '\n' | grep -Fxn -f columns
> } | cut -f1 -d: | paste -sd,)" giga.dat
colname_1 colname_3
row_1 1 3
row_2 4 9
row_3 2 4
You can actually do this by keeping track of the array indexes for the columns that match the column names in your file containing the column list. After you have found the array indexes in the data file for the column names in your column list file, you simply read your data file (beginning at the second line) and output the row_label plus the data for the columns at the array index you determined in matching the column list file to the original columns.
There are probably several ways to approach this and the following assumes the data in each column does not contain any whitespace. The use of arrays presumes bash (or other advanced shell supporting arrays) and not POSIX shell.
The script takes two file names as input. The first is your original data file. The second is your column list file. An approach could be:
#!/bin/bash
declare -a cols ## array holding original columns from original data file
declare -a csel ## array holding columns to select (from file 2)
declare -a cpos ## array holding array indexes of matching columns
cols=( $(head -n 1 "$1") ) ## fill cols from 1st line of data file
csel=( $(< "$2") ) ## read select columns from file 2
## fill column position array
for ((i = 0; i < ${#csel[#]}; i++)); do
for ((j = 0; j < ${#cols[#]}; j++)); do
[ "${csel[i]}" = "${cols[j]}" ] && cpos+=( $j )
done
done
printf " "
for ((i = 0; i < ${#csel[#]}; i++)); do ## output header row
printf " %s" "${csel[i]}"
done
printf "\n" ## output newline
unset cols ## unset cols to reuse in reading lines below
while read -r line; do ## read each data line in data file
cols=( $line ) ## separate into cols array
printf "%s" "${cols[0]}" ## output row label
for ((j = 0; j < ${#cpos[#]}; j++)); do
[ "$j" -eq "0" ] && { ## handle format for first column
printf "%5s" "${cols[$((${cpos[j]}+1))]}"
continue
} ## output remaining columns
printf "%13s" "${cols[$((${cpos[j]}+1))]}"
done
printf "\n"
done < <( tail -n+2 "$1" )
Using your example data as follows:
Data File
$ cat dat/col+data.txt
colname_1 colname_2 colname_3 colname_4
row_1 1 2 3 5
row_2 4 6 9 1
row_3 2 3 4 2
Column Select File
$ cat dat/col.txt
colname_1 colname_3
Example Use/Output
$ bash colnum.sh dat/col+data.txt dat/col.txt
colname_1 colname_3
row_1 1 3
row_2 4 9
row_3 2 4
Give it a try and let me know if you have any questions. Note, bash isn't known for its blinding speed handling large files, but as long as the column list isn't horrendously long, the script should be reasonably fast.
I have a text file like below.
1 1223 abc
2 4234 weroi
0 3234 omsder
1 1111 abc
2 6666 weroi
I want to have unique values for the column 3. So I want to have the below file.
1 1223 abc
2 4234 weroi
0 3234 omsder
Can I do this using some basic commands in Linux? without using Java or something.
You could do this with some awk scripting. Here is a piece of code I came up with to address your problem :
awk 'BEGIN {col=3; sep=" "; forbidden=sep} {if (match(forbidden, sep $col sep) == 0) {forbidden=forbidden $col sep; print $0}}' input.file
The BEGIN keyword declares the forbidden string, which is used to monitor the 3rd column values. Then, the match keyword check if the 3rd column of the current line contains any forbidden value. If not, it adds the content of the column to the forbidden list and print the whole line.
Here, sep=" " instantiate the separator. We use sep between each forbidden value in order to avoid words created by putting several values next to one another. For instance :
1 1111 ta
2 2222 to
3 3333 t
4 4444 tato
In this case, without a separator, t and tato would be considered a forbidden value. We use " " as a separator as it is used by default to separate each column, thus a column cannot include a space in its name.
Note that if you want to change the number of the column in which you need to remove duplicate, just adapt col=3 with the number of the column you need (0 for the whole line, 1 for the first column, 2 for the second, ...)