Pre-pending and appending to a shell variable - bash

My goal is to load an external tables log file into a CLOB column in an oracle database. I've been having issues with the max size you can insert at once but I am able to insert the whole file if I to_clob each line of the log file, concatenate and then insert them (as far as I'm aware this seems to be the quickest and easiest way?):
insert into clob_insert_test values (to_clob('hfsdjhfjsdhfjksd')||chr(10)||to_clob('jhfklsdjfklsdjklfjdsjlk'));
My question is:
I'm reading the file into a shell variable as below so what I need to do is pre-pend to_clob(' to the beginning of each line of the variable and then append ')||chr(10)|| and remove the last ||chr(10)|| from the variable to finish. I can then use that variable in the SQL insert statement for the clob column. Is there a way I can directly do this on the variable rather than modifying the log file before reading it in?
log_content=$(<"$log_file")
Edit:
Sorry I don't think I was clear. Given the example log file I would expect the following variable contents.
Input file:
LOG file opened at 05/05/15 15:12:24
Field Definitions for table ext_loading
Record format DELIMITED BY NEWLINE
Variable contents:
to_clob('LOG file opened at 05/05/15 15:12:24')||char(10)||to_clob('Field Definitions for table ext_loading')||char(10)||to_clob('Record format DELIMITED BY NEWLINE')

I assume you have a file like:
this is me||chr(10)||adfasdf
asdas||chr(10)||asdfasdfasdas
And you want it to become something like:
to_clob('this is meadfasdf')||chr(10)||
to_clob('asdasasdfasdfasdas')||chr(10)||
If so, you can use sed like this:
sed -e "s/||chr(10)||//" -e "s/^/to_clob('/" -e "s/$/')||chr(10)||/" file
That is:
remove ||chr(10)|| once from each line.
add to_clob(' to the begining of each line.
add ')||chr(10)|| to the end of each line.
And to store it in a variable:
log_content=$(sed -e "s/||chr(10)||//" -e "s/^/to_clob('/" -e "s/$/')||chr(10)||/" "$log_file")
Update
To match what you really need, you can also do this:
line=$(sed -e "/./s/^/to_clob('/" -e "/./s/$/')||chr(10)||/" "$log_file")
Then the output is:
$ echo $line # note, without quotes to have all of it together!
to_clob('LOG file opened at 05/05/15 15:12:24')||chr(10)|| to_clob('Field Definitions for table ext_loading')||chr(10)|| to_clob('Record format DELIMITED BY NEWLINE')||chr(10)||
And remove the last ||chr(10)|| with:
$ echo $line | sed 's/||chr(10)||$//'
to_clob('LOG file opened at 05/05/15 15:12:24')||chr(10)|| to_clob('Field Definitions for table ext_loading')||chr(10)|| to_clob('Record format DELIMITED BY NEWLINE')

Related

Replacing placeholder with multiple lines

I have a template that looks like:
<mydata>
<tag1>
<tag2> etc.
DATAHERE
</mydata>
I want to run a query on a DB and fetch a number of records and place them one below the other in the file at the place where there is the string DATAHERE.
My records are already fetched in mydata.txt. How do i replace the single line of DATAHERE ? I do not want to hardcode the number of lines in the template to skip. DATAHERE is my only marker.
This might work for you (GNU sed):
sed -e '/DATAHERE/{r dataFile' -e 'd}' file
Focus on the line DATAHERE and read the file dataFile then delete the current line.

Populate a value in a particular column in csv

I have a folder where there are 50 excel sheets in CSV format. I have to populate a particular value say "XYZ" in the column I of all the sheets in that folder.
I am new to unix and have looked for a couple of pages Here and Here . Can anyone please provide me the sample script to begin with?
For example :
Let's say column C in this case:
A B C
ASFD 2535
BDFG 64486
DFGC 336846
I want to update column C to value "XYZ".
Thanks.
I would export those files into csv format
- with semikolon as field separator
- eventually by leaving out column descriptions (otherwise see comment below)
Then the following combination of SHELL and SED script could more or less do already the trick
#! /bin/sh
for i in *.csv
do
sed -i -e "s/$/;XZY/" $i
done
-i means to edit the file in place, here you could append the value to all lines
-e specifies the regular expresssion for substitution
You might want to use a similar script like this, to rename "XYZ" to "C" only in the 1st line if the csv files should contain also column descriptions.

Delete set of lines from a file in AIX by passing variables

I am trying to delete a set of lines from a file by passing variables.
Below is my file :
$ cat checking.txt
Starting1
DELETE /*+NESTED_TABLE_SET_REFS+*/ FROM tables1
Ending1
Starting2
update table
set col1=2
where val2=685
Ending2
Starting3
update table
set col1=1
where val1=44
Ending3
so in above files I need to delete lines from 1st line to 4th line.
I used below command and it was working fine.
sed '1,4d' checking.txt
Now I gave variable a a value, like a=4
echo $a
4
Now I tried the sed command like
sed "1,${a}d" checking.txt
sed: 0602-404 Function 1, 4d cannot be parsed.
Can someone please tell me how to pass variable here?
Thanks in Advance
One way you could attempt this problem is in the following way. Assuming you want to delete lines between start and stop, you could write
awk '(NR<start) || (stop<NR)' start=1 stop=4 file
The way you are proposing it to work also works,
start=1
stop=4
sed "${start},${stop}d" file
The reason why it failed in your case is because the variable a seems to have blanks in front of it. You notice this from the errormessage and the blanks space in front of the 4.

Create CSV from specific columns in another CSV using shell scripting

I have a CSV file with several thousand lines, and I need to take some of the columns in that file to create another CSV file to use for import to a database.
I'm not in shape with shell scripting anymore, is there anyone who can help with pointing me in the correct direction?
I have a bash script to read the source file but when I try to print the columns I want to a new file it just doesn't work.
while IFS=, read symbol tr_ven tr_date sec_type sec_name name
do
echo "$name,$name,$symbol" >> output.csv
done < test.csv
Above is the code I have. Out of the 6 columns in the original file, I want to build a CSV with "column6, column6, collumn1"
The test CSV file is like this:
Symbol,Trading Venue,Trading Date,Security Type,Security Name,Company Name
AAAIF,Grey Market,22/01/2015,Fund,,Alternative Investment Trust
AAALF,Grey Market,22/01/2015,Ordinary Shares,,Aareal Bank AG
AAARF,Grey Market,22/01/2015,Ordinary Shares,,Aluar Aluminio Argentino S.A.I.C.
What am I doing wrong with my script? Or, is there an easier - and faster - way of doing this?
Edit
These are the real headers:
Symbol,US Trading Venue,Trading Date,OTC Tier,Caveat Emptor,Security Type,Security Class,Security Name,REG_SHO,Rule_3210,Country of Domicile,Company Name
I'm trying to get the last column, which is number 12, but it always comes up empty.
The snippet looks and works fine to me, maybe you have some weird characters in the file or it is coming from a DOS environment (use dos2unix to "clean" it!). Also, you can make use of read -r to prevent strange behaviours with backslashes.
But let's see how can awk solve this even faster:
awk 'BEGIN{FS=OFS=","} {print $6,$6,$1}' test.csv >> output.csv
Explanation
BEGIN{FS=OFS=","} this sets the input and output field separators to the comma. Alternatively, you can say -F=",", -F, or pass it as a variable with -v FS=",". The same applies for OFS.
{print $6,$6,$1} prints the 6th field twice and then the 1st one. Note that using print, every comma-separated parameter that you give will be printed with the OFS that was previously set. Here, with a comma.

Running a number of hive queries and writing output to file

I'm trying to make use of the DESCRIBE function via Hive to output the column descriptions of each of the tables out to individual files. I've discovered the -f option so I can just read from a file and write the output back out:
hive -f nameOfSqlQueryFile.sql > out.txt
However, if I open the output file, it throws all the descriptions back to back and it's unclear where one description starts for a table and where it ends.
So, I've tried making a batch file that uses -e to describe each of the tables individually and output to a file:
#!/bin/bash
nameArr=( $(hive -e 'show tables;') )
count=0
for i in "${nameArr[#]}"
do
echo 'Working on table('$count'): '$i
hive -e 'describe '$i > $i'_.txt';
count=$(($count+1))
done
However, because this needs to reconnect for each query, it's remarkably slow, taking hours to process several hundred queries.
Does anyone have an idea of how else I might run each of these DESCRIBE functions, and ideally output to separate files?
You can probably use one of these, depending on how you process the output:
Just use the OK line as a separator and search for it using a script.
Use DESCRIBE EXTENDED which adds a line at the end with info on the table, including its location, which can be used to extract the table name (using sed, for example)
If you're just using the output file as a manual reference, insert a SQL statement that prints a separator of your choice between each table, e.g.:
DESCRIBE table;
SELECT '-----------------' FROM table;

Resources