I have a requirement to check all text fields in a Database schema for any Illegal XML characters and replace them with a predefined set of acceptable values. This is to form part of a Data transformation rule, than can be called from other functions. So this function could be asked to called over a billion times on our dataset, so I need it to operate really efficiently.
i.e. & = AND ,
' = APOS
An example of what needs to be achieved by the function should be:
Update sometable set somefield = functioncall('f&re'd');
should result in
somefield having the value of ' fANDreAPOSd'
This is to carried out by a generic type PL/SQL function that takes an input of a text field and iterate through that field and replace all illegal values.
I have had a look at http://asktom.oracle.com/pls/asktom/f?p=100:11:0::NO::P11_QUESTION_ID:2612348048
http://decipherinfosys.wordpress.com/2007/11/27/removing-un-wanted-text-from-strings-in-oracle/
For some ideas, but I have my concerns over efficiency and the flexibility of these soltuions.
The way the client wants to handle the solution is to have a table configured to contain an illegal character and it's prefered replacement. The function then uses the values selected from this table to preform the replacements.
well, not exactly what you want, but consider this:
create type xmltest is object (s clob);
select XMLTYPE.createXml(xmltest('a& and ''')) from dual;
XMLTYPE.CREATEXML(XMLTEST('A&'''))
-------------------------------------------------------------------------------------
<XMLTEST>
<S>a& '</S>
</XMLTEST>
However the list of predefined of XML entities is quite small, so there wouldn't be an issue replacing them with replace
If this is because of XML I'll support hal9000 - you should let Oracle do that for you. E.g. XML functions do it automatically:
SQLPLUS> set define off
SQLPLUS> select xmlelement("e", 'foo & bar <> bar "hemingway''s"') from dual;
XMLELEMENT("E",'FOO&BAR<>BAR"HEMINGWAY''S"')
--------------------------------------------------------------------------------
<e>foo & bar <> bar "hemingway's"</e>
Related
I wasn't sure what my title should be, feel free to edit my post and change it if you can come up with something better.
There aren't many resources available on the use of Foxpro and what I'm trying to do is understand what is going on.
lldisdead=.t.
Select .f. as chkbox, * from a_counties ;
order by cn_area, cn_desc ;
into dbf (StrTmpFile1)
scan while !EOF()
IF ChkBox
selected_some_cnty = .t
endif
endscan
Here is my understanding:
Do the following as long as you are not in the last record of the
table:
IF ChkBox
Set selected_some_cnty equal to .t
Stop Check next record
Keep doing this until you are out of records.
What does IF CHKBOX mean?
Is it saying if the column CHKBOX is not null, do the following,
otherwise, do nothing.
Edit: Added additional code
If chkBox
in VFP, means:
if (chkBox)
also in all other well known languages, like C, C++, C#, Java, Go, Dart, Ruby, ... you name it - some languages parentheses are mandatory and some not. It simply mean "if chkBox is true". Sometimes you would see it written as:
If chkBox = .T.
like:
If chkBox == true
as in other languages, but it is more verbose than needed, and seasoned developers do not write it like that (after all writing like "if true is true" is awkward, simply "if true" is fine).
This is explained with comments placed in code:
* Initialize a memory variable named lldisdead as .t. (true)
lldisdead=.t.
* Select some fields into a table named m.StrTmpFile1
* StrTmpFile1 is a variable holding a string name of the table
* selecting all fields of a_counties table
* plus a boolean field named "chkBox" which is initially
* filled with .F. (false) value
Select .f. as chkbox, * from a_counties ;
order by cn_area, cn_desc ;
into dbf (StrTmpFile1)
* select's result table is table open in the current
* work area and by default located on first record.
* scanning the whole table
* with an unnecessary "while !EOF()" addition
* Default scope of scan is until EOF
scan while !EOF()
* Checking if chkBox field has a value of true
IF ChkBox
* if it has, than set "selected_some_cnty" memory variable to true
selected_some_cnty = .t
endif
endscan
Having said that, this part:
scan while !EOF()
IF ChkBox
selected_some_cnty = .t.
endif
endscan
could be written as:
scan
IF ChkBox
selected_some_cnty = .t
endif
endscan
further:
LOCATE FOR ChkBox
selected_some_cnty = !EOF()
However, since we know all chkBox values are .F., that piece of code is totally useless and could be deleted all together.
From the SQL query, the data is going into a physical table based on whatever the name "StrTmpFile1" variable is pointing to. Also note, the first column in this select statement is ".f. as ChkBox". So this is prepping EVERY RECORD in the query with a leading column that is ALWAYS False (hence .f.)
Select .f. as chkbox, * from a_counties ;
order by cn_area, cn_desc ;
into dbf (StrTmpFile1)
Now, I would suspect there is some other user interface action that is using this result table such as presenting in a grid in a form and allowing a checkbox on a column to let a user pick one or more entries to do something further.
After said selection (again, speculating intent), it is going through the loop to only find those records where the "ChkBox" COLUMN IN THE TABLE has been set to true and setting a flag as .t. that something WAS selected.
Overall, a very novice approach, but that is a different issue. A shortcut to getting the answer if a record as marked would be
select (the table)
Locate for ChkBox
selected_some_cnty = found()
Hope this helps, and if you need additional clarification, shoot a comment.
I have created one table named as test in KDB using following statement
test:([doc_id:`int$()];doc_displayid:`symbol$();doc_created_date:`date$();doc_aacess_date:`timestamp$();is_native_exist:`boolean$();file_size:`real$())
Now I want to insert a record in this
I've tried many ways like
insert['test; (1;`D_30;.z.d;.z.P;T;8.5)]
insert['test ([];`D_30;2018.8.8;2018.8.9T12:00:00.123;T;8.5)]
insert['test (1;`D_30;.z.d;2018.8.9T12:00:00.123;T;8.5)]
insert['test (1;`D_30;.z.d;2018.8.9T12:00:00.123;T;8.5)]
'test insert (1;`D_30;2018.8.8;2018.7.8T12:00:00.123;T;8.5)
But it doesn't work.
So please help me to solve this problem.
Thanks in advance.
Check the type of your input variables before insert in your test table.
Basically:
your 1 is of type long, not int;
2018.07.08T12:00:00.123 is of type datetime, not timestamp;
T does not exist, for boolean you should write 1b for true;
8.5 is of type float, not real.
When converting the above to the appropriate format, the insert works provided that you use backtick `test not 'test:
`test insert (1i;`D_30;2018.08.08;"p"$2018.07.08T12:00:00.123;1b;8.5e)
doc_id| doc_displayid doc_created_date doc_aacess_date is_native_exist file_size
------| --------------------------------------------------------------------------------------
1 | D_30 2018.08.08 2018.07.08D12:00:00.123000000 1 8.5
There are multiple issues with your insert statements, please check out the KDB datatypes wiki for examples
symbol should be defined with ` (uptick) i.e. `test , should not be 'test
doc_id is defined as int, so you need to pass an explicit i .e.g. 1i, 2i
There is no T boolean variable defined.
Real should be defined with an explicit e at the end. (8.5e)
timestamp is defined as dateDtimespan (D)
You can use either insert or upsert. upsert allows you to overwrite the record for already inserted record, however, the insert will ensure that you always enter a unique key, otherwise, it will throw an error.
upsert[`test; (1i;`D_30;.z.d;.z.P;0b;8.5e)]
insert[`test ;(2i;`D_30;2018.08.08;2018.08.09D12:00:00.123123123;0b;8.5e)]
doc_id| doc_displayid doc_created_date doc_aacess_date is_native_exist file_size
------| --------------------------------------------------------------------------------------
1 | D_30 2018.08.21 2018.08.21D07:33:40.630975000 0 8.5
2 | D_30 2018.08.08 2018.08.09D12:00:00.123123123 0 8.5
I am out of necessity using the SQLite3 shell tool to maintain a small database. I'm using -header -ascii flags, although this applies—as far as I can tell—to any of the output choices. I'm looking a way to avoid ambiguity over the type of any one value returned. Consider the following:
Create Table `things` (`number` Integer, `string` Text, `binary` Blob);
Insert Into `things` (`number`,`string`,`binary`) Values (4,'4',X'34');
Select * From `things`;
This returns (using caret notation):
number^_string^_binary^^4^_4^_4^^
As is evident, there is no way to infer the type of any of the '4' characters from the response alone as none of them have distinguishing delimiters.
Is there any way to coerce the inclusion of type metadata into the response?
I'd like to avoid:
Altering query statements to also include types as that would be obfuscatory and would be superfluous in the event I did switch interfaces;
Prefixing TEXT and BLOB values prior to insert as this would have to be uniform for all TEXT and BLOB interaction (in saying that, this is still my preferred choice should it come to that).
What I'm looking for is a switch of some kind that indicates type as part of SQLite's response, e.g.:
number^_string^_binary^^4^_'4'^_X'4'^^
number^_string^_binary^^4^_text:4'^blob:4^^
Or some variation thereof. Fundamental to this is the response alone contains enough information to discern the type and value of each element of that response (much in the same way sqlite3_column_type() allows in the SQLite Library API).
Update: I've refined this question since the first answer by #mike-sherrill-cat-recall to clarify expectations.
In SQLite, it doesn't always make sense to echo the data type of a column. SQLite doesn't have column-wise data types in the traditional sense. You can use typeof(X) in SQL to show the "datatype of the expression X".
sqlite> create table test (n integer, d decimal(8, 2));
sqlite> insert into test (n, d) values (8, 3.14);
sqlite> insert into test (n, d) values ('wibble', 'wibble');
Inserting text into an integer column succeeds.
sqlite> select n, typeof(n), d, typeof(d) from test;
n typeof(n) d typeof(d)
---------- ---------- ---------- ----------
8 integer 3.14 real
wibble text wibble text
You can concatenate anything you like--even producing caret notation--but it's kind of clumsy.
sqlite> select '(' || typeof(n) || ')^_' || n as caret_n from test;
caret_n
-------------------------
(integer)^_8
(text)^_wibble
SQLite Core Functions
The shell always converts printed values to strings. (That's what "print" means.)
If you don't want to add separate output columns for the types, you could use the quote function to output all values according to SQL syntax rules:
sqlite> with v(x) as (values (null), (1), (2.3), ('hello'), (x'00')) select quote(x) from v;
NULL
1
2.3
'hello'
X'00'
I need someone could help me out on how to trace the error of "mismatched data type" in visual foxpro 6.0 When I issues a command like this "insert into tmpcur from memvar".
tmpcur is a cursor having bulk numbers of columns and it is ready hard to trace which one is having mismatch in data type for insertion problem.
It is pretty difficult to trace the insertion loop of each record into VFP tables one by one unliked MSSQL profiler.
Appreciate to someone could help. Thanks.
This should help you. I have a temp cursor created with some bogus field / column names testing for types of character, integer, double, currency, date and time. Trying to follow what is the result of your scenario, I am taking the memory variable of "bbbb" which should be double (or numeric at the least), and changed it to a string.
I am then HOLDING the error trapping routine that MAY be in effect, then setting my own (as I don't think try/catch existed in VFP6.. it may, but I just don't remember. So, I did an ON ERROR, set a variable to true. Then, I default it to false, try the insert, then check the flag. If the flag IS set, then I go into a loop and try for each column in the given table/alias (in my example it is "C_Tmp", so replace with your table/alias). It goes through each variable, and if the data type is different from the table structure, it will dump the column name and table / memory value for you to review.
You could put this to a log file or something.
Now, another consideration. Some types are completely valid and common for implied conversion, such as character and memo fields can both get strings. Integer, double, float, currency can all work with generic "numeric" values.
So, if you encounter these differences, then we can go one level further and look for comparable types, but let me know and we can adjust as needed.
At least this should give you a huge jump to your insert issue.
CREATE CURSOR C_tmp ( cccc c(10), iiii i, bbbb b(2), ccyyyy y, ddd d, tttt t )
SCATTER MEMVAR memo
m.bbbb = "wrong data type, was double with 2 decimal"
lcHoldError = ON("ERROR")
ON ERROR lFailInsert = .t.
lFailInsert = .f.
INSERT INTO C_Tmp FROM memvar
IF lFailInsert
FOR lnI = 1 TO FCOUNT( "C_Tmp" )
lcTmp = FIELD( lnI, "C_Tmp" )
IF NOT TYPE( "C_Tmp." + lcTmp ) == TYPE( "m.&lcTmp" )
? "Invalid " + lcTmp + ", C_Tmp.&lcTmp, m.&lcTmp
ENDIF
ENDFOR
ENDIF
ON ERROR &lcHoldError
NOTE:, I don't need help with the generic concept of inserting data to a database, just sorting through the contents of an array depending on the content of the "line" and how to determine which "items" in the array correspond to a field in the database
I have a glob of data posted to me by a desktop application that I need to sort through. My old solution worked, but was far less than elegant (INSERT each line of glob into database, then query for, reINSERT, and delete old).
How can I get the following chunk of information (POSTED to me as "f_data") into an array and insert the data into a database?
f_data Contents:
Open~notepad.exe~7/14/2011 2:28:46 PM~COMPUTER01
Open~mspaint.exe~7/14/2011 2:28:55 PM~COMPUTER01
Close~notepad.exe~7/14/2011 2:30:06 PM~COMPUTER01
Close~mspaint.exe~7/14/2011 2:30:06 PM~COMPUTER01
Session~7/14/2011~336~COMPUTER01
Startup~7/18/2011 11:23:12 AM~COMPUTER01
Please keep in mind that I have never used arrays before. 15 years of ASP and I've never had to use an array. How I've been so lucky I don't know, but I think that it may be required for this solution. Here is my current code to put "f_data" into an array:
Example of what I want to do:
var_logdata = request.form("f_data")
arr_logdata = Split(var_logdata,"~")
for var_arrayitem = 0 to ubound(arr_logdata)
'Do some stuff here depending on the log type
'If type is "Open"
'insert to tb_applicationlog
'Elseif type is "Close"
'insert to tb_applicationlog
'Elseif type is "Session"
'insert to tb_sessions
'End if
next
What I don't know how to do is to determine what "type" of log entry the item in the array is. If you look at the code above, I need to insert to different tables in the database depending on the "type" of log entry. For example, an "Open" or "Close" entry goes into the tb_applicationlog table. Once I determine what type the log entry is, how do I align the items in the array "row" to fields in the database?
Thanks very much in advance,
Beems
I think it would be better to split 'logdata' using another character first, then spilt the fields in the array created by 'logdata' using '~', as below (code not tested) -
var_logdata = request.form("f_data")
arr_logdata = Split(var_logdata,vbCrLf)
'split request.form("f_data") using newline so we have an array containing each line
for var_arrayitem = 0 to ubound(arr_logdata)
'now we can split each line by "~"
arr_linelogdata = Split(arr_logdata(var_arrayitem),"~")
'now arr_linelogdata(0) is log type, arr_linelogdata(1) is next field etc
'linetype = arr_linelogdata(0) etc
'use variables derived from array to do what you need to
next