I am implementing CDC based replication between Postgres and Oracle databases. For the source connector I am using Debezium's Postgres Connector and for the sink connector Confluent's JDBC Sink Connector.
In the Postgres database, all tables are created in lowercase, but in the Oracle database tables are in uppercase. How can I dynamically set config in the JDBC sink connector, so that I don't need to set a hardcoded uppercase table name in transforms.route.replacement? Currently if I don't set the destination table name in uppercase manually in the config, double quotes are being added by the connector. Then, I receive the following error:
Caused by: org.apache.kafka.connect.errors.ConnectException: Table "TEST"."dummy_table" is missing and auto-creation is disabled
I have disabled auto-creation, because I don't need it.
Below is an example of my JDBC sink connector config. I have omitted some fields like connection url etc. If I leave the config like this, every time I need to add a new table, I would need to create a new JDBC sink connector.
{
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "test.public.dummy_table",
"transforms": "route,unwrap",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": "([^.]+)\\.([^.]+)\\.([^.]+)",
"transforms.route.replacement": "TEST.DUMMY_TABLE", // I want to use TEST.$3 here, but apply uppercase. Is it possible?
"transforms.unwrap.drop.tombstones": "false",
"transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"pk.mode": "record_key",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"pk.fields": "id",
"quote.sql.identifiers": "never"
}
Is it possible to achieve table name uppercase transformation using transforms.route.replacement or to tell the sink connector not to add double quotes? If not, could you please suggest how to implement it? Thank you in advance.
I know almost nothing about what you described.
However: as far as Oracle is concerned, table names are stored in uppercase in data dictionary, but Oracle lets you access them using any letter case you want (upper, lower, mixed):
SQL> select count(*) from emp union all --> lower
2 select count(*) from EMP union all --> upper
3 select count(*) from EmP; --> mixed
COUNT(*)
----------
14
14
14
SQL>
That stands as long as you didn't use double quotes while creating the table; in that case, you must use the same letter case and enclose table name into double quotes every time you access it:
SQL> create table "teST" (id number);
Table created.
SQL> select * from test;
select * from test
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> select * from TEST;
select * from TEST
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> select * from teST;
select * from teST
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> select * from "teST";
no rows selected
SQL>
So: if you weren't that unfortunate to use double quotes, perhaps you don't have to bother (as my first example shows). Just don't use double quotes while referencing the table and you're fine.
To avoid the problem with quotes in kafka connect sink, try setting quote_sql_identifiers to never.
https://docs.confluent.io/kafka-connect-jdbc/current/sink-connector/index.html#jdbc-sink-identifier-quoting
Related
I have a table named "libisatz" in my database, and there is no table, nor view with name "libiSatz" (with capital S instead of s) and I don't find any kind of object in my schema having name "libiSatz". But, surprisingly selecting from "libiSatz" results in the same result as if I had written "libisatz". If I change any other letter in this name from lower case to upper case (e.g. I write "Libisatz", then I get an error. How can this be?.
ADDENDUM
I've checked the ideas of #Jon Heller with the following result:
Both select * from DBA_OBJECTS where OBJECT_NAME like 'libi_atz'; and select * from DBA_OBJECTS where lower(OBJECT_NAME) = 'libisatz'; returns one single row (it is the "libisatz" table)
select * from DBA_OBJECTS where OBJECT_NAME = 'libisatz'; returns one row and select * from DBA_OBJECTS where OBJECT_NAME = 'libiSatz'; returns no row.
Both select * from DBA_SQL_TRANSLATIONS; and select * from DBMS_ADVANCED_REWRITE; results in ORA-00942: table or view does not exist
select * from DBA_REWRITE_EQUIVALENCES; results in no rows selected
select DUMP(OBJECT_NAME) from DBA_OBJECTS where lower(OBJECT_NAME) = 'libisatz'; returns Typ=1 Len=8: 108,105,98,105,115,97,116,122
So it seems that the puzzle is not yet solved.
ADDENDUM 2.
When I try to create a view named "libiSatz" then I get
ORA-00955: name is already used by an existing object
in spite of the results above.
After I renamed "libiSatz" to "oraclepuzzle",
select count(*) from "libisatz"; and "select count(*) from "oraclepuzzle"; works, but select count(*) from "libiSatz"; doesn't.
select * from dba_objects where object_name in ( 'oraclepuzzle', 'libisatz', 'libiSatz'); returns one row with object_name "oraclepuzzle".
Do you have SQL translations, rewrite equivalances, or UTF8 characters?
The SQL translation framework was built to allow applications designed for other database types to seamlessly query Oracle. But instead of translating syntaxes, the feature could also be used to silently change your queries. For example, here is an example of querying a table that does not exist in order to workaround table size limitations. Check the view DBA_SQL_TRANSLATIONS.
Rewrite equivalences can be created by the package DBMS_ADVANCED_REWRITE, and can be used to silently change the results of wrong queries or for some rare performance problems. (In my simple tests I wasn't able to get this query to work for invalid queries, but I bet there is a way to make it work.) Check the view DBA_REWRITE_EQUIVALENCES.
UTF8 characters may be silently swapped out for similar ASCII characters. This probably depends on your database and client character set settings. The below example is pretty obviously not an ASCII "S", but you may have a more subtle problem. Check the source of your values, retype them manually, and use the DUMP function to evaluate the binary of the characters.
SQL> create table "libisatz"(a number);
Table created.
SQL> -- An uppercase "S" does not work
SQL> select * from "libiSatz";
select * from "libiSatz"
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> -- But a "LATIN SMALL LETTER S WITH CARON" works.
SQL> select * from "libišatz";
no rows selected
Lastly, are you 100% sure that the object doesn't exist on your database? The above examples are possible, but extremely rare. Triple-check DBA_OBJECTS.OBJECT_NAME for the mystery object.
My deafult Oracle object names are case insensitive.
So, when you issue teh command:
CREATE TABLE the_table ..
CREATE TABLE The_Table ..
CREATE TABLE THE_TABLE ..
The underlying object name in all cases is THE_TABLE, but you can access it using any mix of upper/lower case:
SELECT * FROM THE_TABLE
SELECT * FROM tHe_TaBlE
SELECT * FROM the_table
Will all work.
When a table is created with the name wrapped in double-quotes, the name becomes case-sensitive.
So, if you issue:
CREATE TABLE "libiSatz" ..
Then the table name in all SQL statements must match the exact name in the double quotes, i.e.
SELECT * FROM "libiSatz" is OK (Correction following comments)
SELECT * FROM libisatz is NOT OK
SELECT * FROM LibiSatz is NOT OK
When you search for this in USER_OBJECTS or USER_TABLES you will have to match the creation name:
SELECT * FROM USER_OBJECTS WHERE OBJECT_NAME = 'libiSatz';
I am working with an application using asp.net core 2.2 and efcore database first approach with Oracle database. I am using Oracle.EntityFrameworkCore (2.19.60) nuget package, successfully mapped db model, but when I try to fetch data from DBContext , getting error
ORA-00904: "m"."Id": invalid identifier
Oracle Database version: Oracle Database 12c Standard Edition Release 12.2.0.1.0 - 64bit Production
code:
var fetched = await myDatabaseContext.MyTableVersions.ToListAsync();
LinQ is generating following query :
SELECT "m"."Id", "m"."MAJORVERSION" FROM "MyTableVersions" "m"
Since It's not a correct syntax for PL/Sql query so getting error ORA-00904: "m"."Id": invalid identifier.
Is there any way to fix this error? Thanks.
I have searched on the web and found
Bug 30352492 - EFCORE: ORA-00904: INVALID IDENTIFIER
but that issue is related to schema.
The query is perfectly valid (db<>fiddle here), assuming your table looks something like
CREATE TABLE "MyTableVersions"
("Id" NUMBER,
MAJORVERSION NUMBER)
However, I suspect your table looks like
CREATE TABLE "MyTableVersions"
(ID NUMBER,
MAJORVERSION NUMBER)
I don't know what the class looks like that you're trying to fetch into, but I suspect it has a field named Id. If you can change the name of that field to ID (in other words, so that its capitalization matches the capitalization of the related database column) you might find it works then.
Double quotes are something you should avoid in Oracle world.
Your query:
SELECT "m"."Id", "m"."MAJORVERSION" FROM "MyTableVersions" "m"
means that column name is exactly Id (capital I followed by d). Only if that's really so, you should use double quotes. Otherwise, simply remove them:
SELECT m.id, m.MAJORVERSION FROM MyTableVersions m
The same goes for the table name - if it was created using mixed case (and you can't do that without double quotes), you'll have to use double quotes and exactly same letter case always. Otherwise, don't use them.
Oracle is case-insensitive regarding object and column names and are UPPERCASE by default:
SQL> create table test (id number);
Table created.
SQL> desc test
Name Null? Type
----------------------------------------- -------- -----------------
ID NUMBER
SQL> select table_name, column_name
2 from user_tab_columns
3 where table_name = 'TEST';
TABLE_NAME COLUMN_NAME
------------------------------ ------------------------------
TEST ID
SQL> insert into test (id) values (1);
1 row created.
SQL>
But, if you use mixed case with double quotes, then use them always:
SQL> create table "teST" ("Id" number);
Table created.
SQL> insert into test (id) values (1);
insert into test (id) values (1)
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> insert into "test" (id) values (1);
insert into "test" (id) values (1)
*
ERROR at line 1:
ORA-00942: table or view does not exist
SQL> insert into "teST" (id) values (1);
insert into "teST" (id) values (1)
*
ERROR at line 1:
ORA-00904: "ID": invalid identifier
SQL> insert into "teST" ("Id") values (1);
1 row created.
SQL>
So: first make sure what table and column names really are, then use them appropriately.
To avoid problems with case-sensitivity, add the option to eliminate EF to add quotes to the generated queries. For instance, if you are using Devart Oracle provider, this is done in the following way:
public override void ConfigureServices(ServiceConfigurationContext context)
{
Configure<AbpDbContextOptions>(options =>
{
var config = Devart.Data.Oracle.Entity.Configuration.OracleEntityProviderConfig.Instance;
config.Workarounds.DisableQuoting = true;
...
}
}
If you are using official provider - just inherit from DbCommandInterceptor (do quotes replacement with empty character in dbCommand in ...Executing methods), add this interceptor to DbContextOptionsBuilder instance.
I run the shell tool of H2:
java -cp h2-1.4.199.jar org.h2.tools.Shell
I do:
sql> show tables
...> ;
TABLE_NAME | TABLE_SCHEMA
flyway_schema_history | PUBLIC
(1 row, 19 ms)
Then I want to select from that table:
sql> select * from public.flyway_schema_history;
Error: org.h2.jdbc.JdbcSQLSyntaxErrorException: Table "FLYWAY_SCHEMA_HISTORY" not found; SQL statement:
select * from public.flyway_schema_history [42102-199]
I tried with out the schema name also, same error.
Why can't it select from that table, when it shows it exists?
H2 database is by default case sensitive.
So when you write the table name with lower case letters, and query it without double quotes, it will make it uppercase and try to match it with that.
Since the current table name is lower case you should but it into quotes so that H2 wouldn't force it to uppercase.
select * from public."flyway_schema_history";
In my main table having table structure column without double quotes but when in View column with double quotes. While inserting data from view into table then getting error.
Here is the table structure of tbl
create table tbl (ID number(10),
name varchar2(50),
addr varchar2(200));
While View is-
create or replace view t_view as
select "ID", "name", "addr" from tbl;
While inserting data into tbl from t_view -
insert into tbl
select * from t_view;
Then getting error ORA- 00904: "addr": Invalid identifier.
So how to resolve this issue, can i remove the double quotes from creation of view.
Remove all double quotes from everywhere in your code.
If you use them while creating objects, you'll have to use them always, specifying exactly same letter case.
Get rid of those, Oracle is - by default - case insensitive and treats all names as uppercase (but you can reference them any way you want, just don't use double quotes!).
Oracle stores the name of any object in UPPERCASE by default. If you have provided double quotes then only Oracle stores the name of the object as it is.
Double quotes - Case sensitive
No quotes - Case insensitive - Stores the name in UPPERCASE
In your case, While writing the DDL of the table, you have not provided the name of the column in double quotes so your table name and column names are stored in UPPERCASE in the metadata.
You can see the same using the following query
select table_name, column_name from user_tab_columns where table_name = 'TBL';
For giving your answer, create the view either of the following syntaxes:
create or replace view t_view as
select ID, name, addr from tbl; -- no double quotes
create or replace view t_view as
select "ID", "NAME", "ADDR" from tbl; -- UPPERCASE column names in double quotes
See the demo here
Cheers!!
What is it you are trying to accomplish. Your request just doesn't make sense. Your intended statement
insert into tbl select * from t_view;
accomplishes exactly the same thing as
insert into tbl select * from tbl;
It's not that you cann't do this (you can), but it can only cause massive duplicate data issues or (hopefully) unique constraint violations.
How to create a global temporary table with same table structure to that of a existing table?
I know this concept is available in SQL server like "select * into #temp123 from abc". But I want to perform the same in Oracle.
Create global temporary table mytemp
as
select * from myTable
where 1=2
Global temporary tables in Oracle are very different from temporary tables in SQL Server. They are permanent data structures, it is merely the data in them which is temporary (limited to the session or transaction, depending on how a table is defined).
Therefore, the correct way to use global temporary tables is very different to how we use temporary tables in SQL Server. The CREATE GLOBAL TEMPORARY TABLE statement is a one-off exercise (like any other table). Dropping and recreating tables on the fly is bad practice in Oracle, which doesn't stop people wanting to do it.
Given the creation of a global temporary table should a one-off exercise, there is no real benefit to using the CREATE TABLE ... AS SELECT syntax. The statement should be explicitly defined and the script stored in source control like any other DDL.
You have tagged your question [oracle18c]. If you are really using Oracle 18c you have a new feature open to you, private temporary tables, which are closer to SQL Server temporary tables. These are tables which are genuinely in-memory and are dropped automatically at the end of the transaction or session (again according to definition). These are covered in the Oracle documentation but here are the headlines.
Creating a private temporary table data with a subset of data from permanent table T23:
create table t23 (
id number primary key
, txt varchar2(24)
);
insert into t23
select 10, 'BLAH' from dual union all
select 20, 'MEH' from dual union all
select 140, 'HO HUM' from dual
/
create private temporary table ORA$PTT_t23
on commit preserve definition
as
select * from t23
where id > 100;
The ORA$PTT prefix is mandatory (although it can be changed by setting the init.ora parameter PRIVATE_TEMP_TABLE_PREFIX, but why bother?
There after we can execute any regular DML on the table:
select * from ORA$PTT_t23;
The big limitation is that we cannot use the table in static PL/SQL. The table doesn't exist in the data dictionary as such, and so the PL/SQL compiler hurls - even for anonymous blocks:
declare
rec t23%rowtype;
begin
select *
into rec
from ORA$PTT_t23';
dbms_output.put_line('id = ' || rec.id);
end;
/
ORA-06550: line 6, column 10: PL/SQL: ORA-00942: table or view does not exist
Any reference to a private temporary table in PL/SQL must be done with dynamic SQL:
declare
n pls_integer;
begin
execute immediate 'select id from ORA$PTT_t23' into n;
dbms_output.put_line('id = ' || n);
end;
/
Basically this restricts their usage to SQL*Plus (or sqlcl scripts which run a series of pure SQL statements. So, if you have a use case which fits that, then you should check out private temporary tables. However, please consider that Oracle is different from SQL Server in many aspects, not least its multi-version consistency model: readers do not block writers. Consequently, there is much less need for temporary tables in Oracle.
In the SQL Server's syntax the prefix "#" (hash) in the table name #temp123 means - create temporary table that is only accessible via the current session ("##" means "Global").
To achive exactly the same thing in Oracle you can use private temporary tables:
SQL> show parameter private_temp_table
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
private_temp_table_prefix string ORA$PTT_
create table mytab as
select 1 id, cast ('aaa' as varchar2 (32)) name from dual
;
create private temporary table ora$ptt_mytab on commit preserve definition as
select * from mytab where 1=0
;
Private TEMPORARY created.
Afterwards you can use these tables in SQL and PL/SQL blocks:
declare
r mytab%rowtype;
begin
insert into ora$ptt_mytab values (2, 'bbb');
select id + 1, name||'x' into r from ora$ptt_mytab where rownum = 1;
insert into ora$ptt_mytab values r;
end;
/
select * from mytab
union all
select * from ora$ptt_mytab;
ID NAME
---------- --------------------------------
1 aaa
2 bbb
3 bbbx
Some important restrictions on private temporary tables:
The name must always be prefixed with whatever is defined with the parameter PRIVATE_TEMP_TABLE_PREFIX. The default is ORA$PTT_.
You cannot reference PTT in the static statements of the named PL/SQL blocks, e.g. packages, functions, or triggers.
The %ROWTYPE attribute is not applicable to that table type.
You cannot define column with default values.