In clojure/java jdbc, i have learned that using getMetaData i can return massive amounts of interesting information about a database connected to through jdbc. This can be filtered based on the catalog, schema, and table name.
(defn get-db-metadata
[db-spec ]
(with-connection (get-db-connection-map db-spec)
; get columns returs the following:
; TABLE_CAT String => table catalog (may be null)
; TABLE_SCHEM String => table schema (may be null)
; TABLE_NAME String => table name
; COLUMN_NAME String => column name
; DATA_TYPE int => SQL type from java.sql.Types
; TYPE_NAME String => Data source dependent type name, for a UDT the type name is fully qualified
; COLUMN_SIZE int => column size. For char or date types this is the maximum number of characters, for numeric or decimal types this is precision.
; BUFFER_LENGTH => not used.
; DECIMAL_DIGITS int => the number of fractional digits
; NUM_PREC_RADIX int => Radix (typically either 10 or 2)
; NULLABLE int => is NULL allowed.
; columnNoNulls => might not allow NULL values
; columnNullable => definitely allows NULL values
; columnNullableUnknown => nullability unknown
; REMARKS String => comment describing column (may be null)
; COLUMN_DEF String => default value (may be null)
; SQL_DATA_TYPE int => unused
; SQL_DATETIME_SUB int => unused
; CHAR_OCTET_LENGTH int => for char types the maximum number of bytes in the column
; ORDINAL_POSITION int => index of column in table (starting at 1)
; IS_NULLABLE String => "NO" means column definitely does not allow NULL values; "YES" means the column might allow NULL values. An empty string means nobody knows.
; SCOPE_CATLOG String => catalog of table that is the scope of a reference attribute (null if DATA_TYPE isn't REF)
; SCOPE_SCHEMA String => schema of table that is the scope of a reference attribute (null if the DATA_TYPE isn't REF)
; SCOPE_TABLE String => table name that this the scope of a reference attribure (null if the DATA_TYPE isn't REF)
; SOURCE_DATA_TYPE short => source type of a distinct type or user-generated Ref type, SQL type from java.sql.Types (null if DATA_TYPE isn't DISTINCT or user-generated REF)
(into #{}
(map #(str (% :table_name) "." (% :column_name) "\n")
(resultset-seq (->
(connection)
(.getMetaData)
; Params in are catalog, schemapattern, tablenamepattern
;(.getColumns "stuff" "public" nil "%")
(.getColumns "db_catalog" "schema_name" "some_table" "%")
)
)
)
)
)
)
I'm interested in this information not for every table in the database, but rather for the result set returned by a specific query. My needs specifically at this point for doing this is to know the max length of a particular column that is retrieved, to print to the screen for instance in fixed-width format.
What i have considered so far (not ideal i'm sure):
Trying to parse the in-bound sql statement to figure out what tables
are being queried and then fetching the metadata for those tables
specifically. This would prove to be complicated with the results
of a select statement with functions, or common table expressions, etc.
I think it could get messy (and inaccurate) quickly.
What also may work would be to create a temporary view based on the
inbound query... then i could fetch the metadata on this view. This
however wouldn't work if i only had a read-only connection to the
database i was working with. often the case i believe for what i am trying to do.
Fetching the results, and then for each column returned finding the
max length of the values, and then creating my fixed-width grid
accordingly. This wouldn't be great if i was looking at large
result sets.....
Is there a better way to determine the types of everything i'm returning from my query? How do other programs do this? It seems like i should be able to get the metadata from the time i issue the query request somehow here:
(defn fetch-results
"Treat lazy result sets in whole for returning a database query"
[db-spec query]
(with-connection
(get-db-connection-map db-spec)
(with-query-results res query
; (get the medata here somehow for columns returned ????)
(doall res))
)
)
thanks in advance.
You can do like this in scala, may be this can help you out:
var stmt: PreparedStatement = null
var rs: ResultSetMetaData = null
try {
stmt = conn.prepareStatement(query)
rs = stmt.getMetaData()
} finally {
cleanup(stmt)
}
}
Related
I'm trying to dynamically unpivot each column provided in a list.
(Table as table, ColumnNameList as list) as table =>
List.Accumulate(
List.Zip(ColumnNameList, {1..List.Count(ColumnNameList)}),
Table,
(state, column) => Table.Unpivot(state, column{0}, "Option" & column{1} & "Name", "Option" & column{1} & "Value")
)
This code doesn't work though, how can I do what I'm trying to do?
I ended up creating a function to provide a unique column name, somewhat like how the power query visual editor will automatically create names if an existing column already has one.
Haven't been able to test how it works once you get past more than one with the same name though.
(Table as table, ColumnName as text) =>
if Table.HasColumns(Table, ColumnName) then
let
NewName =
if Text.At(ColumnName, 3) <> "." then
ColumnName & ".01"
else
Text.Start(
ColumnName,
Text.Length(ColumnName) - 2
)
& Text.PadStart(
Text.From(
Number.FromText(Text.End(ColumnName, 2))
+ 1
),
2,
"0"
)
in
#FnTableNewColumnName(Table, NewName)
else
ColumnName
Then I changed the fuction I was working on, to call the naming function for both of the two extra parameters.
(Table as table, ColumnNameList as list) as table =>
List.Accumulate(
ColumnNameList,
Table,
(state, column) =>
Table.Unpivot(
state,
{ column },
FnTableNewColumnName(state, "attribute"),
FnTableNewColumnName(state, "value")
)
)
I am trying to translate the following query from SQL to EF Core. I can easily just use a stored procedure (I already have the SQL), but am trying to learn how some of the linq queries work. Unfortunately this is not by any means an ideal database schema that I inherited and I don't have the time to convert it to something better.
DECLARE #userId INT = 3
SELECT *
FROM dbo.CardGamePairs
WHERE EXISTS (SELECT 1
FROM dbo.Users
WHERE Users.Id = CardGamePairs.player1Id
AND Users.userId = #userId)
UNION
SELECT *
FROM dbo.CardGamePairs
WHERE EXISTS (SELECT 1
FROM dbo.Users
WHERE Users.Id = TableB.player2Id
AND Users.userId = #userId)
So basically I have an id that can exist in one of two separate columns in table b and I don't know in which column it may be in, but I need all rows that have that ID in either column. The following is what I tried to make this work:
//Find data from table A where id matches (part of the subquery from above)
var userResults = _userRepository.GetAllAsQueryable(x => x.userId == userId).ToList();
//Get data from table b
var cardGamePairsResults = _cardGamePairsRepository.GetAllAsQueryable(x => userResults .Any(y => y.userId == x.player1Id || y.userId == x.player2Id));
When I run the code above I get this error message:
predicate: (y) => y.userId == x.player1Id || y.userId == x.player2Id))' could not be translated. Either rewrite the query in a form that can be translated, or switch to client evaluation explicitly by inserting a call to either AsEnumerable(), AsAsyncEnumerable(), ToList(), or ToListAsync().
Any ideas on how I can make this work? (I tried changing the column and table names to something that would actually make sense, hopefully I didn't miss any spots and make it more confusing.)
Because you are already have user id use it to query both columns.
var userResults = _userRepository
.GetAllAsQueryable(x => x.userId == userId)
.ToList();
var cardGamePairsResults = _cardGamePairsRepository
.GetAllAsQueryable(x => x.player1Id == userId || x.player2Id == userId));
if I have a LINQ to SQL query like this:
var query = from x in db.XTable
select x.y == 123; // the y column is nullable!
List<bool> list = query.ToList();
it will generate a SQL statement that will contain this:
(CASE
WHEN [t0].[y] = 123 THEN 1
WHEN NOT ([t0].[y] = 123) THEN 0
ELSE NULL
END)
which will throw an error cause null could not get assigned to bool. I know why this happens (because a comparison in SQL with null is always false) but I don't know why LINQ to SQL does not use a stetement like this:
(CASE
WHEN [t0].[y] = 123 THEN 1
ELSE 0
END)
which would work.
Can I push LINQ to SQL to do this?
Probably, as you state, because you must think the SQL way with linq to sql, not the object way...
It might be considered as a bug or as a feature, by the way...
Especially with null values.
For example, concatenation of nullable string is different in linq to sql and linq to objects.
Assuming a and b are strings :
from n in db
select n.a + n.b
in linq to sql, if a is null and b is not, a + b = null
in linq to object if a is null anb b is not a + b = b
to get the same result in linq to sql, you'll have to use the coalesce operator select (a ?? string.Empty) + b
Anyway, you can either return a list of Nullable<bool> and a list of bool which would be :
from x in db.XTable
select x.y != null && x.y == 123
or
from x in db.XTable
select (x.y ?? 0) == 123
But to get what you want in linq to objects, you would have to do
from x in db.XTable.ToList()
select (x.y== null ? (bool?)null : x.y== 123))
EDIT
This might change in future versions (not sure if the given case will be included)
Recently I developed a Hive Generic UDF getad. It accepts a map type and a string type parameter and outputs a string value. But I found the UDF output really confusing in different conditions.
Condition A:
select
getad(map_col, 'tp') as tp,
getad(map_col, 'p') as p,
getad(map_col, 'sp') as sp
from
table_name
where
id = xxxx;
The output is right: 'tp', 'p', 'sp'.
Condition B:
select
array(tp, p, sp) as ps
from
(
select
getad(map_col, 'tp') as tp,
getad(map_col, 'p') as p,
getad(map_col, 'sp') as sp
from
table_name
where
id = xxxx
) t;
The output is wrong: 'tp', 'tp', 'tp'.
Could you please provide me some hints on this? Thanks!
After setting hive.cache.expr.evaluation=false, all queries output expected results.
And I found that it's related to the getDisplayString function in the UDF. At first the function returns a string regardless of its parameters. And I had to set hive.cache.expr.evaluation = false.
But after I changed the function to return string in depend of parameters, all queries returned expected results even when the hive.cache.expr.evaluation was set to true.
For multiple data insertion we have an efficient way: RecordSortedList
RecordSortedList rsl;
MyTable myTable;
;
rsl = new RecordSortedList(myTable.tableid);
rsl.sortOrder(fieldname2id(myTable.tableId,'RecId'));
myTable.field1 = 'Value1';
rsl.ins(myTable);
myTable.field1 = 'Value2';
rsl.ins(myTable);
rsl.insertDatabase();
Is the same possible for multiple records retrieval from db in one go? Something like
int i =1;
while(i<10000)
{
//enter records from db into a buffer in db
i++
}
//now bring the buffer from db in a single trip
//and do the data manipulation in AX
My intention is to optimize the db trip to the least.
Please Suggest.
Yes, it's called RecordLinkList - http://msdn.microsoft.com/en-us/library/aa643250(v=ax.50).aspx
A recordLinkList is a double linked list that can hold records of
different types at the same time. It is not keyed or sorted.
The recordLinkList is particularly useful for passing records from
different tables as a parameter instead of retrieving the same records
again.
There is no limit to the size of a recordSortedList; it is the
responsibility of the programmer to control its size and, therefore,
memory consumption.
You can also add different types of records.
static void RecordLinkList(Args _args)
{
RecordLinkList rll = new RecordLinkList();
SalesTable salesTable;
CustTable custTable;
InventTrans inventTrans;
Address address;
boolean iterate;
;
select firstonly salesTable;
select firstonly custTable;
select firstonly inventTrans;
select firstonly address;
rll.ins(salesTable);
rll.ins(custTable);
rll.ins(inventTrans);
rll.ins(address);
iterate = rll.first();
while (iterate)
{
switch (rll.fileId()) // FileId == TableId
{
case tablenum(SalesTable):
salesTable = rll.peek();
info(strfmt("SalesTable");
break;
case tablenum(CustTable):
custTable = rll.peek();
info("CustTable");
break;
case tablenum(InventTrans):
inventTrans = rll.peek();
info("InventTrans");
break;
default:
error(strfmt("Table %1 (%2) not expected", tableid2name(rll.fileId()), rll.fileId()));
}
iterate = rll.next();
}
info("Done");
}
The insertDatabase method as stated (use the RecordInsertList class instead of RecordSortedList, if you do not need the sorted order):
inserts multiple records on a single trip to the database.
However this is mostly from the programmers perspective. The operation from the SQL goes like this:
INSERT INTO MyTable ( Column1, Column2 )
VALUES ( Value1, Value2 ),
( Value1, Value2 ), ...
There are limits to the number of records inserted this way, so the AX kernel may split the list to make several calls to the SQL server.
The other way from DB to AX is easy:
while select myTable where ...
Which is translated to SQL as:
SELECT T1.Column1, T1.Column2 FROM MyTable T1 WHERE...
This transports the data from the table to AX as efficient as possible.
You may choose to use a QueryRun object instead, but the call to SQL stays the same.
If you do simple updates on the table, consider using update_recordset as this may move the updates to the SQL server and eliminating the round-trip.