I have an Oracle database that contains multiple users/schemas and I would like to generate Slick Schemas automatically for a specific user. This is what I've tried so far :
import scala.concurrent.ExecutionContext.Implicits.global
val profileInstance: JdbcProfile =
Class.forName("slick.jdbc.OracleProfile$")
.getField("MODULE$")
.get(null).asInstanceOf[JdbcProfile]
val db = profileInstance.api.Database
.forURL("jdbc:oracle:thin:#//myhost:myport/servicename","user","pass")
val modelAction = OracleProfile.createModel(Some(OracleProfile.defaultTables))
val model = Await.result(db.run(modelAction), Duration.Inf)
model.tables.foreach(println)
This doesn't print anything, I guess I have to provide the current schema to use, but I don't know how to do this.
On the other hand, I am able to list all the schemas of the database, using the following code :
val resultSet = db.createSession().metaData.getSchemas.getStatement.getResultSet
while(resultSet.next()) {
println(resultSet.getString(1))
}
How can I specify which schema I want to use with Slick ?
I've found out how to do it. Instead of using OracleProfile.defaultTable I manually defined the tables and views I needed like this :
val modelAction = OracleProfile.createModel(
Some(MTable.getTables(None, Some("MYSCHEMA"), None, Some(Seq("TABLE", "VIEW"))))
)
Related
I have the following code
#Indexed
#PrimaryKeyColumn(name = "x", ordinal = 1, type = PrimaryKeyType.PARTITIONED)
#Column(value="x")
private String x;
#Indexed
#PrimaryKeyColumn(name = "code", ordinal = 2, type = PrimaryKeyType.PARTITIONED)
#Column(value="code")
private String code;
#Query(value = "select * from customers where code = ?0")
Optional<Customer> findByCode(String code);
When this is executed, I get Caused by: com.datastax.driver.core.exceptions.InvalidQueryException: Cannot execute this query as it might involve data filtering and thus may have unpredictable performance. If you want to execute this query despite the performance unpredictability, use ALLOW FILTERING.
Is there a way to avoid this just from spring-data-cassandra? I do not want to add ALLOW FILTERING in my query. I tried creating a separate index on the code column but this haven't solved the issue. I think it stops in the spring data configuration. If I execute the same query in cqlsh, it works.
You must specify partition key on your query, unless you create index or use ALLOW FILTERING
Executing query with allow filtering might not be a good idea as it can use a lot of your computing resources and Might not return any result because of timeout. Don't use allow filtering in production Read the datastax doc about using ALLOW FILTERING
https://docs.datastax.com/en/cql/3.3/cql/cql_reference/select_r.html?hl=allow,filter
When using a no-sql database, you need to properly design your data to avoid filtering. You can add a secondary index to optimize retrieval by a specific field. More details here: https://docs.datastax.com/en/archived/cql/3.3/cql/cql_using/useSecondaryIndex.html
If you are sure that the query is what you need, you can use the allowFiltering parameter on the #Query annotation to explicitly indicate that ALLOW FILTERING be used.
#Query(value = "select * from customers where code = ?0", allowFiltering = true)
Optional<Customer> findOneByCode(String code);
I am setting up a daily cron job that appends a row to BigQuery table (using Python), however, duplicate data is being inserted. I have searched online and I know that there is a way to manually remove duplicate data, but I wanted to see if I could avoid this duplication in the first place.
Is there a way to check a BigQuery table to see if a data record already exists first in order to avoid inserting duplicate data? Thanks.
CODE SNIPPET:
import webapp2
import logging
from googleapiclient import discovery
from oath2client.client import GoogleCredentials
PROJECT_ID = 'foo'
DATASET_ID = 'bar'
TABLE_ID = 'foo_bar_table’
class UpdateTableHandler(webapp2.RequestHandler):
def get(self):
credentials = GoogleCredentials.get_application_default()
service = discovery.build('bigquery', 'v2', credentials=credentials)
try:
the_fruits = Stuff.query(Stuff.fruitTotal >= 5).filter(Stuff.fruitColor == 'orange').fetch();
for fruit in the_fruits:
#some code here
basket = dict()
basket['id'] = fruit.fruitId
basket['Total'] = fruit.fruitTotal
basket['PrimaryVitamin'] = fruit.fruitVitamin
basket['SafeRaw'] = fruit.fruitEdibleRaw
basket['Color'] = fruit.fruitColor
basket['Country'] = fruit.fruitCountry
body = {
'rows': [
{
'json': basket,
'insertId': str(uuid.uuid4())
}
]
}
response = bigquery_service.tabledata().insertAll(projectId=PROJECT_ID,
datasetId=DATASET_ID,
tableId=TABLE_ID,
body=body).execute(num_retries=5)
logging.info(response)
except Exception, e:
logging.error(e)
app = webapp2.WSGIApplication([
('/update_table', UpdateTableHandler),
], debug=True)
The only way to test whether the data already exists is to run a query.
If you have lots of data in the table, that query could be expensive, so in most cases we suggest you go ahead and insert the duplicate, and then merge duplicates later on.
As Zig Mandel suggests in a comment, you can query over a date partition if you know the date when you expect to see the record, but that may still be expensive compared to inserting and removing duplicates.
I should extract data from an oracle database.
How can I find out which schema are defined in the database?
When I do not define any schema in the description of Metadata(), I find no tables.
thanks for your help,
Default Oracle schema matches the username that was used in Oracle connection.
If you don't see any tables - it means the tables are created in another schema.
Looks like you have two questions here:
1) about Oracle schemas - how to find schema and tables in Oracle
2) about SQLAlchemy reflections - how to specify Oracle schema for table
You can find answer for the first question in many places. I.e. here: https://stackoverflow.com/a/2247758/1296661
Answering second question:
Table class constructor has schema argument to specify table's schema if it is different from default user's schema. See more here
http://docs.sqlalchemy.org/en/rel_0_7/core/schema.html#sqlalchemy.schema.Table
Here is the python code to answer second question. You will need to setup db connection and table name values to match your case:
from sqlalchemy import Table
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
engine = create_engine('oracle://<user_name>:<password>#<hostname>:1521/<instance name>', echo=True)
Base = declarative_base()
reflected_table = Table('<Table name>',
Base.metadata,
autoload=True,
autoload_with=engine,
schema='<Schema name other then user_name>')
print [c.name for c in reflected_table.columns]
p = engine.execute("SELECT OWNER,count(*) table_count FROM ALL_OBJECTS WHERE OBJECT_TYPE = 'TABLE' GROUP BY OWNER");
for r in p:
print r
Good luck with using sqlschema and reflection feature - it is a lot of fun. You get your python program working with existing database almost without defining schema information in your program.
I'm using this feature with oracle db in production - the only thing I have to define were relations between tables explicitly setting foreign and primary keys.
I found two other ways to set an alternate Oracle schema.
Using MetaData and passing it into the creation of the SqlAlchemy instance or engine.
Or using __table_args__ when defining your models and pass in {'schema': 'other_schema' }
http://docs.sqlalchemy.org/en/latest/core/metadata.html#sqlalchemy.schema.MetaData.params.schema
How to specify PostgreSQL schema in SQLAlchemy column/foreign key mixin?
But both of these don't help the foreign key calls. For those, you still have to manually prefix with other_schema. before the table name.
The snippet below is in the context of a Flask app with Flask-SqlAlchemy in use.
from sqlalchemy import schema
from flask_sqlalchemy import SQLAlchemy
import os
os.environ['TNS_ADMIN'] = os.path.join( 'oracle_tools', 'network', 'admin')
oracle_connection_string = 'oracle+cx_oracle://{username}:{password}#{tnsname}'.format(
username='user',
password='pass',
tnsname='TNS_SPACE',
)
oracle_schema = 'user_schema_1'
foreign_key_prefix = oracle_schema + '.'
app.config['SQLALCHEMY_DATABASE_URI'] = oracle_connection_string
oracle_db_metadata = schema.MetaData(schema=oracle_schema)
db = SQLAlchemy(app, metadata=oracle_db_metadata)
user_groups_table = db.Table(db_table_prefix + 'user_groups', db.Model.metadata,
db.Column('...', db.Integer, db.ForeignKey(foreign_key_prefix + 'group.id')),
# schema=oracle_schema # Not necessary because of the use of MetaData above
)
class User(db.Model):
__tablename__ = 'user'
# __table_args__ = {'schema': oracle_schema } # Not necessary because of the use of MetaData above
user_name = db.Column('USER_NAME', db.String(250))
And... two other related links:
https://github.com/mitsuhiko/flask-sqlalchemy/issues/172
How to specify PostgreSQL schema in SQLAlchemy column/foreign key mixin?
Hope that helps.
I have an Abstract type called Product, and five "Types" that inherit from Product in a table per type hierarchy fashion as below:
I want to get all of the information for all of the Products, including a smattering of properties from the different objects that inherit from products to project them into a new class for use in an MVC web page. My linq query is below:
//Return the required products
var model = from p in Product.Products
where p.archive == false && ((Prod_ID == 0) || (p.ID == Prod_ID))
select new SearchViewModel
{
ID = p.ID,
lend_name = p.Lender.lend_name,
pDes_rate = p.pDes_rate,
pDes_details = p.pDes_details,
pDes_totTerm = p.pDes_totTerm,
pDes_APR = p.pDes_APR,
pDes_revDesc = p.pDes_revDesc,
pMax_desc = p.pMax_desc,
dDipNeeded = p.dDipNeeded,
dAppNeeded = p.dAppNeeded,
CalcFields = new DAL.SearchCalcFields
{
pDes_type = p.pDes_type,
pDes_rate = p.pDes_rate,
pTFi_fixedRate = p.pTFi_fixedRate
}
}
The problem I have is accessing the p.pTFi_fixedRate, this is not returned with the Products collection of entities as it is in the super type of Fixed. How do I return the "super" type of Products (Fixed) properties using Linq and the Entity Framework. I actually need to return some fields from all the different supertypes (Disc, Track, etc) for use in calculations. Should I return these as separate Linq queries checking the type of "Product" that is returned?
This is a really good question. I've had a look in the Julie Lerman book and scouted around the internet and I can't see an elegant answer.
If it were me I would create a data transfer object will all the properties of the types and then have a separate query for each type and then union them all up. I would insert blanks into the DTO properies where the properties aren't relevant to that type. Then I would hope that the EF engine makes a reasonable stab at creating decent SQL.
Example
var results = (from p in context.Products.OfType<Disc>
select new ProductDTO {basefield1 = p.val1, discField=p.val2, fixedField=""})
.Union(
from p in context.Products.OfType<Fixed>
select new ProductDTO {basefield1 = p.val1, discField="", fixedField=p.val2});
But that can't be the best answer can it. Is there any others?
So Fixed is inherited from Product? If so, you should probably be querying for Fixed instead, and the Product properties will be pulled into it.
If you are just doing calculations and getting some totals or something, you might want to look at using a stored procedure. It will amount to fewer database calls and allow for much faster execution.
Well it depends on your model, but usually you need to do something like:
var model = from p in Product.Products.Include("SomeNavProperty")
.... (rest of query)
Where SomeNavProperty is the entity type that loads pTFi_fixedRate.
I have run sqlmetal.exe agaisnt my database.
SqlMetal.exe /server:server /database:dbname /code:mapping.cs
I have included this into my solution. So I can now create an object for each of the database tables. Great. I now wish to use ling to query by database. Can I presume that none of the connection etc is handled by the output of sqlmetal.exe. If this is correct what ways can I use ling to query my database?
Does the generated code include a Data Context (a class which inherits from System.Data.Linq.DataContext)? If so, then that's probably what you're looking for. Something like this:
var db = new SomeDataContext();
// You can also specify a connection string manually in the above constructor if you want
var records = db.SomeTable.Where(st => st.id == someValue);
// and so on...