Screens:
Now
Earlier
We've had to fork https://github.com/jinzhu/gorm (gorm v1) because of database driver. And now gorm prints logs without full log path. You can see details in screenshots.
How can we make correct output from the gorm logs?
$GOPATH/src/your_git.com/test/gorm/utils.go
var goSrcRegexp = regexp.MustCompile(`jinzhu/gorm(#.*)?/.*.go`)
var goTestRegexp = regexp.MustCompile(`jinzhu/gorm(#.*)?/.*test.go`)
replace jinzhu/gorm with your own
Related
I already set up key vault scope in the notebooks and I established the connection to the storage account using the following steps:
spark.conf.set("fs.azure.account.auth.type."+StorageAccountName+".dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type."+StorageAccountName+".dfs.core.windows.net","org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id."+StorageAccountName+".dfs.core.windows.net",clientId)
spark.conf.set("fs.azure.account.oauth2.client.secret."+StorageAccountName+".dfs.core.windows.net",clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint."+StorageAccountName+".dfs.core.windows.net","https://login.microsoftonline.com/mytenantid/oauth2/token")
The values of "StorageAccountName", "clientId", "clientSecret" all come from key vault and I am able to get their value properly. In my storage account access control I also assigned the
Storage Blob Data Contributor role to my service principal.
After these configurations, I assigned a connection variable:
var apptable = "abfss://container#"+StorageAccountName+".dfs.core.windows.net/path/to/data"
If I run the following command, I am able to see the files in the blob storage
display(dbutils.fs.ls(apptable))
I am also able to check the schema:
var df = spark.read.format("delta").load(apptable)
df.printSchema()
but if I tried to run the following query:
var last_appt = spark.sql(s"""select max(updateddate) from apptable""").collect()(0).getTimestamp(0)
I got the error:
KeyProviderException: Failure to initialize configuration
Caused by: InvalidConfigurationValueException: Invalid configuration value detected for fs.azure.account.key
I researched online and seems there are some issues in the spark configs. But if it failed to get access to the storage, how come the above display command is running well? What could be possibly missing in such scenario?
I have limited experience on databricks. Appreciate any help.
I tried to reproduce the same in my environment and got the below results and I configure same as mentioned above.
Please follow below code:
Read spark dataframe df.
var df = spark.read.format("delta").load(apptable)
Create temp table:
%scala
temp_table_name = "demtb"
df.createOrReplaceTempView(temp_table_name)
Now, using below code. I got this output.
%scala
val aa= spark.sql("""select max(marks) from demtb""")
display(aa)
Update:
As mentioned, in below comment its working fine for me.
df1.write.mode("overwrite").format("parquet").option("path","/FileStore/dd/").option("overwriteschema","true").saveAsTable("app")
And also, you can try this syntax for configuring azure gen2.As per requirement you can change file format. For demo I'm using csv.
spark.conf.set("fs.azure.account.key.<storage_account_name>.dfs.core.windows.net","Access_key")
Scala
%scala
val df1 = spark.read.format("csv").option("header", "true").load("abfss://pool#vamblob.dfs.core.windows.net/")
display(df1)
Python
df1 = spark.read.format("csv").option("header", "true").load("abfss://pool#vamblob.dfs.core.windows.net/")
display(df1)
I have a following conditional update returning false. But when I check in the database the columns I was trying to update are in fact updated.
def deliver(d: Delivery, placedDate: java.time.LocalDate, locationKey: String, vendorId: String, orderId: String, code: String, courierId: String, courierName: String) = {
update.
where(_.placedDate eqs placedDate).
and(_.locationKey eqs locationKey).
and(_.vendorId eqs vendorId).
and(_.orderId eqs orderId).
modify(_.status setTo "DELIVERED").
and(_.deliveredTime setTo LocalDateTime.now()).
onlyIf(_.status is "COLLECTED").and(_.deliveryCode is code).future().map(_.wasApplied)
}
Thank you
This is a pass through value for the phantom driver, which means that the Datastax Java Driver underneath is the one generating this. If you want to follow this up, could you please post a full bug on GitHub?
Meanwhile, I would suggest not relying on wasApplied if you are simply trying to test, and instead doing a direct read.
You generate some test data and the updated values, perform the update, and compare the final results in Cassandra by reading back. There are known problems with wasApplied with conditional batch updates, but aside from that I'm expecting this to work.
I have created f# solution and added one class library. Only one project in the solution and 5 files and 20 lines of code in each file. Still it will take more 2 minutes to build each time.
I have tried to clean solution.
Also created new solution and project and includes same files, still it will take same time to build it.
Note : First I have created it as a Console Application then convert it into the Class Library.
Edit: Code Sample `
open System
open Configuration
open DBUtil
open Definitions
module DBAccess =
let GetSeq (sql: string) =
let db = dbSchema.GetDataContext(connectionString)
db.DataContext.CommandTimeout <- 0
(db.DataContext.ExecuteQuery(sql,""))
let GetEmployeeByID (id: EMP_PersonalEmpID) =
GetSeq (String.Format("EXEC [EMP_GetEntityById] {0}",id.EmployeeID)) |> Seq.toList<EMP_PersonalOutput>
let GetEmployeeListByIDs (id : Emp_PersonalInput) =
GetSeq (String.Format("EXEC [EMP_GetEntityById] {0}",id.EmployeeID)) |> Seq.toList<EMP_PersonalOutput>`
configuration code snippets : `open Microsoft.FSharp.Data.TypeProviders
module Configuration =
let connectionString = System.Configuration.ConfigurationManager.ConnectionStrings.["EmpPersonal"].ConnectionString
//for database,then stored procedure, the getting the context,then taking the employee table
type dbSchema = SqlDataConnection<"", "EmpPersonal">
//let db = dbSchema.GetDataContext(connectionString)
type tbEmpPersonal = dbSchema.ServiceTypes.EMP_Personal`
Okay, seeing your actual code, I think the main problem is that the type provider connects to the database every time to retrieve the schema. The way to fix this is to cache the schema in a dbml file.
type dbSchema = SqlDataConnection<"connection string...",
LocalSchemaFile = "myDb.dbml",
ForceUpdate = false>
The first time, the TP will connect to the database as usual, but it will also write the schema to myDb.dbml. On subsequent compiles, it will load the schema from myDb.dbml instead of connecting to the database.
Of course, this caching means that changes to the database are not reflected in the types. So every time you need to reload the schema from the database, you can set ForceUpdate to true, do a compile (which will connect to the db), and set it back to false to use the updated myDb.dbml.
Edit: you can even commit the dbml file to your source repository if you want. This will have the additional benefit to allow collaborators who don't have access to a development version of the database to compile the solution anyway.
This answer about NGEN helped me once, but the build time of F# is still terrible compared to C#, just not minutes.
I've installed titan-0.5.0-hadoop2 with hbase and elasticsearch support
I've loaded the graph with
g = TitanFactory.open('conf/titan-hbase-es.properties')
==>titangraph[hbase:[127.0.0.1]]
and a then I loaded the test application
GraphOfTheGodsFactory.load(g)
Now when I'm trying to create a new index key with:
g.makeKey('userId').dataType(String.class).indexed(Vertex.class).unique().make()
and I got this error:
No signature of method: groovy.lang.MissingMethodException.makeKey() is applicable for argument types: () values: []
Possible solutions: every(), any()
Display stack trace? [yN]
Can someone help me with this ?
when I want to see the indexed keys I see this
g.getIndexedKeys(Vertex.class)
==>reason
==>age
==>name
==>place
I'm not completely following what you are trying to do. It appears that you loaded Graph of the Gods to g and then you want to add userId as a new property to the schema. If that's right, then i think your syntax is wrong, given the Titan 0.5 API. The method for managing the schema is very different from previous versions. Changes to the schema are performed through the ManagementSystem interface which you can get an instance of through:
mgmt = g.getManagementSystem()
The syntax for adding a property then looks something like:
birthDate = mgmt.makePropertyKey('birthDate').dataType(Long.class).cardinality(Cardinality.SINGLE).make()
mgmt.commit()
Note that g.getIndexKeys(Class) is not the appropriate way to get schema information either. You should use the ManagementSystem for that too.
Please see the documentation here for more information.
I'm working on a ruby on rails app which connects to Jira through jira soap. I browsed through the Jira SOAP API documentation but could not figure out a way to log time for an issue. What method(s) do I've to use to allow users to log time for some issue?
addWorklogAndAutoAdjustRemainingEstimate is the one I'm using in my scripts
This is a snippet of python code that might be useful to you:
d = datetime.datetime.now()
timestamp = "%s-%02d-%02dT%02d:%02d:00+01:00"%(d.year,d.month,d.day,d.hour,d.minute);
log = self.client.factory.create("ns0:RemoteWorklog")
log.comment = comment # a string
log.timeSpent = time # in the usual format (2h30m )
log.startDate = timestamp
# task is the name of the tast (a string)
self.client.service.addWorklogAndAutoAdjustRemainingEstimate(self.auth,task,log)
Via the addWorkLogXxx methods, where Xxx is what to do with the estimate.
for ruby developers,
worklog class from jira4r gem can be accessed via : Jira4R::V2::RemoteWorklog