Can Azure Data Factory transform a string from Oracle (an xml) to a JSon Object to save in a collection in Cosmos as a Object?
I've tried, but I only get a string (json object) as a simple attribute in Cosmos DB.
Per my experience, azure data factory can transfer data, but will not help you do some serialization steps. So, you data stored in cosmos db maybe like "info": "{\"Id\":\"1\",\"Name\":\"AA\",\"Address\":\"HQ - Main Line\"}".
To handle with this solution, I suggest you using Azure Function Cosmos DB Trigger. Please refer to my code:
using System;
using System.Collections.Generic;
using Microsoft.Azure.Documents;
using Microsoft.Azure.Documents.Client;
using Microsoft.Azure.WebJobs;
using Microsoft.Azure.WebJobs.Host;
using Newtonsoft.Json.Linq;
namespace ProcessJson
{
public class Class1
{
[FunctionName("DocumentUpdates")]
public static void Run(
[CosmosDBTrigger(databaseName:"db",collectionName: "item", ConnectionStringSetting = "CosmosDBConnection",LeaseCollectionName = "leases",
CreateLeaseCollectionIfNotExists = true)]
IReadOnlyList<Document> documents,
TraceWriter log)
{
log.Verbose("Start.........");
String endpointUrl = "https://***.documents.azure.com:443/";
String authorizationKey = "***";
String databaseId = "db";
String collectionId = "import";
DocumentClient client = new DocumentClient(new Uri(endpointUrl), authorizationKey);
for (int i = 0; i < documents.Count; i++)
{
Document doc = documents[i];
String info = doc.GetPropertyValue<String>("info");
JObject o = JObject.Parse(info);
doc.SetPropertyValue("info", o);
client.ReplaceDocumentAsync(UriFactory.CreateDocumentUri(databaseId, collectionId, doc.Id), doc);
log.Verbose("Update document Id " + doc.Id);
}
}
}
}
In addition,you could refer to my previous case:Azure Cosmos DB SQL - how to unescape inner json property
Hope it helps you.
Related
I'm using LINQ to execute a query on a List type variable with a large amount of data (over a million). For performance purposes I'm using IEnumerable to store the results but when I try to access it there is a slight delay.
Specifically I want to see if the query produced any results, but when I use the .Count() or .Any() functions the performance drops.
I read that for IEnumerable types the execution of the query happens at the time of need, hence the delay. Is there a way to see if the IEnumerable has elements inside it without having that much delay?
This is what I'm trying to run.
IEnumerable<Entity> matchingEntities = entities.Where(e => e.Names.Any(n => myEntity.Names.Any(entityName => entityName.CompareNameObjects(n))));
and here are my classes
public class Entity
{
public string EntityIdentifier { get; set; }
public List<Name> Names { get; set; }
}
public class Name
{
public string FullName { get; set; }
public string NameType { get; set; }
public bool CompareNameObjects(Name name2)
{
return FullName == name2.FullName &&
NameType == name2.NameType;
}
}
entities is a list of all my objects and I want to check if myEntity has any Names identical with another entity in the set.
EDITED:
The data structure is similar to the 2 classes (Entity and Name). The entities are created by selecting all the entities, along with their names, from the database in XML format and then I convert the XML to a List as such:
List<Entity> entities = new List<Entity>();
using (SqlConnection conn = new SqlConnection(ConfigurationManager.ConnectionStrings["myCS"].ConnectionString))
{
conn.Open();
SqlCommand cmd = new SqlCommand("GetAllEntities", conn);
cmd.CommandType = CommandType.StoredProcedure;
string entitiesXml = "";
using (SqlDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
{
entitiesXml += rdr["XmlString"].ToString();
}
}
using (TextReader reader = new StringReader(entitiesXml))
entities = (Entity)xmlSerializer.Deserialize(reader);
conn.Close();
}
GetAllEntities (Stored Procedure):
declare #xmlString nvarchar(max) =(
select e.EntityIdentifier,
(
select n.[Full Name] as 'FullName',
n.[Name Type] as 'NameType'
from tblNames n
where e.EntityID=n.[Entity_ID]
for xml path('Name'), type
)
from tblEntities e
order by e.EntityID
for xml path('Entity')
)
select #xmlString as XmlString
Basically, you should avoid getting all data from your database then filter it with C# code. It consumes a lot of effort.
However, for quick solution, you can improve performance by preparing your conditions in a Dictionary form firstly.
// Let's say you have myEntity here
var myEntity = new Entity();
var entities = new List<Entity>();
// You should prepare the list of name that you wanna to find before you do it so that you don't have to make it repeatedly for every iteration
var names = myEntity.Names.Select(p=> p.FullName + p.NameType ).ToDictionary(p=>p, p=>p);
IEnumerable<Entity> matchingEntities = entities.Where(e => e.Names.Any(n => names.ContainsKey(n.FullName + n.NameType)));
This is just an example that may give you more idea. You can improve much more. I hope it can help you
I am new to Web API, Entity Framework and OData. I asked a similar question in another forum but haven't gotten a relevant response.
We have a OData compliant web api service for use in Salesforce. We have a custom complex query in Oracle that we need to expose.
I am not sure how to use a custom query like we want to also allow for odata parameter filtering to occur? ($filter, $top, $skip, etc) For example, when a $filter is used i want to apply that filter to the custom query and then send it back to the database to have it return the result set. How can i do this?
The issue i seem to have is that I can see the parameters as they come in but they are not translating to the query being passed to oracle. It seems that it will fire the query returning the full result set and then apply the parameters. This is very slow as the result set is very large.
I am hoping 2 figure out 2 things
1. How can i use custom sql and apply odata parameters to the underlying query?
2. When using EF or a custom query, how can i apply odata parameters to the query so that when the query is sent to the database that the $filter parameter, for example, is included in the query? I don't want the full result returned then apply the filter.
Can anyone give me some pointers on how to make this happen?
private static ODataValidationSettings _validationSettings = new ODataValidationSettings();
//public IHttpActionResult GetName()
//{ }
// GET: odata/ShareData
[ODataRoute("Orders")]
[EnableQuery(PageSize = 50)]
public IHttpActionResult GetOrders(ODataQueryOptions<Orders> queryOptions)
{
// validate the query.
try
{
queryOptions.Validate(_validationSettings);
}
catch (ODataException ex)
{
return BadRequest(ex.Message);
}
try
{
string connectionString = ConfigurationManager.ConnectionStrings["DNATestConnectionString"].ConnectionString;
var items = GetDataItems(connectionString);
return Ok<IEnumerable<Orders>>(items);
}
catch (Exception ex)
{
return StatusCode(HttpStatusCode.InternalServerError);
}
}
#region Load Data Methods
private static List<Orders> GetDataItems(string connectionString)
{
List<Orders> items = new List<Orders>();
using (OracleConnection con = new OracleConnection(connectionString))
{
con.Open();
using (OracleCommand cmd = con.CreateCommand())
{
cmd.CommandText = "select po_header_id, segment1, vendor_id, vendor_site_id from po_headers_all where vendor_id=4993";
using (OracleDataReader rdr = cmd.ExecuteReader())
{
while (rdr.Read())
items.Add(ToOrders(rdr));
}
}
}
return items;
}
private static Orders ToOrders(OracleDataReader rdr)
{
Orders data = new Orders();
data.VENDOR_ID = ToInt32(rdr, "VENDOR_ID");
data.VENDOR_SITE_ID = ToInt32(rdr, "VENDOR_SITE_ID");
data.PO_HEADER_ID = ToInt32(rdr, "PO_HEADER_ID");
data.SEGMENT1 = Convert.ToString(rdr["SEGMENT1"]);
return data;
}
private static int ToInt32(OracleDataReader rdr, string name)
{
int index = rdr.GetOrdinal(name);
return rdr.IsDBNull(index) ? 0 : Convert.ToInt32(rdr[index]);
}
#endregion
I don't think this is possible.
How can i use custom sql and apply odata parameters to the underlying query?
As far as I'm aware, you can't. The whole point of the OData library is that it needs to work off an IQueryable. By using custom SQL in a string like you have in your example, you can't combine it with the OData parameters that are being passed in.
One approach would be to have your custom SQL in a SQL view, then add the SQL view to your EF model in the same way as you would add a table - it will be represented as a DbSet just like tables are.
You can then get an IQueryable to represent the dataset and then apply the OData parameters as follows:
public IHttpActionResult GetOrders(ODataQueryOptions<OrdersView> queryOptions)
{
IQueryable<OrdersView> allData = // ... get the DbSet from entity framework...
// this will apply the OData query to the data set and only pull the data you want from the database
var filteredResults = queryOptions.ApplyTo(allData) as IQueryable<OrdersView>;
return Ok<IQueryable<OrdersView>>(filteredResults);
}
Note: I'm specifically not using Fluent NHibernate but am using 3.x's built-in mapping style. However, I am getting a blank recordset when I think I should be getting records returned.
I'm sure I'm doing something wrong and it's driving me up a wall. :)
Background / Setup
I have an Oracle 11g database for a product by IBM called Maximo
This product has a table called workorder which lists workorders; that table has a field called "wonum" which represents a unique work order number.
I have a "reporting" user which can access the table via the maximo schema
e.g. "select * from maximo.workorder"
I am using Oracle's Managed ODP.NET DLL to accomplish data tasks, and using it for the first time.
Things I've Tried
I created a basic console application to test this
I added the OracleManagedClientDriver.cs from the NHibernate.Driver on the master branch (it is not officially in the release I'm using).
I created a POCO called WorkorderBriefBrief, which only has a WorkorderNumber field.
I created a class map, WorkorderBriefBriefMap, which maps only that value as a read-only value.
I created a console application with console output to attempt to write the lines of work orders.
The session and transaction appear to open correct,
I tested a standard ODP.NET OracleConnection to my connection string
The Code
POCO: WorkorderBriefBrief.cs
namespace PEApps.Model.WorkorderQuery
{
public class WorkorderBriefBrief
{
public virtual string WorkorderNumber { get; set; }
}
}
Mapping: WorkorderBriefBriefMap.cs
using NHibernate.Mapping.ByCode;
using NHibernate.Mapping.ByCode.Conformist;
using PEApps.Model.WorkorderQuery;
namespace ConsoleTests
{
public class WorkorderBriefBriefMap : ClassMapping<WorkorderBriefBrief>
{
public WorkorderBriefBriefMap()
{
Schema("MAXIMO");
Table("WORKORDER");
Property(x=>x.WorkorderNumber, m =>
{
m.Access(Accessor.ReadOnly);
m.Column("WONUM");
});
}
}
}
Putting it Together: Program.cs
namespace ConsoleTests
{
class Program
{
static void Main(string[] args)
{
NHibernateProfiler.Initialize();
try
{
var cfg = new Configuration();
cfg
.DataBaseIntegration(db =>
{
db.ConnectionString = "[Redacted]";
db.Dialect<Oracle10gDialect>();
db.Driver<OracleManagedDataClientDriver>();
db.KeywordsAutoImport = Hbm2DDLKeyWords.AutoQuote;
db.BatchSize = 500;
db.LogSqlInConsole = true;
})
.AddAssembly(typeof(WorkorderBriefBriefMap).Assembly)
.SessionFactory().GenerateStatistics();
var factory = cfg.BuildSessionFactory();
List<WorkorderBriefBrief> query;
using (var session = factory.OpenSession())
{
Console.WriteLine("session opened");
Console.ReadLine();
using (var transaction = session.BeginTransaction())
{
Console.WriteLine("transaction opened");
Console.ReadLine();
query =
(from workorderbriefbrief in session.Query<WorkorderBriefBrief>() select workorderbriefbrief)
.ToList();
transaction.Commit();
Console.WriteLine("Transaction Committed");
}
}
Console.WriteLine("result length is {0}", query.Count);
Console.WriteLine("about to write WOs");
foreach (WorkorderBriefBrief wo in query)
{
Console.WriteLine("{0}", wo.WorkorderNumber);
}
Console.WriteLine("DONE!");
Console.ReadLine();
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName.ToString());
con.Dispose();
Console.WriteLine("Press RETURN to exit.");
Console.ReadLine();
}
catch (Exception ex)
{
Console.WriteLine("Error : {0}", ex);
Console.ReadLine();
}
}
}
}
Thanks in advance for any help you can give!
Update
The following code (standard ADO.NET with OracleDataReader) works fine, returning the 16 workorder numbers that it should. To me, this points to my use of NHibernate more than the Oracle Managed ODP.NET. So I'm hoping it's just something stupid that I did above in the mapping or configuration.
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new Oracle.ManagedDataAccess.Client.OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName);
var cmd = new OracleCommand();
cmd.Connection = con;
cmd.CommandText = "select wonum from maximo.workorder where upper(reportedby) = 'MAXADMIN'";
cmd.CommandType = CommandType.Text;
Oracle.ManagedDataAccess.Client.OracleDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
con.Dispose();
When configuring NHibernate, you need to tell it about your mappings.
I found the answer -- thanks to Oskar's initial suggestion, I realized it wasn't just that I hadn't added the assembly, I also needed to create a new mapper.
to do this, I added the following code to the configuration before building my session factory:
var mapper = new ModelMapper();
//define mappingType(s) -- could be an array; in my case it was just 1
var mappingType = typeof (WorkorderBriefBriefMap);
//use AddMappings instead if you're mapping an array
mapper.AddMapping(mappingType);
//add the compiled results of the mapper to the configuration
cfg.AddMapping(mapper.CompileMappingForAllExplicitlyAddedEntities());
var factory = cfg.BuildSessionFactory();
I was wondering if we can convert a Linq Query on the Entity Framework and save the query to the database by converting it to an Expression Tree and Serializing. Can someone please help me on this and point me in a right direction whether this can be done or not. Any help is greatly appreciated on this.
Thanks,
Ajay.
i released a library for that purpose just yesterday. Serialize.Linq. It serializes linq expressions to xml, json or binary.
using System.Linq.Expressions
using Serialize.Linq.Extensions;
Expression<Func<Person, bool>> query = p => p.LastName == "Miller"
&& p.FirstName.StartsWith("M");
Console.WriteLine(query.ToJson());
Console.WriteLine(query.ToXml());
You could turn the query into a string and then save the string.
This is from an answer by Nick Berardi:
var result = from x in appEntities
where x.id = 32
select x;
var sql = ((System.Data.Objects.ObjectQuery)result).ToTraceString();
The sql generated by the query could be stored and re-used.
Use Sprint.Filter.OData. It converts a Func<T,bool> into string and back to code.
Sample:
public class TestSprintOData
{
public static void Run()
{
// Parse a Func into string
var query = Filter.Serialize<User>(u => u.IsActive && u.Email.Contains("#gmail.com"));
// It'll generate the string "IsActive and substringof('#gmail.com', Email)"
// Convert back to Expression, perhaps on server
var query2 = Filter.Deserialize<User>(query);
// Compiles to Func, so you can use as delegate to Where
var f = query2.Compile();
var list = new List<User>
{
new User{Name="Johnny", IsActive = true, Email = "johnny#gmail.com"},
new User{Name="abc", IsActive = false, Email = ""},
new User{Name="dude", IsActive=true, Email = "dude#gmail.com"}
};
var result = list.Where(f);
}
}
class User
{
public string Name;
public string Phone;
public string Login;
public string Email;
public bool IsActive;
}
You can also use it as a Nuget Package
You may want to consider using Entity SQL rather than LINQ in this case. Entity SQL is a string query that works against your EF conceptual model rather than directly against the database.
I have a Flex application that uses blazeds to connect to a Java backend. Using remoting, I give a call to an API to run a SELECT statement on a table (using conventional JDBC classes) in a Oracle database.
The table has 2 columns:
PRODUCT_CODE of type NVARCHAR2(32) and
DEMAND of type NUMBER(10, 0)
My Java API is as follows:
public List<?> getQueryResult(String query) {
Connection conn = DriverManager.getConnection(connStr, userName, password);
Statement stmt = conn.createStatement();
ResultSet rs = stmt.executeQuery(query);
ArrayList<?> result = new ArrayList<?>();
while(rs.next()) {
Object[] itemArray = new Object[2];
itemArray[0] = rs.getObject(1);
itemArray[1] = rs.getObject(2);
result.add(itemArray);
}
return result;
}
In my Flex side, I have a handler for result event of this remote operation:
private function onResult(e:ResultEvent) : void {
var result:ArrayCollection = (e.result as ArrayCollection);
}
Strangely, the values corresponding to DEMAND column are automatically converted to string (I debugged to find out that in backend, these were BigDecimal)
Any suggestions?
Yes indeed, the BigDecimal from Java is converted to String in ActionScript, as you can read in the developer guide. There is no BigDecimal in ActionScript so it was the only option - you cannot convert any BigDecimal to int or float.
If you are sure that your value represented as NUMBER(10,0) has values in the interval -2,147,483,648 - 2,147,483,647 you can convert it to an int in java - see the code below:
itemArray[1] = ((BigDecimal)rs.getObject(2)).intValue();