Oracle decode logic implementation using Slick - oracle

I have following problem - there is sql with DECODE oracle function:
SELECT u.URLTYPE, u.URL
FROM KAA.ENTITYURLS u
JOIN KAA.ENTITY e
ON decode(e.isurlconfigured, 0, e.urlparentcode, 1, e.CODE,
NULL)=u.ENTITYCODE
JOIN CASINO.Casinos c ON e.casinocode = c.code
WHERE e.NAME = $entityName
AND C.NAME = $casinoName
I'm trying to realize this sql in my slick code , like:
val queryUrlsEntityName = for {
entityUrl <- entityUrls
entity <- entities.filter(e => e.name.trim===entityName &&
entityUrl.entityCode.asColumnOf[Option[Int]]==(e.isURLConfigured match
{
case Some(0) => e.urlParentCode
case Some(1) => e.code.asColumnOf[Option[Int]]
case _ => None
}
)
)
casino <- casinos.filter(_.name.trim===casinoName) if
entity.casinoCode==casino.code
} yield (entityUrl)
But I don't understand how can I implement of matching of values in line
case Some(0) => e.urlParentCode
because I'm getting error
constructor cannot be instantiated to expected type;
[error] found : Some[A]
[error] required: slick.lifted.Rep[Option[Int]]
[error] case Some(0) => e.urlParentCode
Thanks for any advice

You should rewrite your code in pattern-matching section so you could compare required Rep[Option[Int]] - to left type, in your case it's Option[Int], or transform Rep[Option[Int]] to Option[Int] type. Rep is only the replacement to the column datatype in slick. I would prefer the first variant - this answer shows how to make the transformation from Rep, or you can use map directly:
map(u => (u.someField)).result.map(_.headOption.match {
case Some(0) => .....
})

Related

read json key-values with hive/sql and spark

I am trying to read this json file into a hive table, the top level keys i.e. 1,2.., here are not consistent.
{
"1":"{\"time\":1421169633384,\"reading1\":130.875969,\"reading2\":227.138275}",
"2":"{\"time\":1421169646476,\"reading1\":131.240628,\"reading2\":226.810211}",
"position": 0
}
I only need the time and readings 1,2 in my hive table as columns ignore position.
I can also do a combo of hive query and spark map-reduce code.
Thank you for the help.
Update , here is what I am trying
val hqlContext = new HiveContext(sc)
val rdd = sc.textFile(data_loc)
val json_rdd = hqlContext.jsonRDD(rdd)
json_rdd.registerTempTable("table123")
println(json_rdd.printSchema())
hqlContext.sql("SELECT json_val from table123 lateral view explode_map( json_map(*, 'int,string')) x as json_key, json_val ").foreach(println)
It throws the following error :
Exception in thread "main" org.apache.spark.sql.hive.HiveQl$ParseException: Failed to parse: SELECT json_val from temp_hum_table lateral view explode_map( json_map(*, 'int,string')) x as json_key, json_val
at org.apache.spark.sql.hive.HiveQl$.createPlan(HiveQl.scala:239)
at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:50)
at org.apache.spark.sql.hive.ExtendedHiveQlParser$$anonfun$hiveQl$1.apply(ExtendedHiveQlParser.scala:49)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:136)
at scala.util.parsing.combinator.Parsers$Success.map(Parsers.scala:135)
at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
This would work, if you rename "1" and "2" (key names) to "x1" and "x2" (inside the json file or in the rdd):
val resultrdd = sqlContext.sql("SELECT x1.time, x1.reading1, x1.reading1, x2.time, x2.reading1, x2.reading2 from table123 ")
resultrdd.flatMap(row => (Array( (row(0),row(1),row(2)), (row(3),row(4),row(5)) )))
This would give you an RDD of tuples with time, reading1 and reading2. If you need a SchemaRDD, you would map it to a case class inside the flatMap transformation, like this:
case class Record(time: Long, reading1: Double, reading2: Double)
resultrdd.flatMap(row => (Array( Record(row.getLong(0),row.getDouble(1),row.getDouble(2)),
Record(row.getLong(3),row.getDouble(4),row.getDouble(5)) )))
val schrdd = sqlContext.createSchemaRDD(resultrdd)
Update:
In the case of many nested keys, you can parse the row like this:
val allrdd = sqlContext.sql("SELECT * from table123")
allrdd.flatMap(row=>{
var recs = Array[Record]();
for(col <- (0 to row.length-1)) {
row(col) match {
case r:Row => recs = recs :+ Record(r.getLong(2),r.getDouble(0),r.getDouble(1));
case _ => ;
}
};
recs
})

Doing Sum() on a set of anonymous type: is it possible?

If I select an anonymous type (select new) from a query:
var smsPerGroup = from g in db.Groups
select new
{
GroupName = g.Name,
ReceivedSMS = g.Members.SelectMany(p => p.ReceivedSMS).Count()
};
It happends that I can't sum() on it:
int max = smsPerGroup.Max(g => g.ReceivedSMS); // ERROR!
The error thrown is:
There was an error parsing the query.
What is the most correct solution for this?
I don't want to create a class to use ONLY one time, on this query.
Is it possible to have a "not very anonymous" type? this is: to define the type of its properties but not the type of the class itself
Something like this: (which is syntactically incorrect)
...
select new
{
string GroupName = g.Name,
int ReceivedSMS = g.Members.SelectMany(p => p.ReceivedSMS).Count()
};
Edit: the error in detail is the following
There was an error parsing the query. [ Token line number = 6,Token line offset = 5,Token in error = SELECT ]
Description: An unhandled exception occurred during the execution of the current web request. Please review the stack trace for more information about the error and where it originated in the code.
Exception Details: System.Data.SqlServerCe.SqlCeException: There was an error parsing the query. [ Token line number = 6,Token line offset = 5,Token in error = SELECT ]
Source Error:
Line 159: };
Line 160:
Line 161: int max = smsPerGroup.Max(g => g.ReceivedSMS);
A simple solution would be to materialize the query with .ToList() before calling .Sum():
var test = (from g in db.Groups
select new
{
GroupName = g.Name,
ReceivedSMS = g.Members.SelectMany(p => p.ReceivedSMS).Count()
}).ToList();
var sum = test.Sum(t => t.ReceivedSMS);

sphinx and multilanguage search || search by attribute

I'm trying to get results from sphinx by attr_string. Here is sphinx configuration:
source db
{-
type = mysql
sql_query = \
SELECT id,language,text,page_url \
FROM content
sql_attr_string = language
sql_attr_string = page_url
}
index content
{
source = db
charset_type = utf-8
min_word_len = 3
}
The results that i'm getting are like this:
[matches] => Array
(
[106] => Array
(
[weight] => 4
[attrs] => Array
(
[page_url] => en/example.gtml
[language] => en
)
)
What I want to do is to filter all results by "language"=en.
$sphinx->SetFilter() is working by integers where in this case I'll need only string "en".
Any help is appreciated!
I found solution...
If anybody need it.
Configure "source" to use crc32, Ex:
source db
{
type = mysql
sql_query = \
SELECT id,crc32(language) as language,text,page_url \
FROM content
sql_attr_uint = language
sql_attr_string = page_url
}
And in client, modify setFilter method to use crc32(). ex:
$s->SetFilter('language',array(crc32('en')));
$result = $s->query('bird is a word','content');
I hope it helps somebody...
more information: http://sphinxsearch.com/docs/current.html#attributes

LINQ Performance Issue on only a few hundred records

in the following code i've commented on the line that SLOWS my page right down. I did some speed test to reveal the CONTAINS LINQ expression is the problem.
Does anyone know how to change this one line to be more efficient using something else instead. I'm also curious as to why its so slow.
Any ideas (thanks in advance):
var allWaste = _securityRepository.FindAllWaste(userId, SystemType.W);
var allWasteIndicatorItems = _securityRepository.FindAllWasteIndicatorItems();
// First get all WASTE RECORDS
var searchResults = (from s in allWaste
join x in allWasteIndicatorItems on s.WasteId equals x.WasteId
where (s.Description.Contains(searchText)
&& s.Site.SiteDescription.EndsWith(searchTextSite)
&& (s.CollectedDate >= startDate && s.CollectedDate <= endDate))
&& x.EWC.EndsWith(searchTextEWC)
select s).Distinct();
var results = searchResults.AsEnumerable();
if (hazardous != "-1")
{
// User has requested to filter on Hazardous or Non Hazardous only rather than Show All
var HazardousBoolFiltered = (from we in _db.WasteIndicatorItems
.Join(_db.WasteIndicators, wii => wii.WasteIndicatorId, wi => wi.WasteIndicatorId, (wii, wi) => new { wasteid = wii.WasteId, wasteindicatorid = wii.WasteIndicatorId, hazardtypeid = wi.HazardTypeId })
.Join(_db.HazardTypes, w => w.hazardtypeid, h => h.HazardTypeId, (w, h) => new { wasteid = w.wasteid, hazardous = h.Hazardous })
.GroupBy(g => new { g.wasteid, g.hazardous })
.Where(g => g.Key.hazardous == true && g.Count() >= 1)
select we).AsEnumerable(); // THIS IS FAST
// Now join the 2 object to eliminate all the keys that do not apply
if (bHazardous)
results = (from r in results join x in HazardousBoolFiltered on r.WasteId equals x.Key.wasteid select r).AsEnumerable(); //This is FAST
else
results = (from r in results.Where(x => !HazardousBoolFiltered
.Select(y => y.Key.wasteid).Contains(x.WasteId)) select r).AsEnumerable(); // This is DOG SLOW 10-15 seconds !--- THIS IS SLOWING EXECUTION by 10 times --!
}
return results.AsQueryable();
I suggest using a logging / tracing framework like smart inspect or log4net combined with a debug text writer.
http://www.codesprouts.com/post/View-LINQ-To-SQL-Statements-Using-A-Debug-TextWriter.aspx
another possibility is to use the sql server profiler and see what sql linq2sql produces.
also a very nice way is to use the mvc mini profiler in combination with the Profiled DB Connection and the SqlFormatters.SqlServerFormatter.
Try Any (MSDN)
Try this:
results = (from r in results
.Where(x => !HazardousBoolFiltered
.Any(y => y.Key.wasteid == r.WasteId)))
.AsEnumerable()
Or Count:
results = (from r in results
.Where(x => HazardousBoolFiltered
.Count(y => y.Key.wasteid == r.WasteId) == 0))
.AsEnumerable()

Error trying to exclude records with a JOIN to another object

In my code below, is there any way I can use the results in the object 'WasteRecordsExcluded' to join with searchResults, essentially excluding the WasteId's I don't want.
If I debug to the last line I get the error :
base {System.SystemException} = {"The query contains references to items defined on a different data context."}
Or if joining is impossible then i could change bHazardous from TRUE to FALSE and FALSE to TRUE and do some kind of 'NOT IN' comparison.
Going bananas with this one, anyone help? Kind Regards :
var allWaste = _securityRepository.FindAllWaste(userId, SystemType.W);
var allWasteIndicatorItems = _securityRepository.FindAllWasteIndicatorItems();
// First get all WASTE RECORDS
var searchResults = (from s in allWaste
join x in allWasteIndicatorItems on s.WasteId equals x.WasteId
where (s.Description.Contains(searchText)
&& s.Site.SiteDescription.EndsWith(searchTextSite)
&& (s.CollectedDate >= startDate && s.CollectedDate <= endDate))
&& x.EWC.EndsWith(searchTextEWC)
select s).Distinct();
var results = searchResults;
if (hazardous != "-1")
{
// User has requested to filter on Hazardous or Non Hazardous only rather than Show All
var WasteRecordsExcluded = (from we in _db.WasteIndicatorItems
.Join(_db.WasteIndicators, wii => wii.WasteIndicatorId, wi => wi.WasteIndicatorId, (wii, wi) => new { wasteid = wii.WasteId, wasteindicatorid = wii.WasteIndicatorId, hazardtypeid = wi.HazardTypeId })
.Join(_db.HazardTypes, w => w.hazardtypeid, h => h.HazardTypeId, (w, h) => new { wasteid = w.wasteid, hazardous = h.Hazardous })
.GroupBy(g => new { g.wasteid, g.hazardous })
.Where(g => g.Key.hazardous == bHazardous && g.Count() >= 1)
select we);
// Now join the 2 object to eliminate all the keys that do not apply
results = results.Where(n => WasteRecordsExcluded.All(t2 => n.WasteId == t2.Key.wasteid));
}
return results;
Maybe something like this:
.....
var results = searchResults.ToList();
.....
.....
.Where(g => g.Key.hazardous == bHazardous && g.Count() >= 1)
select we).ToList();
.....

Resources