I want to run Spark with Kryo serialisation. Therefore I set spark.serializer=org.apache.spark.serializer.KryoSerializer and spark.kryo.registrationRequired=true
When I then run my code I get the error:
Class is not registered: org.apache.spark.sql.catalyst.InternalRow[]
According to this post I used
sc.getConf.registerKryoClasses(Array( classOf[ org.apache.spark.sql.catalyst.InternalRow[_] ] ))
But then the error is:
org.apache.spark.sql.catalyst.InternalRow does not take type parameters
you should use an external class as
class MyRegistrator extends KryoRegistrator {
override def registerClasses(kryo: Kryo) {
kryo.register(classOf[Array[org.apache.spark.sql.catalyst.InternalRow]])
}
}
source : http://spark.apache.org/docs/0.6.0/tuning.html
Or if you want to register in your spark class
val cls: Class[Array[InternalRow]] = classOf[Array[org.apache.spark.sql.catalyst.InternalRow]]
spark.sparkContext.getConf.registerKryoClasses(Array(cls))
I use the first one and works perfectly, I haven't tested the second one.
Related
I'm using Infinispan 13.0.0.final with the default marshaller (protobuf). When I try to use UUID fields in my datatypes
data class CounterState(
#get:ProtoField(number = 1) var index: Long? = null,
#get:ProtoField(number = 2) var uuid: UUID? = null
)
I get the following error at build time:
.../gradle-kotlin-protobuf/build/tmp/kapt3/stubs/main/io/radiosphere/ProtoSchema.java:8: error: org.infinispan.protostream.annotations.ProtoSchemaBuilderException: The class java.util.UUID must be instantiable using an accessible no-argument constructor.
public abstract interface ProtoSchema extends org.infinispan.protostream.GeneratedSchema {
It seems like I'm not allowed to use UUID in my types unless I generate a protoschema for it, but since UUID is a class outside of my control I can't do this.
Previous questions on the topic have gotten the suggestion to use the JavaSerializationMarshaller, but I want to solve this while still using the Protostream Marshaller. It has also been suggested that this would be fixed in version 12.0.0 here.
An example of this not working can be found here. Note that this project will not build because of the annotation processing failing as mentioned above. If it would build the proof that it is working would be shown by running the main project (ie. not the tests).
The question becomes: What do I need to do to configure UUID to be usable in my protobuf marshalled classes in Infinispan 13? Both for embedded and for a program using the hotrod client?
EDIT:
Based on a given answer I have also tried doing the following:
#AutoProtoSchemaBuilder(
includeClasses = [UUIDAdapter::class, CounterState::class],
schemaPackageName = "tutorial")
interface ProtoSchema : GeneratedSchema {
}
This makes the build work, but when starting Quarkus I get the following error:
Caused by: org.infinispan.protostream.DescriptorParserException: Duplicate type id 1005 for type org.infinispan.protostream.commons.UUID. Already used by tutorial.UUID
at org.infinispan.protostream.descriptors.ResolutionContext.checkUniqueTypeId(ResolutionContext.java:151)
at org.infinispan.protostream.descriptors.ResolutionContext.addGenericDescriptor(ResolutionContext.java:97)
at org.infinispan.protostream.descriptors.FileDescriptor.collectDescriptors(FileDescriptor.java:313)
at org.infinispan.protostream.descriptors.FileDescriptor.resolveDependencies(FileDescriptor.java:245)
at org.infinispan.protostream.descriptors.FileDescriptor.resolveDependencies(FileDescriptor.java:210)
at org.infinispan.protostream.descriptors.ResolutionContext.resolve(ResolutionContext.java:57)
at org.infinispan.protostream.impl.SerializationContextImpl.registerProtoFiles(SerializationContextImpl.java:127)
at org.infinispan.protostream.types.java.CommonTypesSchema.registerSchema(CommonTypesSchema.java:49)
at org.infinispan.client.hotrod.RemoteCacheManager.registerSerializationContextInitializer(RemoteCacheManager.java:422)
at org.infinispan.client.hotrod.RemoteCacheManager.registerDefaultSchemas(RemoteCacheManager.java:437)
at org.infinispan.client.hotrod.RemoteCacheManager.initializeProtoStreamMarshaller(RemoteCacheManager.java:409)
at org.infinispan.client.hotrod.RemoteCacheManager.actualStart(RemoteCacheManager.java:365)
at org.infinispan.client.hotrod.RemoteCacheManager.start(RemoteCacheManager.java:334)
at org.infinispan.client.hotrod.RemoteCacheManager.<init>(RemoteCacheManager.java:192)
at org.infinispan.client.hotrod.RemoteCacheManager.<init>(RemoteCacheManager.java:149)
at io.quarkus.infinispan.client.runtime.InfinispanClientProducer.initialize(InfinispanClientProducer.java:68)
If I instead change to use dependsOn like this:
#AutoProtoSchemaBuilder(
includeClasses = [CounterState::class],
dependsOn = [org.infinispan.protostream.types.java.CommonTypes::class, org.infinispan.protostream.types.java.CommonContainerTypes::class],
schemaPackageName = "tutorial")
I'm back to the build failing with:
error: org.infinispan.protostream.annotations.ProtoSchemaBuilderException: The class java.util.UUID must be instantiable using an accessible no-argument constructor.
public abstract interface ProtoSchema extends org.infinispan.protostream.GeneratedSchema {
It seems to be like Quarkus and the Annotation processor are getting in each others way here when it comes to having a simple working solution for UUID marshalling.
You have to include the org.infinispan.protostream.types.java.util.UUIDAdapter class in your annotation:
#AutoProtoSchemaBuilder(includeClasses = [CounterState::class, UUIDAdapter::class] , schemaPackageName = "tutorial")
For more info, check the documentation page.
I have already declared Define_Module(veins:: TraCIDateDissemination) and go through with various suggestions provided but getting this error again. Any suggestion is welcome. please help.
How to register a new class (new application) in Omnet++
Error in module during network setup:
Class "simpleModule" not found in omnetpp
https://doc.omnetpp.org/omnetpp/manual/
In manual I checked -
7.12.3 Class Registration
You should also use the Register_Class() macro to register the new class. It is used by the createOne() factory function, which can create any object given the class name as a string. createOne() is used by the Envir library to implement omnetpp.ini options such as rng-class="..." or scheduler-class="...". (see Chapter [17])
But not able to understand. pls help.
A runtime error occurred:
Class "TraCIDataDissemination" not found -- perhaps its code was not linked in, or the class wasn't registered with Register_Class(), or in the case of modules and channels, with Define_Module()/Define_Channel() -- in module (omnetpp::cModule) RSUExampleScenario.node[0] (id=13), at t=1s, event #2
Launch a debugger with the following command?
nemiver --attach=20070 &
OMNET.INI
##########################################################
# App Layer #
##########################################################
#*.node[*].applType = "TraCIDemo11p"
*.node[*].applType = "TraCIDataDissemination"
*.node[*].appl.headerLength = 80 bit
TraCIDataDissemination.h
namespace veins {
class TraCIDataDissemination : public DemoBaseApplLayer {
public:
{
//my code
}
}
TraCIDataDissemination.cc
#include "veins/modules/application/traci/TraCIDataDissemination.h"
#include<math.h>
using namespace veins;
Define_Module(veins::TraCIDataDissemination);
//code
TraCIDataDissemination.ned
package org.car2x.veins.modules.application.traci;
import org.car2x.veins.modules.application.ieee80211p.DemoBaseApplLayer;
simple TraCIDataDissemination extends DemoBaseApplLayer
{
#class(TraCIDataDissemination);
#display("i=block/app2");
}
Please Help!!
I followed the instructions in the Template repository to create the Resources database. But now I want to use a custom ResourceInitializer to add Resources to the ResourceGraph. (see my code below)
namespace MyApplication.Resources.MyResource
{
[ResourceInitializer(nameof(DummyInitializer))]
public class DummyInitializer : ResourceInitializerBase
{
...
}
}
But when I go into the runtime console and execute exec ResourceManager initialize, the initializer is not shown in the list:
Output in the runtime window
How do I get the runtime to show me the DummyInitializer?
The Initializers you see in the console, are the ones configured in the resource managers initializer collection. In the MaintenanceWeb you can add an entry for your initializer under: http://localhost/maintenanceweb/#/modules/ResourceManager/configuration?path=ModuleConfig/Initializers
or you can add it the underlying config file.
{
"Initializers": [
{
"PluginName": "DummyInitializer"
}
]
}
Alternatively you can execute it using the resource managers console at: http://localhost/maintenanceweb/#/modules/ResourceManager/console
I hope that helps, let me know if you need further help.
I'm trying to load Configuration from YML. I can load value and I can also load list if these are comma seperated values. But i can't load a typical YML List.
Configuration Class
#Component
#PropertySource("classpath:routing.yml")
#ConfigurationProperties
class RoutingProperties(){
var angular = listOf("nothing")
var value: String = ""
}
Working routing.yml
angular: /init, /home
value: Hello World
Not Working routing.yml
angular:
- init
- home
value: Hello World
Why can't i load the second version of yml / do I have a syntaxt error?
ENV: Kotlin, Spring 2.0.0.M3
As #flyx say, #PropetySource not worked with yaml files. But in spring you may override almost everything :)
PropertySource has additional parameter: factory. It's possible to create your own PropertySourceFactory base on DefaultPropertySourceFactory
open class YamlPropertyLoaderFactory : DefaultPropertySourceFactory() {
override fun createPropertySource(name: String?, resource: EncodedResource?): org.springframework.core.env.PropertySource<*> {
if (resource == null)
return super.createPropertySource(name, resource)
return YamlPropertySourceLoader().load(resource.resource.filename, resource.resource, null)
}
}
And when use this factory in propertysource annotation:
#PropertySource("classpath:/routing.yml", factory = YamlPropertyLoaderFactory::class)
Last that you need is to initialized variable angular with mutableList
Full code sample:
#Component
#PropertySource("classpath:/routing.yml", factory = YamlPropertyLoaderFactory::class)
#ConfigurationProperties
open class RoutingProperties {
var angular = mutableListOf("nothing")
var value: String = ""
override fun toString(): String {
return "RoutingProperties(angular=$angular, value='$value')"
}
}
open class YamlPropertyLoaderFactory : DefaultPropertySourceFactory() {
override fun createPropertySource(name: String?, resource: EncodedResource?): org.springframework.core.env.PropertySource<*> {
if (resource == null)
return super.createPropertySource(name, resource)
return YamlPropertySourceLoader().load(resource.resource.filename, resource.resource, null)
}
}
#SpringBootApplication
#EnableAutoConfiguration(exclude = arrayOf(DataSourceAutoConfiguration::class))
open class Application {
companion object {
#JvmStatic
fun main(args: Array<String>) {
val context = SpringApplication.run(Application::class.java, *args)
val bean = context.getBean(RoutingProperties::class.java)
println(bean)
}
}
}
Kinda old post, i know. But i am at the very same topic right now.
As of now, it seems that PropertySource does indeed work with yaml Files. Given the restriction that it only allows for primitive types (it seems) and it cant handle nested elements. I'm probably gonna dig a bit deeper and update my answer accordingly, but as of now, the accepted answer seems like a functioning workaround.
Well, according to the docs, your YAML file will be rewritten into a property file. The first YAML file becomes:
angular=/init, /home
value=Hello World
While the second one becomes:
angular[0]=init
angular[1]=home
value=Hello World
These are obviously two very different things and therefore behave differently.
Moreover, later in the docs, it is stated that YAML does not even work with #PropertySource:
24.6.4 YAML shortcomings
YAML files can’t be loaded via the #PropertySource annotation. So in the case that you need to load values that way, you need to use a properties file.
That makes me kind of wonder why the first case works for you at all.
The docs say this about the generated …[index] properties:
To bind to properties like that using the Spring DataBinder utilities (which is what #ConfigurationProperties does) you need to have a property in the target bean of type java.util.List (or Set) and you either need to provide a setter, or initialize it with a mutable value, e.g. this will bind to the properties above
So, let's have a look at Kotlin docs: listOf returns a new read-only list of given elements. So the list is not mutable as required by the docs, which I assume is why it doesn't work. Try using a mutable list (since I have never used Kotlin, I cannot give you working code). Also try to declare it as java.util.List if that's possible in Kotlin.
I try out the platform-core-1.0 rc5 Plugin to services by events. Now I write a service in the grails-plugin "listadmin":
package listadmin
class SECO_ListenService {
#grails.events.Listener(topic='getEntriesOfList', namespace='listadmin')
def getEntriesOfList(String intnalListName) {
println "SECO_ListenService"
def Liste aList = Liste.findByInternal_name(intnalListName)
return aList.eintrage.toList()
}
}
This service should return a list for dropdown in an other grails-plugin called "institutionadmin". I want to use this list of the service for a dropdown of a domain-model. I should mention that I use dynamic scaffolding. Now I try to call this event in the domain-model:
package institutionadmin
import org.springframework.dao.DataIntegrityViolationException
class Einrichtung {
Long einrichtungs_type
Long type_of_conzept
int anzahl_gruppen
int anzahl_kinder_pro_Gruppe
String offnungszeiten
static hasMany = [rooms : Raum]
static constraints = {
def aList = []
def reply = event(for:"listadmin", topic:"getEntriesOfList", data:"einrichtung_type").waitFor()
aList = reply.value.toList()
einrichtungs_type(inList: aList)
}
}
If I try to run this application i get the following error:
Caused by MissingMethodException: No signature of method: institutionadmin.Einrichtung.event() is applicable for argument types: (java.util.LinkedHashMap) values: [[for:listadmin, topic:testEventBus]]
Possible solutions: ident(), every(), every(groovy.lang.Closure), count(), get(java.io.Serializable), print(java.lang.Object)
If call this event in a controller everything is fine and the documentation of this plugin describe that I can call events also in domain-models and services... This error-method tell me, that the class don't know the event method.
Do I have to configure anything else?
Should call the event in another way or where is my mistake?
Has anybody experiences with this module?
The event(...) dynamic methods are not available on class (static) level.
You can pull the grailsEvents spring bean and call its event() method alternatively. You still have to get the bean from the application context statically though.
You could also use a custom validator instead, as you can get the current domain instance as a parameter, which should have the event() method injected.
something like this :
static myList = []
static constraints = {
einrichtungs_type validator: { value, instance ->
if(!myList){
// cache it the first time you save/validate the domain
// I would probably recommend you NOT to do this here though in
// real life scenario
def reply = instance.event('blabla').get()
myList = reply.value.toList()
}
return value in myList
}
}
Anyway, In my case I would probably load the list elsewhere (in the Bootstrap.groovy for instance) and use it / inject it in my domain instead of doing in the constraints closure.
I faced similar kind of problem, I wanted to use the event call inside a service class which is going to call the listener in other service class. When I started my application I got the same error.What I did was, added the plugin(platform-core:1.0.RC5) entries in BuildConfig.groovy like below
plugins {
build(":tomcat:$grailsVersion",
":platform-core:1.0.RC5") {
export = false
}
compile ':platform-core:1.0.RC5'
runtime ':platform-core:1.0.RC5'
}
Then I ran grails > clean and grails > compile on that project and restarted the server.It started working. Might be you can give a try.