I'm trying to build a recommendation engine using predictionIO's similar product template. I modified the template to recommend user's instead of items by following this article. Now if I train my model for the first time by importing the sample data provided it works successfully.
If I add a couple of more entries to the event server and if I try to train my model again by using command pio train, I'm getting the following error :
[WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeSystemLAPACK
[WARN] [LAPACK] Failed to load implementation from: com.github.fommil.netlib.NativeRefLAPACK
Exception in thread "dag-scheduler-event-loop" java.lang.StackOverflowError
at java.io.ObjectStreamClass$WeakClassKey.<init>(ObjectStreamClass.java:2451)
at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:336)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:468)
at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1128)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1496)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
at scala.collection.immutable.List$SerializationProxy.writeObject(List.scala:468)
How does pio train works? Do I need to delete the previously trained model and train again?
Can someone please help with this issue?
I faced the same issue.
From my understanding this is due to lack of resources from a hardware perspective. (Probably you need more CPUs or RAM).
What you can do is reduce the number of iterations. (WARNING: This will most probably reduce the accuracy of your output).
To do this, edit your engine.json file. There you can find a value called "numIterations".
Decrease that value.
I initially halved the value (from 10 to 5), just to make sure it worked.
I also faced the same problem and found the solution in the following article.
https://medium.freecodecamp.org/building-an-recommendation-engine-with-apache-prediction-io-ml-server-aed0319e0d8
Decrease the number of iterations in engine.json and rebuild the app before training.
"numIterations": 5,
Related
I'm repeatedly converting an object to a String using the Jackson writeValueAsString method, like a couple of thousand times repeatedly. The size of JSON would be something around 1KB. But after a while, my program exits and throws OOM Exception. Following is the stacktrace:
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:51)
Caused by: java.lang.OutOfMemoryError: Java heap space
at com.fasterxml.jackson.core.util.TextBuffer.carr(TextBuffer.java:864)
at com.fasterxml.jackson.core.util.TextBuffer.expand(TextBuffer.java:825)
at com.fasterxml.jackson.core.util.TextBuffer.append(TextBuffer.java:590)
at com.fasterxml.jackson.core.io.SegmentedStringWriter.write(SegmentedStringWriter.java:58)
at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator._writeString2(WriterBasedJsonGenerator.java:1013)
at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator._writeString(WriterBasedJsonGenerator.java:982)
at com.fasterxml.jackson.core.json.WriterBasedJsonGenerator.writeString(WriterBasedJsonGenerator.java:377)
at com.fasterxml.jackson.databind.ser.std.StringSerializer.serialize(StringSerializer.java:41)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serializeFields(MapSerializer.java:718)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:639)
at com.fasterxml.jackson.databind.ser.std.MapSerializer.serialize(MapSerializer.java:33)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider._serialize(DefaultSerializerProvider.java:480)
at com.fasterxml.jackson.databind.ser.DefaultSerializerProvider.serializeValue(DefaultSerializerProvider.java:319)
at com.fasterxml.jackson.databind.ObjectMapper._configAndWriteValue(ObjectMapper.java:3893)
at com.fasterxml.jackson.databind.ObjectMapper.writeValueAsString(ObjectMapper.java:3207)
at com.ad2pro.neonmigration.neondatamigration.utils.NeonMetricsProducerUtil.produceImpressions(NeonMetricsProducerUtil.java:121)
at com.ad2pro.neonmigration.neondatamigration.scheduler.NeonScheduler.gerMetrics(NeonScheduler.java:100)
at com.ad2pro.neonmigration.neondatamigration.NeonDataMigrationApplication.main(NeonDataMigrationApplication.java:18)
... 8 more
java.lang.OutOfMemoryError: Java heap space
at javax.crypto.CipherSpi.bufferCrypt(CipherSpi.java:814)
at javax.crypto.CipherSpi.engineUpdate(CipherSpi.java:555)
at javax.crypto.Cipher.update(Cipher.java:2002)
at sun.security.ssl.CipherBox.decrypt(CipherBox.java:544)
at sun.security.ssl.EngineInputRecord.decrypt(EngineInputRecord.java:200)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:974)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at com.amazon.support.channels.TLSSocketChannel.read(Unknown Source)
at com.amazon.jdbc.communications.InboundMessagesThread.run(Unknown Source)
~
There is 1GB of free memory before my program starts. Is objectmapper holding onto a lot of memory that even 1GB is not sufficient to convert objects to String. Any help is appreciated.
You could try to set JVM heap passing the parameters -Xmx and -Xms. For example tell to the Jvm to take a maximium of 512 mb of heap -Xms512m.
Default max heap size is 64 mb, maybe that's not enough for your program.
There are multiple aspects to this issue:
As mentioned in above answer ,Increase the JVM heap to take 512 mb.
Check whether conversion to String using objectMapper object is not
created everytime it is called.
writeAsString should not result in unending string.
use visualVM and check what exactly is causing increase in heap.
Using Command line edition for Processing 0269 (Java Mode) running in macOS 10.14.6's Terminal.app.
I'm using processing-java to run a Processing program which is producing an error. Accordingly the following stack trace is printed:
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
at processing.core.PApplet.runSketch(PApplet.java:10845)
at processing.core.PApplet.main(PApplet.java:10613)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at processing.core.PApplet.runSketch(PApplet.java:10839)
... 1 more
Caused by: java.lang.NullPointerException
at treibsand$Runtime.averageHistory(treibsand.java:72)
at treibsand$Analyzer.<init>(treibsand.java:204)
at treibsand.<init>(treibsand.java:105)
... 6 more
Now this information would probably be useful if it weren't for the literal ... 6 more being shown - instead of the rest of the stack trace.
The same stack trace is shown when running the program in the Processing IDE.
How can I see the entire stack trace?
Processing is not cutting anything off. This is part of the Java standard. Please see here for more info.
It shouldn't really matter though. The top of the stack trace almost always contains the information you're looking for. In your case, you're hitting a NullPointerException on line 72 of treibsand.java (which, by the way, should be named with an upper-case letter to follow standard naming conventions).
If you really need more information about exactly what's happening, then you can add your own logging just above that line and print out anything you want, including the full stack trace. But my guess is that won't actually help, as the problem is happening on the line at the top of your stack trace.
This could be caused by something like trying to use a Processing function before the sketch has been initialized. But again, that information will not be lower in the stack trace.
Can I control the max memory consumed by chronicle queue?
I encounter the following exception on 32-bit java process with Xmx1200m parameter:
java.nio.BufferOverflowException
at net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore(MappedBytes.java:147)
at net.openhft.chronicle.bytes.MappedBytes.writeCheckOffset(MappedBytes.java:135)
at net.openhft.chronicle.bytes.AbstractBytes.compareAndSwapInt(AbstractBytes.java:165)
at net.openhft.chronicle.wire.AbstractWire.writeFirstHeader(AbstractWire.java:402)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:514)
at net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:65)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.storeForCycle(SingleChronicleQueue.java:262)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.cycle(SingleChronicleQueueExcerpts.java:1249)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.moveToIndex(SingleChronicleQueueExcerpts.java:1094)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.moveToIndexResult(SingleChronicleQueueExcerpts.java:1080)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.moveToIndex(SingleChronicleQueueExcerpts.java:1073)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.next(SingleChronicleQueueExcerpts.java:828)
at net.openhft.chronicle.queue.impl.single.SingleChronicleQueueExcerpts$StoreTailer.readingDocument(SingleChronicleQueueExcerpts.java:808)
at net.openhft.chronicle.queue.ExcerptTailer.readingDocument(ExcerptTailer.java:41)
at net.openhft.chronicle.wire.MarshallableIn.readBytes(MarshallableIn.java:38)
at com.pingway.platform.tb.InboundQueue.pop(InboundQueue.java:74)
at com.pingway.platform.tb.RecordUpdateExecutor$1.run(RecordUpdateExecutor.java:23)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: java.lang.OutOfMemoryError: Map failed
at net.openhft.chronicle.core.OS.asAnIOException(OS.java:306)
at net.openhft.chronicle.core.OS.map(OS.java:282)
at net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:186)
at net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:141)
at net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore(MappedBytes.java:143)
... 23 more
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method)
at sun.reflect.GeneratedMethodAccessor131.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at net.openhft.chronicle.core.OS.map0(OS.java:290)
at net.openhft.chronicle.core.OS.map(OS.java:278)
... 26 more
If I decrease Xmx to 768m, the exception will disappear.
When running on 32-bit processes, you do need to allow some space for memory mappings. A heap of 1.2 GB is close to the maximum heap size you can have on Win XP (and thus on the 32-bit emulation system on later versions of windows)
What you can do is reduce the block/chunk size from the default of 64 MB to say 1 MB. This will reduce the size of memory mapping.
However, a much better/simpler/faster solution is to use a 64-bit JVM. This will give you about 100,000x more virtual memory in practice.
If you can't use a 64-bit JVM just yet, you can use a Java client connection to Chronicle Engine. This would allow you to run a server with Chronicle Queue running 64-bit, and have a 32-bit client access that data.
on my hortonworks HDP 2.6 cluster, I'm using kite-dataset tool to import data:
./kite-dataset -v csv-import ml-100k/u.data ratings
I'm getting this error:
java.lang.Exception: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:522)
Caused by: java.lang.OutOfMemoryError: Java heap space
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:986)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:402)
at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.<init>(MapTask.java:698)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:770)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
My cluster nodes have 16 GB or RAM, some of which is listed as available.
What can I do to avoid this error?
My first impulse would be to ask what your startup parameters are. Typically, when you run MapReduce and experience an out-of-memory error, you would use something like the following as your startup params:
-Dmapred.map.child.java.opts=-Xmx1G -Dmapred.reduce.child.java.opts=-Xmx1G
The key here is that these two amounts are cumulative. So, the amounts you specificy added together should not come close to exceeding the memory available on your system after you start MapReduce.
So im new to Java programming and i was trying to find an example code of Ball animation in 3D with accurate physics involved.
This is the source:
http://www.java2s.com/Code/Java/3D/AnimationandInteractionaBouncingBall.htm
And here is the error:
Exception in thread "main" java.lang.UnsatisfiedLinkError: no J3D in java.library.path
at java.lang.ClassLoader.loadLibrary(Unknown Source)
at java.lang.Runtime.loadLibrary0(Unknown Source)
at java.lang.System.loadLibrary(Unknown Source)
at javax.media.j3d.MasterControl$22.run(MasterControl.java:889)
at java.security.AccessController.doPrivileged(Native Method)
at javax.media.j3d.MasterControl.loadLibraries(MasterControl.java:886)
at javax.media.j3d.VirtualUniverse.<clinit>(VirtualUniverse.java:229)
at BouncingBall.<init>(BouncingBall.java:81)
at BouncingBall.main(BouncingBall.java:140)
The native libraries are missing and I fear that you use an obsolete version of Java 3D, please follow these instructions.