Gradle build is too slow - performance
for example if I changed small code I will get this result in gradle console x
processing com/google/android/gms/internal/c$b.class...
processing com/google/android/gms/internal/f.class...
processing com/google/android/gms/internal/iy$1.class...
processing com/google/android/gms/internal/kj.class...
processing com/google/android/gms/internal/qn.class...
processing com/google/android/gms/internal/jl$a.class...
processing com/google/android/gms/internal/qo.class...
processing com/google/android/gms/internal/os.class...
processing com/google/android/gms/internal/c$d.class...
processing com/google/android/gms/internal/qw.class...
processing com/google/android/gms/internal/ke$1.class...
processing com/google/android/gms/internal/qt.class...
processing com/google/android/gms/internal/jt$a$a.class...
processing com/google/android/gms/internal/jv$1.class...
processing com/google/android/gms/internal/ju$a.class...
processing com/google/android/gms/internal/li.class...
processing com/google/android/gms/internal/ih$a$a.class...
processing com/google/android/gms/internal/jq.class...
processing com/google/android/gms/internal/jl$c.class...
processing com/google/android/gms/internal/op.class...
processing com/google/android/gms/internal/r$a$a.class...
processing com/google/android/gms/internal/ke$a.class...
processing com/google/android/gms/internal/jp.class...
processing com/google/android/gms/internal/f$h.class...
processing com/google/android/gms/internal/js.class...
processing com/google/android/gms/internal/qp.class...
processing com/google/android/gms/internal/j.class...
processing com/google/android/gms/internal/kv$b.class...
processing com/google/android/gms/internal/oq$a$a.class...
processing com/google/android/gms/internal/oq.class...
processing com/google/android/gms/internal/o.class...
processing com/google/android/gms/internal/ox.class...
processing com/google/android/gms/internal/iy$2.class...
processing com/google/android/gms/internal/kg.class...
processing com/google/android/gms/internal/r.class...
processing com/google/android/gms/internal/f$i.class...
processing com/google/android/gms/internal/kr.class...
processing com/google/android/gms/internal/jo.class...
processing com/google/android/gms/internal/jc.class...
processing com/google/android/gms/internal/la.class...
processing com/google/android/gms/internal/jl$f.class...
processing com/google/android/gms/internal/ra$d.class...
processing com/google/android/gms/internal/i$a.class...
processing com/google/android/gms/internal/ig.class...
processing com/google/android/gms/internal/p.class...
processing com/google/android/gms/internal/lj.class...
processing com/google/android/gms/internal/f$d.class...
processing com/google/android/gms/internal/jr$a.class...
processing com/google/android/gms/internal/kw.class...
processing com/google/android/gms/internal/jm.class...
processing com/google/android/gms/internal/jb.class...
processing com/google/android/gms/internal/kf$a.class...
processing com/google/android/gms/internal/qu.class...
processing com/google/android/gms/internal/ld.class...
processing com/google/android/gms/internal/kq.class...
processing com/google/android/gms/internal/jr.class...
processing com/google/android/gms/internal/iz$1.class...
processing com/google/android/gms/internal/kc.class...
processing com/google/android/gms/internal/ko.class...
processing com/google/android/gms/internal/kh.class...
processing com/google/android/gms/internal/kn.class...
processing com/google/android/gms/internal/ji.class...
processing com/google/android/gms/internal/qq.class...
processing com/google/android/gms/internal/s$a$a.class...
processing com/google/android/gms/internal/c$f.class...
processing com/google/android/gms/internal/kb.class...
processing com/google/android/gms/internal/jn$a.class...
processing com/google/android/gms/internal/c$i.class...
processing com/google/android/gms/internal/lf.class...
processing com/google/android/gms/internal/n.class...
processing com/google/android/gms/internal/qs.class...
processing com/google/android/gms/internal/iy$3.class...
processing com/google/android/gms/internal/jl$h.class...
processing com/google/android/gms/internal/jj.class...
processing com/google/android/gms/internal/qy.class...
processing com/google/android/gms/internal/ra$c.class...
processing com/google/android/gms/internal/lh.class...
processing com/google/android/gms/internal/jn.class...
processing com/google/android/gms/internal/ke.class...
processing com/google/android/gms/internal/ju.class...
processing com/google/android/gms/internal/jm$a.class...
processing com/google/android/gms/internal/oq$a.class...
processing com/google/android/gms/internal/ko$a.class...
processing com/google/android/gms/internal/j$1.class...
processing com/google/android/gms/internal/f$c.class...
processing com/google/android/gms/internal/jq$a.class...
processing com/google/android/gms/internal/kh$a$a.class...
processing com/google/android/gms/internal/c$c.class...
processing com/google/android/gms/internal/iz.class...
processing com/google/android/gms/internal/c$j.class...
processing com/google/android/gms/internal/lm.class...
processing com/google/android/gms/internal/q.class...
processing com/google/android/gms/internal/lb.class...
processing com/google/android/gms/internal/f$k.class...
processing com/google/android/gms/internal/kv.class...
processing com/google/android/gms/internal/js$a$a.class...
processing com/google/android/gms/internal/f$a.class...
processing com/google/android/gms/internal/kh$a.class...
processing com/google/android/gms/internal/jn$a$a.class...
processing com/google/android/gms/internal/jl$e.class...
processing com/google/android/gms/internal/g.class...
processing com/google/android/gms/internal/iz$b.class...
processing com/google/android/gms/internal/ra$b.class...
processing com/google/android/gms/internal/f$g.class...
processing com/google/android/gms/internal/os$a.class...
processing com/google/android/gms/internal/jm$b.class...
processing com/google/android/gms/internal/qv.class...
processing com/google/android/gms/internal/f$e.class...
processing com/google/android/gms/internal/kl.class...
processing com/google/android/gms/internal/qz.class...
processing com/google/android/gms/internal/kc$1.class...
processing com/google/android/gms/internal/jt$a.class...
processing com/google/android/gms/internal/f$1.class...
processing com/google/android/gms/internal/d$a.class...
processing com/google/android/gms/internal/jh.class...
processing com/google/android/gms/internal/je.class...
processing com/google/android/gms/internal/kv$a.class...
processing com/google/android/gms/internal/f$b.class...
processing com/google/android/gms/internal/b.class...
processing com/google/android/gms/internal/ra.class...
processing com/google/android/gms/internal/iy$b.class...
processing com/google/android/gms/internal/kk.class...
processing com/google/android/gms/internal/ky.class...
processing com/google/android/gms/internal/pu$a.class...
processing com/google/android/gms/internal/ra$a.class...
processing com/google/android/gms/internal/iy$a.class...
processing com/google/android/gms/internal/lg.class...
processing com/google/android/gms/internal/m.class...
processing com/google/android/gms/internal/kr$b.class...
processing com/google/android/gms/internal/c$a.class...
processing com/google/android/gms/internal/qr.class...
processing com/google/android/gms/internal/hb.class...
processing com/google/android/gms/internal/jd.class...
processing com/google/android/gms/internal/iz$a$a.class...
processing com/google/android/gms/internal/ki.class...
processing com/google/android/gms/internal/oo.class...
processing com/google/android/gms/internal/kx.class...
processing com/google/android/gms/internal/c$h.class...
processing com/google/android/gms/internal/l.class...
processing com/google/android/gms/internal/jz.class...
processing com/google/android/gms/internal/pu.class...
processing com/google/android/gms/internal/f$j.class...
processing com/google/android/gms/internal/kp.class...
processing com/google/android/gms/internal/r$a.class...
processing com/google/android/gms/internal/ka.class...
processing com/google/android/gms/internal/ks.class...
processing com/google/android/gms/internal/s$a.class...
processing com/google/android/gms/internal/h.class...
processing com/google/android/gms/internal/i.class...
processing com/google/android/gms/internal/ki$a.class...
processing com/google/android/gms/internal/kz.class...
processing com/google/android/gms/internal/f$l.class...
processing com/google/android/gms/internal/jl$d.class...
processing com/google/android/gms/internal/jv.class...
processing com/google/android/gms/internal/kf.class...
processing com/google/android/gms/internal/f$f.class...
processing com/google/android/gms/internal/c.class...
processing com/google/android/gms/internal/jb$a.class...
processing com/google/android/gms/internal/kt.class...
processing com/google/android/gms/internal/s.class...
processing com/google/android/gms/internal/lc.class...
processing com/google/android/gms/internal/km.class...
processing com/google/android/gms/internal/kd.class...
processing com/google/android/gms/internal/jf.class...
processing com/google/android/gms/internal/kr$a.class...
processing com/google/android/gms/internal/lk.class...
processing com/google/android/gms/internal/qp$a.class...
processing com/google/android/gms/internal/ju$a$a.class...
processing com/google/android/gms/internal/o$a.class...
processing com/google/android/gms/internal/jc$a.class...
processing com/google/android/gms/internal/c$e.class...
processing com/google/android/gms/internal/iy.class...
processing com/google/android/gms/ads/identifier/AdvertisingIdClient.class...
processing com/google/android/gms/ads/identifier/AdvertisingIdClient$Info.class...
processing com/google/android/gms/ads/identifier/AdvertisingIdClient$a.class...
processing com/google/android/gms/dynamic/a.class...
processing com/google/android/gms/dynamic/d.class...
processing com/google/android/gms/dynamic/e.class...
processing com/google/android/gms/dynamic/f.class...
processing com/google/android/gms/dynamic/a$3.class...
processing com/google/android/gms/dynamic/a$5.class...
processing com/google/android/gms/dynamic/a$4.class...
processing com/google/android/gms/dynamic/LifecycleDelegate.class...
processing com/google/android/gms/dynamic/a$7.class...
processing com/google/android/gms/dynamic/a$a.class...
processing com/google/android/gms/dynamic/c$a$a.class...
processing com/google/android/gms/dynamic/a$1.class...
processing com/google/android/gms/dynamic/g.class...
processing com/google/android/gms/dynamic/a$6.class...
processing com/google/android/gms/dynamic/d$a$a.class...
processing com/google/android/gms/dynamic/d$a.class...
processing com/google/android/gms/dynamic/b.class...
processing com/google/android/gms/dynamic/c$a.class...
processing com/google/android/gms/dynamic/g$a.class...
processing com/google/android/gms/dynamic/h.class...
processing com/google/android/gms/dynamic/a$2.class...
processing com/google/android/gms/dynamic/c.class...
processing com/google/android/gms/gcm/GoogleCloudMessaging$1.class...
processing com/google/android/gms/gcm/GoogleCloudMessaging.class...
processing com/google/android/gms/analytics/ExceptionParser.class...
processing com/google/android/gms/analytics/k.class...
processing com/google/android/gms/analytics/Tracker.class...
processing com/google/android/gms/analytics/ad.class...
processing com/google/android/gms/analytics/w$b.class...
processing com/google/android/gms/analytics/w$2.class...
processing com/google/android/gms/analytics/ac.class...
processing com/google/android/gms/analytics/h$1.class...
processing com/google/android/gms/analytics/CampaignTrackingReceiver.class...
processing com/google/android/gms/analytics/w$1.class...
processing com/google/android/gms/analytics/a.class...
processing com/google/android/gms/analytics/StandardExceptionParser.class...
processing com/google/android/gms/analytics/HitBuilders.class...
processing com/google/android/gms/analytics/d.class...
processing com/google/android/gms/analytics/e.class...
processing com/google/android/gms/analytics/w$a.class...
processing com/google/android/gms/analytics/Logger.class...
processing com/google/android/gms/analytics/HitBuilders$HitBuilder.class...
processing com/google/android/gms/analytics/c$b.class...
processing com/google/android/gms/analytics/f.class...
processing com/google/android/gms/analytics/aa.class...
processing com/google/android/gms/analytics/HitBuilders$ExceptionBuilder.class...
processing com/google/android/gms/analytics/n$a.class...
processing com/google/android/gms/analytics/z$a.class...
processing com/google/android/gms/analytics/GoogleAnalytics$a.class...
processing com/google/android/gms/analytics/HitBuilders$SocialBuilder.class...
processing com/google/android/gms/analytics/j.class...
processing com/google/android/gms/analytics/GoogleAnalytics$b.class...
processing com/google/android/gms/analytics/w$d.class...
processing com/google/android/gms/analytics/y.class...
processing com/google/android/gms/analytics/o.class...
processing com/google/android/gms/analytics/HitBuilders$EventBuilder.class...
processing com/google/android/gms/analytics/v$1.class...
processing com/google/android/gms/analytics/r.class...
processing com/google/android/gms/analytics/ai.class...
processing com/google/android/gms/analytics/y$a.class...
processing com/google/android/gms/analytics/x$1.class...
processing com/google/android/gms/analytics/p.class...
processing com/google/android/gms/analytics/Logger$LogLevel.class...
processing com/google/android/gms/analytics/ExceptionReporter.class...
processing com/google/android/gms/analytics/Tracker$a.class...
processing com/google/android/gms/analytics/x$d.class...
processing com/google/android/gms/analytics/TrackerHandler.class...
processing com/google/android/gms/analytics/al.class...
processing com/google/android/gms/analytics/n.class...
processing com/google/android/gms/analytics/z.class...
processing com/google/android/gms/analytics/u.class...
processing com/google/android/gms/analytics/k$1.class...
processing com/google/android/gms/analytics/w.class...
processing com/google/android/gms/analytics/c$c.class...
processing com/google/android/gms/analytics/q.class...
processing com/google/android/gms/analytics/t.class...
processing com/google/android/gms/analytics/CampaignTrackingService.class...
processing com/google/android/gms/analytics/ag$a.class...
processing com/google/android/gms/analytics/ak.class...
processing com/google/android/gms/analytics/x$c.class...
processing com/google/android/gms/analytics/g.class...
processing com/google/android/gms/analytics/v$2.class...
processing com/google/android/gms/analytics/x$b.class...
processing com/google/android/gms/analytics/HitBuilders$TimingBuilder.class...
processing com/google/android/gms/analytics/ab.class...
processing com/google/android/gms/analytics/af.class...
processing com/google/android/gms/analytics/aj.class...
processing com/google/android/gms/analytics/am.class...
processing com/google/android/gms/analytics/b.class...
processing com/google/android/gms/analytics/x$a.class...
processing com/google/android/gms/analytics/ae.class...
processing com/google/android/gms/analytics/v.class...
processing com/google/android/gms/analytics/m.class...
processing com/google/android/gms/analytics/c$a.class...
processing com/google/android/gms/analytics/w$c.class...
processing com/google/android/gms/analytics/l.class...
processing com/google/android/gms/analytics/h.class...
processing com/google/android/gms/analytics/i.class...
processing com/google/android/gms/analytics/w$e.class...
it is normal to take
16:03:08.530 [LIFECYCLE] [org.gradle.BuildResultLogger] Total time: 3 mins 9.729 secs
These are some libraries that I use
compile 'com.android.support:multidex:1.0.0'
compile 'com.android.support:support-v4:21.+"'
/*compile project(':viewPagerIndicator241')*/
compile 'com.google.code.gson:gson:2.2.4'
compile 'com.google.android.gms:play-services-maps:6.5.87'
compile 'com.actionbarsherlock:actionbarsherlock:4.4.0#aar'
compile files('libs/achartengine-1.1.0.jar')
compile files('libs/activation.jar')
compile files('libs/additionnal.jar')
/*compile files('libs/androidannotations-api-3.2.jar')*/
compile files('libs/commons-io-2.4.jar')
/* compile files('libs/commons-lang3-3.2.jar')*/
compile files('libs/jsoup-1.8.1.jar')
compile files('libs/mail.jar')
compile files('libs/mpandroidchartlibrary-1-7-4.jar')
compile files('libs/prettytime-3.2.5.Final.jar')
Try this new build.gradle:
I would highly recommend trying to move away from MultiDex as it is very slow.
repositories {
maven { url "https://jitpack.io" }
}
android {
...
dependencies {
// Gradle dependencies
compile 'com.android.support:multidex:1.0.1' // <-- was 1.0.1
compile 'com.android.support:support-v4:22.0.0' // <-- was 21+
compile 'com.google.code.gson:gson:2.3.1' // <-- was 2.2.4
compile 'com.google.android.gms:play-services-maps:7.0.0' // <-- was 6.5.87
compile 'com.actionbarsherlock:actionbarsherlock:4.4.0#aar' // <-- deprecated, move to AppCompat
compile 'commons-io:commons-io:2.4' // <-- was compile files('libs/commons-io-2.4.jar')
compile 'org.jsoup:jsoup:1.8.1' // <-- was compile files('libs/jsoup-1.8.1.jar')
compile 'com.github.PhilJay:MPAndroidChart:v2.0.8' // <-- was compile files('libs/mpandroidchartlibrary-1-7-4.jar')
compile 'org.ocpsoft.prettytime:prettytime-nlp:3.2.5.Final' // <-- was files('libs/prettytime-3.2.5.Final.jar')
// Jars
compile files('libs/achartengine-1.1.0.jar')
compile files('libs/mail.jar')
compile files('libs/additionnal.jar')
compile files('libs/activation.jar')
}
}
I'd suggest trying to remove some of these dependencies:
Jake Whartons's ActionBarSherlock is DEPRECATED, switch to App Compat
You have 2 chart libraries achartengine and mpandroidchartlibrary
Do you really need Google's gson vs Android's built-in json libs?
Do you really need Common's IO?
Related
Bulk load aborted with some files not yet loaded
I am doing bulk load in Hbase But getting error while loading Hfiles. 11:02:49.909 ERROR org.apache.hadoop.hbase.tool.LoadIncrementalHFiles - ------------------------------------------------- Bulk load aborted with some files not yet loaded: ------------------------------------------------- hdfs://10.13.10.19:8020/madhuri/userOutput/personalDetails/6255a0ffd4e14f70ad2146bb756f004c java.io.IOException: Retry attempted 10 times without completing, bailing out at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:420) at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:343) at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:256) at com.clouds.dbtest.test.HbaseBulkLoad.doBulkLoad(HbaseBulkLoad.java:26) at com.clouds.dbtest.test.HBaseBulkLoadDriver.run(HBaseBulkLoadDriver.java:84) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at com.clouds.dbtest.test.HBaseBulkLoadDriver.main(HBaseBulkLoadDriver.java:41) I have taken code from link.
SystemTap script compilation fails to recognize the kernel.xxx probes
I'm running SystemTap on ArchLinux 4.7.4-1, with the rebuilt kernel as instructed on ArchLinux Wiki. The version of SystemTap is:
$ stap -V
Systemtap translator/driver (version 3.0/0.167, non-git sources)
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 LIBXML2 NLS NSS TR1_UNORDERED_MAP READLINE
$ uname -a
Linux eva-dummy 4.7.4-1-ARCH #1 SMP PREEMPT Sun Sep 25 14:32:13 EDT 2016 x86_64 GNU/Linux
I have the SystemTap script:
#!/usr/bin/stap
probe kernel.function("*#net/socket.c").call
{
printf ("%s -> %s\n", thread_indent(1), probefunc())
}
probe kernel.function("*#net/socket.c").return
{
printf ("%s <- %s\n", thread_indent(-1), probefunc())
}
The output of running with -vvv option is:
$ stap -vvv hello-world.stp
Systemtap translator/driver (version 3.0/0.167, non-git sources)
Copyright (C) 2005-2015 Red Hat, Inc. and others
This is free software; see the source for copying conditions.
enabled features: AVAHI LIBSQLITE3 LIBXML2 NLS NSS TR1_UNORDERED_MAP READLINE
Created temporary directory "/tmp/stap4mY4hm"
Session arch: x86_64 release: 4.7.4-1-ARCH
Parsed kernel "/lib/modules/4.7.4-1-ARCH/build/.config", containing 5396 tuples
Parsed kernel "/lib/modules/4.7.4-1-ARCH/build/Module.symvers", containing 7295 vmlinux exports
Kernel symbol table /lib/modules/4.7.4-1-ARCH/build/System.map unavailable, (No such file or directory)
Parsed kernel "/boot/System.map-4.7.4-1-ARCH", containing 32456 symbols
Processing tapset "/usr/share/systemtap/tapset/linux/context.stpm"
Processing tapset "/usr/share/systemtap/tapset/linux/json.stpm"
Processing tapset "/usr/share/systemtap/tapset/linux/nfs_proc.stpm"
Processing tapset "/usr/share/systemtap/tapset/linux/syscalls.stpm"
Processing tapset "/usr/share/systemtap/tapset/linux/task.stpm"
Searched for library macro files: "/usr/share/systemtap/tapset/linux/*.stpm", found: 5, processed: 5
Processing tapset "/usr/share/systemtap/tapset/choose_defined.stpm"
Processing tapset "/usr/share/systemtap/tapset/container_of.stpm"
Processing tapset "/usr/share/systemtap/tapset/macros.stpm"
Processing tapset "/usr/share/systemtap/tapset/offsetof.stpm"
Processing tapset "/usr/share/systemtap/tapset/print_stats.stpm"
Processing tapset "/usr/share/systemtap/tapset/private30.stpm"
Processing tapset "/usr/share/systemtap/tapset/try_assign.stpm"
Processing tapset "/usr/share/systemtap/tapset/type_defined.stpm"
Searched for library macro files: "/usr/share/systemtap/tapset/*.stpm", found: 8, processed: 8
Processing tapset "/usr/share/systemtap/tapset/linux/x86_64/aux_syscalls.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/x86_64/nd_syscalls.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/x86_64/syscalls.stp"
Searched: "/usr/share/systemtap/tapset/linux/x86_64/*.stp", found: 3, processed: 3
Processing tapset "/usr/share/systemtap/tapset/linux/atomic.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/aux_syscalls.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/context-caller.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/context-envvar.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/context.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/context-symbols.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/context-unwind.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/conversions-guru.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/conversions.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ctime.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/dentry.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/dev.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/endian.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/guru-delay.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/guru-signal.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/inet_sock.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/inet.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ioblock.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ioscheduler.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ipmib-filter-default.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ipmib.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ip.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/irq.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/json.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/kprocess.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/kretprobe.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/linuxmib-filter-default.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/linuxmib.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/loadavg.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/logging.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/memory.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nd_syscalls2.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nd_syscalls.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/netfilter.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/networking.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nfsderrno.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nfsd.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nfs_proc.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/nfs.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/panic.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/perf.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/proc_mem.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/pstrace.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/rcu.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/rlimit.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/rpc.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/scheduler.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/scsi.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/signal.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/socket.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/syscalls2.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/syscalls_cfg_trunc.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/syscalls.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/target_set.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/task_ancestry.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/task.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/task_time.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/tcpmib-filter-default.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/tcpmib.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/tcp.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/timestamp_gtod.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/timestamp_monotonic.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/timestamp.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/tty.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/tzinfo.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ucontext.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ucontext-symbols.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/ucontext-unwind.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/udp.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/utrace.stp"
Processing tapset "/usr/share/systemtap/tapset/linux/vfs.stp"
Searched: "/usr/share/systemtap/tapset/linux/*.stp", found: 71, processed: 71
Processing tapset "/usr/share/systemtap/tapset/x86_64/registers.stp"
Searched: "/usr/share/systemtap/tapset/x86_64/*.stp", found: 1, processed: 1
Processing tapset "/usr/share/systemtap/tapset/ansi.stp"
Processing tapset "/usr/share/systemtap/tapset/argv.stp"
Processing tapset "/usr/share/systemtap/tapset/context.stp"
Processing tapset "/usr/share/systemtap/tapset/errno.stp"
Processing tapset "/usr/share/systemtap/tapset/indent-default.stp"
Processing tapset "/usr/share/systemtap/tapset/indent.stp"
Processing tapset "/usr/share/systemtap/tapset/init.stp"
Processing tapset "/usr/share/systemtap/tapset/java.stp"
Processing tapset "/usr/share/systemtap/tapset/logging.stp"
Processing tapset "/usr/share/systemtap/tapset/null.stp"
Processing tapset "/usr/share/systemtap/tapset/oneshot.stp"
Processing tapset "/usr/share/systemtap/tapset/pn.stp"
Processing tapset "/usr/share/systemtap/tapset/queue_stats.stp"
Processing tapset "/usr/share/systemtap/tapset/random.stp"
Processing tapset "/usr/share/systemtap/tapset/registers.stp"
Processing tapset "/usr/share/systemtap/tapset/speculative.stp"
Processing tapset "/usr/share/systemtap/tapset/stap_staticmarkers.stp"
Processing tapset "/usr/share/systemtap/tapset/stopwatch.stp"
Processing tapset "/usr/share/systemtap/tapset/string.stp"
Processing tapset "/usr/share/systemtap/tapset/switchfile.stp"
Processing tapset "/usr/share/systemtap/tapset/system.stp"
Processing tapset "/usr/share/systemtap/tapset/timers.stp"
Processing tapset "/usr/share/systemtap/tapset/tokenize.stp"
Processing tapset "/usr/share/systemtap/tapset/tzinfo.stp"
Processing tapset "/usr/share/systemtap/tapset/uconversions.stp"
Searched: "/usr/share/systemtap/tapset/*.stp", found: 25, processed: 25
Pass 1: parsed user script and 113 library scripts using 116784virt/38632res/6676shr/32184data kb, in 100usr/0sys/239real ms.
Attempting to extract kernel debuginfo build ID from /lib/modules/4.7.4-1-ARCH/build/vmlinux.id
Attempting to extract kernel debuginfo build ID from /sys/kernel/notes
blacklist regexps:
blfn: ^(atomic_notifier_call_chain|default_do_nmi|__die|die_nmi|do_debug|do_general_protection|do_int3|do_IRQ|do_page_fault|do_sparc64_fault|do_trap|dummy_nmi_callback|flush_icache_range|ia64_bad_break|ia64_do_page_fault|ia64_fault|io_check_error|mem_parity_error|nmi_watchdog_tick|notifier_call_chain|oops_begin|oops_end|program_check_exception|single_step_exception|sync_regs|unhandled_fault|unknown_nmi_error|xen_[gs]et_debugreg|xen_irq_.*|xen_.*_fl_direct.*|check_events|xen_adjust_exception_frame|xen_iret.*|xen_sysret64.*|test_ti_thread_flag.*|inat_get_opcode_attribute|system_call_after_swapgs|HYPERVISOR_[gs]et_debugreg|HYPERVISOR_event_channel_op|hash_64|hash_ptr|native_set_pte|.*raw_.*_lock.*|.*raw_.*_unlock.*|.*raw_.*_trylock.*|.*read_lock.*|.*read_unlock.*|.*read_trylock.*|.*write_lock.*|.*write_unlock.*|.*write_trylock.*|.*write_seqlock.*|.*write_sequnlock.*|.*spin_lock.*|.*spin_unlock.*|.*spin_trylock.*|.*spin_is_locked.*|rwsem_.*lock.*|.*mutex_.*lock.*|raw_.*|atomic_.*|atomic64_.*|get_bh|put_bh|.*apic.*|.*APIC.*|.*softirq.*|.*IRQ.*|.*_intr.*|__delay|.*kernel_text.*|get_current|current_.*|.*exception_tables.*|.*setup_rt_frame.*|.*preempt_count.*|preempt_schedule|__switch_to|special_mapping_.*|.*_pte_.*)$
blfn_ret: ^(do_exit|sys_exit|sys_exit_group)$
blfile: ^(kernel/kprobes\.c|arch/.*/kernel/kprobes\.c|.*/include/asm/io\.h|.*/include/asm/io-defs\.h|.*/include/asm/io_64\.h|.*/include/asm/bitops\.h|drivers/ide/ide-iops\.c|arch/.*/kernel/paravirt\.c|.*/include/asm/paravirt\.h|fs/seq_file\.c)$
blsection: ^(\.init\.|\.exit\.|\.devinit\.|\.devexit\.|\.cpuinit\.|\.cpuexit\.|\.meminit\.|\.memexit\.)
parse '*#net/socket.c', func '*', file 'net/socket.c'
focused on module 'kernel' = [0xffffffff81000000-0xffffffff81b7e000, bias 0 file /lib/modules/4.7.4-1-ARCH/build/vmlinux ELF machine |x86_64 (code 62)
focused on module 'kernel'
selected source file 'net/socket.c'
suggesting 29554 dwarf functions from modules:
kernel
semantic error: while resolving probe point: identifier 'kernel' at hello-world.stp:3:7
thrown from: elaborate.cxx:1068
source: probe kernel.function("*#net/socket.c").call
^
semantic error: no match (similar functions: f, lf, up, die, gcd)
thrown from: tapsets.cxx:8329
parse '*#net/socket.c', func '*', file 'net/socket.c'
focused on module 'kernel' = [0xffffffff81000000-0xffffffff81b7e000, bias 0 file /lib/modules/4.7.4-1-ARCH/build/vmlinux ELF machine |x86_64 (code 62)
focused on module 'kernel'
selected source file 'net/socket.c'
suggesting 29554 dwarf functions from modules:
kernel
semantic error: while resolving probe point: identifier 'kernel' at :8:7
thrown from: elaborate.cxx:1068
source: probe kernel.function("*#net/socket.c").return
^
semantic error: no match (similar functions: f, lf, up, die, gcd)
thrown from: tapsets.cxx:8329
Pass 2: analyzed script: 0 probes, 0 functions, 0 embeds, 0 globals using 118068virt/41172res/7648shr/33468data kb, in 560usr/20sys/967real ms.
Pass 2: analysis failed. [man error::pass2]
Running rm -rf /tmp/stap4mY4hm
Spawn waitpid result (0x0): 0
Removed temporary directory "/tmp/stap4mY4hm"
It seems that SystemTap cannot recognize the kernel.xxxx probes. How to make the script run successfully on ArchLinux?
Spark NullPointerException on SQLListener.onTaskEnd while finishing task
I have a Spark application using Scala which perform series of transformation, then writing the result to parquet file. The transformation part finished without problem, the result output is written to HDFS correctly. The application is running on top of YARN cluster of 30 nodes. However, the Spark application itself will not complete and exit the YARN. It will remain in resource manager. After hanging for about an hour (consuming resources and vcores), then either it finishes or throw an error and killed itself. Here is the error log of the application. Appreciate if anyone can shed some light on this matter. 16/08/24 14:51:12 INFO impl.ContainerManagementProtocolProxy: Opening proxy : phhdpdn013x.company.com:8041 16/08/24 14:51:22 INFO cluster.YarnClusterSchedulerBackend: Registered executor NettyRpcEndpointRef(null) (phhdpdn013x.company.com:54175) with ID 1 16/08/24 14:51:22 INFO storage.BlockManagerMasterEndpoint: Registering block manager phhdpdn013x.company.com:24700 with 2.1 GB RAM, BlockManagerId(1, phhdpdn013x.company.com, 24700) 16/08/24 14:51:29 INFO cluster.YarnClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after waiting maxRegisteredResourcesWaitingTime: 30000(ms) 16/08/24 14:51:29 INFO cluster.YarnClusterScheduler: YarnClusterScheduler.postStartHook done 16/08/24 15:11:00 ERROR scheduler.LiveListenerBus: Listener SQLListener threw an exception java.lang.NullPointerException at org.apache.spark.sql.execution.ui.SQLListener.onTaskEnd(SQLListener.scala:167) at org.apache.spark.scheduler.SparkListenerBus$class.onPostEvent(SparkListenerBus.scala:42) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.scheduler.LiveListenerBus.onPostEvent(LiveListenerBus.scala:31) at org.apache.spark.util.ListenerBus$class.postToAll(ListenerBus.scala:55) at org.apache.spark.util.AsynchronousListenerBus.postToAll(AsynchronousListenerBus.scala:37) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(AsynchronousListenerBus.scala:80) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(AsynchronousListenerBus.scala:65) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at org.apache.spark.util.AsynchronousListenerBus$$anon$1$$anonfun$run$1.apply$mcV$sp(AsynchronousListenerBus.scala:64) at org.apache.spark.util.Utils$.tryOrStopSparkContext(Utils.scala:1181) at org.apache.spark.util.AsynchronousListenerBus$$anon$1.run(AsynchronousListenerBus.scala:63) 16/08/24 15:11:46 ERROR scheduler.LiveListenerBus: Listener SQLListener threw an exception java.lang.NullPointerException aa
What is your version of Spark? Your ERROR looks a lot like this issue https://issues.apache.org/jira/browse/SPARK-12339
Spark Streaming UpdateStateByKey
I am running a spark streaming 24X7 and using updateStateByKey function to save the computed historical data like in the case of NetworkWordCount Example.. I am tried to stream a file with 3lac records with 1 sec sleep for every 1500 records. I am using 3 workers Over a period updateStateByKey is growing, then the program throws the following exception ERROR Executor: Exception in task ID 1635 java.lang.ArrayIndexOutOfBoundsException: 3 14/10/23 21:20:43 ERROR TaskSetManager: Task 29170.0:2 failed 1 times; aborting job 14/10/23 21:20:43 ERROR DiskBlockManager: Exception while deleting local spark dir: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232 java.io.IOException: Failed to delete: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232/24 14/10/23 21:20:43 ERROR Executor: Exception in task ID 8037 java.io.FileNotFoundException: /var/folders/3j/9hjkw0890sx_qg9yvzlvg64cf5626b/T/spark-local-20141023204346-b232/22/shuffle_81_0_1 (No such file or directory) at java.io.FileOutputStream.open(Native Method) How to handle this? I guess updateStateByKey should be periodically reset as its growing in a rapid rate, please share some example on when and how to reset the updateStateByKey.. or i there any other problem? shed some light. Any help is much appreciated. Thanks for your time
Did you set the CheckPoint
ssc.checkpoint("path to checkpoint")
Pig permission denied
I wrote a simple script in pig and it runs good when run from terminal. However when i try to run the script from the browser using apache server, it throws the following error. [main] ERROR org.apache.pig.tools.grunt.Grunt - You don't have permission to perform the operation. Error from the server: dummy (Permission denied) [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. dummy (Permission denied) Any ideas where to fix the problem? Well if it helps, nowhere in my script have I used anything called "dummy"!!! P.S.: changed the permissions in the /tmp hdfs directory to 777, didn't help!! checked the permissions in the /tmp local directory has 777, didn't matter!! tried to use -Dpig.temp.dir to a directory in which apache has permission to write, didn't help!! Please help, did I miss something? Edit: Well worth mentioning: this error comes while i am running the script while checking for compilation of the pig script, i.e., pig -c ScriptName 2013-09-06 10:41:19,344 [main] INFO org.apache.pig.Main - Apache Pig version 0.10.0 (r1328203) compiled Apr 19 2012, 22:54:12 2013-09-06 10:41:19,344 [main] INFO org.apache.pig.Main - Logging error messages to: /data/storage/pig-0.10.0/logs/pig_1378444279340.log 2013-09-06 10:41:19,658 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to hadoop file system at: hdfs://[master]:9000 2013-09-06 10:41:19,743 [main] INFO org.apache.pig.backend.hadoop.executionengine.HExecutionEngine - Connecting to map-reduce job tracker at: [master]:9001 2013-09-06 10:41:19,866 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - created tmp python.cachedir=/tmp/pig_jython_4266457116882300725 *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/lib/tools.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/hadoop-core-0.20.203.0.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/aspectjrt-1.6.5.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/aspectjtools-1.6.5.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-beanutils-1.7.0.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-beanutils-core-1.8.0.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-cli-1.2.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-codec-1.4.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-collections-3.2.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-configuration-1.6.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-configurationhadoop-1.6.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-daemon-1.0.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-digester-1.8.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-el-1.0.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-httpclient-3.0.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-lang-2.4.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-logging-1.1.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-logging-api-1.0.4.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-math-2.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/commons-net-1.4.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/core-3.1.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/guava-r09.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/hadoop-fairscheduler-0.20.203.0.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/hbase-0.94.4.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/hsqldb-1.8.0.10.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/hstreaming-all.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jackson-core-asl-1.0.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jackson-mapper-asl-1.0.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jasper-compiler-5.5.12.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jasper-runtime-5.5.12.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jets3t-0.6.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jetty-6.1.26.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jetty-util-6.1.26.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jsch-0.1.42.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/junit-4.5.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/kfs-0.2.2.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/log4j-1.2.15.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/mockito-all-1.8.5.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/oro-2.0.8.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/protobuf-java-2.4.0a.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/servlet-api-2.5-20081211.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/slf4j-api-1.4.3.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/slf4j-log4j12-1.4.3.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/xmlenc-0.52.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/zookeeper-3.4.5.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jsp-2.1/jsp-2.1.jar' *sys-package-mgr*: processing new jar, '/home/hadoop/hadoop-0.20.203.0/lib/jsp-2.1/jsp-api-2.1.jar' *sys-package-mgr*: processing new jar, '/data/storage/pig-0.10.0/lib/automaton.jar' *sys-package-mgr*: processing new jar, '/data/storage/pig-0.10.0/lib/jython-2.5.0.jar' *sys-package-mgr*: processing new jar, '/data/storage/pig-0.10.0/pig-0.10.0-withouthadoop.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/resources.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/rt.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/jsse.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/jce.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/charsets.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/sunpkcs11.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/sunec.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/sunjce_provider.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/zipfs.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/localedata.jar' *sys-package-mgr*: processing new jar, '/usr/java/jdk1.7.0_03/jre/lib/ext/dnsns.jar' 2013-09-06 10:41:28,086 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: pythonDefpyUDF.[UDF1] 2013-09-06 10:41:28,087 [main] INFO org.apache.pig.scripting.jython.JythonScriptEngine - Register scripting UDF: pythonDefpyUDF.[UDF2] 2013-09-06 10:41:28,266 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2013-09-06 10:41:28,421 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2013-09-06 10:41:28,578 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2013-09-06 10:41:29,665 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2013-09-06 10:41:29,712 [main] WARN org.apache.pig.tools.grunt.GruntParser - 'rm/rmf' statement is ignored while processing 'explain -script' or '-check' 2013-09-06 10:41:29,774 [main] WARN org.apache.pig.PigServer - Encountered Warning IMPLICIT_CAST_TO_FLOAT 1 time(s). 2013-09-06 10:41:29,814 [main] ERROR org.apache.pig.tools.grunt.Grunt - You don't have permission to perform the operation. Error from the server: dummy (Permission denied) 2013-09-06 10:41:29,815 [main] ERROR org.apache.pig.tools.grunt.Grunt - ERROR 2997: Encountered IOException. dummy (Permission denied) Details at logfile: /data/storage/pig-0.10.0/logs/pig_1378444279340.log
It seems to be permission issues for apache user. While running following command pig -c ScriptName, check the output from both places. Output From Shell Output from script while executing through Apache server Check the permission for log files as well, where pig is trying to write. This should help you to resolve this issue.