Sonarqube version: 9.7.1.62043 (Developer Edition)
Database: PostgreSQL
Installed using: Docker
Integraion with: Gitlab Community Edition 15.5.2
Cannot go to project dashboard in sonarqube - getting exceptions as mentioned
Issue: Getting a pop up after clicking on project with the message ‘An error has occurred. Please contact your administrator’
We have configured gitlab-ci to run sonarscanner on every pull request.
In the docker logs I can see logs as:
web.log
2023.01.30 07:24:06 ERROR web[AYX0AdNMD90rem68AAMM][o.s.s.w.WebServiceEngine] Fail to process request http://MY_SONAR_URL/api/project_pull_requests/list?project=MY_PROJECT_ID
java.lang.NullPointerException: Pull request data should be available for branch type PULL_REQUEST
at java.base/java.util.Objects.requireNonNull(Objects.java:246)
ce.log
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.s.ComputationStepExecutor] Compute new coverage | status=FAILED | time=99ms
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Webhooks | globalWebhooks=0 | projectWebhooks=0 | status=SUCCESS | time=2ms
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][c.s.G.D.F.B] GitLab's instance URL from the scanner ('https://gitlab.MYDOMAIN/api/v4') is overridden by the settings ('https://gitlab.MYDOMAIN/api/v4')
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][c.s.G.D.F.B] GitLab's project ID from the scanner ('74') is overridden by the settings ('74')
2023.01.31 13:14:45 ERROR ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Execution of task class com.sonarsource.G.D.d failed
java.lang.NullPointerException: null
at org.sonar.db.protobuf.DbProjectBranches$PullRequestData$Builder.mergeFrom(DbProjectBranches.java:864)
at org.sonar.db.protobuf.DbProjectBranches$PullRequestData.newBuilder(DbProjectBranches.java:697)
at com.sonarsource.G.D.d.A(Unknown Source)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at com.sonarsource.G.D.d.A(Unknown Source)
at com.sonarsource.G.D.d.B(Unknown Source)
at org.sonar.ce.async.SynchronousAsyncExecution.addToQueue(SynchronousAsyncExecution.java:27)
at com.sonarsource.G.D.d.A(Unknown Source)
at java.base/java.util.Optional.ifPresent(Optional.java:183)
at com.sonarsource.G.D.d.finished(Unknown Source)
at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.executeTask(PostProjectAnalysisTasksExecutor.java:102)
at org.sonar.ce.task.projectanalysis.api.posttask.PostProjectAnalysisTasksExecutor.finished(PostProjectAnalysisTasksExecutor.java:93)
at org.sonar.ce.task.step.ComputationStepExecutor.executeListener(ComputationStepExecutor.java:89)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:61)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Pull Request decoration | status=FAILED | time=10ms
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.p.a.p.PostProjectAnalysisTasksExecutor] Report branch Quality Gate status to devops platforms | status=SUCCESS | time=0ms
2023.01.31 13:14:45 ERROR ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.CeWorkerImpl] Failed to execute task AYYH9x3NGuaUdN9D31Cm
org.sonar.ce.task.projectanalysis.component.VisitException: Visit failed for Component {key=KEY:SOME_PATH/PricingServiceImpl.java,type=FILE} located KEY:SOME_PATH(type=DIRECTORY)->KEY:FILE_PATH(type=DIRECTORY)->KEY:FILE_PATH(type=DIRECTORY)->KEY:quote(type=DIRECTORY)->KEY(type=PROJECT)
at org.sonar.ce.task.projectanalysis.component.VisitException.rethrowOrWrap(VisitException.java:44)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:52)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitChildren(PathAwareCrawler.java:87)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:70)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
at org.sonar.ce.task.projectanalysis.step.NewCoverageMeasuresStep.execute(NewCoverageMeasuresStep.java:90)
at org.sonar.ce.task.step.ComputationStepExecutor.executeStep(ComputationStepExecutor.java:79)
at org.sonar.ce.task.step.ComputationStepExecutor.executeSteps(ComputationStepExecutor.java:70)
at org.sonar.ce.task.step.ComputationStepExecutor.execute(ComputationStepExecutor.java:57)
at org.sonar.ce.task.projectanalysis.taskprocessor.ReportTaskProcessor.process(ReportTaskProcessor.java:75)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.executeTask(CeWorkerImpl.java:212)
at org.sonar.ce.taskprocessor.CeWorkerImpl$ExecuteTask.run(CeWorkerImpl.java:194)
at org.sonar.ce.taskprocessor.CeWorkerImpl.findAndProcessTask(CeWorkerImpl.java:160)
at org.sonar.ce.taskprocessor.CeWorkerImpl$TrackRunningState.get(CeWorkerImpl.java:135)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:87)
at org.sonar.ce.taskprocessor.CeWorkerImpl.call(CeWorkerImpl.java:53)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException: null
at org.sonar.ce.task.projectanalysis.scm.ScmInfoRepositoryImpl.removeAuthorAndRevision(ScmInfoRepositoryImpl.java:104)
at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
at java.base/java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
at java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
at java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
at org.sonar.ce.task.projectanalysis.scm.ScmInfoRepositoryImpl.removeAuthorAndRevision(ScmInfoRepositoryImpl.java:99)
at org.sonar.ce.task.projectanalysis.scm.ScmInfoRepositoryImpl.generateAndMergeDb(ScmInfoRepositoryImpl.java:119)
at org.sonar.ce.task.projectanalysis.scm.ScmInfoRepositoryImpl.getScmInfoForComponent(ScmInfoRepositoryImpl.java:74)
at java.base/java.util.HashMap.computeIfAbsent(HashMap.java:1134)
at org.sonar.ce.task.projectanalysis.scm.ScmInfoRepositoryImpl.getScmInfo(ScmInfoRepositoryImpl.java:65)
at org.sonar.ce.task.projectanalysis.source.NewLinesRepository.computeNewLinesFromScm(NewLinesRepository.java:72)
at org.sonar.ce.task.projectanalysis.source.NewLinesRepository.getNewLines(NewLinesRepository.java:64)
at org.sonar.ce.task.projectanalysis.step.NewCoverageMeasuresStep$NewCoverageCounter.initialize(NewCoverageMeasuresStep.java:205)
at org.sonar.ce.task.projectanalysis.formula.FormulaExecutorComponentVisitor.processLeaf(FormulaExecutorComponentVisitor.java:148)
at org.sonar.ce.task.projectanalysis.formula.FormulaExecutorComponentVisitor.process(FormulaExecutorComponentVisitor.java:125)
at org.sonar.ce.task.projectanalysis.formula.FormulaExecutorComponentVisitor.visitFile(FormulaExecutorComponentVisitor.java:105)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitNode(PathAwareCrawler.java:102)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visitImpl(PathAwareCrawler.java:73)
at org.sonar.ce.task.projectanalysis.component.PathAwareCrawler.visit(PathAwareCrawler.java:50)
... 35 common frames omitted
2023.01.31 13:14:45 INFO ce[AYYH9x3NGuaUdN9D31Cm][o.s.c.t.CeWorkerImpl] Executed task | project=KEY | type=REPORT | pullRequest=8837 | id=AYYH9x3NGuaUdN9D31Cm | submitter=sonarqube11609 | status=FAILED | time=2908ms
2023.01.31 13:26:06 INFO ce[][o.s.c.t.CeWorkerImpl] Execute task | project=KEY | type=REPORT | pullRequest=8837 | id=AYYIAY0jGuaUdN9D31Co | submitter=sonarqube11609
2023.01.31 13:26:07 INFO ce[AYYIAY0jGuaUdN9D31Co][o.s.c.t.s.ComputationStepExecutor] Extract report | status=SUCCESS | time=876ms
2023.01.31 13:26:07 INFO ce[AYYIAY0jGuaUdN9D31Co][o.s.c.t.s.ComputationStepExecutor] Persist scanner context | status=SUCCESS | time=35ms
2023.01.31 13:26:07 INFO ce[AYYIAY0jGuaUdN9D31Co][o.s.c.t.s.ComputationStepExecutor] Propagate analysis warnings from scanner report | status=SUCCESS | time=6ms
This does not fail for all the pull requests, but only a few.
I have a Django project. It has some files with cyrillic chars in their names:
SonarQube gives error:
java.nio.file.InvalidPathException: Malformed input or input contains
unmappable characters:
src/apps/orders/sap_data_examples/ZLO_CANCEL_ORDERS/??????????
?????????????? ??????????????.json at
java.base/sun.nio.fs.UnixPath.encode(Unknown Source) at
java.base/sun.nio.fs.UnixPath.(Unknown Source) at
java.base/sun.nio.fs.UnixFileSystem.getPath(Unknown Source) at
java.base/java.nio.file.Path.resolve(Unknown Source) at
org.sonar.scm.git.IncludedFilesRepository.indexFiles(IncludedFilesRepository.java:65)
at
org.sonar.scm.git.IncludedFilesRepository.(IncludedFilesRepository.java:40)
at org.sonar.scm.git.GitIgnoreCommand.init(GitIgnoreCommand.java:37)
at
org.sonar.scanner.scan.filesystem.ProjectFileIndexer.index(ProjectFileIndexer.java:104)
at
org.sonar.scanner.scan.ProjectScanContainer.doAfterStart(ProjectScanContainer.java:352)
at
org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
at
org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
at
org.sonar.scanner.bootstrap.GlobalContainer.doAfterStart(GlobalContainer.java:150)
at
org.sonar.core.platform.ComponentContainer.startComponents(ComponentContainer.java:137)
at
org.sonar.core.platform.ComponentContainer.execute(ComponentContainer.java:123)
at org.sonar.batch.bootstrapper.Batch.doExecute(Batch.java:72) at
org.sonar.batch.bootstrapper.Batch.execute(Batch.java:66) at
org.sonarsource.scanner.api.internal.batch.BatchIsolatedLauncher.execute(BatchIsolatedLauncher.java:46)
at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown
Source) at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
Source) at java.base/java.lang.reflect.Method.invoke(Unknown Source)
at
org.sonarsource.scanner.api.internal.IsolatedLauncherProxy.invoke(IsolatedLauncherProxy.java:60)
at com.sun.proxy.$Proxy0.execute(Unknown Source) at
org.sonarsource.scanner.api.EmbeddedScanner.doExecute(EmbeddedScanner.java:189)
at
org.sonarsource.scanner.api.EmbeddedScanner.execute(EmbeddedScanner.java:138)
at org.sonarsource.scanner.cli.Main.execute(Main.java:112) at
org.sonarsource.scanner.cli.Main.execute(Main.java:75) at
org.sonarsource.scanner.cli.Main.main(Main.java:61)
SonarQube in gitlab-ci is being started as follows:
analyze:
stage: analyze
needs: ["tests"]
dependencies:
- tests
image:
name: docker-sre.site.ru/sonarqube/scanner:${SONAR_TAG}
entrypoint: [""]
before_script:
- echo code analysis will be executed on ${SONAR_HOST}
script:
- /entrypoint.sh ${SONAR_HOST} ${SONAR_LOGIN} -X
allow_failure: true
artifacts:
when: always
reports:
junit: report.xml
coverage_report:
coverage_format: cobertura
path: coverage.xml
I supposed that the error related to cyrillic chars in file names. If so how to fix this? Maybe specify encoding some how?
RUN THIS COMMAND
/bin/nifi.sh stateless RunFromRegistry Once --file ./test/stateless_test1.json
LOG
Note: Use of this command is considered experimental. The commands and approach used may change from time to time.
Java home (JAVA_HOME): /home/deltaman/software/jdk1.8.0_211
Java options (STATELESS_JAVA_OPTS): -Xms1024m -Xmx1024m
13:48:39.835 [main] INFO org.apache.nifi.StatelessNiFi - Unpacking 100 NARs
13:50:51.513 [main] INFO org.apache.nifi.StatelessNiFi - Finished unpacking 100 NARs in 131671 millis
Exception in thread "main" java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.nifi.StatelessNiFi.main(StatelessNiFi.java:103)
... 5 more
Caused by: java.nio.file.NoSuchFileException: ./test/stateless_test1.json
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.Files.readAllBytes(Files.java:3152)
at org.apache.nifi.stateless.runtimes.Program.runLocal(Program.java:119)
at org.apache.nifi.stateless.runtimes.Program.launch(Program.java:67)
... 10 more
it seems no exist file,but i can find the file as follows:
$ cat ./test/stateless_test1.json
{
"registryUrl": "http://10.148.123.12:9991",
"bucketId": "ec1b291e-c3f1-437c-a4e4-c069bd2f6ed1",
"flowId": "b1f73fe8-2874-47a5-970c-6b25eea19497",
"parameters": {
"text" : "xixixixi"
}
}
CONFIGURATION
IDK WHAT IS THE PROBLEM?
ANY SUGGESTION IS APPRECIATION!
/bin/nifi.sh stateless RunFromRegistry Once --file ./test/stateless_test1.json
it is relative path,must use full path,such as
/home/NiFi/nifi-1.10.0/bin/nifi.sh stateless RunFromRegistry Once --file /home/NiFi/nifi-1.10.0/test/stateless_test1.json
I have loaded JSON data to my HDFS, I created the table with required columns in MySQL database as follows.
How to create table with row formatter for accepting JSON?
My HDFS data
{
"Employees" : [
{
"userId":"rirani",
"jobTitleName":"Developer",
"firstName":"Romin",
"lastName":"Irani",
"preferredFullName":"Romin Irani",
"employeeCode":"E1",
"region":"CA",
"phoneNumber":"408-1234567",
"emailAddress":"romin.k.irani#gmail.com"
},
{
"userId":"nirani",
"jobTitleName":"Developer",
"firstName":"Neil",
"lastName":"Irani",
"preferredFullName":"Neil Irani",
"employeeCode":"E2",
"region":"CA",
"phoneNumber":"408-1111111",
"emailAddress":"neilrirani#gmail.com"
},
{
"userId":"thanks",
"jobTitleName":"Program Directory",
"firstName":"Tom",
"lastName":"Hanks",
"preferredFullName":"Tom Hanks",
"employeeCode":"E3",
"region":"CA",
"phoneNumber":"408-2222222",
"emailAddress":"tomhanks#gmail.com"
}
]
}
My SQL table structure
mysql> create table employee(userid int,jobTitleName varchar(20),firstName varchar(20),lastName varchar(20),preferrredFullName varchar(20),employeeCode varchar(20),region varchar(20),phoneNumber varchar(20), emailAddress varchar(20),modifiedDate timestamp DEFAULT CURRENT_TIMESTAMP);
mysql> desc employee;
+--------------------+-------------+------+-----+-------------------+-------+
| Field | Type | Null | Key | Default | Extra |
+--------------------+-------------+------+-----+-------------------+-------+
| userid | int(11) | YES | | NULL | |
| jobTitleName | varchar(20) | YES | | NULL | |
| firstName | varchar(20) | YES | | NULL | |
| lastName | varchar(20) | YES | | NULL | |
| preferrredFullName | varchar(20) | YES | | NULL | |
| employeeCode | varchar(20) | YES | | NULL | |
| region | varchar(20) | YES | | NULL | |
| phoneNumber | varchar(20) | YES | | NULL | |
| emailAddress | varchar(20) | YES | | NULL | |
| modifiedDate | timestamp | NO | | CURRENT_TIMESTAMP | |
+--------------------+-------------+------+-----+-------------------+-------+
10 rows in set (0.00 sec)
I am trying to load data from my HDFS to MySQL for the above table using sqoop export as follows
sqoop export --connect jdbc:mysql://localhost/emp_scheme --username root --password adithyan --table employee --export-dir /user/adithyan/filesystem/employee.txt
it has end up with exception as follows
17/02/18 19:35:35 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
17/02/18 19:35:35 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
17/02/18 19:35:35 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
17/02/18 19:35:35 INFO tool.CodeGenTool: Beginning code generation
17/02/18 19:35:36 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
17/02/18 19:35:36 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM `employee` AS t LIMIT 1
17/02/18 19:35:36 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /home/adithyan/hadoop_dir/hadoop-1.2.1
Note: /tmp/sqoop-adithyan/compile/35afadf151a1dd1626a3658577cbc2dd/employee.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
17/02/18 19:35:41 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-adithyan/compile/35afadf151a1dd1626a3658577cbc2dd/employee.jar
17/02/18 19:35:41 INFO mapreduce.ExportJobBase: Beginning export of employee
17/02/18 19:35:45 INFO input.FileInputFormat: Total input paths to process : 1
17/02/18 19:35:45 INFO input.FileInputFormat: Total input paths to process : 1
17/02/18 19:35:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library
17/02/18 19:35:45 WARN snappy.LoadSnappy: Snappy native library not loaded
17/02/18 19:35:46 INFO mapred.JobClient: Running job: job_201702181051_0002
17/02/18 19:35:47 INFO mapred.JobClient: map 0% reduce 0%
17/02/18 19:36:17 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_0, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
... 12 more
17/02/18 19:36:18 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_0, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
... 12 more
17/02/18 19:36:29 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
... 12 more
17/02/18 19:36:29 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_1, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
... 12 more
17/02/18 19:36:42 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000000_2, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '"firstName":"Tom"'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: ""firstName":"Tom""
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:569)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
... 12 more
17/02/18 19:36:42 INFO mapred.JobClient: Task Id : attempt_201702181051_0002_m_000001_2, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:145)
at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:364)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: java.lang.RuntimeException: Can't parse input data: '{'
at employee.__loadFromFields(employee.java:596)
at employee.parse(employee.java:499)
at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:83)
... 10 more
Caused by: java.lang.NumberFormatException: For input string: "{"
at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
at java.lang.Integer.parseInt(Integer.java:580)
at java.lang.Integer.valueOf(Integer.java:766)
at employee.__loadFromFields(employee.java:548)
Can somebody help me on this?
you may have to look at multiple options..
JSON_SET/REPLACE/INSERT - these options may not be directly supported by sqoop yet.
Another options is to pre-process data using pig then stage the data in HDFS before sqooping to RDBMS.
I get the following exception when I try to run Grails from the terminal in OSX:
| Loading Grails 2.3.6
| Error java.lang.reflect.InvocationTargetException
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| Error at java.lang.reflect.Method.invoke(Method.java:597)
| Error at org.codehaus.groovy.grails.cli.support.GrailsStarter.rootLoader(GrailsStarter.java:235)
| Error at org.codehaus.groovy.grails.cli.support.GrailsStarter.main(GrailsStarter.java:263)
| Error Caused by: java.lang.IllegalAccessError: class sun.reflect.GeneratedConstructorAccessor2 cannot access its superclass sun.reflect.ConstructorAccessorImpl
| Error at sun.misc.Unsafe.defineClass(Native Method)
| Error at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:45)
| Error at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:381)
| Error at java.security.AccessController.doPrivileged(Native Method)
| Error at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:377)
| Error at sun.reflect.MethodAccessorGenerator.generateConstructor(MethodAccessorGenerator.java:76)
| Error at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:30)
| Error at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
| Error at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
| Error at org.codehaus.groovy.reflection.CachedConstructor.invoke(CachedConstructor.java:77)
| Error at org.codehaus.groovy.runtime.callsite.ConstructorSite$ConstructorSiteNoUnwrapNoCoerce.callConstructor(ConstructorSite.java:102)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callConstructor(AbstractCallSite.java:202)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor.addRuleForModuleId(EnhancedDefaultDependencyDescriptor.groovy:135)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor$addRuleForModuleId$0.callCurrent(Unknown Source)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor.excludeForMap(EnhancedDefaultDependencyDescriptor.groovy:113)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor.this$3$excludeForMap(EnhancedDefaultDependencyDescriptor.groovy)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor$this$3$excludeForMap.callCurrent(Unknown Source)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor.<init>(EnhancedDefaultDependencyDescriptor.groovy:76)
| Error at org.codehaus.groovy.grails.resolve.EnhancedDefaultDependencyDescriptor.<init>(EnhancedDefaultDependencyDescriptor.groovy:80)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies.registerDependency(GrailsIvyDependencies.groovy:69)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies.registerDependencies(GrailsIvyDependencies.groovy:58)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| Error at java.lang.reflect.Method.invoke(Method.java:597)
| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite$StaticMetaMethodSiteNoUnwrapNoCoerce.invoke(StaticMetaMethodSite.java:148)
| Error at org.codehaus.groovy.runtime.callsite.StaticMetaMethodSite.call(StaticMetaMethodSite.java:88)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies$_createDeclaration_closure1_closure3.doCall(GrailsIvyDependencies.groovy:117)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies$_createDeclaration_closure1_closure3.doCall(GrailsIvyDependencies.groovy)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| Error at java.lang.reflect.Method.invoke(Method.java:597)
| Error at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
| Error at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.call(PogoMetaMethodSite.java:64)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:112)
| Error at org.codehaus.groovy.grails.resolve.config.DependencyConfigurationConfigurer.dependencies(DependencyConfigurationConfigurer.groovy:150)
| Error at org.codehaus.groovy.grails.resolve.config.DependencyConfigurationConfigurer$dependencies$1.call(Unknown Source)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies$_createDeclaration_closure1.doCall(GrailsIvyDependencies.groovy:102)
| Error at org.codehaus.groovy.grails.resolve.GrailsIvyDependencies$_createDeclaration_closure1.doCall(GrailsIvyDependencies.groovy)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
| Error at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
| Error at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
| Error at java.lang.reflect.Method.invoke(Method.java:597)
| Error at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
| Error at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1086)
| Error at groovy.lang.ExpandoMetaClass.invokeMethod(ExpandoMetaClass.java:1110)
| Error at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:910)
| Error at groovy.lang.Closure.call(Closure.java:411)
| Error at groovy.lang.Closure.call(Closure.java:405)
| Error at org.codehaus.groovy.grails.resolve.AbstractIvyDependencyManager.doParseDependencies(AbstractIvyDependencyManager.java:676)
| Error at org.codehaus.groovy.grails.resolve.AbstractIvyDependencyManager.parseDependencies(AbstractIvyDependencyManager.java:577)
| Error at org.codehaus.groovy.grails.resolve.DependencyManager$parseDependencies.call(Unknown Source)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:45)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
| Error at org.codehaus.groovy.grails.resolve.DependencyManagerConfigurer.configureIvy(DependencyManagerConfigurer.groovy:157)
| Error at grails.util.BuildSettings.configureDependencyManager(BuildSettings.groovy:1281)
| Error at grails.util.BuildSettings.postLoadConfig(BuildSettings.groovy:1219)
| Error at grails.util.BuildSettings.loadConfig(BuildSettings.groovy:1075)
| Error at grails.util.BuildSettings.loadConfig(BuildSettings.groovy)
| Error at grails.util.BuildSettings$loadConfig$0.callCurrent(Unknown Source)
| Error at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
| Error at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:141)
| Error at grails.util.BuildSettings.loadConfig(BuildSettings.groovy:1053)
| Error at org.codehaus.groovy.grails.cli.GrailsScriptRunner.loadConfigEnvironment(GrailsScriptRunner.java:249)
| Error at org.codehaus.groovy.grails.cli.GrailsScriptRunner.main(GrailsScriptRunner.java:210)
| Error ... 6 more
I can run it fine from within IntelliJ. I know it's something with my environment configuration, but I haven't been able to figure out what, yet. Doesi anyone have any ideas?
I'm running Java:
java version "1.6.0_65"
Java(TM) SE Runtime Environment (build 1.6.0_65-b14-462-11M4609)
Java HotSpot(TM) 64-Bit Server VM (build 20.65-b04-462, mixed mode)
OSX: 10.8.4
Are you exporting JAVA_HOME, I know on Linux I have to everytime I want to use grails command line..
Take a look here, maybe try updating to jdk since it does say jre .. (not a mac expert though) - also a few things to try:
http://liberalsprouts.blogspot.co.uk/2012/12/how-to-install-jdk-7-and-set-up.html
echo $PATH
echo $JAVA_HOME
java --version
and
$JAVA_HOME/bin/java -version
see if it is all the same.. I guess it will be, if you follow the instructions maybe see how latest jdk treats it.
to figure out what your intelliJ is using:
jps
3735
7588 GrailsStarter
7660 ForkedTomcatServer
7783 Jps
in my case its ggts if i now run:
lsof -p 7588|grep -i java|grep jdk|head -n 3
java 7588 mx1 txt REG 8,1 38568 18745123 /usr/lib/jvm/java-6-openjdk-i386/jre/bin/java
java 7588 mx1 mem REG 8,1 71084 12982357 /usr/lib/jvm/java-6-openjdk-i386/jre/lib/i386/libj2pkcs11.so
java 7588 mx1 mem REG 8,1 85518 18745490 /usr/lib/jvm/java-6-openjdk-common/jre/lib/jce.jar
which is -p {process id} I piped it into head to minimise output - but it gives me an idea of what jdk it is using - maybe you can trace what is going on and exports etc from using this method..