I'm trying to create an ageRange target for a line item using the DV360 API. However, I require targetOptionId. Where can I get it?
(https://developers.google.com/display-video/api/reference/rest/v1/advertisers.lineItems.targetingTypes.assignedTargetingOptions/create?apix_params=%7B%22advertiserId%22% 3A613810781% 2C% 22lineItemId% 22% 3A15014994889% 2C% 22targetingType% 22% 3A% 22TARGETING_TYPE_AGE_RANGE% 22% 2C% 22resource% 22% 3A% 7B% 22ageRangeDetails24% 22% 3Aagne% 22GE_RANAGE% 22% 3Aagne% 22GE_RANAGE% 22 7D% 7D% 7D)
Related
I have integrated elastic search with hadoop using elasticsearch-hadoop-2.3.2.jar.I am Querying my hive table using beeline.when i create or insert normal table , it works fine.But, when i create a external table for using elastic search like following
CREATE EXTERNAL TABLE flogs2 (
name STRING,
city STRING,
status STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.nodes' = '192.168.18.79','es.port' = '9200','es.index.auto.create' = 'true', 'es.resource' = 'mylog/log', 'es.query' = '?q=*');
The table is created.But, when i insert data into it like below,
INSERT OVERWRITE TABLE flogs2 SELECT s.name,s.city,s.status FROM logs
s;
i am stuck at the following lines
0: jdbc:hive2://192.168.18.197:10000> INSERT OVERWRITE TABLE flogs2 SELECT s.name,s.city,s.status FROM logs s;
INFO : Number of reduce tasks is set to 0 since there's no reduce operator
INFO : number of splits:1
INFO : Submitting tokens for job: job_1464067651503_0014
INFO : The url to track the job: http://vasanthakumar-virtual-machine:8088/proxy/application_1464067651503_0014/
INFO : Starting Job = job_1464067651503_0014, Tracking URL = http://vasanthakumar-virtual-machine:8088/proxy/application_1464067651503_0014/
INFO : Kill Command = /home/vasanthakumar/Desktop/software/hadoop-2.7.1/bin/hadoop job -kill job_1464067651503_0014
INFO : Hadoop job information for Stage-0: number of mappers: 1; number of reducers: 0
INFO : 2016-05-24 16:26:51,866 Stage-0 map = 0%, reduce = 0%
INFO : 2016-05-24 16:27:52,372 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 1.48 sec
INFO : 2016-05-24 16:28:52,498 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 1.48 sec
INFO : 2016-05-24 16:29:52,562 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 1.48 sec
INFO : 2016-05-24 16:30:52,884 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 1.48 sec
INFO : 2016-05-24 16:31:53,103 Stage-0 map = 0%, reduce = 0%, Cumulative CPU 1.48 sec
Note:
1.I have tried both HiveCLI and Beeline
2.Increased my memory space
3.normal hive queries working fine
Please help me to get rid from this.
I'm trying out something simple in Hive on HDFS.
The problem is that the queries are not running map reduce when I'm running a "where clause". However, it runs map reduce for count(*), and even group by clauses.
Here's data and queries with result:
Create External Table:
CREATE EXTERNAL TABLE testtab1 (
id STRING, org STRING)
row format delimited
fields terminated by ','
stored as textfile
location '/usr/ankuchak/testtable1';
Simple select * query:
0: jdbc:hive2://> select * from testtab1;
15/07/01 07:32:46 [main]: ERROR hdfs.KeyProviderCache: Could not find uri with key [dfs.encryption.key.provider.uri] to create a keyProvider !!
OK
+---------------+---------------+--+
| testtab1.id | testtab1.org |
+---------------+---------------+--+
| ankur | idc |
| user | idc |
| someone else | ssi |
+---------------+---------------+--+
3 rows selected (2.169 seconds)
Count(*) query
0: jdbc:hive2://> select count(*) from testtab1;
Query ID = ankuchak_20150701073407_e7fd66ae-8812-4e02-87d7-492f81781d15
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
15/07/01 07:34:08 [HiveServer2-Background-Pool: Thread-40]: ERROR mr.ExecDriver: yarn
15/07/01 07:34:08 [HiveServer2-Background-Pool: Thread-40]: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
Starting Job = job_1435425589664_0005, Tracking URL = http://slc02khv:8088/proxy/application_1435425589664_0005/
Kill Command = /scratch/hadoop/hadoop/bin/hadoop job -kill job_1435425589664_0005
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
15/07/01 07:34:16 [HiveServer2-Background-Pool: Thread-40]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2015-07-01 07:34:16,291 Stage-1 map = 0%, reduce = 0%
2015-07-01 07:34:23,831 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.04 sec
2015-07-01 07:34:30,102 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.41 sec
MapReduce Total cumulative CPU time: 2 seconds 410 msec
Ended Job = job_1435425589664_0005
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 2.41 sec HDFS Read: 6607 HDFS Write: 2 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 410 msec
OK
+------+--+
| _c0 |
+------+--+
| 3 |
+------+--+
1 row selected (23.527 seconds)
Group by query:
0: jdbc:hive2://> select org, count(id) from testtab1 group by org;
Query ID = ankuchak_20150701073540_5f20df4e-0bd4-4e18-b065-44c2688ce21f
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=
In order to set a constant number of reducers:
set mapreduce.job.reduces=
15/07/01 07:35:40 [HiveServer2-Background-Pool: Thread-63]: ERROR mr.ExecDriver: yarn
15/07/01 07:35:41 [HiveServer2-Background-Pool: Thread-63]: WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
Starting Job = job_1435425589664_0006, Tracking URL = http://slc02khv:8088/proxy/application_1435425589664_0006/
Kill Command = /scratch/hadoop/hadoop/bin/hadoop job -kill job_1435425589664_0006
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 1
15/07/01 07:35:47 [HiveServer2-Background-Pool: Thread-63]: WARN mapreduce.Counters: Group org.apache.hadoop.mapred.Task$Counter is deprecated. Use org.apache.hadoop.mapreduce.TaskCounter instead
2015-07-01 07:35:47,200 Stage-1 map = 0%, reduce = 0%
2015-07-01 07:35:53,494 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 1.05 sec
2015-07-01 07:36:00,799 Stage-1 map = 100%, reduce = 100%, Cumulative CPU 2.53 sec
MapReduce Total cumulative CPU time: 2 seconds 530 msec
Ended Job = job_1435425589664_0006
MapReduce Jobs Launched:
Stage-Stage-1: Map: 1 Reduce: 1 Cumulative CPU: 2.53 sec HDFS Read: 7278 HDFS Write: 14 SUCCESS
Total MapReduce CPU Time Spent: 2 seconds 530 msec
OK
+-------+------+--+
| org | _c1 |
+-------+------+--+
| idc | 2 |
| ssi | 1 |
+-------+------+--+
2 rows selected (21.187 seconds)
Now the simple where clause:
0: jdbc:hive2://> select * from testtab1 where org='idc';
OK
+--------------+---------------+--+
| testtab1.id | testtab1.org |
+--------------+---------------+--+
+--------------+---------------+--+
No rows selected (0.11 seconds)
It would be great if you could provide me with some pointers.
Please let me know if you need further information in this regard.
Regards,
Ankur
Map job is occuring in your last query. So it's not that map reduce is not happening. However, some rows should be returned in your last query. The likely culprit here is that for some reason it is not finding a match on the value "idc". Check your table and ensure that the group for Ankur and user contain the string idc.
Try this to see if you get any results:
Select * from testtab1 where org rlike '.*(idc).*';
or
Select * from testtab1 where org like '%idc%';
These will grab any row that has a value containing the string 'idc'. Good luck!
Here, details of the same error and fixed recently. Try verifying the version you are using
I am using Golang to implement naive bayesian classification for a dataset with over 30000 possible tags. I have built the model and I am in the classification phase. I am working on classifying 1000 records and this is taking up to 5 minutes. I have profiled the code with pprof functionality; the top10 are shown below:
Total: 28896 samples
16408 56.8% 56.8% 24129 83.5% runtime.mapaccess1_faststr
4977 17.2% 74.0% 4977 17.2% runtime.aeshashbody
2552 8.8% 82.8% 2552 8.8% runtime.memeqbody
1468 5.1% 87.9% 28112 97.3% main.(*Classifier).calcProbs
861 3.0% 90.9% 861 3.0% math.Log
435 1.5% 92.4% 435 1.5% runtime.markspan
267 0.9% 93.3% 302 1.0% MHeap_AllocLocked
187 0.6% 94.0% 187 0.6% runtime.aeshashstr
183 0.6% 94.6% 1137 3.9% runtime.mallocgc
127 0.4% 95.0% 988 3.4% math.log10
Surprisingly the map access seems to be the bottleneck. Has anyone experienced this. What other key, value datastructure can be used to avoid this bottleneck? All the map access is done in the following piece of code given below:
func (nb *Classifier) calcProbs(data string) *BoundedPriorityQueue{
probs := &BoundedPriorityQueue{}
heap.Init(probs)
terms := strings.Split(data, " ")
for class, prob := range nb.classProb{
condProb := prob
clsProbs := nb.model[class]
for _, term := range terms{
termProb := clsProbs[term]
if termProb != 0{
condProb += math.Log10(termProb)
}else{
condProb += -6 //math.Log10(0.000001)
}
}
entry := &Item{
value: class,
priority: condProb,
}
heap.Push(probs,entry)
}
return probs
}
The maps are nb.classProb which is map[string]float64 while the nb.model is a nested map of type
map[string]map[string]float64
In addition to what #tomwilde said, another approach that may speed up your algorithm is string interning. Namely, you can avoid using a map entirely if you know the domain of keys ahead of time. I wrote a small package that will do string interning for you.
Yes, the map access will be the bottleneck in this code: it's the most significant operation inside the two nested loops.
It's not possible to tell for sure from the code that you've included, but I expect you've got a limited number of classes. What you might do, is number them, and store the term-wise class probabilities like this:
map[string][NumClasses]float64
(ie: for each term, store an array of class-wise probabilities [or perhaps their logs already precomputed], and NumClasses is the number of different classes you have).
Then, iterate over terms first, and classes inside. The expensive map lookup will be done in the outer loop, and the inner loop will be iteration over an array.
This'll reduce the number of map lookups by a factor of NumClasses. This may need more memory if your data is extremely sparse.
The next optimisation is to use multiple goroutines to do the calculations, assuming you've more than one CPU core available.
The hive does need to process 45 files. Each size is about 1GB. After the mapper execution finished 100%, hive Failed with the error message above.
Driver returned: 1. Errors: OK
Hive history file=/tmp/hue/hive_job_log_hue_201308221004_1738621649.txt
Total MapReduce jobs = 3
Launching Job 1 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1376898282169_0441, Tracking URL = http://SH02SVR2882.hadoop.sh2.ctripcorp.com:8088/proxy/application_1376898282169_0441/
Kill Command = //usr/lib/hadoop/bin/hadoop job -kill job_1376898282169_0441
Hadoop job information for Stage-1: number of mappers: 236; number of reducers: 0
2013-08-22 10:04:40,205 Stage-1 map = 0%, reduce = 0%
2013-08-22 10:05:07,486 Stage-1 map = 1%, reduce = 0%, Cumulative CPU 121.28 sec
.......................
2013-08-22 10:09:18,625 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 7707.18 sec
MapReduce Total cumulative CPU time: 0 days 2 hours 8 minutes 27 seconds 180 msec
Ended Job = job_1376898282169_0441
Ended Job = -541447549, job is filtered out (removed at runtime).
Ended Job = -1652692814, job is filtered out (removed at runtime).
Launching Job 3 out of 3
Number of reduce tasks is set to 0 since there's no reduce operator
Job Submission failed with exception
'java.io.IOException(Max block location exceeded for split: Paths:/tmp/hive-beeswax-logging/hive_2013-08-22_10-04-32_755_6427103839442439579/-ext-10001/000009_0:0+28909,....,/tmp/hive-beeswax-logging/hive_2013-08-22_10-04-32_755_6427103839442439579/-ext-10001/000218_0:0+45856
Locations:10.8.75.17:...:10.8.75.20:; InputFormatClass: org.apache.hadoop.mapred.TextInputFormat
splitsize: 45 maxsize: 10)'
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MapRedTask
MapReduce Jobs Launched:
Job 0: Map: 236 Cumulative CPU: 7707.18 sec HDFS Read: 63319449229 HDFS Write: 8603165 SUCCESS
Total MapReduce CPU Time Spent: 0 days 2 hours 8 minutes 27 seconds 180 msec
But I didn't set the maxsize. Executed many times but got the same error. I tried to add mapreduce.jobtracker.split.metainfo.maxsize property for hive. But in this case, hive failed without any map work.
set mapreduce.job.max.split.locations > 45
In our situation it solved problem.
I have a node API which normally handles traffic quite well. However, in our peak times, it gets into this state where it starts using 100% CPU and needs to be restarted. After restarting, it returns to a regular state for the next few days.
Using a load testing site, I have been able to reproduce this issue. The request I am load testing is extremely simple, so I fear the problem is in a third-party library I'm using.
I'm new to debugging node and I'm not sure what to make of the following output from node-tick-processor. Can anyone decipher this?
Update: I'm running node v0.10.4
[Unknown]:
ticks total nonlib name
5 0.0%
[Shared libraries]:
ticks total nonlib name
11943 49.1% 0.0% /lib64/libc-2.12.so
10754 44.2% 0.0% /usr/local/bin/node
314 1.3% 0.0% /lib64/libpthread-2.12.so
50 0.2% 0.0% 7fff318b4000-7fff318b5000
5 0.0% 0.0% /lib64/libm-2.12.so
3 0.0% 0.0% /usr/lib64/libstdc++.so.6.0.17
[JavaScript]:
ticks total nonlib name
40 0.2% 3.2% LazyCompile: ~read tls.js:397
36 0.1% 2.8% LazyCompile: *EventEmitter.addListener events.js:126
29 0.1% 2.3% LazyCompile: *Readable.read _stream_readable.js:226
<clipped>
[Bottom up (heavy) profile]:
Note: percentage shows a share of a particular caller in the total
amount of its parent calls.
Callers occupying less than 2.0% are not shown.
ticks parent name
11943 49.1% /lib64/libc-2.12.so
10754 44.2% /usr/local/bin/node
8270 76.9% LazyCompile: *use tls.js:222
5162 62.4% LazyCompile: ~read tls.js:397
5074 98.3% LazyCompile: *Readable.read _stream_readable.js:226
3396 66.9% LazyCompile: ~write tls.js:315
3396 100.0% LazyCompile: *Writable.write _stream_writable.js:155
1063 20.9% LazyCompile: *write tls.js:315
1063 100.0% LazyCompile: *Writable.write _stream_writable.js:155
370 7.3% LazyCompile: *Writable.write _stream_writable.js:155
370 100.0% LazyCompile: ~write _stream_readable.js:546
186 3.7% LazyCompile: ~callback tls.js:753
180 96.8% LazyCompile: *onclienthello tls.js:748
6 3.2% LazyCompile: ~onclienthello tls.js:748
2417 29.2% LazyCompile: *read tls.js:397
2417 100.0% LazyCompile: *Readable.read _stream_readable.js:226
2320 96.0% LazyCompile: *Writable.write _stream_writable.js:155
2315 99.8% LazyCompile: ~write _stream_readable.js:546
57 2.4% LazyCompile: ~callback tls.js:753
57 100.0% LazyCompile: *onclienthello tls.js:748
691 8.4% LazyCompile: *Readable.read _stream_readable.js:226
675 97.7% LazyCompile: *write tls.js:315
675 100.0% LazyCompile: *Writable.write _stream_writable.js:155
674 99.9% LazyCompile: ~write _stream_readable.js:546
For those still coming here, this was caused somewhere in the SSL-handling code in a system node module. I re-configured my stack to do SSL termination at nginx, and use basic HTTP handling at the node level, and this problem went away entirely.