How can I detect these x coordinates? - algorithm

The blue points are a scatterplot (x-y points), not "lines"
The question is, how can I detect the x coordinates at which the "lines" end (marked with green arrows)
I think that the coordinates are proportional to powers of 2 (here is the same chart in logarithmic scale)
The red line shows the accumulated number of points. There is not a fall on the point density where the lines end.
The "lines" do not end all at the same x coordinate, and the y coordinates depend on the dataset (not all are at the same scales, so I cannot just hardcode one y coordinate)
Maybe if I could detect the periodicity of the y coordinate, I could group the points by their approximate maximum x value. (note that the minimum y separation is also unknown)
I'm not asking about a particular language but a suggestion for an algorithm. (although I prefer R or python)
I don't even know what tag to use for this question, I welcome comments on the appropriate tags.
Here is a sample file, and here is a snippet from it:
0.000035048,-0.000012875
0.000035107,0.000000060
0.000039995,0.000004888
0.000036120,-0.000003874
0.000036716,0.000000596
0.000037014,0.000000298
0.000038803,0.000001788
0.000039697,0.000000894
0.000039995,0.000000298
0.000041783,0.000001788
0.000039101,-0.000002682
0.000041485,0.000002384
0.000041008,-0.000000477
0.000039995,-0.000001013
0.000040054,0.000000060
0.000040889,0.000000834
0.000041485,0.000000596
0.000044882,0.000003397
0.000044882,0.000000000
0.000047863,0.000002980
0.000045002,-0.000002861
0.000045002,0.000000000
0.000047684,0.000002682
0.000047326,-0.000000358
0.000047207,-0.000000119
0.000047207,0.000000000
0.000047386,0.000000179
0.000047386,0.000000000
0.000047445,0.000000060
0.000047982,0.000000536
0.000048995,0.000001013
0.000050008,0.000001013
0.000053227,0.000003219
0.000054479,0.000001252
0.000054777,0.000000298
0.000053108,-0.000001669
0.000050008,-0.000003099
0.000051796,0.000001788
0.000051618,-0.000000179
0.000052810,0.000001192
0.000052810,0.000000000
0.000051975,-0.000000834
0.000051558,-0.000000417
0.000053465,0.000001907
0.000053406,-0.000000060
0.000053406,0.000000000
0.000053406,0.000000000
0.000052810,-0.000000596
0.000052810,0.000000000
0.000052571,-0.000000238
0.000052571,0.000000000
0.000051975,-0.000000596
0.000050008,-0.000001967
0.000048280,-0.000001729
0.000047445,-0.000000834
0.000047445,0.000000000
0.000047445,0.000000000
0.000047445,0.000000000
0.000049531,0.000002086
0.000049353,-0.000000179
0.000049412,0.000000060
0.000049412,0.000000000
0.000048876,-0.000000536
0.000048518,-0.000000358
0.000047982,-0.000000536
0.000047803,-0.000000179
0.000047803,0.000000000
0.000048935,0.000001132
0.000047743,-0.000001192
0.000047624,-0.000000119
0.000047445,-0.000000179
0.000047147,-0.000000298
0.000046015,-0.000001132
0.000044882,-0.000001132
0.000043213,-0.000001669
0.000039995,-0.000003219
0.000039995,0.000000000
0.000039279,-0.000000715
0.000039995,0.000000715
0.000039995,0.000000000
0.000041008,0.000001013
0.000041068,0.000000060
0.000044882,0.000003815
0.000041544,-0.000003338
0.000041544,0.000000000
0.000043392,0.000001848
0.000042021,-0.000001371
0.000042021,0.000000000
0.000043929,0.000001907
0.000044763,0.000000834
0.000042200,-0.000002563
0.000044465,0.000002265
0.000044465,0.000000000
0.000044465,0.000000000
0.000043511,-0.000000954
0.000043988,0.000000477
0.000043929,-0.000000060
0.000043929,0.000000000
0.000043929,0.000000000
0.000043929,0.000000000
0.000043869,-0.000000060
0.000043809,-0.000000060
0.000043809,0.000000000
0.000045300,0.000001490
0.000045300,0.000000000
0.000045717,0.000000417
0.000045478,-0.000000238
0.000045478,0.000000000
0.000045478,0.000000000
0.000044763,-0.000000715
0.000044763,0.000000000
0.000044465,-0.000000298
0.000042796,-0.000001669
0.000042856,0.000000060
0.000042856,0.000000000
0.000044405,0.000001550
0.000044286,-0.000000119
0.000044286,0.000000000
0.000044286,0.000000000
0.000044286,0.000000000
0.000044405,0.000000119
0.000044465,0.000000060
0.000045478,0.000001013
0.000045478,0.000000000
0.000045419,-0.000000060
0.000045002,-0.000000417
0.000043631,-0.000001371
0.000043631,0.000000000
0.000044882,0.000001252
0.000044882,0.000000000
0.000044823,-0.000000060
0.000044703,-0.000000119
0.000044644,-0.000000060
0.000044644,0.000000000
0.000048399,0.000003755
0.000048161,-0.000000238
0.000048101,-0.000000060
0.000048161,0.000000060
0.000048161,0.000000000
0.000048161,0.000000000
0.000048459,0.000000298
0.000048459,0.000000000
0.000048459,0.000000000
0.000048280,-0.000000179
0.000048280,0.000000000
0.000048459,0.000000179
0.000048280,-0.000000179
0.000047982,-0.000000298
0.000048280,0.000000298
0.000048220,-0.000000060
0.000047982,-0.000000238
0.000047982,0.000000000
0.000047982,0.000000000
0.000048399,0.000000417
0.000048399,0.000000000
0.000048459,0.000000060
0.000048399,-0.000000060
0.000048399,0.000000000
0.000046253,-0.000002146
0.000046194,-0.000000060
0.000047982,0.000001788
0.000048161,0.000000179
0.000048161,0.000000000
0.000048161,0.000000000
0.000047982,-0.000000179
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047088,-0.000000894
0.000047982,0.000000894
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047266,-0.000000715
0.000047982,0.000000715
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000048399,0.000000417
0.000048399,0.000000000
0.000048399,0.000000000
0.000048399,0.000000000
0.000047982,-0.000000417
0.000047982,0.000000000
0.000047266,-0.000000715
0.000047982,0.000000715
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047505,-0.000000477
0.000047505,0.000000000
0.000047982,0.000000477
0.000047982,0.000000000
0.000047982,0.000000000
0.000048339,0.000000358
0.000048339,0.000000000
0.000048399,0.000000060
0.000047803,-0.000000596
0.000048280,0.000000477
0.000048280,0.000000000
0.000048280,0.000000000
0.000048280,0.000000000
0.000048280,0.000000000
0.000048280,0.000000000
0.000048161,-0.000000119
0.000048161,0.000000000
0.000048101,-0.000000060
0.000048101,0.000000000
0.000047922,-0.000000179
0.000047863,-0.000000060
0.000047028,-0.000000834
0.000047028,0.000000000
0.000047982,0.000000954
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047088,-0.000000894
0.000047982,0.000000894
0.000047982,0.000000000
0.000047505,-0.000000477
0.000047922,0.000000417
0.000047505,-0.000000417
0.000047207,-0.000000298
0.000047147,-0.000000060
0.000047207,0.000000060
0.000047088,-0.000000119
0.000047088,0.000000000
0.000047028,-0.000000060
0.000047028,0.000000000
0.000047028,0.000000000
0.000047028,0.000000000
0.000047803,0.000000775
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000047922,0.000000119
0.000047863,-0.000000060
0.000047863,0.000000000
0.000047863,0.000000000
0.000047922,0.000000060
0.000047922,0.000000000
0.000047922,0.000000000
0.000047863,-0.000000060
0.000047863,0.000000000
0.000047028,-0.000000834
0.000046611,-0.000000417
0.000046611,0.000000000
0.000046611,0.000000000
0.000046611,0.000000000
0.000046611,0.000000000
0.000046611,0.000000000
0.000046015,-0.000000596
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000045002,-0.000001013
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000043869,-0.000001132
0.000043690,-0.000000179
0.000043690,0.000000000
0.000045776,0.000002086
0.000045776,0.000000000
0.000045776,0.000000000
0.000045776,0.000000000
0.000043690,-0.000002086
0.000043690,0.000000000
0.000043690,0.000000000
0.000043750,0.000000060
0.000043750,0.000000000
0.000043690,-0.000000060
0.000045776,0.000002086
0.000045776,0.000000000
0.000045776,0.000000000
0.000045776,0.000000000
0.000045776,0.000000000
0.000045776,0.000000000
0.000045776,0.000000000
0.000045717,-0.000000060
0.000045717,0.000000000
0.000045717,0.000000000
0.000044107,-0.000001609
0.000044107,0.000000000
0.000044107,0.000000000
0.000044107,0.000000000
0.000044107,0.000000000
0.000045776,0.000001669
0.000046015,0.000000238
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000046015,0.000000000
0.000045002,-0.000001013
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000044167,-0.000000834
0.000044167,0.000000000
0.000044167,0.000000000
0.000045300,0.000001132
0.000045300,0.000000000
0.000045300,0.000000000
0.000046790,0.000001490
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046790,0.000000000
0.000046730,-0.000000060
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000046730,0.000000000
0.000045121,-0.000001609
0.000045121,0.000000000
0.000045121,0.000000000
0.000045121,0.000000000
0.000045121,0.000000000
0.000045002,-0.000000119
0.000045002,0.000000000
0.000045002,0.000000000
0.000044644,-0.000000358
0.000043750,-0.000000894
0.000042975,-0.000000775
0.000042796,-0.000000179
0.000042796,0.000000000
0.000042796,0.000000000
0.000042140,-0.000000656
0.000042140,0.000000000
0.000042140,0.000000000
0.000042200,0.000000060
0.000042200,0.000000000
0.000045002,0.000002801
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000042379,-0.000002623
0.000042379,0.000000000
0.000044882,0.000002503
0.000044882,0.000000000
0.000044882,0.000000000
0.000044823,-0.000000060
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044882,0.000000060
0.000044882,0.000000000
0.000044882,0.000000000
0.000044823,-0.000000060
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044823,0.000000000
0.000044882,0.000000060
0.000042677,-0.000002205
0.000042737,0.000000060
0.000044763,0.000002027
0.000044763,0.000000000
0.000044763,0.000000000
0.000043511,-0.000001252
0.000043511,0.000000000
0.000044763,0.000001252
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044763,0.000000000
0.000044882,0.000000119
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000045002,0.000000119
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000046492,0.000001490
0.000046492,0.000000000
0.000047028,0.000000536
0.000047505,0.000000477
0.000047803,0.000000298
0.000047863,0.000000060
0.000047922,0.000000060
0.000047803,-0.000000119
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000047922,0.000000119
0.000047982,0.000000060
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047982,0.000000000
0.000047803,-0.000000179
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000047803,0.000000000
0.000045359,-0.000002444
0.000045359,0.000000000
0.000045300,-0.000000060
0.000045359,0.000000060
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000044525,-0.000000834
0.000044525,0.000000000
0.000044525,0.000000000
0.000044525,0.000000000
0.000044525,0.000000000
0.000044525,0.000000000
0.000044525,0.000000000
0.000044525,0.000000000
0.000045359,0.000000834
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000043988,-0.000001371
0.000043988,0.000000000
0.000043988,0.000000000
0.000045359,0.000001371
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000044048,-0.000001311
0.000045359,0.000001311
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000045359,0.000000000
0.000044525,-0.000000834
0.000045121,0.000000596
0.000045121,0.000000000
0.000045121,0.000000000
0.000045121,0.000000000
0.000045121,0.000000000
0.000045121,0.000000000
0.000044048,-0.000001073
0.000043988,-0.000000060
0.000043750,-0.000000238
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043809,0.000000060
0.000044882,0.000001073
0.000044882,0.000000000
0.000044882,0.000000000
0.000044882,0.000000000
0.000043809,-0.000001073
0.000043750,-0.000000060
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043750,0.000000000
0.000043631,-0.000000119
0.000043631,0.000000000
0.000043631,0.000000000
0.000043571,-0.000000060
0.000043631,0.000000060
0.000043631,0.000000000
0.000045359,0.000001729
0.000047028,0.000001669
0.000047028,0.000000000
0.000047028,0.000000000
0.000047028,0.000000000
0.000043988,-0.000003040
0.000045955,0.000001967
0.000045955,0.000000000
0.000045955,0.000000000
0.000045955,0.000000000
0.000045955,0.000000000
0.000045955,0.000000000
0.000045002,-0.000000954
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000045002,0.000000000
0.000044107,-0.000000894
0.000044107,0.000000000
0.000044107,0.000000000
0.000044107,0.000000000
0.000044107,0.000000000
0.000043690,-0.000000417
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043690,0.000000000
0.000043988,0.000000298
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000
0.000043988,0.000000000


I had a go on analysing the numbers, starting with this Python script:
file = open(r"C:\Users\timok\Downloads\sample_.csv")
d = {}
for line in file:
x, y = map(float, line.split(","))
d[y] = max(x, d.get(y, x))
for y, x in sorted(d.items()):
print(y, x)
This aggregates the data by Y-coordinate, keeping the maximum X-coordinate. This seemed already a good push in the right direction.
Looking at these results, I could see that these coordinates were multiples of a common (small) factor. After playing around, I found that factor to be near 1/16777215, so that allowed me to map the coordinates to integers:
d = {}
c = 16777215
for line in file:
x, y = map(float, line.split(","))
y = round(y*c)
x = round(x*c)
d[y] = max(x, d.get(y, x))
for y, x in sorted(d.items()):
print(y, x)
Looking at the Y-values around 0, the output is:
...
1 2044
2 4096
3 2050
4 8192
5 2052
6 4100
7 2039
8 16384
9 1991
10 4104
11 2041
12 8200
13 1896
14 4108
15 2011
16 32768
...
And here we see the pattern arise: where Y is a power of 2, the corresponding maximum X coordinate is also a power of 2 and also doubling each time. We see a little noise on the first value (2044 instead of 2048), but the next few values are powers of 2:
...
1 2044
2 4096
4 8192
8 16384
16 32768
...
Looking further, we see this pattern:
The (mapped) X-coordinate is 211+z, where z represents the number of trailing zeroes that the (mapped) Y-coordinate has in its binary representation. So:
y
binary
z
x = 211+z
1
00001
0
2048
2
00010
1
4096
3
00011
0
2048
4
00100
2
8192
5
00101
0
2048
6
00110
1
4096
7
00111
0
2048
8
01000
3
16384
9
01001
0
2048
10
01010
1
4096
11
01011
0
2048
12
01100
2
8192
13
01101
0
2048
14
01110
1
4096
15
01111
0
2048
16
10000
4
32768
So the boundary values you were looking for are 211+z for any natural number z. But this is after we did the mapping. So without the mapping, we get:
211+z / 16777215

Related

Job error on a multi-chain chainlink node version:1.2.0

I have a chainlink node running on two chains evmChainID=4(rinkeby) and evmChainID=80001(Mumbai-polygon)
I have deployed a TestnetConsumer contract on Mumbai which fetches the latest Ethereum price.
The Jobspec created for the consuming contract is as follows:
type = "directrequest"
schemaVersion = 1
name = "Get > Uint256-MUMBAI-POLYGON"
externalJobID = "1d77fe57-fd0c-4ee7-ba6e-e5e40a0e72c4"
maxTaskDuration = "0s"
contractAddress = "0x-my-oracle-contract"
evmChainID = "80001"
minIncomingConfirmations = 0
observationSource = """
decode_log [type="ethabidecodelog"
abi="OracleRequest(bytes32 indexed specId, address requester, bytes32 requestId, uint256 payment, address callbackAddr, bytes4 callbackFunctionId, uint256 cancelExpiration, uint256 dataVersion, bytes data)"
data="$(jobRun.logData)"
topics="$(jobRun.logTopics)"]
decode_cbor [type="cborparse" data="$(decode_log.data)"]
fetch [type="http" method=GET url="$(decode_cbor.get)"]
parse [type="jsonparse" path="$(decode_cbor.path)" data="$(fetch)"]
multiply [type="multiply" input="$(parse)" times=100]
encode_data [type="ethabiencode" abi="(uint256 value)" data="{ \\"value\\": $(multiply) }"]
encode_tx [type="ethabiencode"
abi="fulfillOracleRequest(bytes32 requestId, uint256 payment, address callbackAddress, bytes4 callbackFunctionId, uint256 expiration, bytes32 data)"
data="{\\"requestId\\": $(decode_log.requestId), \\"payment\\": $(decode_log.payment), \\"callbackAddress\\": $(decode_log.callbackAddr), \\"callbackFunctionId\\": $(decode_log.callbackFunctionId), \\"expiration\\": $(decode_log.cancelExpiration), \\"data\\": $(encode_data)}"
]
submit_tx [type="ethtx" to="0x-my-oracle-contract" data="$(encode_tx)" evmChainID="80001"]
decode_log -> decode_cbor -> fetch -> parse -> multiply -> encode_data -> encode_tx -> submit_tx
"""
The job runs keep getting success and error alternatively. The error happens because the node tries to send the txn using the key bound to evmChainID=4 as shown in the screenshot.
Errors:
while creating transaction: cannot send transaction on chain ID 80001; eth key with address 0xED3413f20d7F16B5E8934aFf935FD763a3Bd63Cf is pegged to chain ID 4: task run failed
It seems its an error related to some default config. Please help. Thanks
The expectation is the Job spec written specific for evmChainID=80001 will use the key bound to the same evmChainID. The keys were created with CLI.

Chainlink Node Operator Create Job, Unknown Task Type "ETHTX"

I am a Chainlink beginner attempting to set up a Chainlink node on Rinkeby. I have the node running and can access the operator GUI. When I attempt to set up a job (following the documentation here: https://docs.chain.link/docs/fulfilling-requests/) I copy and paste the TOML blob into the field, update my oracle address.
type = "directrequest"
schemaVersion = 1
name = "Get > Uint256"
contractAddress = "updated_addr"
maxTaskDuration = "0s"
observationSource = """
decode_log [type="ethabidecodelog"
abi="OracleRequest(bytes32 indexed specId, address requester, bytes32 requestId, uint256 payment, address callbackAddr, bytes4 callbackFunctionId, uint256 cancelExpiration, uint256 dataVersion, bytes data)"
data="$(jobRun.logData)"
topics="$(jobRun.logTopics)"]
decode_cbor [type="cborparse" data="$(decode_log.data)"]
fetch [type="http" method=GET url="$(decode_cbor.get)"]
parse [type="jsonparse" path="$(decode_cbor.path)" data="$(fetch)"]
multiply [type="multiply" input="$(parse)" times=100]
encode_data [type="ethabiencode" abi="(uint256 value)" data="{ \\"value\\": $(multiply) }"]
encode_tx [type="ethabiencode"
abi="fulfillOracleRequest(bytes32 requestId, uint256 payment, address callbackAddress, bytes4 callbackFunctionId, uint256 expiration, bytes32 data)"
data="{\\"requestId\\": $(decode_log.requestId), \\"payment\\": $(decode_log.payment), \\"callbackAddress\\": $(decode_log.callbackAddr), \\"callbackFunctionId\\": $(decode_log.callbackFunctionId), \\"expiration\\": $(decode_log.cancelExpiration), \\"data\\": $(encode_data)}"
]
submit_tx [type="ethtx" to="updated_addr" data="$(encode_tx)"]
decode_log -> decode_cbor -> fetch -> parse -> multiply -> encode_data -> encode_tx -> submit_tx
"""
When I attempt to create the job, I get the following error message:
UnmarshalTaskFromMap: unknown task type: "ethtx"
I did some digging in their documentation to see if they had changed the task name, and found this link to this specific task, and it seems to match the example code above.
https://docs.chain.link/docs/jobs/task-types/eth-tx/
I also attempted "eth tx", "eth_tx" and also ethtx without quotes without luck. I assume this has been depreciated and the documentation has not caught up.
Thank you for your time.

It's ethtx, you should just update your Chainlink Node to version 1.0.0 or above

WM_CHAR and strange characters with Alt-key

I have an MFC CWnd derived text editor and when I (accidentally) press Alt+A, WM_CHAR inserts a character '☺'.
Spy++ logs (I can't copy/paste - I had to type this):
WM_SYSKEYDOWN nVirtKey:VK_MENU cRepeat:1 ScanCode:38 fExtended:0 AltDown:1 fRepeat:0 fUp:0
WM_SYSKEYDOWN nVirtKey:VK_MENU cRepeat:1 ScanCode:38 fExtended:0 AltDown:1 fRepeat:1 fUp:0
WM_SYSKEYDOWN nVirtKey:'A' cRepeat:1 ScanCode:1 fExtended:0 AltDown:1 fRepeat:0 fUp:0
WM_SYSCHAR chCharCode:97 cRepeat:1 ScanCode:1E fExtended:0 AltDown:1 fRepeat:0 fUp:0
WM_SYSKEYUP nVirtKey:'A' cRepeat:1 ScanCode:1E fExtended:0 AltDown:1 fRepeat:1 fUp:1
WM_KEYUP nVirtKey:VK_MENU cRepeat:1 ScanCode:38 fExtended:0 AltDown:0 fRepeat:1 fUp:1
WM_CHAR chCharCode:63 cRepeat:1 ScanCode:38 fExtended:0 AltDown:1 fRepeat:1 fUp:1
My debugger says OnChar() gets nChar = 0x0000263a, nFlags = 0x0000c038, nRepCnt=1.
How can I distinguish this code from a proper code like AltGr+m = µ on a German keyboard?

Interpreting Section object in kernel dump

I'm trying to track down issues with a 3thParty application. The path currently being investigated is to look into a Section object that get's created in each process: rpsPdf10.mutex.
If the name of the object is any indication for it's intended usage, I'm not sure why they choose a Section object and use it as a Mutex but that's likely largely irrelevant.
Using LiveKd I've issued following command's trying to get detailed info of the Section object
0: kd>!process 0 0 3thParty.exe
...
PROCESS fffffa800ea80060
SessionId: 0 Cid: 0a00 Peb: fffdf000 ParentCid: 014c
DirBase: 99349000 ObjectTable: fffff8a004448bf0 HandleCount: 338.
Image: 3thParty.exe
...
0: kd> !handle 0 7 fffffa800ea80060
...
08 fffff8a012e26710 Section rpsPdf10.mutex
...
0: kd> !object fffff8a012e26710
Object: fffff8a012e26710 Type: (fffffa800cd7cea0) Section
ObjectHeader: fffff8a012e266e0 (new version)
HandleCount: 38 PointerCount: 39
Directory Object: fffff8a00a980080 Name: rpsPdf10.mutex
0: kd> dt nt!_FILE_OBJECT fffff8a012e26710
+0x000 Type : 0n256
+0x002 Size : 0n0
+0x008 DeviceObject : 0x000000000008dfb0 _DEVICE_OBJECT
+0x010 Vpb : 0xfffffa80c0000001 _VPB
+0x018 FsContext : (null)
+0x020 FsContext2 : 0xfffffa8000000034 Void
+0x028 SectionObjectPointer : 0xfffff8a0102d7820 _SECTION_OBJECT_POINTERS
+0x030 PrivateCacheMap : 0x0000000000001000 Void
+0x038 FinalStatus : 0n73728
+0x040 RelatedFileObject : 0x63536153030a040c _FILE_OBJECT
+0x048 LockOperation : 0x74 't'
+0x049 DeletePending : 0 ''
+0x04a ReadAccess : 0x65 'e'
+0x04b WriteAccess : 0 ''
+0x04c DeleteAccess : 0x73 's'
+0x04d SharedRead : 0 ''
+0x04e SharedWrite : 0x74 't'
The string 't' 'e' 's' 't' in the output definitely sticks out so
either I'm following a wrong path -> tx to Blabb, this is certain. It's not a fileobject but the question remains how to find out more information about the Section object. It's still curious and/or a rather unfortunate coincidence that following the section and control area pointers I derived from the fileobject info do seem correct?!
or there's something wrong with the Section object
or ... ?
tldr;
Following the _SECTION_OBJECT_POINTERS of the _FILE_OBJECT structure above, I arrive at a
NumberOfMappedViews of 0x26 (= HandleCount: 38)
NumberOfUserReferences of 0x27 (= PointerCount: 39)
so for the moment I assume the path I've followed is correct.
0: kd> dt nt!_SECTION_OBJECT_POINTERS 0xfffff8a0102d7820
+0x000 DataSectionObject : 0xfffffa800fbed900 Void
+0x008 SharedCacheMap : 0x0008000000000001 Void
+0x010 ImageSectionObject : 0x0000000000000001 Void
0: kd> dt nt!_CONTROL_AREA 0xfffffa800fbed900
+0x000 Segment : 0xfffff8a0102d7820 _SEGMENT
+0x008 DereferenceList : _LIST_ENTRY [ 0x0000000000000000 - 0x0000000000000000 ]
+0x018 NumberOfSectionReferences : 1
+0x020 NumberOfPfnReferences : 0
+0x028 NumberOfMappedViews : 0x26
+0x030 NumberOfUserReferences : 0x27
Edit
The object header looks like this
0: kd> dt nt!_OBJECT_HEADER fffff8a012e266e0
+0x000 PointerCount : 0n39
+0x008 HandleCount : 0n38
+0x008 NextToFree : 0x00000000`00000026 Void
+0x010 Lock : _EX_PUSH_LOCK
+0x018 TypeIndex : 0x21 '!'
+0x019 TraceFlags : 0 ''
+0x01a InfoMask : 0xa ''
+0x01b Flags : 0 ''
+0x020 ObjectCreateInfo : 0xfffffa80`0e505140 _OBJECT_CREATE_INFORMATION
+0x020 QuotaBlockCharged : 0xfffffa80`0e505140 Void
+0x028 SecurityDescriptor : 0xfffff8a0`1ba076a8 Void
+0x030 Body : _QUAD
Edit 2
following #blabb's answer adjusting for architecture
0: kd> ? #$proc
Evaluate expression: -6047068061600 = fffffa80`0ea80060
0: kd> dx (char *)#$proc->ImageFileName
(char *)#$proc->ImageFileName : 0xfffffa800ea80340 : [Type: char *] : "3thParty.exe"
0: kd> !handle 0 0 #$proc section
...
0474: Object: fffff8a012e26710 GrantedAccess: 000f0007
...
0: kd> !object fffff8a012e26710
Object: fffff8a012e26710 Type: (fffffa800cd7cea0) Section
ObjectHeader: fffff8a012e266e0 (new version)
HandleCount: 38 PointerCount: 39
Directory Object: fffff8a00a980080 Name: rpsPdf10.mutex
0: kd> ?? (unsigned long) (#FIELD_OFFSET(nt!_OBJECT_HEADER, Body))
unsigned long 0x30
0: kd> dt nt!_object_header 0xfffff8a012e26710-0x30
+0x000 PointerCount : 0n39
+0x008 HandleCount : 0n38
+0x008 NextToFree : 0x00000000`00000026 Void
+0x010 Lock : _EX_PUSH_LOCK
+0x018 TypeIndex : 0x21 '!'
+0x019 TraceFlags : 0 ''
+0x01a InfoMask : 0xa ''
+0x01b Flags : 0 ''
+0x020 ObjectCreateInfo : 0xfffffa80`0e505140 _OBJECT_CREATE_INFORMATION
+0x020 QuotaBlockCharged : 0xfffffa80`0e505140 Void
+0x028 SecurityDescriptor : 0xfffff8a0`1ba076a8 Void
+0x030 Body : _QUAD
0: kd> x nt!ObTypeIndexTable
fffff800`01a70c00 nt!ObTypeIndexTable = <no type information>
0: kd> dt -r1 nt!_SECTION_OBJECT 0xfffff8a012e26710
+0x000 StartingVa : 0x00000022`00000100 Void
+0x008 EndingVa : 0x00000000`0008dfb0 Void
+0x010 Parent : 0xfffffa80`c0000001 Void
+0x018 LeftChild : (null)
+0x020 RightChild : 0xfffffa80`00000034 Void
+0x028 Segment : 0xfffff8a0`102d7820 _SEGMENT_OBJECT
+0x000 BaseAddress : 0xfffffa80`0fbed900 Void
+0x008 TotalNumberOfPtes : 1
+0x010 SizeOfSegment : _LARGE_INTEGER 0x1
+0x018 NonExtendedPtes : 0x1000
+0x01c ImageCommitment : 0
+0x020 ControlArea : (null)
+0x028 Subsection : (null)
+0x030 MmSectionFlags : 0xfffffa80`10987b10 _MMSECTION_FLAGS
+0x038 MmSubSectionFlags : 0x00000000`03400000 _MMSUBSECTION_FLAGS
0: kd> dc 0xfffff8a012e26710-0x30-0x50
fffff8a0`12e26690 030c0408 f4636553 0e1a02e0 fffffa80 ....Sec.........
fffff8a0`12e266a0 00000048 000000b8 0000001c fffffa80 H...............
fffff8a0`12e266b0 0e505140 fffffa80 00000000 00000000 #QP.............
fffff8a0`12e266c0 0a980080 fffff8a0 001c001c 00000000 ................
fffff8a0`12e266d0 10eb8770 fffff8a0 00000000 00000008 p...............
fffff8a0`12e266e0 00000027 00000000 00000026 00000000 '.......&.......
fffff8a0`12e266f0 00000000 00000000 000a0021 fffff8a0 ........!.......
fffff8a0`12e26700 0e505140 fffffa80 1ba076a8 fffff8a0 #QP......v......
0: kd> !pool 0xfffff8a012e26710-0x30-0x50 2
Pool page fffff8a012e26690 region is Paged pool
*fffff8a012e26690 size: c0 previous size: 80 (Allocated) *Sect (Protected)
Pooltag Sect : Section objects

This is a 32 bit machine running windows 7
Commands used are architecture agnostic but pointer arithmetic are arch dependent
Current process
kd> ? #$proc
Evaluate expression: -2061895528 = 8519f898
Process Name From EPROCESS->ImageFileName
kd> dx (char *)#$proc->ImageFileName
(char *)#$proc->ImageFileName : 0xffffffff8519fa04 : "windbg.exe" [Type: char *]
lets Search For Some Section Handles in this process
the TypeName is CaseSensitive
kd> !handle 0 3 #$proc Section
Searching for handles of type Section
PROCESS 8519f898 SessionId: 1 Cid: 0138 Peb: 7ffd8000 ParentCid: 0d04
DirBase: 7e257560 ObjectTable: b91a3520 HandleCount: 254.
Image: windbg.exe
Handle table at b91a3520 with 254 entries in use
00c0: Object: 9a10bc58 GrantedAccess: 00000004 Entry: 9945b180
Object: 9a10bc58 Type: (84eb6040) Section
ObjectHeader: 9a10bc40 (new version)
HandleCount: 6 PointerCount: 6
!handle 0 3 flag dumps object specific information which can be reverified using !object {object address}
kd> !object 9a10bc58
Object: 9a10bc58 Type: (84eb6040) Section
ObjectHeader: 9a10bc40 (new version)
HandleCount: 6 PointerCount: 6
each object has an objectheader in 32 bit it is 18 bytes prior to object address that is sizeof(nt!_OBJECT_HEADER- sizeof(obheader->Body)) body is embedded in HEADER as the last member and is variable sized
kd> ?? (unsigned long ) (#FIELD_OFFSET(nt!_OBJECT_HEADER , Body))
unsigned long 0x18
_OBJECT_HEADER is as follows (though sizes haven't changed there are differences between new version header and old version header)
kd> dt nt!_object_header 9a10bc58-0x18
+0x000 PointerCount : 0n6
+0x004 HandleCount : 0n6
+0x004 NextToFree : 0x00000006 Void
+0x008 Lock : _EX_PUSH_LOCK
+0x00c TypeIndex : 0x21 '!'
+0x00d TraceFlags : 0 ''
+0x00e InfoMask : 0x8 ''
+0x00f Flags : 0 ''
+0x010 ObjectCreateInfo : 0x82f7aa00 _OBJECT_CREATE_INFORMATION
+0x010 QuotaBlockCharged : 0x82f7aa00 Void
+0x014 SecurityDescriptor : (null)
+0x018 Body : _QUAD
the old version header had _OBJECT_TYPE directly in the header
the new version is an index into an array
here the type index is 0x21
the array of Type is at
kd> x nt!ObTypeIndexTable
82f88580 nt!ObTypeIndexTable = <no type information>
you can write a script like this to dump all the types
function log(instr)
{
host.diagnostics.debugLog(instr + "\n");
}
function exec (cmdstr)
{
return host.namespace.Debugger.Utility.Control.ExecuteCommand(cmdstr);
}
function dumptypeindex()
{
var cpob = host.createPointerObject
var titab = exec("x nt!ObTypeIndexTable").First().substr(0,8)
var obtype = cpob(host.parseInt64(titab , 16),"nt","_OBJECT_TYPE **")
var i = 2
while(obtype[i] !=0 )
{
log("index = "+i+"\t"+ host.memory.readWideString(obtype[i].Name.Buffer))
i++
}
}
executing this script would yield the types as follows
kd> .scriptload c:\wdscr\dumptypeindex.js
JavaScript script successfully loaded from 'c:\dumptypeindex.js'
kd> dx #$scriptContents.dumptypeindex()
index = 2 Type
index = 3 Directory
index = 4 SymbolicLink
index = 5 Token
index = 6 Job
index = 7 Process
index = 8 Thread
index = 9 UserApcReserve
index = 10 IoCompletionReserve
index = 11 DebugObject
index = 12 Event
index = 13 EventPair
index = 14 Mutant
index = 15 Callback
index = 16 Semaphore
index = 17 Timer
index = 18 Profile
index = 19 KeyedEvent
index = 20 WindowStation
index = 21 Desktop
index = 22 TpWorkerFactory
index = 23 Adapter
index = 24 Controller
index = 25 Device
index = 26 Driver
index = 27 IoCompletion
index = 28 File
index = 29 TmTm
index = 30 TmTxȂ؃扏楄
index = 31 TmRm
index = 32 TmEn
index = 33 Section
index = 34 Session
index = 35 Key
index = 36 ALPC Port
index = 37 PowerRequest
index = 38 WmiGuid
index = 39 EtwRegistration
index = 40 EtwConsumer
index = 41 FilterConnectionPort
index = 42 FilterCommunicationPort
index = 43 PcwObject
notice 0x21 = 0n33 = Section
given that we have a Section
we can dump the Section Object
kd> dt -r1 nt!_SECTION_OBJECT 9a10bc58
+0x000 StartingVa : 0x90f87b44 Void
+0x004 EndingVa : 0x82efb58a Void
+0x008 Parent : 0xc0802000 Void
+0x00c LeftChild : (null)
+0x010 RightChild : 0xc0c0a280 Void
+0x014 Segment : 0x995ed8d8 _SEGMENT_OBJECT
+0x000 BaseAddress : 0x86b65740 Void
+0x004 TotalNumberOfPtes : 0xdf
+0x008 SizeOfSegment : _LARGE_INTEGER 0x000000df`00080000
+0x010 NonExtendedPtes : 0xdf000
+0x014 ImageCommitment : 0
+0x018 ControlArea : (null)
+0x01c Subsection : (null)
+0x020 MmSectionFlags : 0x869f52a8 _MMSECTION_FLAGS
+0x024 MmSubSectionFlags : 0x02ea0000 _MMSUBSECTION_FLAGS
an object is preceded by object header which is preceded by the pool_header
kd> dc 9a10bc58-0x18-0x18
9a10bc28 060b0204 f4636553 00000720 00000070 ....Sec. ...p...
9a10bc38 00000000 00000000 00000006 00000006 ................
9a10bc48 00000000 00080021 82f7aa00 00000000 ....!...........
9a10bc58 90f87b44 82efb58a c0802000 00000000 D{....... ......
9a10bc68 c0c0a280 995ed8d8 000df000 00000000 ......^.........
9a10bc78 00012000 00000004 0670020b 6666744e . ........p.Ntff
9a10bc88 00f00702 00000a48 0000c0fe 00020000 ....H...........
9a10bc98 00000000 00000002 00000000 00000000 ................
notice the Sec tag Sect is used by SectionObjects
d> !pool 9a10bc58-0x18-0x18 2
Pool page 9a10bc28 region is Paged pool
*9a10bc28 size: 58 previous size: 20 (Allocated) *Sect (Protected)
Pooltag Sect : Section objects

Ruby - Hash to Array - Accessing Keys

I'm a little uncomfortable using chain of methods in Ruby. Here, I'm using such a chain.
So the question is, Tell me a better way to do the same.
I have this line run,
p["All_Products"]["Product"].map{|h| h["Categories"]["Category"] rescue nil }.compact.flatten.uniq
p equals this,
{"All_Products"=>
{"Product"=>
[{"ProductCode"=>"GFT",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1553", "CategoryName"=>"Bath and Body"}]}},
{"ProductCode"=>"GF-honey",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"GF-malbec",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"GF-olives",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"HC-flatiron",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1554", "CategoryName"=>"Hair Care"}]}}
]
}
}
Finally I'm getting this,
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1553", "CategoryName"=>"Bath and Body"},
{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"},
{"CategoryID"=>"1554", "CategoryName"=>"Hair Care"},
{"CategoryID"=>"1673", "CategoryName"=>"Sweets"},
{"CategoryID"=>"1477", "CategoryName"=>"Home Decor"},
{"CategoryID"=>"1516", "CategoryName"=>"Furniture"},
{"CategoryID"=>"1517", "CategoryName"=>"Home Accessories"},
{"CategoryID"=>"1814", "CategoryName"=>"Apparel"},
{"CategoryID"=>"1815", "CategoryName"=>"Women"},
{"CategoryID"=>"1816", "CategoryName"=>"Men"}]

Will this help?:
products = {"All_Products"=>
{"Product"=>
[{"ProductCode"=>"GFT",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1553", "CategoryName"=>"Bath and Body"}]}},
{"ProductCode"=>"GF-honey",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"GF-malbec",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"GF-olives",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"}]}},
{"ProductCode"=>"HC-flatiron",
"Categories"=>
{"Category"=>
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1554", "CategoryName"=>"Hair Care"}]}}
]
}
}
then you can:
puts products['All_Products']['Product'].
collect{ |product| product['Categories'] && product['Categories']['Category'] }.
flatten.uniq
result:
[{"CategoryID"=>"1513", "CategoryName"=>"Beauty"},
{"CategoryID"=>"1515", "CategoryName"=>"Specials"},
{"CategoryID"=>"1553", "CategoryName"=>"Bath and Body"},
{"CategoryID"=>"1514", "CategoryName"=>"Gourmet Food"},
{"CategoryID"=>"1632", "CategoryName"=>"Specialty Items"},
{"CategoryID"=>"1554", "CategoryName"=>"Hair Care"}]
Also, please try not to create variable names such as p, puts, do etc. They're reserved Ruby methods.

Resources