I've written a couple contracts for L1 (Ethereum) and L2 (Starknet) and have them communicate here.
I can see that L1 sent the message I'm expecting, see this TX on etherscan. The last message in there however never executed on my L2 contract. I'm trying to figure out whether the L2 Sequencer has invoked the handler function of my contract and if so, whether/how it failed.
Does anyone here know how to find the TX that handles the invoke on L2? Or any other ideas/tools that would help figuring out why the l1_handler never executed/failed?
First thing, transactions coming from L1 are regular transactions and therefore their hash can be computed the same way as invoke transactions. To have more information on this you can check the documentation here. Now this is helpful to understand the theory but not that much to actually compute the tx hash.
Here is the L1 event that send a message to StarkNet and this is where I get the needed information to compute the hash
Address 0xde29d060d45901fb19ed6c6e959eb22d8626708e
Name LogMessageToL2 (index_topic_1 address fromAddress, index_topic_2 uint256 toAddress, index_topic_3 uint256 selector, uint256[] payload, uint256 nonce)View Source
Topics
0 0x7d3450d4f5138e54dcb21a322312d50846ead7856426fb38778f8ef33aeccc01
1 0x779b989d7358acd6ce64237f16bbef09f35f6ecc
2 1524569076953457512425355396075576585145183562308719695739798372277154230742
3 1285101517810983806491589552491143496277809242732141897358598292095611420389
Data
payload :
1393428179030720295440092695193628168230707649901849797435563042612822742693
11819812303435348947619
0
nonce :
69106
Here is the script I use applied to your transaction (this may change in the future)
from starkware.cairo.lang.vm.crypto import pedersen_hash
from starkware.cairo.common.hash_state import compute_hash_on_elements
from starkware.crypto.signature.fast_pedersen_hash import pedersen_hash
from typing import List
def calculate_transaction_hash_common(
tx_hash_prefix,
version,
contract_address,
entry_point_selector,
calldata,
max_fee,
chain_id,
additional_data,
hash_function=pedersen_hash,
) -> int:
calldata_hash = compute_hash_on_elements(data=calldata, hash_func=hash_function)
data_to_hash = [
tx_hash_prefix,
version,
contract_address,
entry_point_selector,
calldata_hash,
max_fee,
chain_id,
*additional_data,
]
return compute_hash_on_elements(
data=data_to_hash,
hash_func=hash_function,
)
def tx_hash_from_message(
from_address: str, to_address: int, selector: int, nonce: int, payload: List[int]
) -> str:
int_hash = calculate_transaction_hash_common(
tx_hash_prefix=510926345461491391292786, # int.from_bytes(b"l1_handler", "big")
version=0,
contract_address=to_address,
entry_point_selector=selector,
calldata=[int(from_address, 16), *payload],
max_fee=0,
chain_id=1536727068981429685321, # StarknetChainId.TESTNET.value
additional_data=[nonce],
)
return hex(int_hash)
print(
tx_hash_from_message(
from_address="0x779b989d7358acd6ce64237f16bbef09f35f6ecc",
to_address=1524569076953457512425355396075576585145183562308719695739798372277154230742,
selector=1285101517810983806491589552491143496277809242732141897358598292095611420389,
nonce=69106,
payload=[
1393428179030720295440092695193628168230707649901849797435563042612822742693,
11819812303435348947619,
0,
],
)
)
This outputs 0x4433250847579c56b12822a16205e12410f6ad35d8cfc2d6ab011a250eae77f that we can find here which was properly executed.
Related
I am currently trying to increase the Compute-Budget on my Solana Program on devnet. I am using solana version 1.9.9.
I have checked the anchor discord, and found this implementation to request a larger compute-budget. however, when I run this code-snippet
const data = Buffer.from(
Uint8Array.of(0, ...new BN(256000).toArray("le", 4))
);
const additionalComputeIx: TransactionInstruction = new TransactionInstruction({
keys: [],
programId: new PublicKey("ComputeBudget111111111111111111111111111111"),
data,
});
...
I get invalid instruction data. any idea why this could be?
I was getting Compute Budget Exceeded before adding this instruction to the transaction.
You're very close! The instruction definition for requesting an increase from the compute budget program contains a little-endian u32 for the units, but then also a little-endian u32 for the additional fee to add, which you must include, even if it's 0. So instead, you should try:
const data = Buffer.from(
Uint8Array.of(0, ...new BN(256000).toArray("le", 4), ...new BN(0).toArray("le", 4))
);
More information about the instruction at https://github.com/solana-labs/solana/blob/a6742b5838ffe6f37afcb24ab32ad2287a1514cf/sdk/src/compute_budget.rs#L10
If you look at how this go-cardano-client is making it's handshake request payload:
https://github.com/gocardano/go-cardano-client/blob/master/shelley/handshake.go#L64
versionTable.Add(cbor.NewPositiveInteger8(1), cbor.NewPositiveInteger(764824073))
versionTable.Add(cbor.NewPositiveInteger16(32770), cbor.NewPositiveInteger(764824073))
versionTable.Add(cbor.NewPositiveInteger16(32771), cbor.NewPositiveInteger(764824073))
But the grpc generated struct is:
type HandshakeRequest struct {
// Nonce for the server to authenticate its node ID with.
Nonce []byte `protobuf:"bytes,1,opt,name=nonce,proto3" json:"nonce,omitempty"`
}
And this []byte needs to come through for the nonce referenced:
https://github.com/input-output-hk/jormungandr/blob/master/jormungandr/src/network/service.rs#L60
It's length 32:
https://github.com/input-output-hk/jormungandr/blob/master/jormungandr/src/network/client/connect.rs#L58
https://github.com/input-output-hk/jormungandr/blob/6f324b706a13273afb6a0808e589735020bb59da/jormungandr/src/network/mod.rs#L73
So this line in the golang code:
versionTable.Add(cbor.NewPositiveInteger8(1), cbor.NewPositiveInteger(764824073))
can't be length 32 []byte right? How do I encode this:
req := HandshakeRequest{}
req.Nonce = []byte{}
for i := 0; i < 32; i++ {
req.Nonce = append(req.Nonce, byte(rand.Intn(256)))
}
into this versionTable "params"?
see also and proto
Edit: You seem to assume that the the handshake from gocardano/go-cardano-client and the one described in node.proto somehow are related to the same implementation. Actually, I don't think they do.
The TCP-based handshake follows the Shelley protocol specs and sends a payload with the encoded versionTable. The gRPC-based HandshakeRequest instead is, as you also considered, just a nonce. There's nothing in the proto schema that hints to the Shelley protocol. The comments on the Nonce field also say that quite explicitly: "Nonce for the server to authenticate its node ID with."
So it would be a bit strange to assume that this nonce and the versionTable payload have anything in common at all.
Edit 2: In addition, it seems the "Jormungandr" rust node implementation does not support Shelley at all, so when you say you can't connect to the nodes in the relay topology, I think you shouldn't look for answers in the Jormungandr repository. Instead, I think the relays run the Haskell implementations of the Ouroboros network.
Now as for why you can't connect, the go-cardano client panics on some unchecked type assertions, because after the QueryTip Shelley message chainSyncBlocks.RequestNext, the relay servers respond with a different mini-protocol altogether, transactionSubmission.msgRequestTxIds as shown by running the client with TCP and tracing the messages:
MiniProtocol: 4 / MessageMode: 1 / f1bb7f80800400058400f50003
Array: [4]
PositiveInteger8(0)
False
PositiveInteger8(0)
PositiveInteger8(3)
You'll also have the same result when sending a sync chain request with MiniProtocol 2 (ChainSyncHeaders). I checked the Shelley protocol specs but couldn't find an explicit indication about why the server would switch protocols... Unfortunately I'm not familiar enough with Haskell to gain further insight from the Ouroboros sources.
In the unexpected case that the nonce in the proto HandshakeRequest is indeed related to the Shelley protocol, its content might be the CBOR array in your linked Cardano client (speculations follows):
arr := cbor.NewArray()
arr.Add(cbor.NewPositiveInteger8(handshakeMessagePropose))
versionTable := cbor.NewMap()
arr.Add(versionTable)
versionTable.Add(...)
versionTable.Add(...)
versionTable.Add(...)
return []cbor.DataItem{arr}
By inspecting the client where the handshake request is used, we can see:
messageResponse, err := c.queryNode(multiplex.MiniProtocolIDMuxControl, handshakeRequest())
and then in queryNode:
sdu := multiplex.NewServiceDataUnit(miniProtocol, multiplex.MessageModeInitiator, dataItems)
...
c.socket.Write(sdu.Bytes())
The sdu.Bytes() method serializes the entire payload, in particular:
// EncodeList return CBOR representation for each item in the list
func EncodeList(list []DataItem) []byte {
result := []byte{}
for _, item := range list {
result = append(result, item.EncodeCBOR()...)
}
return result
}
The EncodeCBOR() method is implemented by both the Array and the Map used in the handshakeRequest() []cbor.DataItem function. Watch out that the handshake function returns a slice []cbor.DataItem which contains one Array item which contains (as documented) the handshakeMessagePropose and the versionTable map.
If you carefully follow how the serialization proceeds, you'll eventually obtain the breakdown of the byte array — hereafter in decimal:
[130 0 163 1 26 45 150 74 9 25 128 2 26 45 150 74 9 25 128 3 26 45 150 74 9]
Where:
130 is the array data item prefix
0 is the handshakeMessagePropose
163 is the map data item prefix
and the subsequent bytes are the versionTable map
It is 25 bytes in total. At this point, I don't know if the multiplex wrapper built in queryNode function is part of the nonce or not. With the full wrapper, the length of the serialized byte array goes up to 33. So excluding some control bits or whatnot, this might be what you're supposed to write into the HandshakeRequest.Nonce.
I am using Spark Streaming for creating a system to enrich incoming data from a cloudant database. Example -
Incoming Message: {"id" : 123}
Outgoing Message: {"id" : 123, "data": "xxxxxxxxxxxxxxxxxxx"}
My code for the driver class is as follows:
from Sample.Job import EnrichmentJob
from Sample.Job import FunctionJob
import pyspark
from pyspark.streaming.kafka import KafkaUtils
from pyspark import SparkContext, SparkConf, SQLContext
from pyspark.streaming import StreamingContext
from pyspark.sql import SparkSession
from kafka import KafkaConsumer, KafkaProducer
import json
class SampleFramework():
def __init__(self):
pass
#staticmethod
def messageHandler(m):
return json.loads(m.message)
#staticmethod
def processData(rdd):
if (rdd.isEmpty()):
print("RDD is Empty")
return
# Expand
expanded_rdd = rdd.mapPartitions(EnrichmentJob.enrich)
# Score
scored_rdd = expanded_rdd.map(FunctionJob.function)
# Publish RDD
def run(self, ssc):
self.ssc = ssc
directKafkaStream = KafkaUtils.createDirectStream(self.ssc, QUEUENAME, \
{"metadata.broker.list": META,
"bootstrap.servers": SERVER}, \
messageHandler= SampleFramework.messageHandler)
directKafkaStream.foreachRDD(SampleFramework.processData)
ssc.start()
ssc.awaitTermination()
Code for the the Enrichment Job is as follows:
class EnrichmentJob:
cache = {}
#staticmethod
def enrich(data):
# Assume that Cloudant Connector using the available config
cloudantConnector = CloudantConnector(config, config["cloudant"]["host"]["req_db_name"])
final_data = []
for row in data:
id = row["id"]
if(id not in EnrichmentJob.cache.keys()):
data = cloudantConnector.getOne({"id": id})
row["data"] = data
EnrichmentJob.cache[id]=data
else:
data = EnrichmentJob.cache[id]
row["data"] = data
final_data.append(row)
cloudantConnector.close()
return final_data
My question is - Is there someway to maintain [1]"a global cache on the main memory that is accessible to all workers" or [2]"local caches on each of the workers such that they remain persisted in the foreachRDD setting"?
I have already explored the following -
Broadcast Variables - Here we go the [1] way. As I understand, they are meant to be read-only and immutable. I have checked out this reference but it cites an example of unpersisting/persisting the broadcasted variable. Is this a good practice?
Static Variables - Here we go the [2] way. The class that is being referred to ("Enricher" in this case) maintains a cache in the form of a static variable dictionary. But it turns out that the ForEachRDD function spawns a completely new process for each incoming RDD and this removes the previously initiated static variable. This is the one coded above.
I have two possible solutions right now -
Maintain an offline cache on the file system.
Do the entire computation of this enrichment task on my driver node. This would cause the entire data to end up on driver and be maintained there. The cache object will be sent to the enrichment job as an argument to the mapping function.
Here obviously the first one looks better than the second, but I wish to conclude that these two are the only ways around, before committing to them. Any pointers would be appreciated!
Is there someway to maintain [1]"a global cache on the main memory that is accessible to all workers"
No. There is no "main memory" which can be accessed by all workers. Each worker runs in a separate process and communicates with external world with sockets. Not to mention separation between different physical nodes in non-local mode.
There are some techniques that can be applied to achieve worker scoped cache with memory mapped data (using SQLite being the simplest one) but it takes some additional effort to implement the right way (avoid conflicts and such).
or [2]"local caches on each of the workers such that they remain persisted in the foreachRDD setting"?
You can use standard caching techniques with scope limited to the individual worker processes. Depending on the configuration (static vs. dynamic resource allocation, spark.python.worker.reuse) it may or may not be preserved between multiple tasks and batches.
Consider following, simplified, example:
map_param.py:
from pyspark import AccumulatorParam
from collections import Counter
class CounterParam(AccumulatorParam):
def zero(self, v: Counter) -> Counter:
return Counter()
def addInPlace(self, acc1: Counter, acc2: Counter) -> Counter:
acc1.update(acc2)
return acc1
my_utils.py:
from pyspark import Accumulator
from typing import Hashable
from collections import Counter
# Dummy cache. In production I would use functools.lru_cache
# but it is a bit more painful to show with accumulator
cached = {}
def f_cached(x: Hashable, counter: Accumulator) -> Hashable:
if cached.get(x) is None:
cached[x] = True
counter.add(Counter([x]))
return x
def f_uncached(x: Hashable, counter: Accumulator) -> Hashable:
counter.add(Counter([x]))
return x
main.py:
from pyspark.streaming import StreamingContext
from pyspark import SparkContext
from counter_param import CounterParam
import my_utils
from collections import Counter
def main():
sc = SparkContext("local[1]")
ssc = StreamingContext(sc, 5)
cnt_cached = sc.accumulator(Counter(), CounterParam())
cnt_uncached = sc.accumulator(Counter(), CounterParam())
stream = ssc.queueStream([
# Use single partition to show cache in work
sc.parallelize(data, 1) for data in
[[1, 2, 3], [1, 2, 5], [1, 3, 5]]
])
stream.foreachRDD(lambda rdd: rdd.foreach(
lambda x: my_utils.f_cached(x, cnt_cached)))
stream.foreachRDD(lambda rdd: rdd.foreach(
lambda x: my_utils.f_uncached(x, cnt_uncached)))
ssc.start()
ssc.awaitTerminationOrTimeout(15)
ssc.stop(stopGraceFully=True)
print("Counter cached {0}".format(cnt_cached.value))
print("Counter uncached {0}".format(cnt_uncached.value))
if __name__ == "__main__":
main()
Example run:
bin/spark-submit main.py
Counter cached Counter({1: 1, 2: 1, 3: 1, 5: 1})
Counter uncached Counter({1: 3, 2: 2, 3: 2, 5: 2})
As you can see we get expected results:
For "cached" objects accumulator is updated only once per unique key per worker process (partition).
For not-cached objects accumulator is update each time key occurs.
I am trying to make asynchronous kernel calls to my GPGPU using CUDAfy .NET.
When I pass values to the kernel and copy them back to the host, I do not always get the value I expect.
I have a structure Foo with a byte Bar:
[Cudafy]
public struct Foo {
public byte Bar;
}
And I have a kernel I want to call:
[Cudafy]
public static void simulation(GThread thread, Foo[] f)
{
f[0].Bar = 3;
thread.SyncThreads();
}
I have a single thread with streamID = 1 (I tried using multiple threads, and noticed the issue. Reducing to a single thread didn't seem to fix the issue though).
//allocate
streamID = 1;
count = 1;
gpu.CreateStream(streamID);
Foo[] sF = new Foo[count];
IntPtr hF = gpu.HostAllocate<Foo>(count);
Foo[] dF = gpu.Allocate<Foo>(sF);
while (true)
{
//set value
sF[0].Bar = 1;
byte begin = sF[0].Bar;
//host -> pinned
GPGPU.CopyOnHost<Foo>(sF, 0, hF, 0, count);
sF[0].Bar = 2;
lock (gpu)
{
//pinned -> device
gpu.CopyToDeviceAsync<Foo>(hF, 0, dF, 0, count, streamID);
//run
gpu.Launch().simulation(dF);
//device -> pinned
gpu.CopyFromDeviceAsync<Foo>(dF, 0, hF, 0, count, streamID);
}
//WAIT
gpu.SynchronizeStream(streamID);
//pinned -> host
GPGPU.CopyOnHost<Foo>(hF, 0, sF, 0, count);
byte end = sF[0].Bar;
}
//de-allocate
gpu.Free(dF);
gpu.HostFree(hF);
gpu.DestroyStream(streamID);
First I create a stream on the GPU.
I am creating a regular structure Foo array of size 1 (sF) and setting it's Bar value to 1. Then I create pinned memory on the host (hF) for Foo as well. I also create memory on the device for Foo (dF).
I initialize the structure's Bar value to 1 then I copy it to the pinned memory (As a check, I set the value to 2 for the structure after copying to pinned, you'll see why later). Then I use a lock to ensure I have full access to the GPU and I queue a copy to dF, a run for the kernel, and a copy from dF. At this point I don't know when this will all actually run on the GPU... so I can call SynchronizeStream to wait on the host until the device is done.
When it's done, I can copy the pinned memory (hF) to the shared memory (sF). When I get the value, it's usually a 3 (which was set on the device) or a 1 (which means either the value wasn't set in the kernel, or the new value wasn't copied to the pinned memory). I do know that the pinned memory is copied to the structure because the structure never has the value of 2.
Over many runs, a small percentage is runs results in something other than begin=1 and end=3. It would always be begin=1, end=1 and it happens about 5-10% of the time.
I have no idea why this happens. I know it generally highlights a race condition, but by calling the sync calls, I would expect the async calls to work in a predictable fashion.
Why would I be encountering this kind of issue with this code?
Thank you so much!
-Phil
I just figured out the issue that was occurring. While the launch was being done asynchronously... I didn't include the stream for the launch.
Changing my launch to be:
gpu.Launch(gridsize,blocksize,streamID).simulation(dF);
resolved the problem. It seems that the launches were occurring on stream 0 and the stream 1 and 2 were being synced. So sometimes the data gets set, sometimes it doesn't. A race condition.
I've the following problem that is begging a zmq solution. I have a time-series data:
A,B,C,D,E,...
I need to perform an operation, Func, on each point.
It makes good sense to parallelize the task using multiple workers via zmq. However, what is tripping me up is how do I synchronize the result, i.e., the results should be time-ordered exactly the way the input data came in. So the end result should look like:
Func(A), Func(B), Func(C), Func(D),...
I should also point out that time to complete,say, Func(A) will be slightly different than Func(B). This may require me to block for a while.
Any suggestions would be greatly appreciated.
You will always need to block for a while in order to synchronize things. You can actually send requests to a pool of workers, and when a response is received - to buffer it if it is not a subsequent one. One simple workflow could be described in a pseudo-language as follows:
socket receiver; # zmq.PULL
socket workers; # zmq.DEALER, the worker thread socket is started as zmq.DEALER too.
poller = poller(receiver, workers);
next_id_req = incr()
out_queue = queue;
out_queue.last_id = next_id_req
buffer = sorted_queue;
sock = poller.poll()
if sock is receiver:
packet_N = receiver.recv()
# send N for processing
worker.send(packet_N, ++next_id_req)
else if sock is workers:
# get a processed response Func(N)
func_N_response, id = workers.recv()
if out_queue.last_id != id-1:
# not subsequent id, buffer it
buffer.push(id, func_N_rseponse)
else:
# in order, push to out queue
out_queue.push(id, func_N_response)
# also consume all buffered subsequent items
while (out_queue.last_id == buffer.min_id() - 1):
id, buffered_N_resp = buffer.pop()
out_queue.push(id, buffered_N_resp)
But here comes the problem what happens if a packet is lost in the processing thread(the workers pool).. You can either skip it after a certain timeout(flush the buffer into the out queue), amd continue filling the out queue, and reorder when the packet comes later, if ever comes.