I have a Chainlink node and I'm hitting this error:
number of unconfirmed transactions exceeds ETH_MAX_QUEUED_TRANSACTIONS. WARNING: Hitting ETH_MAX_QUEUED_TRANSACTIONS is a sanity limit and should never happen under normal operation. This error is very unlikely to be a problem with Chainlink, and instead more likely to be caused by a problem with your eth node's connectivity. Check your eth node: it may not be broadcasting transactions to the network, or it might be overloaded and evicting Chainlink's transactions from its mempool. Increasing ETH_MAX_QUEUED_TRANSACTIONS is almost certainly not the correct action to take here unless you ABSOLUTELY know what you are doing, and will probably make things worse: cannot create transaction; too many unstarted transactions in the queue (250/250). WARNING: Hitting ETH_MAX_QUEUED_TRANSACTIONS is a sanity limit and should never happen under normal operation. This error is very unlikely to be a problem with Chainlink, and instead more likely to be caused by a problem with your eth node's connectivity. Check your eth node: it may not be broadcasting transactions to the network, or it might be overloaded and evicting Chainlink's transactions from its mempool. Increasing ETH_MAX_QUEUED_TRANSACTIONS is almost certainly not the correct action to take here unless you ABSOLUTELY know what you are doing, and will probably make things worse
What is the best action item to take to solve this?
This means, that there are more than ETH_MAX_QUEUED_TRANSACTIONS in the eth_txes table in the psql database. You can do a few things, but it's important to know what they all do.
Delete the unstarted txes
Log into your database:
psql <DATABASE_URL_STRINGusing a psql client
Run the following:
DELETE FROM eth_txes WHERE state = 'unstarted';
Or find all the stuck transactions with:
select * from eth_txes;
Note, this will drop all transactions in the memepool of your node.
Bump up ETH_MAX_QUEUED_TRANSACTIONS
Edit your .env with:
ETH_MAX_QUEUED_TRANSACTIONS=<SOME_NUMBER_MORE_THAN_250>
Restart your node
This won't drop any transactions, but it'll just bump up how many can be "in flight" at once. Usually this isn't the right move.
Related
For the last few weeks or so, we are having the following issue:
Some of our transactions, when sent via sendRawTransaction() never get picked up by the network (if we look up the txid in the explorer, it's never there), and yet web3js doesn't error out.
We use "#solana/web3.js": "^1.44.1"
This has started happening to us like 2-3 weeks ago:
This issue affects some sets of transactions that all share the same instructions + amount of signers and accounts.
It repros 100% of the time for all transactions in those sets. No matter the state of the network or how many times we retry, they never get picked up.
We don't get any error back from web3.js, such as transaction limit hit
They all work in devnet, but not in mainnet!
For one of these tx, I removed one instruction+signer and it started working, so I imagine there's some limit we're hitting, but I can't tell which or how to even determine the limit.
When network congestion is high, validators can drop transactions without any error. To fix your issue you could send more of the same transaction on some interval while you're waiting for confirmation, and while your transaction blockhash is valid. This way you'll raise a chances for your transaction to been processed by the validator.
I've written a basic rpc client which polls the state of an Solana account to look for a specific condition (i.e. a unique int64 Id being written to it). When the condition arises, I call a smart contract which takes the same account as a mutable argument.
Before doing anything, the program checks for the same condition. However this check fails. I understand we're dealing with a distributed system and that state maybe inconsistent for a period of time, but I can repeatedly call for over 30 secs and it fails each time, before ultimately succeeding.
I've read about the concept of commitment levels but always assumed the account state passed into the smart contract would be the latest state of the world (i.e. processed)? What I appear to be observing is it's more like the finalised state.
Can anyone shed some light on what might be going on here?
I will try and come up with a minimal code example to demonstrate the problem but just wanted to ask the question first, to see if anyone can point me in the right direction.
Thanks
So if you look at the docs you linked, processed has a note:
the block may still be skipped by the cluster
This is a very important note if you're only looking for account state changes and don't want some that may be false. There's a number of reasons that a slot can be skipped, or a transaction could be rejected by the cluster.
If any of the above happens, then the account state that is accepted by the cluster as a whole may not be reflected in processed, but finalized.
In the end my specific problem came down to pre-flight checks using the 'finalized' commitment level when my logic for polling the account was using 'confirmed'. Modifying the preflightCommitment argument on sendTransaction fixed the problem for me.
I am new to the topic. Having read a handful of articles on it, and asked a couple of persons, I still do not understand what you people do in regard to one problem.
There are UI clients making requests to several backend instances (for now it's irrelevant whether sessions are sticky or not), and those instances are connected to some highly available DB cluster (may it be Cassandra or something else of even Elasticsearch). Say the backend instance is not specifically tied to one or cluster's machines, and instead its every request to DB may be served by a different machine.
One client creates some record, it's synchronously of asynchronously stored to one of cluster's machines then eventually gets replicated to the rest of DB machines. Then another client requests the list or records, the request ends up served by a distant machine not yet received the replicated changes, and so the client does not see the record. Well, that's bad but not yet ugly.
Consider however that the second client hits the machine which has the record, displays it in a list, then refreshes the list and this time hits the distant machine and again does not see the record. That's very weird behavior to observe, isn't it? It might even get worse: the client successfully requests the record, starts some editing on it, then tries to store the updates to DB and this time hits the distant machine which says "I know nothing about this record you are trying to update". That's an error which the user will see while doing something completely legitimate.
So what's the common practice to guard against this?
So far, I only see three solutions.
1) Not actually a solution but rather a policy: ignore the problem and instead speed up the cluster hard enough to guarantee that 99.999% of changes will be replicated on the whole cluster in, say, 0.5 secord (it's hard to imagine some user will try to make several consecutive requests to one record in that time; he can of course issue several reading requests, but in that case he'll probably not notice inconsistency between results). And even if sometimes something goes wrong and the user faces the problem, well, we just embrace that. If the loser gets unhappy and writes a complaint to us (which will happen maybe once a week or once an hour), we just apologize and go on.
2) Introduce an affinity between user's session and a specific DB machine. This helps, but needs explicit support from the DB, and also hurts load-balancing, and invites complications when the DB machine goes down and the session needs to be re-bound to another machine (however with proper support from DB I think that's possible; say Elasticsearch can accept routing key, and I believe if the target shard goes down it will just switch the affinity link to another shard - though I am not entirely sure; but even if re-binding happens, the other machine may contain older data :) ).
3) Rely on monotonic consistency, i.e. some method to be sure that the next request from a client will get results no older than the previous one. But, as I understand it, this approach also requires explicit support from DB, like being able so pass some "global version timestamp" to a cluster's balancer, which it will compare with it's latest data on all machines' timestamps to determine which machines can serve the request.
Are there other good options? Or are those three considered good enough to use?
P.S. My specific problem right now is with Elasticsearch; AFAIK there is no support for monotonic reads there, though looks like option #2 may be available.
Apache Ignite has primary partition for a key and backup partitions. Unless you have readFromBackup option set, you will always be reading from primary partition whose contents is expected to be reliable.
If a node goes away, a transaction (or operation) should be either propagated by remaining nodes or rolled back.
Note that Apache Ignite doesn't do Eventual Consistency but instead Strong Consistency. It means that you can observe delays during node loss, but will not observe inconsistent data.
In Cassandra if using at least quorum consistency for both reads and writes you will get monotonic reads. This was not the case pre 1.0 but thats a long time ago. There are some gotchas if using server timestamps but thats not by default so likely wont be an issue if using C* 2.1+.
What can get funny is since C* uses timestamps is things that occur at "same time". Since Cassandra is Last Write Wins the times and clock drift do matter. But concurrent updates to records will always have race conditions so if you require strong read before write guarantees you can use light weight transactions (essentially CAS operations using paxos) to ensure no one else updates between your read to update, these are slow though so I would avoid it unless critical.
In a true distributed system, it does not matter where your record is stored in remote cluster as long as your clients are connected to that remote cluster. In Hazelcast, a record is always stored in a partition and one partition is owned by one of the servers in the cluster. There could be X number of partitions in the cluster (by default 271) and all those partitions are equally distributed across the cluster. So a 3 members cluster will have a partition distribution like 91-90-90.
Now when a client sends a record to store in Hazelcast cluster, it already knows which partition does the record belong to by using consistent hashing algorithm. And with that, it also knows which server is the owner of that partition. Hence, the client sends its operation directly to that server. This approach applies on all client operations - put or get. So in your case, you may have several UI clients connected to the cluster but your record for a particular user is stored on one server in the cluster and all your UI clients will be approaching that server for their operations related to that record.
As for consistency, Hazelcast by default is strongly consistent distributed cache, which implies that all your updates to a particular record happen synchronously, in the same thread and the application waits until it has received acknowledgement from the owner server (and the backup server if backups are enabled) in the cluster.
When you connect a DB layer (this could be one or many different types of DBs running in parallel) to the cluster then Hazelcast cluster returns data even if its not currently present in the cluster by reading it from DB. So you never get a null value. On updating, you configure the cluster to send the updates downstream synchronously or asynchronously.
Ah-ha, after some even more thorough study of ES discussions I found this: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-preference.html
Note how they specifically highlight the "custom value" case, recommending to use it exactly to solve my problem.
So, given that's their official recommendation, we can summarise it like this.
To fight volatile reads, we are supposed to use "preference",
with "custom" or some other approach.
To also get "read your
writes" consistency, we can have all clients use
"preference=_primary", because primary shard is first to get all
writes. This however will probably have worse performance than
"custom" mode due to no distribution. And that's quite similar to what other people here said about Ignite and Hazelcast.
Right?
Of course that's a solution specifically for ES. Reverting to my initial question which is a bit more generic, turns out that options #2 and #3 are really considered good enough for many distributed systems, with #3 being possible to achieve with #2 (even without immediate support for #3 by DB).
I have a Laravel (Lumen 5.2) project that run against a MariaDB Galera Cluster. When running the app it seems to work just fine. But when I run the PHPUnit tests, they randomly fails.
The problem is that I populate the database and then trying to get the data (ids) to populate other tables with a foreign key. But when trying to get the data immediately after, the data is null.
The Laravel database connection is used with a READ user and a WRITE user. (Laravel automatically uses the correct one when inserting or reading). And I think this is the problem somehow. When I only use the WRITE user, the tests works just fine.
SET SESSION wsrep_sync_wait = 1;
before the SELECT will guarantee that the writes have caught up.
Although Galera is "synchronous", it is not quite. The writes are guaranteed to be sent to all the other nodes, and that they will work there. However, in the case of a "critical" read, a SELECT can get to the recipient node too fast to see the write. The above setting solves that problem.
Now, let's get real about how you should implement, say, a web site with Galera under the covers.
When practical, such as in a transaction, do all the commands against the same node. There is no penalty for this. But, check for errors after the COMMIT.
When not practical -- such as starting a new web page from a new HTTP connection -- use that SET.
While there is, potentially, a slight delay in the SELECT to wait for replication, the delay is usually very close to zero. I would suggest that your tests are effectively stress tests will deceptively say that the wait is high. That is, a benchmark is usually designed to find the "worst", not the "typical".
How much delay is there between the write and the failing SELECT? Perhaps only 1ms. How fast can a 'user' post a 'blog', then get to the next page and find it "missing"? Perhaps over 100ms.
Your stress test has discovered the need for the SET, not that Galera is 'broken'.
More Galera Caveats.
I have a WCF service that uses ODP.NET to read data from an Oracle database. The service also writes to the database, but indirectly, as all updates and inserts are achieved through an older layer of business logic that I access via COM+, which I wrap in a TransactionScope. The older layer connects to Oracle via ODBC, not ODP.NET.
The problem I have is that because Oracle uses a two-phase-commit, and because the older business layer is using ODBC and not ODP.NET, the transaction sometimes returns on the TransactionScope.Commit() before the data is actually available for reads from the service layer.
I see a similar post about a Java user having trouble like this as well on Stack Overflow.
A representative from Oracle posted that there isn't much I can do about this problem:
This maybe due to the way OLETx
ITransaction::Commit() method behaves.
After phase 1 of the 2PC (i.e. the
prepare phase) if all is successful,
commit can return even if the resource
managers haven't actually committed.
After all the successful "prepare" is
a guarantee that the resource managers
cannot arbitrarily abort after this
point. Thus even though a resource
manager couldn't commit because it
didn't receive a "commit" notification
from the MSDTC (due to say a
communication failure), the
component's commit request returns
successfully. If you select rows from
the table(s) immediately you may
sometimes see the actual commit occur
in the database after you have already
executed your select. Your select will
not therefore see the new rows due to
consistent read semantics. There is
nothing we can do about this in Oracle
as the "commit success after
successful phase 1" optimization is
part of the MSDTC's implementation.
So, my question is this:
How should I go about dealing with the possible delay ("asyc" via the title) problem of figuring out when the second part of the 2PC actually occurs, so I can be sure that data I inserted (indirectly) is actually available to be selected after the Commit() call returns?
How do big systems deal with the fact that the data might not be ready for reading immediately?
I assume that the whole transaction has prepared and a commit outcome decided by the TransactionManager, therefore eventually (barring heuristic damage) the Resource Managers will receive their commit message and complete. However, there are no guarantees as to how long that might take - could be days, no timeouts apply, having voted "commit" in the Prepare the Resource Manager must wait to hear the collective outcome.
Under these conditions, the simplest approach is to take "an understood, we're thinking" approach. Your request has been understood, but you actually don't know the outcome, and that's what you tell the user. Yes, in all sane circumstances the request will complete, but under some conditions operators could actually choose to intervene in the transaction manually (and maybe cause heuristic damage in doing so.)
To go one step further, you could start a new transaction and perform some queries to see if the data is there. Now, if you are populating a result screen you will naturally be doing such as query. The question would be what to do if the expected results are not there. So again, tell the user "your recent request is being processed, hit refresh to see if it's complete". Or retry automatically (I don't much like auto retry - prefer to educate the user that it's effectively an asynch operation.)