I've just upgraded to HortonWorks HBase client 1.1.2.2.4.2.0-258 from version 0.98. All works fine locally, but under production load after a couple of minutes I'm starting to have TableNotFoundException:
Caused by: org.apache.hadoop.hbase.TableNotFoundException
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegionInMeta(ConnectionManager.java:1264)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.locateRegion(ConnectionManager.java:1162)
at org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:300)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:152)
at org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:60)
at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:200)
at org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:320)
at org.apache.hadoop.hbase.client.ClientScanner.nextScanner(ClientScanner.java:295)
at org.apache.hadoop.hbase.client.ClientScanner.initializeScannerInConstruction(ClientScanner.java:160)
at org.apache.hadoop.hbase.client.ClientScanner.<init>(ClientScanner.java:155)
at org.apache.hadoop.hbase.client.HTable.getScanner(HTable.java:821)
The behaviour seems random and unpredictable, e.g. repeating the same request just works (i.e. no exception is thrown and data are successfully retrieved)
I was trying to understand what was changed from 0.98 to 1.1 and the only significant thing I found was https://issues.apache.org/jira/browse/HBASE-16973
Playing around with those values didn't help.
Any thing else I should take into account? Any pointers are highly appreciated!
Thanks!
The problem was in the way how we used HConnection object, there were several instances of them (which, I agree, is not the right way to use it, IIUC). In the new version of the hbase client, there are also several "hconnection" threads for meta lookup per HConnection. All of this accumulated leading to the extremely high number of threads that JVM were not able to handle. After using a single HConnection instance, the problem went away and everything works as expected.
Related
I'm trying to implement MonetDB in three machines, one master and two replicas in lazy logical replication.
For now I'm trying to implement in only machine with the following commands I took from this old issue in only one machine for now.
Everything goes according to plan until the first problem I have: When trying to create tables or inserting stuff I get the following errors I was not able to find on google:
Error in optimizer wlc: TypeException:user.main[17]:'wlc.predicate' undefined in: X_0:any := wlc.predicate("alpha":str, "id":str);
Error in optimizer wlc: TypeException:user.main[50]:'wlc.predicate' undefined in: X_0:any := wlc.predicate("beta":str, "id":str);
Error in optimizer wlc: TypeException:user.main[77]:'wlc.depend' undefined in: X_0:any := wlc.depend("beta":str, X_1:lng);
I got around this by setting optpipe to minimal_pipe but I wanted to know why this is happening so I don't have to do this.
The second problem I have when trying CALL wlr.replicate:
Perhaps a missing wlr.master() call.
How do I correctly set-up replication?
Thanks in advance.
The wlc/wlr features are experimental and de facto deprecated in current releases of MonetDB and completely removed starting from the next major release. Replication in MonetDB is a topic currently under revision. You might be better off formulating a feature request on MonetDB's githup page.
You might also consider looking into the concepts of replicate and remote tables. But those are definitely not solutions by themselves and if used as such, implement replication on the SQL layer instead of the infrastructural layer.
But on the short run, I do not expect that the open source community can help you out here much. Consider commercial support otherwise if feasible.
Sorry if this question has been asked elsewhere, I can't find it. I'm working through some basic examples in MonetDBLite.
> dbGetQuery(dbcon, "SELECT MAX(mpg) FROM mtcars WHERE cyl = 8")
L3
1 19.2
works, but
> ms <- MonetDBLite::src_monetdblite("./DB")
> t <- tbl(ms, "mtcars")
Error in UseMethod("tbl") :
no applicable method for 'tbl' applied to an object of class
"c('src_monetdb', 'src_sql', 'src')"
It seems that it's trying to assign the db to t not the table.
Any suggestions would be greatly appreciated.
I've been perusing resources and found a useR2016 presentation and noticed a difference here:
> ms
src: MonetDBEmbeddedConnection
tbls: mtcars
Curious...
I'm a huge fan of using MonetDBLite together with dplyr. My addition to Hannes Mühleisen's (thanks for the package!) answer would be that it appears that the order you load the packages can matter. Loading MonetDBLite after dplyr and dbplyr seems to be the key for me. Loading MonetDBLite first causes errors similar to the one nzgwynn noted.
Sometimes I could connect to the database with no problems. Other times I would get error messages like:
Error in UseMethod("db_query_fields") : no applicable method for 'db_query_fields' applied to an object of class "MonetDBEmbeddedConnection"
Like nzgwynn, I was puzzled about why it would work sometimes but not others. Restarting and reinstalling wouldn't necessarily fix it for me.
This clue, from an issue filed about sparklyr, lead me to explore the package loading order:
https://github.com/rstudio/sparklyr/issues/38
Like noted there with sparklyr, and I've noticed with other R database packages, MonetDBLite will load and attach automatically if the Global Environment already contains a connection object. My problem was that I had an src_monetdb object in my workspace, which was causing MonetDBLite to load upon starting RStudio. So I while I thought I was loading it after dplyr and dbplyr, it was really loading first. If I clear the workspace and then restart, I can load the packages in the preferred order. So far, this method has worked.
I've seen starting with a clean workspace advised as good practice generally, e.g.: https://twitter.com/hadleywickham/status/561146907519500288. Starting with a fresh workspace loses you no time either given MonetDBLite's speedy query ability.
Lastly, I would put a enthusiastic pitch in for using MonetDBLite. I saw it mentioned on RStudio's database page and was immediately impressed on how easy it was to setup and how fast it is. It's the best way I've found for working with a ~2 GB dataset in R. When exploring the data interactively, the dplyr queries run so quickly that it feels like I'm working with the data in memory. And if all I want to do is load the whole dataset into memory, MonetDBLite is as fast or faster than other methods I've tried like read.fst() from the fst package.
I closed R and opened it again and the same coding worked fine...
You need to call library("dplyr") before using tbl and friends. Also make sure you have dbplyr installed.
Update: Also, please make sure there is no connection object (src) in a stored workspace loaded at startup. Loading connections from .Rdata files does not work! Instead, create the connection/src from scratch every time you run a script.
When trying to run a fairly basic query using the driver provided by Simba, I'm running into issues where the "nanosecond" value is negative, causing IllegalArgumentException.
When writing a simple query that returns a Timestamp value, what comes back is an epoch value that is initially stored in a Double. Going through and debugging for example, I can see that the value coming back from the query is "1.498571169792E9". This corresponds to a timestamp of "Tuesday, June 27, 2017 1:46:09.792 PM" according to epochconverter.com, which is exactly what it should be.
Continuing to step through the code, we eventually try to use BQCoreUtils.convertUnixToTimestamp(). Now, while I've tried to disassemble the class (thanks IntelliJ!), I can't quite figure out what's going on. It eventually tries to create a new TimestampTz() which is an extension of java.sql.Timestamp, but the value getting passed for nanos is negative. This of course prompts Java to throw an IllegalArgumentException, but I can't figure out what I need to do to avoid this.
I also don't know if there's a simpler explanation for what's going on. Ultimately though, it appears that there's a driver bug. BQCoreUtils.convertUnixToTimestamp doesn't properly safeguard against the nanos calculation being non-negative.
The dumb question then is: has anyone else experience issues querying BigQuery where Timestamp values are triggering exceptions?
Update: Since this is happening in Timestamp created by JDBC driver, it does appear to be a bug in JDBC driver itself. Please file it under https://issuetracker.google.com/issues?q=componentid:187149.
Original:
The timestamp resolution in BigQuery is microseconds, and it looks like the value you are providing is in seconds, so you should multiply it by 1000000.
With reference to Google issue tracker,
This should be resolved with versions newer than 1.1.1 of the drivers, which also addressed other timestamp-related issues.
If any issue persists, please report at Google issue tracker they will re-open to examine.
I've just completed the upgrade from v2.23 to v3.00 and run a successful LIVE transaction, passing 3.00 as the value for VPSProtocol.
However, I've been having issues when running SIMULATED transactions using this value as SagePay returns the following error:
The <b>VPSProtocol</b> field you supplied contains invalid data. It should contain the value <b>2.23</b>
I'm also still seeing the big red bar at the top of MySagePay with the following scary message:
You will have to cease trading on this version (v2.23) unless you upgrade by the end of this month.
So, a couple of questions:
Do I need to do anything in MySagePay to tell the system that I'm
now using v3.00, or will that red bar disappear eventually?
Is there a current issue with simulated transactions because the error is still telling me that I need to pass 2.23 as the value for VPSProtocol despite this version being dropped?
Finally, How do I know that these last transactions I've made while passing 3.00 as the value for VPSProtocol have actually been made under v3.00? There's just a lot of confusing things going on here and no clear way to know exactly what's happening (ie, should I be seeing a completely new version of MySagePay somewhere? Or should I be able to distinguish which transactions were made under v2.23 and which were made under v3.00? Etc, etc.)
Thanks
Few answers for you:
This will disappear eventually if you are processing transactions at 3.00. It seems to behave fairly randomly.
You might as well forget about ever using the simulator. It hasn't been updated for protocol 3.00 (and a long time before that).
You will be using the current version of MSP, so no worry there. MSP won't actually tell you what protocol a transaction was carried out using, so if you want to be sure, you could make a 'getTransactionDetail' API call. Alternatively, call Sage Pay, give them a TransactionId and get them to confirm the protocol the transaction was processed using.
I have a Rails 3 app running an older version of Spree (an open source shopping cart). I am in the process of updating it to the latest version. This requires me to run numerous migrations on the database to be compatible with the latest version. However the apps current database is roughly around 300mb and to run the migrations on my local machine (mac os x 10.7, 4gb ram, 2.4GHz Core 2 Duo) takes over three days to complete.
I was able to decrease this time to only 16 hours using an Amazon EC2 instance (High-I/O On-Demand Instances, Quadruple Extra Large). But 16 hours is still too long as I will have to take down the site to perform this update.
Does anyone have any other suggestions to lower this time? Or any tips to increase the performance of the migrations?
FYI: using Ruby 1.9.2, and Ubuntu on the Amazon instance.
Dropping indices beforehand and adding them again afterwards is a good idea.
Also replacing .where(...).each with .find_each and perhaps adding transactions could help, as already mentioned.
Replace .save! with .save(:validate => false), because during the migrations you are not getting random inputs from users, you should be making known-good updates, and validations account for much of the execution time. Or using .update_attribute would also skip validations where you're only updating one field.
Where possible, use fewer AR objects in a loop. Instantiating and later garbage collecting them takes CPU time and uses more memory.
Maybe you have already considered this:
Tell the db not to bother making sure everything is on disk (no WAL, no fsync, etc), you now have an in memory db which should make a very big difference. (Since you have taken the db offline you can just restore from a backup in the unlikely event of power loss or similar). Turn fsync/WAL on when you are done.
It is likely that you can do some of the migrations before you take the db offline. Test this in staging env of course. That big user migration might very well be possible to do live. Make sure that you don't do it in a transaction, you might need to modify them a bit.
I'm not familiar with your exact situation but I'm sure there are even more things you can do unless this isn't enough.
This answer is more about approach than a specific technical solution. If your main criteria is minimum downtime (and data-integrity of course) then the best strategy for this is to not use rails!
Instead you can do all the heavy work up-front and leave just the critical "real time" data migration (i'm using "migration" in the non-rails sense here) as a step during the switchover.
So you have your current app with its db schema and the production data. You also (presumably) have a development version of the app based on the upgraded spree gems with the new db schema but no data. All you have to do is figure out a way of transforming the data between the two. This can be done in a number of ways, for example using pure SQL and temporary tables where necessary or using SQL and ruby to generate insert statements. These steps can be split up so that data that is fairly "static" (reference tables, products, etc) can be loaded into the db ahead of time and the data that changes more frequently (users, sesssions, orders, etc) can be done during the migration step.
You should be able to script this export-transform-import procedure so that it is repeatable and have tests/checks after it's complete to ensure data integrity. If you can arrange access to the new production database during the switchover then it should be easy to run the script against it. If you're restricted to a release process (eg webistrano) then you might have to shoe-horn it into a rails migration but you can run raw SQL using execute.
Take a look at this gem.
https://github.com/zdennis/activerecord-import/
data = []
data << Order.new(:order_info => 'test order')
Order.import data
Unfortunaltly the downrated solution is the only one. What is really slow in rails are the activerecord models. The are not suited for tasks like this.
If you want a fast migration you will have to do it in sql.
There is an other approach. But you will always have to rewrite most of the migrations...