I just upgraded my Postgres DB on Heroku and used pg:copy to copy everything. However in my original DB it shows there was ~32,000 rows and in the new one there are ~28,000 rows. Does this mean some rows weren't copied or is this common ?
The variance in row counts can be attributed to stale data, table bloat, or both. pg:copy uses pg_dump and pg_restore under the hood. If pg:copy completed without errors you can rest assured that data was not lost in transit.
Related
I am having issues with getting the correct number of rows. I ran this code a while back and according to the data, this is the number of rows I should be getting.
Original
But i just ran it again after probably a few weeks and i am sure the data is not messed with because the tables have the same data as before
Ran Again
I have an issue in production env, one of the work flow is running more tgan one day and inserting records in to sql server db. It s just direct load mapping, there is no sq over ride as well. Monitor shows sq count as 7 million and inseting same no of records inyo target. But source db shows around 3 million records only. How can this be possible?
Have you checked if the source qualifier is joining more than one table? A screenshot of the affected mapping pipeline and obfuscated logfile would help.
Another thought... given your job ran for a day, were there any jobs ran in that time to purge old records from the source table?
Cases when I saw this kind of things happening:
There's a SQL Query override doing something different than I thought (eg. joining some tables)
I'm looking at a different source - verify the connections and make sure to check the same object on the same database at the same server the PowerCenter is connecting to.
It's a reusable session being executed multiple times by different workflows. In such case in workflow monitor it may happen that Source/Target Statistics will refer to another execution.
I am trying to update from 4.0 to 4.5.1 but the process always fails at UpdateMeasuresDebtToMinutes. I am using MySQL 5.5.27 as a database with InnoDB as table engine.
Basically the problem looks like this problem
After the writeTimeout exceeds (600 seconds) there is an exception in the log
Caused by: java.io.EOFException: Can not read response from server. Expected to read 81 bytes, read 15 bytes before connection was unexpectedly lost.
at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3166) ~[mysql-connector-java-5.1.27.jar:na]
at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3676) ~[mysql-connector-java-5.1.27.jar:na]
Adding the indexes as proposed in the linked issue did not help.
Investigating further I noticed several things:
the migration step reads data from a table and wants to write back to the same table (project_measures)
project_measures contains more than 770000 rows
the process always hangs after 249 rows
the hanging happens in org.sonar.server.migrations.MassUpdate when calling update.addBatch() which after the BatchSession.MAX_BATCH_SIZE (250) forces an execute and a commit
is there a way to configure the DB connection to allow this to proceed?
First of all, could you try to revert your db to 4.0 and try again ?
Then, could you please give us the JDBC url (sonar.jdbc.url) you're using ?
Thanks
As I need that sonar server to run I finally implemented a workaround.
It seems I cannot write to the database at all, as long as a big result set is still open (I tried with a second table but the same issue as before).
Therefore I changed all migrations that need to read and write the project_measurestable (org.sonar.server.db.migrations.v43.TechnicalDebtMeasuresMigration, org.sonar.server.db.migrations.v43.RequirementMeasuresMigration, org.sonar.server.db.migrations.v44.MeasureDataMigration) to load the changed data into a memory structure and after closing the read resultset write it back.
This is as hacky as it sounds and will not work for larger datasets where you would need to this with paging through the data or storing everything into a secondary datastore.
Furthermore I found that later on (in 546_inverse_rule_key_index.rb) an index needs to be created on the rules table which is larger than the max key length on mysql (2 varchar(255) columns with UTF-8 is more than 1000bytes .. ) so I had to limit the key length on that too ..
As I said, it is a workaround and therefore I will not accept it as an answer ..
I have two Rails apps running on Heroku, each has its own PostgreSQL 9.1.5 database (with Amazon endpoints accessible by me)
Both apps are running the same codebase, so they initialise the two database using the same set of schema. But App 1 only uses say Table A,B and C while App 2 only uses Table D and E.
(e.g. App 1's database's table D and E are empty)
Now, I need to move/copy all the data (table D and E ) from App 2's database to App 1's database (and then reconfig App 2 to use App 1's database from now on).
If I just take a pg_dump on App 2's database and restore it on App 1's database, it will erase the existing Table A,B and C rows, I believe. Or is there any flag/option that I can set so that it will preserve the existing data? Or what other methods I should look into?
Thanks!
P.S.
This post suggested pg_dump or database link, but after reading the pages, I am still not confident that I know how to use them so that my existing data won't be erased/overwritten
how to copy data from one database to another database in postgresql?
I would fork your database. https://devcenter.heroku.com/articles/heroku-postgres-fork
Create a backup of your db
Fork the db
Hook up App2 to your 2nd databse
Verify app1 and app2 are working find (and not writing to each others tables)
delete columns ABC from the new (app2) db
delete columns DE from the original (app1) db
I've been using heroku for one of my applications and it got shutdown today because the row count has exceeded 10,000 rows.
I don't understanding how this figure is arrived at though, as rails tells me I only have around 2000 records in the db.
Running a pg:info, I see the following:
Plan: Dev
Status: available
Connections: 1
PG Version: 9.1.5
Created: 2012-09-25 03:12 UTC
Data Size: 11.0 MB
Tables: 9
Rows: 15686/10000 (Write access revoked)
Fork/Follow: Unavailable
Can anyone explain to me how I seem to have 15,000 rows despite only have 2,000 records in the database?
Thanks!
Rails alone is not enough. Heroku has a nice SQL console that you can access with:
heroku pg:psql YOUR_DB_URL
then you can write this query to obtain a rank of records per table:
SELECT schemaname,relname,n_live_tup
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;
If you need only the updated num. of rows, you can use
SELECT sum(n_live_tup) FROM pg_stat_user_tables;
Please note that you can have both the new dev plan db and the old SHARED one in your config (access it by heroku pg:info). You have to insert the correct db url, probably the one with a color.
Allow a 30 mins delay between any sql truncate and the Rows count to update.
BTW the web console on http://heroku.com in my case was updated with the correct num. during my sql queries. May be heroku toolbelt console updates, are slower.
I contacted Heroku Support on this and they ran the following command to get my numbers...
$ heroku pg:info
30 mins weren't enough for me, so I took a backup and restored the database. Then my app came back online.