I am running gradle flywayMigrate and getting this output that doesn't show any errors although it is not running my migration scripts:
Database: jdbc:mysql://localhost:3306 (MySQL 8.0)
Successfully validated 1 migration (execution time 00:00.006s)
Current version of schema `userdb`: null
Schema `userdb` is up to date. No migration necessary.
:flywayMigrate (Thread[Daemon worker Thread 3,5,main]) completed. Took 1.025 secs.
my configuration in gradle is as follows:
flyway{
url = 'jdbc:mysql://localhost:3306?&serverTimezone=UTC'
user = 'root'
password = 'password'
schemas = ['userdb']
locations = ['filesystem:src/main/resources/db/migration/']
}
and my scripts are in: F:......\src\main\resources\db\migration\v1__Create_user_table.sql
create table USERS (
ID int not null,
NAME varchar(100) not null
);
can't figure out why it is not carrying out the migration. It did however create the flyway history table.
I realized what the problem was. I had to capitalize the 'v' in my scriptname "v1__Create_user_table.sql". Amazing waste of time spent debugging.
Related
I am trying to self host Supabase with Postgersql (AWS RDS). I have got the db up and running.
I download their docker diretory. It's working fine with the default settings, but when I do it with my postgreSQL,
supabase-auth doesnt seem to work. It give me following error
supabase-auth | time="2022-07-15T14:09:51Z" level=fatal msg="running db migrations: Migrator: problem creating schema migrations: CREATE TABLE \"schema_migrations\" (\n\"version\" VARCHAR (14) NOT NULL\n);\nCREATE UNIQUE INDEX \"schema_migrations_version_idx\" ON \"schema_migrations\" (version);: ERROR: no schema has been selected to create in (SQLSTATE 3F000)"
supabase-auth | [POP] 2022/07/15 14:09:51 info - 2.6074 seconds
supabase-auth exited with code 1
Any help will be appreciated!
Thanks in advance.
Problem & Question
I am working on a docker-compose stack with Minio for object storage, a Hive Standalone Metastore, and Trino as the query engine. My repo is store here - https://github.com/rylativity/trino_hive-meta_minio and steps to reproduce the issue are included at the end of this post.
When I attempt to use the Hive Standalone Metastore, I get an error when attempting to create a table. I am able to create the schema with CREATE SCHEMA hive.taxi;, and I can see the taxi.db folder created in the /opt/warehouse folder in my Hive Metastore container when I run the CREATE SCHEMA command. However, when I run CREATE TABLE hive.taxi.trips (...<col_type_info>...) WITH (external_location = 's3a://test/trips, format = 'PARQUET'), I immediately see an NoSuchObjectException error thrown by the ThriftMetastoreClient in the Trino container's logs that says the hive.taxi.trips table cannot be found and then a timeout occurs 60s later:
trino_1 | 2022-06-07T15:09:41.496Z DEBUG dispatcher-query-13 io.trino.plugin.hive.metastore.thrift.ThriftHiveMetastoreClient Invocation of get_table_req(req=GetTableRequest(dbName:taxi, tblName:trips, capabilities:ClientCapabilities(values:[INSERT_ONLY_TABLES]))) took 29.53ms and failed with NoSuchObjectException(message:hive.taxi.trips table not found)
However, if I use a FileHiveMetastore (stored within the Trino container), I am able to create and query the table successfully, which leads me initailly to believe that the issue is with my Hive Standalone Metastore setup and not with the other services.
Does anyone know why I might be receiving a NoSuchObjectException - hive.taxi.trips table not found when attempting to create a table and after I am able to successfully create the hive.taxi schema?
Steps to Reproduce
Clone the repo - https://github.com/rylativity/trino_hive-meta_minio
Run docker-compose up -d to bring up the containers. The minio-init container will create a 'test' bucket and an access-key/secret to access the bucket.
Run ./dataload_scripts/load_taxidata_to_minio.sh. This will download parquet files containing yellow-cab trip data from the public NYC-TLC bucket and load it into the 'test' bucket in Minio.
Run docker-compose exec trino trino to open a Trino shell inside the Trino container.
Inside the Trino shell, run CREATE SCHEMA hive.taxi; and then run
CREATE TABLE IF NOT EXISTS filehive.taxi.trips(
VendorID BIGINT,
tpep_pickup_datetime TIMESTAMP,
tpep_dropoff_datetime TIMESTAMP,
passenger_count DOUBLE,
trip_distance DOUBLE,
PULocationID BIGINT,
DOLocationID BIGINT,
RatecodeID DOUBLE,
store_and_fwd_flag VARCHAR,
dropoff_longitude DOUBLE,
dropoff_latitude DOUBLE,
payment_type BIGINT,
fare_amount DOUBLE,
improvement_surcharge DOUBLE,
congestion_surcharge DOUBLE,
mta_tax DOUBLE,
tip_amount DOUBLE,
tolls_amount DOUBLE,
extra DOUBLE,
airport_fee DOUBLE,
total_amount DOUBLE
)
WITH (
external_location = 's3a://test/taxi',
format = 'PARQUET'
);
If you run docker-compose exec metastore ls /opt/warehouse/, you will see that the taxi.db folder gets created after you run the CREATE SCHEMA... command. If you look at the Trino logs after running the CREATE TABLE... command, you will see the "NoSuchObjectException - hive.taxi.trips does not exist" error.
If you replace "hive" with "filehive" in the CREATE SCHEMA... and CREATE TABLE... commands, you will see that there is no issue when using a Trino FileHiveMetastore.
At any point, to start over, you can run docker-compose down -v and then return to step 1
I'm using SpringBoot 2.4.x app with SpringBatch 4.3.x. I've created a simple job.
Where I've FlatFileItemReader which reads from CSV file. I've ImportKafkaItemWriter which writes to Kafka topic. One step where I combines these. I'm using SimpleJobLauncher and I've set ThreadPoolTaskExecutor as TasKExecutor of the JobLauncher. It is working fine as I've expected. But one resilience use case I've which is if I kill the app and then restart the app and trigger the job then it would carry on and finish the remaining job. Unfortunately it is not happening. I did further investigate and found that when I forcibly close the app SpringBatch job repository key tables look like this:
job_execution_id
version
job_instance_id
create_time
start_time
end_time
status
exit_code
exit_message
last_updated
job_configuration_location
1
1
1
2021-06-16 09:32:43
2021-06-16 09:32:43
STARTED
UNKNOWN
2021-06-16 09:32:43
and
step_execution_id
version
step_name
job_execution_id
start_time
end_time
status
commit_count
read_count
filter_count
write_count
read_skip_count
write_skip_count
process_skip_count
rollback_count
exit_code
exit_message
last_updated
1
4
productImportStep
1
2021-06-16 09:32:43
STARTED
3
6
0
6
0
0
0
0
EXECUTING
2021-06-16 09:32:50
If I manually update these tables where I set a valid end_time and status to FAILED then I can restart the job and works absolutely fine. May I know what I need to do so that Spring Batch can update those relevant repositories appropriately and I can avoid this manual steps. I can provide more information about code if needed.
If I manually update these tables where I set a valid end_time and status to FAILED then I can restart the job and works absolutely fine. May I know what I need to do so that Spring Batch can update those relevant repositories appropriately and I can avoid this manual steps
When a job is killed abruptly, Spring Batch won't have a chance to update its status in the Job repository, so the status is stuck at STARTED. Now when the job is restarted, the only information that Spring Batch has is the status in the job repository. By just looking at the status in the database, Spring Batch cannot distinguish between a job that is effectively running and a job that has been killed abruptly (in both cases, the status is STARTED).
The way to go in indeed manually updating the tables to either mark the status as FAILED to be able to restart the job or ABANDONED to abandon it. This is a business decision that you have to make and there is no way to automate it on the framework side. For more details, please refer to the reference documentation here: Aborting a Job.
You can add a faked parameter example Version a counter to increment for every new job execution so you don't have to check for the table database job.
What I mean mvn clean package
Then you try to launch the program like this :
java my-jarfile.jar dest=/tmp/foo Version="0"
java my-jarfile.jar dest=/tmp/foo Version="1"
java my-jarfile.jar dest=/tmp/foo Version="2"
etc ... Or
You Can use jobParameters to launch thé job programatically via jobLauncher and use date paramèter date = new Date().toString() which gives date with New stamp on every New job execution
You can use "JVM Shutdown Hook":
Something like this:
Runtime.getRuntime().addShutdownHook(new Thread(() -> {
if (jobExecution.isRunning()) {
jobExecution.setEndTime(new Date());
jobExecution.setStatus(BatchStatus.FAILED);
jobExecution.setExitStatus(ExitStatus.FAILED);
jobRepository.update(jobExecution);
}
}));
I am facing an error while creating an External Table to push the data from Hive to ElasticSearch.
What I have done so far:
1) Successfully set up ElasticSearch-1.4.4 and is running.
2) Successfully set up Hadoop1.2.1, all the daemons are up and running.
3) Successfully set up Hive-0.10.0.
4) Configured elasticsearch-hadoop-1.2.0.jar in both Hadoop/lib and Hive/lib as well.
5) Successfully created few internal tables in Hive.
Error coming when executing following command:
CREATE EXTERNAL TABLE drivers_external (
id BIGINT,
firstname STRING,
lastname STRING,
vehicle STRING,
speed STRING)
STORED BY 'org.elasticsearch.hadoop.hive.EsStorageHandler'
TBLPROPERTIES('es.nodes'='localhost','es.resource' = 'drivers/driver');
Error is:
Failed with exception org.apache.hadoop.hive.ql.metadata.HiveException: Error in loading storage handler.org.elasticsearch.hadoop.hive.EsStorageHandler
FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask
Any Help!
Finally found the resolution for it...
1) The "elasticsearch-hadoop-1.2.0.jar" jar I was using was bugged one. It didn't have any hadoop/hive packages inside it. (Found this jar on internet and just downloaded it).
Now replaced it by jar from Maven repository "elasticsearch-hadoop-1.3.0.M1.jar".
2) The class "org.elasticsearch.hadoop.hive.**EsStorageHandler**" has been renamed in new elasticsearch jar as "org.elasticsearch.hadoop.hive.**ESStorageHandler**". Note that capital 'S' in 'ES'.
So the new hive command to create External table is :
CREATE EXTERNAL TABLE drivers_external (
id BIGINT,
firstname STRING,
lastname STRING,
vehicle STRING,
speed STRING)
STORED BY 'org.elasticsearch.hadoop.hive.ESStorageHandler'
TBLPROPERTIES('es.nodes'='localhost','es.resource' = 'drivers/driver');
It Worked!
I am getting following error while running a workflow in informatica.
Session task instance [worklet.session] : [TM_6775 The master DTM process was unable to connect to the master service process to update the session status with the following message: error message [ERROR: The session run for [Session task instance [worklet.session]] and [ folder id = 206, workflow id = 16042, workflow run id = 65095209, worklet run id = 65095337, task instance id = 13272 ] is not yet registered with this service process.] and error code [1417].]
This error comes randomly for many other sessions, when they are ran through workflow as a whole. However if I "start task" that failed task next time, it runs successfully.
Any help is much appreciated.
Just an idea to try if you use versioning. Check that everthing is checked in correctly. If the mapping, worflow or worklet is checked out then you and informatica will run different versions wich may cause the behaivour to differ when you start it manually.
Infromatica will allways use the checked in version and you will allways use the checked out version.