Getting project root in Liquibase sql changeset - oracle

I have a requirement in my project where I want to read files into table CLOB data. The database is oracle 12g. We are using liquibase to maintain the data.
The project is in springboot. Liquibase uses .sql files to load changes.
The hurdle I am facing is that sql needs the absolute path to the file rather than the path relative to the current changeset file.
Any pointers about how I can use a "." or the "classpath:" in the sql of the changeset?
One of the attempts I have made are sending a property in the changeset through the property tag withen the databaseChangeLog tag. Values of such properties are available in the changeset file, but they dont interpret placeholders like "classpath:". Hence I am stuck.
The
dbms_lob.fileopen
only accepts a directory object with the absolute path. No "." etc is understood. The . in a filepath starts at the database installation root which is of no use to me. I would like the path to where all the changeset files are stored.
Please help If you have solved this in any way..
I will post more details if required..
Appreciate the help..

As example what work for me: i'va put this argument on command line as reference for the master changelog
--changeLogFile=src/main/resources/changelog/db.changelog-master.xml
and in the master file
<include file="./changes/db.changelog-ddl-.....xml" relativeToChangelogFile="true"/>
It will be the same for sql file in changeset - they will be referenced relative to the .xml

What about setting up a directory object in oracle in advance and then use symbolic linking on OS level. Depending on your exact requirements you can use symbolic linking on directory level, or you can even directly link the files into the oracle directory, so that Oracle can see files from many different directories at once in the directory object. They need to have different filenames then. I have not tried this idea, but I guess this approach is what I would try out first. HTH KR

Liquibase supports properties in changelogs that can be set in various ways - more information:
http://www.liquibase.org/documentation/changelog_parameters.html
Here is a snippet of that documentation:
Liquibase allows dynamic substitution of parameters in a changelog.
The parameters to replace are described using the ${} syntax.
Parameter values are looked up in the following order:
Passed as a parameter to your Liquibase runner (see Ant, command_line, etc. documentation for how to pass them)
As a JVM system property
In the parameters block ( Tag) of the DatabaseChangeLog file itself.
Example of how could be used:
<databaseChangeLog
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ext="http://www.liquibase.org/xml/ns/dbchangelog-ext"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-2.0.xsd
http://www.liquibase.org/xml/ns/dbchangelog-ext http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-ext.xsd">
<property name="clob.type" value="clob" dbms="oracle"/>
<property name="clob.type" value="longtext" dbms="mysql"/>
<changeSet id="1" author="joe">
<createTable tableName="table1">
<column name="id" type="int"/>
<column name="${columnname}" type="varchar(20)"/>
</createTable>
</changeSet>
easily you can set file path inside SQL change-set via a property while property value passed with JVM or Build-tools.
#echo off
call Liquibase --changeLogFile=myChangeLogFile.xml update -Dcolumnname=myclm
However I think you can use ${java.class.path} without set it's value from outside.

Related

Specifying files by absolute path was removed in Liquibase 4.0

I am getting the following liquibase error when I run my Spring Boot application:
Specifying files by absolute path was removed in Liquibase 4.0. Please use a relative path or add '/' to the classpath parameter.
Here is the class path in application.yaml:
liquibase:
change-log: classpath:db/changelog/db-changelog-master.xml
I also tried:
liquibase:
change-log: classpath:/db/changelog/db-changelog-master.xml
Here is folder structure:
Changlog master:
<?xml version="1.0" encoding="UTF-8"?>
<databaseChangeLog
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="http://www.liquibase.org/xml/ns/dbchangelog"
xsi:schemaLocation="http://www.liquibase.org/xml/ns/dbchangelog
http://www.liquibase.org/xml/ns/dbchangelog/dbchangelog-3.5.xsd">
<include file="db-changelog-1.0.xml"/>
</databaseChangeLog>
I got this issue when putting the changelog files outside the resources folder, but if I include them under resources/db/changelog, then it would work fine with setting the bellow config.
spring.liquibase.change-log=classpath:/db/changelog/changelog-master.xml
Tested under 4.6.2
This is still an open issue unfortunately. See 2281.
Looks like this was fixed in v4.4.3
As explained here
How the Liquibase classpath worked before version 4.0
Before version 4.0, one of the default locations Liquibase added to
the classpath was the root directory in your filesystem (/). The
change caused issues because of a machine-dependent changelog path,
such as /home/my-user/projects/liquibase/changelog.xml, found under
the / directory. This way, Liquibase uses the given path as part of
the changeset identifier stored in the DATABASECHANGELOG table, and
when you run Liquibase from
/home/other-user/projects/liquibase/changelog.xml, Liquibase sees it
as a different changelog and tries to rerun all the previously run
changesets.
To prevent identification issues from happening, a / was removed as a
default part of the classpath. How the Liquibase classpath works in
4.0 and later versions
Starting with Liquibase 4.0, the root directory (/) is no longer a
default part of the classpath because of the issue mentioned in the
previous section.
...
The message "Please use a relative path or add '/' to the classpath parameter." refers to the root directory '/', and does not mean to add a slash to the start of your classpath path. Afaik, classpath:x and classpath:/x are the same.
Also, the message appears when the master changelog is not found, for whatever reason, so also a typo can cause this message. It's only a hint telling you that it might not be found because the file is not on the classpath, because they removed the root directory from the classpath, but it could also not be found because you specified the wrong path (I just did that).
To configure it correctly, the master changelog must be on the Liquibase classpath. In Spring Boot, the Liquibase classpath is set to the application's classpath, i.e. you can use src/main/resources.
Tl;dr: When your file is src/main/resources/db/changelog/db.changelog-master.xml use
spring.liquibase.change-log=classpath:/db/changelog/db.changelog-master.xml
I don't know, whether there was or is a bug in regards to that with certain Liquibase versions, but that's how it's supposed to work, anyway.
Just try to remove "classpath:" from "change-log:" parameter.
Also try to check your pom.xml ("changeLogFile" tag in configuration):
there should not be a "${basedir}" before change log file path.
<changeLogFile>
/src/main/resources/liquibase/changelog.xml
</changeLogFile>

Liquibase databasechangelog table stores relative path if run spring boot and let liquibase do the migration automatically

I know that liquibase will create a databasechangelog to check if one change set has been applied. But if i let the app to run it automatically, the FILENAME column will store the relative path of the changelog file. But if I run mvn liquibase:status, it said my change set had not been applied, the cause is that in liquibase.properties file I defined the path of the changelog as absolute path (i.e changeLogFile=src/main/resources/db/changelog/db-changeLog-master.xml) while for spring boot app automatic migration the path was defined as relative path (spring.liquibase.change-log=classpath:db/changelog/db-changeLog-master.xml) in application.properties file and "db/changelog/db-changeLog-master.xml" was stored in the database. If you do it manually (by running mvn liquibase:update), "src/main/resources/db/changelog/db-changeLog-master.xml" would be stored instead.
How could I solve this problem? Since I wanna have control over the migration (i.e. do the migration manually) or check the status of migration in development environment.
after I changed the property: changeLogFile=/db/changelog/db-changeLog-master.xml, in liquibase.properties, mvn command works as expected: now liquibase consider the path of my changelog to be "db/changelog/db-changeLog-master.xml"

How to Configure Eclipse to Work with `ehcache`

Summary/Question
I've rolled onto a project that uses ehcache. The project is maven enabled and when I run mvn clean install from cli all ends well.
The project can also be opened in NetBeans and everything displays properly, however when I open the project in eclipse, I'm seeing some errors related to ehcache. In NetBeans the spring-cache.xml is located under a "Web Pages" folder. This made me think that the eclipse project may not be recognized as a dynamic web project, however following the directions here: https://www.mkyong.com/java/how-to-convert-java-project-to-web-project-in-eclipse/ I was able to verify that the project is indeed setup to support Dynamic Web Module, Java and JavaScript.
I also opened the project in IntelliJ and also did not see any issues with the spring-cache.xml.
I also didn't find documentation related to this version of ehcache on http://www.ehcache.org/documentation/ but the fact that things work in NetBeans, IntelliJ and via mvn clean install make me think its my eclipse setup. Thoughts?
More Details:
The errors I see in eclipse are:
Multiple annotations found at this line:
- cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'ehcache:annotation-driven'.
- schema_reference.4: Failed to read schema document 'http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring-1.1.xsd, because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
And
cvc-complex-type.2.4.c: The matching wildcard is strict, but no declaration can be found for element 'ehcache:config'.
My maven dependency looks like:
<dependency>
<groupId>com.googlecode.ehcache-spring-annotations</groupId>
<artifactId>ehcache-spring-annotations</artifactId>
<version>1.2.0</version>
</dependency>
Which matches what it should be, based on: https://mvnrepository.com/artifact/com.googlecode.ehcache-spring-annotations/ehcache-spring-annotations/1.2.0
My spring-cache.xml file looks like:
<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:ehcache="http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring"
xsi:schemaLocation="
http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd">
<ehcache:annotation-driven />
<ehcache:config cache-manager="cacheManager">
<ehcache:evict-expired-elements interval="60" />
</ehcache:config>
<bean id="cacheManager" class="org.springframework.cache.ehcache.EhCacheManagerFactoryBean">
<property name="configLocation" value="${ehcache.config.location}"/>
</bean>
</beans>
Looks like current versions of NetBeans and IntelliJ aren't as strict on checking things as eclipse is. :) The following is what I found out.
From ehcache with Spring. google code xsd file not found I found out that the top error was related to the fact that the xsd is no longer hosted at the url provided.
Doing some googles, I found a reference to the version on github at: https://raw.githubusercontent.com/agentgt/ehcache-spring-annotations/master/core/src/main/resources/com/googlecode/ehcache/annotations/ehcache-spring-1.2.xsd by changing my bottom xsi schemalocation to the above url, it resolved the issue in eclipse.
The stack overflow questions referenced, also provides a link to the google code archive of: https://code.google.com/archive/p/ehcache-spring-annotations/source/default/source
If you download the zip, you can then find the offical xsd in the zip at: /schema/ehcache-spring/ehcache-spring-1.2.xsd (it also has ehcache-spring-1.0.xsd and ehcache-spring1.1.xsd in case anyone else has issues with even earlier versions).
I found a way to reference the local schema file here: How to reference a local XML Schema file correctly?
I'm going to post a question on how to reference it from a relative path of the project, as we have both windows and mac developers on this project. If this is of use to you, the details will be here: How to Reference Local XSD File By Relative Path
Summary of issue:
xsi-schemaLocation reference:
http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd
results in validation-issues in eclipse, on commandline maven there are no issues.
replacing the ref with:
https://raw.githubusercontent.com/agentgt/ehcache-spring-annotations/master/core/src/main/resources/com/googlecode/ehcache/annotations/ehcache-spring-1.1.xsd
solves issue in eclipse. In maven the saxparser will complain however if you 're not connected to internet.
By putting a proxy ref :
-Dhttp.proxyHost=[proxyhost] -Dhttp.proxyPort=[proxyport] -Dhttps.proxyHost=[proxyhost] -Dhttps.proxyPort=[proxyport]
the maven build succeeds....
I investigated this difference in behaviour:
in the ehcache-spring jar, ehcache-spring-annotations-1.1.2.jar,
a META-INF/spring.schemas file is present.
Content of that file:
http\://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.0.xsd=com/googlecode/ehcache/annotations/ehcache-spring-1.0.xsd
http\://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd=com/googlecode/ehcache/annotations/ehcache-spring-1.1.xsd
cmdline maven uses this content (and finds the xsd in the jar).
eclipse does not.
Solution I used:
go to eclipse preferences,
XML -> XML Catalog and add a new user entry:
Location: https://raw.githubusercontent.com/agentgt/ehcache-spring-annotations/master/core/src/main/resources/com/googlecode/ehcache/annotations/ehcache-spring-1.1.xsd
Key type: Public id
Key: http://ehcache-spring-annotations.googlecode.com/svn/schema/ehcache-spring/ehcache-spring-1.1.xsd
Alternative web adress: https://raw.githubusercontent.com/agentgt/ehcache-spring-annotations/master/core/src/main/resources/com/googlecode/ehcache/annotations/ehcache-spring-1.1.xsd
Re-validating spring context files solved issues in my eclipse workspace.
I like this solution better because it is a shortcoming of eclipse which i solve with an eclipse-specific solution.

Maven install:install-file : specified file not exists

I tried to add custom jar into my project (to pom.xml). When I made an operation
mvn install:install-file -Dfile=<path-to-file> -DgroupId=<myGroup> -DartifactId=<myArtifactId> -Dversion=<myVersion> -Dpackaging=<myPackaging> -DlocalRepositoryPath=<path>
I actually received an error: the specified file not exists. But the file is in the folder - that's the problem. How I can resolve this? Or how I can add custom jar without this step? I know about scope , but it's not working properly - maven variable ${basedir} cannot be used in systemPath.
Maven 3.3.9
UPDATE 1: Actually, now I find smth new: Maven doesn't like dots in groupId and version. After some tries I installed file, but it's wrong path, because instead of (e.g) org.my.local.file tree I received file in org_my_local_file folder in maven repo.
I did this many times and it worked as expected.
What do you exactly mean with :
Maven doesn't like dots in groupId and version.
What exception do you get ?
You get org_my_local_file because I think you exachanged '.' in the groupId against '_' ?
You find a good tutorial here where you can see how it is done correctly :
How to include custom library into maven local repository?
EDIT :
It looks like you are using windows. Especially powershell is known to cause problems. Have a look here, may be you have to escape some characters, there are some possible solutions :
install maven artifact
I encountered this issue when trying to initialize our base development environment. We use an Ant script to initialize Maven, our developer server and other tools and includes installing a few 3rd party jars into the local Maven repository. This error, "The specified file '<path>\true' not exists" was occurring when invoking Maven from within Ant 1.9.11.
I also used this answer to specify paths identified in the Ant script and set them in the batch script template before running the batch script.
Specifically, this answer offered a lot to get the batch script constructed: https://stackoverflow.com/a/36298099/2336934. Didn't take much:
<!-- generate the batch file -->
<tempfile property="temp.file" suffix=".properties" deleteonexit="true" destdir="${env.TEMP}" />
<echoproperties destfile="${temp.file}" />
<replaceregexp file="${temp.file}" match="([^=]*)=" replace="#\1#=" byline="true" />
<copy file="${basedir}/scripts/install-jars.template" tofile="${basedir}/scripts/install-jars.bat" />
<replace file="${basedir}/scripts/install-jars.bat" replacefilterfile="${temp.file}" />
<echo message="Running batch file for Maven setup" />
<exec executable="${basedir}/scripts/install-jars.bat" />
<delete file="${basedir}/scripts/install-jars.bat" deleteonexit="true" />
There weren't any issues with the parameters being passed, but this must be specified correctly. Running the command you're trying to execute in a script temporarily in a command window is useful to verify the command syntax and function before trying to embed in a script.
I even created a empty true file to satisfy the error which complained the file was not a valid POM file. Filling it with basic POM content led to another error. So reader beware, this business about "true not exists" is a Red Herring.
Moving the Maven install:install-file commands out to the batch file was the key to get past the issue. It also allowed me to specify the quotes in the command as " rather than " due to the commands originally being specified in the Ant script (xml).

Liquibase duplicate identifiers using inclueAll tag

I'm new to liquibase and trying to setup a working solution for our software. At the moment i'm struggling with the 'includeAll' tag for the master changelog.
If i'm using includeAll it allways says:
Validation Error:
1 change sets had duplicate identifiers
install/data/changelog-data.xml::1::wild.a
Liquibase 'validate' Successful
The whole files to reproduce are included in the database.zip which can be downloaded here:
https://docs.google.com/file/d/0B8W-n_j7omQ4UUhzQjdrdnlMdk0/edit?usp=sharing
You only have to modify liquibase.properties.
The curious thing is that the following line work:
<include relativeToChangelogFile="true" file="install/data/changelog-data.xml"/>
While this line won't work:
<includeAll relativeToChangelogFile="true" path="install/data/"/>
The question is how to use includeAll to work without having duplicate identifiers.
Thanks in advance.
It is a bug in 3.0.0 that has been fixed in 3.0.2.

Resources