I have a use case wherein I need to fetch data from GCP BigQuery database into my Springboot application and subsequently perform some operations on it. I'm unable to understand how to go about doing it. For example, how the application properties need to be configured for using BQ database, etc, nor was I able to find any good resource for the same.
Request you all to kindly guide me a bit on this. Would be great even if you could point me to a relevant resource!
Indeed there are no examples on Spring Cloud documentation. However there is nice sample on spring-cloud-gcp github.
There is small tutorial how to run it, so I think this will be good starting point.
Related
I´m developing a Jokenpo Game using React with Spring Rest, but I can´t have a database to store all the information needed(create and delete moves, create and delete players).
I don´t know the best practice of development, or if there is some design pattern on how to store that kind of information. I know there is the folder src/main/resources where maybe I can store a text file there and thought about on the startup of the api it loads that file with the begin of the game, maybe, and after changing it during the game.
Trying to be more clear: I just would like to know the simplest way of storing information without being a database inside of a Spring Rest application. I really appreciate any helps. Thanks.
Take a look at SQLite. It's a very light database library that you can include as a dependency of your Spring application, It doesn't require a separate database server to run, and the entire database is stored in a single file, that you can choose where to store in the connection string.
It offers the flexibility of a standard database, so you can use Spring Data / JPA to access the data. It has some limitations compared with robust databases like MySQL, specially related with concurrent writes that you should investigate and be aware of. Usually it works very well for small applications or embedded applications.
I don't have any experience on any ETL tool. However I want to know if it is possible to do the followings using any ETL tool or we need to write a java or any other batch job to do this:
Scenario 1:
The source system has different REST APIs. I need to get the data, transform it, then store the data in a MongoDB.
The hardest part is the transformation. There can be situation where I need to call a REST API of source, and based on its data I need to call several other REST APIs using the 1st API data. After that we need to format the entire data in different format and store it in Mongo.
Scenario 2:
The source system has a DB. I need to transform the data using my custom logic and store it in MongoDB.
Here the custom logic can include things like this:
From table1 of source I created collection1. After that I need to consult table2 and previously created collection1, process the data and then create collection2.
Is this possible using any ETL tool? If possible then which tool? If possible please mention in as short as possible, how it can be done using different terminology so that I can search internet, learn things and implement it.
Briefly speaking: yes, that is what ETL tools are exactly for. You can Extract data from REST sources, Transform using sophisticated logic and Load to target, like MongoDB.
Exact implementation depends on the tool. While I guess you will get help if you run across problems implementing the solution in any of the tools, I don't think anyone will prepare complete, detailed solutions for you.
I have googled so much how to save data persistent (meaning it is still there after I shutdown / restart my apllication).
But every guide there is is "Hey we show you how to use spring-boot with jpa real quick and we use h2 database (in-memory)"
I am looking for a guide who shows the setup to make it possible to use a database that stores the data somewhere that it can be retrieved at any later point in time, even after the application was shut down.
If any of you can provide a link to something like this, I am very grateful!
I think Google can give a lot of examples about that. I am not sure what was your query.
Here are some examples:
https://www.mkyong.com/spring-boot/spring-boot-spring-data-jpa-mysql-example/
https://www.callicoder.com/spring-boot-rest-api-tutorial-with-mysql-jpa-hibernate/
https://spring.io/guides/gs/accessing-data-mysql/
HTH
I have seen the sample projects on your website for Dexie.Syncable such as sync-server and sync-client and they all seem to write to a datbase directly vs interacting with a web api. I am looking for a little help in where to get started beyond the examples on the website. The api I am trying to write a gateway for is dreamfactory
Also it looks like version 2 beta has had many improvements to Dexie.Syncable
I would recommend to build a new server-project based on either WebSocketSyncServer.js or the github repo of sync-server. However, I cannot give the details on how to call REST APIs instead of working directly towards database or memory. I would suggest using ES2016 async/await since your API calls are asynchronic.
Maybe you could try getting more help on https://github.com/nponiros/sync_server by filing an issue there.
We are starting to use Amazon web services. As I was searching for a Spring library, but couldn't find anything. Is this a case of there has been a decision made to not include this functionality or has no one provided any code. Is there any interested in having a spring-data-dynamodb?
there are plenty of examples out of there. Take a look at this one: It should be very interesting: https://github.com/michaellavelle/spring-data-dynamodb
[UPDATE]
That project is out to date right now, you should use this one: https://github.com/derjust/spring-data-dynamodb