My question boils down to the title: are rugged Repository instances (intended to be) threadsafe?
While I'm at it, I may be able to settle a question I've been having longer: is access to a git repository using rugged (intended to be) threadsafe when using different Repository instances?
Context
I'm using Rugged to access a git repository that stores documents for multiple users that can access the repo via a shared web frontend. So far I created a new Repository instance for each access, as that performs well enough and seems to be safe (I can't find guarantees in the docs or determine obvious safety from the way libgit2 is used, but no tests found problems and I'm assuming libgit2 itself is safe).
However, I ran into an issue, which limits the amount of repository instances that you can open near-simultaneously, which causes problems for some scripts that reuse some of the code that create Repository instances for each git repository access. An easy solution would be to share Repository instances between all users. However, that will cause problems if repository instances are not thread safe. Do I need to guard all of these shared instances with a Mutex or can I do without, because rugged/libgit2 already solves this problem for me?
Yes, libgit2 (and thus rugged as well) should be threadsafe, as long as you don't use the same repository instance (or any other object created from libgit2) across different threads.
But as indicated by the second part of your question, you actually want to use the same repository instance across different threads. Here, the answe is it depends. Most, but not all, of the functions provided by libgit2 should be threadsafe, but I can't give you a definite list. See https://github.com/libgit2/libgit2/issues/2491 for a bit of more information.
Related
I wonder how people are handling the repository of the documentation of their modules (I am mostly doing python/sphinx/git).
Facing two choices:
include the doc sources in the module repository
has the advantage of having a all in one unique place for the project, but the inconvenience of bringing non-vital data with the code (for instance images for the doc)
make a separate repository for the documentation
has the advantage of having a separate life cycle between the doc and the module code. When correcting a doc typo for instance one would like to avoid a new repository version for the code. The drawback is to maintain two repositories with maybe equivalent branching.
Are there clear guidelines somewhere? Am I missing some important points?
I have question about microservices and repositories. We are a small team (5 people) and we creating new project in microservices. Expected microservice applications in our project is between 10-15.
We are thinking about one repository for all microservices in structure like that:
-/
--/app1
--/app2
--/app3
-./script.sh
-./script.bat
What do you think about this design? Can you recommend something better? We think if we will have repository per app it will be overkill for that small project in one team. As our applications you can imagine spring boot or spa applications in angular. Thank you in advice.
In general you can have all your micro-services in one repository but I think while the code grows for each of them it can be difficult to manage that.
Here are some things that you might want to consider before deciding to put all your micro-services in one repository:
Developer discipline:
Be careful with coupling of code. Since the code for all your micro-services is in one repository you don't have a real physical boundary between them, so developers can just use some code from other micro-services like adding a reference or similar. Having all micro-services in one repository will require some discipline and rules for developers not to cross boundaries and misuse them.
Come into temptation to create and misuse shared code.
This is not a bad thing if you do it in a proper and structured way. Again this leaves a lot of space for doing it the wrong way. If people just start using the same shared jar or similar that could lead to a lot of problems. In order to have something shared it should be isolated and packaged and ideally should have some versioning with support for backwards compatibility. This way each micro-service when this library is updated would still have a working code with the previous version. Still it is doable in the same repository but as with the 1. point above it requires planing and managing.
Git considerations:
Managing a lot of pull requests and branches in one repository can be challenging and can lead to the situation: "I am blocked by someone else". Also as possibly more people are going to work on the project and will commit to your source branch you will have to do rebase and/or merge source branch to your development or feature branch much more often(even if you do not need the changes from other services). Email notifications configured for the repository can be very annoying as you will receive Emails for things which are not in your micro-service code. In this case you need to create some Filters/Rules in your Email clients to avoid the Emails that you are not interested in.
Number of micro-services grow even further then your initial 10-15. The number can grow? If not, all fine. But if it does, at some point you could maybe consider to split each micro-service in a dedicated repository. Doing this at the point where you are in later stage of project can be challenging and could require some work and in worst case you will find out that there are some couplings that people made over time which you will have to resolve at this stage.
CI pipelines considerations:
If you use something like Jenkins to build, test and/or deploy your code
you could encounter some small configuration difficulties like the integration between Jenkins and GitHub. You would need to configure a pipeline which would only build/test a specific part of the code (or one micro-service) if someone creates a merge/pull request against that micro-service. I never tried to do such a thing but I guess you will have to figure out how to do it (script and automate this). It is doable I guess but will required some work to achieve it.
Conclusion
Still all or most of these points can be resolved with some extra management and configuration but it is still worth knowing what additional effort you could encounter. I guess there are some other points to be taken into considerations as well but my general advice would be to use separate repositories for each micro-service if you can (Private Repository pricing and similar reasons). This is a decision which is made project by project.
In my company, we have a system organized with microservices with a dedicated git repository per service. We would like to introduce gRPC and we were wondering how to share protobuf files and build libs for our various languages. Based on some examples we collected, we decided at the end to go for a single repository with all our protobuf inside, it seems the most common way of doing it and it seems easier to maintain and use.
I would like to know if you have some examples on your side ?
Do you have some counter examples of companies doing the exact opposite, meaning hosting protobuf in a distributed way ?
We have a distinct repo for protofiles (called schema) and multiple repos for every microservice. Also we never store generated code. Server and client files are generated from scratch by protoc during every build on CI.
Actually this approach works and fits our needs well. But there are two potential pitfalls:
Inconsistency between schema and microservice repositories. Commits to two different git repos are not atomic, so, at the time of schema updates, there is always a little time period when schema is updated, while microservice's repo is not yet.
In case if you use Go, there is a potential problem of moving to Go modules introduced in Go 1.11. We didn't make a comprehensive research on it yet.
Each of our microservices has it's own API (protobuf or several protobuf files). For each API we have separate repository. Also we have CI job which build protoclasses into jar (and not only for Java but for another language too) and publish it into our central repository. Than you just add dependencies to API you need.
For example, we have microservice A, we also have repository a-api (contains only protofiles) which build by job into jar (and to another languages) com.api.a-service.<version>
R-Osgi provides us a way to call service from a remote OGSi container. WebSite: http://r-osgi.sourceforge.net.
I'm new to R-OSGi and now I want to split my OSGi container into small ones and interact each other by R-Osgi because it's too huge. But it seems R-OSGi only provides a way for Registered Service. We know that the most popular way to interaction between 2 bundles besides Service, "exported-package" is also widely used.
So, is there anyone familiar with R-OSGi and know how to use "exported-package" from remote OSGi container?
Thanks for any response.
If you think about it, attempting to handle the remote importing/exporting of packages is very complex, fragile and prone to error; you'd need to send all bundle lifecycle events over the wire, and honour them in the importing system (which would require caching).
Additionally the framework would need to know ahead of time to use these class definitions (you cannot instantiate a class that references classes that aren't available to your classloader). A remote bundle's classloader may depend on classes from another classloader, this chain could go round and round a network making classloading take ages.
Put another way; your local bundles will never resolve without the class definitions they depend on, and considering there could be thousands+ of potential remote exporters on a network/hardware with very poor SLA, this wouldn't scale well or be very robust considering the fallacies of distributed computing.
If we tried to do remote packages, the framework would need to import all exported packages from all available remote nodes and then select just one to import each bundle export from (this would be arbitrary and if the select node goes down, the whole import remote package process would have to triggered again).
What you need to do is separate you api/interfaces from your implementation, you then distribute the api bundle to all nodes that need it and then use dOSGi to import services.
Apologies if this unclear or waffly but it should explain why it's simply not practical to have remote exported packages.
On a side note; I'm not sure if r-osgi is being actively maintained or is up-to-date with the latest Remote Services Admin spec, from looking at the last commit to SVN trunk was 14/02/2011. There's some alternative dOSGi implementations listed here (but avoid CXF).
EDIT: In terms of deployment, distributing your bundles (and configuration) can be done from an OBR (there are number of public ones and several implementations Felix/Eclipse), or a maven repository can be reappropriated with pax url handler.
I just started using subclipse for my class projects after a re-writing half of a project due to going off on the wrong logical branch. Since I'm using it on my home systems, I have a couple questions whose answers I haven't found in searching.
Say I have local separate repositories on my computers. First, can I use a file sync to keep them synced until I figure out what I need to do to access the university's network? (aside- only information available is about accessing the network remotely is thru PUTTY and FTP, which are not valid URL strings)
Second, in the same local system, am I correct in that once I check out a project and apply commits regularly, I don't have to keep checking out the project?
I ended up using SlikSVN for hosting the repository, since it is free for <100MB storage, which is perfect for my classwork. Also, I don't have to bother with tunneling through the school's network.
You can access SVN repositories via ssh, using urls of the form:
svn+ssh://me#myhost.com:/mypath/to/repo
From the sounds of things with you saying you have 'putty' access this should be very possible.
I would avoid the "separate local repositories" route as much as possible since keeping them in sync introduces problems that having one, definitive location for the whole repository avoids..