in company where I am working we are starting to use artifactory like tool of repositories managment, and then I'm reading the user guide of this tool. We started in the configuration creating a virtual repository, a few local and remote repositories. On the use guide i found the following thing:
Prevent disclosing sensitive business information derived from your artifact queries to whomever can intercept the queries, including the
owners of the remote repository itself.
I saw that this could be avoided through
exclude pattern
functionality on the virtual repository. Can you give us some suggestion about this? What kinds of request we should avoided to do?
You should avoid requests for internal artifacts being sent to remote repositories (directly or via virtuals). This can happen when projects depends on internal libraries or within multi module projects where modules depends on each other. When working with virtual repositories Artifactory will always search for such artifacts in local repositories first. However, if someone asked for a wrong version or had a typo in the artifact name, the artifact will not be found in a local repository and Artifactory will try to look for it in the remote repositories configured in this virtual.
To avoid exposing sensitive business information as described above, we strongly recommend the following best practices:
The list of remote repositories used in an organization should be managed under a single virtual repository to which all requests are directed
All internal artifacts should be specified in the Excludes Pattern field of the virtual repository (or alternatively, of each remote repository) using wildcard characters to encapsulate the widest possible specification of internal artifacts.
Assuming all of your projects/modules are using some kind of namespace, for example com.mycompany, you can configure an exclusion pattern for artifacts under this namespace: com/mycompany/**.
For more information take a look at avoiding security risks with an excludes pattern
Related
We are looking into setting up a local maven repository server at our institute. We have several teams that are completely independent and therefore should not have write access to each others artifacts.
Ideally we would like a similar permission concept as e.g. Gitlab, where every user can create a project (or in this case repository) and give permissions to it, but not have write access to any other repositories.
I tried out Artifactory and Nexus, but as far as I could tell, giving permissions to a group for a specific project includes manual work for a user with full admin rights, namely creating that repository.
Is there a freely available maven repository server, that allows creation and administration of repositories by users without giving them access to other repositories? If not, is there one, that can restrict write access for an artifact to the person who first deployed it?
Even better if Artifactory or Nexus can in fact do something like that and I just missed it.
You can create an arbitrary number of repositories in one Nexus 2.x (professional) server. Then you can set up different user accounts with different writing rights, e.g. you can create a projectA repository and define a projectAuser which has writing rights only for this particular repository.
I guess this is possible in Artifactory and Nexus 3 as well, but I do not have experience.
In Artifactory you have many options, I recommend you:
Create a virtual maven repository with permissions for a group of users.
Configure the settings.xml of this group of users with this virtual repository.
Create remote repositories to access to Github or any other websites.
Add these remote repositories to the virtual repository.
Create local repositories if you need and add to the virtual repository.
Do this with every group of users that you want and you will have independent artifacts and repositories.
I want to setup a development environment that allows reusing some artifacts from public Maven repositories like Maven Central, Code Haus. Specifically, I like the concept of transitive dependencies.
In our company, our production network cannot export any data outside, but we can push data inside. We already have some gateways to copy file from the outside into our network. Therefore, I could use this to copy the required packages manually but we would miss the power of maven. In our case, the perfect solution would be to be able to get data from public repository but be forbidden to deploy to the external repo.
So I would like to have your expert view on this problem.
We can use various means, as long as the capability to export data outside our network is guarantee:
External packages are created on a disk area that is read-only from production servers.
Some HTTP requests are filtered.
Using a repository manager, as Nexus.
In the repository management guide, Nexus talks about this possibility (http://books.sonatype.com/nexus-book/reference/confignx-sect-manage-repo.html). I would like a confirmation from you guys about how secure it is. Specifically, this has to be updated only by the IT manager.
Regards,
Loïc.
This is completely feasible and a common setup with Nexus. Here are the steps roughly.
Lock all developers and CI server inside the network disallowing direct access to outside servers
Setup Nexus to proxy external repositories like Central as desired
Allow Nexus to reach to those external repositories via the proxy
Configure developers and CI server machines to access Nexus to get the dependencies (and transitive dependencies) as desired
Optionally you can also
Configure CI servers to deploy any internal packages to Nexus
Configure deployment tools to get components for deployment from Nexus
Also note this can be done via different repository formats and toolchains. The common one is Maven, but Nexus also supports NPM, Nuget, Rubygems, sites, YUM and others.
And if you want to make some of your packages in Nexus available to the outside you can configure this as well following multiple options.
Also note that a proxy repository is by definition read only in terms of deployments to it directly. Thats what a hosted repository is for...
I'm trying to get the netty-codec-hhtp going in my maven project. I have a completely standard Sonatype Nexus set up to proxy requests to Maven Central.
<dependency>
<groupId>io.netty</groupId>
<artifactId>netty-codec-http</artifactId>
<version>4.0.9.Final</version>
</dependency>
This fails when building using maven. If I search for it manually in Nexus I find it, but if I go to download the jar it tells me:
404 - Not Found
Automatic routing filter rejected remote request for path /io/netty/netty-codec-http/4.0.9.Final/netty-codec-http-4.0.9.Final.jar from M2Repository(id=central)
What does this even mean, why am I getting it, and maybe more importantly, how do I fix it?
I am using Nexus 2.5.0-04 with Maven 3.0.4
Downloading other artifacts seems to work just fine.
UPDATE: This turned out to be an issue with CDN configuration that should now be resolved. The steps below for forcing and/or disabling remote discovery are left for reference.
It means that Automatic Routing for Central is active, and that the discovered rules does not contain io.jetty as allowed prefix.
This should not happen, as the default configuration should update the rules on daily basis (as seen on screenshot below, showing the default Automatic Routing configuration for Central).
The remedy is to either force the update of rules (I did check, prefix /io is among rules Central publishes), or disable remote discovery completely.
Try steps as shown below on screenshot:
From Repository Management with Nexus - 6.4. Managing Routing:
Routing can be considered the internal activities Nexus perform in order to determine, where to look for a specific component in a Maven repository. The routing information has an impact on the performance of component retrieval as well as determining the availability of components.
(...) Automatic Routing is handled by Nexus on a per repository basis.(...)The Routing information consists of the top two levels of the directory structure of the repository and is stored in a prefixes.txt file. It allows Nexus to automatically route only component requests with the corresponding groupId values to a repository to avoid unnecessary index or even remote repository access.
Since Maven central repo contains that artifact I assume that the automatic routing rules forbids remote download for that artifact. The error message you posted suggests it also.
You can read how to add a routing rule under 6.4.2. Manual Routing Configuration. If my assumption is correct, this situation shoud be resolved by adding an inclusive rule type with ^/io/netty/.* route for central repo.
I faced the same issue when configuring a bridge between 2 Nexus.
In the first:
I added 2 proxy repo, targeting 2 public repo groups in the second,
then aggregate those 2 proxy repos in a new group,
that I finally added to the repo list of the first Nexus public repo group
Using the first Nexus, each time I requested a dependency only present in the 2nd, I encountered the Automatic routing filter rejected remote request for path... exception.
As mentionned by Tamas, this issue comes from the Routing >> Discovery feature that was enabled in my 2 proxy repos, that were targeting remote Nexus repository groups.
Once disabled, the issue was solved.
The exact explanation of this need is in some way explained in the official Sonatype documentation for Repository Management with Nexus, in chapter 6.4.1 on Automatic Routing:
The Routing tab for a proxy repository displayed in Figure 6.18,
“Automatic Routing for a Proxy Repository” contains the Discovery
section. It displays the Status and a more detailed Message about the
prefix file access. The Last run field displays the date and time of
the last execution of the prefix file discovery. Such an execution can
be triggered by pressing the Update now button. Otherwise, the Update
Interval allows you to trigger a new discovery every one, two, three,
six, nine or twelve hours or as a daily or weekly execution.
[...]
For a proxy repository, the prefix file is either downloaded from the
remote repository or a generation is attempted by scraping the remote
repository. This generation is not attempted for remote Nexus
repository groups, since they are too dynamic in nature and should not
be proxied directly. Scraping of hosted or proxy repositories as well
as Subversion-based repositories is supported.
Regards,
Thomas
I am looking for an article which describes a set of guidelines to follow when creating repositories in an artifact repository manager.
I know that:
You need to keep snapshots in snapshot repositories.
You need to keep releases in release repositories.
Third-party artifacts should be in a separate repository (the same goes for forked/patched
versions of third-party libraries).
It's generally a good idea to prefix the names with int-* and ext-*.
Usually different product lines end up having their own repositories as sometimes their artifacts don't depend on each other.
I've been trying to find an article on this to illustrate to a client how this artifact separation abstraction is done by other companies and organizations using repositories.
Many thanks in advance!
I am not aware of existence of such an article, but as #tieTYT mentioned, you can look at Artifactory default repositories. They reflect years of experience in binaries management, continuous integration and delivery.
Those practices still apply even if you use Nexus (and you can observe them even without installing Artifactory, by looking at JFrog public Artifactory instance http://repo.jfrog.org)
For your convenience, here are the defaults (important usage emphasised):
Local Repositories:
libs-snapshot-local: Deploy here your local snapshots
libs-release-local: Deploy here your local releases
ext-snapshot-local: Deploy here 3rd-party snapshots which aren't available in remote repos
ext-release-local: Deploy here 3rd-party releases which aren't available in remote repos
plugins-snapshot-local: Deploy here your plugin (usually, maven) snapshots
plugins-release-local: Deploy here your plugin (usually, maven) releases
Remote Repositories:
jcenter: proxy of http://jcenter.bintray.com. Normally, that's the only remote repo you'll need. It includes whatever exists in maven central plus all other major maven repositories
Virtual Repositories:
remote-repos: aggregation of all the remote repositories
libs-release: this is the resolution repository for release builds. It includes remote-repos, libs-release-local and ext-release-local
libs-snapshot: this is the resolution repository for snapshot builds. It includes remote-repos, libs-snapshot-local and ext-shapshot-local
repo: this is special virtual repository, that aggregates everything. Generally, do not use it, if you ever plan building release pipeline using binary repository.
I'll be glad to advice on specific question.
As is the case with many questions about best practices, the answer is: It depends.
Technically there are only two distinctions that are required:
Snapshot vs release repo
Hosted vs proxy repository
Snapshots vs release repositories as a distinction is required since the Maven repository format and therefore Maven and other build tools differentiate how they work with the the meta data and what they do during upload.
For proxy repositories you will just have to add as many you need to proxy. This will depend on what components you require and will be separate for proxying snapshot and release repos.
For hosted repositories you also have to have separate snapshot and release repos. Beyond that is is all up for grabs. Having a separate third party repo as preconfigured in Nexus (and Artifactory) and other setups are certainly useful, but not really necessary. You can have all those distinctions sorted out by internal meta data where required.
Along the same lines you can have one release repo for everyone or one for each team or whatever. You can still apply access rights within those repositories to separate access and so on in Nexus with repository targets. I assume Artifactory and Archiva can do something similar. The question here mostly boils down to ease of administration, backups, security setup and access for users.
Naming conventions like you mentioned can help if you want to have separate repositories, but technically none of this is necessary.
Other things I have seen are e.g. migration repos that are used to migrate legacy project libraries into a repo but become frozen after the migration is done, separate repos per team, separate repos per project and so on. Another aspect are separate repos for different levels of approval and so on (e.g. check out problems with that on http://blog.sonatype.com/people/2013/10/golden-repository/)
In the end however this all hinges really on usability and meta data and is not required. Ultimately these repositories will in most cases grouped together and accessed via one group, which flattens out the whole separation. And access rights still carry through into the group so everything can still be controlled as you like. So it turns to be a matter of taste on how you want to slice and dice and manage it.
PS: I am referring to the Maven repositories and format. Once you add a whole bunch of other formats into the mix and wrappers around them exposing them in other formats, everything gets more complicated, but the ideas behind things stay similar.
Do you know a way to configure Nexus OSS so that it publishes the artifact repository to a remote server in a form that can be statically served, e.g. by Apache Httpd? I'd like to use this static copy to serve only my own artifacts, so the nexus server could actively trigger an update in case there is something new published.
Technically, I think it should be possible to create the metadata for the repo and store them in a static file, but I'm not sure with that. Any hints appreciated.
If there is another repo manager to achieve that, it would be fine for me as well.
I clearly understand the advantages to use the repo manager directly, but due to IT rules I can run Nexus only internally and it would be necessary to have these artifacts available in a (private) repo copy on the Internet as well.
A typical way to solve this IT requirement of only exposing known servers like Apache httpd is to setup Apache httpd as a reverse proxy as documented here.
You can use that approach in a more restrictive way by only exposing a specific repository or better repository group (so you can combine snapshots and releases) and tying that together with a specific user or a specifically restricted setup of the anonymous user that is used by default when no credentials are passed through.
Also if you need more help feel free to contact us in the user mailinglist or on hipchat.