Structure of winget source repositories - windows

Microsoft announced and open-sourced winget, a package manager for Windows.
When entering winget source (after installing it), it can be seen that it supports adding sources.
winget source provides the following subcommands:
add Add a new source
list list current sources
update updates current sources
remove removes current sources
reset resets sources
By entering winget source list, the default source is displayed (as long as the sources were not changed):
C:\Windows\System32>winget source list
Name Arg
-----------------------------------------
winget https://winget.azureedge.net/cache
As sources can be added using winget source add, how can source repositories be created?
How does a repository has to be structured or is there just not enough documentation to answer this question?
I think this question is on-topic for Stack Overflow as the main reason to add sources is (in my point of view) to create alpha/beta channels for programs or similar.

winget 1.0 introduced a new REST API for custom repositories. The API and a reference implementation are provided by Microsoft at github.com/microsoft/winget-cli-restsource/. The reference implementation uses C# and allows self-hosting on Azure. Hosted offerings are also becoming available, such as https://winget.pro.
Before winget 1.0, the structure of winget repositories looked roughly as follows:
The remote repository should have a source.msix file. You may refer to the example at the default repository: https://winget.azureedge.net/cache/source.msix
One can find details about MSIX itself on the Microsoft Docs website.
Actually the source.msix is a zip package with contents structured in a pre-defined fashion (just rename it to source.zip and unpack):
Assets/
Public/
AppxBlockMap.xml
AppxManifest.xml
AppcSignature.p7x
[Content_Typex.xml]
The main data file seems to be Public/index.db. It is an SQLite database containing information derived from Community Repo Manifests. It has a quite simple structure to understand.
Another concern is the MSIX should be signed by the developer. One should change Windows settings in order winget to accept packages signed by third parties. See below for details.
Note that winget doesn't accept an HTTP repository, it requires only the HTTPS one with a trusted certificate.
The most interesting tables in index.db are manifest and pathparts. The first matches application's name, version, etc. to pathparts, and the latter points to the manifest YAML-file.
For example: https://winget.azureedge.net/cache/manifests/RubyInstallerTeam/Ruby/e70d-2.7.2.yaml (a cache of github/winget-pkgs/manifests/RubyInstallerTeam/Ruby/2.7.2.yaml).
winget uses this cached manifest for application installation.
In order winget could add the third-party source repository (when the source.msix signed by third-party certificate) one should allow installing sideload apps.
Summarizing the above the overall sequence seems to be the following:
Download the source.msix from the winget's default repo: https://winget.azureedge.net/cache/source.msix
Unpack it as a ZIP-package or using the MSIX Packaging Tool to get the index.db file from Public directory.
Edit this SQLite DB (I've used the DB Browser for SQLite for that) leaving only your application (tables ids, monikers, names, versions).
Set the path to YAML-manifest in pathparts table (one record per each path element). E.g. for /manifests/MyCompany/MyProduct/1.0.0.yaml the table should be like:
Edit the manifest table by adding the record that unites all related records.
Save the edited index.db, overwrite it in source.msix using the MSIX Packaging Tool, sign the MSIX with your own code-signing certificate.
Upload both the manifest (see Manifest Specification for details about manifest itself) and source.msix to your server (e.g. to myserver.net/repo/source.msix and myserver.net/repo/manifests/MyCompany/MyProduct/1.0.0.yaml).
Use:
winget source add myrepo https://myserver.net/repo
winget install MyProduct

You can deploy a private REST source to Azure using the reference implementation.
It is also possible to create your own implementation referencing the Swagger API.

WinGet 1.0 has just been released, and Microsoft has also released a reference implementation of the REST API source so it’s possible to host your own private repository.
It's called WinGet.RestSource and can be cloned from here; because it’s an Azure solution, next to Visual Studio 2019 you also need an Azure subscription.
I found that getting WinGet.RestSource up and running and pushing packages to it (currently) isn't that easy, but for anyone who's interested, I've written 2 blog posts about this:
Hosting your own WinGet private repository
Adding a package to your private WinGet.RestSource feed using its API

As of July 2020, as clarified by How to Update Winget Source List:
Winget is a really young application in terms of software age and currently, only has a single source/repo available for it, Microsoft's official Winget repository.
Thus sub-commands such as add and remove aren't of use yet.

Related

How to list NuGet packages used in my project programatically with Xamarin.Forms?

In order to print them in the legal info of the app, I'd like to recover all the used packages in the app along with their version and the repo that contains them.
So, is there anything within the framework that allows me to recover programmatically (from c#) the info you see on the Nuget Package Manager for each package in the project?
This is the info that we would like:
But only the name, version, and Project URL would be alright too...
You can get all the needed information from YourProject.csproj file.
Thanks!
even you can use NuGet Server API for get extra information API

Sign and update Visual Studio Extensions

So it seems as a pretty easy Task, I created a VSIX which I uploaded a few days ago. Some days later I fixed some issues and now I want to update the extension on the marketplace.
I went ahead and took a look at the MSDN which states the following:
Unsigned all user extensions do not get updated automatically. You should always sign your extensions.
So therefor I went on an tried to sign my VSIX as explained here. Which tells me to do the following:
Search for VSIXSignTool from VisualStudioExtensibility and install the NuGet package.
After downloading the package, I searched for the given location which should be in your project's local packages location. Which makes me wonder, cause I do not have any local packages folder for this project. Furthermore I should sign my extension with the following command:
VSIXSignTool.exe sign /f <certfile> /p <password> <VSIXfile>
Which makes me wonder, what kind of certificate that should be, what should be the password be, a new password?
Anyway, lets assume I somehow mange to do that, how would I 'update' the extension than? Just upload it again as a new package with a new VSIX-Id?
To sum it up, this are my questions:
How/Where do I download the VSIXSignTool
What parameters do I need to pass into the VSIXSignTool command provided above
How do you really update the extension on the marketplace and do I need a new VSIX Id
I suggest first try to update your extension without signing. Just increase the version number and on the marketplace site click Manage - ... (More Actions) - Edit - Upload Extension/Edit package.
VSIXSignTool requires a code signing certificate that needs to be purchased.

Incremental auto online update with Install4J

I need some help regarding Install4J and the auto online update function.
We have a JRE bundled Install4J installer which works fine. We use the auto online update functionality which works fine, too. All this functionality in integrated to a continuous integration process in TeamCity.
Unfortunately the update installation setup file has a size of around 100MB which is inconvenient to download if we only want to apply a patch or a minor update.
As far as I have understood from other similar questions and the Install4J documentation the right way to deal with that is using a second project file with another application ID and setting up an Add-On-Installer which only contains relevant files and points to the full installer application ID.
To enable the auto online update functionality I have to upload the update.xml file from the Add-On installer, am I right?
By the way, does the Add-on online auto update procedure automatically detect the locally installed version correctly?
But in that case we have to adapt the add-on installer each and every time we are deploying a new patch or minor feature which is difficult to implement in the CI process.
Is there a way to configure an Install4J installer so that it automatically gets the information online which elements have to be updated so that the download size is as small as possible and we could just use one default add-on installer which contains all sources beside the JRE bundle?
Something like an online incremental update procedure which is managed internally by Install4J?
Thank you.
You should never use the add-on installer for updating, because it cannot change the version number.
In order to get smaller update installers, you create a second project where the distribution tree only contains the changed files - or at least does not contain some large files that did not change. To do that, you simply copy the original installer and delete the files in the distribution tree that you do not need to update.
Then you build two sets of installers, those for a new installation and those for the update.

Making OS X web Installer Packages

I have an installer implemented with "Packages" which contains the payload and after running some plugins and a post install script it successfully installs the product.
The same package bundle is used for making updates too, as we run it in background with root privileges and it overwrites the current/old installation files.
We now have the requirement to make a lightweight installer having the components (different Packages inside the product) in a web location to be downloaded and installed.
As I know that this is possible in other systems as Windows/InstallShield with "Releases of Web Type", I would like to know if this is possible in OS X. Otherwise, I will have implement it all from scratch (packages management, download, packages versions compare to make selective updates, privileges escalation, etc).
Well, it was long ago and I almost started doing a web installer from scratch but then I found out that the option is in Packages itself. Hope this helps.
It is the "Package Reference" option.
According to the Packages documentation
A Package Reference lets you use a package that is hosted on a web
server or a removable media and to which you may not have a direct
access. This package will not be built during the built phase.
I think that any referenced package will have its own pre-post-scripts so there would be no problem because of the limited options in the Package Reference. But I need to test it.

DVCS with a Windows central repository

We are currently using VSS for version control. Quite few of our developers are interested in a distributed model (And want to get rid of VSS). Our network is full of Windows machines and while our IT department has experience maintaining Linux machines they would prefer not to.
What DVCS systems can host their central repository on Windows while providing..
Push access to the repository.
Basic authentication. Mostly just a way to allow or deny access to the whole repository. No need for fine grained access.
Server process so users don't need write right to the repository reducing the risk of accidentally messing with it.
On the client side a GUI such as Tortoise would be more or less a requirement (Sorry, Windows shell sucks. :|). Ease of installation would be a huge plus as our IT department is already quite low on resources. And using windows credentials for authentication would be an advantage but not a requirement as long as the client is able to store the credentials.
I have had a (really) quick look at Git, Mercurial and Bazaar.
Git seemed to use ssh or simple WebDAV for repository access, requiring write permission for the users.
Mercurial had a built in http server, but this seemed to be only for pull purposes. Update: Mercurial supports push as well.
Bazaar Seemed to use sftp for repository access, again requiring a write permission for the users.
Are there windows server processes for any DVCS systems and has anyone managed to set one up in a Windows land?
And apologies if this is a duplicate question. I couldn't find one.
Update
Got Mercurial working for push purposes! Detailed list what was required can be found as an answer below.
Mercurial's almost certainly your easiest option on Windows.
If you didn't care about authentication, you actually can trivially allow hg serve to permit push. To do so, you merely need to add the following to the .hg/hgrc file in the repository you wish to serve:
[web]
allow_push = *
push_ssl = false
The first line says that anyone may push to this repository. The second tells Mercurial to allow pushing without SSL, since hg serve does not currently natively support HTTPS. At this point, users can push to your repository without having an account anywhere. If you're simply a small shop, that's probably fine--especially since you can use Mercurial's ability to sign changesets to guarantee a much higher level of verifiability than HTTP Basic will provide, anyway.
For a larger, shop, though, you'd be totally right in wanting at least a simple barrier for committing. To do that, you need to make two changes. First, you'll need to put Mercurial behind a web server with either reverse proxy support or CGI support. Thankfully, recent versions of IIS support both. You can consult the CGI directions in the Mercurial Redbook for Mercurial-specific steps, and Microsoft's guide to setting up CGI applications in IIS 6 for help on the IIS side.
Next, you'll need to set up some basic authentication. IIS provides HTTP Basic out-of-the-box, which, as a bonus, can authenticate directly against your domain, keeping administrative overhead to a minimum.
Finally, you'll want to change the allow_push line to support only specific users by specifying a comma-delimited list of user names. For example:
allow_push = benjamin, ted, the_cow
That's it. Mercurial will now allow push from users who can authenticate via HTTP Basic authentication, and allow pull from everyone else.
After Benjamin pointed out the HTTP serving CGI scripts I decided to try those out and managed to get a repository hosted over HTTP. The Redbook which Benjamin linked was of much help as were two Mercurial wiki articles. One which describes Mercurial publishing in general and another containing step by step instructions for setting up the HgWebDir CGI script.
These instructions weren't completely foolproof though so I had to poke around a bit. Most likely as I'm running 64bit Vista. The instructions below document what I did. Now that I've done it once I'd probably do things in another order so don't consider these step by step instructions.
Mercurial
First I acquired the Mercurial binary from http://mercurial.berkwood.com/ which got installed into d:\dev\Mercurial. I created a repository for testing under d:\dev\testRepo repository using hg init. The d:\dev\Mercurial\library.zip contains Mercurial library files required by the CGI script so they were extracted to d:\dev\Mercurial\library. Something which confused me at first is that when I opened the zip file I received an error message and saw no contents. Just extracting the file to a directory worked though.
For the web script, I downloaded Mercurial source which contained the hgwebdir.cgi which got moved and renamed to d:\dev\Mercurial\webroot\hgwebdir.py. The step by step article contains good instructions for modifying the hgwebdir script for Windows. They also contain instructions for hgweb.config which in my case ended up looking like this:
[paths]
/hg/hgwebdir.py/test = D:\dev\Mercurial\testRepo
Also the repository wanted the following config so I could push there without SSL. Note I am using Basic Authentication to authenticate users currently. I had to create the config in D:\dev\Mercurial\testRepo\.hg\hgrc and add the following lines to it:
[web]
allow_push = *
push_ssl = false
Python
The CGI script is a Python script so it requires Python. It's seems pretty picky on which Python version executes it. One of the articles mentioned that running it requires same version that was used to build the Mercurial. In the end I got it working on Python 2.5 x86 after trying Python 2.6 x64, Python 2.4, Python 2.5 x64.
IIS
Two things I missed and had to install were CGI support and Basic Authentication. Both of these were installed through Control Panel, Programs and Features. Once done with installation I created a virtual directory (Which I later changed to an Application) in IIS pointing to D:\dev\Mercurial\webroot. The virtual directory required an CGI handler for *.py files which could be added from Handler Mappings. The executable was D:\dev\SDKs\Python25_x86\Python.exe %s. Once IIS had permissions to the webroot directory I could navigate to http://localhost/hg/hgwebdir.py/test and see the repository.
So now the read access was working. When I tried pushing to the repository I received weird error messages telling me it wasn't a real repository.
After an hour of debugging I ended up copying the whole D:\dev\Mercurial\library\mercurial tree under webroot so that Python could find D:\dev\Mercurial\webroot\mercurial\hgweb\hgwebdir_mod.pyc. After this Wireshark was reportting Access Denied errors in the stack trace. No idea what the real reason to this was but changing the virtual directory into an Application in IIS and moving it on top of an application pool which ran using Local System account the access denied errors went away.
Also at some point I gave HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\WinSock2\Parameters registry key more permissions so IIS could access it. Doubt that it requires these after using Local System account.
Once these were done pushing stuff to the repository using hg push http://localhost/hg/hgwebdir.cgi/test was working!
Problems and solutions
Where to find the library files.
They were in the library.dll under Mercurial installation folder. I just had to extract them even if my unzip program refused to view me its contents.
How to get the Python script to run
Download the correct Python version for x86 architecture as the script uses some x86 libraries. The correct Python version depends on the Mercurial version. For 1.2.1 it was Python 2.5 x86.
Alternatively you could try building Mercurial from sources with whatever Python version you want but in my case this failed when building extensions.
How to set CGI up in IIS
First make sure CGI is installed in IIS. This wasn't assumed to be true in the IIS instructions Benjamin posted.
Create a new Module Mapping for *.py in IIS Handler Mappings. The correct Module is CgiModule and the executable is your Python executable + %s
How to allow the CGI script to write to the repository
Make sure the script has everything it requires. I had to move the library\mercurial\hgweb\hgwebdir_mod.pyc to another place.
Make sure the script has permissions to everywhere it wants. I solved this by Creating a new Application Pool for the CGI script that used Local System account, converting the Virtual Directory to an Application in IIS and selecting the new Application Pool.
After reading Mikko's Answer which almost worked for me, I came up with my own notes for installation. My setup was designed to be a "non protected and open" repository that members of my team could use installed on a Windows 2008 Server.
1. Install Python.
The version of Python I used was Python 2.6.2 and I used the Windows x86 MSI Installer.
Install for all Users.
Install to C:\Mercurial\Python
Use Default Feature Options.
2. Install MinGW.
The version of Minimalist GNU for Windows I used was MinGW 5.1.4
Install the MinGW-5.1.4.exe.
Choose the Download and Install Option.
Choose the Current Package Option to Install.
For the Components to Install Select the "Minimal" option.
Install to C:\Mercurial\MinGW
3. Modify your path.
You need to add in locations to your environmental path at this point.
Add 'C:\Mercurial\Python26;C:\Mercurial\MinGW\bin' to the path (Order Matters.)
4. Install Mercurial.
The version of mercurial that I used was the latest release in the stable branch and I did not use the binaries, but used the source code. I wanted to compile mercurial myself so that it would work with whatever version of Python I had installed so I didn't have to worry about any compatability issues which I found to be the biggest challenge with other install methods. The easist way to get the source is by downloading the "zip" file.
Mercurial Stable Release
Extract Zip File to C:\Mercurial\Source.
Build the Source at command prompt.
python setup.py build --force -c mingw32
python setup.py install --force --skip-build
5. Modify your path.
You need to insert into your environmental path another location for the 'hg' command.
Add 'C:\Mercurial\Python26\Scripts;C:\Mercurial\Python26;C:\Mercurial\MinGW\bin' to the path (Order Matters.)
6. Create your Config file.
You need to have a default user name set if your going to do any commits locally on this server.
Create file '"C:\Documents and Settings{username}.hgrc"'
[ui]
editor = Notepad
username = your_name
6. Test your Install.
Open up a new command window and test with 'hg debuginstall' to validate. You should see something like the following.
Checking encoding (cp1252)...
Checking extensions...
Checking templates...
Checking patch...
Checking commit editor...
Checking username...
No problems detected
7. Setup Web Directory.
Create Directory 'C:\Mercurial\Web'
Copy the hgwebdir.cgi file from the 'C:\Mercurial\Source' to 'C:\Mercurial\Web'
8. Configure IIS7 for Centralized Repository.
I used the DefaultAppPool which is using .Net 2.0, Pipeline=Integrated, Identity = ApplicationPoolIdentity.
Ensure CGI features are available in IIS7.
Control Panel/Programs/Windows Features/IIS/App Development Features/CGI
Add App into IIS on the Website you wish.
Alias=Mercurial -- Physical Path=C:\Mercurial\Web
On the App select HTTP Modules and add a new Module Mapping.
Request Path=*.cgi, Module=CgiModule, Executable=C:\Mercurial\Python26\python.exe %s, Name=Mercurial.
When Prompted to add entry to ISAPI and CGI restrictions list say yes.
9. Test your Web Setup.
You should now be able to browse http://localhost/Mercurial/hgwebdir.cgi and see and empty repository list.
10. Configure IIS7 for Friendly URL
I did not like having the unfriendly URL and this step allows us to remap the URL to something more friendly. Install the URL Rewrite Moduel 1.1 Extension for IIS.
On the Mercurial IIS Application in IIS Manager featurs View select URL Rewrite Component and install a new Rule.
Choose Add Rules, then the Template 'Rule with rewrite map.' Rule Action=Rewrite, Specify Rewrite Map=Mercurial
Add a mapping Entry. OriginalValue='/Mercurial/Repo', New Value='/Mercurial/hgwebdir.cgi'
11. Create Mercurial Repository
You can now create a test repository.
Create a Directory C:\Mercurial\Repository and ensure IUSR account has the permissions to write to the directory. (If on Domain account is more like IUSR_{ComputerName}.
Create file C:\Mercurial\Web\hgweb.config to list the repositories.
[paths]
/ = C:\Mercurial\Repository\**
Add a directory C:\Mercurial\Repository\Test and initialize the repository with 'hg init'
** If you want now to be able to push without ssl create in the .hg directory of the repository a hgrc file the following lines.
[web]
allow_push = *
push_ssl = false
References:
Mercurial Wiki Windows Install
HG Book
Step by Step
Publishing Mercurial Repositories
For a team taking the first step away from VSS I would have suggested using SubVersion for source control and either TortoiseSVN or VisualSVN for the client.
But if the team has made the decision to switch to a DVCS then I'd suggest Mercurial because of it's better support for HTTP and windows on the client via TortoiseHg.
If you're looking for:
Distributed development support
Run Windows servers seamlessly
And a great GUI
You're exactly describing Plastic SCM
Excuse my necroposting and shameless self-promotion, but I've just released an alpha version of HgLab, which is a Mercurial Server for Windows with full pull-push support and Active Directory integration.
SCM-agnostic (to some degree) Windows-solution with Repository-frontent and management today may be SCM-Manager (Git, Mercurial, SVN repo out of a box with a single requirement of JVM)

Resources