download NuGet Licenses from VS Project and export it as plaintext - visual-studio

Question

is there a way to convert html to plaintext?
I have a script that exports all NuGet-Licenses which been used in a visual studio project to a textfile.
Unfortunately the exports are mostly in HTML, and I found no way to solve it.
# Run in Package Manager Console with `./download-packages-license.ps1`.
# If access denied, execute `Set-ExecutionPolicy -Scope Process -ExecutionPolicy RemoteSigned`.
# Save licenses to One text file and one csv file instead of individual files
$LicensesFile = (Join-Path (pwd) 'licenses\Licenses.txt')
$LicensesFile_csv = (Join-Path (pwd) 'licenses\Licenses.csv')
$results = #()
# Below 2 lines to comment if you uncomment Split-Path ..
$solutionFile = "d:\Solutions\SolFile.sln"
cd "d:\Solutions"
# Uncomment below line if you wish to want to use above 2 lines
# Split-Path -parent $dte.Solution.FileName | cd;
New-Item -ItemType Directory -Force -Path ".\licenses";
#( Get-Project -All | ? { $_.ProjectName } | % {
Get-Package -ProjectName $_.ProjectName | ? { $_.LicenseUrl }
} ) | Sort-Object Id -Unique | % {
$pkg = $_;
Try
{
if ($pkg.Id -notlike 'microsoft*' -and $pkg.LicenseUrl.StartsWith('http'))
{
Write-Host ("Download license for package " + $pkg.Id + " from " + $pkg.LicenseUrl);
#Write-Host (ConvertTo-Json ($pkg));
$licenseUrl = $pkg.LicenseUrl
if ($licenseUrl.contains('github.com')) {
$licenseUrl = $licenseUrl.replace("/blob/", "/raw/")
}
$extension = ".txt"
if ($licenseUrl.EndsWith(".md"))
{
$extension = ".md"
}
(New-Object System.Net.WebClient).DownloadFile($licenseUrl, (Join-Path (pwd) 'licenses\') + $pkg.Id + $extension);
$licenseText = get-content "$((Join-Path (pwd) 'licenses\') + $pkg.Id + $extension)"
Remove-Item $((Join-Path (pwd) 'licenses\') + $pkg.Id + $extension) -ErrorAction SilentlyContinue -Force
$data = '' | select PkgId, LicenseText
$data.PkgId = $pkg.Id
$data.LicenseText = $licenseText | Out-String
$results += $data
# save in txt file
"Designation: NugetPackage $($pkg.Id)" | Add-Content $LicensesFile
$licenseText | Add-Content $LicensesFile
"" | Add-Content $LicensesFile
"" | Add-Content $LicensesFile
"" | Add-Content $LicensesFile
"" | Add-Content $LicensesFile
Write-Host "Package $($pkg.Id): License Text saved to $LicensesFile" -ForegroundColor Green
}
}
Catch [system.exception]
{
Write-Host ("Could not download license for " + $pkg.Id)
}
}
# save in .csv file
$results | Export-Csv $LicensesFile_csv -nti
Source of the Script here
A user also said ,,Unfortunately, most license URLs now point to HTML-only versions (early 2020). For example, licenses.nuget.org ignores any "Accept: text/plain" (or json) headers and returns html regardless"
So is there even a way to get the license information in plaintext?
Thanks and stay healthy!

Answer 1

So is there even a way to get the license information in plaintext?
Actually, we do not recommend that you convert the html file into plaintext format. And when you get the license data from nuget.org, it is the data returned from the site in full HTML format, which is designed by that.
The returned data also contains various formats for the license field, so we should not easily modify the accepted data format(such as plaintext ). And if possible, the only way to do this is to get rid of the HTML format fields from the source data, but however, it is impossible by Powershell and it cannot be done so far.
Therefore, in order to strictly follow the format of the returned data, it is best to use an HTML file to receive license info. It can maintain consistency with the website in the form of html.
Suggestion
1) change these in powershell:
$LicensesFile = (Join-Path (pwd) 'licenses\Licenses.html')
$LicensesFile_csv = (Join-Path (pwd) 'licenses\Licenses_csv.html')
And then you can get what you want.
Hope it could help you.