Hi,
I have been working on the validation of Rubygem licenses with the SPDX format.
all work done so far lives in my fedorapeople space: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/
It is WIP, including the scripts. (I got a bit sidetracked with validating MIT variations.) Note that you have to be in the directory where `rpm-specs-latest.tar.xz` was upacked into.
From the total of about 500 rubygems (and with my incomplete script) we have 232 SPDX licenses:
~~~ $ cat current_ok_rb.csv | wc -l 232 ~~~
The real count is probably higher, as some gems are simply MIT or have correct SPDX, however are older so they do not have the license metadata included that I can compare to.
See this list for complete info: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/rbge... You can see the `current_ok_rb.csv` in the same repository.
So far, I have validated mostly MIT (as those are the easiest to get right), I will need to fix the script before my next update to account for other licenses too.
If your gem has multiple valid SPDX license identifiers, but are in conjunction using "and" "or" instead of "AND" "OR", the script will be able to catch that and I'll make a PR to the gem with the converted form should that be the case.
There are also a few gems excluded from the search as they do not have a gem in Fedora's sources cache. These gems are: rubygem-morph-cli rubygem-krb5-auth rubygem-asciidoctor rubygem-rgen rubygem-net-irc
I have not run the script against them, they are explicitly excluded, since so far I am only comparing the RPM Spec against gem2rpm output.
== Plan ==
I plan to do this in 2 main phases:
1) Initial pass This is just a pass through the packages to note all packages that are already SPDX compatible.
2a) Inspecting and fixing This probably will be mostly manual work to identify correct licenses for gems that do not have license in the gemspec metadata. Also convert Callaway convention to the correct SPDX identifier. This cannot be reliably automated for example the BSD license can be converted to 4 different SPDX license identifiers. However, tooling can give an idea of what the licensing probably is.
At that point I will be mainly looking on Fedora spec's license, whether it has valid SPDX or not.
2b) Validate MIT licenses MIT license has multiple variants. While it mostly seems that the gem MIT licenses are under what SPDX also considers MIT, I'd like to validate that assumption.
== How to help ==
Make sure your rubygem package has a license in Fedora's specfile that is a valid SPDX license identifier.
You can also take a look at the gems where license does not match between gem2rpm and fedora (RubyGems should use SPDX, even though it is older version, that's a good pointer for us) and fix them. On closing note, be wary of licenses like "LGPL-2.1+", it is valid, however deprecated by current SPDX version and the `license-validate` tool won't accept it.
See https://spdx.org/licenses/ for complete license list including the list of deprecated identifiers.
Thanks, Jarek
Hi all,
I have fixed up the script a bit and sprinkled some more processing on top, so I come back with more precise data (still rpm-specs as of 2022-11-14, as with previous email)
The state is as follows:
* Total of rubygem packages in Fedora: ~~~ $ ls rubygem-*.spec | wc -l 495 ~~~
* Total of packages I ran script against (-1 line for the header): ~~~ $ cat rubygems_fedora_spdx_state.csv | wc -l 487 ~~~
* Ruby gems where /Fedora License field/ and /gem2rpm/ output matches and /license-validate/ says they are OK SPDX: *291/495*
https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
* Ruby gems where only /license-validate/ says they are OK SPDX, but licenses may or may not match between Fedora and upstream: *334/495*
https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
* Ruby gems where the "or" and "and" are converted to "OR" and "AND" and /license-validate/ says they are OK SPDX: *337/495* ~~~ $ cat rubygems_try_convert_conjunctions.csv | grep -E "(true|false);0" | wc -l 337 ~~~ https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
I will follow up with RPM specs from this week later.
Regards, Jarek
On 11/21/22 14:25, Jarek Prokop wrote:
Hi,
I have been working on the validation of Rubygem licenses with the SPDX format.
all work done so far lives in my fedorapeople space: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/
It is WIP, including the scripts. (I got a bit sidetracked with validating MIT variations.) Note that you have to be in the directory where `rpm-specs-latest.tar.xz` was upacked into.
From the total of about 500 rubygems (and with my incomplete script) we have 232 SPDX licenses:
$ cat current_ok_rb.csv | wc -l 232The real count is probably higher, as some gems are simply MIT or have correct SPDX, however are older so they do not have the license metadata included that I can compare to.
See this list for complete info: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/rbge... You can see the `current_ok_rb.csv` in the same repository.
So far, I have validated mostly MIT (as those are the easiest to get right), I will need to fix the script before my next update to account for other licenses too.
If your gem has multiple valid SPDX license identifiers, but are in conjunction using "and" "or" instead of "AND" "OR", the script will be able to catch that and I'll make a PR to the gem with the converted form should that be the case.
There are also a few gems excluded from the search as they do not have a gem in Fedora's sources cache. These gems are: rubygem-morph-cli rubygem-krb5-auth rubygem-asciidoctor rubygem-rgen rubygem-net-irc
I have not run the script against them, they are explicitly excluded, since so far I am only comparing the RPM Spec against gem2rpm output.
== Plan ==
I plan to do this in 2 main phases:
- Initial pass
This is just a pass through the packages to note all packages that are already SPDX compatible.
2a) Inspecting and fixing This probably will be mostly manual work to identify correct licenses for gems that do not have license in the gemspec metadata. Also convert Callaway convention to the correct SPDX identifier. This cannot be reliably automated for example the BSD license can be converted to 4 different SPDX license identifiers. However, tooling can give an idea of what the licensing probably is.
At that point I will be mainly looking on Fedora spec's license, whether it has valid SPDX or not.
2b) Validate MIT licenses MIT license has multiple variants. While it mostly seems that the gem MIT licenses are under what SPDX also considers MIT, I'd like to validate that assumption.
== How to help ==
Make sure your rubygem package has a license in Fedora's specfile that is a valid SPDX license identifier.
You can also take a look at the gems where license does not match between gem2rpm and fedora (RubyGems should use SPDX, even though it is older version, that's a good pointer for us) and fix them. On closing note, be wary of licenses like "LGPL-2.1+", it is valid, however deprecated by current SPDX version and the `license-validate` tool won't accept it.
See https://spdx.org/licenses/ for complete license list including the list of deprecated identifiers.
Thanks, Jarek
Hi,
current statistics for specs from 24. 11. are as follows: * Total rubygems in Fedora: 492 * Total rubygems checked: 485 * Fedora License field and gem2rpm license match and license-validate succeeds: 288/485 * license-validate says they are OK SPDX, but licenses may vary: 331/485 * license-validate says they are OK SPDX when conjunctions were converted: 334/485
There were 3 rubygem packages dropped in Fedora, otherwise the stats are the same.
Regards, Jarek
On 11/22/22 14:06, Jarek Prokop wrote:
Hi all,
I have fixed up the script a bit and sprinkled some more processing on top, so I come back with more precise data (still rpm-specs as of 2022-11-14, as with previous email)
The state is as follows:
- Total of rubygem packages in Fedora:
$ ls rubygem-*.spec | wc -l 495
- Total of packages I ran script against (-1 line for the header):
$ cat rubygems_fedora_spdx_state.csv | wc -l 487
- Ruby gems where /Fedora License field/ and /gem2rpm/ output matches
and /license-validate/ says they are OK SPDX: *291/495*
https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
- Ruby gems where only /license-validate/ says they are OK SPDX, but
licenses may or may not match between Fedora and upstream: *334/495*
https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
- Ruby gems where the "or" and "and" are converted to "OR" and "AND"
and /license-validate/ says they are OK SPDX: *337/495*
$ cat rubygems_try_convert_conjunctions.csv | grep -E "(true|false);0" | wc -l 337https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/ruby...
I will follow up with RPM specs from this week later.
Regards, Jarek
On 11/21/22 14:25, Jarek Prokop wrote:
Hi,
I have been working on the validation of Rubygem licenses with the SPDX format.
all work done so far lives in my fedorapeople space: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/
It is WIP, including the scripts. (I got a bit sidetracked with validating MIT variations.) Note that you have to be in the directory where `rpm-specs-latest.tar.xz` was upacked into.
From the total of about 500 rubygems (and with my incomplete script) we have 232 SPDX licenses:
$ cat current_ok_rb.csv | wc -l 232The real count is probably higher, as some gems are simply MIT or have correct SPDX, however are older so they do not have the license metadata included that I can compare to.
See this list for complete info: https://fedorapeople.org/cgit/jackorp/public_git/spdx_rubygems.git/tree/rbge... You can see the `current_ok_rb.csv` in the same repository.
So far, I have validated mostly MIT (as those are the easiest to get right), I will need to fix the script before my next update to account for other licenses too.
If your gem has multiple valid SPDX license identifiers, but are in conjunction using "and" "or" instead of "AND" "OR", the script will be able to catch that and I'll make a PR to the gem with the converted form should that be the case.
There are also a few gems excluded from the search as they do not have a gem in Fedora's sources cache. These gems are: rubygem-morph-cli rubygem-krb5-auth rubygem-asciidoctor rubygem-rgen rubygem-net-irc
I have not run the script against them, they are explicitly excluded, since so far I am only comparing the RPM Spec against gem2rpm output.
== Plan ==
I plan to do this in 2 main phases:
- Initial pass
This is just a pass through the packages to note all packages that are already SPDX compatible.
2a) Inspecting and fixing This probably will be mostly manual work to identify correct licenses for gems that do not have license in the gemspec metadata. Also convert Callaway convention to the correct SPDX identifier. This cannot be reliably automated for example the BSD license can be converted to 4 different SPDX license identifiers. However, tooling can give an idea of what the licensing probably is.
At that point I will be mainly looking on Fedora spec's license, whether it has valid SPDX or not.
2b) Validate MIT licenses MIT license has multiple variants. While it mostly seems that the gem MIT licenses are under what SPDX also considers MIT, I'd like to validate that assumption.
== How to help ==
Make sure your rubygem package has a license in Fedora's specfile that is a valid SPDX license identifier.
You can also take a look at the gems where license does not match between gem2rpm and fedora (RubyGems should use SPDX, even though it is older version, that's a good pointer for us) and fix them. On closing note, be wary of licenses like "LGPL-2.1+", it is valid, however deprecated by current SPDX version and the `license-validate` tool won't accept it.
See https://spdx.org/licenses/ for complete license list including the list of deprecated identifiers.
Thanks, Jarek
Hi all,
Current stats for 2022-12-05
I have also included a list of packages for the field "Action required", where license-validate returns error and the licenses need to be converted to proper SPDX ID.
If you'd like to see some other info generated with this, feedback is welcomed.
~~~ * Total rubygems in Fedora: 494 * Total rubygems checked: 486 * Fedora License field and gem2rpm license match and license-validate succeeds: 288/486 * license-validate says they are OK SPDX, but licenses may or may not match between Fedora and upstream: 332/486 * license-validate with converted conjunctions: 335/486 * Action required: 154/486
Packages that need to convert to valid SPDX: rubygem-gem_name, fedora_license ---------------- -------------- rubygem-abstract, GPLv2 or Ruby rubygem-addressable, ASL 2.0 rubygem-allison, AFL rubygem-ansi, BSD rubygem-archive-tar-minitar, GPLv2+ or Ruby rubygem-aruba, MIT and CC-BY and (MIT or GPLv2) and (MIT or BSD or GPLv2) rubygem-atk, LGPLv2 rubygem-atomic, ASL 2.0 rubygem-aws-sigv4, ASL 2.0 rubygem-bcrypt, MIT and Public Domain and ISC rubygem-bcrypt_pbkdf, MIT and BSD and ISC rubygem-between_meals, ASL 2.0 rubygem-boxgrinder-build, LGPLv3+ rubygem-boxgrinder-core, LGPLv3+ rubygem-byebug, BSD rubygem-cairo, GPLv2 or Ruby rubygem-cairo-gobject, LGPLv2+ rubygem-clutter, LGPLv2+ rubygem-clutter-gdk, LGPLv2+ rubygem-clutter-gstreamer, LGPLv2+ rubygem-clutter-gtk, LGPLv2+ rubygem-colorize, GPLv2 rubygem-contracts, BSD rubygem-cookiejar, BSD rubygem-creole, GPLv2 or Ruby rubygem-daemons, MIT and (GPLv2+ or Ruby) rubygem-dnsruby, ASL 2.0 rubygem-domain_name, BSD and (MPLv1.1 or GPLv2+ or LGPLv2+) rubygem-elasticsearch-transport, ASL 2.0 rubygem-escape, BSD rubygem-eventmachine, GPLv2 or Ruby rubygem-fattr, BSD or Ruby rubygem-ffi, BSD rubygem-file-tail, ASL 2.0 rubygem-gdk3, LGPLv2+ rubygem-gdk_pixbuf2, LGPLv2 rubygem-gem2rpm, GPLv2+ rubygem-gio2, LGPLv2 rubygem-glib2, LGPLv2 rubygem-gobject-introspection, LGPLv2+ rubygem-goocanvas, LGPLv2 rubygem-gstreamer, LGPLv2 rubygem-gtk2, LGPLv2 rubygem-gtk3, LGPLv2+ rubygem-gtksourceview2, LGPLv2 rubygem-gtksourceview3, LGPLv2+ rubygem-haml, MIT and WTFPL rubygem-hashery, BSD rubygem-hashicorp-checkpoint, MPLv2.0 rubygem-hiera-vault, ASL 2.0 rubygem-highline, GPLv2 or Ruby or BSD rubygem-hiredis, BSD rubygem-hocon, ASL 2.0 rubygem-hpricot, MIT and ASL 2.0 rubygem-hrx, ASL 2.0 rubygem-httpclient, (Ruby or BSD) and Public Domain rubygem-i18n, MIT and (BSD or Ruby) rubygem-icaro, GPLv3 rubygem-idn, ASL 2.0 and LGPLv2+ rubygem-image_size, Ruby or GPLv2 rubygem-imagesize, GPLv2 or Ruby rubygem-jmespath, ASL 2.0 rubygem-json, Ruby or BSD rubygem-levenshtein, GPLv2 rubygem-locale, GPLv2 or Ruby rubygem-lockfile, GPLv2 or Ruby rubygem-log4r, LGPLv3 rubygem-logstash-event, ASL 2.0 rubygem-macaddr, Ruby or BSD rubygem-marcel, MIT and ASL 2.0 rubygem-memcache-client, BSD rubygem-mixlib-cli, ASL 2.0 rubygem-mixlib-config, ASL 2.0 rubygem-mixlib-log, ASL 2.0 rubygem-mixlib-shellout, ASL 2.0 rubygem-mocha, MIT or Ruby or BSD rubygem-mongo, ASL 2.0 rubygem-msgpack, ASL 2.0 rubygem-narray, BSD and Ruby rubygem-native-package-installer,LGPLv3+ rubygem-ncursesw, LGPLv2+ rubygem-net-sftp, MIT or LGPLv2 rubygem-nifti, LGPLv3+ rubygem-nio4r, MIT and (BSD or GPLv2+) rubygem-nokogiri, MIT and ASL 2.0 rubygem-open4, BSD or Ruby rubygem-opennebula, ASL 2.0 rubygem-openscap, GPLv2+ rubygem-pango, LGPLv2 rubygem-pathspec, ASL 2.0 rubygem-pdf-core, GPLv2 or GPLv3 or Ruby rubygem-pdf-inspector, GPLv2 or GPLv3 or Ruby rubygem-pg, (BSD or Ruby) and PostgreSQL rubygem-pkg-config, LGPLv2+ rubygem-poppler, LGPLv2 rubygem-power_assert, Ruby or BSD rubygem-prawn, (GPLv2 or GPLv3 or Ruby) and APAFML rubygem-prawn-icon, Ruby or GPLv2 or GPLv3 rubygem-prawn-table, Ruby or GPLv2 or GPLv3 rubygem-prawn-templates, Ruby or GPLv2 or GPLv3 rubygem-protobuf, MIT and BSD rubygem-public_suffix, MIT and MPLv2.0 rubygem-puppet-resource_api, ASL 2.0 rubygem-rabbit, GPLv2+ and CC-BY rubygem-racc, BSD rubygem-rack, MIT and BSD rubygem-rb-readline, BSD rubygem-rchardet, LGPLv2 rubygem-rdiscount, ASL 1.1 rubygem-rdoc, GPL-2.0 AND Ruby AND BSD-3-Clause AND CC-BY-2.5 AND OFL-1.1-RFN rubygem-rdtool, GPLv2+ or Ruby rubygem-redcarpet, MIT and ISC rubygem-rmail, BSD rubygem-rouge, MIT and BSD rubygem-rsvg2, LGPLv2 rubygem-rubeyond, GPLv3+ rubygem-ruby-dbus, LGPLv2+ and MIT rubygem-ruby-libvirt, LGPLv2+ rubygem-ruby-shadow, Public Domain rubygem-rubyzip, Ruby or BSD rubygem-sass-twitter-bootstrap, ASL 2.0 rubygem-sassc-rails, MIT and OFL rubygem-scanf, BSD rubygem-selenium-webdriver, ASL 2.0 rubygem-semantic_puppet, ASL 2.0 rubygem-serialport, GPLv2 rubygem-sinatra-rabbit, ASL 2.0 rubygem-sqlite3, BSD rubygem-stomp, ASL 2.0 rubygem-sugarjar, ASL 2.0 rubygem-syntax, BSD rubygem-sys-uname, ASL 2.0 rubygem-test-unit, (BSD or Ruby or Python) and (BSD or Ruby) rubygem-test-unit-notify, LGPLv2+ and (LGPLv2+ or GFDL or CC-BY-SA) rubygem-test-unit-rr, LGPLv2+ rubygem-thread_safe, ASL 2.0 and Public Domain rubygem-tk, BSD or Ruby rubygem-ttfunk, GPLv2 or GPLv3 or Ruby rubygem-uglifier, MIT and BSD rubygem-unf, BSD rubygem-uuid, MIT or CC-BY-SA rubygem-uuidtools, ASL 2.0 rubygem-vault, MPLv2.0 rubygem-vte, LGPLv2 rubygem-vte3, LGPLv2+ rubygem-webkit2-gtk, LGPLv2+ rubygem-webrick, Ruby and BSD-2-Clause rubygem-webrobots, BSD rubygem-websocket-driver, ASL 2.0 rubygem-xmlparser, GPLv2+ and ( Ruby or GPLv2+ or MIT ) and ( GPLv2+ or Artistic ) rubygem-xmlrpc, Ruby or BSD rubygem-yard, MIT and (BSD or Ruby) rubygem-zoom, LGPLv2+ ~~~
Thanks, Jarek Prokop
On 11/24/22 18:21, Jarek Prokop wrote:
Hi,
current statistics for specs from 24. 11. are as follows: * Total rubygems in Fedora: 492 * Total rubygems checked: 485 * Fedora License field and gem2rpm license match and license-validate succeeds: 288/485 * license-validate says they are OK SPDX, but licenses may vary: 331/485 * license-validate says they are OK SPDX when conjunctions were converted: 334/485
There were 3 rubygem packages dropped in Fedora, otherwise the stats are the same.
Regards, Jarek
ruby-sig@lists.fedoraproject.org
-
Jarek Prokop