Hi and very much thank you for looking into this issue.

Dne 10. 08. 23 v 18:24 Jarek Prokop napsal(a):

Hi,

I recently had an idea of faking load paths [0] to satisfy bundler's loading and help bring our behavior closer to upstream while still unbundling the libraries.

To automatically test for the usual user's workflow with bundler and a default gem (only RDoc so far),
I have created small TMT test suite [1]. It pulls the Ruby RPMs from my copr repo https://copr.fedorainfracloud.org/coprs/jackorp/ruby-rdoc-default-gem/ .
It is adjustable to fit your own copr if you wish to do your own changes [2], TMT can also install RPMs from a local directory, but that's going off-topic.

One of my TODO item here is to validate this against some rbenv Ruby install.

To the loading itself and my examination.
Default gems seem to skip bundler somewhere and go their own path which is more similar to regular `Kernel.require` without bundler.
Therefore, just having it on the default load path, where default gems are expected to exist is enough.
However, regular Ruby loading also means that it expects a certain layout that it won't search for itself if it can't find a subfile, see Rake and `rdoc/task` below.

So, if we place `rdoc.rb` into `/usr/share/ruby/` that adjusts the load path accordingly and then just passes on the require, that's enough if the code is either
~~~
$ cat foo.rb
require 'rdoc'
p RDoc::VERSION # Demonstrate that RDoc is available
~~~
and run with bundler `bundle exec ruby ./foo.rb`.

or in CLI like this: `bundle exec ruby -rrdoc -e 'p RDoc::VERSION'`.

In both cases, the associated Gemfile is empty. No gems are recored in the file.

My approach, at this point in time, falls apart with Rake and RDoc using `require rdoc/task`, as there is no /usr/share/ruby/rdoc/task.rb. I am unable to track down the exact code path,
but it seems to skip some places inside bundler, even rubygems, that could otherwise make it possible to work (with a few minor adjustments of course).


At this place it is necessary to understand what Bundler actually does. Please note that it is possible to run not only `bundle exec rdoc`, but it also is possible to run `bundle exec bash` and there are several interesting environment variables injected by Bundler:

~~~

$ bundle exec bash

[mockbuild@fbbdeb5a66084b7eae24c24f33923368 ~]$ env | sort
BUNDLER_ORIG_BUNDLER_SETUP=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLER_VERSION=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_BIN_PATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_BUNDLE_GEMFILE=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_GEM_HOME=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_GEM_PATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_MANPATH=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/sbin
BUNDLER_ORIG_RB_USER_INSTALL=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYLIB=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_ORIG_RUBYOPT=BUNDLER_ENVIRONMENT_PRESERVER_INTENTIONALLY_NIL
BUNDLER_SETUP=/usr/share/gems/gems/bundler-2.4.10/lib/bundler/setup
BUNDLER_VERSION=2.4.10
BUNDLE_BIN_PATH=/usr/share/gems/gems/bundler-2.4.10/libexec/bundle
BUNDLE_GEMFILE=/builddir/Gemfile


... snip ...


RUBYLIB=/usr/share/gems/gems/bundler-2.4.10/lib
RUBYOPT=-r/usr/share/gems/gems/bundler-2.4.10/lib/bundler/setup


... snip ...

~~~


As you can see, Bundler places itself on Ruby `$LOAD_PATH` via `$RUBYLIB` and it loads itself via `$RUBYOPT`. Now should also be clear, that you actually don't have to run `bundle exec`, it is enough to setup the right env variables.

If I am not mistaken, historically Bundler already at this stage injected the load paths via `RUBYLIB`. Apparently, it does not doing so anymore (I think that is probably because they are handling too many corner cases).

Anyway, what `bundler/setup` does at this stage is that it populates the Ruby `$LOAD_PATH` (and loads the libraries, unless there is `:require => false`, but this is not really important). As a last step, Bundler actually unloads the customized `require` method and resorts back to the plain Ruby (!!!) version of require. Not even the RubyGems loader is active. Nothing. So what is not on load path is not loaded.

These are references to the key steps:

https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd66fe786c0d/bundler/lib/bundler/runtime.rb#L21

https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd66fe786c0d/bundler/lib/bundler/rubygems_integration.rb#L389


This demostrated a problem, we'd have to fake out the whole structure of any gem we'd like to unbundle this way.
That, of course, is a possibility.


Given the above, I think this is necessity.


So the pros / cons of this approach currently seem:
+ we'd unbundle default gems away
+ no more symlinks.
+ closer to upstream behavior for the users
- We'd have to retain the structure and provide each "fake" path to a file.
  * Though we are doing something a small bit similar with the symlinks.
- added complexity of writing correct paths into the files on their correct places.


One cons is that still, the `$LOAD_PATH` with this approach will be different to what would be the default.


Alternatively, this is just an academic idea, populate the default load path with default gems.
I, however, do not know where would the right place for that be, or what would be the cons of this approach
(apart from potentially breaking expectations set accross the whole Ruby environment for loadables).


Not sure about this. We would need to ensure that the paths has lower priority in `$LOAD_PATH`.


Also to adjust the Kernel.require, but that feels very wrong (also potentially unsafe, as the code is in C AFAICT, and very divergent from upstream).


Just given the two links above, this is asking for even more troubles.


But I have quickly dismissed these as I have high doubts such patches would ever be accepted upstream and maintainability of such approaches is quetionable at best.

Additionally, I am not sure what should be the behaviour with the plain rdoc executable and bundler executed like this: `bundle exec rdoc`


From the `env` output you can see, that at minimum, Bundler preserves some of the original env variables, such as `BUNDLER_ORIG_PATH`, because it obviously can modify `$PATH` env variable. It can also influence `MANPATH`:

https://github.com/rubygems/rubygems/blob/556349fd863ba586930804140fbcdd66fe786c0d/bundler/lib/bundler/runtime.rb#L33

I was not aware of this behavior and it seems to complicate the matter :/

BTW you are using `find` and what not to create the stubs, but it could be possible to use the already built Ruby and use them to e.g. evaluate the .gemspec files to get the list of gem files. Or even try to identify the unbundled default gems.


Vít



I will try to apply the approach of faking the whole gem directory and file tree in /usr/share/ruby for RDoc and record my findings.

I am open to discussion, should you want to share your thoughts or ideas regarding this topic.

Regards,
Jarek Prokop

[0] https://src.fedoraproject.org/fork/jackorp/rpms/ruby/c/1f660500090ad6d176a98a2e0883d69810b98da8?branch=rdoc_require_stub
[1] https://gitlab.com/jackorp/test_default_gem_fedora_ruby
[2] https://gitlab.com/jackorp/test_default_gem_fedora_ruby/-/blob/76581956440fc465dd37c13f62d114b18943b901/plans/main.fmf#L7


_______________________________________________
ruby-sig mailing list -- ruby-sig@lists.fedoraproject.org
To unsubscribe send an email to ruby-sig-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/ruby-sig@lists.fedoraproject.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue