performance problems with RubyGems and Artifactory

[coldnebo]

we are seeing a massive performance hit when trying to do a bundle install where the source is an Artifactory based repository.

For a Gemfile where one gem changed some minor versions, 'bundle install' recalculation took about 5 seconds. Now it takes 11 minutes.

#292 seems like an explanation from Artifactory's side, but rubygems discounts this as a documentation issue for the compact API.

However, the compact API should be invoked by the latest version of bundler (2.4.13 at this writing) but apparently isn't. Instead two fallback apis are called resulting in 404s, which then trigger the third fallback requesting gemspecs for all gem versions (which takes about 11 minutes on my system).

This is an unacceptable change in performance that seems related to the changes rolled out in the compact API. We are urgently searching for solutions or workarounds to this issue.

references

• Review API documentation on guides.rubygems.org #292
• https://jfrog.atlassian.net/browse/RTFACT-13635
• https://jfrog.com/help/r/artifactory-rubygems-org-dependency-api-deprecation
• https://blog.rubygems.org/2023/02/22/dependency-api-deprecation.html

how the issue presents

• a call to artifactory/api/gems/gems-repos/versions fails with a 404.
• a fallback to artifactory/api/gems/gems-repos/api/v1/dependencies fails with a 404.
• finally, the third fallback initiates a download of all the gemspecs for all gem versions in order to resolve the dep tree locally

$ bundle install --verbose
Running `bundle install --verbose` with bundler 2.4.13
Found changes from the lockfile, re-resolving dependencies because the dependencies in your gemfile changed
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
HTTP 404 Not Found [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
Bundler::Fetcher::FallbackError: Net::HTTPNotFound: [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/versions)
/DONOTDELETE/shared/rvm/gems/ruby-3.1.2@app/gems/bundler-2.4.12/lib/bundler/fetcher/downloader.rb:45:in `fetch'
  /DONOTDELETE/shared/rvm/gems/ruby-3.1.2@app/gems/bundler-2.4.12/lib/bundler/fetcher/compact_index.rb:124:in `call'
...
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/api/v1/dependencies)
HTTP 404 Not Found [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/api/v1/dependencies)
...
Fetching source index from [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/)
HTTP GET [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/quick/Marshal.4.8/amq-protocol-0.8.3.gemspec.rz)
HTTP 200 OK [http://repository/artifactory/api/gems/gems-repo...](http://repository/artifactory/api/gems/gems-repos/quick/Marshal.4.8/amq-protocol-0.8.3.gemspec.rz)
...
# thousands more gemspec requests

environment

$ ruby -v
ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [x86_64-linux]
$ bundle --version
Bundler version 2.4.12
$ curl https://repository/artifactory/api/system/version
{
  "version" : "7.46.11",
  "revision" : "74611900",
  "addons" : [ "ha", "build", "docker", "vagrant", "replication", "filestore", "plugins", "gems", "composer", "npm", "bower", "git-lfs", "nuget", "debian", "opkg", "rpm", "cocoapods", "conan", "vcs", "pypi", "release-bundle", "jf-connect", "jf-event", "replicator", "keys", "alpine", "cargo", "chef", "federated", "git", "observability", "pub", "rest", "swift", "conda", "terraform", "tracker", "license", "puppet", "ldap", "sso", "layouts", "properties", "search", "securityresourceaddon", "filtered-resources", "p2", "watch", "webstart", "support", "xray", "retention" ],
  "license" : "0431f9d1fbae223e0bfa12c8ec1430fb0ca7ec1c3",
  "entitlements" : {
    "EVENT_BASED_PULL_REPLICATION" : true,
    "SMART_REMOTE_TARGET_FOR_EDGE" : false,
    "REPO_REPLICATION" : true,
    "MULTIPUSH_REPLICATION" : true
  }
}

[deivid-rodriguez]

I think this is the same problem being worked on at rubygems/rubygems#6632?

[coldnebo]

I think this is the same problem being worked on at rubygems/rubygems#6632?

I think you might be right. I'll see if the PR fixes this issue on my side with a patch.

[coldnebo]

I don't think there is any way to patch this because of when it is loaded, so I'm waiting for the fix version I guess.

[coldnebo]

we are tracking this through the Artifactory side issue: https://jfrog.atlassian.net/browse/RTFACT-13635

[martinemde]

Hey, just came across this. This is fixed in rubygems. Are you seeing the improvement?

[deivid-rodriguez]

I'm pretty sure this was indeed fixed, let's close! @coldnebo if you're still seeing issues please open a separate issue at https://github.com/rubygems/rubygems.