Ability to override tarball URL

[vania-pooh]

Feature: Having an S3 or another reliable store with HTTP protocol it would be great to be able to override host name in tarball URL in package.json.

Use case: Very often S3 or similar storage is a powerful infrastructure with wide network channel and a lot of servers behind load balancer. Being used by frontend developers only private NPM installation could be slightly less powerful. So proxying all NPM download traffic through NPM could potentially slow it down. Possible solution is to replace host name http://my-npm.example.com/ in tarball URL to e.g. http://my-storage.example.com/. With such replacement all network traffic would to go directly to storage host and decrease the traffic through NPM.

Possible implementation: add one more configuration field and update convertDistRemoteToLocalTarballUrls function accordingly.

verdaccio/src/lib/utils.ts

Lines 167 to 178 in 05c9d05

	export function convertDistRemoteToLocalTarballUrls(pkg: Package, req: Request, urlPrefix: string | void): Package {
	for (const ver in pkg.versions) {
	if (Object.prototype.hasOwnProperty.call(pkg.versions, ver)) {
	const distName = pkg.versions[ver].dist;
	if (_.isNull(distName) === false && _.isNull(distName.tarball) === false) {
	distName.tarball = getLocalRegistryTarballUri(distName.tarball, pkg.name, req, urlPrefix);
	}
	}
	}
	return pkg;
	}

[DanielRuf]

override host name in tarball URL in package.json

I guess you mean changing the registry / hostname in package-lock.json?

You can easily do this with some tools like sedor search & replace.

[DanielRuf]

override host name in tarball URL in package.json

s3 does not provide the same URL structure and uses hashed file URLs afaik. I highly doubt that this is possible.

[DanielRuf]

In theory this would only work if http://my-storage.example.com/ also proxies to npmjs because it can not have every single tarball from npmjs (there are many, more than 6TB).

But this would be like another verdaccio instance or a reverse proxy like nginx before verdaccio.

With such replacement all network traffic would to go directly to storage host and decrease the traffic through NPM.

So this is probably not completely true except you sync your verdaccio and storage server (in realtime).

I see more problems this would produce than solve. What do you think @juanpicado?

[juanpicado]

I see more problems this would produce than solve. What do you think @juanpicado?

Seems to me a particular request, so, for such reason we introduced the filter plugin, it is not pretty popular but might do the work.

https://verdaccio.org/docs/en/next/dev-plugins#filter-plugin

If this can be achieved with a plugin, I'm ok with it, but that sort of approach might bring more issues than benefits. But I particularly would not change the core unless there is a public general benefit of it.

[aokhotin]

@DanielRuf What if it try to solve on aws-s3-plugin side? Is it possible to send 302 response code with url pointed to s3 location instead of sending the data of tarball? How do npm clients will process 302 response code?
Is it possible to solve the traffic problem here (https://github.com/verdaccio/verdaccio-aws-s3-storage/blob/master/src/s3PackageManager.ts#L282)?

[DanielRuf]

I'm not really sure tbh.

[juanpicado]

Sounds to me like a workaround, I would insist if the whole idea is mutate the metadata (packument) the ideal would be a filter plugin, which was the initial idea (there are some unit test as prove of concept), worth research of that fills these requirements or might be extended. Somehow I think workaround on plugins only will introduce more complexity and users confusion.

[favoyang]

Possible duplicated with verdaccio/monorepo#250.

I implemented verdaccio/monorepo#320 + #1688 for redirecting tarball, and verdaccio/monorepo#249 for changing ACL on S3.