Choco search not returning all results from Nexus and Artifactory repositories #3270
Comments
|
Can you confirm what you mean by 'part of the result'?
|
|
Hey, Is it consistently 50 packages for example? No, have run a few times and the results change from time to time. Sometimes it returns approx 250 items other in the 300s. Is it different result number each time? Mostly, yeah. I can run this a few times if we need to determine a pattern. Let me know. How many are you expecting? Approx 700 from mem. Are you going through a proxy? No proxy, direct access via PowerShell. |
|
@git-day to try diving into the possible issue, could I ask of you to provide us the response from the servers for the calls that you are making? Either as part of using This will include the information about the URL to the server, so I recommend cleaning those URLs up if you want to keep them private (there will be a lot of them). With that said, I also did compare the queries being used for Chocolatey CLI v1.4.0 and Chocolatey CLI v2.1.0, and except for the use of |
|
@AdmiringWorm pls see attached Fiddler files.
Let me know if you need any further info. |
|
I've the same issue, I'm using chocolatey v2.2.2 and Nexus 3.58.1 Additional info:
|
|
Just upgraded from 1.1.0 thru 1.40 to 2.2.2 and I am having exactly the same problem. Choco search using two sources in Artifactory with both 1.x.x versions producing the list of all the available packages. Choco search using these two exactly same sources in Artifactory with choco 2.2.2 version produces list of only third of the packages that do exist in the source repos. ( I am expecting close to 100 results, only getting 36) Playing with commands: choco search --page=1 and/or choco search --page=2 etc produces different results and I am seeing some packages there that aren't visible in other page search. Also, going thru 2.2.2 release notes and choco search documentation doesn't provide any useful info. This is pretty much useless |
|
After thinking about this, I understand the pagination, which is obviousy good against the default CCR repository which has a lot of packages. But even with pagination you face the challenge to browse the content using --page= -command. You still can browse the entire content of the command choco search even when its not paginated, just scroll using the mouse! Maybe the pagination should be a feature/option to turn on- or off - if user wants it to be in use. |
|
Additional information about pagination. As I already said, also using the pagination, I have missing packages. |
Same here. Also choco search --page=0 produces random results starting with packages in alphabetic order and has random packages not in any alphabetic order (36 results) choco search --page=1 produces again random results starting with packages starting with letter "C" and has random packages not in any alphabetic order (24 results) choco search --page=2 produces 8 packages, listing packages random stuff (atleast these are different packages than in page 1) choco search --page=3 produces 13 packages, listing packages random stuff (also these are different packages than in page 2) choco search --page=4 produces 6 packages, listing packages random stuff (also these are different packages than in page 3) choco search --page=5 produces 7 packages, listing packages random stuff (also these are different packages than in page 4) choco search --page=6 produces 13 packages, listing packages random stuff (also these are different packages than in page 5) etc. Also found out that page 7 search produces an error with a certain package in this page w/ out displaying the erroneous page or list or the packages in this page; just the error text below: All in all, yes - how to use choco search with version 2.2.2 is getting clearer, just wished this was documented somewhere a bit wider. This is cumbersome for users - however as a non-business user I think such handicap is not critical |
|
@tehautanop @misteriosojo Can you provide the same information as requested in this comment? Without that, we won't be able to help you further with this. |
|
@pauby do you know if the fiddler output I supplied was useful? Do you need anything else from me to help solve the problem? |
|
@git-day That will be picked up by @AdmiringWorm as soon as he can. I know he's very busy at the moment, but he'll come back to you if he needs more. |
|
@pauby all good, reach out if you need anything! |
pauby
changed the title
|
@pauby - I'm sorry I cannot run Fiddler in my environment. But basically I have the same situation what @misteriosojo described, with version 2.2.2 |
|
@git-day I am sorry for taking so long to respond. Unfortunately, the fiddler session you provided is not what we are after. You have exported session files that are to be used when replaying entire session with the absolute bare minimum to do so, in Visual Studio (pre version 2022). What we are after are the |
|
We are experiencing the same issue. Request: And even to get this, I had to tinker with Fiddler options (e.g. adding tls1.2 among the protocols under the HTTPS tab). |
|
To give more context, we have 2 repo on Nexus, a Test repo and a Production repo. PS C:\temp\test> $prodPkgs = choco search --source $ProdRepo --all-versions --limit-output | ConvertFrom-Csv -Delimiter '|' -Header Name, Version PS C:\temp\test> $testPkgs = choco search --source $TestRepo --all-versions --limit-output | ConvertFrom-Csv -Delimiter '|' -Header Name, Version PS C:\temp\test> $divergence = Compare-Object -ReferenceObject $testpkgs -DifferenceObject $prodpkgs -Property name, version | Where-Object SideIndicator -EQ '<=' PS C:\temp\test> "In the prod repo there are:$(($prodPkgs | Measure).Count) packages" In the prod repo there are:251 packages PS C:\temp\test> "In the prod repo there are:$(($testPkgs | Measure).Count) packages" In the prod repo there are:1609 packages PS C:\temp\test> "The divergence is $(($divergence | Measure).Count) packages" The divergence is 1480 packages PS C:\temp\test> |
|
@aledeniz if setting up fiddler is causing a problem, then there is also a different alternative (as we are mostly interested in what the response from the servers that are returned). You can also do the following (only valid for Chocolatey CLI 2.x)
Do note these files may include information you may not want the public to know (especially for the production source). EDIT: Or reach out to support through the usual channels if you have a business license available. |
|
@AdmiringWorm I did follow your suggestion, I have sent the logs to Stephen and Chris under ticket 19810. These are the command line that generated the 4 folders in those logs: `PS C:\Windows\system32> ((choco search --source $ProdRepo --limit-output --proxy='http://127.0.0.1:8888' | ConvertFrom-Csv -Delimiter '|' -Header Name, Version)|measure).Count 29 PS C:\Windows\system32> ((choco search --source $ProdRepo --limit-output --all-versions --proxy='http://127.0.0.1:8888' | ConvertFrom-Csv -Delimiter '|' -Header Name, Version)|measure).Count 251 PS C:\Windows\system32> ((choco search --source $TestRepo --limit-output --proxy='http://127.0.0.1:8888' | ConvertFrom-Csv -Delimiter '|' -Header Name, Version)|measure).Count 242 PS C:\Windows\system32> ((choco search --source $TestRepo --limit-output --all-versions --proxy='http://127.0.0.1:8888' | ConvertFrom-Csv -Delimiter '|' -Header Name, Version)|measure).Count 1600` |
|
@aledeniz Thank you, I have just now had a look at this response. With this, we have identified where the problem is (at least for Nexus), but have yet to determine the proper fix for these situations. I cannot share anything more at this moment, but @pauby may share more information when ready. Additionally, while this identifies the issue when Nexus is used as the source, it would be good if someone who uses Artifactory, or other providers and sees similar issues could provide the same responses from the servers through fiddler. |
Just to clarify that I am observing this behaviour also on the licensed version (2.0.0 to 2.2.2). |
|
Any news on this one? We are facing the same issue. |
|
We investigated this issue with Sonatype Nexus and JFrog Artifactory using v2 (OData) repositories and we identified an error in the responses coming back from those repository managers. Ordinarily we'd suggest raising issues with the repository managers directly. However, with NuGet v2 repository support being reduced / deprecated in Nexus and Artifactory, there's unlikely to be much traction in those being fixed. During the investigations we looked at the responses for NuGet v3 repositories with Chocolatey CLI and can confirm that we didn't find the same issues. Our suggestion would be to use Chocolatey CLI with NuGet v3 repositories when using Sonatype Nexus or JFrog Artifactory, if you are seeing these issues. There is some work for us to do around this and I've raised an issue to ensure each area is picked up. |
What is the suggested path for users on Nexus that are proxying CCR? As Nexus can't mix v2 and v3 feeds when a proxy feed is in the mix. |
|
That would be a better question to ask over on the doc repository issue I linked to so we can ensure we document that. |
|
@pauby I've created a new repo using nuget v3. When I execute the following command: choco search --source https://nexus.local/repository/Choco-group-v3/ I get the following error: Failed to fetch results from V2 feed at 'https://nexus.local/repository/Choco-group-v3/Packages()?$filter=((((Id%20ne%20null)%20and%20substringof('',tolower(Id)))%20or%20((Description%20ne%20null)%20and%20substringof('',tolower(Description))))%20or%20((Tags%20ne%20null)%20and%20substringof('%20%20',tolower(Tags))))%20and%20IsLatestVersion&$orderby=Id&$skip=0&$top=30&semVerLevel=2.0.0' with following message : Response status code does not indicate success: 502 (The group repository Choco-group-v3 contains a proxy with different nuget version).
|
|
@git-day the error indicates you're using a repository group containing both NuGet v2 and v3 sources which Nexus does not support. We documented this in our upgrade guide. My understanding is that a NuGet v2 repository, on an up-to-date version of Nexus, also works as a NuGet v3 repository without any changes in Nexus itself. All you need to do is add Just to be transparent, I'm writing this on my phone and I have no links to point you to that confirms this. If you find any, please add them here and I'll do the same, and update this comment if necessary, when I'm back at my desk on Monday. But I hope that at least helps. |
|
@pauby perfect. The addition of /index.json worked. |
|
Awesome @git-day. Hopefully that helps others. It does help us add that to any documentation with confidence it's been tested and confirmed! I will also look to update the upgrade documentation with that to. |
|
@pauby, thanks to you and your team for finding a solution. I have now tried to implement this against the new NuGet version 3, but am getting some odd output, and in some instances no output. Not sure if this is related to the “Choco search” problem originally identified or if it is a new issue. I’ll do my best to highlight what I have done. If something doesn’t make sense, let me know. Current Setup
V2 repo.
V3 repo.
Let me know if I can help with debug output or further testing. |
|
Based on the information in this issue I started to add /index.json to my Nexus source. After this all packages are returned when executing a chocolatey search. But I just found out that when /index.json is added to my source that dependencies in my packages are completely ignored. I am not sure if the addition of /index.json will cause any other odd behavior but not not honouring the dependencies as specified in the .nuspec is certainly an issue for my situation. |
|
the dependencies issues looks like a specific chocolatey issues with nuget v3 feeds. |
There isn't enough information here to do anything with this. Can you raise a new issue for this and provide all the information requested? Can you also ensure your dependencies are available on the v3 feed. |
|
I will go ahead and do that. I do have one more question regarding the Choco vs Nexus issue. I did some tests and it looks like the issue (not returning all results) is related to Grouped Repo's. When I connect Choco to one ore several individual Nexus repo instead of a Group all results are returned according to my tests. Would connecting Choco to multiple individual Nexus repo's (Choco Sources) be a feasible workaround for now or would I introduce other issue's that I do not oversee right now? thanks again, Martijn |
|
That's interesting that it only occurs on grouped repositories. Chocolatey CLI just queries a URL endpoint and doesn't have any special handling for grouped or single repositories or even know the difference. It just interprets the results of the query. With this limited information, this looks like a Nexus issue. Adding each individual repository is feasible. It may not be practical depending on how many there are. Chocolatey CLI has to query each of them, so there will be a difference between 3 and 30. But YMMV so it is something I'd try and see how well it works. |
|
Thanks for the response, in the mean time I already started migrating all the endpoints to use the individual repo's instead of the group. The groups we have only contain two repo's so we should be fine regarding the queries Choco needs to do. I would also like to take the opportunity to warn people not to go the /index.json route. I have seen the most odd behavior with this. Dependencies not working is one of them. But the one that gave me the bigest headache is the fact that after configuring a repo with /index.json every choco command can trigger an reinstall or forced upgrade of all the installed packages, ultimately leading to broken installations and packages ending up in the lib-bad. So be warned ;) |
|
@martinisoft1 Are you a Chocolatey for Business customer or are you using open-source? |
|
Open-source |
|
I have been having a chat internally. Once I have some public links, I'll be able to share more.
Do you have logs you can share for this? And just to confirm, you've only tested this with Chocolatey CLI and Nexus single and grouped repositories?
Are there logs you can share for this? Just to be clear, I have never seen this behaviour ever. If you want to share the logs in private, could you please reach out to me ( |
|
There are more Fiddler logs available on customer ticket #20460 |
|
Recently I noticed something... I think this issue happens only when there is a package which has only a "pre" release.
IMO this issue should be corrected by nexus, because if we have a repo with ~50 "pre" packages, then we cannot expect that chocolatey tries to query for next pages until it finds something new. The artifact repository should return always the requested amount of items. If "pre releases" are not requested, it should simply skip those packages and return the expected amount of items regardless of only "pre" versions available or not. |
|
Before I start, I'd like to apologise for the delay on this. We have put together our findings in this area and I will share highlights shortly (as in within a week). The intention was to publish them before now, but there is always something else that needs attention and the priority on this slipped. I've removed the |
|
@pauby all good Paul! Thanks for your support! Note that I have since moved away from the 'Grouped' option with Nexus and have gone direct to the repo using v3. All is working as expected. |
|
Following on from my comment above, we have completed the review again after the recent major change to Nexus, and they are currently being peer-reviewed. My apologies again for not updating the issue when I said I would! |
|
I wanted to provide an update here on this again and to confirm we have not forgotten about this. I won't offer any reasons or excuses, or promises of dates, as it's a bit late for that, I think. The team has something prepared that we are continuing to review to ensure it answers the questions asked, and I want to publish it in January, likely as a blog post. I finally would like to clarify that I am the bottleneck for this, nobody else. |
Checklist
What You Are Seeing?
Using "choco search --source $LocalRepo --limit-output" under chocolatey 1.4.0 returns all expected packages, whilst chocolatey 2.1.0 seems to return only part of the result.
What is Expected?
Expecting to see all returned packages that are queried using the following command line.
Note that the "choco list" worked pre 2.1.0 release as well.
How Did You Get This To Happen?
Running the following command against a Nexus repo results in a part results returned.
"choco search --source $LocalRepo --limit-output"
System Details
PS C:> [System.Environment]::OSVersion.version.tostring()
10.0.22621.0
PS C:> $PSVersionTable
Name Value
PSVersion 7.3.6
PSEdition Core
GitCommitId 7.3.6
OS Microsoft Windows 10.0.22621
Platform Win32NT
PSCompatibleVersions {1.0, 2.0, 3.0, 4.0…}
PSRemotingProtocolVersion 2.3
SerializationVersion 1.1.0.1
WSManStackVersion 3.0
PS C:> choco --version
2.1.0
Installed Packages
Output Log
Additional Context
No response
The text was updated successfully, but these errors were encountered: