-
Notifications
You must be signed in to change notification settings - Fork 122
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Alldebrid puts files in silly places sometimes, causing rdt-client to be unable to find them #648
Alldebrid puts files in silly places sometimes, causing rdt-client to be unable to find them #648
Comments
I have the same problem! It can be cool to have a solution for that. |
@rogerfar that PR does not fully close this issue - the single file torrents are still a problem. I can open a new issue if you'd rather they be separate. |
It must've closed this automatically when I merged the PR. @Sculas is this something you could have a look at? |
If I understand you correctly, @Cucumberrbob, is this still the behavior after my PR? I don't see a reason at all why RDT would check the |
I just tried and this work now if I put /mnt/rclone/magnets/*/ in the settings |
Wait but, so now it's |
This one magnet:?xt=urn:btih:73609fad5cef86743734fcf242cbffe69edb3a2f is for this one magnet:?xt=urn:btih:4c39706481b8887a5f1e08347c273d87debf30d0 is for |
Oh, that's interesting. A torrent only allows a single file or a directory with multiple files. It seems AllDebrid makes you blindly follow that assumption because the API doesn't show the root folder if it has one. After much reading, I think I've been able to find the root cause: rdt-client/server/RdtClient.Service/Helpers/DownloadHelper.cs Lines 59 to 99 in 79164d5
This function always adds the torrent filename, which is actually the root directory of the torrent, to the file path. @rogerfar Do you know if this behavior also applies to RealDebrid? Does it place single-file torrents in a directory or not? |
I'm testing a bit and I'm not sure if this is what we're trying to accomplish. I attached a couple of test torrents I uploaded myself. When I download them with qBittorrent, I get these results with
Before your patch I get:
After your patch it's the same. This isn't really good behaviour anyway as I want to match the output of RDT similar to qBittorrent. If you run these test torrents in AllDebrid, how do they end up? |
That was indeed my intention as well, to have the same output as qBittorrent. I will test the torrents you've provided tomorrow! (currently night for me) |
Since the last version 2.0.94 everything is working pretty well! |
I found 1 snag, this is the Webdav drive from Real-Debrid when adding those test torrents:
It seems it never places it in the root, even when it's a single file, and it ignores my sub-directory. Which seems to not matter, the symlink is what matters. |
I added some code in my latest commit if you want to do a snifftest. I would like to see the results from AllDebrid and TorBox too to compare. |
Okay, so RealDebrid always stores it in a directory. AllDebrid doesn't. That's frustrating since we'll have to handle that.
It does matter, though? The issue is that RDT always assumes the files are in a directory, which is the case for RealDebrid, but not for AllDebrid, and then tries to access the file from the WebDAV drive at a place it thinks it exists, but doesn't. See it like this. Let's say we're downloading
Now, let's download
Let's use
|
With those two torrents downloaded, I currently have this in
I don't have the WebDAV drive mounted so I can't show a tree output, but I've checked and it's the same as above. |
That's really weird, because I would expect that file to appear at {
"data": {
"magnets": {
"filename": "Name Mismatch (Single File)",
"links": [
{
"filename": "single-file.txt",
"files": [
{
"n": "Name Mismatch (Single File)",
"e": [
{
"n": "single-file.txt",
}
]
}
],
}
],
}
},
} I can confirm this is still not found by the latest version ( I have at least one single-file torrent that is even weirder: its {
"data": {
"magnets": {
"filename": "A",
"links": [
{
"filename": "B.mkv",
"files": [
{
"n": "B.mkv",
}
],
}
],
}
},
} And is available at |
Also worth noting that the {
"data": {
"magnets": {
"filename": "TestFileNameMismatch.txt", // <-- same as links[0].files[0].n or filename, doesn't demonstrate mismatch
"links": [
{
"filename": "TestFileNameMismatch.txt",
"files": [
{
"n": "TestFileNameMismatch.txt"
}
]
}
]
}
}
} But {
"status": "success",
"data": {
"magnets": {
"filename": "TestSingleInSub", // <-- not the same as links[0].files[0].e[0].e[0].n or filename
"links": [
{
"filename": "TestSingleInSub.txt",
"files": [
{
"n": "TestSingleInSub",
"e": [
{
"n": "TestSingleInSub",
"e": [
{
"n": "TestSingleInSub.txt",
}
]
}
]
}
]
}
]
}
}
} |
AllDebrid's way of placing files works exactly like qBittorrent, with As for the torrent you provided, the WebDAV server reports that the file is at So why would you expect this file to be at |
I think we might be misunderstanding each other. I'm not making any assertions as to where files should end up in /data/downloads. I'm saying that when a torrent like But I disagree with your first sentence:
For |
The version 2.0.94 of yesterday is good with Alldebrid. No problem to create symlinks anymore |
Oh wait, are we experiencing the same issue then? I've just been working around it since I thought it was a weird issue only I had. As you can see, the torrents aren't directories for some reason, but instead something... weird? It sounds wrong that the file is placed at Actually, I have a Linux VM now to test this with, so let me mount it via rclone as well and see if I get the same issue as you. |
Yes! spot on! These files are actually files not directories despite how they're named, |
I'm using TestMultiple:Stored as: {
"status": "success",
"data": {
"magnets": {
"filename": "TestMultiple",
"size": 74,
"nbLinks": 2,
"files": [
{
"n": "TestMultiple",
"e": [
{
"n": "TestMultiple2.txt",
"s": 37,
"l": "<>"
},
{
"n": "TestMultiple.txt",
"s": 37,
"l": "<>"
}
]
}
]
}
}
} Sword Art Online:Stored as: {
"status": "success",
"data": {
"magnets": {
"filename": "[Tenrai-Sensei] Sword Art Online [BD][1080p][HEVC 10bit x265][Dual Audio]",
"size": 58880130060,
"nbLinks": 159,
"files": [
{
"n": "Season 1",
"e": [
{
"n": "Sword Art Online - S01E01 - The World Of Swords.mkv",
"s": 489401205,
"l": "<>"
}
]
},
{
"n": "poster.jpg",
"s": 494325,
"l": "<>"
}
]
}
}
} TestSingleInSub:Stored as: {
"status": "success",
"data": {
"magnets": {
"filename": "TestSingleInSub",
"size": 86,
"nbLinks": 1,
"files": [
{
"n": "TestSingleInSub",
"e": [
{
"n": "TestSingleInSub",
"e": [
{
"n": "TestSingleInSub.txt",
"s": 86,
"l": "<>"
}
]
}
]
}
]
}
}
} |
Yep, these are all constructible from the responses: For TestMultiple, our file nodes (object with
|
Yeah. It's confusing then why qBittorrent shows something different. Oh well. rdt-client/server/RdtClient.Service/Helpers/DownloadHelper.cs Lines 63 to 73 in ee9e80f
I think I'll have to unfortunately leave it at this though. This is something AllDebrid has to fix on their end if we ever want to see this issue closed unless there's some kind of workaround I'm not aware of at the moment. In either case, this doesn't seem to be an issue in RDT. |
My rclone parameters are
I have mine running as a systemd mount, so it's in daemon mode, but this shouldn't matter
I think you're pretty much right - without AD fixing their webDAV some of the single file torrents will be unplayable. It's worth looking a bit more to see if this is consistently predictable based on the |
old comment (ignore)Oh, it is predictable. If we fall into the block where we set the rdt-client/server/RdtClient.Service/Helpers/DownloadHelper.cs Lines 63 to 73 in ee9e80f
I'm currently reading the verbose logs of WinSCP to figure out what it's doing. I'll update this message as I go.
WinSCP knows it's a directory...? The WebDAV response from SabreDAV looks correct: SabreDAV Response<d:multistatus
xmlns:d="DAV:"
xmlns:s="http://sabredav.org/ns">
<d:response>
<d:href>/magnets/Name%20Mismatch%20(Single%20File)/</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/Name%20Mismatch%20(Single%20File)/single-file.txt</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
<d:getcontentlength>16</d:getcontentlength>
<d:resourcetype/>
<d:getetag>"cfcd208495d565ef66e7dff9f98764da"</d:getetag>
<d:getcontenttype>text/plain</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus> Okay, that response seems to be later. Not sure why WinSCP suddenly figures out it's a directory, that's next to figure out. SabreDAV Response<d:multistatus
xmlns:d="DAV:"
xmlns:s="http://sabredav.org/ns">
<d:response>
<d:href>/magnets/</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 15:10:37 GMT</d:getlastmodified>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/Name%20Mismatch%20(Single%20File)</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
<d:getcontentlength>16</d:getcontentlength>
<d:resourcetype/>
<d:getetag>"f0555d9391cbc8e3203080cfff2900a9"</d:getetag>
<d:getcontenttype>application/octet-stream</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/TestSingleInSub</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 11:28:29 GMT</d:getlastmodified>
<d:getcontentlength>86</d:getcontentlength>
<d:resourcetype/>
<d:getetag>"e1d4fb2e024f90822d69113ccd844a94"</d:getetag>
<d:getcontenttype>application/octet-stream</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/TestSingle.txt</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 12:30:47 GMT</d:getlastmodified>
<d:getcontentlength>36</d:getcontentlength>
<d:resourcetype/>
<d:getetag>"7b2ff460e337db1199b514fc666e7c7a"</d:getetag>
<d:getcontenttype>text/plain</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/TestMultiple/</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 13:29:24 GMT</d:getlastmodified>
<d:resourcetype>
<d:collection/>
</d:resourcetype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
<d:response>
<d:href>/magnets/TestFileNameMismatch.txt</d:href>
<d:propstat>
<d:prop>
<d:getlastmodified>Mon, 06 Jan 2025 10:42:46 GMT</d:getlastmodified>
<d:getcontentlength>77</d:getcontentlength>
<d:resourcetype/>
<d:getetag>"2501e66fa1e86f4f6a17a6bf80dbfc31"</d:getetag>
<d:getcontenttype>text/plain</d:getcontenttype>
</d:prop>
<d:status>HTTP/1.1 200 OK</d:status>
</d:propstat>
</d:response>
</d:multistatus> Now here's the interesting part, the broken directories return |
{
"data": {
"magnets": {
"filename": "A",
"links": [
{
"filename": "B.mkv",
"files": [
{
"n": "B.mkv",
}
],
}
],
}
},
} For all the torrents I have which have responses like this, |
No. In your example the file will be stored at Here's an example that will cause the bug: {
"status": "success",
"data": {
"magnets": {
"filename": "A",
"files": [
{
"n": "B",
"e": {
"n": "C.mkv"
}
}
]
}
}
} This file should be stored at |
I don't think this is true. For all the single file torrents I've posted in here, none of them follow that rule. They are all always at But if the bug is just for single file torrents where the file is in a directory, it might be OK just to error out and say we can't download the files. If my library is representative, 0.2% error rate isn't terrible - but even if it's not, those torrents would have errored (less helpfully) anyway |
You're right, it is stored that way. So then why does that mkv file work, but |
They're not quite identical - the |
You're right, qBittorrent's seemingly inconsistent (or AllDebrid's) file listing got me again. Why can these things just not be consistent with each other... Now, why is AllDebrid missing the file extension on the file... |
Sorry it's been night here so haven't fully caught up. But it might be worth it to make a list of all potential iterations and create test torrents for those, because I also see the same discrepancies with Real-Debrid too. qBittorrent seems to have some naming rules itself too which conflicts with expectation and the output of the debrid providers. |
Honestly I think it's AD again - the first bit of any .torrent file is pretty much human readable if you
Definitely, but I can't figure out how to create torrents of the working kind - where they appear like this in the But between that nyaa one and the |
The TLDR pretty much is that this is an AllDebrid issue:
|
To clarify, this is only slightly true. This only works because
|
To sum this all up into something actionable: The logic should be as follows:
If it's not clear what I mean by "all the 'n's ", I mean all the @Sculas can you spot an issue with this logic? |
That looks correct! I'm not sure what's the best way to report this to the Alldebrid developers. Just asking them to test the Nyaa torrent you sent earlier @Cucumberrbob is enough to show something's definitely wrong (missing file extension and broken content/mime type). And hopefully, with that, we can also tell them about the issue with the broken directories (which may be harder to figure out). Their GitHub profile hasn't seen activity in quite some time, unfortunately. I'll send an email to their support. I may otherwise consider forking rclone_rd and porting it to Alldebrid if I also encounter this issue myself. I'm not using the rclone mount setup right now, but I'm working on switching to it. I'll have to see though, since I've already spent hours trying to fix this. |
I've sent an email to their support mail ( |
Just got a response from AD support - they've said they've found the issue and "will fix it most likely tomorrow". Let's see what we end up with then. |
Oh wow, that's a quick response! I guess you used the website? I'll take a look at your PR and continue there :) |
Yeah! I won't lie I'm impressed. I did use the contact form on their website. With regards to the PR, given that the single-file torrent situation will likely change in the near future, only the changes in |
Yeah, I was about to mention that in the PR, but we may want to wait with my PR until we know it's fixed. Otherwise, we push out a release now that works around a part of the bug, but then tomorrow it's fixed for example, and we have to fix it again to remove the workaround. |
Just heard back from AD (they're insanely quick). As I understand it, the logic is now as follows:
|
I've been checking occasionally and am happy to report that it's just been fixed! I can finally access the directories again.
Perfect! |
On your PR (ee9e80f) with my changes to the Dockerfile made, a few are still broken, I'll keep updating these as I get the logs through TestSingleInSub [not working]
TestSingle.txt [not working]
TestMultiple [not working]
Notably in every case, the All single-file-at-the-root torrents work now, since they're found by searching for |
In the setting of RDclient, did you put /*/ |
I suppose that could work, but since the path in |
No - I've intentionally left that out. That really feels like a workaround to deal with the reasonably common case of torrents containing one level of directories. It doesn't actually solve the problem of torrents with directories in them unless it's only one directory deep.
|
I've fixed the symlink search by adding the |
What version are you using?
docker 2541993a2fe6
What OS are you running?
NixOS
Are you using Docker or as a service?
Docker
Which debrid provider are you using?
Alldebrid
Which downloader are you using?
Symlink
Please attach a log file here with the log setting set to debug
See below for excerpts where relevant
Edit: [Note] AD changed how single file torrents are stored in the webDAV drive on 2025-01-07T12:30:00Z ish - many comments about single-file torrents are no longer true
Variables in
<>
referenced below are from thedata.magnets
field in a response from a/api/v4/magnet/status
request.Alldebrid is doing some weird stuff with their webdav. I've identified a few cases where
rdt-client
won't pick up the files. Below are the cases I've identified/mnt-path/magnets/<filename>
(notably, notlinks[0].filename
)./mnt-path/magnets/<filename>/<links[I].files[0].n]>/<links[0].files[0].e[0].n>
.As an example of the first case: say
<filename> = "A"
and<links[0]> = {"filename": "B.mkv", "files": [{"n": "B.mkv"}]}
Then the file is at/mnt-path/magnets/A
(no extension). Yes this is really an mkv file despite having no extension. . The logs for the files checked are as followsFor the second case, say
<filename> = "A"
, and there are filesC.mkv
andD.mkv
in the subdirectory B. Then the files are at/mnt-path/magnets/A/B/C.mkv
and/mnt-path/magnets/A/B/D.mkv
. The logs for the files checked are the same as above.The
data.magnets
in the response from/api/v4/magnet/status
looks something likewhich, when constructed as above , does produce the correct filename. This construction also works for torrents which do not contain subdirectories, just that
<links[0].files[0]>
is{ "n": "B.mkv"}
, with noe
, so the correct filename is/mnt-path/magnets/<filename>/<links[I].files[0].n]>
.However, this construction fails for single file torrents, in a similar way to the current setup: most single file torrents are named the same as the file they contain, so it's fine most of the time, but if the
filename
is not the same aslinks[0].n
, the construction fails. . For single file torrents,/mnt-path/magnets/<filename>
is always the correct path.Effectively, I think the logic is that if there is one file in the torrent (even if it is in a subdirectory), it appears at
/mnt-path/magnets/<filename>
, and if there is more than one file, each file's path is/mnt-path/magnets/<filename>/<all the
"n"s of its parent>/<its "n">
The text was updated successfully, but these errors were encountered: