Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Alldebrid puts files in silly places sometimes, causing rdt-client to be unable to find them #648

Open
Cucumberrbob opened this issue Dec 31, 2024 · 59 comments · Fixed by #650 or #651 · May be fixed by #652
Open

Alldebrid puts files in silly places sometimes, causing rdt-client to be unable to find them #648

Cucumberrbob opened this issue Dec 31, 2024 · 59 comments · Fixed by #650 or #651 · May be fixed by #652

Comments

@Cucumberrbob
Copy link

Cucumberrbob commented Dec 31, 2024

What version are you using?
docker 2541993a2fe6

What OS are you running?
NixOS

Are you using Docker or as a service?
Docker

Which debrid provider are you using?
Alldebrid

Which downloader are you using?
Symlink

Please attach a log file here with the log setting set to debug
See below for excerpts where relevant

Edit: [Note] AD changed how single file torrents are stored in the webDAV drive on 2025-01-07T12:30:00Z ish - many comments about single-file torrents are no longer true

Variables in <> referenced below are from the data.magnets field in a response from a /api/v4/magnet/status request.

Alldebrid is doing some weird stuff with their webdav. I've identified a few cases where rdt-client won't pick up the files. Below are the cases I've identified

  • Single file torrent whose name does not end in a file extension, but the file inside does: The file is available at /mnt-path/magnets/<filename> (notably, not links[0].filename).
  • Torrents containing a subdirectory. Say there was a file in this subdirectory, then it would be available at /mnt-path/magnets/<filename>/<links[I].files[0].n]>/<links[0].files[0].e[0].n>.

As an example of the first case: say <filename> = "A" and <links[0]> = {"filename": "B.mkv", "files": [{"n": "B.mkv"}]} Then the file is at /mnt-path/magnets/A (no extension). Yes this is really an mkv file despite having no extension. . The logs for the files checked are as follows

[DBG] Searching /mnt/rclone/magnets for B.mkv (attempt #3)...
[DBG] Searching /mnt/rclone/magnets/A/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B.mkv/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B.mkv...

For the second case, say <filename> = "A", and there are files C.mkv and D.mkv in the subdirectory B. Then the files are at /mnt-path/magnets/A/B/C.mkv and /mnt-path/magnets/A/B/D.mkv. The logs for the files checked are the same as above.

The data.magnets in the response from /api/v4/magnet/status looks something like

{
  "filename": "A",
  "links": [
    {
      "filename": "C.mkv",
      "files": [{
        "n": "B",
        "e": [{
          "n": "C.mkv"
        }]
      }]
    },
   {
      "filename": "D.mkv",
      "files": [{
        "n": "B",
        "e": [{
          "n": "D.mkv"
        }]
      }]
    }
  ]
}

which, when constructed as above , does produce the correct filename. This construction also works for torrents which do not contain subdirectories, just that <links[0].files[0]> is { "n": "B.mkv"}, with no e, so the correct filename is /mnt-path/magnets/<filename>/<links[I].files[0].n]>.
However, this construction fails for single file torrents, in a similar way to the current setup: most single file torrents are named the same as the file they contain, so it's fine most of the time, but if the filename is not the same as links[0].n, the construction fails. . For single file torrents, /mnt-path/magnets/<filename> is always the correct path.

Effectively, I think the logic is that if there is one file in the torrent (even if it is in a subdirectory), it appears at /mnt-path/magnets/<filename>, and if there is more than one file, each file's path is /mnt-path/magnets/<filename>/<all the "n"s of its parent>/<its "n">

@bob1321
Copy link

bob1321 commented Jan 3, 2025

I have the same problem! It can be cool to have a solution for that.

@Cucumberrbob
Copy link
Author

@rogerfar that PR does not fully close this issue - the single file torrents are still a problem. I can open a new issue if you'd rather they be separate.

@rogerfar
Copy link
Owner

rogerfar commented Jan 5, 2025

It must've closed this automatically when I merged the PR.

@Sculas is this something you could have a look at?

@rogerfar rogerfar reopened this Jan 5, 2025
@Sculas
Copy link
Contributor

Sculas commented Jan 5, 2025

As an example of the first case: say <filename> = "A" and <links[0]> = {"filename": "B.mkv", "files": [{"n": "B.mkv"}]} Then the file is at /mnt-path/magnets/A (no extension). Yes this is really an mkv file despite having no extension. The logs for the files checked are as follows

[DBG] Searching /mnt/rclone/magnets for B.mkv (attempt #3)...
[DBG] Searching /mnt/rclone/magnets/A/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B.mkv/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B/B.mkv...
[DBG] Searching /mnt/rclone/magnets/B.mkv...

If I understand you correctly, A is the name of the torrent, and B.mkv is the only file in it.
This means the file should be at /mnt/rclone/magnets/B.mkv. I've confirmed this by downloading a torrent from Nyaa that had a different name than the single file in it. Connecting to the WebDAV server via WinSCP showed the file with a different name in the magnets directory, as I expected. Requesting the torrent via Alldebrid's API shows a single file with a different name. RDT does the same, it sees the file and then sends that directly to the downloader. This should all just work as expected?

@Cucumberrbob, is this still the behavior after my PR? I don't see a reason at all why RDT would check the A directory here.
RDT knows the exact path each file will be at (by asking Alldebrid beforehand) in the rclone mount, so it technically shouldn't even need to "search" for it. I don't have an rclone mount setup, so if you could check for me that would be really appreciated :)

@bob1321
Copy link

bob1321 commented Jan 5, 2025

I just tried and this work now if I put /mnt/rclone/magnets/*/ in the settings
[DBG] Searching /mnt/rclone/magnets for B.mkv
[DBG] Searching /mnt/rclone/magnets/A/A/B.mkv...

@Sculas
Copy link
Contributor

Sculas commented Jan 5, 2025

Wait but, so now it's /mnt/rclone/magnets/<torrent name>/<torrent name>/<file name>, right?
And why does that work?! It should be /mnt/rclone/magnets/<file name> for all I know.
Do you have a torrent or magnet I can reproduce this with? If needed, you can encode it in base64 so it doesn't get indexed.

@bob1321
Copy link

bob1321 commented Jan 5, 2025

This one magnet:?xt=urn:btih:73609fad5cef86743734fcf242cbffe69edb3a2f is for /mnt/rclone/magnets/<torrent name>/<torrent name>/<file name.mkv>

this one magnet:?xt=urn:btih:4c39706481b8887a5f1e08347c273d87debf30d0 is for /mnt/rclone/magnets/<torrent name>/<file name>/<file name.mkv>

@Sculas
Copy link
Contributor

Sculas commented Jan 5, 2025

Oh, that's interesting. A torrent only allows a single file or a directory with multiple files. It seems AllDebrid makes you blindly follow that assumption because the API doesn't show the root folder if it has one.

After much reading, I think I've been able to find the root cause:

public static String? GetDownloadPath(Torrent torrent, Download download)
{
var fileUrl = download.Link;
if (String.IsNullOrWhiteSpace(fileUrl) || torrent.RdName == null)
{
return null;
}
var uri = new Uri(fileUrl);
var torrentPath = RemoveInvalidPathChars(torrent.RdName);
var fileName = download.FileName;
if (String.IsNullOrWhiteSpace(fileName))
{
fileName = uri.Segments.Last();
fileName = HttpUtility.UrlDecode(fileName);
}
var matchingTorrentFiles = torrent.Files.Where(m => m.Path.EndsWith(fileName)).Where(m => !String.IsNullOrWhiteSpace(m.Path)).ToList();
if (matchingTorrentFiles.Count > 0)
{
var matchingTorrentFile = matchingTorrentFiles[0];
var subPath = Path.GetDirectoryName(matchingTorrentFile.Path);
if (!String.IsNullOrWhiteSpace(subPath))
{
subPath = subPath.Trim('/').Trim('\\');
torrentPath = Path.Combine(torrentPath, subPath);
}
}
var filePath = Path.Combine(torrentPath, fileName);
return filePath;
}

This function always adds the torrent filename, which is actually the root directory of the torrent, to the file path.
This is evident when using a downloader like aria2c because the file gets placed at <download path>/<torrent filename>/<the file that's in the torrent>. So the fix I'm thinking is that we check if the torrent has multiple files, and if so don't add the torrent filename to the file path.

@rogerfar Do you know if this behavior also applies to RealDebrid? Does it place single-file torrents in a directory or not?

@rogerfar
Copy link
Owner

rogerfar commented Jan 6, 2025

I'm testing a bit and I'm not sure if this is what we're trying to accomplish.

I attached a couple of test torrents I uploaded myself. When I download them with qBittorrent, I get these results with tree /F:

C:.
│   TestFileNameMismatch.txt
│   TestSingle.txt
│
├───TestMultiple
│       TestMultiple.txt
│       TestMultiple2.txt
│
└───TestSingleInSub
    └───TestSingleInSub
            TestSingleInSub.txt

Before your patch I get:

C:.
├───TestFileNameMismatch.txt
│       TestFileNameMismatch.txt
│
├───TestMultiple
│       TestMultiple.txt
│       TestMultiple2.txt
│
├───TestSingle.txt
│       TestSingle.txt
│
└───TestSingleInSub.txt
    └───TestSingleInSub
            TestSingleInSub.txt

After your patch it's the same.

This isn't really good behaviour anyway as I want to match the output of RDT similar to qBittorrent.

If you run these test torrents in AllDebrid, how do they end up?

TestTorrents.zip

@rogerfar rogerfar reopened this Jan 6, 2025
@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

That was indeed my intention as well, to have the same output as qBittorrent. I will test the torrents you've provided tomorrow! (currently night for me)

@bob1321
Copy link

bob1321 commented Jan 6, 2025

Since the last version 2.0.94 everything is working pretty well!
With this in the setting /mnt/rclone/magnets/*/

@rogerfar
Copy link
Owner

rogerfar commented Jan 6, 2025

I found 1 snag, this is the Webdav drive from Real-Debrid when adding those test torrents:

Z:.
├───TestSingleInSub.txt
│       TestSingleInSub.txt
│
├───TestMultiple
│       TestMultiple.rar
│
└───TestSingle.txt
        TestSingle.txt

It seems it never places it in the root, even when it's a single file, and it ignores my sub-directory.

Which seems to not matter, the symlink is what matters.

@rogerfar
Copy link
Owner

rogerfar commented Jan 6, 2025

I added some code in my latest commit if you want to do a snifftest. I would like to see the results from AllDebrid and TorBox too to compare.

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Okay, so RealDebrid always stores it in a directory. AllDebrid doesn't. That's frustrating since we'll have to handle that.

Which seems to not matter, the symlink is what matters.

It does matter, though? The issue is that RDT always assumes the files are in a directory, which is the case for RealDebrid, but not for AllDebrid, and then tries to access the file from the WebDAV drive at a place it thinks it exists, but doesn't.

See it like this. Let's say we're downloading RDT.Client.v2.0.94.exe.torrent, which contains a single file: RDT.Client.v2.0.94.exe

  • RealDebrid stores this at: <webdav>/RDT.Client.v2.0.94.exe/RDT.Client.v2.0.94.exe
  • AllDebrid stores this at: <webdav>/RDT.Client.v2.0.94.exe
  • RDT will assume this file is at: <webdav>/RDT.Client.v2.0.94.exe/RDT.Client.v2.0.94.exe

Now, let's download RDT.Client.All.Versions.torrent, which looks like this:

C:.
└───RDT.Client.All.Versions
        RDT.Client.v2.0.92.exe
        RDT.Client.v2.0.93.exe
        RDT.Client.v2.0.94.exe

Let's use RDT.Client.v2.0.92.exe as the file here we're targeting.

  • RealDebrid stores this at: <webdav>/RDT.Client.All.Versions/RDT.Client.v2.0.92.exe
  • AllDebrid stores this at: <webdav>/RDT.Client.All.Versions/RDT.Client.v2.0.92.exe
  • RDT will assume this file is at: <webdav>/RDT.Client.All.Versions/RDT.Client.v2.0.92.exe

If you run these test torrents in AllDebrid, how do they end up?

It seems the torrents are stalled. Are they still being seeded? 2/4 have downloaded now, but the others are still stalled.

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

With those two torrents downloaded, I currently have this in /data/downloads:

C:.
│   TestFileNameMismatch.txt
│
└───TestSingleInSub
    └───TestSingleInSub
            TestSingleInSub.txt

I don't have the WebDAV drive mounted so I can't show a tree output, but I've checked and it's the same as above.
So, currently, RDT and AllDebrid match in their file structure. So, it seems like we'll have to add a check for RealDebrid, then.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

If I understand you correctly, A is the name of the torrent, and B.mkv is the only file in it. This means the file should be at /mnt/rclone/magnets/B.mkv. I've confirmed this by downloading a torrent from Nyaa that had a different name than the single file in it. Connecting to the WebDAV server via WinSCP showed the file with a different name in the magnets directory, as I expected. Requesting the torrent via Alldebrid's API shows a single file with a different name. RDT does the same, it sees the file and then sends that directly to the downloader. This should all just work as expected?

That's really weird, because I would expect that file to appear at /mnt/rclone/magnets/A - in fact Name Mismatch (Single File).torrent.zip demonstrates this behaviour: It appears at /mnt/rclone/magnets/Name\ Mismatch\ \(Single\ File\), not /mnt/rclone/magnets/single-file.txt. I created that torrent using mktorrent Name\ Mismatch\ \(Single\ File\) where Name\ Mismatch\ \(Single\ File\) is a directory with single-file.txt inside.
Here's the /v4/magnets/status response with irrelevant fields omitted.

{
    "data": {
        "magnets": {
            "filename": "Name Mismatch (Single File)",
            "links": [
                {
                    "filename": "single-file.txt",
                    "files": [
                        {
                            "n": "Name Mismatch (Single File)",
                            "e": [
                                {
                                    "n": "single-file.txt",
                                }
                            ]
                        }
                    ],
                }
            ],
        }
    },
}

I can confirm this is still not found by the latest version (docker build run on d5ee5d2)

I have at least one single-file torrent that is even weirder: its /api/v4/status looks like:

{
    "data": {
        "magnets": {
            "filename": "A",
            "links": [
                {
                    "filename": "B.mkv",
                    "files": [
                        {
                            "n": "B.mkv",
                        }
                    ],
                }
            ],
        }
    },
}

And is available at /mnt/rclone/magnets/A. I'm not sure how this torrent was made - I've been unable to figure out how to recreate this structure with either qBittorrent's torrent creator or mktorrent. But I think it confirms my suspicion that it's magnets.filename, not magnets.links[0].n which is used to make the name of the file that will appear in /mnt/rclone/magnets

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

Also worth noting that the TestFileNameMismatch.txt torrent doesn't actually demonstrate this mismatch behaviour:

{
    "data": {
        "magnets": {
            "filename": "TestFileNameMismatch.txt", // <-- same as links[0].files[0].n or filename, doesn't demonstrate mismatch
            "links": [
                {
                    "filename": "TestFileNameMismatch.txt",
                    "files": [
                        {
                            "n": "TestFileNameMismatch.txt"
                        }
                    ]
                }
            ]
        }
    }
}

But TestSingleInSub does

{
    "status": "success",
    "data": {
        "magnets": {
            "filename": "TestSingleInSub", // <-- not the same as links[0].files[0].e[0].e[0].n or filename
            "links": [
                {
                    "filename": "TestSingleInSub.txt",
                    "files": [
                        {
                            "n": "TestSingleInSub",
                            "e": [
                                {
                                    "n": "TestSingleInSub",
                                    "e": [
                                        {
                                            "n": "TestSingleInSub.txt",
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        }
    }
}

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

AllDebrid's way of placing files works exactly like qBittorrent, with Save at set to /magnets.
At the root, a torrent must only contain 1 file or directory. If you want multiple files, it needs to be in a directory.

As for the torrent you provided, the WebDAV server reports that the file is at magnets/Name Mismatch (Single File)/single-file.txt, is this also what you see? Because this is what I expect. You can verify this by opening the torrent in qBittorrent, and seeing that it's structured like this: Name Mismatch (Single File)/single-file.txt. If I now download this file using RDT, it's placed at /data/downloads/Name Mismatch (Single File)/single-file.txt. This is again the result I expect to see.

So why would you expect this file to be at /data/downloads/single-file.txt, if you're the one that placed the file in a directory to begin with? (if I'm supposed to believe what qBittorrent is telling me about your torrent)

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

I think we might be misunderstanding each other. I'm not making any assertions as to where files should end up in /data/downloads. I'm saying that when a torrent like Name Mismatch (Single File) (which has this mismatch in /v4/magnets/status) is added to AD, it will appear at /mnt/rclone/magnets/<filename> . Rdt-client probably should create a symlink at /data/downloads/<filename>/<links[0].files[0].e[0].n>.

But I disagree with your first sentence:

AllDebrid's way of placing files works exactly like qBittorrent, with Save at set to /magnets.

For Name Mismatch (Single File), when I add it to qBitTorrent, it will be saved at /qBit-dl/Name\ Mismatch\ \(Single\ File\)/single-file.txt. But when I add it to AD, it appears at /mnt/rclone/magnets/Name\ Mismatch\ \(Single\ File\)

@bob1321
Copy link

bob1321 commented Jan 6, 2025

The version 2.0.94 of yesterday is good with Alldebrid. No problem to create symlinks anymore

@Cucumberrbob
Copy link
Author

@bob1321 are you able to test with this file? Maybe this is something on my end instead

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Oh wait, are we experiencing the same issue then? I've just been working around it since I thought it was a weird issue only I had.
This is what I see if I connect to the WebDAV server via WinSCP:

image

As you can see, the torrents aren't directories for some reason, but instead something... weird?
I can't open the directories in WinSCP either, I have to right-click them and press "Open", only then does it download the directory to a temp directory and does it properly show the folder in Windows Explorer.

It sounds wrong that the file is placed at /mnt/rclone/magnets/Name Mismatch (Single File), since it should obviously be at /mnt/rclone/magnets/Name Mismatch (Single File)/single-file.txt. Can you check if this is also the case for you @Cucumberrbob?


Actually, I have a Linux VM now to test this with, so let me mount it via rclone as well and see if I get the same issue as you.

@Cucumberrbob
Copy link
Author

Yes! spot on! These files are actually files not directories despite how they're named, TestSingleInSub is actually a .txt - can you cat it (or the windows equivalent of cat it)?

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

I'm using v4.1 here since v4 is deprecated and imo a bit harder to read, but that doesn't affect the end result.
I've removed fields that aren't necessary, and the SAO torrent had way too much files, so I culled them as well.


TestMultiple:

Stored as: /mnt/rclone/magnets/TestMultiple/TestMultiple/TestMultiple.txt

{
  "status": "success",
  "data": {
    "magnets": {
      "filename": "TestMultiple",
      "size": 74,
      "nbLinks": 2,
      "files": [
        {
          "n": "TestMultiple",
          "e": [
            {
              "n": "TestMultiple2.txt",
              "s": 37,
              "l": "<>"
            },
            {
              "n": "TestMultiple.txt",
              "s": 37,
              "l": "<>"
            }
          ]
        }
      ]
    }
  }
}

Sword Art Online:

Stored as: /mnt/rclone/magnets/[Tenrai-Sensei] Sword Art Online [BD][1080p][HEVC 10bit x265][Dual Audio]/poster.jpg

{
  "status": "success",
  "data": {
    "magnets": {
      "filename": "[Tenrai-Sensei] Sword Art Online [BD][1080p][HEVC 10bit x265][Dual Audio]",
      "size": 58880130060,
      "nbLinks": 159,
      "files": [
        {
          "n": "Season 1",
          "e": [
            {
              "n": "Sword Art Online - S01E01 - The World Of Swords.mkv",
              "s": 489401205,
              "l": "<>"
            }
          ]
        },
        {
          "n": "poster.jpg",
          "s": 494325,
          "l": "<>"
        }
      ]
    }
  }
}

TestSingleInSub:

Stored as: /mnt/rclone/magnets/TestSingleInSub (broken, cannot access as file, cannot cd into it)

{
  "status": "success",
  "data": {
    "magnets": {
      "filename": "TestSingleInSub",
      "size": 86,
      "nbLinks": 1,
      "files": [
        {
          "n": "TestSingleInSub",
          "e": [
            {
              "n": "TestSingleInSub",
              "e": [
                {
                  "n": "TestSingleInSub.txt",
                  "s": 86,
                  "l": "<>"
                }
              ]
            }
          ]
        }
      ]
    }
  }
}

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

Yep, these are all constructible from the responses:

For TestMultiple, our file nodes (object with n but no e) are as follows:

{
  "n": "TestMultiple2.txt",
  "s": 37,
  "l": "<>"
},
{
  "n": "TestMultiple.txt",
  "s": 37,
  "l": "<>"
}

And they have the same parent,:

{
  "n": "TestMultiple",
  "e": [ /* our file nodes */ ]
}

The shared parent node has no parent (it's an element of files), so we can now assemble the file's path: <magnets.filename>/<files[0].n>/<files[0].e[i].n> with i=0 for the first file TestMultiple2.txt and I=1 for the second file TestMultiple.txt. Which is

TestMultiple/TestMultiple/TestMultiple.txt
TestMultiple/TestMultiple/TestMultiple2.txt

For the SAO one, we do similar, but since files.length == 2, let's do each one individually:

The first element of files

 {
  "n": "Season 1",
  "e": [
    {
      "n": "Sword Art Online - S01E01 - The World Of Swords.mkv",
      "s": 489401205,
      "l": "<>"
    }
  ]
}

So again, combining the root filename with all the ns gives us <filename>/<files[0].n>/<files[0].e[0].n> - in this case `

[Tenrai-Sensei] Sword Art Online [BD][1080p][HEVC 10bit x265][Dual Audio]/Season 1/Sword Art Online - S01E01 - The World Of Swords.mkv

And

 {
  "n": "poster.jpg",
  "s": 494325,
  "l": "<>"
}

Corresponds to <filename>/<files[1].n>, which is

 [Tenrai-Sensei] Sword Art Online [BD][1080p][HEVC 10bit x265][Dual Audio]/poster.jpg

In essence, the file's path is always <filename>/<all the 'n's> in a multi file torrent

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Yeah. It's confusing then why qBittorrent shows something different. Oh well.
Also, @Cucumberrbob what are your rclone parameters? Since I'm not able to access Name Mismatch (Single File) at all.
Regardless, I think AllDebrid is missing this edge case we explicitly check in RDT, which causes the bug:

var torrentFile = torrent.Files[0];
var subPath = Path.GetDirectoryName(torrentFile.Path);
// What we think is a single file torrent may also be a folder with a single file in it.
// So make sure we handle that here. If this is not the case, torrentPath will be empty below.
if (!String.IsNullOrWhiteSpace(subPath))
{
subPath = subPath.Trim('/').Trim('\\');
torrentPath = Path.Combine(torrentPath, subPath);
}


I think I'll have to unfortunately leave it at this though. This is something AllDebrid has to fix on their end if we ever want to see this issue closed unless there's some kind of workaround I'm not aware of at the moment. In either case, this doesn't seem to be an issue in RDT.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

My rclone parameters are

--cache-dir /home/cucumberrbob/.cache/rclone --dir-cache-time 10s --cutoff-mode cautious --vfs-cache-mode full  --buffer-size=0 --read-only --allow-non-empty  --vfs-cache-max-size 120G --vfs-cache-max-age 72h --allow-other --config /run/secrets/rclone.conf

I have mine running as a systemd mount, so it's in daemon mode, but this shouldn't matter

I think I'll have to unfortunately leave it at this though. This is something AllDebrid has to fix on their end if we ever want to see this issue closed unless there's some kind of workaround I'm not aware of at the moment. In either case, this doesn't seem to be an issue in RDT.

I think you're pretty much right - without AD fixing their webDAV some of the single file torrents will be unplayable. It's worth looking a bit more to see if this is consistently predictable based on the /magnets/status response,
Maybe just erroring out for single file torrents which have this mismatch is the best option

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

old comment (ignore)

Oh, it is predictable. If we fall into the block where we set the torrentPath, it's guaranteed failure:

var torrentFile = torrent.Files[0];
var subPath = Path.GetDirectoryName(torrentFile.Path);
// What we think is a single file torrent may also be a folder with a single file in it.
// So make sure we handle that here. If this is not the case, torrentPath will be empty below.
if (!String.IsNullOrWhiteSpace(subPath))
{
subPath = subPath.Trim('/').Trim('\\');
torrentPath = Path.Combine(torrentPath, subPath);
}


I'm currently reading the verbose logs of WinSCP to figure out what it's doing. I'll update this message as I go.
This message especially caught my eye:

2025-01-06 15:47:31.277 Getting size of directory "/magnets/Name Mismatch (Single File)"

WinSCP knows it's a directory...? The WebDAV response from SabreDAV looks correct:

SabreDAV Response
<d:multistatus
	xmlns:d="DAV:"
	xmlns:s="http://sabredav.org/ns">
	<d:response>
		<d:href>/magnets/Name%20Mismatch%20(Single%20File)/</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
				<d:resourcetype>
					<d:collection/>
				</d:resourcetype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/Name%20Mismatch%20(Single%20File)/single-file.txt</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
				<d:getcontentlength>16</d:getcontentlength>
				<d:resourcetype/>
				<d:getetag>&quot;cfcd208495d565ef66e7dff9f98764da&quot;</d:getetag>
				<d:getcontenttype>text/plain</d:getcontenttype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
</d:multistatus>

Okay, that response seems to be later. Not sure why WinSCP suddenly figures out it's a directory, that's next to figure out.
I scrolled a bit up and found this response. This is returned when WinSCP asks for the contents of the /magnets directory:

SabreDAV Response
<d:multistatus
	xmlns:d="DAV:"
	xmlns:s="http://sabredav.org/ns">
	<d:response>
		<d:href>/magnets/</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 15:10:37 GMT</d:getlastmodified>
				<d:resourcetype>
					<d:collection/>
				</d:resourcetype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/Name%20Mismatch%20(Single%20File)</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 12:47:02 GMT</d:getlastmodified>
				<d:getcontentlength>16</d:getcontentlength>
				<d:resourcetype/>
				<d:getetag>&quot;f0555d9391cbc8e3203080cfff2900a9&quot;</d:getetag>
				<d:getcontenttype>application/octet-stream</d:getcontenttype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/TestSingleInSub</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 11:28:29 GMT</d:getlastmodified>
				<d:getcontentlength>86</d:getcontentlength>
				<d:resourcetype/>
				<d:getetag>&quot;e1d4fb2e024f90822d69113ccd844a94&quot;</d:getetag>
				<d:getcontenttype>application/octet-stream</d:getcontenttype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/TestSingle.txt</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 12:30:47 GMT</d:getlastmodified>
				<d:getcontentlength>36</d:getcontentlength>
				<d:resourcetype/>
				<d:getetag>&quot;7b2ff460e337db1199b514fc666e7c7a&quot;</d:getetag>
				<d:getcontenttype>text/plain</d:getcontenttype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/TestMultiple/</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 13:29:24 GMT</d:getlastmodified>
				<d:resourcetype>
					<d:collection/>
				</d:resourcetype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
	<d:response>
		<d:href>/magnets/TestFileNameMismatch.txt</d:href>
		<d:propstat>
			<d:prop>
				<d:getlastmodified>Mon, 06 Jan 2025 10:42:46 GMT</d:getlastmodified>
				<d:getcontentlength>77</d:getcontentlength>
				<d:resourcetype/>
				<d:getetag>&quot;2501e66fa1e86f4f6a17a6bf80dbfc31&quot;</d:getetag>
				<d:getcontenttype>text/plain</d:getcontenttype>
			</d:prop>
			<d:status>HTTP/1.1 200 OK</d:status>
		</d:propstat>
	</d:response>
</d:multistatus>

Now here's the interesting part, the broken directories return application/octet-stream as their content type.
Meanwhile, the working directories return <d:resourcetype><d:collection/></d:resourcetype>, which is correct.
I did some searching, and it appears application/octet-stream is used when the content type is null:
https://github.com/sabre-io/dav/blob/58be83aae10a244372f113b63624c48034378094/lib/DAV/FS/File.php#L76-L86
This is in File.php though, not Directory.php. Seems like there's still a bit more debugging to do...

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

I might be confused on where the variables come from in that snippet, but am I right in saying that the if condition on L68 would be true if /magnets/status looked like: Edit: I need my eyes checked, I did not see that !

{
    "data": {
        "magnets": {
            "filename": "A",
            "links": [
                {
                    "filename": "B.mkv",
                    "files": [
                        {
                            "n": "B.mkv",
                        }
                    ],
                }
            ],
        }
    },
}

For all the torrents I have which have responses like this, head /mnt/rclone/magnets/<filename> works fine - it's only for ones like Name Mismatch (Single File), which have subdirectories, that the files seem to have this file/directory weirdness . In my ~500 torrent library, I only have one non-test torrent with this structure, and 10 of the structure above
<nyaa dot si>/view/1509241 is a good example of a torrent with this behaviour in the json above - it ends up at /mnt/rclone/magnets/<filename> with no .mkv extension - but it plays fine as an mkv

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

but am I right in saying that the if condition on L68 would be true if /magnets/status looked like

No. In your example the file will be stored at /mnt/rclone/magnets/B.mkv.
The filename under magnets is only ever used when length(links) > 1.

Here's an example that will cause the bug:

{
  "status": "success",
  "data": {
    "magnets": {
      "filename": "A",
      "files": [
        {
          "n": "B",
          "e": {
            "n": "C.mkv"
          }
        }
      ]
    }
  }
}

This file should be stored at /mnt/rclone/magnets/B/C.mkv, but the WebDAV server thinks /mnt/rclone/magnets/B is a file of type application/octet-stream (see my comment above as to why), which makes it inaccessible. However, it appears if you force the WebDAV server into thinking it's a directory by asking for the file listing of it, it fixes its mistake and properly returns it as a directory. This is also why the "Open" workaround works in WinSCP.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

The filename under magnets is only ever used when length(links) > 1.

I don't think this is true. For all the single file torrents I've posted in here, none of them follow that rule. They are all always at /mnt/rclone/magnets/<filename>. Could you try the nyaa torrent I posted in the comment above? That ends up at /mnt/rclone/magnets/Akira.1988.UHD.BluRay.2160p.TrueHD.Atmos.7.1.DV.HEVC.HYBRID.REMUX-FraMeSToR, but is a playable mkv

But if the bug is just for single file torrents where the file is in a directory, it might be OK just to error out and say we can't download the files. If my library is representative, 0.2% error rate isn't terrible - but even if it's not, those torrents would have errored (less helpfully) anyway

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

You're right, it is stored that way. So then why does that mkv file work, but Name Mismatch (Single File) does not, which is literally the exact same thing? I am able to run file on the broken mkv "directory", but not the name mismatch one.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

They're not quite identical - the /magnets/status responses aren't identical in structure - Name Mismatch (Single File) has a directory in it, but that nyaa torrent has no directories - it's just a single file at the root.

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

You're right, qBittorrent's seemingly inconsistent (or AllDebrid's) file listing got me again. Why can these things just not be consistent with each other... Now, why is AllDebrid missing the file extension on the file...

@rogerfar
Copy link
Owner

rogerfar commented Jan 6, 2025

Sorry it's been night here so haven't fully caught up.

But it might be worth it to make a list of all potential iterations and create test torrents for those, because I also see the same discrepancies with Real-Debrid too. qBittorrent seems to have some naming rules itself too which conflicts with expectation and the output of the debrid providers.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

You're right, qBittorrent's seemingly inconsistent (or AllDebrid's) file listing got me again. Why can these things just not be consistent with each other... Now, why is AllDebrid missing the file extension on the file...

Honestly I think it's AD again - the first bit of any .torrent file is pretty much human readable if you head it , and I think single-file torrents that end up with different structures in AD's webDAV have the same structure in this frontmatter section.

But it might be worth it to make a list of all potential iterations and create test torrents for those, because I also see the same discrepancies with Real-Debrid too. qBittorrent seems to have some naming rules itself too which conflicts with expectation and the output of the debrid providers.

Definitely, but I can't figure out how to create torrents of the working kind - where they appear like this in the /magnets/status response, and I've been struggling to find GitHub-safe torrents to link to directly.

But between that nyaa one and the Name Mismatch (Single File) torrent I uploaded, I think all cases are covered for single-file torrents

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Sorry it's been night here so haven't fully caught up.

The TLDR pretty much is that this is an AllDebrid issue:

  1. Torrents with a directory as root and 2 or more files in them: works (confirmed by TestMultiple.torrent)
  2. Torrents with a directory as root and 1 file that may be N directories deep: broken (confirmed by TestSingleInSub.torrent)
  3. Torrents with a file as root: works (confirmed by TestSingle.txt.torrent)

Name Mismatch (Single File).torrent also confirms case 2.
TestFileNameMismatch.txt.torrent is the same as TestSingle.txt.torrent.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

Torrents with a file as root: works (confirmed by TestSingle.txt.torrent)

To clarify, this is only slightly true. This only works because TestSingle.txt.torrent has the name TestSingle.txt, which is the same as the file's name inside it. If the torrent were called anything else, the file would be at /mnt/rclone/magnets/<anything else>, and it would not be usable because of this weird directory/file mixup that AD's webDAV server seems to be doing

/magnets/status response for TestSingle.txt.torrent
{
  "data": {
      "magnets": {
          "filename": "TestSingle.txt", // <-- this is the same
          "links": [
              {
                  "filename": "TestSingle.txt", // <-- as this, which I think is always the same 
                  "size": 36,
                  "files": [
                      {
                          "n": "TestSingle.txt",  // <-- as this
                          "s": 36
                      }
                  ]
              }
          ]
      }
  }
}

TestFileNameMismatch.txt.torrent is the same as TestSingle.txt.torrent.

This is only true because likewise, TestFilefnameMismatch.txt.torrent has the same name as the file inside it. Despite its name, it's not actually a mismatch according to //magnets/status

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 6, 2025

To sum this all up into something actionable:

The logic should be as follows:

if the torrent has multiple files then
    Each file's path  is `/mnt-path/<data.magnets.filename>/<all the "n"s>`
else
    // ↓ equivalent to if <data.magnets.links[0].files[0].e> exists
    if the file is inside a directory then
        Error out - these types of torrents confuse AD's webDAV and they aren't usable
    else
      We can use this file! it will appear at `/mnt-path/<data.magnets.filename>`, 
      but we should ensure that it ends up being renamed to `<data.magnets.links[0].files[0].n>`
      (which in my experience has always been the same as `<data.magnets.links[0].filename>`

If it's not clear what I mean by "all the 'n's ", I mean all the "n" attributes of the ancestor nodes of the file node joined with /. A file node is an object that has an "n" but has no "e", and is in its parent's "e" array (this is what I mean by the parent/child relationship). See this comment (the status responses are in the comment above it) for a breakdown of how we construct the file path from the /magnets/status response

@Sculas can you spot an issue with this logic?

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

That looks correct! I'm not sure what's the best way to report this to the Alldebrid developers. Just asking them to test the Nyaa torrent you sent earlier @Cucumberrbob is enough to show something's definitely wrong (missing file extension and broken content/mime type). And hopefully, with that, we can also tell them about the issue with the broken directories (which may be harder to figure out). Their GitHub profile hasn't seen activity in quite some time, unfortunately. I'll send an email to their support.

I may otherwise consider forking rclone_rd and porting it to Alldebrid if I also encounter this issue myself. I'm not using the rclone mount setup right now, but I'm working on switching to it. I'll have to see though, since I've already spent hours trying to fix this.

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

I've sent an email to their support mail (support [at] alldebrid [dot] com). Time to wait for their response now.
If I don't get a response in a few days, I'll try sending a support ticket via their site (since I'm not sure if their mail is still active).
Also, I'm not sure who sent me an email 2 days ago with a test torrent, but sorry I didn't see it. It went into my spam folder 😅

@Cucumberrbob
Copy link
Author

Just got a response from AD support - they've said they've found the issue and "will fix it most likely tomorrow". Let's see what we end up with then.
In the meantime, the multi-file torrents are still an issue in the latest version: TestMultiple.torrent still is not handled properly. I got something working based on @Sculas' most recent PR. The multi-file part is here - which shouldn't actually need the commit from @Sculas' most recent PR. I'm not sure if adding another potentialFilePath is a good idea though

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Oh wow, that's a quick response! I guess you used the website? I'll take a look at your PR and continue there :)

@Cucumberrbob
Copy link
Author

Yeah! I won't lie I'm impressed. I did use the contact form on their website.

With regards to the PR, given that the single-file torrent situation will likely change in the near future, only the changes in DownloadHelper.cs are actually useful now.

@Sculas
Copy link
Contributor

Sculas commented Jan 6, 2025

Yeah, I was about to mention that in the PR, but we may want to wait with my PR until we know it's fixed. Otherwise, we push out a release now that works around a part of the bug, but then tomorrow it's fixed for example, and we have to fix it again to remove the workaround. My PR as it is now should work just fine afaik, given that the WebDAV bug is fixed in Alldebrid.
Actually, there might still be some behavior that might be incorrect which we discovered today. I'll take another look over it all once it's fixed though.

@Cucumberrbob
Copy link
Author

Just heard back from AD (they're insanely quick). As I understand it, the logic is now as follows:
All files are at /mnt-path/magnets/<filename>/<all the "n"s> unless it's a single-file-at-the-root torrent, in which case it's /mnt-path/magnets/<files[0].n>.

lsing the relevant paths:
> eza --tree /mnt/rclone/magnets/{Test*,Name\ Mismatch\ \(Single\ File\)}
/mnt/rclone/magnets/'Name Mismatch (Single File)'
└──  single-file.txt
/mnt/rclone/magnets/TestFileNameMismatch.txt
/mnt/rclone/magnets/TestMultiple
└── TestMultiple
    ├── TestMultiple.txt
    └── TestMultiple2.txt
/mnt/rclone/magnets/TestSingle.txt
/mnt/rclone/magnets/TestSingleInSub
└── TestSingleInSub
    └── TestSingleInSub.txt

@Sculas
Copy link
Contributor

Sculas commented Jan 7, 2025

I've been checking occasionally and am happy to report that it's just been fixed! I can finally access the directories again.
Well isn't that a coincidence...

All files are at /mnt-path/magnets/<filename>/<all the "n"s> unless it's a single-file-at-the-root torrent, in which case it's /mnt-path/magnets/<files[0].n>.

Perfect! I'll update my PR to replicate this behavior. Seems like there's no need, the current behavior looks correct.
@Cucumberrbob When you have the time, could you please test if my PR works as expected for you now?

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 7, 2025

On your PR (ee9e80f) with my changes to the Dockerfile made, a few are still broken, I'll keep updating these as I get the logs through

TestSingleInSub [not working]
[DBG] Starting symlink resolving of [url redacted]/TestSingleInSub.txt, writing to path: TestSingleInSub/TestSingleInSub/TestSingleInSub.txt
[DBG] Searching /mnt/rclone/magnets for TestSingleInSub.txt (attempt #0)...
[DBG] Searching /mnt/rclone/magnets/TestSingleInSub/TestSingleInSub.txt...
[DBG] Searching /mnt/rclone/magnets/TestSingleInSub.txt/TestSingleInSub.txt...
[DBG] Searching /mnt/rclone/magnets/TestSingleInSub.txt...
TestSingle.txt [not working]
[DBG] Starting symlink resolving of [url redacted]/TestSingle.txt, writing to path: TestSingle.txt
[DBG] Searching /mnt/rclone/magnets for TestSingle.txt (attempt #0)...
[DBG] Searching /mnt/rclone/magnets/magnets/TestSingle.txt...
[DBG] Searching /mnt/rclone/magnets/rclone/TestSingle.txt...
[DBG] Searching /mnt/rclone/magnets/mnt/TestSingle.txt...
[DBG] Searching /mnt/rclone/magnets/TestSingle.txt/TestSingle.txt...
[DBG] Searching /mnt/rclone/magnets/TestSingle/TestSingle.txt...
[DBG] Searching /mnt/rclone/magnets/TestSingle.txt...
TestMultiple [not working]
[DBG] Starting symlink resolving of [url redacted]/TestMultiple2.txt, writing to path: TestMultiple/TestMultiple/TestMultiple2.txt
[DBG] Searching /mnt/rclone/magnets for TestMultiple2.txt (attempt #0)...
[DBG] Searching /mnt/rclone/magnets/TestMultiple/TestMultiple2.txt...
[DBG] Searching /mnt/rclone/magnets/TestMultiple2.txt/TestMultiple2.txt...
[DBG] Searching /mnt/rclone/magnets/TestMultiple2/TestMultiple2.txt...
[DBG] Searching /mnt/rclone/magnets/TestMultiple2.txt...

Notably in every case, the path passed to SymlinkDownloader.Download() (logged on the first line as writing to path <path>) is always the correct path within /mnt/rclone/magnets

All single-file-at-the-root torrents work now, since they're found by searching for new FileInfo(path).Name in rcloneMountPath (which is done with potentialFilePath "")

@bob1321
Copy link

bob1321 commented Jan 7, 2025

In the setting of RDclient, did you put /*/
With that its work to search in subfolders

@Sculas
Copy link
Contributor

Sculas commented Jan 7, 2025

I suppose that could work, but since the path in writing to path <path> is always correct anyway, why not skip the whole search and just get the file directly, since we know it's there anyway? We know the exact behavior of Alldebrid placing the torrent files.

@Cucumberrbob
Copy link
Author

Cucumberrbob commented Jan 7, 2025

In the setting of RDclient, did you put /*/

No - I've intentionally left that out. That really feels like a workaround to deal with the reasonably common case of torrents containing one level of directories. It doesn't actually solve the problem of torrents with directories in them unless it's only one directory deep.

I suppose that could work, but since the path in writing to path is always correct anyway, why not skip the whole search and just get the file directly, since we know it's there anyway? We know the exact behavior of Alldebrid placing the torrent files.

This is pretty much what I did in my PR on your fork in the SymlinkDownloader - I just added the directory from path to potentialFilePaths - they get deduped anyway so no harm if this overlaps with an existing potentialFilePath
Yes, that seems much more efficient. With the Kind in your PR, we can easily check right at the top if it's an AD torrent and skip the whole potentialFilePath thing
Also might be worth (in a separate issue) seeing if this can be done for other debrid services - presumably they all have some way of figuring out the file path from their equivalent to /magnets/status. The whole searching in multiple places for a file feels a little janky to me

@Sculas
Copy link
Contributor

Sculas commented Jan 7, 2025

I've fixed the symlink search by adding the searchPath to the directories to search in (8fe277f).
I've also added the optimization for Alldebrid downloads (38a3bc1).
@rogerfar The PR (#652) should be ready to merge now if it works for @Cucumberrbob.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment