Copyparty as a Network File System
·
cyclicircuit's blog
How to use copyparty's WebDAV server as a network file system with rclone and systemd
Table of Contents
- What I've Tried, and Why They All Suck
- Copyparty as a Network File System
A very common requirement in homelabs (and all sorts of other networks if we're being honest) is the ability to access files over the network using a remote file system. In my case, this is even more essential because I have one very large storage server which many different applications (typically running in Proxmox VMs/CTs) need to access with different permissions. However, despite the demonstrably clear need for such tools, I've found the actual software available in this space (SMB, NFS, and SSHFS) to be extremely limited and frankly unfit for an environment like a homelab.
Meanwhile, I had started using copyparty for all sorts of things: file drops for friends, a searchable document archive, a quick way to get files on and off of machines from a browser, etc. If you haven't come across it, it's a self-contained file server written in Python that speaks HTTP (with an excellent web UI), WebDAV, SFTP, FTP, TFTP, and even SMB.
So after the latest round of frustration with SSHFS and SMB (to be honest, I never got to the point of successfully running NFS for any meaningful period of time) I decided to try something truly unhinged - what if I just used Copyparty as the network file system?
At some point it occurred to me that copyparty's WebDAV support is good enough that, paired with rclone mount and a couple of systemd units, it makes a really effective network file system for Linux machines. Copyparty's own README even lists rclone over WebDAV as the fastest way to mount it as a drive, and has a dedicated guide for it. Over the course of a few months, I moved every server in my homelab off of Samba and onto it, and I haven't looked back.
What I've Tried, and Why They All Suck #
- NFS is the "proper" UNIX answer, but it is way, way too complicated to make secure in a homelab. The default security flavor,
sec=sys, has "no cryptographic security" at all: the server bases its access control on the UID and GID provided in each request, and simply assumes the client's kernel has verified them, which the NFS man page itself describes as "an easy system to spoof". So any machine that is allowed to mount the export can claim to be any user it wants (other than root, thanks toroot_squash). The fix for that issec=krb5, which means running an entire Kerberos KDC and issuing principals to every server, just to get what amounts to username/password authentication. And without Kerberos, NFSv4's name-based ID mapping is disabled by default (look fornfs4_disable_idmapping), so raw numeric UIDs go over the wire and you get to keep them in sync across every machine by hand. - SSHFS is wonderful for one person mounting one thing, and I used it for years. What eventually drove me off of it is that it occasionally just locks up, and it turns out that's by design. Network operations run without timeouts by default, so if the connection is interrupted, "operations on files or directories under the mountpoint will block until the connection is either restored or closed altogether", and "applications that try to access such files or directories will generally appear to 'freeze'". When that happens, every process that touches the mount (including
umount) ends up in uninterruptible sleep, and the only way out iskill -9on the sshfs process. The official workaround,-o ServerAliveInterval=15 -o reconnect, only trades the hang for guaranteed loss of whatever was being read or written at the time. - Samba is a disaster zone. I've already written an entire article about the incantations needed to unfuck it for Linux and macOS, and even with all of that in place, it has arcane character encoding issues and files that silently disappear. Either they collapse into each other thanks to case-insensitivity, or they're hidden from directory listings entirely: when I finally moved Jellyfin off of Samba, a dozen or so old Soviet movies whose names contained characters remapped into the Unicode private use area showed up for the first time, having been invisible to Jellyfin for over a year. And then there's the security track record that everyone involved should be ashamed of: Samba's security page lists over 180 CVEs, including one that let clients make the server execute a shared library they'd uploaded (in every version from 3.5.0 onwards), and a CVSS 9.9 root RCE in
vfs_fruit, the very same Mac-compatibility module my previous article tells you to enable.
Copyparty as a Network File System #
The general idea is as follows:
- On the server, we run a copyparty instance (using a docker container in the example below, but you can deploy it however you like) that exposes one share per consumer, with a username/password for every client and granular per-share permissions (read-only for media servers for example and read-write for the things like immich which also upload information).
- On the client,
rclonemounts the WebDAV share as a FUSE file system, driven by systemd.mount/.automountunits, with its cache tuned to whatever the client is doing. - The result is pretty performant, and in my experience, considerably less painful than any of the alternatives above (with some numbers to back that up).
Server-Side Configuration #
I should clarify outright that I deploy copyparty slightly differently than what is described below. The objective is to get readers to the point where they can understand how this works, and then deploy it themselves in a way that makes sense to them. I purposely demonstrate using a docker container because I feel that this is the easiest to set up for experimentation, and from there, you should be able to adapt this to however you manage your own setup.
There are, however, key elements of the configuration that need to be called out. Specifically, I do not describe how to manage HTTPS below, and the example explicitly uses HTTP. This is not ideal, and undermines my claims of better security than SMB above. Those claims are only valid if you have proper HTTPS, permissions, and authentication configured. The last two are described below, but you have to configure HTTPS in a manner that works best for your configuration.
The Container #
Before the first start, create the db directory and give it to the user the container runs as. Otherwise Docker creates it as root, and copyparty (running as 1000:1000) can't write to it. I personally store this stuff in the /var/lib/docker-apps/ directory, but you can obviously use whichever directory works for you:
1mkdir -p /var/lib/docker-apps/copyparty-network-fs/db
2chown 1000:1000 /var/lib/docker-apps/copyparty-network-fs/db
1services:
2 copyparty:
3 image: copyparty/ac:latest
4 container_name: copyparty-network-fs
5 user: "1000:1000"
6 read_only: true
7 tmpfs:
8 - /tmp
9 security_opt:
10 - no-new-privileges:true
11 cap_drop:
12 - ALL
13 environment:
14 HOME: /db
15 XDG_CONFIG_HOME: /db/.config
16 ports:
17 - "192.168.XXX.XXX:3920:3923"
18 volumes:
19 - /etc/docker-compose-projects/copyparty-network-fs/copyparty.conf:/cfg/copyparty.conf:ro
20 - /var/lib/docker-apps/copyparty-network-fs/db:/db
21 - "/storage:/w/storage:rw"
22 - "/storage/media/audio/books:/w/audiobooks:ro"
23 - "/storage/media/audio/podcasts:/w/podcasts:rw"
24 restart: unless-stopped
Note the following:
image: copyparty/ac:latestfrom Docker Hub: Notghcr.io/9001/copyparty, which returns "denied" for anonymous pulls.user: "1000:1000": WebDAV carries no UID or GID, so the only thing that decides on-disk ownership is the UID the server process runs as. Pinning it to my own user means every write lands ascyclic:cyclic, regardless of which copyparty account or client made it. If you need UID/GID mapping, simply run multiple copyparty instances as different accounts.HOMEandXDG_CONFIG_HOMEpoint at/dbbecause the root file system is read-only and copyparty wants somewhere to persist filekeys and dirkeys. If you don't do this, it falls back to/tmpwith a warning, which means those keys change on every restart, which is less than helpful.- Bind the port to one specific address, not
127.0.0.1(LAN clients couldn't reach it). If you want it available on all interfaces (including something like Tailscale), use0.0.0.0instead. - Each share is a bind mount at
/w/<name>, withroorrwmatching what the ACL says. Belt and suspenders: if you mess up a permission in the config, the kernel will still refuse to write.
The Configuration File #
Here's what you need to feed to copyparty as a configuration:
1[global]
2 p: 3923
3 ah-alg: argon2
4 ah-salt: <random-salt>
5 no-db-ip
6 ups-who: 0
7 shr-who: no
8 no-script
9 no-logues
10 no-readme
11 xvol
12 xdev
13 no-reload
14 no-rescan
15 ed
16 see-dots
17
18[accounts]
19 r2-d2: <hash>
20 mikoshi: <hash>
21 brainiac: <hash>
22 jn-66: <hash>
23
24[/storage]
25 /w/storage
26 accs:
27 rwmd.: r2-d2, mikoshi
28 r.: brainiac
29 flags:
30 -dedup
31
32[/audiobooks]
33 /w/audiobooks
34 accs:
35 r.: jn-66
36 flags:
37 -dedup
38
39[/podcasts]
40 /w/podcasts
41 accs:
42 rwmd.: jn-66
43 flags:
44 -dedup
What the Global Options Do #
I'm not going to go over every single configuration element (the copyparty documentation is extensive), but I will cover the important ones:
ah-alg: argon2andah-salt: Hash account passwords, which is why the[accounts]block above shows hashes rather than plaintext. The salt is pinned in the config on purpose; hashing account passwords below explains why, and how to generate the hashes.no-db-ip,ups-who: 0,no-script,no-logues,no-readme,no-reload,no-rescan, andshr-who: no: We cannot disable or block the copyparty UI since WebDAV operates on the same port; it's the same thing. These options, however, limit the copyparty UI to the point where it leaks as little information as possible and won't render or run anything from uploaded files, and everything is still behind authentication to boot.xvol: Prevents someone from following a symlink that leads out of the volume, unless it lands inside another volume the same account can already read. Without it, a symlink from a share to a file anywhere else on the server simply serves that file, whatever it happens to be. Symlinks that stay inside the volume still work.xdev: Prevents someone from descending into another file system as part of the volume.edandsee-dots: Make dotfiles visible. Individual users also need the.permission, which is what the trailing dots in theaccs:blocks are doing.
Two operational notes:
- Changes to
[global]need a restart, but account and volume changes can be reloaded in place by sending copypartySIGUSR1usingdocker kill -s USR1 copyparty-network-fs. - Watch out for the ban thresholds: By default copyparty bans a client for 24 hours after more than 9 failed passwords in an hour, and (the one that's much easier to trip) after more than 9
403responses within two minutes. That is reasonable for a file server, but a real annoyance when you're iterating through a client config. I accidentally banned myself for a day while testing and the sign is a mount that suddenly returns I/O errors for no discernible reason.
Hashing Account Passwords #
You can skip this entirely at first: put a plaintext password in [accounts] and copyparty will still accept it, hash it on startup, and log a warning with the hashed version to paste back into the config. But it's good to never have the plaintext in the file at all, and copyparty is on PyPI, so uvx can generate hashes on any machine without installing anything:
1# once: generate a salt, and put it in [global] as `ah-salt:`
2python3 -c 'import secrets; print(secrets.token_urlsafe(18))'
3
4# per account: prompts without echoing, prints the line to paste into [accounts]
5read -rsp 'password: ' PW; echo
6printf '%s' "$PW" | uvx --with argon2-cffi copyparty --ah-alg argon2 --ah-salt '<your-salt>' --ah-gen - 2>/dev/null | grep '^+'
7unset PW
Note the following:
--ah-gen -reads the password from stdin, so it never appears in your shell history or inps. It can also hash several at once, one per line.--with argon2-cffiis not optional. Argon2 support is an optional dependency in copyparty; without it, hashing fails withNo module named 'argon2'. Thecopyparty/acDocker image already includes it, which is why the server itself doesn't need anything extra.- The output is a
+-prefixed string. The leading+is how copyparty tells a hash from a plaintext password, so paste the whole thing, plus included, as the account's password. - The salt has to match the server's exactly. A hash generated with any other salt never matches, and the login just falls through to anonymous with no error on either end. That's why the config pins
ah-saltinstead of letting copyparty generate its own: it gives you the exact value to pass to--ah-salthere, and it can't change underneath you if copyparty's config directory is ever lost or unwritable.
Username/Password Authentication and Permissions #
Now the other primary section of that file: [accounts] creates logins, and each [/share] block maps a URL to a directory and lists who may do what to it. The flags: -dedup at the end of each block isn't a permission: it disables copyparty's deduplication, which replaces duplicate uploads with symlinks. This is not something you want in a file system mount. Dedup is off by default anyway; this just makes it explicit and prevents it from being accidentally enabled later.
The permission letters are single characters, and the full set is worth reading, but these are the ones that matter for file system use:
| letter | meaning |
|---|---|
r |
read: browse listings, download files |
w |
write: upload only, plus move/copy files into this folder |
m |
move files out of this folder (i.e. rename) |
d |
delete |
. |
may show dotfiles in listings |
Be aware that w in copyparty configurations means "upload", not "write" in the POSIX sense. I originally granted rw and everything looked fine until something tried to rename or delete a file: rclone's MOVE and DELETE returned HTTP 500-series errors from copyparty's permission check. A mount that behaves like a file system needs rwmd, and the trailing . on top of that if you want dotfiles. So the config above uses rwmd. for writers and r. for readers.
I personally tend to create one account per client.
Client-Side Configuration #
On each client, rclone mounts a share as a FUSE file system, and systemd takes care of mounting it at boot. The example below is r2-d2, which mounts the /storage share read-write at /infosphere.
Prerequisites #
1# rclone itself; distro packages tend to be old, so use the official script
2curl https://rclone.org/install.sh | sudo bash
3sudo apt install fuse3
4
5# lets systemd use rclone as a mount helper (Type=rclone below)
6sudo ln -s /usr/bin/rclone /sbin/mount.rclone
7
8# lets users other than root read the mount (needed for allow-other below)
9echo user_allow_other | sudo tee -a /etc/fuse.conf
The rclone Config #
I use one config file per mount, which keeps each mount's credentials separate:
1# /etc/rclone/infosphere.conf
2[infosphere]
3type = webdav
4url = http://192.168.XXX.XXX:3920/storage
5vendor = owncloud # don't ask me why this is the right option
6pacer_min_sleep = 0.01ms
7user = r2-d2
8pass = <output of: rclone obscure 'the-password'>
Note the following:
- The share goes in the
url, not in the mount path. rclone asks the root of the URL for free space, and copyparty only reports it for the root of a share, so with the share anywhere elsedfshows a meaningless 1 PiB. vendor = owncloudmakes rclone send each file's modification time along with the upload, which copyparty honors, so files keep their mtimes. It's what copyparty's rclone guide recommends, as mentioned above, no clue why "owncloud" is the name of the setting.pacer_min_sleep = 0.01ms: by defaultrclonewaits 10 ms between WebDAV requests, and copyparty's guide recommends removing that. It makes a big difference with small files: rsyncing 3,000 of them through a mount with the cache off went from 289 seconds to 19 (see Performance Metrics).passis obscured, not encrypted.rclone obscureonly stops the password being readable at a glance, sochmod 600the file.
The systemd Mount Unit #
systemd has a very useful concept called a mount unit similar to /etc/fstab but a little bit more flexible, which allows us to configure a program like rclone mount as a file system that gets mounted reliably on a given machine.
1# /etc/systemd/system/infosphere.mount
2[Unit]
3Description=rclone WebDAV mount of infosphere:/storage
4After=network-online.target
5Wants=network-online.target
6
7[Mount]
8What=infosphere:
9Where=/infosphere
10Type=rclone
11Options=_netdev,nofail,args2env,config=/etc/rclone/infosphere.conf,cache-dir=/var/cache/rclone/infosphere,vfs-cache-mode=full,vfs-cache-max-size=20G,vfs-cache-max-age=24h,uid=1000,gid=1000,allow-other,umask=022
12TimeoutSec=60
13
14[Install]
15WantedBy=multi-user.target
Then systemctl daemon-reload && systemctl enable --now infosphere.mount.
Note the following:
- The file name has to match
Where=./infospheremeansinfosphere.mount, and/mnt/audiobookswould bemnt-audiobooks.mount(systemd-escape --pathwill tell you). If they don't match, systemd fails with aBadUnitSettingerror that looks like it's complaining aboutWhat=. Avoid dashes in the path, since systemd escapes them as\x2d. args2envpasses the options to rclone as environment variables instead of command-line arguments, which keeps them out ofps.- Use absolute paths for
config=andcache-dir=. Mount units don't run with your usual environment, so~and$HOMEwon't work. uidandgidare only applicable on the client. They decide how files appear on this machine; who owns them on the server is dictated by the container'suser:setting.- Add
read-onlyto the options for read-only shares. Copyparty enforces it anyway, but this way the client knows too.
If you'd rather mount on first access than at boot, there's a related concept called an automount unit and you can use that that instead of the .mount:
1# /etc/systemd/system/infosphere.automount
2[Unit]
3Description=rclone WebDAV automount of /infosphere
4
5[Automount]
6Where=/infosphere
7TimeoutIdleSec=0
8
9[Install]
10WantedBy=multi-user.target
Don't copy the network-online.target lines into the automount unit. Automount units start very early in boot, and ordering them after the network creates a dependency cycle that systemd resolves by not starting NetworkManager at all. The .mount unit keeps those lines, and that's enough.
Caching #
In the torrent of settings in the rclone configuration above, you may have noticed several related to caching. Sadly, these are both complicated and important to get right. Its probably the most annoying part of using rclone/webDAV in this manner, but I consider it an acceptable price to pay given the limitations on other file systems I've tried.
rclone manages its own cache, which essentially copies files onto the local disk in cache-dir. vfs-cache-mode is the setting that controls this behavior. Please refer to the Performance Metrics section to see what kind of impact this has on performance:
| mode | reads | writes | good for |
|---|---|---|---|
off |
straight from the server | streamed straight to the server; files can't be opened read-write or seeked while writing | read-write mounts by default, and large files read front to back by one reader |
writes |
straight from the server | written to local disk first, uploaded after the file is closed | read-write mounts where something needs to open files read-write |
full |
cached on local disk as they're read | same as writes |
read-only mounts, especially with several readers of one file or random reads |
(There's also minimal, somewhere between off and writes, which none of my clients use.)
There is no mode that caches reads but sends writes straight to the server. The closest thing is full on a mount with the read-only option: nothing is ever written through it, so it works as a pure read cache, with none of the write caveats below.
Key caveats:
- With
writesandfull, writes are asynchronous.close()returns as soon as the file is in the local cache, and the upload happens afterwards: by default 5 seconds after the file is closed (vfs-write-back), plus however long the upload takes. Until then, the server and every other client see the old version of the file, or nothing at all. Settingvfs-write-back=0sstarts the upload as soon as the file is closed, but it's still asynchronous. - Pending uploads survive a crash, but not a dead disk. If rclone dies or the machine reboots before an upload finishes, rclone uploads it the next time it starts with the same flags, as long as
cache-dirsurvives (so don't put it on a tmpfs). If the client's disk dies first, whatever was still waiting is gone. For read-write mounts,offis the safer choice unless something needs to open files read-write. fullis slow on the first read. Everything it reads is also written tocache-dir, which made a cold read less than half as fast asoffin my tests. Setvfs-cache-max-sizeandvfs-cache-max-age, or it will fill the client's disk: one of mine hit 15 GB within 20 minutes.- The write cache is slow with lots of small files. Writing 3,000 small files took 19 seconds with the cache off, but 100–160 seconds with
writesorfull. If a mount mostly receives lots of small files and nothing on it needs to open files read-write,offis much faster. offfalls apart with several readers of the same file. See the concurrency result in Performance Metrics. If more than one thing reads the same file at once, usefull.buffer-sizeis memory per open file (16 MB by default). An application that opens lots of files at once can eat a lot of RAM this way: on a 2 GB container, Audiobookshelf pushed rclone to 1.4 GB and got it OOM-killed, and droppingbuffer-sizeto 4 MB fixed it.vfs-read-aheadonly does anything infullmode.- Don't raise
dir-cache-timejust because the library is big. It's how long rclone trusts a directory listing (5 minutes by default), so a file added on the server, or by another client, stays invisible for that long. I set it to 72 hours for Jellyfin once, on the theory that library scans would be faster. They weren't, since a scan reads each directory once anyway, and new media would have been hidden for up to three days.
For reference, this is what my clients use:
| client | workload | mode | notable options |
|---|---|---|---|
| Media servers (like Jellyfin) | streams large media files, never writes | off |
read-only, buffer-size=32M |
| Audiobookshelf | opens lots of files at once on a 2 GB container | full |
buffer-size=4M, vfs-read-ahead=0 |
| general use | a bit of everything | full |
vfs-cache-max-size=20G, vfs-cache-max-age=24h |
Why It's So Effective in a Homelab #
Going back to the problems from the start of this article:
- NFS trusts whatever UID the client claims. Here, every client logs in with its own username and password, and the server decides what each account can do, share by share.
- NFS makes you keep UIDs in sync across machines. WebDAV doesn't carry UIDs at all, so there's nothing to keep in sync: everything lands on disk as the server's user.
- SSHFS hangs when the connection drops. rclone has connection and I/O timeouts (1 minute and 5 minutes by default), so a server going away turns into errors instead of processes stuck until you kill them.
- Samba rewrites and hides file names. File names travel as plain UTF-8 over HTTP and are served exactly as they are on disk. In fact I've had serious issues with SMB hiding media files.
- Samba's security track record. Copyparty is a single unprivileged process in a locked-down container, configured by one file you can read top to bottom (with the HTTPS caveat from the server section).
It's also fast enough, and on my array faster than the SMB mount it replaced. The numbers are below, along with the places where it isn't fast.
Performance Metrics #
I measured this twice, in two different environments, and the results disagree:
- Ideal conditions, in a pair of throwaway containers with the storage taken out of the path, to isolate protocol overhead. This is where WebDAV's weaknesses show up.
- My real storage array, spinning disks and all. Here WebDAV comes out ahead of the SMB mount it replaced.
Part 1: Ideal Conditions (Where WebDAV Struggles) #
The setup, loosely following this NAS performance comparison:
- Two throwaway Proxmox CTs (privileged, which NFS needs), one on each of two different physical nodes, so traffic crosses the 10GbE LAN instead of looping back through a bridge on one host.
- The server exports one directory over four protocols at once: copyparty (WebDAV), Samba (SMB3, configured exactly as my Samba article prescribes), SSH (for sshfs), and NFSv4.2.
- The client mounts that same export six ways: the three rclone cache modes, SMB3, SSHFS, and NFSv4.2.
- The export lives on tmpfs, so the disks aren't the bottleneck and the protocol stack is.
- The WebDAV mounts use
pacer_min_sleep = 0.01ms, as recommended in the rclone config above. My first run used rclone's default, which made several WebDAV results look far worse than they are, so I re-ran the WebDAV tests on identical containers. The WebDAV numbers below are from that second run. - Versions: copyparty 1.20.24, rclone 1.75.1, Samba 4.19.5, sshfs 3.7.3, kernel 7.0.14-pve.
One 2 GiB file, conv=fsync on the write so the timer includes getting the bytes to the server, and the mount restarted before each read so nothing comes from a local cache:
| mount | write | read |
|---|---|---|
copyparty WebDAV (rclone, cache off) |
442 MB/s | 590 MB/s |
copyparty WebDAV (rclone, cache writes) |
301 MB/s | 563 MB/s |
copyparty WebDAV (rclone, cache full) |
477 MB/s | 247 MB/s |
| SMB3 | 918 MB/s | 1084 MB/s |
| SSHFS | 393 MB/s | 326 MB/s |
| NFSv4.2 | 806 MB/s | 1089 MB/s |
When the server can saturate the link, SMB and NFS are roughly twice as fast as WebDAV, but copyparty isn't the reason: a plain curl of the same file, with no FUSE in the path, does 1081 MB/s down and 394 MB/s up, so the slowdown comes from the rclone mount.
There are also three specific areas where WebDAV does much worse.
Writing small files is slow, especially through a write cache. rsync -a of ~3000 files of 600K each, in both directions:
| mount | read (server → client) | write (client → server) |
|---|---|---|
copyparty WebDAV (cache off) |
7s | 19s |
copyparty WebDAV (cache writes) |
8s | 104s |
copyparty WebDAV (cache full) |
15s | 162s |
| SMB3 | 7s | 7s |
| SSHFS | 9s | 11s |
| NFSv4.2 | 6s | 6s |
Reading three thousand small files is as fast over WebDAV as over SMB or NFS, and writing them with the cache off takes 19 seconds against 6–7. That's with the pacer fix: with rclone's default, the same write took 289 seconds and the read 32, because every file costs about five WebDAV requests (the upload, the rename rsync does afterwards, setting its modification time, and a couple of checks) and rclone waited 10 ms before each one. The write cache is a different story. With writes or full, the same write takes anywhere from 100 to 160 seconds across runs, and the pacer makes no difference there. I haven't pinned down why, but if you're writing lots of small files, a mount with the cache off is much faster. If your workload is a Maildir, a git checkout or node_modules, stop reading and use something else.
Random reads without a local cache are catastrophic. 4K random, queue depth 32:
| file system | random write | random read |
|---|---|---|
copyparty WebDAV (cache off) |
not supported | 0.8 MB/s (194 IOPS) |
copyparty WebDAV (cache writes) |
58 MB/s (14.8k IOPS) | 5.4 MB/s (1.4k IOPS) |
copyparty WebDAV (cache full) |
70 MB/s (17.9k IOPS) | 264 MB/s (68k IOPS) |
| SMB3 | 286 MB/s (73k IOPS) | 316 MB/s (81k IOPS) |
| SSHFS | 3.2 MB/s (821 IOPS) | 166 MB/s (42.5k IOPS) |
| NFSv4.2 | 295 MB/s (75k IOPS) | 356 MB/s (91k IOPS) |
194 IOPS, because every 4K read becomes its own HTTP range request (with rclone's default pacer it was half that). Turning the cache to full makes the same test about 350x faster, and close to SMB and NFS, because rclone fetches chunks once and then serves them off local disk. Note also that --vfs-cache-mode off cannot open a file read-write at all (fio gets EPERM), which is why there is no random-write figure for it.
Concurrent readers on one file collapse without a cache. With --vfs-cache-mode off, one process reading a file sequentially gets 898 MB/s. Four processes reading the same file get 55 MB/s, 16x slower, because rclone streams a single HTTP response per handle and readers at different offsets defeat it. With --vfs-cache-mode full the same test is fine, because the cache absorbs the concurrency.
That last one is the main reason not to leave the cache off by default, and why I spend so much time on the cache configuration above: the right cache mode depends on how many things read the same file at once, as well as how much disk you're willing to give it.
Finally, SMB is faster on metadata. Listing a 110-entry directory takes ~17–19 ms over SMB against ~33 ms for a WebDAV PROPFIND.
Part 2: The Real Array (Where WebDAV Wins) #
Everything above ran with the storage removed from the path. My real server is 12 × 22 TB spinning disks in raidz2 with a 15.5 GiB ARC, and it can't come close to saturating 10GbE on a cold read. So I ran the same kind of test against it: one 22 GiB movie file, a different cold 2 GiB region for each method so nothing is served out of ARC twice, from a client over the same 10GbE.
This is what ends up happening on my system in practice:
| configuration | 2 GiB cold read |
|---|---|
local dd on the server, no network at all |
338–393 MB/s |
copyparty WebDAV via rclone mount (cache off, 32M buffer) |
358 MB/s |
copyparty over raw HTTP via curl, no FUSE |
365 MB/s |
| SMB3 via cifs mount, same share, same file | 242 MB/s |
| (ARC-warm local read, for reference) | 899 MB/s |
Two conclusions from this, which undermine the results from Part 1:
- The array is the bottleneck here, not the protocol or FUSE. 358 MB/s through a FUSE mount, against a server whose own local read is 338–393 MB/s, means WebDAV delivers essentially everything the disks can, and the mount is only ~2% slower than raw
curl(365 MB/s) on the same file. In the tmpfs run that same comparison showed nearly a 2x penalty (590 vs 1081), but only because the mount was the bottleneck there. - Against the real array, WebDAV beat SMB by ~50%: 358 vs 242 MB/s, on the same file, share, and client, with each method reading a cold region. I'm not certain why, but my best guess is that one long streaming HTTP
GETlets ZFS prefetch further ahead than SMB's chunked request/response pattern does. Either way, it's reproducible, and it matches what I saw when I moved my services over.
What I Take Away from This #
I expected throughput to be a painful tradeoff, and it turned out not to be. On top of being more performant than Samba, the copyparty/WebDAV/rclone approach has way better permissions management, authentication, and (when using HTTPS) is actually encrypted - none of the other options give me this combination of features nearly as easily.