Copyparty as a Network File System

· cyclicircuit's blog
How to use copyparty's WebDAV server as a network file system with rclone and systemd


Table of Contents

A very common requirement in homelabs (and all sorts of other networks if we're being honest) is the ability to access files over the network using a remote file system. In my case, this is even more essential because I have one very large storage server which many different applications (typically running in Proxmox VMs/CTs) need to access with different permissions. However, despite the demonstrably clear need for such tools, I've found the actual software available in this space (SMB, NFS, and SSHFS) to be extremely limited and frankly unfit for an environment like a homelab.

Meanwhile, I had started using copyparty for all sorts of things: file drops for friends, a searchable document archive, a quick way to get files on and off of machines from a browser, etc. If you haven't come across it, it's a self-contained file server written in Python that speaks HTTP (with an excellent web UI), WebDAV, SFTP, FTP, TFTP, and even SMB.

So after the latest round of frustration with SSHFS and SMB (to be honest, I never got to the point of successfully running NFS for any meaningful period of time) I decided to try something truly unhinged - what if I just used Copyparty as the network file system?

At some point it occurred to me that copyparty's WebDAV support is good enough that, paired with rclone mount and a couple of systemd units, it makes a really effective network file system for Linux machines. Copyparty's own README even lists rclone over WebDAV as the fastest way to mount it as a drive, and has a dedicated guide for it. Over the course of a few months, I moved every server in my homelab off of Samba and onto it, and I haven't looked back.

What I've Tried, and Why They All Suck #

Copyparty as a Network File System #

The general idea is as follows:

  1. On the server, we run a copyparty instance (using a docker container in the example below, but you can deploy it however you like) that exposes one share per consumer, with a username/password for every client and granular per-share permissions (read-only for media servers for example and read-write for the things like immich which also upload information).
  2. On the client, rclone mounts the WebDAV share as a FUSE file system, driven by systemd .mount/.automount units, with its cache tuned to whatever the client is doing.
  3. The result is pretty performant, and in my experience, considerably less painful than any of the alternatives above (with some numbers to back that up).

Server-Side Configuration #

I should clarify outright that I deploy copyparty slightly differently than what is described below. The objective is to get readers to the point where they can understand how this works, and then deploy it themselves in a way that makes sense to them. I purposely demonstrate using a docker container because I feel that this is the easiest to set up for experimentation, and from there, you should be able to adapt this to however you manage your own setup.

There are, however, key elements of the configuration that need to be called out. Specifically, I do not describe how to manage HTTPS below, and the example explicitly uses HTTP. This is not ideal, and undermines my claims of better security than SMB above. Those claims are only valid if you have proper HTTPS, permissions, and authentication configured. The last two are described below, but you have to configure HTTPS in a manner that works best for your configuration.

The Container #

Before the first start, create the db directory and give it to the user the container runs as. Otherwise Docker creates it as root, and copyparty (running as 1000:1000) can't write to it. I personally store this stuff in the /var/lib/docker-apps/ directory, but you can obviously use whichever directory works for you:

1mkdir -p /var/lib/docker-apps/copyparty-network-fs/db
2chown 1000:1000 /var/lib/docker-apps/copyparty-network-fs/db
 1services:
 2  copyparty:
 3    image: copyparty/ac:latest
 4    container_name: copyparty-network-fs
 5    user: "1000:1000"
 6    read_only: true
 7    tmpfs:
 8      - /tmp
 9    security_opt:
10      - no-new-privileges:true
11    cap_drop:
12      - ALL
13    environment:
14      HOME: /db
15      XDG_CONFIG_HOME: /db/.config
16    ports:
17      - "192.168.XXX.XXX:3920:3923"
18    volumes:
19      - /etc/docker-compose-projects/copyparty-network-fs/copyparty.conf:/cfg/copyparty.conf:ro
20      - /var/lib/docker-apps/copyparty-network-fs/db:/db
21      - "/storage:/w/storage:rw"
22      - "/storage/media/audio/books:/w/audiobooks:ro"
23      - "/storage/media/audio/podcasts:/w/podcasts:rw"
24    restart: unless-stopped

Note the following:

The Configuration File #

Here's what you need to feed to copyparty as a configuration:

 1[global]
 2  p: 3923
 3  ah-alg: argon2
 4  ah-salt: <random-salt>
 5  no-db-ip
 6  ups-who: 0
 7  shr-who: no
 8  no-script
 9  no-logues
10  no-readme
11  xvol
12  xdev
13  no-reload
14  no-rescan
15  ed
16  see-dots
17
18[accounts]
19  r2-d2:         <hash>
20  mikoshi:       <hash>
21  brainiac:      <hash>
22  jn-66:         <hash>
23
24[/storage]
25  /w/storage
26  accs:
27    rwmd.: r2-d2, mikoshi
28    r.: brainiac
29  flags:
30    -dedup
31
32[/audiobooks]
33  /w/audiobooks
34  accs:
35    r.: jn-66
36  flags:
37    -dedup
38
39[/podcasts]
40  /w/podcasts
41  accs:
42    rwmd.: jn-66
43  flags:
44    -dedup
What the Global Options Do #

I'm not going to go over every single configuration element (the copyparty documentation is extensive), but I will cover the important ones:

Two operational notes:

  1. Changes to [global] need a restart, but account and volume changes can be reloaded in place by sending copyparty SIGUSR1 using docker kill -s USR1 copyparty-network-fs.
  2. Watch out for the ban thresholds: By default copyparty bans a client for 24 hours after more than 9 failed passwords in an hour, and (the one that's much easier to trip) after more than 9 403 responses within two minutes. That is reasonable for a file server, but a real annoyance when you're iterating through a client config. I accidentally banned myself for a day while testing and the sign is a mount that suddenly returns I/O errors for no discernible reason.
Hashing Account Passwords #

You can skip this entirely at first: put a plaintext password in [accounts] and copyparty will still accept it, hash it on startup, and log a warning with the hashed version to paste back into the config. But it's good to never have the plaintext in the file at all, and copyparty is on PyPI, so uvx can generate hashes on any machine without installing anything:

1# once: generate a salt, and put it in [global] as `ah-salt:`
2python3 -c 'import secrets; print(secrets.token_urlsafe(18))'
3
4# per account: prompts without echoing, prints the line to paste into [accounts]
5read -rsp 'password: ' PW; echo
6printf '%s' "$PW" | uvx --with argon2-cffi copyparty --ah-alg argon2 --ah-salt '<your-salt>' --ah-gen - 2>/dev/null | grep '^+'
7unset PW

Note the following:

Username/Password Authentication and Permissions #

Now the other primary section of that file: [accounts] creates logins, and each [/share] block maps a URL to a directory and lists who may do what to it. The flags: -dedup at the end of each block isn't a permission: it disables copyparty's deduplication, which replaces duplicate uploads with symlinks. This is not something you want in a file system mount. Dedup is off by default anyway; this just makes it explicit and prevents it from being accidentally enabled later.

The permission letters are single characters, and the full set is worth reading, but these are the ones that matter for file system use:

letter meaning
r read: browse listings, download files
w write: upload only, plus move/copy files into this folder
m move files out of this folder (i.e. rename)
d delete
. may show dotfiles in listings

Be aware that w in copyparty configurations means "upload", not "write" in the POSIX sense. I originally granted rw and everything looked fine until something tried to rename or delete a file: rclone's MOVE and DELETE returned HTTP 500-series errors from copyparty's permission check. A mount that behaves like a file system needs rwmd, and the trailing . on top of that if you want dotfiles. So the config above uses rwmd. for writers and r. for readers.

I personally tend to create one account per client.

Client-Side Configuration #

On each client, rclone mounts a share as a FUSE file system, and systemd takes care of mounting it at boot. The example below is r2-d2, which mounts the /storage share read-write at /infosphere.

Prerequisites #

1# rclone itself; distro packages tend to be old, so use the official script
2curl https://rclone.org/install.sh | sudo bash
3sudo apt install fuse3
4
5# lets systemd use rclone as a mount helper (Type=rclone below)
6sudo ln -s /usr/bin/rclone /sbin/mount.rclone
7
8# lets users other than root read the mount (needed for allow-other below)
9echo user_allow_other | sudo tee -a /etc/fuse.conf

The rclone Config #

I use one config file per mount, which keeps each mount's credentials separate:

1# /etc/rclone/infosphere.conf
2[infosphere]
3type = webdav
4url = http://192.168.XXX.XXX:3920/storage
5vendor = owncloud  # don't ask me why this is the right option
6pacer_min_sleep = 0.01ms
7user = r2-d2
8pass = <output of: rclone obscure 'the-password'>

Note the following:

The systemd Mount Unit #

systemd has a very useful concept called a mount unit similar to /etc/fstab but a little bit more flexible, which allows us to configure a program like rclone mount as a file system that gets mounted reliably on a given machine.

 1# /etc/systemd/system/infosphere.mount
 2[Unit]
 3Description=rclone WebDAV mount of infosphere:/storage
 4After=network-online.target
 5Wants=network-online.target
 6
 7[Mount]
 8What=infosphere:
 9Where=/infosphere
10Type=rclone
11Options=_netdev,nofail,args2env,config=/etc/rclone/infosphere.conf,cache-dir=/var/cache/rclone/infosphere,vfs-cache-mode=full,vfs-cache-max-size=20G,vfs-cache-max-age=24h,uid=1000,gid=1000,allow-other,umask=022
12TimeoutSec=60
13
14[Install]
15WantedBy=multi-user.target

Then systemctl daemon-reload && systemctl enable --now infosphere.mount.

Note the following:

If you'd rather mount on first access than at boot, there's a related concept called an automount unit and you can use that that instead of the .mount:

 1# /etc/systemd/system/infosphere.automount
 2[Unit]
 3Description=rclone WebDAV automount of /infosphere
 4
 5[Automount]
 6Where=/infosphere
 7TimeoutIdleSec=0
 8
 9[Install]
10WantedBy=multi-user.target

Don't copy the network-online.target lines into the automount unit. Automount units start very early in boot, and ordering them after the network creates a dependency cycle that systemd resolves by not starting NetworkManager at all. The .mount unit keeps those lines, and that's enough.

Caching #

In the torrent of settings in the rclone configuration above, you may have noticed several related to caching. Sadly, these are both complicated and important to get right. Its probably the most annoying part of using rclone/webDAV in this manner, but I consider it an acceptable price to pay given the limitations on other file systems I've tried.

rclone manages its own cache, which essentially copies files onto the local disk in cache-dir. vfs-cache-mode is the setting that controls this behavior. Please refer to the Performance Metrics section to see what kind of impact this has on performance:

mode reads writes good for
off straight from the server streamed straight to the server; files can't be opened read-write or seeked while writing read-write mounts by default, and large files read front to back by one reader
writes straight from the server written to local disk first, uploaded after the file is closed read-write mounts where something needs to open files read-write
full cached on local disk as they're read same as writes read-only mounts, especially with several readers of one file or random reads

(There's also minimal, somewhere between off and writes, which none of my clients use.)

There is no mode that caches reads but sends writes straight to the server. The closest thing is full on a mount with the read-only option: nothing is ever written through it, so it works as a pure read cache, with none of the write caveats below.

Key caveats:

For reference, this is what my clients use:

client workload mode notable options
Media servers (like Jellyfin) streams large media files, never writes off read-only, buffer-size=32M
Audiobookshelf opens lots of files at once on a 2 GB container full buffer-size=4M, vfs-read-ahead=0
general use a bit of everything full vfs-cache-max-size=20G, vfs-cache-max-age=24h

Why It's So Effective in a Homelab #

Going back to the problems from the start of this article:

It's also fast enough, and on my array faster than the SMB mount it replaced. The numbers are below, along with the places where it isn't fast.

Performance Metrics #

I measured this twice, in two different environments, and the results disagree:

  1. Ideal conditions, in a pair of throwaway containers with the storage taken out of the path, to isolate protocol overhead. This is where WebDAV's weaknesses show up.
  2. My real storage array, spinning disks and all. Here WebDAV comes out ahead of the SMB mount it replaced.
Part 1: Ideal Conditions (Where WebDAV Struggles) #

The setup, loosely following this NAS performance comparison:

One 2 GiB file, conv=fsync on the write so the timer includes getting the bytes to the server, and the mount restarted before each read so nothing comes from a local cache:

mount write read
copyparty WebDAV (rclone, cache off) 442 MB/s 590 MB/s
copyparty WebDAV (rclone, cache writes) 301 MB/s 563 MB/s
copyparty WebDAV (rclone, cache full) 477 MB/s 247 MB/s
SMB3 918 MB/s 1084 MB/s
SSHFS 393 MB/s 326 MB/s
NFSv4.2 806 MB/s 1089 MB/s

When the server can saturate the link, SMB and NFS are roughly twice as fast as WebDAV, but copyparty isn't the reason: a plain curl of the same file, with no FUSE in the path, does 1081 MB/s down and 394 MB/s up, so the slowdown comes from the rclone mount.

There are also three specific areas where WebDAV does much worse.

Writing small files is slow, especially through a write cache. rsync -a of ~3000 files of 600K each, in both directions:

mount read (server → client) write (client → server)
copyparty WebDAV (cache off) 7s 19s
copyparty WebDAV (cache writes) 8s 104s
copyparty WebDAV (cache full) 15s 162s
SMB3 7s 7s
SSHFS 9s 11s
NFSv4.2 6s 6s

Reading three thousand small files is as fast over WebDAV as over SMB or NFS, and writing them with the cache off takes 19 seconds against 6–7. That's with the pacer fix: with rclone's default, the same write took 289 seconds and the read 32, because every file costs about five WebDAV requests (the upload, the rename rsync does afterwards, setting its modification time, and a couple of checks) and rclone waited 10 ms before each one. The write cache is a different story. With writes or full, the same write takes anywhere from 100 to 160 seconds across runs, and the pacer makes no difference there. I haven't pinned down why, but if you're writing lots of small files, a mount with the cache off is much faster. If your workload is a Maildir, a git checkout or node_modules, stop reading and use something else.

Random reads without a local cache are catastrophic. 4K random, queue depth 32:

file system random write random read
copyparty WebDAV (cache off) not supported 0.8 MB/s (194 IOPS)
copyparty WebDAV (cache writes) 58 MB/s (14.8k IOPS) 5.4 MB/s (1.4k IOPS)
copyparty WebDAV (cache full) 70 MB/s (17.9k IOPS) 264 MB/s (68k IOPS)
SMB3 286 MB/s (73k IOPS) 316 MB/s (81k IOPS)
SSHFS 3.2 MB/s (821 IOPS) 166 MB/s (42.5k IOPS)
NFSv4.2 295 MB/s (75k IOPS) 356 MB/s (91k IOPS)

194 IOPS, because every 4K read becomes its own HTTP range request (with rclone's default pacer it was half that). Turning the cache to full makes the same test about 350x faster, and close to SMB and NFS, because rclone fetches chunks once and then serves them off local disk. Note also that --vfs-cache-mode off cannot open a file read-write at all (fio gets EPERM), which is why there is no random-write figure for it.

Concurrent readers on one file collapse without a cache. With --vfs-cache-mode off, one process reading a file sequentially gets 898 MB/s. Four processes reading the same file get 55 MB/s, 16x slower, because rclone streams a single HTTP response per handle and readers at different offsets defeat it. With --vfs-cache-mode full the same test is fine, because the cache absorbs the concurrency.

That last one is the main reason not to leave the cache off by default, and why I spend so much time on the cache configuration above: the right cache mode depends on how many things read the same file at once, as well as how much disk you're willing to give it.

Finally, SMB is faster on metadata. Listing a 110-entry directory takes ~17–19 ms over SMB against ~33 ms for a WebDAV PROPFIND.

Part 2: The Real Array (Where WebDAV Wins) #

Everything above ran with the storage removed from the path. My real server is 12 × 22 TB spinning disks in raidz2 with a 15.5 GiB ARC, and it can't come close to saturating 10GbE on a cold read. So I ran the same kind of test against it: one 22 GiB movie file, a different cold 2 GiB region for each method so nothing is served out of ARC twice, from a client over the same 10GbE.

This is what ends up happening on my system in practice:

configuration 2 GiB cold read
local dd on the server, no network at all 338–393 MB/s
copyparty WebDAV via rclone mount (cache off, 32M buffer) 358 MB/s
copyparty over raw HTTP via curl, no FUSE 365 MB/s
SMB3 via cifs mount, same share, same file 242 MB/s
(ARC-warm local read, for reference) 899 MB/s

Two conclusions from this, which undermine the results from Part 1:

  1. The array is the bottleneck here, not the protocol or FUSE. 358 MB/s through a FUSE mount, against a server whose own local read is 338–393 MB/s, means WebDAV delivers essentially everything the disks can, and the mount is only ~2% slower than raw curl (365 MB/s) on the same file. In the tmpfs run that same comparison showed nearly a 2x penalty (590 vs 1081), but only because the mount was the bottleneck there.
  2. Against the real array, WebDAV beat SMB by ~50%: 358 vs 242 MB/s, on the same file, share, and client, with each method reading a cold region. I'm not certain why, but my best guess is that one long streaming HTTP GET lets ZFS prefetch further ahead than SMB's chunked request/response pattern does. Either way, it's reproducible, and it matches what I saw when I moved my services over.
What I Take Away from This #

I expected throughput to be a painful tradeoff, and it turned out not to be. On top of being more performant than Samba, the copyparty/WebDAV/rclone approach has way better permissions management, authentication, and (when using HTTPS) is actually encrypted - none of the other options give me this combination of features nearly as easily.