I am a big fan of magic-wormhole which I routinely use to send large files to people. Its a great tool. However, it does have one notable problem - if the transfer fails for any reason, you basically have to start over. This isn't much of a problem under normal circumstances, but it does pose a challenge when transferring large files half-way around the world.
There is, however, a very similar tool called croc, which is directly inspired by magic-wormhole, works functionally the same way, but it has interruptible transfers by virtue of each file being hashed before transfer. I still use magic-wormhole from time to time because I find that it has a better GUI (and some people need that) and it handles complex networking configurations better whereas croc sometimes just can't connect.
Anyway, recently, I was transferring some files over to someone, and we collectively came up with the idea to use a loop in bash with croc to basically restart transfers until they complete. I think this is worth publishing so other people can use it too.
Essentially, on the side of the sender, we execute this loop:
1CROC_SECRET="<KEY>" until croc send "<FILE_OR_DIRECTORY>"; do sleep 30; done
Note the following:
- We set the key manually, for this I recommend writing a small script to generate a strong random key (example below). Without this,
crocwill generate a different key each time the transfer is interrupted. - We use
untilto basically run the whole thing on a loop until the command actually exits with code zero indicating success. - There is a 30 second delay after a failure.
On the receiving side, given the same key, we execute the following:
1CROC_SECRET="<KEY>" until croc --yes; do sleep 30; done
The same basic principles apply with an until loop and a 30 second delay, but after a failure, the sender and receiver will invariably re-align, re-hash the files, and resume the transfer.
Generating a Random Key #
To generate a strong random key, you can use a script like this one in Python:
1import string
2from secrets import choice
3import sys
4
5def random_str(length: int):
6 return ''.join(choice(string.ascii_lowercase + string.ascii_uppercase + string.digits) for i in range(length))
7
8print(random_str(int(sys.argv[1])))
A previous version of this blog post used random.choice() instead of secrets.choice(), but it was pointed out to me that random.choice() is not cryptographically secure.
Alternatively, you can use openssl like so, for an equally secure random key:
1openssl rand -hex 16
Navi Cheat #
navi is one of my all-time favorite productivity tools in the terminal. If you haven't heard of it, I strongly recommend it. I wrote two navi cheats for these loops which you can see below. Feel free to use them:
% croc, file-transfer
# Use croc to send a file in a persistent way that re-starts the send in case of a crash
export CROC_SECRET="<random_password>"
until croc send "<local_file>"; do sleep 30; done
# Receive a file with croc in a persistent way that re-starts in case of a crash
export CROC_SECRET="<password>"
until croc --yes; do sleep 30; done
$ local_file: ls -1
$ random_password: python3 "`navi info cheats-path`/navi-cheats-repo/network/random_string.py" 24