This page permanently redirects to gemini://complete.org/filespooler-append-only-queues/.

Filespooler Append-Only Queues

Purpose of Append-Only Queues

In the Using Filespooler over Syncthing[1] example, we synced the entire b64queue directory. This is unnecessary.

=> 1: /using-filespooler-over-syncthing/

It would be better to sync only the jobs subdirectory of it, to prevent it looking like a valid queue for processing on the sender.

Implications of Append-Only Queues

The Filespooler Reference[2] discusses append-only queues. An append-only queue will allow commands that append (fspl queue-write) and commands that inspect (for instance, fspl queue-ls) to succeed. Commands that require a sequence number, such as fspl queue-process, will fail.

=> 2: /filespooler-reference/

The idea is that often times, the jobs directory will be synced. It would be convenient to be able to write to this directory on the sender side, while still preventing the sender from accidentally trying to process the queue. That is exactly the point of an append-only queue.

Set up

1. On the sender, use fspl queue-init --append-only to create an append-only queue.

2. On the receiver, use fspl queue-init without --append-only to create a regular queue.

3. Both queues will contain a jobs subdirectory. Tell your software to sync just that directory.

Links to this note

• Using Filespooler over Rclone and S3, Rsync.Net, Etc.[3]

=> 3: /using-filespooler-over-rclone-and-s3-rsync-net-etc/

You can use Filespooler with a number of other filesystems and storage options. s3fs, for instance, lets you mount S3 filesystems locally. I can't possibly write about every such option, so I'll write about one: rclone.

• Only Syncing the Jobs/ Subdirectory With an Append-Only Queue[4]

=> 4: /using-filespooler-over-syncthing/

Filespooler[5] is a way to execute commands in strict order on a remote machine, and its communication method is by files. This is a perfect mix for Syncthing[6] (and others, but this page is about Filespooler and Syncthing).

=> 5: /filespooler/ | 6: /syncthing/

• Filespooler[7]

=> 7: /filespooler/

Filespooler lets you request the remote execution of programs, including stdin and environment. It can use tools such as S3, Dropbox, Syncthing[8], NNCP[9], ssh, UUCP[10], USB drives, CDs, etc. as transport; basically, a filesystem is the network for Filespooler.

Filespooler is particularly suited to distributed and Asynchronous Communication[11].

=> 8: /syncthing/ | 9: /nncp/ | 10: /uucp/ | 11: /asynchronous-communication/

More on www.complete.org

=> Homepage
=> Interesting Topics
=> How This Site is Built
=> About John Goerzen
=> Web version of this site

(c) 2022-2024 John Goerzen