Solving miscellaneous automation problems

While automation and scripting can save time and headaches, with it comes strange edge cases.

pip infamously does not have a dependency resolver[1]. This means in order to upgrade all packages at once you need an unruly command like this:

=> 1: https://github.com/pypa/pip/issues/988

pip3 list --user --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1

Upgrading all pip packages at once seems like an anti-pattern because different packages have different requirements and upgrading dependencies can break things. For this reason, dependencies are not updated automatically. However, because there is no convenient way to keep your packages up-to-date due to the lack of a dependency resolver, you have a bit of a catch-22. Installing Python packages via your system package manager resolves this problem because the packages are curated but you run the risk of poorly maintained packages and developers prefer moving at a faster pace than system package managers.

But I digress. There were two interesting things I ran into here. First, pip install will complain if you try to pass it no requirements. It does not complain about an empty requirements.txt file. The former behavior was changed to also have pip install return a non-zero exit status upon error. Ironically, this poses a problem for me I use chronic combined with MAILTO to e-mail me an error upon output. I do not wish to receive an e-mail for this error because it will happen the majority of the time. This means I need to do something like this:

# Fish shell syntax
pip3 install --user -U -r (pip3 list --user --outdated --format=freeze | grep -v '^\-e' | cut -d = -f 1 | psub) 

This uses process substitution with the requirements flag. Therefore we suppress the empty requirements error.

Secondly, I discovered that because I was not using the -e switch with chronic my errors were being suppressed. By default, chronic operates based on return status instead of stderr. My offending crontab line was this:

15 22 * * * chronic fish -c 'borgmatic -v -1 --syslog-verbosity 1; borgmatic -c /home/tom/.config/borgmatic/rsync_net.yaml -v -1 --syslog-verbosity 1; enable_suspend'

Here we have Borgmatic only output errors but log everything else to syslog. Because fish -c uses the return status of the last command, chronic did not trigger.