released a new version of scrapy-contrib-bigexporter (https://github.com/ZuInnoTe/scrapy-contrib-bigexporters/releases/tag/scb-0.5.0) for exporting web crawled data with scrapy to big data formats, such as parquet, orc and avro.
now also with provenance using pypi trusted publisher (https://blog.trailofbits.com/2024/11/14/attestations-a-new-generation-of-signatures-on-pypi/). You can be now ensure which CI job build the artifact and how it was built.
It uses short-living token to publish to pypi and not static credentials.
See the provenance e.g. for the *.whl on pypi: https://pypi.org/project/scrapy-contrib-bigexporters/#scrapy_contrib_bigexporters-0.5.0-py3-none-any.whl
[#]pypi
=> More informations about this toot | View the thread | More toots from jornfranke@mastodon.online
=> View pypi tag This content has been proxied by September (ba2dc).Proxy Information
text/gemini