Skip to content

Support zstd compression #7457

Open

Description

@hutch3232

I really appreciate how fread and fwrite support gzip compression with basically no effort on the users part. I think it would be awesome if zstandard was also supported. https://github.com/facebook/zstd

Various online benchmarks have shown compression and decompression to be faster than gzip and with better compression ratios. Another big benefit is that zstd, unlike gzip, is splittable. That means you could take these CSVs and read them with Spark in parallel.

Thanks for considering!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions