Conversation
|
First, on my end I need |
|
Then, my functional tests crash because your buffer here: https://github.com/juliusgeo/csimdv-rs/blob/master/src/constants.rs#L3 is too short for real-life CSV data (any file containing raw text in a cell, like press articles, will overflow). |
|
Now with the files I can test (the ones which don't overflow the buffer), |
|
@Yomguithereal I bumped the buffer size in csimdv to 1mb, which should (hopefully) be large enough to accommodate real world files. I also realized that in the benchmark I was not only counting the number of rows in the CSV file, but also the length of each cell in the rows. I ran the benchmarks from this repo on the examples from the csimdv repo, and the results align much better after that change. Seeing roughly the same speed from zero-copy and csimdv on aarch64, will have to test later on AVX-512. |
Adds csimdv to benchmark file.
On my aarch64 machine, it works without any rustflags. On x86_64 (Windows), I had to provide
RUSTFLAGS="target-cpu=native".