Writing a guide to SIMD

[workingjubilee]

We are going to want to produce a guide on SIMD in Rust that doesn't just cover the API but also covers a standard vocabulary about this, so as to hopefully standardize on a vendor-ambivalent vocabulary. We are also going to want to link to vendors, of course, but we want a way to discuss our own portable types that meshes reasonably well with Rust.

It's important to meet people where they are at here. I believe that people mostly either:

1. know much of what there is to say about SIMD but may mostly have experience with particular vendor intrinsics or even a more abstract model like GPGPU... for them, we want an option to basically read the "front and back pages" because they may be used to certain terms that are not reflected in our API
2. have an idea of what SIMD is but mostly as an awareness of e.g. autovectorization, they may know some basics but don't have experience with things like (GP)GPU or intrinsic function programming
3. have no idea whatsoever, and may not even know why they might care!

• Standard vocabulary will likely include these terms or alternates to them:

  • SIMD
  • vector
  • vector register
  • scalar (by contrast)
  • vectorize / autovectorization
  • "lane"?
  • "field"?
  • "element"?
  • "intrinsics"?

• We will likely need to also cover subtle differences (or the absence thereof)

  • intrinsics vs. instructions
  • instructions vs. operations
  • lane vs. field vs. element
  • scalar vs. vector

• Differences between CPU SIMD and GPU SIMT

• Differences between SIMD vectors and SPMD threads

• Differences between "wide vectors" (x86 SSE/AVX/AVX512, Arm Neon, etc.) and "vector architectures" (Cray, RISCV-V, Arm SVE)

• How to get the most out of Rust SIMD code

  • feature levels, including is_x86_feature_detected!
  • #[target_feature]
  • #[cfg(target_feature)]
  • how to do "multiversioning"
  • multiversioning, why or why not?
  • when do you use std::arch instead?

And we might want to talk briefly about (abstract) tensors vs. (SIMD) vectors due to matrix multiplication being related to this.

[programmerjake]

Additional things to potentially include is Cray-style vectors (they allow dynamically picking any arbitrary vector length below some hardware limit), there's probably a better name. This is important since Cray-style vectors are used on RISC-V V, Libre-SOC, and probably more. Also, the first time I heard of vectorization, it was talking about Cray-style vectors, so, even if we don't implement them yet, we should at least mention them.

[programmerjake]

IIRC, ARM SVE is somewhere in between fixed-length SIMD and Cray-style vectors.

[workingjubilee]

As far as I understand it, but I am by no means an expert, Cray vectors worked intrinsically differently than RISCV-V vectors (that is, they did not have "vector registers" in our sense, so the implications on performance, speed, and et cetera were very different). I definitely would be happy to include discussion of RVV, SVE, and other dynamically sized vectors though, especially when we actually start compiling to them.

[programmerjake]

Cray totally has vector registers. it was the computer that Cray worked on right before that that didn't. (I think it was the STAR-100, though could be way off)

[programmerjake]

Quoting the wikipedia article on the Cray-1:

The vector system added another eight 64-element by 64-bit vector (V) registers, as well as a vector length (VL) and vector mask (VM).

Note that RISC-V V is designed to be able to work more like Cray computers did (though they don't have to), in that vector operations might take many clock cycles and the first vector elements might be finished waay before the last.

[workingjubilee]

Ahhhh. Noted, then, I agree that Cray vectors and RISCV-V vectors are similar enough at that point.

[dylanede]

Should this guide mention something about using SIMD in conjunction with runtime CPU feature detection? As far as I can tell, it seems like the only way to do that is to move all of an application's SIMD code into a separate crate, compile multiple versions of that crate with different RUSTFLAGS, then link to all of those versions from the main application and decide which version of each function to call based on runtime CPU feature detection?

[BurntSushi]

@dylanede For x86, you can use is_x86_feature_detected. And you can tag functions with #[target_feature(enable = "...")]. There's no need to put things into separate crates or use RUSTFLAGS.

@workingjubilee I'm not sure if this is in scope or not, but it would be good to have practical matters discussed in the guide. The runtime CPU feature detection thing is actually a good example, since I see it come up a lot. There's a lot of confusion around that, how it works and whether they should be using that or compile time CPU feature detection. (Which is often easier to do, but less generally useful when writing libraries.)

Another practical matter is to how to tell whether your target feature settings are correct or not. My heuristic for this is to look at a profile and see whether trivial vendor intrinsics are getting inlined or not. If they're not, it's likely because of a target feature mismatch between caller and callee.

[dylanede]

@BurntSushi thanks!

[workingjubilee]

Added relevant desirables to the list. Such advice is absolutely relevant and in-scope, though it had come up in separate issues so I had not yet gotten around to adding it here. "Multiversioning" as a Rust-level code feature is not in scope for the overall Portable SIMD API project, i.e. we will make it easier, but only by providing a portable API. But "how to do it in your own code" is in scope for this guide, as it is essentially supposed to be "what is this library even about and how do I use it?"

Some theory is required to be able to understand the meta-level concerns and make any use of the crate, but also some practical details should be covered as well. Most of the practical details will be in the API documentation itself, but anything that is "up one meta level" relative to the crate can go in the overall guide.