diff options
Diffstat (limited to 'content/blog')
| -rw-r--r-- | content/blog/2019-02-10-handmade-rust-1-allocators/aligned_alloc.png | bin | 0 -> 38684 bytes | |||
| -rw-r--r-- | content/blog/2019-02-10-handmade-rust-1-allocators/index.md | 149 | ||||
| -rw-r--r-- | content/blog/2019-02-11-handmade-rust-2-unq.md | 107 | ||||
| -rw-r--r-- | content/blog/2019-03-23-handmade-rust-3-containers.md | 347 | ||||
| -rw-r--r-- | content/blog/2019-05-04-handmade-rust-4-vulkan-bindings.md | 724 | ||||
| -rw-r--r-- | content/blog/_index.md | 6 |
6 files changed, 1333 insertions, 0 deletions
diff --git a/content/blog/2019-02-10-handmade-rust-1-allocators/aligned_alloc.png b/content/blog/2019-02-10-handmade-rust-1-allocators/aligned_alloc.png Binary files differnew file mode 100644 index 0000000..92ef9de --- /dev/null +++ b/content/blog/2019-02-10-handmade-rust-1-allocators/aligned_alloc.png diff --git a/content/blog/2019-02-10-handmade-rust-1-allocators/index.md b/content/blog/2019-02-10-handmade-rust-1-allocators/index.md new file mode 100644 index 0000000..dec7054 --- /dev/null +++ b/content/blog/2019-02-10-handmade-rust-1-allocators/index.md @@ -0,0 +1,149 @@ ++++ +title = "Handmade Rust Part 1: Introduction & Allocators" +[taxonomies] +tags = ["Rust", "Programming"] ++++ + +Welcome to Handmade Rust, a series (hopefully) where I will be developing a Vulkan rendering engine in Rust the Handmade way. By this, I mean using no external libraries, not even the Rust standard library, only the core lib. I am doing this mainly for my own enjoyment but also because I want to get better at writing Rust, and sometimes the best way to really understand something is to just do it yourself. The project will be available on GitHub at [stevenlr/HandmadeRust](https://github.com/stevenlr/HandmadeRust). + +<!-- more --> + +The first step will be to build a foundation library for memory allocation, containers, and other utilities that are not provided by the core lib. + +The `Allocator` trait +================== + +Let's begin with memory allocation. We'll need to somehow be able to acquire memory. Usually this is done using some kind of global allocator (think `malloc`, the `new` keyword, etc.). However for performance reasons, it is really important to be able to control how memory is allocated, this justifies writing specialized allocators for different tasks. For instance you can have a global allocator which can be slow but general purpose, or a temporary allocator for when you need small allocations very quickly, or a composable allocator that uses different underlying allocators depending on the requested size. This is quite different from how most languages choose to do dynamic memory allocation, where nobody cares when and how the memory is allocated. + +First of all we'll need a `Layout` that defines the size and alignment of the memory we want to allocate. We'll also define the `Allocator` trait, which is going to be underwhelmingly unsurprising. + +```rust +pub struct Layout +{ + pub size: usize, + pub align: usize, +} + +impl Layout +{ + pub fn new(size: usize) -> Self + { + Self + { + size, + align: 4, + } + } + + pub fn from_type<T>() -> Self + { + Self + { + size: size_of::<T>(), + align: align_of::<T>(), + } + } +} + +pub trait Allocator +{ + unsafe fn alloc(&mut self, layout: Layout) -> Option<*mut c_void>; + unsafe fn dealloc(&mut self, ptr: *mut c_void); +} + +``` + +Implementing an allocator +============================ + +Usually a container that is allocation-aware will be defined like this: + +```rust +struct Container<A: Allocator> +{ + alloc: A, + // ... +} +``` + +Now an issue arises from the fact that the `alloc` and `dealloc` methods from the `Allocator` trait use mutable references to `self`: in Rust, there can either be one mutable reference to an object or many immutable references. In other words you have to choose between mutability and aliasing. In the current configuration, this means that we have to create one allocator per container, which kind of defeats the purpose of having them is the first place. + +The solution here is to implement the `Allocator` trait for an immutable reference and then use interior mutabilty when necessary. Then the `A` type parameter from our previous type can be `&MyAllocator` and problem solved! + +```rust +impl Allocator for &MyAllocator +{ + // ... +} + +fn foo() +{ + let my_allocator = MyAllocator::new(); + let container1 = Container::new(&my_allocator); + let container2 = Container::new(&my_allocator); +} +``` + +You can see the example allocator I implemented using Windows' `HeapAlloc` and `HeapFree` there: [fnd::alloc::Win32HeapAllocator](https://github.com/stevenlr/HandmadeRust/blob/60d301e4163e85870b8d13b1b1c3ad5528788a0c/fnd/src/alloc/win32_heap.rs). + +Aligned memory +========================= + +It is often desirable to request aligned memory. For this, we can provide two convenience methods in the `Allocator` trait. + +```rust +impl Layout +{ + // ... + + pub fn align_up(&self, i: usize) -> usize + { + let p = i + self.align - 1; + return p - (p % self.align); + } +} + +pub trait Allocator +{ + // ... + + unsafe fn alloc_aligned(&mut self, layout: Layout) -> Option<*mut c_void> + { + let actual_size = layout.size + layout.align - 1 + size_of::<usize>(); + let ptr = match self.alloc(Layout::new(actual_size)) + { + Some(p) => p as usize, + None => return None, + }; + + let aligned_ptr = layout.align_up(ptr + size_of::<usize>()); + let actual_ptr_ptr = aligned_ptr - size_of::<usize>(); + + write_unaligned(actual_ptr_ptr as *mut usize, ptr); + + Some(aligned_ptr as *mut c_void) + } + + unsafe fn dealloc_aligned(&mut self, ptr: *mut c_void) + { + let aligned_ptr = ptr as usize; + let actual_ptr_ptr = aligned_ptr - size_of::<usize>(); + let actual_ptr = read_unaligned(actual_ptr_ptr as *const usize); + self.dealloc(actual_ptr as *mut c_void); +} +``` + +The way this works is by allocating excess memory: We know that by allocating an excess of `align - 1` bytes, we'll be able to properly align the returned pointer, and still have enough room to fit the size that was requested. + +However when deallocating, we have to remember what was the actual pointer that was returned by the allocation in order to deallocate it. To do this, we also allocate an excess of 8 bytes (the size of `usize` in a 64 bits environment) so we can fit the pointer to the original memory right before the returned aligned pointer. + + + +In the above (wonderful) drawing, the user requests a memory space of size `size`. We allocate `size + 8 + align - 1` bytes (the entire black rectangle). Its address is `ptr`. We offset it by 8 and align this up to `align`, which gives us `aligned ptr` (red). This is the pointer we return to the user. Finally we offset this aligned pointer by -8 bytes and write `ptr` there. + +To deallocate we simply offset the pointer by -8 bytes which gives us the address of `ptr`. We read that and this gives us the pointer we have to deallocate. + +Wrap up +=================== + +That's it for today. If you have any question or remark about this post, you can tweet at me at [@steven_lr](https://twitter.com/steven_lr). Next time we'll work on a `Box` replacement using our `Allocator` trait. diff --git a/content/blog/2019-02-11-handmade-rust-2-unq.md b/content/blog/2019-02-11-handmade-rust-2-unq.md new file mode 100644 index 0000000..cd6b213 --- /dev/null +++ b/content/blog/2019-02-11-handmade-rust-2-unq.md @@ -0,0 +1,107 @@ ++++ +title = "Handmade Rust Part 2: Unq, an allocator-aware Box" +[taxonomies] +tags = ["Rust", "Programming"] ++++ + +In the Rust standard library, `Box` is a RAII wrapper for an object on the heap. It's actually a special type that's not implemented purely in the library, but also use special features called lang items. It uses the global allocator to allocate its memory. We want a similar type that also has an allocator associated to it. We'll call it `Unq`, which mirror C++'s `unique_ptr`. + +<!-- more --> + +Defining `Unq` +===================== + +```rust +pub struct Unq<T: ?Sized, A: Allocator> +{ + ptr: NonNull<T>, + alloc: A, +} +``` + +`T` is the wrapped type. It is optionally sized because we want to allow trait objects to be wrapped in there. For those unfamiliar with traits, they are basically Rust's interfaces, but they allow both static and dynamic dispatch. A trait object is an object and its accompanying virtual functions table that allows dynamic dispatch. They use the syntax `dyn Trait`. For example, `&mut dyn Trait` and `Unq<dyn Trait, A>` are both trait objects. + +Creating and dropping a `Unq` +=================== + +When creating a `Unq`, we pass to it a value of type `T` and an allocator. The necessary memory is allocated. Then we can write the value using `core::ptr::write`, which moves a value into a potentially uninitialized memory location. + +*Edit 2019-03-25*: I originally used `forget(replace(ptr, value))` to write the value, but [@myrrlyn](https://twitter.com/myrrlyn) pointed out that `ptr::write` is actually a better choice here because it doesn't require initialized memory like `replace` does. + +```rust +impl<T, A: Allocator> Unq<T, A> +{ + pub fn new(value: T, mut alloc: A) -> Self + { + let mut ptr = /* Allocate memory */; + + unsafe + { + core::ptr::write(ptr.as_mut(), value); + } + + // ... + } +} +``` + +Note that this means that the value first has to live on the stack before being copied to the heap. Maybe in the future we'll implement a method to create a `Unq` from a type that implements `Default` in place. + +When dropping the `Unq`, before deallocating the memory, we first want to drop the contained value. To do so we could use `core::mem::drop`, but we would first need to copy the value, which we don't want. Instead we can use `core::ptr::drop_in_place` to drop the value at a given memory location. + +```rust +impl<T: ?Sized, A: Allocator> Drop for Unq<T, A> +{ + fn drop(&mut self) + { + unsafe + { + drop_in_place(self.ptr.as_ptr()); + // Deallocate + } + } +} +``` + +Making `Unq` usable +========================= + +We can now create a `Unq`, but then there is no way to access the underlying value. To do so, we'll implement `core::ops::Deref` and `core::ops::DerefMut`. Those are basically operator overloadings that tells the compiler what to do when dereferencing a `Unq`. By returning `self.ptr.as_ref()` and `self.ptr.as_mut()` respectively, we allow the user to access members and methods of the wrapped object simply by writing `my_unq.member = new_value;` or `my_unq.do_something();`. + +Finally, we need `Unq` to be able to do dynamic dispatch, so we need to tell the compiler that `Unq` is covariant over `T`, meaning that when `T` is a subtype of `U`, `Unq<T>` is a subtype of `Unq<U>`. We'll be able to write something like this: + +```rust +trait MyTrait +{ + fn do_something(&self); +} + +struct MyObject; + +impl MyTrait for MyObject +{ + fn do_something(&self) {} +} + +fn foo() +{ + let alloc = ...; + let my_unq : Unq<dyn MyTrait, ...> = Unq::new(MyObject{}, ...); + my_unq.do_something(); +} +``` + +Do to so, we need to implement `core::ops::CoerceUnsized` which is not stable (so we have to use the nightly toolchain). + +```rust +impl<T: ?Sized + Unsize<U>, U: ?Sized, A: Allocator> + CoerceUnsized<Unq<U, A>> + for Unq<T, A> {} +``` + +In this snippet, `T` would be `MyObject` and `U` `dyn MyTrait`. `T: Unsize<U>` basically means that `T` can be seen as the dynamically sized type `U`. I won't go into too much details because I'm not sure I understand all of it myself, but there are plenty of resources to learn about DSTs. + +Wrap up +===================== + +The current implementation of `Unq` is available here: [fnd::unique::Unq](https://github.com/stevenlr/HandmadeRust/blob/fb4084420c3263d800b1674527403095ddea20ea/fnd/src/unique.rs). I'm sure we'll have to add some stuff to it in the future, but this is the minimal implementation for now. Next time we'll implement a growable array type. As always don't hesitate to ping me on [Twitter](https://twitter.com/steven_lr). diff --git a/content/blog/2019-03-23-handmade-rust-3-containers.md b/content/blog/2019-03-23-handmade-rust-3-containers.md new file mode 100644 index 0000000..e9b48c7 --- /dev/null +++ b/content/blog/2019-03-23-handmade-rust-3-containers.md @@ -0,0 +1,347 @@ ++++ +title = "Handmade Rust Part 3: Containers" +[taxonomies] +tags = ["Rust", "Programming"] ++++ + +The most commonly used kinds of containers are arrays and maps. Pretty much any other container type can be built using those two, so that's what we'll build today! + +The Rust standard library has `Vec` which is a dynamic array and `HashMap` and `BTreeMap` which are two kind of maps, the first one relying on the key being hashable, and the second one relying on the key being sortable. + +The hash map has the cost of hashing the key but then lookup is constant time (in theory at least), whereas the ordered map only compares the keys, but lookup is logarithmic time. I chose to implement a hash map because it's the one I find myself using most of the time, but ordered map are interesting as well. + +Of course, just like for `Unq`, we won't be making simple replacements, instead we'll be making the most minimal containers necessary for now and add features later as needed, but we'll be make them allocator aware. + +As always, full source code is available at [github.com/stevenlr/HandmadeRust](https://github.com/stevenlr/HandmadeRust). + +<!-- more --> + +`RawArray`: the dumbest container +================= + +All containers will somehow need to allocate aggregates of data. We'll be making the `RawArray` type for allocating contiguous regions of memory, resizing them, and then deallocating the memory. This container will not be taking care of initializing memory, calling destructors, or anything like that. + +```rust +pub struct RawArray<T, A: Allocator> +{ + pub ptr: *mut T, + pub capacity: usize, + alloc: A, + _phantom: PhantomData<T>, +} +``` + +Let's analyze this. First we have two type parameters. `T` is the type contained in the array. `A` is an allocator, as defined in the first part of this series. `ptr` is the pointer to the memory region containing our data. `capacity` is the size of that region. Finally, `_phantom` is used to tell the compiler that this struct acts like it contains some data of type `T`. + +When creating a `RawArray`, we simply set the pointer to null and the capacity to 0. Unless `T` is zero-sized. In this case, we set the capacity to the max value of a `usize`. That way, the array will never grow, because we don't need it to grow. + +We also define a `reserve` method that takes in a new capacity. If the current capacity is larger that the new capacity, we don't do anything. Otherwise we allocate a new memory region of the desired size, copy the old data using `ptr::copy_from`, and dealloc the old region. Simple! Note that I don't know how this container will behave with the `Pin` type, and actually I don't care for now. + +Finally, when dropped, the raw array simply deallocates its memory if its pointer is not null. + +`Array`: the less dumb container +================== + +Now that we can allocate raw arrays, we can define the proper `Array` type that can be pushed to and poped from. + +```rust +pub struct Array<T, A: Allocator> +{ + size: usize, + buffer: RawArray<T, A>, +} +``` + +As you can see, the `Array` simply wraps a raw array and keeps track of the number of elements actually contained. Unlike `RawArray`, `Array` will guarantee that memory is always properly initialized for the `size` first elements of the memory region, and it will drop the content of the elements when needed. + +First let's define the `push` operation. It needs to grow the container when its max capacity is reached, and then write the new value to the end of the used region. + +```rust +pub fn push(&mut self, value: T) +{ + if self.size == self.buffer.capacity + { + self.grow_auto(); + } + + unsafe + { + self.buffer.ptr.offset(self.size as isize).write(value); + } + + self.size += 1; +} +``` + +`grow_auto` is a simple method that calls `reserve` on the raw array and doubles the current capacity (or sets it to 1 if the array was empty). Note that when writing the value, we don't use `write_unaligned` because we made sure that `RawArray` properly aligns its memory region. + +```rust +pub fn pop(&mut self) -> Option<T> +{ + if self.size == 0 + { + None + } + else + { + let value = unsafe + { + self.buffer.ptr.offset((self.size - 1) as isize).read() + }; + + self.size -= 1; + Some(value) + } +} +``` + +Nothing very exciting here. Compared to C++, `pop` not only resizes the array, but at the same time returns the element. In C++ you need to access the last element, then `pop` to have the same effect, and you also need to check if the array is empty or not. We can avoid all of this simply because we have the `Option` type! + +```rust +pub fn clear(&mut self) +{ + if needs_drop::<T>() + { + unsafe + { + for i in 0..self.size + { + drop_in_place(self.buffer.ptr.offset(i as isize)); + } + } + } + + self.size = 0; +} +``` + +When clearing, we need to drop the elements contained in the array. We do this by calling `drop_in_place` which calls `drop` without needing any copy. This is of course `unsafe`. Then, drop for the array can be implemented by simply calling `clear`. + +Finally, let's talk about `resize`. There are two cases here: + + - When resizing to a smaller size, we need to drop all elements above the new size. + - When resizing to a bigger size, we may need to reserve to the new size, and then to write a default value to each new slot. For this `T` needs to implement `Clone`. However we don't really want our array to only be usable with cloneable types. Instead we'll provide a resize method that takes in a closure that generates new `T`s. When `T` is cloneable, that closure can simply be `|| value.clone()`. We could also provide a `resize_default` method that only takes in the new size when `T` implements `Default`. + + ```rust +pub fn resize_with<F>(&mut self, new_size: usize, f: F) + where F: Fn() -> T +{ + if new_size < self.size && needs_drop::<T>() + { + for i in new_size..self.size + { + unsafe + { + drop_in_place(self.buffer.ptr.offset(i as isize)); + } + } + } + else if new_size > self.size + { + if new_size > self.buffer.capacity + { + self.reserve(new_size); + } + + for i in self.size..new_size + { + unsafe + { + self.buffer.ptr.offset(i as isize).write(f()); + } + } + } + + self.size = new_size; +} + +pub fn resize(&mut self, new_size: usize, value: T) + where T: Clone +{ + self.resize_with(new_size, || value.clone()); +} +``` + +Now, that's all fine, but we'd like our array to behave like any other contiguous region of memory: we want to index in it (possibly even index ranges), iterate over it, and basically be able to do anything a slice can. There is a very simple way of doing this. + +```rust +impl<T, A: Allocator> Deref for Array<T, A> +{ + type Target = [T]; + + #[inline] + fn deref(&self) -> &Self::Target + { + unsafe + { + slice::from_raw_parts(self.buffer.ptr, self.size) + } + } +} +``` + +Also implement `DerefMut` and voilà! Our array can behave just like a slice, and that was free! + +Actually we may want to implement `Index` so we don't have to pay for creating a slice everytime we need to index. Careful generated assembly examination will tell if that's needed. + +`HashMap`s need to `Hash` +======================== + +Before implementing the hash map, we need to somehow be able to hash some data. The Rust core library defines the `Hash` trait. This trait indicates that a type can be hashed using a `Hasher`. No hasher is provided by default in the core library so let's make one! I chose to implement [SipHash 2-4](https://en.wikipedia.org/wiki/SipHash) because that's what the standard library uses, and it looked simple enough. + +I won't bother you with the implementation, just know that the `Hasher` trait requires two methods: + + - `write` to run a few rounds of hashing on a slice of bytes + - `finish` to actually finish hashing and return the hash. + +It is also a good idea for the hasher to implement `Default` so we can use `BuildHasherDefault` which implements `BuildHasher`, basically a hasher factory. + +`HashMap`, finally +==================== + +Now we have everything we need to build our hash map. I won't go into details of the implementation, but just go through some interesting implementation points. + +I chose to implement something that resembles [Google's swiss table](https://www.youtube.com/watch?v=ncHmEUmJZf4). The basic idea is that you have a first array that stores 7 bits of the hash of each element, the 8th bit is used to know when an element is present. When looking up a slot, this is used to quickly find a suitable slot without having to iterate though the main array that contains the keys and values. Yay for cache locality. Note that with some SIMD magic, this can be made even faster. In my implementation, this 8 bits hint is called a `ShortBucket`. + +```txt + 1xxx xxxx = occupied, xxx xxxx = first 7 bits of the key hash. + 0000 0001 = free + 0000 0010 = deleted, required for linear probing +``` + +In my implementation, the main array is defined as containing `Option<Bucket>`. `Bucket` itself contains the hash of the key (so we don't need to re-hash it), the key, and the value. This is not really efficient, but I didn't want to have to deal with uninitialized values, and `Option` takes care of that. Note that I could have used `RawArray` instead, but I'd have had more boilerplate code to write. I'll probably revise all of this one day, but for now it works well enough. + +```rust +struct Bucket<K, V> +where + K: Sized + Eq + Hash, + V: Sized, +{ + hash: u64, + key: K, + value: V, +} + +struct HashMapInner<K, V, A> +where + K: Sized + Eq + Hash, + V: Sized, + A: Allocator + Clone, +{ + shortb: Array<ShortBucket, A>, + buckets: Array<Option<Bucket<K, V>>, A>, +} + +pub struct HashMap<K, V, A> +where + K: Sized + Eq + Hash, + V: Sized, + A: Allocator + Clone, +{ + alloc: A, + hash: BuildHasherDefault<SipHash>, + inner: HashMapInner<K, V, A>, +} +``` + +The reason for having `HashMapInner` is for resizing: the simplest way of doing that is to create a new `HashMapInner` with the new capacity, then iterate over all elements and swap them into the new container, and finally replace the old `HashMapInner` with the new one. + +For finding a suitable slot, we define the `find_bucket` method. For a given key, it returns either the matching slot, or the first one that the key can be inserted into. + + - It first hashes the key to find the start position for linear probing, then iterates over the short buckets array. + - If the first 7 bits match the short bucket, we check the full hash in the buckets array. If that matches too we check the keys for equality. If that matches too, we found the correct bucket, we're done. + - If we find a deleted slot, we mark it as acceptable for insertion if none was marked as such before and then continue. + - If we find a free slot, linear probing is done. If we had an acceptable bucket for insertion, we return it, otherwise we return the current one. + - If we reach back the starting index, we're done and we return the acceptable slot for insertion (which may be none). + + This method alone is enough for lookup, insertion, deletion, etc. + + When we want to count the elements in the map, we just need to iterate over the short buckets and count the ones that are occupied. + +Currently, resizing only happens when we run out of space. Ideally it should be based on load factor. + +There are many things that this hash map doesn't do such as iterating over elements, but for now that's enough... + +`HashSet` is free +========================= + +Since `Array` supports zero-sized types, ... + +```rust +pub struct HashSet<T, A> +where + T: Sized + Eq + Hash , + A: Allocator + Clone, +{ + map: HashMap<T, (), A>, +} +``` + +... done! (Well you still need to implement a few methods but you know...) + +Dynamic `String`s +======================= + +Dynamic strings just wrap a `Array<u8, A>`. We implement a method for creating them from `str`. + +```rust +pub fn from_str(s: &str, alloc: A) -> Self +{ + let slice = s.as_bytes(); + let mut buf = Array::new(alloc); + buf.resize(slice.len(), 0); + + unsafe + { + copy_nonoverlapping(s.as_ptr(), buf.as_mut_ptr(), slice.len()); + } + + Self { buf } +} +``` + +Just like `Array` can be coerced to a slice, we would like this string to coerce to a `str` so we can do anything `str` can with `String`. We implement `Deref` and `DerefMut`, and a `as_str` method. + +```rust +impl<A: Allocator> String<A> +{ + #[inline] + pub fn as_str(&self) -> &str + { + self + } +} + +impl<A: Allocator> Deref for String<A> +{ + type Target = str; + + #[inline] + fn deref(&self) -> &str + { + unsafe + { + str::from_utf8_unchecked(&self.buf) + } + } +} +``` + +Now we can very easily implement useful traits such as `Display`, `Hash` or `PartialEq`: + +```rust +impl<A: Allocator> hash::Hash for String<A> +{ + fn hash<H: hash::Hasher>(&self, h: &mut H) + { + hash::Hash::hash(self.as_str(), h); + } +} +``` + +And that's it for this container, for now. We'll probably want to be able to concatenate and do other mutating operations on it eventually, but the basic structure is here. + +Wrap up +=============== + +The containers we have implemented so far should be enough for most things we need. Of course they lack some very useful features at the moment, but we'll get back to them as needed. I don't suggest anybody use those in production because, me not being an expert at Rust, there probably are many bugs remaining. I guess time will tell. Next time we'll take a look at initializing Vulkan. Until then, you can contact me on [Twitter](https://twitter.com/steven_lr). diff --git a/content/blog/2019-05-04-handmade-rust-4-vulkan-bindings.md b/content/blog/2019-05-04-handmade-rust-4-vulkan-bindings.md new file mode 100644 index 0000000..07f5740 --- /dev/null +++ b/content/blog/2019-05-04-handmade-rust-4-vulkan-bindings.md @@ -0,0 +1,724 @@ ++++ +title = "Handmade Rust Part 4: Generating Vulkan bindings" +[taxonomies] +tags = ["Rust", "Programming", "Vulkan"] ++++ + +Vulkan is a C API so we'll need some kind of bindings to be able to use it in Rust. The API is defined in a XML file distributed by Khronos. This file describes the structs, enums, constants, and functions for each version of the API, and all published extensions. The functions can then be loaded from the Vulkan dynamic library and other functions from the API. + +<!-- more --> + +However using a raw C API isn't easy in Rust because it requires using a lot of unsafe code. This is why we'll also generate builders for all structs so we can for instance fill in pointer/size pairs using slices, but we'll also generate methods that return Rust's `Result`s and take in Rust-friendly types like references instead of raw C types. Finally we'll also generate loaders so we don't have to manually load the function we need. + +Parsing the spec +======================= + +The XML spec for Vulkan is notoriously annoying to use outside of the C/C++ ecosystem because it's basically C code with tags around it. For instance, good luck parsing this: + +```xml +<member>const <type>void</type>* <name>pNext</name></member> +``` + +For my purpose, I wanted to transform this awkward representation into some kind of AST. For example, given the code above, this is what the parser outputs: + +```xml +<member name="pNext" optional="False"> + <pointer const="False"> + <type const="True">void</type> + </pointer> +</member> +``` + +This is much easier to use for generating code in any other language than C. + +On a higher level, the parser is also able to resolve type dependencies. The goal is to tell it what version of the API and what extensions to use. With that, the parser visits all necessary types and commands, possibly extends some enums, and this is what is then written to the output file. + +With that in mind, I won't go into too much details into how the parser works. Just know that it's written in Python and that it was both incredibly boring and annoying to do, I never want to do it again! I'll now show a few examples of the generated XML file. + +The source code for the parser and generators is available on [GitHub](https://github.com/stevenlr/VkXml). + +Type aliases +--------------- + +```xml +<alias name="VkInstanceCreateFlags" type="VkFlags"/> +<base name="VkDeviceSize" type="uint64_t"/> +<handle name="VkBufferView" type="uint64_t"/> +``` + +The `alias` tag is a Vulkan-to-Vulkan type alias while `base` is a Vulkan-to-native alias. In this instance `VkInstanceCreateFlags` is an alias for `VkFlags` because it is an empty enum. + +Commands +------------- + +```xml +<command name="vkGetPhysicalDeviceSurfacePresentModesKHR" success-codes="VK_SUCCESS,VK_INCOMPLETE" type="instance"> + <return-type> + <type const="False">VkResult</type> + </return-type> + <arg name="physicalDevice" optional="False"> + <type const="False">VkPhysicalDevice</type> + </arg> + <arg name="surface" optional="False"> + <type const="False">VkSurfaceKHR</type> + </arg> + <arg name="pPresentModeCount" optional="True"> + <pointer const="False"> + <type const="False">uint32_t</type> + </pointer> + </arg> + <arg length="pPresentModeCount" name="pPresentModes" optional="True"> + <pointer const="False"> + <type const="False">VkPresentModeKHR</type> + </pointer> + </arg> +</command> +``` + +`command` define their name of course, but also their success codes for functions that return a `VkResult` (this will be used for error checking) and their type, which defines how the associated function pointer must be loaded. + +The command has one `return-type` and many `arg`s with a type AST inside. + +Each argument defines whether it is optional or not (this is useful for pointers) and pointer arguments can indicate what argument defines their length. + +- For instance if `length` is `pPresentModeCount`, it means that the `pPresentModeCount` argument defines the length of the array pointed to by the pointer. +- If the length is `pCreateInfo::somethingCount`, it means that the lenth of the array is the `somethingCount` field from the struct in the `pCreateInfo` argument. + +These info are useful to generate functions that take in slices instead of pointer/length pairs, which is much more idiomatic in Rust. Also, whether the pointer type is const or not indicates whether it is an input argument or not. + +Enums +----------- + +```xml +<enum name="VkPresentModeKHR" type="enum"> + <entry name="VK_PRESENT_MODE_IMMEDIATE_KHR">0</entry> + <entry name="VK_PRESENT_MODE_MAILBOX_KHR">1</entry> + <entry name="VK_PRESENT_MODE_FIFO_KHR">2</entry> + <entry name="VK_PRESENT_MODE_FIFO_RELAXED_KHR">3</entry> +</enum> +``` + +Not much to say here, everything is pretty obvious. :) + +Bit flags +-------------- + +```xml +<bitmask flags="VkShaderStageFlagBits" name="VkShaderStageFlags"/> +<enum name="VkShaderStageFlagBits" type="bitmask"> + <entry name="VK_SHADER_STAGE_VERTEX_BIT">1</entry> + <entry name="VK_SHADER_STAGE_TESSELLATION_CONTROL_BIT">2</entry> + <entry name="VK_SHADER_STAGE_TESSELLATION_EVALUATION_BIT">4</entry> + <entry name="VK_SHADER_STAGE_GEOMETRY_BIT">8</entry> + <entry name="VK_SHADER_STAGE_FRAGMENT_BIT">16</entry> + <entry name="VK_SHADER_STAGE_ALL_GRAPHICS">31</entry> + <entry name="VK_SHADER_STAGE_COMPUTE_BIT">32</entry> + <entry name="VK_SHADER_STAGE_ALL">2147483647</entry> +</enum> +``` + +For bit flags, we associate a top-level `bitmask` tag which indicates the association between the enum that defines the flags and the type that combines those flags. + +In Vulkan, the flags enum are always called `XxxFlagBits` and the combined flags `XxxFlags`. + +Function pointers +--------------------- + +```xml +<function-pointer name="PFN_vkFreeFunction"> + <return-type> + <type const="False">void</type> + </return-type> + <arg name="pUserData" optional="False"> + <pointer const="False"> + <type const="False">void</type> + </pointer> + </arg> + <arg name="pMemory" optional="False"> + <pointer const="False"> + <type const="False">void</type> + </pointer> + </arg> +</function-pointer> +``` + +Function pointers follow the exact same syntax as commands. + +Constants +-------------- + +```xml +<integer-constant name="VK_SUBPASS_EXTERNAL" size="32">4294967295</integer-constant> +<real-constant name="VK_LOD_CLAMP_NONE" size="32">1000.0</real-constant> +<string-constant name="VK_KHR_WIN32_SURFACE_EXTENSION_NAME">VK_KHR_win32_surface</string-constant> +``` + +Structures and unions +----------- + +```xml +<struct name="VkInstanceCreateInfo"> + <member default_value="VK_STRUCTURE_TYPE_INSTANCE_CREATE_INFO" name="sType" optional="False"> + <type const="False">VkStructureType</type> + </member> + <member name="pNext" optional="False"> + <pointer const="False"> + <type const="True">void</type> + </pointer> + </member> + <member name="flags" optional="True"> + <type const="False">VkInstanceCreateFlags</type> + </member> + <member name="pApplicationInfo" optional="True"> + <pointer const="False"> + <type const="True">VkApplicationInfo</type> + </pointer> + </member> + <member name="enabledLayerCount" optional="True"> + <type const="False">uint32_t</type> + </member> + <member length="enabledLayerCount" name="ppEnabledLayerNames" optional="False"> + <pointer const="False"> + <pointer const="True"> + <type const="True">char</type> + </pointer> + </pointer> + </member> + <member name="enabledExtensionCount" optional="True"> + <type const="False">uint32_t</type> + </member> + <member length="enabledExtensionCount" name="ppEnabledExtensionNames" optional="False"> + <pointer const="False"> + <pointer const="True"> + <type const="True">char</type> + </pointer> + </pointer> + </member> +</struct> +``` + +On their top-level tag, structs can define an `extends` attribute which indicates what structures can receive it in the `pNext` chain. + +Another interesting note here is that pointer fields can have a `length` attribute, which serves the exacts same purpose as on function arguments. + +Unions are the exact same thing as structures, so I won't detail them here. + +Generating Rust types +======================== + +Handles +-------------- + +```rust +#[repr(transparent)] +#[derive(Default, Copy, Clone, PartialEq, Eq)] +pub struct VkPhysicalDevice(usize); +impl VkPhysicalDevice +{ + #[inline] + pub fn null() -> Self + { + Self(0) + } + + #[inline] + pub fn from_raw(r: usize) -> Self + { + Self(r) + } + + #[inline] + pub fn as_raw(&self) -> usize + { + self.0 + } +} +``` + +Handles are generated as a newtypes with the `transparent` attribute, which guarantees that it has the exact same layout as the underlying type, allowing us to pass it to the API directly. We also generate a few helper methods. + +Enums +----------- + +```rust +#[repr(transparent)] +#[derive(Default, PartialOrd, Copy, Clone, Ord, PartialEq, Eq, Hash)] +pub struct VkSystemAllocationScope(u32); +impl VkSystemAllocationScope +{ + pub const COMMAND: VkSystemAllocationScope = VkSystemAllocationScope(0); + pub const OBJECT: VkSystemAllocationScope = VkSystemAllocationScope(1); + pub const CACHE: VkSystemAllocationScope = VkSystemAllocationScope(2); + pub const DEVICE: VkSystemAllocationScope = VkSystemAllocationScope(3); + pub const INSTANCE: VkSystemAllocationScope = VkSystemAllocationScope(4); +} +``` + +Enums are also generated as newtypes with `transparent`. Each value is an associated constant. It was not possible to use an `enum` here because there can be multiple symbols with the same value, which is not allowed in Rust enums. + +Bitflags +------------ + +```rust +#[repr(transparent)] +#[derive(Default, Copy, Clone, PartialOrd, Ord, PartialEq, Eq, Hash)] |
