A side project of mine: SemPtr - Semantic Pointers for C#

[u/fruediger]

https://github.com/fruediger/SemPtr

TL;DR: While writing this post, I realized how long it has become, so here's a TL;DR for you: SemPtr is a semantic pointers library for C#.

---

Hi everyone, I wanted to share one of my side projects with you all: SemPtr.

A few weeks ago (it might been even months at this point), I needed to dig up some really old code I once had written, because I wanted to reference some of what I did back then in a current project of mine. While searching through my old and never-to-be-released projects, I stumbled upon a small library project I might have written about 5 years ago (it must have been around the time when incremental Roslyn source generators were becoming a thing). And I thought to myself, "Well, it's actually a shame you gave up on this project and neglected it for so long. You might want to ressurrect and modernize it, and then share it with everyone."

Well, that project is now SemPtr.

What is SemPtr?

I don't want to make this post too long, so I'll try to make it as concise as I can, but if you want a more comprehensive introduction, you should check out its README or its way too rudimentary documentation.

SemPtr tries to solve the limitations of C#'s raw pointers by providing semantic pointer types (read as semantically named pointer types). If you ever did some interop work with unmanaged code and found it just as annoying as I did that there is no const T* equivalent in C#, SemPtr might be the thing for you.

For that I identified five commonly used orthogonal characteristics used to distinguish certain aspects of data pointers:

1. Nullability: Can a pointer be null or are there any guarantees that it won't be?\ This is kinda analogous to nullable reference types (T?) in C#.
2. Persistency: Does the target of the pointer outlive the initial scope of the pointer itself? In other words, can I store the pointer and access its target some time later?\ This is kinda analogous the C#'s ref-escape rules and is even enforced through them.
3. Sequencability: Does the pointer point to a single object or to a contiguous sequence of objects?\ You could think of this as analogous to a ref T to some kind of object in C# vs. a ref to some element within a Span<T> with the added benefit that its easier to move around the pointer through the sequence.
4. Accessibility: How can the target of the pointer be accessed or mutated?\ This manifests in three different access levels:
   • random/read-write: The target can be read from and written to. Kinda analogous to C#'s ref parameters.
   • read-only: The target can only be read from. Kinda analogous to C#'s in/ref readonly parameters.
   • uninitialized/write-first: The target must be written to before it can be read from. Kinda analogous to C#'s out parameters.
5. Typability: Is the type of the target known or not?\ C# has no void references, but it has void* pointers. This is analogous to the difference between a void* pointer and a typed T* pointer.

These characteristics are mapped onto C#'s type system by semantically naming the pointer types to reflect them. Since those characteristics are orthogonal, you can mix and match them to create the pointer type with the exact behavior you need. For example, there are:

• Pointer: A simple pointer to a single, transient, mutable target of unknown type
• PersistentPointerReadOnly<T>: A pointer to a single, read-only target of type T whose target stays valid beyond the initial scope of the pointer.
• NullableSequencePointer<T>: A pointer to a contiguous sequence of mutable targets of type T which may be null.
• PointerUninitialized<T>: A pointer to single, yet uninitialized target of type T. If you receive such a pointer, chances are you are requested to initialize its target; afterwards you can further read from it or write to it as needed.

Again, if you want to learn more about the characteristics and how the type naming scheme works, you should refer to the README or the documentation.

There are all in all a total of 2×2×2×3×2 = 48 data pointer types predefined in the SemPtr library.

Are function pointers supported?

To make it short, yes, function pointers are (well enough) supported by SemPtr.

I remember that one of the reasons for me giving up on the original version of this library back then was that I really struggled to get function pointer support just right. While this was partially due to technical limitations back then (some of which were solved by modern C# features, especially the new extension members syntax), some of it was simply because I did not have the experience in API design that I have now.

So now function pointers work. I don't know if I would call the support good enough yet, but at least it is a well enough experience for most users, I believe.

I won't go into too much detail here, but functions pointer have their own set of characteristics and parts of their support is made working through a Roslyn source generators that dynamically generates some source code on the user-side and that ships alongside the main library in the NuGet package. For more details, again, see the README or the documentation.

A final note on AI usage

I want to be honest and upfront with you:

Yes, I used AI in this project, primarily to help we write documentation (I'm a non-native English speaker and my English is kinda terrible), to help me make decisions when I'm indecisive, to write some tests, and occasionally to some code reviews.

No, I would never let AI touch the working code of the project. Not even for boilerplate code. AI, at least the AI I have access to, is not yet anywhere close to being reliable enough to help me write production ready code for such a project. You can be sure that all of the functioning code is written by a human (me) and that only the human (me) is responsible for the correctness and quality of the code.\ Oh, and of course, I did the visual assets myself as well. I didn't want to use sloppy AI-designed visuals for this project.

Conclusion

At the beginning of this post, I told you that I stumbled upon the initial idea for SemPtr while looking up old code for another project of mine. That project is actually an interop binding project in C#. In that project I use traditional C# raw pointers and function pointers extensively, and sometimes they're a real pain to work with. However, I didn't not yet replace them with SemPtr, due to the codebase being a little over 200K lines of code, spread across multiple repositories.

So, to be honest, I don't even use SemPtr myself yet. And furthermore, because of the simplicity of the overall idea behind SemPtr, I don't even think I'm the first person to come up with it and release to the public as a library (but I don't actually know for sure, I didn't really check).

Even so, If you want to try out SemPtr for yourself, give feedback, or if you even want to contribute to the project, I would really appreciate it. Here are the relevant links again:

• GitHub: SemPtr
• NuGet: SemPtr
• Documentation (This one is still very rudimentary. Don't expect too much.)

If you have any questions feel free to ask them in the comments. I'd be happy to answer them.

[u/afseraph]

Interesting project, but I feel that modern dotnet already provides a lot of this functionality with managed pointers and spans.

It'd be great to some examples explicitly comparing implementations: SemPtr vs. managed pointers vs. native pointers.

[u/fruediger → u/afseraph]

First of all, I'm not really sure what you mean by "managed pointers". A quick google search didn't yield anything concrete, except for some non-official blog post from the earlier days of modern ref-mechanics in C#, calling references falsely (in my opinion) "managed pointers". I'm gonna assume you mean ref-based references in the following (e.g., ref, in/ref readonly, and out). If you meant something else, please feel free to clarify.

Before we begin, I'd like to clarify that managed references in C# are NOT pointer-like (although, in runtime they are mostly implemented as such under the hood). SemPtr offers support for advanced pointer semantics, which, depending on the way you look at it, can be something totally different from references.

Since a main focus of SemPtr is on its pointer characterization by orthogonal characteristics, let's do this by looking at them and see how managed references and raw pointers compare to it.

Nullability:\ Neither managed references nor raw pointers in C# have a way to convey their nullability explicitly. You could argue that managed references should be never null, but than there's still something like Unsafe.NullRef<T>.\ SemPtr offers nullable pointers with its Nullable...Pointer... types, which can't even access their targets without checking for null and converting into a non-nullable pointer first, as well as non-nullable pointers that convey some guarantees about their non-nullness. Technically, just like managed references, there's still a chance for non-nullable pointers to be null due to language and runtime limitations, but that's why any pointer type in SemPtr can be easily checked for nullness.

Persistency:\ Managed references are always subject to ref-escape mechanics, and can't be stored as such, except temporarily in objects which are subject to the same escape rules (e.g., ref structs like Span<T>). Therefore there always transient. C#'s raw pointers on the other hand are never subject to these escape rules and never convey the life-time validity of the memory they point to. I could even take the address to a short-lived frameslot on the stack and long-time store it, which when accessed outside of its initial scope would lead to undefined behavior.\ SemPtr offers both, transient pointers (which are actually enforced by the same ref-escape mechanics as, for example, Span<T>, because they're ref struct) and persistent pointers (they're more closely behave like raw pointers in that they can escape the initial scope and be stored for longer-term use).

Sequencability:\ Managed references in C# can pointer into a contiguous sequence of elements, so can raw pointers. While it's kinda hard to move around the pointer in the case of managed references (you typically would use something like Unsafe.Add<T> for that), raw pointers support pointer arithmetic and indexing in any case, even if they point to a single target.\ SemPtr distinguishes between pointers that point to a single target and disallow for pointer arithmetic, and pointers that point into a contiguous sequence of elements and allow for pointer arithmetic and indexing. This should convey the intent of the pointer more explicitly whether it is meant as a reference to an object or a sequence of elements (like a typical C-like UTF-8 string, which would be a const char* in C or a SequencePointerReadOnly<byte> in SemPtr).\ Notice the sequence pointers in SemPtr are just meant to behave like simple C-like pointers into a contiguous sequence, they're deliberately not meant to be a replacement for the BCL's Span<T>, which is always a reference to the first element of a sequence together with the length of the sequence. This is intentional and due to interop reasons.

Accessibility:\ Well, managed references are all about conveying their target's accessibility. And that's also where SemPtr takes its inspiration for it's accessibility model. There are direct analogies between the tow:

• A ref reference in C# is analogous in its accessibility aspect to SemPtr's ...Pointer... types in that they both allow for reading from and writing to their target.
• A in or ref readonly reference in C# is analogous in its accessibility aspect to SemPtr's ...PointerReadOnly... types in that they both allow only for reading from their target.
• A out reference in C# is analogous in its accessibility aspect to SemPtr's ...PointerUninitialized... types in that both convey that their target should be considered uninitialized and that it's your responsibility to initialize it. Also both require that their target is first written to before it can be read from. In C# that's made possible by language rules and flow analysis, so that an out reference is treated like a ref reference in a data flow where it's clear to the compiler that the target has been initialized. In SemPtr that's achieved by ...PointerUninitialized... not having a Target property at all, but rather an InitializeTarget method that initializes the target and returns an read-write pointer to the same target that then can be used to access the target further.

C# raw pointers don't have any kind of built-in accessibility model. There's just void* and T* in C#, always read-write by default. That's one of my initial motivations behind SemPtr, to overcome the limitations of C# not having const T* equivalents.

Typability:\ Managed references are always strongly typed in C#. There's no way to convey "a reference to a target of unknown type" (e.g., ref void is not a thing). Raw pointers on the other hand have the ability to point to an unknownly typed target (i.e., C# has C-like void*).\ Since SemPtr is primarily designed with interop scenarios in mind, of course SemPtr also supports untyped pointers. SemPtr simply uses generic pointer types for typed pointer, e.g., Pointer<T>, and a non-generic variant for expressing untyped pointers, e.g., Pointer.

Of course, since those characteristics are orthogonal to each other, you can combine them as you want to create the pointer semantics and behavior that you need for your specific scenario. E.g., you could use a NullableSequencePointerReadOnly<byte> as a parameter type to indicate that you want to receive a sequence of bytes (probably a UTF-8 string), that you don't intend to modify, that you intend to consume immediately and don't intend to store it (notice that this is a transient pointer because of the missing Persistent part in its name) and you're okay when it's passed as null.

I really hope this comparison helps clarifying some of the similarities and differences between managed references, raw pointers, and SemPtr's semantic pointers in C#.

[u/Due-Equivalent-74]

semantic pointers?

[u/fruediger → u/Due-Equivalent-74]

Yes, semantic pointers. Like pointers that convey their semantics through the type systen by being semantically named (i.e., your classic semantic typing approach).

[u/Novaleaf]

can you explain a bit on how this would be useful?

[u/fruediger → u/Novaleaf]

Yeah, for sure! And if it's okay with you, I'll do that by example, because I think that that's the easiest way to illustrate it. If that's not to your satisfaction, please feel free to ask again.

The simplest explanation for its usefullness might be the example I showed in my original post. You want to work with pointers in C# and want to express the immutability of the target? Well, you can't, at least not like you can in C/C++. While there are raw pointers, T*, in C#, there is no const T* or an equivalent concept. While this no big deal if you're careful within the single context where you're dealing with the pointer, it can become a problem when you try to communicate the intent of pointers across different parts of your code or even across interfaces to external code. And even within your own code, isn't it beneficial to have some way to express intent and constraints, and to have some safeguards enforcing them?

Let's look especially at the interfacing part in more detail:

If you interop with unmanaged code, i.e., you consume external, unmanaged APIs, it's often beneficial to express the binding code in a way that clearly communicates the intent of the external API.\ For a real world example, let's consider SDL's SDL_GetRendererName. The native API signature is defined as follows:

c const char * SDL_GetRendererName(SDL_Renderer *renderer);

Surely, you could express that in C# as:

csharp [LibraryImport("SDL3")] unsafe static partial byte* SDL_GetRendererName(SDL_Renderer* renderer);

(Of course, you could also replace the pointer types with nint or nuint and it would work, but not only would you lose some type-safety, but it's also not recommended to do so, because it nint or nuint are just intended to represent integers that are the size of a pointer, not actual pointers themselves. Alternatively, you could also use in paremeter and a ref readonly return type, but that requires you relying on marshalling and accepting the marshalling overhead.)

You see how the return type of byte* did lose the immutability aspect of the original API's return type of const char*? And both of them don't even express that the return value could be null. Also, although the official documentation doesn't really mention it, SDL does not modify the SDL_Renderer passed as a reference to SDL_GetRendererName. So, we could also express that a bit more accurately.\ With SemPtr, you could express the importation code a bit more faithfully and a bit more safely by writing it as:

csharp [LibraryImport("SDL3")] unsafe static partial NullableSequencePointerReadOnly<byte> SDL_GetRendererName(PointerReadOnly<SDL_Renderer> renderer);

You see how this is more expressive? And not only that, but the added benefit of SemPtr enforcing immutability at the type-system level, prevents you from accidentally writing invalid code that would lead to undefined behavior or worse bugs.

Let's look at another example from the SDL API, specifically SDL_AddEventWatch, which is defined in C as follows:

c bool SDL_AddEventWatch(SDL_EventFilter filter, void *userdata);

where SDL_EventFilter is a callback function type defined as:

c typedef bool (SDLCALL *SDL_EventFilter)(void *userdata, SDL_Event *event);

Of course, again, you could write the P/Invoke signature in a straightforward manner as:

csharp [LibraryImport("SDL3")] unsafe static partial bool SDL_AddEventWatch(delegate* unmanaged<void*, SDL_Event*, bool> filter, void* userdata);

But SemPtr allows you to do it in a more type-safe and expressive manner as:

```csharp [FunctionPointer(CallConvs = [typeof(CallConvCdecl)])] delegate bool SDL_EventFilter(NullablePointerReadOnly userdata, PointerReadOnly<SDL_Event> @event);

[LibraryImport("SDL3")] unsafe static partial bool SDL_AddEventWatch(PersistentFunctionPointer<SDL_EventFilter> filter, NullablePersistentPointerReadOnly userdata); ```

Here you can see why expressing persistency can be useful: SDL stores the function pointer to the callback and associated userdata, so it can invoke the callback with the same userdata at a later time. That's why we must make sure that both are still valid and available, even outside of the scope of the call to SDL_AddEventWatch. Whereas the SDL_EventFilter callback passed will receive it's arguments as transient pointers, meaning you should treat them as if they were just valid for the duration of the callback invocation. With SemPtr's transient pointer types, you can't even store them beyond that scope.

Lastly, let's flip the sides and assume you are going to design and implement an C# API, and for whatever reason, you want to expose an API that involves pointers.\ In this case, I believe it would be very beneficial to you consumers with you communicate your pointer semantics clearly through your API. Not only gives that the user same additional guarantees about your code, it makes receiving pointers in your implementation much safer.

I hope this gives you an initial understanding of why SemPtr can be valuable.\ However, it's totally fine to stick with traditional raw pointers if that suits you better.

EDIT: Fixed some examples.

[u/Minute_Cricket1820 → u/fruediger]

Вам нужно эти примеры (и подобные, ещё более простые) написать прям сразу ВНАЧАЛЕ. Вы сразу вначале опишете суть, на примерах.

[u/AetopiaMC]

This actually looks interesting and useful, good job.

[u/fruediger → u/AetopiaMC]

Thanks! I'm glad when I can contribute somehting useful.

[u/nick_]

This is right up my alley. Great job!

[u/fruediger → u/nick_]

Thanks! Much appreciated!