RFC-145 (former RFC-4) host functions implementation

[s0me0ne-unkn0wn]

This PR implements the host functions introduced by the proposed RFC-145.

The allocator and the new versions of the host functions introduced by this PR are not activated; they are just added to the host, so by the time we start using them (#8866), the supermajority of nodes already had the functions ready to use.

[bkchr]

• *_clear_prefix is using prototypes introduced by PPP#7. If these functions hit the limit, they are supposed to fill the output buffer with a cursor, which is a SCALE-encoded Option<last_key_seen>. Now the problem is that it's impossible to predict the size of the buffer the runtime needs to provide, as the storage key length is theoretically unbounded.

While we implemented these methods, they never got used. So, not sure. They provide some value and we could fix it by having some extra function for getting the cursor and we buffer the cursor internally on the node.

[s0me0ne-unkn0wn]

The difference between v1 and v2 is that the state version is passed. This is actually not required and thus, it was v2 was deprecated.

Yeah, I just wasn't sure if I could just get rid of the version and that's it, was afraid to rip something useful. It's clear now, thank you!

Not sure what you don't understand there? You have one function that returns the size of the input and the other one is for reading the data. The read function would take a mut ptr to where the input data is written.

That has already been clarified with @koute, and I think your proposal, which you made in the original RFC discussion, to have a single function instead of two, makes a lot of sense. Even more sense would be made by passing the data length as an argument (as well as passing the fat pointer right now) so the runtime knows at once how much to allocate before calling the "gimme input" function.

While we implemented these methods, they never got used. So, not sure. They provide some value and we could fix it by having some extra function for getting the cursor and we buffer the cursor internally on the node.

I'm currently trying to integrate it into the implementation as much as possible (it's in #8866 but please don't dive there yet, it's far from being ready) cuz you and @tomaka did a great job sorting that out, so why would we lose that, especially given that we're breaking host function signatures anyway. That comes with some challenges because we don't really know how much buffer a storage key would require, as it's unbounded, and we basically cannot repeat the call if the buffer was not enough, because something has already been deleted after the call, but I believe we'll be able to find some common ground here when #8866 comes to the review point.

[bkchr]

That comes with some challenges because we don't really know how much buffer a storage key would require, as it's unbounded, and we basically cannot repeat the call if the buffer was not enough, because something has already been deleted after the call, but I believe we'll be able to find some common ground here when #8866 comes to the review point.

What I mean above is that you create some new extra host function get_cursor and we cache the cursor of the latest clear_prefix* call and return it there.

[s0me0ne-unkn0wn]

What I mean above is that you create some new extra host function get_cursor and we cache the cursor of the latest clear_prefix* call and return it there.

A good idea, I'll give it a try, thank you!

[s0me0ne-unkn0wn]

Okay, I've just found out I'm a dumbass cuz I dropped in the picoalloc in such a way that it implicitly (as a global allocator) frees memory that was explicitly allocated by the old host-side allocator, causing a UB. The easiest way around it I can think of is to merge #8866 into this PR, which we were discussing anyway. @koute I suppose that's exactly what you meant by the necessity to move the function divide check into this PR, but I just didn't understand your reasoning 🤦

[s0me0ne-unkn0wn]

Uh, that escalated quickly. We have a build-runtimes-polkavm CI test that is naturally failing on this PR because the runtime allocator is not implemented for PolkaVM. And, given that we've switched to JAMv1 flavor, to implement the allocation, we need the grow_heap host function, which requires an RFC by itself. I already have the allocator (and a grow_heap workaround) implemented in #6704, so I can just port that here, but in such a way we'd get an out-of-spec host function. @koute @bkchr need your opinion on how to proceed with that.

[koute]

@s0me0ne-unkn0wn You don't need an RFC for a PolkaVM-only host function. You should be able to easily modify the host function macro machinery to support a #[polkavm_only] attribute (so it only gets registered when the executor is PolkaVM), then you can add a PolkaVM-only grow_heap hostcall.