What's going on under the hood?

[rudrOwO]

I am trying to understand how Guidance is able to guide a language model. Can someone please explain to me the implementation details or point me to relevant resources?

[Harsha-Nori]

Happy to expand on more details, but as a first step, you might find this talk I gave recently helpful: https://www.youtube.com/watch?v=qXMNPVVlCMs

[rudrOwO]

Thank You- It was a great watch!

It's fascinating how manipulating the actual inference engine is proving to be a viable steering strategy in addition to the long standing traditional approach of fine-tuning models.

And yes on that note- how exactly is Guidance able to hook into a model's inference layer during runtime?

[Harsha-Nori]

Glad you enjoyed the talk! Totally agree that steering at inference is super interesting :)! For all local models (e.g. anything hosted on Transformers or LlamaCpp), we directly modify the inference stack locally. You can see examples of our integrations here:

guidance/guidance/models/transformers/_transformers.py

Line 139 in e234c56

class TransformersEngine(Engine):

For most remote models (e.g. Anthropic/Gemini/OpenAI Models), we're limited in how much we can manipulate the inference stack by the APIs the model builders expose for us. However -- as mentioned in the video -- we're actively working on bringing guidance support to the cloud :)