-
Notifications
You must be signed in to change notification settings - Fork 749
Removed "cannot answer"
literals and added reset
tool
#698
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
@@ -49,7 +50,7 @@ def settings_to_tools( | |||
embedding_model = embedding_model or settings.get_embedding_model() | |||
tools: list[Tool] = [] | |||
for tool_type in ( | |||
(PaperSearch, GatherEvidence, GenerateAnswer, Complete) | |||
(PaperSearch, GatherEvidence, GenerateAnswer, Reset, Complete) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We'll need to update the tool constructor in the server too.
cc. @nadolskit
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yep agreed
qa_prompt = ( | ||
"Answer the question below with the context.\n\n" | ||
"Context (with relevance scores):\n\n{context}\n\n----\n\n" | ||
"Question: {question}\n\n" | ||
"Write an answer based on the context. " | ||
"If the context provides insufficient information reply " | ||
'"I cannot answer."' | ||
f'"{CANNOT_ANSWER_PHRASE}." ' |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nice!
51482c3
to
8e3d78d
Compare
…tcut to is_sure=False
67ef8e5
to
8b6aa61
Compare
…-exhaustive answers
8b6aa61
to
2e3edb3
Compare
There many of places where we depended on the string literal
"cannot answer"
in theqa
prompt, mainly the environment being done (prior to #684) or theanswer
being considered unsure.This environment check of
"cannot answer"
also has some downsides:qa
prompt"cannot answer"
is present in theqa
promptanswer
)So, what is "unsure"? Really it should be:
gen_answer
tool call updates the answergen_answer
tool) decides if an answer was successfulTo resolve this, we moved the unsure call directly to the
complete
tool. Now:complete(has_successful_answer=True)
complete(has_successful_answer=False)
The
"cannot answer"
check was mostly easy to remove, other than theAnswerSettings.wipe_context_on_answer_failure
, since we no longer have a way of checking unsure withingen_answer
.Since the agent controls unsureness now, we needed to make a new tool:
reset
, which basically performs the use case ofwipe_context_on_answer_failure
.After this PR, we have:
"cannot answer"
string literalAnswerSettings.wipe_context_on_answer_failure
gen_answer
tool