Agentic workflows, locally indexed search, and CLI

[mskarlin]

Description is still a WIP

This PR merges agentic workflows (via an OpenAI-based agent and a 'fake' agent) into a new branch to prep for release. The agent allows PQA to use a "Search" tool via a local full text index of files, a "Gather Evidence" tool, and a "Generate Answer" tool. Another new included feature is a CLI tool which allows PaperQA to be used in its entirety from the command line. Papers are chosen to be added into the Docs objects via full-text search of indexed directories on your machine. After installing via pip install paper-qa[agents] (or if local, pip install -e .[agents]), you can navigate to a directory with some PDF/TXT/HTML documents in it and run:

pqa ask "what is pyroptosis?"

Which, for the first time you run, an index will be created, but further runs will use that same index. If you change some PaperQA settings which would invalidate your current index (like changing the chunksize or your target directory), PaperQA will automatically create a new index for you.

New options can be set like the following (with a relative directory):

pqa set 'agent_tools.paper_directory' 'papers'

and then be viewed similarly

pqa show 'agent_tools.paper_directory'

Or to see everything you've assigned:

pqa show all 

All the settings available to pqa can be found in the QueryRequest object within agents/models/, behind the scenes PaperQA will store a configuration yaml in your (by default) ~/.pqa, folder but this can be changed via setting PQA_HOME.

Additional new stuff:

• After using the client library more heavily to query across 100s of papers, I implemented some minor bugfixes, like stripping https://doi.org/ out of DOIs which often shows up when they are extracted by LLMs
• Optional install for a 'google' environment that includes vertex. It's a crazy large import that takes ~2-3 seconds on its own, so it's best to leave optional.

[jamesbraza]

Huge battle @mskarlin here wow. Amazing work

[whitead]

Per our discussion - LGTM with removal of streaming functions