0.38.0

What's Changed

• Enhance scenario display Web report UI by @takahirom in #210
• Add support for multi-line strings in additional system prompts by @takahirom in #211

Full Changelog: 0.37.0...0.38.0

0.37.0

What's Changed

• Fix file not found crash by @takahirom in #208

Full Changelog: 0.36.0...0.37.0

0.36.0

Add Historical Step Limit option

When the history of agent steps is long, the context consumes a lot of resources and hinders the quality of the AI response. Therefore, I added a Historical Step Limit to restrict the number of steps passed to the AI.

Add option of force disable cache for scenario

In some cases, image information is critical, and we cannot rely on UI tree information for caching. To address this, I added an option called force disable cache for scenario, which allows us to disable the AI decision cache for specific scenarios.

What's Changed

• Add scenario caching options and UI integration by @takahirom in #195
• Update AI options column width for improved layout by @takahirom in #200
• Make test deterministic by setting default scenario ID in expandOptions by @takahirom in #204
• Update timestamp retrieval in AI-decision cache to use TimeProvider by @takahirom in #201
• Update ExpandableSection default expansion logic for screenshots by @takahirom in #202
• Add logging option --ai-api-logging for AI API for CLI by @takahirom in #203
• Refactor scenario options UI to separate fundamental and other options by @takahirom in #206
• Add last step count option to AI options and update related methods by @takahirom in #205
• Rename lastStepCount to historicalStepLimit in AI options and update … by @takahirom in #207

Full Changelog: 0.35.0...0.36.0

0.35.0

You can now check the API logs in the UI by enabling the debug log.

What's Changed

• Add logging option to AI settings and update OpenAIAi constructor by @takahirom in #199

Full Changelog: 0.34.0...0.35.0

0.34.0

We needed to check arbagent.log to see exceptions of APIs. I added it to the step logs.

What's Changed

• Add error handling to agent actions and log failure details by @takahirom in #198

Full Changelog: 0.33.0...0.34.0

0.33.0

To resolve the "java.net.SocketException: Network is unreachable" error, replacing CIO with OkHttp.

What's Changed

• Replace CIO with OkHttp client for improved HTTP handling in OpenAIAi by @takahirom in #196

Full Changelog: 0.32.0...0.33.0

0.32.0

Add AiOptions for Exploring and Improving AI Accuracy

We introduced several options to enhance AI accuracy:

• Temperature: This controls how randomly the AI operates.
• Image Detail: Options like "high" and "low" determine how the AI interprets images.
  Please refer to https://platform.openai.com/docs/guides/vision#low-or-high-fidelity-image-understanding for more details. I don’t think you need to adjust this, but you can experiment to see if it makes a difference.
• Image Format:
  We can now pass lossy WebP images to the AI. I’m not entirely sure why, but I believe it may improve the AI’s image understanding. Feel free to try it out and see the results.

What's Changed

• Add tests for UserPromptTemplate to validate command template handling by @takahirom in #187
• Add index element to UserPromptTemplate for enhanced UI state representation by @takahirom in #188
• Add ai options support by @takahirom in #189
• Update UserPromptTemplate to simplify UI instructions by @takahirom in #190
• Add support for custom image formats in AI options by @takahirom in #192
• Add image detail options to AI settings and enhance UI components by @takahirom in #191
• Add options to e2e by @takahirom in #193
• Refactor screenshot handling to improve image processing and cleanup by @takahirom in #194

Full Changelog: 0.31.0...0.32.0

0.31.0

AIs tend not to see the images, which makes the test inconsistent. We addressed this with several approaches:

• Reduced context size by optimizing prompts
• Added image referencing in the UI_STATE prompt

Breaking Change for Users Using userPromptTemplate

To make the project and prompts consistent, we have renamed {{COMMAND_TEMPLATES}} to {{ACTION_TEMPLATES}}. This will generate an error so you can identify if you are using the incorrect syntax.

This is the latest userPromptTemplate template.

<GOAL>{{USER_INPUT_GOAL}}</GOAL>

<STEP>
Current step: {{CURRENT_STEP}}
Max step: {{MAX_STEP}}

<PREVIOUS_STEPS>
{{STEPS}}
</PREVIOUS_STEPS>
</STEP>

<UI_STATE>
Please refer to the image above.
<ELEMENTS>
{{UI_ELEMENTS}}
</ELEMENTS>
<FOCUSED_TREE>
{{FOCUSED_TREE}}
</FOCUSED_TREE>
</UI_STATE>

<INSTRUCTIONS>
Based on the above, decide on the next action to achieve the goal. Please ensure not to repeat the same action. The action must be one of the following:
</INSTRUCTIONS>

<ACTION_OPTIONS>
{{ACTION_TEMPLATES}}
</ACTION_OPTIONS>

What's Changed

• Improve user prompts by adding default messages for empty focused tree and UI elements by @takahirom in #185
• Refactor agent command types to agent action types for consistency in naming by @takahirom in #186

Full Changelog: 0.30.0...0.31.0

0.30.0

Bugfixes

What's Changed

• Refactor UserPromptTemplate to format UI state elements for improved … by @takahirom in #181
• Add XML content handling in ArbigentHtmlReportTest and escape XML in … by @takahirom in #182
• Remove replace by @takahirom in #184
• Add timestamp formatting to step display in Scenario and Main components by @takahirom in #183

Full Changelog: 0.29.0...0.30.0

0.29.0

Breaking Change for Users Using userPromptTemplate

Now the prompt is using XML format (experimental. We might revert this if performance is not good). The prompt now adds {{UI_ELEMENTS}}, {{FOCUSED_TREE}}, and {{COMMAND_TEMPLATES}} to cover all prompt elements.

<GOAL>{{USER_INPUT_GOAL}}</GOAL>

<STEP>
Current step: {{CURRENT_STEP}}
Max step: {{MAX_STEP}}

<PREVIOUS_STEPS>
{{STEPS}}
</PREVIOUS_STEPS>
</STEP>

<UI_STATE>
<ELEMENTS>{{UI_ELEMENTS}}</ELEMENTS>
<FOCUSED_TREE>{{FOCUSED_TREE}}</FOCUSED_TREE>
</UI_STATE>

<INSTRUCTIONS>
Based on the above, decide on the next action to achieve the goal. Please ensure not to repeat the same action. The action must be one of the following:
</INSTRUCTIONS>

<COMMAND_OPTIONS>
{{COMMAND_TEMPLATES}}
</COMMAND_OPTIONS>

What's Changed

• Refactor UserPromptTemplate to enhance template structure with XML-like tags and update unit tests for new format by @takahirom in #179
• Refactor context prompt generation to include UI elements and command templates, and add unit tests for context functionality by @takahirom in #178
• Add logging for MaestroDevice initialization with screenshots directory path by @takahirom in #180

Full Changelog: 0.28.0...0.29.0