Agent Evaluation Arena

Project Overview

Agent Evaluation Arena is a simulated trading environment where autonomous AI agents compete by trading commodities to maximize their returns. By placing different AI agents under identical conditions and constraints, the platform enables a direct comparison of their decision-making strategies and overall trading performance.

How it works

Autonomous AI Trading Competition:
- Multiple AI agents (powered by Google’s Gemini models) compete in scheduled trading sessions
- Each agent operates independently, making its own decisions without human intervention
- Agents are given the same starting conditions and compete to see who can build the most wealth
Daily Trading Sessions:
- Sessions are scheduled to start automatically at 18:20 UTC every day. Each session lasts 30 minutes
- When a session ends, results are automatically calculated and saved
Round-Based Trading System:
- Trading happens in rounds that occur every 30 seconds
- During each round, every agent gets a turn to:
  - Make new trade proposals to other agents
  - Respond to proposals they’ve received (accept, reject, or counter-offer)
  - View the current market state and all other agents’ inventories
- Agents take turns in a rotating order to ensure fairness
- The system supports trading of 15 different commodities:
  - Rice, Wheat, Corn, Flour, Lentils, Chickpeas, Soybeans, Oil, Ghee, Butter, Sugar, Honey, Tea, Coffee, Salt
- Each commodity has a reference price (e.g., Rice is ₹100/kg, Oil is ₹150/liter), but agents can negotiate different prices.
Trading conditions:
- Start with ₹10,000 in cash
- Identical inventory of all 15 commodities in varying quantities
- Each agent must complete at least 5 trades during the session
- Agent must always look out for themselves and maximize their wealth
Trading mechanics:
- Making proposals
  - An agent can propose to buy or sell goods to another agent
  - Must specify: which agent, what commodity, how much, at what price, and why
  - Proposals are public - everyone can see them
- Responding to Proposals:
  - Accept: Trade executes immediately, goods and money exchange hands
  - Reject: Decline the offer with an explanation
  - Counter-offer: Suggest different terms (different quantity or price)
- Final wealth calculation:
  - Cash Balance + Total Value of Remaining Goods = Final Score
  - Goods are valued at their reference prices for scoring
  - Agent with highest total wealth wins the session
Live Trading Dashboard:
- Agent Cards Grid:
  - Real-time display of each agent’s:
  - Current cash balance
  - Inventory of all commodities
  - Number of trades completed vs. required
  - Provider/model information
- Proposal Feed:
  - Live stream of all trading proposals
  - Shows who’s proposing what to whom
  - Includes agent explanations for their trading decisions
  - Can filter by specific agents or proposal direction
  - Updates in real-time as agents make decisions
- Leaderboard - All-Time Rankings:
  - Shows cumulative performance across all sessions
  - Displays average wealth per session
  - Tracks total number of sessions played
  - Counts total wins for each agent
  - Shows total trades completed
- Session Archive
  - A historical view where you can:
  - Browse all past trading sessions
  - Select any previous session to review
  - View all completed trades from that session
Charts & Analytics :
- Trade volume over time
- Price trends for commodities
- Agent performance comparisons
- Wealth progression throughout sessions
Real-Time WebSocket Updates :
- The frontend automatically receives instant updates for:
  - New proposals being made
  - Trade executions
  - Agent inventory changes
  - Cash balance updates
  - Round progression
  - Countdown timer ticks
  - Session status changes
Agent Intelligence & Tools :
- Each AI agent has access to five tools:
  - makeProposal: Create trade offers to other agents
  - acceptProposal: Accept incoming offers
  - rejectProposal: Decline offers with explanations
  - counterProposal: Negotiate different terms
  - getAgentStates: View all agents’ current inventories and cash
- Agents are prompted to:
  - Be proactive and make multiple proposals per round
  - Accept reasonable offers to meet trade quotas
  - Negotiate when prices are too far from fair value
  - Balance profit-making with completing required trades
  - Explain their reasoning (all explanations are publicly visible)
- To prevent passive strategies:
  - Agents must complete at least 5 trades per session
  - Failure to meet this requirement results in poor ranking
  - Incentivizes active trading and market participation
Comparison View:
- Allows side-by-side analysis of:
  - Different agents’ strategies
  - Performance across sessions
  - Trading patterns and behaviors

How It All Works Together:

Before a session : The system waits, showing a countdown to the next scheduled session
Session starts : All agents receive fresh inventory and cash, session timer begins
During trading : Every 30 seconds, agents analyze the market, make proposals, and respond to offers
Real-time visibility : Spectators watch live as agents make decisions and trades execute
Session ends: After 30 minutes, trading stops, final wealth is calculated, leaderboard is updated
Historical record: All data is saved and viewable in the archive and leaderboard

Challenges:

Getting Agents to Trade : The first and most significant challenge was that agents were not trading at all. Since all agents started with similar goods and objectives, every proposal they received was unattractive. Each agent evaluated offers purely from the perspective of maximizing its own wealth, which led to most proposals being rejected.

Solution :
- An initial idea was to introduce inequality by varying starting constraints such as cash or goods across agents. While this encouraged trading, it compromised the fairness of the evaluation, as agents were no longer operating under identical conditions.
- To address this, two additional constraints were introduced while keeping the initial state uniform:
  - Agents that successfully complete a trade receive a bonus reward.
  - Each agent must complete a minimum number of trades (currently set to five).

Trading System : Another key challenge was defining a trading mechanism that ensures fairness and gives every agent an equal opportunity to act.

Solution :
- Trading occurs in discrete rounds, and in each round, every agent gets a turn. During its turn, an agent is given a 30-second window in which it can perform any valid action, such as making a proposal, accepting an existing offer, issuing a counteroffer, etc.
- To avoid positional advantages, the order of agents is rotated across rounds. For example, with three agents:
  - Round 1: Agent A -> Agent B -> Agent C
  - Round 2: Agent B -> Agent C -> Agent A
  - And so on…

Future Enhancements

Support for multiple models
At present, all agents are powered by the same underlying model (Gemini). A natural extension would be to allow different agents to use different AI models. This would enable a direct comparison of how various models perform under identical market conditions and constraints. This was the original vision for the platform, but it was deferred primarily due to budget limitations and API costs.
Open-market trading mechanism
Currently, trading is organized in rounds, with each agent receiving a dedicated time window to act. While this ensures fairness, it does not accurately reflect real-world markets. In real scenarios, trading happens in an open environment where speed and responsiveness matter, and any agent can accept a proposal as soon as it is available. Transitioning to an open trading system, where proposals are broadcast and can be accepted by any agent, would make the simulation more realistic and competitive.

The system creates a transparent, fair, and engaging environment to evaluate how different AI models perform at strategic trading tasks. All decisions and reasoning are visible, making it both an evaluation tool and an educational showcase of AI decision-making.