arxiv:2508.11987

FutureX: An Advanced Live Benchmark for LLM Agents in Future Prediction

Published on Aug 16

· Submitted by

liujiashuo77 on Aug 21

#2 Paper of the day

Upvote

Authors:

Jiashuo Liu ,

Tianci He ,

Yang Yang ,

Lingyue Yin ,

Zhenwei Zhu ,

Tianle Cai ,

Zehui Chen ,

Jiecao Chen ,

Xiangsheng Li ,

Abstract

FutureX is a dynamic, live benchmark for evaluating LLM agents in future prediction tasks, addressing challenges in real-time updates and data contamination.

AI-generated summary

Future prediction is a complex task for LLM agents, requiring a high level of analytical thinking, information gathering, contextual understanding, and decision-making under uncertainty. Agents must not only gather and interpret vast amounts of dynamic information but also integrate diverse data sources, weigh uncertainties, and adapt predictions based on emerging trends, just as human experts do in fields like politics, economics, and finance. Despite its importance, no large-scale benchmark exists for evaluating agents on future prediction, largely due to challenges in handling real-time updates and retrieving timely, accurate answers. To address this, we introduce FutureX, a dynamic and live evaluation benchmark specifically designed for LLM agents performing future prediction tasks. FutureX is the largest and most diverse live benchmark for future prediction, supporting real-time daily updates and eliminating data contamination through an automated pipeline for question gathering and answer collection. We evaluate 25 LLM/agent models, including those with reasoning, search capabilities, and integration of external tools such as the open-source Deep Research Agent and closed-source Deep Research models. This comprehensive evaluation assesses agents' adaptive reasoning and performance in dynamic environments. Additionally, we provide in-depth analyses of agents' failure modes and performance pitfalls in future-oriented tasks, including the vulnerability to fake web pages and the temporal validity. Our goal is to establish a dynamic, contamination-free evaluation standard that drives the development of LLM agents capable of performing at the level of professional human analysts in complex reasoning and predictive thinking.

View arXiv page View PDF Project page Add to collection

Community

liujiashuo77

Paper author Paper submitter 19 days ago

•

edited 19 days ago

A truly contamination-free benchmark!

The world’s first live benchmark for real future prediction, avoiding any data contamination, covering diverse domains like politics, economy, culture, and sports. A reliable benchmark to test LLM agents' planning, searching, and reasoning capabilities!

yangyang0606

Paper author 19 days ago

This has huge economic potential! Real-world trends (such as stock market fluctuations, epidemic spread, and technology adoption curves) are the result of the interaction and emergence of a large number of heterogeneous individuals (people, institutions, and companies). AI agents should do better.

librarian-bot

18 days ago

This is an automated message from the Librarian Bot. I found the following papers similar to this paper.

The following papers were recommended by the Semantic Scholar API

Please give a thumbs up to this comment if you found it helpful!

If you want recommendations for any Paper on Hugging Face checkout this Space

You can directly ask Librarian Bot for paper recommendations by tagging it in a comment: @librarian-bot recommend

Hualin3

13 days ago

e46f5ccf-292e-a13e-4ead-deee23715e97
You are an agent that can predict future events. The event to be predicted: "When will Elon Musk create a political party? (resolved around 2025-09-01 (GMT+8)).
A. the outcome be By August 31
B. the outcome be By September 30
C. the outcome be By October 31
D. the outcome be By November 30
E. the outcome be By December 31"
IMPORTANT: Your final answer MUST end with this exact format:
listing all plausible options you have identified, separated by commas, within the box. For example: \boxed{A} for a single option or \boxed{B, C, D} for multiple options.
Do not use any other format. Do not refuse to make a prediction. Do not say "I cannot predict the future." You must make a clear prediction based on the best data currently available, using the box format specified above. 这个题目出的不太好，马斯克啥时候成立政党，是一个下周有可能也无法确定的问题。最好是那种在下周可以验证的问题。比如：马斯克在09/01之前会成立政党吗？