142 original papers with authors, known publication dates, concise summaries, artifact notes and downloadable citations.
Inclusion provides research context, not reproduced results. 3 papers have no corresponding reviewed artifact entry and do not add to the catalog entry count. arXiv dates are first submissions; other sources retain their known dates without invented months or days.
Introduces Code2Games for linking scene structure with gameplay generation, plus GameCode4D for studying game-world quality and subsequent engine adaptation.
Hongcheng Gao · Hailong Qu · Yu Lei · Henghui Sun · Haoyang Li · Yipeng Wei · Naihao Xue · Xiaohan Yu · Zhuo Tao · Yihe Zang · Yajiao Wang · Jingyi Tang · Yi Li · Jingjing Zhou · Jie Luo · Bohan Zeng · Chengyu Shen · Hao Jiang · Chong Chen · Bowen Qu · Olive Huang · Zeqiang Wang
Introduces engineering tasks across multiple software domains and GUI/CLI interfaces, with artifact-based checks and a separately identified main-experiment subset.
Alexander Gill · Md Farhan Ishmam · Xuyen Nguyen · Neha Bhat · Parker Henry DeYoung · Fateme Hashemi Chaleshtori · Nathan Stringham · Kenneth Marino · Ana Marasović
Introduces web research tasks ending in documents, spreadsheets or presentations, evaluated through artifact-specific checks combined with model judgments.
Kun Chen · Haorong Hong · Peizhong Gao · Jianfeng Lin · Tongxu Luo · Yuxuan Xie · Chenxu Liu · Jieling He · Zhongyuan Liu · Zeno Zeng
Introduces separate tracks for game generation, defect repair and multi-turn optimization, combining interaction, behavioral tests and regression checks.
Paper record · no cataloged artifact
The repository currently contains a README and project-site scaffold. It says benchmark data, evaluation code and model artifacts are still being prepared; the website template license does not establish benchmark reuse terms.
Introduces dashboard reconstruction through active exploration, the Plotly+Dash DashboardMimic benchmark, and combined code-semantic and interaction-based evaluation.
Brian La · Sejoon Chang · Ben Kim · Junyoung Bae · Aamish Ahmad Beg · Sei Chang · Gonzalo Gonzalez-Pumariega · Kanav Goyal
Introduces scoped C++ modification tasks inside Unreal Engine projects, evaluating compilation and runtime behavior across game-engine subsystems.
Paper record · no cataloged artifact
The paper and author research page are public. No reviewable task/evaluator release was established: the inspected repository URL returned404. This paper record does not imply that the benchmark can be run from public artifacts.
Jian Zhu · Yuzheng Zhang · Zeyao Ma · Bohan Zhang · Armin Schoepf · Daniel Woloch · Peter Yiliu Wang · Guangyu Robert Yang · Samuel Jacob · Siddharth Nagisetty · Abhiram Chundru · Jean Lin · Spencer Mateega · Jing Zhang
Introduces end-to-end spreadsheet generation, debugging and visualization workflows with complex cross-sheet dependencies and separate artifact-evaluation paths.
Mengqi Yuan · Zilong Zhou · Xinzhuang Xiong · Weiming Wu · Jiayang Sun · Jiamin Song · Kaiqian Cui · Bowen Wang · Haoyuan Wu · Yitong Li · Dunjie Lu · Haikong Lu · Qi Zhen · Xinyuan Wang · Jiaqi Deng · Yuhao Yang · Cheng Chen · Boyuan Zheng · Alex Su · Xiao Yu · Hao Zou · Saaket Agashe · Xing Han Lu · Manpreet Kaur · Zhengyang Qi · Vincent Sunn Chen · Frederic Sala · Dayiheng Liu · Junyang Lin · Zhou Yu · Yu Su · Siva Reddy · Xin Eric Wang · Peng Qi · Tianbao Xie · Tao Yu
Introduces a distinct set of long-horizon computer-use workflows that combine realistic artifacts, changing context and multi-application dependencies.
Tongxu Luo · Rongsheng Wang · Jiaxi Bi · Chenming Xu · Zhengyang Tang · Jianlong Chen · Juhao Liang · Ke Ji · Shuqi Guo · Yuhao Du · Fan Bu · Wenyu Du · Xiaotong Zhang · Kyle Li · Shaobo Wang · Linfeng Zhang · Yuxuan Liu · Xin Lai · Chenxin Li · Yiduo Guo · Zhexin Zhang · Xinyuan Wang · Tianyi Bai · Ziniu Li · Benyou Wang
Introduces end-to-end Godot game generation evaluated through complete executable artifacts, replayed demonstrations and rubric-guided multimodal judgments.
Wenkai Wang · Tao Xiong · Jingchen Ni · Yunpeng Bao · Xiyun Li · Tianqi Liu · Hongcan Guo · Zilong Huang · Shengyu Zhang
Introduces professional desktop tasks and a phased interaction protocol in which simulated users can clarify, interrupt or give feedback. The study examines long workflows and proactive collaboration; its released task set and model-backed simulator should be versioned separately from the original OSWorld tasks.
Yilei Jiang · Jinyuan Hu · Qianyin Xiao · Yaozhi Zheng · Ruize Ma · Kaituo Feng · Jiaming Han · Tianshuo Peng · Kaixuan Fan · Manyuan Zhang · Xiangyu Yue
Introduces an agentic web-game generation framework with reusable templates and debugging knowledge, and describes a benchmark based on build, visual and intent criteria.
Paper record · no cataloged artifact
The game-generation framework and example archives are public, but the README still describes the OpenGame-Bench evaluation pipeline as forthcoming. This record preserves the research without claiming a complete public evaluator.
Yuxuan Zhang · Yubo Wang · Yipeng Zhu · Penghui Du · Junwen Miao · Xuan Lu · Zhuofeng Li · Xingwei Qu · Zhengkang Guo · Yuanzhe Shen · Dingjie Song · Han Zhou · Tuney Zheng · Xian Wu · Hao Yu · Songcheng Cai · Yi Lu · Yunzhuo Hao · Minyi Lei · Liang Chen · Kai Zou · Huifeng Yin · Wendong Xu · Dongfu Jiang · Ping Nie · Jiaheng Liu · Wenhu Chen · Kelsey R. Allen
Introduces browser-agent evaluation on everyday production-site workflows, with recorded interaction evidence and task-specific submission interception.
Tests web-agent generalization across historical interface variants and proposes plan distillation to collect training trajectories spanning those variants.
Romain Froger · Pierre Andrews · Matteo Bettini · Amar Budhiraja · Ricardo Silveira Cabral · Virginie Do · Emilien Garreau · Jean-Baptiste Gaya · Hugo Laurençon · Maxime Lecanu · Kunal Malkan · Dheeraj Mekala · Pierre Ménard · Gerard Moreno-Torres Bertran · Ulyana Piterbarg · Mikhail Plekhanov · Mathieu Rita · Andrey Rusakov · Vladislav Vorotilov · Mengjue Wang · Ian Yu · Amine Benhalloum · Grégoire Mialon · Thomas Scialom
Introduces asynchronous simulated-app scenarios with changing events and action-level verification, enabling evaluation and reward-based learning under temporal and coordination demands.
Wayne Chi · Yixiong Fang · Arnav Yayavaram · Siddharth Yayavaram · Seth Karten · Qiuhong Anna Wei · Runkun Chen · Alexander Wang · Valerie Chen · Ameet Talwalkar · Chris Donahue
Introduces Godot project-editing tasks that combine code changes with visual assets, and studies image/video feedback for evaluating multimodal coding agents.
Zhaoyang Wang · Canwen Xu · Boyi Liu · Yite Wang · Siwei Han · Zhewei Yao · Huaxiu Yao · Yuxiong He
Introduces a pipeline for synthesizing executable MCP tool environments backed by databases, then using their state transitions and verifiers for multi-turn agent learning.
Mike A. Merrill · Alexander G. Shaw · Nicholas Carlini · Boxuan Li · Harsh Raj · Ivan Bercovich · Lin Shi · Jeong Yeon Shin · Thomas Walshe · E. Kelly Buchanan · Junhong Shen · Guanghao Ye · Haowei Lin · Jason Poulos · Maoyu Wang · Marianna Nezhurina · Jenia Jitsev · Di Lu · Orfeas Menis Mastromichalakis · Zhiwei Xu · Zizhao Chen · Yue Liu · Robert Zhang · Leon Liangyu Chen · Anurag Kashyap · Jan-Lucas Uslu · Jeffrey Li · Jianbo Wu · Minghao Yan · Song Bian · Vedang Sharma · Ke Sun · Steven Dillmann · Akshay Anand · Andrew Lanpouthakoun · Bardia Koopah · Changran Hu · Etash Guha · Gabriel H. S. Dreiman · Jiacheng Zhu · Karl Krauth · Li Zhong · Niklas Muennighoff · Robert Amanfu · Shangyin Tan · Shreyas Pimpalgaonkar · Tushar Aggarwal · Xiangning Lin · Xin Lan · Xuandong Zhao · Yiqing Liang · Yuanli Wang · Zilong Wang · Changzhi Zhou · David Heineman · Hange Liu · Harsh Trivedi · John Yang · Junhong Lin · Manish Shetty · Michael Yang · Nabil Omi · Negin Raoof · Shanda Li · Terry Yue Zhuo · Wuwei Lin · Yiwei Dai · Yuxin Wang · Wenhao Chai · Shang Zhou · Dariush Wahdany · Ziyu She · Jiaming Hu · Zhikang Dong · Yuxuan Zhu · Sasha Cui · Ahson Saiyed · Arinbjörn Kolbeinsson · Jesse Hu · Christopher Michael Rytting · Ryan Marten · Yixin Wang · Alex Dimakis · Andy Konwinski · Ludwig Schmidt
Introduces Terminal-Bench 2.0, a collection of challenging command-line tasks with isolated environments, human-written reference solutions and executable tests of task completion.
Karen Ullrich · Jingtong Su · Claudia Shi · Arjun Subramonian · Amir Bar · Ivan Evtimov · Nikolaos Tsilivis · Randall Balestriero · Julia Kempe · Mark Ibrahim
Studies UI-agent reliability under changes to simulated app appearance and content, using configurable applications and state-based task rewards.
Qiushi Sun · Mukai Li · Zhoumianze Liu · Zhihui Xie · Fangzhi Xu · Zhangyue Yin · Kanzhi Cheng · Zehao Li · Zichen Ding · Qi Liu · Zhiyong Wu · Zhuosheng Zhang · Ben Kao · Lingpeng Kong
Presents a mobile safety detector combining explicit state checks with contextual model judgments, alongside an Android research environment and annotated risk trajectories.
Nan Huo · Xiaohan Xu · Jinyang Li · Per Jacobsson · Shipei Lin · Bowen Qin · Binyuan Hui · Xiaolong Li · Ge Qu · Shuzheng Si · Linheng Han · Edward Alexander · Xintong Zhu · Rui Qin · Ruihan Yu · Yiyao Jin · Feige Zhou · Weihao Zhong · Yun Chen · Hongyu Liu · Chenhao Ma · Fatma Ozcan · Yannis Papakonstantinou · Reynold Cheng
Introduces interactive database tasks with clarification, supporting knowledge and executable checks, distinguishing a prescribed conversational protocol from open-ended agentic interaction.
Yang Chen · Minghao Liu · Yufan Shen · Yunwen Li · Tianyuan Huang · Xinyu Fang · Tianyu Zheng · Wenxuan Huang · Cheng Yang · Daocheng Fu · Jianbiao Mei · Rong Wu · Yunfei Zhao · Licheng Wen · Xuemeng Yang · Song Mao · Qunshu Lin · Zhi Yu · Yongliang Shen · Yu Qiao · Botian Shi
Introduces interactive webpage reconstruction from user videos and supplied assets, with action-sequence evaluation of visual fidelity and functional behavior.
Xiang Deng · Jeff Da · Edwin Pan · Yannis Yiming He · Charles Ide · Kanak Garg · Niklas Lauffer · Andrew Park · Nitin Pasari · Chetan Rane · Karmini Sampath · Maya Krishnan · Srivatsa Kundurthy · Sean Hendryx · Zifan Wang · Vijay Bharadwaj · Jeff Holm · Raja Aluri · Chen Bo Calvin Zhang · Noah Jacobson · Bing Liu · Brad Kenstler
Introduces long-horizon repository-level software-engineering tasks, separating public and nonpublic collections and evaluating proposed patches against test requirements.
Zhiheng Xi · Jixuan Huang · Chenyang Liao · Baodai Huang · Honglin Guo · Jiaqi Liu · Rui Zheng · Junjie Ye · Jiazheng Zhang · Wenxiang Chen · Wei He · Yiwen Ding · Guanyu Li · Zehui Chen · Zhengyin Du · Xuesong Yao · Yufei Xu · Jiecao Chen · Tao Gui · Zuxuan Wu · Qi Zhang · Xuanjing Huang · Yu-Gang Jiang
Presents AgentGym-RL for training agents through multi-turn environmental interaction and ScalingInter-RL, which progressively increases the interaction horizon during reinforcement learning.
Xinyuan Wang · Bowen Wang · Dunjie Lu · Junlin Yang · Tianbao Xie · Junli Wang · Jiaqi Deng · Xiaole Guo · Yiheng Xu · Chen Henry Wu · Zhennan Shen · Zhuokai Li · Ryan Li · Xiaochuan Li · Junda Chen · Boyuan Zheng · Peihang Li · Fangyu Lei · Ruisheng Cao · Yeqiao Fu · Dongchan Shin · Martin Shin · Jiarui Hu · Yuyan Wang · Jixuan Chen · Yuxiao Ye · Danyang Zhang · Dikang Du · Hao Hu · Huarong Chen · Zaida Zhou · Haotian Yao · Ziwei Chen · Qizheng Gu · Yipu Wang · Heng Wang · Diyi Yang · Victor Zhong · Flood Sung · Y. Charles · Zhilin Yang · Tao Yu
Presents OpenCUA's demonstration collection, AgentNet trajectories and generated state-action reasoning pipeline for computer-use agents. The catalog links the released data and offline action evaluator, while model results and OSWorld evaluation remain separate from the artifact's data scope.
Ori Press · Brandon Amos · Haoyu Zhao · Yikai Wu · Samuel K. Ainsworth · Dominik Krupke · Patrick Kidger · Touqir Sajed · Bartolomeo Stellato · Jisun Park · Nathanael Bosch · Eli Meril · Albert Steppi · Arman Zharmagambetov · Fangzhao Zhang · David Perez-Pineiro · Alberto Mercurio · Ni Zhan · Talor Abramovich · Kilian Lieret · Hanlin Zhang · Shirley Huang · Matthias Bethge · Ofir Press
Introduces numerical-program optimization tasks with problem generators, correctness validators and reference timings, plus an iterative optimization agent.
Victor Barres · Honghua Dong · Soham Ray · Xujie Si · Karthik Narasimhan
Introduces τ²-Bench, where both an agent and a simulated user can change a shared environment through tools, enabling evaluation of task execution, communication and coordination.
Zhiyu Lin · Zhengda Zhou · Zhiyuan Zhao · Tianrui Wan · Yilun Ma · Junyu Gao · Xuelong Li
Breaks webpage-to-code evaluation into visual perception, HTML programming, image–HTML understanding and page generation to diagnose model capabilities.
Introduces more demanding WebArena tasks centered on retaining observations, numerical reasoning and tracking information across extended browsing workflows.
Extends CRMArena to broader sales and service workflows, B2B/B2C settings and persona-guided multi-turn interactions, including confidentiality evaluation.
Zimu Lu · Yunqiao Yang · Houxing Ren · Haotian Hou · Han Xiao · Ke Wang · Weikang Shi · Aojun Zhou · Mingjie Zhan · Hongsheng Li
Introduces instruction-based website generation tasks and browser-agent functional evaluation, together with instruction data for training code-generation systems.
John Yang · Kilian Lieret · Carlos E. Jimenez · Alexander Wettig · Kabir Khandpur · Yanzhe Zhang · Binyuan Hui · Ofir Press · Ludwig Schmidt · Diyi Yang
Constructs executable repository environments and synthesizes test-failing changes to scale software-engineering training tasks and agent trajectories.
Leon Guertler · Bobby Cheng · Simon Yu · Bo Liu · Leshem Choshen · Cheston Tan
Introduces TextArena as a text-game collection and online-play interface for studying and training interactive language-model behavior, including competitive social reasoning.
Yunqi Gu · Ian Huang · Jihyeon Je · Guandao Yang · Leonidas Guibas
Evaluates visual model systems on code-based Blender scene reconstruction across five graphics-editing categories, including the allocation of generation and verification effort.
Shravan Nayak · Xiangru Jian · Kevin Qinghong Lin · Juan A. Rodriguez · Montek Kalsi · Rabiul Awal · Nicolas Chapados · M. Tamer Özsu · Aishwarya Agrawal · David Vazquez · Christopher Pal · Perouz Taslakian · Spandana Gella · Sai Rajeswar
Defines desktop GUI element grounding, layout grounding and action prediction tasks; the inspected public release currently supplies the two grounding components.
Ludovico Mitchener · Jon M Laurent · Alex Andonian · Benjamin Tenmann · Siddharth Narayanan · Geemi P Wellawatte · Andrew White · Lorenzo Sani · Samuel G Rodriques
Introduces bioinformatics data-analysis capsules and a notebook-based agent harness, with open-answer and multiple-choice evaluation of multi-step computational work.
Deepak Nathani · Lovish Madaan · Nicholas Roberts · Nikolay Bashlykov · Ajay Menon · Vincent Moens · Amar Budhiraja · Despoina Magka · Vladislav Vorotilov · Gaurav Chaurasia · Dieuwke Hupkes · Ricardo Silveira Cabral · Tatiana Shavrina · Jakob Foerster · Yoram Bachrach · William Yang Wang · Roberta Raileanu
Introduces MLGym and MLGym-Bench for developing and evaluating agents that implement machine-learning methods, run experiments and iteratively improve task results.
Samuel Miserendino · Michele Wang · Tejal Patwardhan · Johannes Heidecke
Studies freelance software-engineering work through code-fix and implementation-proposal tasks, with end-to-end tests and a public Diamond evaluation split.
Henry Hengyuan Zhao · Kaiming Yang · Wendi Yu · Difei Gao · Mike Zheng Shou
Studies desktop-agent robustness when a workflow starts from varied interface and application states. WorldGUI pairs base tasks with state augmentations and presents an agent that critiques planning and execution; condition variants should not be reported as independent user goals.
Jiayi Pan · Xingyao Wang · Graham Neubig · Navdeep Jaitly · Heng Ji · Alane Suhr · Yizhe Zhang
Introduces executable Python-repository tasks with unit tests for training software-engineering agents, and studies solution verifiers learned from the resulting agent trajectories.
Frank F. Xu · Yufan Song · Boxuan Li · Yuxuan Tang · Kritanjali Jain · Mengxue Bao · Zora Z. Wang · Xuhui Zhou · Zhitong Guo · Murong Cao · Mingyang Yang · Hao Yang Lu · Amaad Martin · Zhe Su · Leander Maben · Raj Mehta · Wayne Chi · Lawrence Jang · Yiqing Xie · Shuyan Zhou · Graham Neubig
Introduces a simulated software company with internal services and task checks for agents that browse websites, run programs and collaborate with simulated coworkers.
Thibault Le Sellier De Chezelles · Maxime Gasse · Alexandre Drouin · Massimo Caccia · Léo Boisvert · Megh Thakkar · Tom Marty · Rim Assouel · Sahar Omidi Shayegan · Lawrence Keunho Jang · Xing Han Lù · Ori Yoran · Dehan Kong · Frank F. Xu · Siva Reddy · Quentin Cappart · Graham Neubig · Ruslan Salakhutdinov · Nicolas Chapados · Alexandre Lacoste
Presents a shared browser-agent interface and experiment tooling for comparing behavior across multiple web benchmarks.
Kung-Hsiang Huang · Akshara Prabhakar · Sidharth Dhawan · Yixin Mao · Huan Wang · Silvio Savarese · Caiming Xiong · Philippe Laban · Chien-Sheng Wu
Introduces professional CRM tasks over interconnected business records to evaluate customer-service agents' reasoning, function calling and rule following.
Zhiyong Wu · Zhenyu Wu · Fangzhi Xu · Yian Wang · Qiushi Sun · Chengyou Jia · Kanzhi Cheng · Zichen Ding · Liheng Chen · Paul Pu Liang · Yu Qiao
Describes cross-platform GUI grounding data collection and action-model training, combining newly collected interfaces with existing sources. Its public corpus supports grounding pretraining; data elements, evaluation tasks and released model variants are distinct units.
Jingxuan Chen · Derek Yuen · Bin Xie · Yuhao Yang · Gongwei Chen · Zhihao Wu · Li Yixing · Xurui Zhou · Weiwen Liu · Shuai Wang · Kaiwen Zhou · Rui Shao · Liqiang Nie · Yasheng Wang · Jianye Hao · Jun Wang · Kun Shao
Introduces Android tasks across single-app and cross-app workflows, agent integrations and automated evaluation of task completion and resource consumption.
Ziru Chen · Shijie Chen · Yuting Ning · Qianheng Zhang · Boshi Wang · Botao Yu · Yifei Li · Zeyi Liao · Chen Wei · Zitong Lu · Vishal Dey · Mingyi Xue · Frazier N. Baker · Benjamin Burns · Daniel Adu-Ampratwum · Xuhui Huang · Xia Ning · Song Gao · Yu Su · Huan Sun
Introduces scientific-analysis tasks requiring standalone programs, with task-specific execution checks and output assessment grounded in published research.
Rogerio Bonatti · Dan Zhao · Francesco Bonacci · Dillon Dupont · Sara Abdali · Yinheng Li · Yadong Lu · Justin Wagle · Kazuhito Koishida · Arthur Bucker · Lawrence Jang · Zack Hui
Adapts desktop-agent evaluation to Windows applications and describes parallel execution for scaling repeatable computer-use experiments.
Jiarui Lu · Thomas Holleis · Yizhe Zhang · Bernhard Aumayer · Feng Nan · Felix Bai · Shuang Ma · Shen Ma · Mengyu Li · Guoli Yin · Zirui Wang · Ruoming Pang
Introduces stateful conversational tool-use evaluation with simulated users, dependencies between tool actions, and checks on intermediate milestones and final outcomes.
Mark Towers · Ariel Kwiatkowski · Jordan Terry · John U. Balis · Gianluca De Cola · Tristan Deleu · Manuel Goulão · Andreas Kallinteris · Markus Krimmel · Arjun KG · Rodrigo Perez-Vicente · Andrea Pierré · Sander Schulhoff · Jun Jet Tai · Hannah Tan · Omar G. Younis
Describes Gymnasium’s shared environment API, reference tasks and tools for reproducible reinforcement-learning experiments.
Florian Felten · Umut Ucak · Hicham Azmani · Gao Peng · Willem Röpke · Hendrik Baier · Patrick Mannion · Diederik M. Roijers · Jordan K. Terry · El-Ghazali Talbi · Grégoire Danoy · Ann Nowé · Roxana Rădulescu
Introduces a standardized collection of multi-objective multi-agent environments and baseline learning methods for comparing policies under differing rewards and utility assumptions.
Ori Yoran · Samuel Joseph Amouyal · Chaitanya Malaviya · Ben Bogin · Ofir Press · Jonathan Berant
Studies realistic web-research tasks and introduces AssistantBench and SeePlanAct to evaluate information gathering, planning and answer reliability on the open web.
Ruisheng Cao · Fangyu Lei · Haoyuan Wu · Jixuan Chen · Yeqiao Fu · Hongcheng Gao · Xinzhuang Xiong · Hanchong Zhang · Yuchen Mao · Wenjing Hu · Tianbao Xie · Hongshen Xu · Danyang Zhang · Sida Wang · Ruoxi Sun · Pengcheng Yin · Caiming Xiong · Ansong Ni · Qian Liu · Victor Zhong · Lu Chen · Kai Yu · Tao Yu
Introduces a computer-use benchmark for data-science and engineering workflows that require coordinated code generation and GUI operations in professional tools.
Wentong Chen · Junbo Cui · Jinyi Hu · Yujia Qin · Junjie Fang · Yue Zhao · Chongyi Wang · Jun Liu · Guirong Chen · Yupeng Huo · Yuan Yao · Yankai Lin · Zhiyuan Liu · Maosong Sun
Presents GUICourse's GUIEnv, GUIAct and GUIChat datasets and a training pipeline for improving visual GUI perception and action prediction.
Dongping Chen · Yue Huang · Siyuan Wu · Jingyu Tang · Liuyi Chen · Yilin Bai · Zhigang He · Chenlong Wang · Huichi Zhou · Yiqiang Li · Tianshuo Zhou · Yue Yu · Chujie Gao · Qihui Zhang · Yi Gui · Zhen Li · Yao Wan · Pan Zhou · Jianfeng Gao · Lichao Sun
Builds video-based GUI understanding tasks across diverse interface scenarios and examines temporal context, frame selection and a GUI-oriented video model.
Kevin Qinghong Lin · Linjie Li · Difei Gao · Qinchen WU · Mingyi Yan · Zhengyuan Yang · Lijuan Wang · Mike Zheng Shou
Uses instructional videos to evaluate visual GUI workflows through high-level planning, subtask planning and atomic actions. These levels expose different failure modes in professional software; the reviewed release contains data, while its evaluation implementation is still listed as pending.
Describes the original PufferLib environment interoperability and vectorization framework; later native Ocean/5.0 components require separate source documentation.
Zhiheng Xi · Yiwen Ding · Wenxiang Chen · Boyang Hong · Honglin Guo · Junzhe Wang · Dingwen Yang · Chenyang Liao · Xin Guo · Wei He · Songyang Gao · Lu Chen · Rui Zheng · Yicheng Zou · Tao Gui · Qi Zhang · Xipeng Qiu · Xuanjing Huang · Zuxuan Wu · Yu-Gang Jiang
Introduces AgentGym's shared environment interfaces, task collections and demonstration trajectories, together with AgentEvol for improving language-model agents through exploration across environments.
Wei Li · William Bishop · Alice Li · Chris Rawles · Folawiyo Campbell-Ajala · Divya Tyamagundlu · Oriana Riva
Uses AndroidControl demonstrations with high- and low-level instructions to study how data scale affects UI action prediction and out-of-domain generalization.
Christopher Rawles · Sarah Clinckemaillie · Yifan Chang · Jonathan Waltz · Gabrielle Lau · Marybeth Fair · Alice Li · William Bishop · Wei Li · Folawiyo Campbell-Ajala · Daniel Toyama · Robert Berry · Divya Tyamagundlu · Timothy Lillicrap · Oriana Riva
Introduces an Android emulator benchmark with parameterized mobile workflows and rewards derived from system state.
Li Zhang · Shihe Wang · Xianqing Jia · Zhihan Zheng · Yunhe Yan · Longxi Gao · Yuanchun Li · Mengwei Xu
Evaluates Android task execution by matching essential application states, allowing successful traces to differ from a single reference action sequence.
Tianbao Xie · Danyang Zhang · Jixuan Chen · Xiaochuan Li · Siheng Zhao · Ruisheng Cao · Toh Jing Hua · Zhoujun Cheng · Dongchan Shin · Fangyu Lei · Yitao Liu · Yiheng Xu · Shuyan Zhou · Silvio Savarese · Caiming Xiong · Victor Zhong · Tao Yu
Introduces a real-computer environment and desktop-task benchmark with configurable initial states and checks of the resulting application and file state.
Yi Gui · Zhen Li · Yao Wan · Yemin Shi · Hongyu Zhang · Yi Su · Bohua Chen · Dongping Chen · Siyuan Wu · Xing Zhou · Wenbin Jiang · Hai Jin · Xiangliang Zhang
Introduces a large dataset of webpage designs paired with code and layout information, together with a code-generation baseline and structural evaluation metric.
Zhicheng Guo · Sijie Cheng · Hao Wang · Shihao Liang · Yujia Qin · Peng Li · Zhiyuan Liu · Maosong Sun · Yang Liu
Addresses instability in tool-use evaluation with a virtual API server that combines cached responses and API simulation, plus revised model-based scoring procedures.
Chenglei Si · Yanzhe Zhang · Ryan Li · Zhengyuan Yang · Ruibo Liu · Diyi Yang
Introduces a real-webpage benchmark with visual metrics and human evaluation for measuring how well multimodal models reconstruct interfaces from screenshots.
Adds descriptions of screens, prior actions and intended next actions to Android demonstrations, studying how these annotations support GUI action prediction.
Introduces paired screen images, instructions and action programs for desktop and web tasks, with sequence and action metrics for evaluating generated GUI-control code.
Introduces a conversational web-navigation dataset and benchmark, and studies methods for selecting relevant webpage elements during multi-turn interaction.
Jing Yu Koh · Robert Lo · Lawrence Jang · Vikram Duvvur · Ming Chong Lim · Po-Yu Huang · Graham Neubig · Shuyan Zhou · Ruslan Salakhutdinov · Daniel Fried
Introduces web tasks that require agents to combine visual and textual information, with execution-based evaluation of task completion.
Chang Ma · Junlei Zhang · Zhihao Zhu · Cheng Yang · Yujiu Yang · Yaohui Jin · Zhenzhong Lan · Lingpeng Kong · Junxian He
Introduces an analytical evaluation framework that measures intermediate progress across multi-turn environments rather than relying only on final task success.
Kanzhi Cheng · Qiushi Sun · Yougang Chu · Fangzhi Xu · Yantao Li · Jianbing Zhang · Zhiyong Wu
Presents the SeeClick visual GUI agent and the ScreenSpot benchmark to investigate how locating instructed screen elements supports interface automation.
Grégoire Mialon · Clémentine Fourrier · Craig Swift · Thomas Wolf · Yann LeCun · Thomas Scialom
Introduces assistant tasks that combine reasoning, multimodal inputs, browsing and tool use, with held-out answers for evaluating general-purpose assistants.
Shuyan Zhou · Frank F. Xu · Hao Zhu · Xuhui Zhou · Robert Lo · Abishek Sridhar · Xianyi Cheng · Tianyue Ou · Yonatan Bisk · Daniel Fried · Uri Alon · Graham Neubig
Introduces realistic, self-hosted web environments and executable checks for evaluating agents on multi-step website tasks.
Christopher Rawles · Alice Li · Daniel Rodriguez · Oriana Riva · Timothy Lillicrap
Introduces a dataset of human Android interactions with instructions, screens and actions for studying device control across tasks, apps and device configurations.
John Yang · Akshara Prabhakar · Karthik Narasimhan · Shunyu Yao
Frames interactive coding as an environment where code is an action and execution feedback is an observation, with benchmark tasks using Bash, SQL and Python.
Maxime Chevalier-Boisvert · Bolun Dai · Mark Towers · Rodrigo de Lazcano · Lucas Willems · Salem Lahlou · Suman Pal · Pablo Samuel Castro · Jordan Terry
Presents the design and shared interfaces of Minigrid and Miniworld for creating configurable two- and three-dimensional goal-directed learning tasks.
Xiang Deng · Yu Gu · Boyuan Zheng · Shijie Chen · Samuel Stevens · Boshi Wang · Huan Sun · Yu Su
Introduces a diverse dataset of real-website tasks and human action sequences for developing web agents that generalize to unfamiliar sites and domains.
Minghao Li · Yingxiu Zhao · Bowen Yu · Feifan Song · Hangyu Li · Haiyang Yu · Zhoujun Li · Fei Huang · Yongbin Li
Introduces executable API tools and evaluation dialogues for measuring planning, tool retrieval and API invocation, alongside separate training dialogues for tool-augmented language models.
John P. Agapiou · Alexander Sasha Vezhnevets · Edgar A. Duéñez-Guzmán · Jayd Matyas · Yiran Mao · Peter Sunehag · Raphael Köster · Udari Madhushani · Kavya Kopparapu · Ramona Comanescu · DJ Strouse · Michael B. Johanson · Sukhdeep Singh · Julia Haas · Igor Mordatch · Dean Mobbs · Joel Z. Leibo
Documents Melting Pot 2.0 substrates, background populations, asymmetric roles and the social-generalization evaluation protocol.
Yuhang Lai · Chengxi Li · Yiming Wang · Tianyi Zhang · Ruiqi Zhong · Luke Zettlemoyer · Scott Wen-tau Yih · Daniel Fried · Sida Wang · Tao Yu
Introduces data-science code-generation problems with execution and code-form checks across seven Python libraries, using adapted real-world programming questions.
Shunyu Yao · Howard Chen · John Yang · Karthik Narasimhan
Introduces a simulated shopping environment for learning how to interpret product requests and complete searches, selections and purchases through web interactions.
Linxi Fan · Guanzhi Wang · Yunfan Jiang · Ajay Mandlekar · Yuncong Yang · Haoyi Zhu · Andrew Tang · De-An Huang · Yuke Zhu · Anima Anandkumar
Introduces a Minecraft task suite and multimodal knowledge collection, alongside an agent-learning method that uses video-language models to provide rewards.
Mikayel Samvelyan · Robert Kirk · Vitaly Kurin · Jack Parker-Holder · Minqi Jiang · Eric Hambro · Fabio Petroni · Heinrich Küttler · Edward Grefenstette · Tim Rocktäschel
Presents a NetHack-based sandbox for creating reinforcement-learning tasks with configurable layouts, objects, dynamics and procedural variation.
Joel Z. Leibo · Edgar Duéñez-Guzmán · Alexander Sasha Vezhnevets · John P. Agapiou · Peter Sunehag · Raphael Koster · Jayd Matyas · Charles Beattie · Igor Mordatch · Thore Graepel
Introduces evaluation of multi-agent generalization to unfamiliar social situations using interactive worlds and reference co-players.
Daniel Toyama · Philippe Hamel · Anita Gergely · Gheorghe Comanici · Amelia Glaese · Zafarali Ahmed · Tyler Jackson · Shibl Mourad · Doina Precup
Presents a reinforcement-learning platform that exposes Android applications through a common touchscreen interface and evaluates agents on configurable tasks.
Mohit Shridhar · Xingdi Yuan · Marc-Alexandre Côté · Yonatan Bisk · Adam Trischler · Matthew Hausknecht
Aligns text-based household tasks with visual embodied environments so agents can learn abstract action policies in TextWorld and transfer them to ALFRED task execution.
J. K. Terry · Benjamin Black · Nathaniel Grammel · Mario Jayakumar · Ananth Hari · Ryan Sullivan · Luis Santos · Rodrigo Perez · Caroline Horsch · Clemens Dieffendahl · Niall L. Williams · Yashas Lokesh · Praveen Ravi
Introduces PettingZoo and the Agent Environment Cycle model for consistent interaction with diverse multi-agent environments.
Karl Cobbe · Christopher Hesse · Jacob Hilton · John Schulman
Introduces procedurally generated game environments and experimental protocols for measuring reinforcement-learning sample efficiency and generalization.
Marc Lanctot · Edward Lockhart · Jean-Baptiste Lespiau · Vinicius Zambaldi · Satyaki Upadhyay · Julien Pérolat · Sriram Srinivasan · Finbarr Timbers · Karl Tuyls · Shayegan Omidshafiei · Daniel Hennes · Dustin Morrill · Paul Muller · Timo Ewalds · Ryan Faulkner · János Kramár · Bart De Vylder · Brennan Saeta · James Bradbury · David Ding · Sebastian Borgeaud · Matthew Lai · Julian Schrittwieser · Thomas Anthony · Edward Hughes · Ivo Danihelka · Jonah Ryan-Davis
Presents a common framework of game environments, algorithms and analysis tools for reinforcement learning, game theory and search.
Ian Osband · Yotam Doron · Matteo Hessel · John Aslanides · Eren Sezener · Andre Saraiva · Katrina McKinney · Tor Lattimore · Csaba Szepesvari · Satinder Singh · Benjamin Van Roy · Richard Sutton · David Silver · Hado Van Hasselt
Defines controlled interactive experiments and standardized analysis for diagnosing core reinforcement-learning capabilities.
William H. Guss · Brandon Houghton · Nicholay Topin · Phillip Wang · Cayden Codel · Manuela Veloso · Ruslan Salakhutdinov
Introduces a simulator-paired dataset of human Minecraft demonstrations for studying learning methods that use examples to reduce interaction requirements.
Arthur Juliani · Vincent-Pierre Berges · Ervin Teng · Andrew Cohen · Jonathan Harper · Chris Elion · Chris Goy · Yuan Gao · Hunter Henry · Marwan Mattar · Danny Lange
Examines game engines as configurable research platforms and uses Unity ML-Agents to illustrate environments with visual, physical, task and social complexity.
Marc-Alexandre Côté · Ákos Kádár · Xingdi Yuan · Ben Kybartas · Tavian Barnes · Emery Fine · James Moore · Ruo Yu Tao · Matthew Hausknecht · Layla El Asri · Mahmoud Adada · Wendy Tay · Adam Trischler
Introduces a controllable text-game generator and learning environment for studying reinforcement learning, generalization and transfer across adventures.
Evan Zheran Liu · Kelvin Guu · Panupong Pasupat · Tianlin Shi · Percy Liang
Proposes using workflows extracted from demonstrations to guide reinforcement-learning exploration on web interfaces and evaluates the method on browser tasks.
Lianmin Zheng · Jiacheng Yang · Han Cai · Weinan Zhang · Jun Wang · Yong Yu
Introduces the original MAgent engine for configurable many-agent worlds and collective-behavior research; MAgent2 continues that implementation lineage.
Charles Beattie · Joel Z. Leibo · Denis Teplyashin · Tom Ward · Marcus Wainwright · Heinrich Küttler · Andrew Lefrancq · Simon Green · Víctor Valdés · Amir Sadik · Julian Schrittwieser · Keith Anderson · Sarah York · Max Cant · Adam Cain · Adrian Bolton · Stephen Gaffney · Helen King · Demis Hassabis · Shane Legg · Stig Petersen
Introduces a programmable first-person 3D platform for studying learning in visually diverse, partially observed worlds.