Tag: what is ai agent benchmark

The Conference and Workshop on Neural Information Processing Systems (NeurIPS) is one of the most influential annual meetings for presenting and sharing research on neural information processing systems in their biological, technological, mathematical, and theoretical aspects. The NeurIPS 2022 organizing committee announced its coveted awards this week, recognizing thirteen Outstanding Papers, two in the Datasets & Benchmarks category, and a Test of Time p

“We have been investing in developing more agentic models, meaning they can understand more about the world around you, think multiple steps ahead, and take action on your behalf, with your supervision,” Pichai explained.Gemini 2.0: Core features and availabilityAt the heart of today’s announcement is the experimental release of Gemini 2.0 Flash, the flagship model of Gemini’s second generation. It builds upon the foundations laid by its predecessors while

Sorting algorithm is one of the most popular foundation algorithms that are used trillions of times on almost every day. But like many algorithms, it has reached a stage whereby human are struggling to improve them further, especially when the demand for computation continue to grow. In a new paper Faster sorting algorithms discovered using deep reinforcement learning, a DeepMind research team introduces AlphaDev, a deep reinforcement learning (DRL) agent

In a new study, Assistant Professor Vlad Ayzenberg and other researchers found preschoolers’ visual object recognition to be much better than AI’s capabilities. Credit: Joseph V. Labolito As artificial intelligence (AI) rapidly grows—a recent UN Trade and Development report projects the global AI market soaring to $4.8 trillion by 2033—the technology seems equipped to handle any task. Driving cars. Analyzing medical images. Making music. Having a conversat

Create a watch on this topic Receive upcoming developments if any, with an AI watch configured from this article. Create my free watch European AI sovereignty, American inference silicon: the RX-1, unveiled by Helsing, a German up-and-comer launched in 2021, exposes in a single technical sheet the two disjointed halves of the European embedded AI stack. This first quadruped robot designed for the battlefield by Helsing was conceived by Area 9, the company'

This latest agentic capability operates through the user interface of ChatGPT; users simply select the “Deep Research” option in the message composer and type their query. Supporting files or spreadsheets can also be uploaded for additional context.Once initiated, the AI embarks on a rigorous multi-step process, which may take 5-30 minutes to complete. A sidebar provides updates on the actions taken and the sources consulted. Users can carry on with other

Generative AI and Large Language Models (LLMs) have achieved remarkable success in Natural Language Processing (NLP) tasks, and their evolution now extends to performing actions beyond text comprehension. This marks a significant shift in the pursuit of Artificial General Intelligence. Evaluating the gaming prowess of LLMs, particularly in the context of Pokemon battles, serves as an effective benchmark for assessing their game-playing capabilities. In a n

Last year, ARC Prize 2024 saw 1,500 competitor teams, resulting in 40 papers of acclaimed industry influence. This year’s increased stakes aim to nurture even greater success.ARC Prize believes progress hinges on novel ideas rather than merely scaling existing systems. The next breakthrough in efficient general systems might not originate from current tech giants but from bold, creative researchers embracing complexity and curious experimentation.(Image cr

Alongside the model, Amazon is releasing a research preview of the Amazon Nova Act SDK. Using the SDK, developers can create agents capable of automating web tasks like submitting out-of-office notifications, scheduling calendar holds, or enabling automatic email replies.The SDK aims to break down complex workflows into dependable “atomic commands” such as searching, checking out, or interacting with specific interface elements like dropdowns or popups. De

AI is giving attackers a head start, Microsoft warns Threat actors are using AI to find bugs, build malware and run intrusions faster than defenders can keep up. Microsoft’s 2026 Digital Defense Report, covering July 2025 to June 2026, describes a near-term period in which attackers collect the benefits of AI first and defenders have to move quickly to close the gap. “AI is changing the physics of cybersecurity,” the company said. Bugs found faster than th

AI vulnerability patching has a grading problem Researchers at Northwestern University tested AI coding agents on 112 real security bugs. They found that a patch which stops a crash often fails the checks the project’s maintainers wrote for the fix. With Claude Opus 4.7, third of the patches that made the crash go away still failed those maintainer tests. If you’re thinking about letting an AI tool patch vulnerabilities for you, this is the number to look

Meta outlines Muse Wonder safeguards for AI agents acting on your behalf Meta’s Muse Wonder and Muse Wonder Full models are built for AI agents that handle tasks such as preparing insurance claims, reviewing leases and developing software. The models combine document analysis and tool use with the ability to retain relevant details across conversations and work through obstacles toward a user’s goal. Intended for commercial and research use, they support w

Getty / Futurism Your pesky remote freelancers demanding more money as inflation soars? You could try replacing them with AI agents instead — but it probably won’t work out well. New research highlighted by Wired shows how these AI models designed to automate tasks — if not entire jobs — turn out to be incredibly unproductive compared to the humans they’re replacing. Conducted by researchers at the nonprofit Center for AI Safety (CAIS) and the massive data

Upwind secures $400 million to protect AI across endpoints, code, and cloud Upwind Security has raised $400 million in Series C funding to expand its cloud and AI security platform, bringing the company’s total funding to more than $800 million. Bessemer Venture Partners led the round, with participation from all existing Upwind investors, including Greylock Partners, Cyberstarts, Craft, Salesforce Ventures, Alta Park, Picture Capital, Swish Ventures, Cerc

OpenAI’s GPT-6 Cyber found 81 Firefox flaws earlier AI testing missed OpenAI has introduced GPT-6 Cyber, a version of GPT-6 Astra trained to find software vulnerabilities, test how attackers could exploit them, and help develop fixes. The model is designed for authorized security testing of websites and applications. Vulnerability research and security testing OpenAI used GPT-6 Cyber to help secure its systems, including the infrastructure used to train it

Large Language Model (LLM) agents have become increasingly sophisticated, particularly in cybersecurity. Modern AI agents can autonomously hack mock “capture-the-flag” style websites and exploit real-world vulnerabilities when provided with descriptions. However, they struggle with real-world vulnerabilities that are unknown ahead of time, which also known as zero-day vulnerabilities. In a new paper Teams of LLM Agents can Exploit Zero-Day Vulnerabilities,

GitHub Agentic Autofix and the question of who owns a bad AI patch In this Help Net Security interview, Marcelo Oliveira, VP of Product Management at GitHub, talks about who is accountable when an AI-generated patch fixes one flaw and introduces another. He explains how Agentic Autofix re-runs detection engines and project tests, and what a CISO should change before pointing an agent at thousands of old alerts. Oliveira also discusses which metrics show a

For artificial intelligence to thrive in a complex, constantly evolving world, it must overcome significant challenges: limited data quality and scale, and a lag in new, relevant information creation. An intriguing question emerges—can language models autonomously generate new tasks to practice on, enabling them to self-improve and better align with human preferences? In a new paper Evolving Alignment via Asymmetric Self-Play, a research team from Google D

Share My Research is Synced’s column that welcomes scholars to share their own research breakthroughs with over 1.5M global AI enthusiasts. Beyond technological advances, Share My Research also calls for interesting stories behind the research and exciting research ideas. Contact us: chain.zhang@jiqizhixin.com Meet the authorsInstitutions: Penn State University, Duke University, Google DeepMind, University of Washington, Meta, Nanyang Technological Univers

Credit: Pixabay/CC0 Public Domain For farmers, every planting decision carries risks, and many of those risks are increasing with climate change. One of the most consequential is weather, which can damage crop yields and livelihoods. A delayed monsoon, for example, can force a rice farmer in South Asia to replant or switch crops altogether, losing both time and income. Access to reliable, timely weather forecasts can help farmers prepare for the weeks ahea