Have you heard about the recent AI-generated paper that was peer reviewed at ICLR 2025 workshop? Yes, it’s by company called Sakana AI! Let’s discuss their product, AI Scientist, briefly in this post.

Contents:

✨ Sakana AI

I first heard about Sakana AI from a friend a week or two ago, and their AI Scientist immediately caught my attention. This AI agent isn’t just another research assistant—it generates novel ideas, writes code, runs experiments, visualizes results, and even composes full scientific papers, complete with a simulated review process for evaluation. After briefly reading the AI Scientist paper, I was intrigued by its potential and, since it’s open-source, I’m already thinking about how to implement it in my own field. The cost of generating these papers is surprisingly low—around $15 per paper. Using just a single 8x NVIDIA H100 machine, they generated hundreds of papers in a week — something that would have definitely made my supervisor very happy if I could do the same!

🌟 AI-Generated Paper

The 3 papers submitted to the ICLR 2025 workshop were generated by The AI Scientist-v2, which was an improved version of the original AI Scientist, although the full details on the new model have yet to be released. The ICLR workshop name is “I Can’t Believe It’s Not Better: Challenges in Applied Deep Learning”. I think this is an interesting workshop which focuses on the challenges and failure modes of deep learning models. Of course, the organizers are fully aware that the paper was AI generated as Sakana AI has previously seeked permission to “test” this experiment. The reviewers were only told that they might be reviewing AI generated papers (3 out of 43 papers) but were not told which ones. One of the papers, titled “Compositional Regularization: Unexpected Obstacles in Enhancing Neural Network Generalization”, received a score of 6.33 which is above the acceptance threshold. Nevertheless, the paper was eventually withdrawn after the reviewing process as it’s still unclear if AI-generated papers should be accepted at these venues. They also noted that none of the three papers met the threshold for acceptance in the ICLR main conference track. An interesting idea being brought up is using the AI Scientist to automate the reproducibility of existing papers instead of just generating new ones. Reproducibility is super important, but still pretty lacking in the research community, so this could actually be a game-changer. Also,looks like there’s another AI-generated paper accepted at the Tiny Papers workshop track at ICLR, this time from an AI agent called Carl. Unlike Sakana’s AI Scientist, though, this one still had some human intervention.

⚠️ Limitations

  • Experiment details can sometimes be incorrect.
  • Related work is incomplete and overly general.
  • Cites incorrect references.
  • Lacks precision in technical mathematical details.
  • Figure captions can be inaccurate.
  • Claims are not always clearly supported by the presented evidence and often lack further explanation.
  • Has a tendency to overclaim.

💭 Questions to Ponder

  1. Should AI-generated papers be submitted to the same venues as human-written ones, or do they need a separate category?
  2. If reviewers can’t distinguish between AI-generated and human-written papers, does it really matter how they were created?
  3. Are AI-generated papers just combining existing ideas, or can they truly create something novel?

📚 References