High cost and verification difficulty for AI mathematical proofs
Problem
An AI model solved only ~5% of 8,000 mathematical problems after a single 3-hour attempt, with a back-of-the-envelope cost of ~$810 per solution. Readers found the generated proofs messy and hard to understand, requiring significant human time to verify. Additionally, reliance on formalization (e.g., Lean) may not validate the original natural language reasoning, creating a trust gap for practitioners.
Pain points
- ~5% success rate on longstanding math problems; ~60 hours and ~$810 compute cost per solution
- AI-generated math proofs are messy and unclear, requiring additional AI assistance to understand, increasing human cognitive load
- Humans spend significant time reading and verifying AI-generated code/proofs, work that is thankless and uncredited
- Lean formalized code may be correct but does not guarantee validity of the natural language argument, leading to trust gaps
- Autoformalization has limitations in verifying mathematical texts, reducing confidence in original NL arguments
Questions to verify
- What are the actual token usage and compute costs compared to the estimated $810 per solution?
- Can current autoformalization processes (e.g., Lean verification) effectively address the trust issue in natural language proofs?
- For typical software engineering teams, does introducing AI-generated proofs actually improve efficiency or mainly increase verification burden?
Evidence
quote
apparently they tried the model on about 8,000 problems. So, right now it “merely” solves ~5% of the longstanding open mathematical problems that it’s asked about, the problems that whole communities have spent years on, after a single 3-hour attempt on them
quote
It feels like something written by someone who’s on psychedelics. So much unclear and doesn’t make sense.
quote
The "OpenAI model" sets up a crazy race among humans to digest and explain a messy AI proof (work that could easily be some combination of thankless, barely-credited, competitive, and unfun)
quote
But it also appears that no human has understood just about any of these proofs yet. Has anyone verified any of the proofs produced by OpenAI or is everyone just assuming that it just be true because the Lean code checks out?
Disclaimer: this card is distilled by AI from community discussions as “pain point evidence”. It is a starting point for need analysis and validation, not a “validated market need”. AI has analytical limits; please open the original discussion above to manually verify before key decisions.