12/2025: Current LLMs compress too much data at the same time, which may be the essential reason for reward hacking. A necessary condition for good generalization requires compressing much less data while achieving comparable performance. Same performance, less compression, better generalization.
10-12/2025: Next-token prediction is a special case of the Yoneda lemma. Knowledge and intelligence are the structure of mapping between objects/functions.
10/2025: I believe the most important factors for super-intelligence are (1) Search in Evolving Environments, (2) Verification, (3) Learning, and (4) Scaling (1)-(3).
09/2024: Artificial intelligence can be very smart, but it still needs to follow the truth. If intelligence continues to evolve in the future, seeking the truth to survive in the universe may serve as a creed for a peaceful transition between generations.
Everything should be made as simple as possible, but not simpler. (Albert Einstein)
Deep learning is based on the audacious conjecture that biological neurons and artificial neurons are not that different. (Ilya Sutskever)
The Bitter Lesson. (Richard Sutton)
Useful or elegant, at least one. (?)
若不披上这件衣裳，众生又怎知我尘缘已断、金海尽干。 (Black Myth)