Mind Crunches #28: Links to Finetune Your LLM
📌 Mind Crunches
OpenAI released two major new preview models last month: o1-preview and o1-mini previously rumored as having the codename “strawberry”. There’s a lot to understand about these models—they’re not as simple as the next step up from GPT-4o, instead introducing some major trade-offs in terms of cost and performance in exchange for improved “reasoning” capabilities. One way to think about these new models is as a specialized extension of the chain of thought prompting pattern—a “think step by step” concept. Pre-trained models are doing next token prediction on an enormous amount of data. They rely on “training-time compute.” An emergent property of scale is basic reasoning, but this reasoning is very limited. What if you could teach a model to reason more directly? This is essentially what’s happening with o-1 models. When we say “inference-time compute” what we mean is asking the model to stop and think before giving you a response, which requires more compute at inference time (hence “inference-time compute”). In case this rings a bell, it’s pretty much the same concept as Kahneman’s System 1/2 thinking. So where do we go from here? I see two major trends:
A rapid rise of an agentic economy built on the next-gen reasoning capabilities of LLMs
More innovation and focus shifting from the Infra stack of AI, which is dominated by Big Tech, to Inference whose costs go down. This means more apps, new use cases, more VC money pouring in the system.
The crypto industry went wild the past weeks with AI agents onchain. Some autonomous agents launched memecoins and demonstrated interesting behaviors like for example tipping humans to convert them into influencers. I am personally cautiously optimistic about the future of agents onchain. There are some use cases that make absolute sense like for example providing crypto payment rails for agents to transact onchain or coordinating thousands of micro-agents to orchestrate an action. There are some other that are still emerging but they are fascinating like attaching a real human behind an agent, embedding an agent in your own wallet to provide personalized financial assistance, hooking agents in prediction markets etc. There is also an innovation race for providing amazing tools to launch agents onchain. Coinbase has an SDK to launch an agent in a few minutes, Brian provides an intent engine that makes agent functional, Giza builds a full stack for ZKML agents. The next months will be interesting.
The world of progress is filled with O-ring problems. For example, driverless cars -a manifestation of brilliant software and hardware engineering- will not be super useful if our roads are bad -something which only requires basic state capacity-. In a similar fashion, next gen AI won’t accelerate Bio sciences if we don’t first address “infra problems” like the cost of raw materials. This is another post from Tyler Cowen on whether AI itself suffers from O-ring problems. I had written about the O-ring theory in one of my previous posts and I have to admit that once you internalize the concept, it becomes like a second nature. You constantly try to identify O-rings that can hinder the success of your idea/project/startup.
How exactly is it working for Palantir? Highly recommend you reading the whole thing. Especially, the part about moral decision making in tech is fascinating. In a way, this applies to crypto as well. For many people in crypto, the “System” (tradfi, institutions etc) is a priori immoral, hence working with the “System” is anti-crypto. A more empirical/realistic/effective approach would be to pursue crypto projects with the “System”, infuse new behaviors, change the paradigm.
Satya’s speech on the new world currency: Tokens x USD x Watt. The Crypto version of the currency is: Tokens x Tokens x Node.
The Startup Pirate on the vibrant Greek crypto ecosystem! A combination of micro and macro factors built the right conditions for many Greeks to play a pivotal role in the future of crypto.
Casey Handmer makes the case for a terraformed American West. The American West is empty. We could have many more metropolitan cities and a better life for millions of people (this is a good book on why need millions of more people in America). The main problem is water scarcity. The key to solving water scarcity is to make more of it using solar powered desalination technology.
The Atlas of live players and institutions by the excellent Samo Burja. Again, do yourself a favor and subscribe to Palladium. It’s definitely expensive for a digital subscription but it’s totally worth it!
Ben Thompson is bullish on Meta’s AI future. His main thesis is that GenAI can make a big impact in advertising and further enhance Meta’s strategy to bypass ATT.
Oldie but goldie. Paul Graham on Makers’ vs Managers’ Schedule. The way it works for me -that I am not a maker- is to block days, or big chunks of days, for back to back meetings and reserve the rest for deep work.
The excellent Jason Zhao on how blockchains will eat the world. Come for the tool, stay for the network.
My observations on the future of prediction markets.
Sotheby’s first AI art exhibition by Botto. Since 2021, Botto has been generating prompts for text-to-image models without direct human intervention, seeking to create what might qualify as true "art". An open, decentralized community votes on Botto's outputs, training Botto in artistic sensibilities. Weekly, the most popular image is published as a canonical Botto artwork that is then sent to auction. To date, Botto has sold over $4M worth of art, from just over 140 unique pieces.
Charter cities and network states are currently one of the hottest startup sectors. My personal thesis is that charter cities/special economic is a fascinating concept and have already proven to deliver interesting results. I am all for more innovation, more charter cities, more experimentation. But the concept of “network states” the way it’s articulated in Balajis’ book is problematic mostly in the sense that it envisions a non-pluralistic future focusing on exitocracy when the future we should aspire to is one of deliberate bridging of antithesis and plural governance.
The brilliant Devon Zuegel founded Esmeralda, a new city outside San Fransisco.
Praxis raised >$500M to build a new city from scratch
Zuitzerland is buying land for 10,000 people in Switzerland, with the eventual goal to be Switzerland's 27th Canton (Swiss version of state)
I have written multiple times in the past how fragile the MSFT<>OAI partnership is. It’s now an established reality that the bromance is over.
What ladders are you climbing? A great read on the importance of carefully choosing which hierarchies we decide to climb and why.
The one video you have to watch if you want to understand memecoins. As the famous lyric goes “Nobody know what it means, but it’s provocative. Gets the people going”
This is a good breakdown of how much smart people are paying to boost their AI productivity. I am using many of these premium services myself. I expect a big consolidation happening pretty soon for most of these services but i still see a market for extremely narrow/specialized LLMs to assist specific professions.
We have now mapped all the 140,000 neurons in the fruit-fly brain. Magic.
Packy on Cuby, the most exciting startup that you have never heard of. Build more housing. End of discussion.
On the importance of being “taught a culture”. Karlsson makes the case that “culture” is a catalyst in the effectiveness of all teaching methods, interventions and tools and if you learn how to “scale culture” you can actually build a new educational movement that will shape individuals throughout their lives the same way that Montessori and Jesuit schools did.
One thing I am convinced about is the natural PMF of stablecoins in emerging markets. Here is a great report from Castle Island Ventures.
Short comments on Draghi’s European competitiveness report.
OnlyFans is a hell of a business.
Why the energy-hungry AI industry is actually good for the climate. tldr: Clean energy technologies are famous for their learning rates - they get cheaper as you build more of them. Data centres create price-insensitive clean power demand that will foot the bill of expensive early projects. Read the whole thing!
Lux’s letters to their LPs are always a must read.
A robot controlled by a fungus. Remember! Fungi are truly amazing.
A fascinating read on the concept of Advanced Market Commitments or aka Economy as a Service.
Recommended Blog: Ben James’ blog about the Climate. One of the best blogs out there.
Recommended Book: The MANIAC - Benjamin Labatut
Recommended Podcast: The MetaScience podcast series
Quote: “Bring forward what is true. Write it so that it is clear. Defend it to your last breath.” Ludwig Boltzmann