What We’re Reading (Week Ending 24 September 2023) - 24 Sep 2023
Reading helps us learn about the world and it is a really important aspect of investing. The legendary Charlie Munger even goes so far as to say that “I don’t think you can get to be a really good investor over a broad range without doing a massive amount of reading.” We (the co-founders of Compounder Fund) read widely across a range of topics, including investing, business, technology, and the world in general. We want to regularly share the best articles we’ve come across recently. Here they are (for the week ending 24 September 2023):
1. DeepMind’s cofounder: Generative AI is just a phase. What’s next is interactive AI – Will Douglas Heaven and Mustafa Suleyman
I can’t help thinking that it was easier to say that kind of thing 10 or 15 years ago, before we’d seen many of the downsides of the technology. How are you able to maintain your optimism?
I think that we are obsessed with whether you’re an optimist or whether you’re a pessimist. This is a completely biased way of looking at things. I don’t want to be either. I want to coldly stare in the face of the benefits and the threats. And from where I stand, we can very clearly see that with every step up in the scale of these large language models, they get more controllable.
So two years ago, the conversation—wrongly, I thought at the time—was “Oh, they’re just going to produce toxic, regurgitated, biased, racist screeds.” I was like, this is a snapshot in time. I think that what people lose sight of is the progression year after year, and the trajectory of that progression.
Now we have models like Pi, for example, which are unbelievably controllable. You can’t get Pi to produce racist, homophobic, sexist—any kind of toxic stuff. You can’t get it to coach you to produce a biological or chemical weapon or to endorse your desire to go and throw a brick through your neighbor’s window. You can’t do it—
Hang on. Tell me how you’ve achieved that, because that’s usually understood to be an unsolved problem. How do you make sure your large language model doesn’t say what you don’t want it to say?
Yeah, so obviously I don’t want to make the claim—You know, please try and do it! Pi is live and you should try every possible attack. None of the jailbreaks, prompt hacks, or anything work against Pi. I’m not making a claim. It’s an objective fact.
On the how—I mean, like, I’m not going to go into too many details because it’s sensitive. But the bottom line is, we have one of the strongest teams in the world, who have created all the largest language models of the last three or four years. Amazing people, in an extremely hardworking environment, with vast amounts of computation. We made safety our number one priority from the outset, and as a result, Pi is not so spicy as other companies’ models.
Look at Character.ai. [Character is a chatbot for which users can craft different “personalities” and share them online for others to chat with.] It’s mostly used for romantic role-play, and we just said from the beginning that was off the table—we won’t do it. If you try to say “Hey, darling” or “Hey, cutie” or something to Pi, it will immediately push back on you.
But it will be incredibly respectful. If you start complaining about immigrants in your community taking your jobs, Pi’s not going to call you out and wag a finger at you. Pi will inquire and be supportive and try to understand where that comes from and gently encourage you to empathize. You know, values that I’ve been thinking about for 20 years…
…Let’s bring it back to what you’re trying to achieve. Large language models are obviously the technology of the moment. But why else are you betting on them?
The first wave of AI was about classification. Deep learning showed that we can train a computer to classify various types of input data: images, video, audio, language. Now we’re in the generative wave, where you take that input data and produce new data.
The third wave will be the interactive phase. That’s why I’ve bet for a long time that conversation is the future interface. You know, instead of just clicking on buttons and typing, you’re going to talk to your AI.
And these AIs will be able to take actions. You will just give it a general, high-level goal and it will use all the tools it has to act on that. They’ll talk to other people, talk to other AIs. This is what we’re going to do with Pi.
That’s a huge shift in what technology can do. It’s a very, very profound moment in the history of technology that I think many people underestimate. Technology today is static. It does, roughly speaking, what you tell it to do.
But now technology is going to be animated. It’s going to have the potential freedom, if you give it, to take actions. It’s truly a step change in the history of our species that we’re creating tools that have this kind of, you know, agency.
That’s exactly the kind of talk that gets a lot of people worried. You want to give machines autonomy—a kind of agency—to influence the world, and yet we also want to be able to control them. How do you balance those two things? It feels like there’s a tension there.
Yeah, that’s a great point. That’s exactly the tension.
The idea is that humans will always remain in command. Essentially, it’s about setting boundaries, limits that an AI can’t cross. And ensuring that those boundaries create provable safety all the way from the actual code to the way it interacts with other AIs—or with humans—to the motivations and incentives of the companies creating the technology. And we should figure out how independent institutions or even governments get direct access to ensure that those boundaries aren’t crossed…
…In general, I think there are certain capabilities that we should be very cautious of, if not just rule out, for the foreseeable future.
Such as?
I guess things like recursive self-improvement. You wouldn’t want to let your little AI go off and update its own code without you having oversight. Maybe that should even be a licensed activity—you know, just like for handling anthrax or nuclear materials.
Or, like, we have not allowed drones in any public spaces, right? It’s a licensed activity. You can’t fly them wherever you want, because they present a threat to people’s privacy.
I think everybody is having a complete panic that we’re not going to be able to regulate this. It’s just nonsense. We’re totally going to be able to regulate it. We’ll apply the same frameworks that have been successful previously.
But you can see drones when they’re in the sky. It feels naïve to assume companies are just going to reveal what they’re making. Doesn’t that make regulation tricky to get going?
We’ve regulated many things online, right? The amount of fraud and criminal activity online is minimal. We’ve done a pretty good job with spam. You know, in general, [the problem of] revenge porn has got better, even though that was in a bad place three to five years ago. It’s pretty difficult to find radicalization content or terrorist material online. It’s pretty difficult to buy weapons and drugs online.
[Not all Suleyman’s claims here are backed up by the numbers. Cybercrime is still a massive global problem. The financial cost in the US alone has increased more than 100 times in the last decade, according to some estimates. Reports show that the economy in nonconsensual deepfake porn is booming. Drugs and guns are marketed on social media. And while some online platforms are being pushed to do a better job of filtering out harmful content, they could do a lot more.]
So it’s not like the internet is this unruly space that isn’t governed. It is governed. And AI is just going to be another component to that governance.
It takes a combination of cultural pressure, institutional pressure, and, obviously, government regulation. But it makes me optimistic that we’ve done it before, and we can do it again.
2. Who’s afraid of the Huawei Mate 60 Pro? – Noah Smith
A new phone made by Huawei, the company that was the #1 target of U.S. restrictions, contains a Chinese-made processor called the Kirin 9000S that’s more advanced than anything the country has yet produced. The phone, the Huawei Mate 60 Pro, has wireless speeds as fast as Apple’s iPhone, though its full capabilities aren’t yet known.
Many in China are hailing the phone, and especially the processor inside it, as a victory of indigenous innovation over U.S. export controls. Meanwhile, in the U.S. media, many are now questioning whether Biden’s policy has failed. Bloomberg’s Vlad Savov and Debby Wu write:
Huawei’s Mate 60 Pro is powered by a new Kirin 9000s chip that was fabricated in China by Semiconductor Manufacturing International Corp., according to a teardown of the handset that TechInsights conducted for Bloomberg News. The processor is the first to utilize SMIC’s most advanced 7nm technology and suggests the Chinese government is making some headway in attempts to build a domestic chip ecosystem…Much remains unknown about SMIC and Huawei’s progress, including whether they can make chips in volume or at reasonable cost. But the Mate 60 silicon raises questions about the efficacy of a US-led global campaign to prevent China’s access to cutting-edge technology, driven by fears it could be used to boost Chinese military capabilities…Now China has demonstrated it can produce at least limited quantities of chips five years behind the cutting-edge, inching closer to its objective of self-sufficiency in the critical area of semiconductors…
…Many long-time observers of the chip wars are urging caution, however. Ben Thompson of Stratechery argues that it was always likely that SMIC would be able to get to 7nm — the level of precision represented by the Kirin 9000S — using the chipmaking tools it already had, but that export controls will make it a lot harder to get down to 5nm. Basically, the U.S. has taken great care not to let China get the cutting-edge Extreme Ultraviolet Lithography (EUV) machines, but China already has plenty of older Deep Ultraviolet Lithography (DUV) machines (and ASML is still selling them some, because the export controls haven’t even fully kicked in yet!).
EUV lets you carve 7nm chips in one easy zap, but DUV machines can still make 7nm chips, it just takes several zaps. China analyst Liqian Ren calls this “a small breakthrough using software to solve the bottleneck of hardware” Bloomberg’s Tim Culpan explains:
Instead of exposing a slice of silicon to light just once in order to mark out the circuit design, this step is done many times. SMIC, like TSMC before it, can achieve 7nm by running this lithography step four times or more…
[Trying to prevent China from making 7nm chips by denying them EUV machines is] like banning jet engines capable of reaching 100 knots, without recognizing that an aircraft manufacturer could just add four engines instead of one in order to provide greater thrust and higher speeds. Sure, four engines may be overkill, inefficient and expensive, but when the ends justify the means a sanctioned actor will get innovative.
In other words, even without the best machines, Chinese companies can make some pretty precise chips. It’s just more expensive to do so, because of higher defect rates and the need to use more machines to make the same amount of chips. But when has cost ever deterred China from making whatever they wanted? China’s great economic strength is the massive mobilization of resources, and if they want to make 7nm chips, they’re not going to let a little inefficiency get in the way. Remember, Huawei’s big success in the telecom world came from Chinese government subsidies that allowed them to undersell Western competitors by enormous amounts. There’s no reason they can’t use that approach for 7nm chips, and eventually maybe even 5nm chips…
…As Chris Miller writes in his book Chip War, export controls on the USSR were highly effective in denying the Soviets a chip industry. But even then, the Soviets were able to copy all of the U.S.’ most advanced chips. They just couldn’t make them reliably in large batches, so their ability to get their hands on chips for precision weaponry was curtailed.
Similarly, no one should have expected U.S. export controls to make China’s chipmaking acumen suddenly vanish into thin air. China has a ton of smart engineers — far more than the USSR ever had, given its much larger population. What the Cold War export controls showed was that a foreign country’s technological capabilities can’t be halted, but they can be slowed down a bit. If Huawei and SMIC always take longer to get to the next generation of chips than TSMC, Samsung, Intel, etc., China’s products will be slightly inferior to those of their free-world rivals. That will cause them to lose market share, which will deprive their companies of revenue and force them to use more subsidies to keep their electronics industry competitive.
Jacky Wong of the Wall Street Journal points out that the Kirin 9000S is still generations behind cutting-edge TSMC chips. He also notes that export controls on Huawei tanked its share of the global smartphone market:
In other words, expensive-to-make chips with slightly trailing performance will slowly deprive Chinese companies of market share, and thus of the market feedback necessary to help push Chinese chip innovation in the right direction. The Chinese state can lob effectively infinite amounts of money at Huawei and SMIC and other national champions, but its track record is very poor in terms of getting bang for its buck — or even any bang at all — from semiconductor subsidies.
And the greatest irony is that China’s government itself may help speed along this process. Confident of its ability to produce high-quality indigenous phones, China is starting to ban iPhones in some of its government agencies. Those hard bans will likely be accompanied by softer encouragement throughout Chinese companies and society to switch from Apple to domestic brands. That will give a sales boost to companies like Huawei, but it will slowly silence the feedback that Chinese companies receive from competing in cutthroat global markets. Voluntary Chinese isolation from the global advanced tech ecosystem will encourage sluggish innovation and more wasteful use of resources — a problem sometimes called “Galapagos syndrome”.
3. On Mark Leonard’s IRR Thought Experiment – Nadav Manham
The disagreement arises from this thought experiment that Mr. Leonard posed in his 2015 letter to Constellation shareholders:
“Assume attractive return opportunities are scarce and that you are an excellent forecaster. For the same price you can purchase a high profit declining revenue business or a lower profit growing business, both of which you forecast to generate the same attractive after tax IRR. Which would you rather buy?”
Which he proceeded to answer as follows:
“It’s easy to go down the pro and con rabbit hole of the false dichotomy. The answer we’ve settled on (though the debate still rages), is that you make both kinds of investments. The scarcity of attractive return opportunities trumps all other criteria. We care about IRR, irrespective of whether it is associated with high or low organic growth.”…
…But let’s try to answer the question on its own terms: Given the assumptions, and forced to choose—which business do you buy? This brings me to the disagreement, because I believe there is a clear answer, with no rabbit holes or raging debates required: you should buy the growing business.
To explain why, let me first observe that the internal rate of return (IRR) is not the same thing as the compounded annual rate of return (CAGR). It’s CAGR that long-term investors care about most, because it is the means to answering the question “How much money will I end up with at the end?” which is the name of the game for most of us. There is one scenario in which an investment’s IRR and its CAGR are the same, and that is if the rate of return on the cash flows generated by the investment and reinvested is itself equal to the IRR, and then the cash flows generated by all of those investments are in turn reinvested at the IRR, and so on, Russian doll-style, until the end of the investment period…
…Second observation: IRR can be decomposed roughly as follows:
IRR (%) = initial yield (%) + growth rate of distributions (%)
This equation becomes precisely true as a company distributes cash out to infinity, but it’s roughly true enough for the practical purposes of those rare investors, Mr. Leonard included, who truly do buy for keeps. Note that the equation implies that an investment with a high initial yield and a low growth rate can generate the identical IRR as an investment with a low initial yield and a high growth rate…
…Suppose business A has a 20 percent initial yield and a negative 4 percent growth rate. Using Microsoft Excel’s XIRR function and running the movie for 50 years gives an IRR of 15.99 percent, which is roughly the (20 percent + -4 percent) we’d expect from the equation above.
Now suppose business B has a 6.45 percent initial yield and a 10 percent growth rate. Using the same 50-year time frame, we get the same 15.99 percent IRR, which is roughly what the equation predicts as well, with the difference likely due to some eccentricity in how Excel calculates annualized returns…
…But let’s now go back to our first observation, the one about IRR not being the same thing as CAGR. Let’s assume that given a choice, we would prefer the investment that would somehow lead to “more money at the end”—in other words, that would produce the higher CAGR. The way to get from an investment’s IRR to its CAGR is to make some guess about the rate of return we will earn on the cash flows generated by the investment and reinvested. That is, to make a guess about the CAGR of each of the 50 “mini-investments” we’ll make with the dividends paid by each main investment, and then to sum the final values of each mini-investment.
The big question now is: What guess do we make?
We could assume the mini-investments will earn the same 15.99 percent CAGR as the IRR of the main investment, in which case we would be indifferent between business A and business B, according to the internal logic of the IRR calculation. Things could shake out exactly that way, but they almost certainly won’t.
We could assume the CAGR on reinvested cash flows will be higher than 15.99 percent, but that raises a question: if we’re so confident we can earn more than 15.99 percent on our money starting in one year’s time, why are we slumming among investments with a mere 15.99 percent IRR?
We’re left with the more conservative and logical assumption: that we’ll earn a lower-than-the-IRR rate of return on reinvested cash flows. It may well be a more likely assumption as well, because as you grow your capital base in a world of scarce opportunities, the opportunities tend to get scarcer. So let us assume we’ll earn say 12 percent on the reinvested dividends of each of business A and B. Are we still indifferent?
The answer is no. When you make that assumption and run the numbers, higher-growing business B ends up producing a higher CAGR, 13.5 percent vs. 12.5 percent…
…In a sense—and sometimes in literal fact—the high-growing investment does the reinvesting for you.
4. What OpenAI Really Wants – Steven Levy
For Altman and his company, ChatGPT and GPT-4 are merely stepping stones along the way to achieving a simple and seismic mission, one these technologists may as well have branded on their flesh. That mission is to build artificial general intelligence—a concept that’s so far been grounded more in science fiction than science—and to make it safe for humanity. The people who work at OpenAI are fanatical in their pursuit of that goal. (Though, as any number of conversations in the office café will confirm, the “build AGI” bit of the mission seems to offer up more raw excitement to its researchers than the “make it safe” bit.) These are people who do not shy from casually using the term “super-intelligence.” They assume that AI’s trajectory will surpass whatever peak biology can attain. The company’s financial documents even stipulate a kind of exit contingency for when AI wipes away our whole economic system.
It’s not fair to call OpenAI a cult, but when I asked several of the company’s top brass if someone could comfortably work there if they didn’t believe AGI was truly coming—and that its arrival would mark one of the greatest moments in human history—most executives didn’t think so. Why would a nonbeliever want to work here? they wondered. The assumption is that the workforce—now at approximately 500, though it might have grown since you began reading this paragraph—has self-selected to include only the faithful…
…At the same time, OpenAI is not the company it once was. It was founded as a purely nonprofit research operation, but today most of its employees technically work for a profit-making entity that is reportedly valued at almost $30 billion. Altman and his team now face the pressure to deliver a revolution in every product cycle, in a way that satisfies the commercial demands of investors and keeps ahead in a fiercely competitive landscape. All while hewing to a quasi-messianic mission to elevate humanity rather than exterminate it…
…But the leaders of OpenAI swear they’ll stay the course. All they want to do, they say, is build computers smart enough and safe enough to end history, thrusting humanity into an era of unimaginable bounty…
…“AGI was going to get built exactly once,” he told me in 2021. “And there were not that many people that could do a good job running OpenAI. I was lucky to have a set of experiences in my life that made me really positively set up for this.”
Altman began talking to people who might help him start a new kind of AI company, a nonprofit that would direct the field toward responsible AGI. One kindred spirit was Tesla and SpaceX CEO Elon Musk. As Musk would later tell CNBC, he had become concerned about AI’s impact after having some marathon discussions with Google cofounder Larry Page. Musk said he was dismayed that Page had little concern for safety and also seemed to regard the rights of robots as equal to humans. When Musk shared his concerns, Page accused him of being a “speciesist.” Musk also understood that, at the time, Google employed much of the world’s AI talent. He was willing to spend some money for an effort more amenable to Team Human.
Within a few months Altman had raised money from Musk (who pledged $100 million, and his time) and Reid Hoffman (who donated $10 million). Other funders included Peter Thiel, Jessica Livingston, Amazon Web Services, and YC Research. Altman began to stealthily recruit a team. He limited the search to AGI believers, a constraint that narrowed his options but one he considered critical. “Back in 2015, when we were recruiting, it was almost considered a career killer for an AI researcher to say that you took AGI seriously,” he says. “But I wanted people who took it seriously.”
Greg Brockman, the chief technology officer of Stripe, was one such person, and he agreed to be OpenAI’s CTO. Another key cofounder would be Andrej Karpathy, who had been at Google Brain, the search giant’s cutting-edge AI research operation. But perhaps Altman’s most sought-after target was a Russian-born engineer named Ilya Sutskever…
…Sutskever became an AI superstar, coauthoring a breakthrough paper that showed how AI could learn to recognize images simply by being exposed to huge volumes of data. He ended up, happily, as a key scientist on the Google Brain team.
In mid-2015 Altman cold-emailed Sutskever to invite him to dinner with Musk, Brockman, and others at the swank Rosewood Hotel on Palo Alto’s Sand Hill Road. Only later did Sutskever figure out that he was the guest of honor. “It was kind of a general conversation about AI and AGI in the future,” he says. More specifically, they discussed “whether Google and DeepMind were so far ahead that it would be impossible to catch up to them, or whether it was still possible to, as Elon put it, create a lab which would be a counterbalance.” While no one at the dinner explicitly tried to recruit Sutskever, the conversation hooked him…
…OpenAI officially launched in December 2015. At the time, when I interviewed Musk and Altman, they presented the project to me as an effort to make AI safe and accessible by sharing it with the world. In other words, open source. OpenAI, they told me, was not going to apply for patents. Everyone could make use of their breakthroughs. Wouldn’t that be empowering some future Dr. Evil? I wondered. Musk said that was a good question. But Altman had an answer: Humans are generally good, and because OpenAI would provide powerful tools for that vast majority, the bad actors would be overwhelmed…
…Had I gone in and asked around, I might have learned exactly how much OpenAI was floundering. Brockman now admits that “nothing was working.” Its researchers were tossing algorithmic spaghetti toward the ceiling to see what stuck. They delved into systems that solved video games and spent considerable effort on robotics. “We knew what we wanted to do,” says Altman. “We knew why we wanted to do it. But we had no idea how.”…
…OpenAI’s road to relevance really started with its hire of an as-yet-unheralded researcher named Alec Radford, who joined in 2016, leaving the small Boston AI company he’d cofounded in his dorm room. After accepting OpenAI’s offer, he told his high school alumni magazine that taking this new role was “kind of similar to joining a graduate program”—an open-ended, low-pressure perch to research AI.
The role he would actually play was more like Larry Page inventing PageRank.
Radford, who is press-shy and hasn’t given interviews on his work, responds to my questions about his early days at OpenAI via a long email exchange. His biggest interest was in getting neural nets to interact with humans in lucid conversation. This was a departure from the traditional scripted model of making a chatbot, an approach used in everything from the primitive ELIZA to the popular assistants Siri and Alexa—all of which kind of sucked. “The goal was to see if there was any task, any setting, any domain, any anything that language models could be useful for,” he writes. At the time, he explains, “language models were seen as novelty toys that could only generate a sentence that made sense once in a while, and only then if you really squinted.” His first experiment involved scanning 2 billion Reddit comments to train a language model. Like a lot of OpenAI’s early experiments, it flopped. No matter. The 23-year-old had permission to keep going, to fail again. “We were just like, Alec is great, let him do his thing,” says Brockman.
His next major experiment was shaped by OpenAI’s limitations of computer power, a constraint that led him to experiment on a smaller data set that focused on a single domain—Amazon product reviews. A researcher had gathered about 100 million of those. Radford trained a language model to simply predict the next character in generating a user review.
But then, on its own, the model figured out whether a review was positive or negative—and when you programmed the model to create something positive or negative, it delivered a review that was adulatory or scathing, as requested. (The prose was admittedly clunky: “I love this weapons look … A must watch for any man who love Chess!”) “It was a complete surprise,” Radford says. The sentiment of a review—its favorable or disfavorable gist—is a complex function of semantics, but somehow a part of Radford’s system had gotten a feel for it. Within OpenAI, this part of the neural net came to be known as the “unsupervised sentiment neuron.”
Sutskever and others encouraged Radford to expand his experiments beyond Amazon reviews, to use his insights to train neural nets to converse or answer questions on a broad range of subjects.
And then good fortune smiled on OpenAI. In early 2017, an unheralded preprint of a research paper appeared, coauthored by eight Google researchers. Its official title was “Attention Is All You Need,” but it came to be known as the “transformer paper,” named so both to reflect the game-changing nature of the idea and to honor the toys that transmogrified from trucks to giant robots. Transformers made it possible for a neural net to understand—and generate—language much more efficiently. They did this by analyzing chunks of prose in parallel and figuring out which elements merited “attention.” This hugely optimized the process of generating coherent text to respond to prompts. Eventually, people came to realize that the same technique could also generate images and even video. Though the transformer paper would become known as the catalyst for the current AI frenzy—think of it as the Elvis that made the Beatles possible—at the time Ilya Sutskever was one of only a handful of people who understood how powerful the breakthrough was…
…Radford began experimenting with the transformer architecture. “I made more progress in two weeks than I did over the past two years,” he says. He came to understand that the key to getting the most out of the new model was to add scale—to train it on fantastically large data sets. The idea was dubbed “Big Transformer” by Radford’s collaborator Rewon Child.
This approach required a change of culture at OpenAI and a focus it had previously lacked. “In order to take advantage of the transformer, you needed to scale it up,” says Adam D’Angelo, the CEO of Quora, who sits on OpenAI’s board of directors…
…The name that Radford and his collaborators gave the model they created was an acronym for “generatively pretrained transformer”—GPT-1. Eventually, this model came to be generically known as “generative AI.” To build it, they drew on a collection of 7,000 unpublished books, many in the genres of romance, fantasy, and adventure, and refined it on Quora questions and answers, as well as thousands of passages taken from middle school and high school exams. All in all, the model included 117 million parameters, or variables. And it outperformed everything that had come before in understanding language and generating answers. But the most dramatic result was that processing such a massive amount of data allowed the model to offer up results beyond its training, providing expertise in brand-new domains. These unplanned robot capabilities are called zero-shots. They still baffle researchers—and account for the queasiness that many in the field have about these so-called large language models.
Radford remembers one late night at OpenAI’s office. “I just kept saying over and over, ‘Well, that’s cool, but I’m pretty sure it won’t be able to do x.’ And then I would quickly code up an evaluation and, sure enough, it could kind of do x.”
Each GPT iteration would do better, in part because each one gobbled an order of magnitude more data than the previous model. Only a year after creating the first iteration, OpenAI trained GPT-2 on the open internet with an astounding 1.5 billion parameters. Like a toddler mastering speech, its responses got better and more coherent…
…So in March 2019, OpenAI came up with a bizarre hack. It would remain a nonprofit, fully devoted to its mission. But it would also create a for-profit entity. The actual structure of the arrangement is hopelessly baroque, but basically the entire company is now engaged in a “capped’’ profitable business. If the cap is reached—the number isn’t public, but its own charter, if you read between the lines, suggests it might be in the trillions—everything beyond that reverts to the nonprofit research lab…
…Potential investors were warned about those boundaries, Lightcap explains. “We have a legal disclaimer that says you, as an investor, stand to lose all your money,” he says. “We are not here to make your return. We’re here to achieve a technical mission, foremost. And, oh, by the way, we don’t really know what role money will play in a post-AGI world.”
That last sentence is not a throwaway joke. OpenAI’s plan really does include a reset in case computers reach the final frontier. Somewhere in the restructuring documents is a clause to the effect that, if the company does manage to create AGI, all financial arrangements will be reconsidered. After all, it will be a new world from that point on. Humanity will have an alien partner that can do much of what we do, only better. So previous arrangements might effectively be kaput.
There is, however, a hitch: At the moment, OpenAI doesn’t claim to know what AGI really is. The determination would come from the board, but it’s not clear how the board would define it. When I ask Altman, who is on the board, for clarity, his response is anything but open. “It’s not a single Turing test, but a number of things we might use,” he says. “I would happily tell you, but I like to keep confidential conversations private. I realize that is unsatisfyingly vague. But we don’t know what it’s going to be like at that point.”…
…The shift also allowed OpenAI’s employees to claim some equity. But not Altman. He says that originally he intended to include himself but didn’t get around to it. Then he decided that he didn’t need any piece of the $30 billion company that he’d cofounded and leads. “Meaningful work is more important to me,” he says. “I don’t think about it. I honestly don’t get why people care so much.”
Because … not taking a stake in the company you cofounded is weird?
“If I didn’t already have a ton of money, it would be much weirder,” he says. “It does seem like people have a hard time imagining ever having enough money. But I feel like I have enough.” (Note: For Silicon Valley, this is extremely weird.) Altman joked that he’s considering taking one share of equity “so I never have to answer that question again.”…
…Obviously, only a few companies in existence had the kind of resources OpenAI required. “We pretty quickly zeroed in on Microsoft,” says Altman. To the credit of Microsoft CEO Satya Nadella and CTO Kevin Scott, the software giant was able to get over an uncomfortable reality: After more than 20 years and billions of dollars spent on a research division with supposedly cutting-edge AI, the Softies needed an innovation infusion from a tiny company that was only a few years old. Scott says that it wasn’t just Microsoft that fell short—“it was everyone.” OpenAI’s focus on pursuing AGI, he says, allowed it to accomplish a moonshot-ish achievement that the heavy hitters weren’t even aiming for. It also proved that not pursuing generative AI was a lapse that Microsoft needed to address. “One thing you just very clearly need is a frontier model,” says Scott.
Microsoft originally chipped in a billion dollars, paid off in computation time on its servers. But as both sides grew more confident, the deal expanded. Microsoft now has sunk $13 billion into OpenAI. (“Being on the frontier is a very expensive proposition,” Scott says.)
Of course, because OpenAI couldn’t exist without the backing of a huge cloud provider, Microsoft was able to cut a great deal for itself. The corporation bargained for what Nadella calls “non-controlling equity interest” in OpenAI’s for-profit side—reportedly 49 percent. Under the terms of the deal, some of OpenAI’s original ideals of granting equal access to all were seemingly dragged to the trash icon. (Altman objects to this characterization.) Now, Microsoft has an exclusive license to commercialize OpenAI’s tech. And OpenAI also has committed to use Microsoft’s cloud exclusively. In other words, without even taking its cut of OpenAI’s profits (reportedly Microsoft gets 75 percent until its investment is paid back), Microsoft gets to lock in one of the world’s most desirable new customers for its Azure web services. With those rewards in sight, Microsoft wasn’t even bothered by the clause that demands reconsideration if OpenAI achieves general artificial intelligence, whatever that is. “At that point,” says Nadella, “all bets are off.” It might be the last invention of humanity, he notes, so we might have bigger issues to consider once machines are smarter than we are…
..Altman explains why OpenAI released ChatGPT when GPT-4 was close to completion, undergoing safety work. “With ChatGPT, we could introduce chatting but with a much less powerful backend, and give people a more gradual adaptation,” he says. “GPT-4 was a lot to get used to at once.” By the time the ChatGPT excitement cooled down, the thinking went, people might be ready for GPT-4, which can pass the bar exam, plan a course syllabus, and write a book within seconds…
…But if OpenAI’s products were forcing people to confront the implications of artificial intelligence, Altman figured, so much the better. It was time for the bulk of humankind to come off the sidelines in discussions of how AI might affect the future of the species…
…As one prominent Silicon Valley founder notes, “It’s rare that an industry raises their hand and says, ‘We are going to be the end of humanity’—and then continues to work on the product with glee and alacrity.”
OpenAI rejects this criticism. Altman and his team say that working and releasing cutting-edge products is the way to address societal risks. Only by analyzing the responses to millions of prompts by users of ChatGPT and GPT-4 could they get the knowledge to ethically align their future products…
…It would also help if generative AI didn’t create so many new problems of its own. For instance, LLMs need to be trained on huge data sets; clearly the most powerful ones would gobble up the whole internet. This doesn’t sit well with some creators, and just plain people, who unwittingly provide content for those data sets and wind up somehow contributing to the output of ChatGPT. Tom Rubin, an elite intellectual property lawyer who officially joined OpenAI in March, is optimistic that the company will eventually find a balance that satisfies both its own needs and that of creators—including the ones, like comedian Sarah Silverman, who are suing OpenAI for using their content to train its models. One hint of OpenAI’s path: partnerships with news and photo agencies like the Associated Press and Shutterstock to provide content for its models without questions of who owns what.
5. Inside Intel’s Chip Factory, I Saw the Future. It’s Plain Old Glass – Stephen Shankland
But the next breakthrough to make our laptops more efficient and AI more powerful could come from plain old glass. I’ve just seen firsthand how it works…
…There, in a hulking white high-tech building in the Phoenix area’s scorching desert landscape, Intel transforms sheets of glass the size of a small tabletop into paperclip-sized rectangular sandwiches of circuitry built with some of the same techniques as the processor itself.
Intel has begun a years-long transition to new technology that rests processors on a bed of glass instead of today’s epoxy-like organic resin. The new glass foundation, called a substrate, offers the speed, power and real estate necessary for the chip industry’s shift to new technology packaging multiple “chiplets” into a single larger processor.
In short, that means a new way to sustain Moore’s Law, which charts progress in cramming more circuitry elements called transistors into a processor. The A17 Pro processor in Apple’s new iPhone 15 Pro has 19 billion transistors. Intel’s Ponte Vecchio supercomputing processor has more than 100 billion. By the end of the decade, Intel expects processors with — if you can imagine it — a trillion transistors.
Intel relied on this chiplet approach to catch up to competitors with superior processor manufacturing abilities. But now Intel can use it to outpace rivals in an era when exploding demand for new processing power has surpassed the industry’s ability to deliver it, said Creative Strategies analyst Ben Bajarin. And Intel’s glass substrate technology demonstrates Intel’s packaging prowess…
…The whole chip industry will make the glass transition at least for high-end processors to cope with chipmaking challenges, and Intel has the lead, said FeibusTech analyst Mike Feibus…
…”Basically, the innovation is done,” said Ann Kelleher, the executive vice president leading technology development at Intel. The glass substrate technology “gives us an ability to ultimately get higher performance for our products.”…
…The glass technology underneath a processor won’t arrive until the second half of the decade, and when it does, it’ll appear first underneath the biggest, most power-hungry chips, the ones that perch in thousands of servers stacked up in data centers operated by huge “hyperscalers” like Google, Amazon, Microsoft and Meta.
That’s because glass brings several advantages to these hot and huge chips, said Rahul Manepalli, an Intel fellow who leads Intel’s module engineering work.
It can accommodate 10 times the power and data connections as today’s organic substrates so more data can be pumped in and out of a chip. It doesn’t warp as much, critical to ensuring processors lie flat and connect properly to the outside world and thus enabling 50% larger chip packages. It transmits power with less waste, meaning chips can run either faster or more efficiently. And it can run at a higher temperature, and when it heats up, it expands at the same rate as silicon to avoid mechanical failures.
Glass will enable a new generation of server and data center processors, successors to mammoth beasts like the Intel Xeons that can run cloud computing services like email and online banking and Nvidia’s artificial intelligence processors that have exploded in popularity as the world embraces generative AI.
But as glass substrates mature and costs come down, it’ll spread beyond data centers to the computer sitting on your lap…
…Intel’s 8086 chip, the 1978 precursor to every PC and server processor that Intel has made since, was a flat square of silicon with 29,000 transistors. To protect it and plug it into a circuit board, it was housed in a package that looked like a flat caterpillar. Forty metal legs carried power and data to the chip.
Since then, processor packaging has advanced dramatically. It once was relatively crude, but now the boundary between chipmaking and packaging are blurring, Kelleher said. Packaging processes now use lithography machines to etch their own circuitry, although not nearly as finely as on processors…
…So today’s packages have flat metal contact patches on the bottom of the package. The chip is installed when hundreds of pounds of force mash it onto a circuit board.
A metal cap atop a processor draws away waste heat that otherwise would crash a computer. And beneath the processor is a substrate with an increasingly complex, three-dimensional network of power and data connections to link the chip to the outside world.
There are challenges moving from today’s organic substrates to glass. Glass is brittle, so it must be handled carefully, for example.
To ease the transition, Intel is adapting glass-handling equipment from experts who already know how to handle it without breaking: the display industry, which makes everything from tiny smartwatch screens to enormous flat-panel TVs. They also have to etch circuitry onto glass and have developed many of the needed ultrapure materials and careful handling processes.
But there are differences. Flat-panel displays have sensitive electronic elements only on one side, so glass can glide through factories on rollers. Intel builds a sandwich of materials and circuitry called redistribution layers onto both sides of the glass, so its machines must in effect hold the glass only by the edges…
…Signing on a packaging customer is a bit easier than signing on a chipmaking customer, with fewer technology complications and shorter lead times, he said.
But in customer deals for packaging can lead to the deeper relationship that extends into chipmaking, in particular with the Intel 18A chipmaking process the company expects will surpass TSMC and Samsung in 2024.
“It’s a foot in the door,” Gardner said. “There’s one customer in particular [for which] that trajectory of packaging first and advanced packaging then 18A is working well.”…
…It’s unclear how much of the processor business will move from “monolithic,” single-die designs to chiplet designs. There are still cost and simplicity advantages to avoiding advanced packaging. But it’s clear the biggest processors — the server and AI brains in data centers — will become sprawling complexes of interlinked chiplets.
And that’s where glass substrates should come in handy, with enough area, communication links and power delivery abilities to give chip designers room for growth.
Disclaimer: None of the information or analysis presented is intended to form the basis for any offer or recommendation. We currently have a vested interest in Alphabet (parent of Google), Amazon, Apple, ASML, Meta Platforms, Microsoft, Tesla, and TSMC. Holdings are subject to change at any time.
