Soulcode #3: The Questions About AI We Should Ask

The last few weeks in AI (from someone with a soul)

---

In a world of algorithms and automation, we need a human compass. This newsletter brings you AI and tech news through the lens of ethics and humanity, where code meets consciousness.

We'll explore AI's biggest developments and their impact on our shared future every two weeks(-ish). No jargon, no hype – just clear insights with a conscience.

---

Wednesday morning, as I was wrestling with these questions, something happened that put everything in perspective.

At 5:45 AM, my room slowly filled with the most extraordinary light. The sunrise painted my walls in shades of amber and rose that no digital screen could capture. I got up from my bed, drawn to the window, and took the photograph I've shared above.

In that moment, something crystallized for me: A machine learning algorithm can analyze millions of sunrise photos, generate stunning images, even predict the exact time the sun will peek over the horizon. But it cannot feel that quiet awe, that spontaneous urge to capture beauty, that deeply human moment of connection with the world around us.

Perhaps that's the most important question of all: In our rush to make machines more like us, are we forgetting what makes us uniquely human?

While the world marvels at the latest AI breakthroughs and trillion-dollar investments or feels utterly outraged, I find myself returning to some uncomfortable questions that challenge everything many BELIEVE they hope artificial intelligence can do.

Let me share what's keeping me up at night.

1. The Machine Metaphor

This week's announcement of America’s $500 billion Stargate AI initiative rests on a fundamental assumption: that the brain is essentially a very sophisticated computer. It's an attractive idea. Neat. Clean. Engineering-focused.

But what if they're wrong?

John Searle's famous Chinese Room thought experiment poses a devastating challenge: a system might perfectly manipulate symbols without understanding anything at all. Like a person following instructions to respond to Chinese characters without knowing Chinese, our most advanced AI systems might be sophisticated mimics rather than truly intelligent entities.

The AGI Mythology

“AGI… that’s not the goal. After that, Artificial Super Intelligece will come and solve issues that mankind never had thought we could solve“

—Masa’s words at the Stargate announcement

What if AGI is more a myth than a destination?

Artificial General Intelligence is the promise of machines that can think, learn, and adapt like humans do.

Consider this: despite exponential increases in computing power and investment, we're no closer to understanding the fundamental nature of general intelligence. Our most advanced AI systems remain narrow specialists, brilliant at specific tasks but lacking the fluid, adaptive intelligence of even a young child.

The Money Trail

Now, let's talk about the elephant in the room.

With over half a trillion dollars committed to AI infrastructure in just the past month, we need to ask: Who benefits from the AI narrative? What hidden motives shape the stories we tell about AI's potential?

This isn't conspiracy thinking – it's due diligence. When Chinese companies demonstrate they can match Western AI capabilities despite chip restrictions, it reveals something crucial: the race for AI dominance is as much about power and economics as it is about technological progress.

2. A Different Path Forward

These questions aren't meant to dismiss AI's remarkable achievements. Google's new weather prediction model, GenCast, shows real, practical value.

But perhaps we need to:

• Challenge the ‘machine paradigm.’ Instead of trying to build bigger, faster ‘brains,’ what if we explored fundamentally different approaches to artificial intelligence?

• Redefine success rather than chasing the AGI myth. What if we focused on developing AI that complements human intelligence rather than replicates it?

• Follow our (human) wisdom. What insights might we gain by engaging more deeply with philosophers, neuroscientists, and others who question our basic assumptions about consciousness and intelligence?

The Stakes

The assumptions we embrace about AI today shape the tools we build as much as they shape the world we’re creating for tomorrow. These tools touch every part of our lives, influencing how we work, think, and connect. That’s why it’s so important to question who we’re listening to when it comes to AI narratives and predictions.

Why is it that technologists—brilliant in their fields of computer science or engineering—often step into roles as philosophers, politicians, or even theologians to guide societal conversations about AI? And why do we let them? Entrepreneurs, too, often present a vision of AI that aligns perfectly with their business agenda. Yet their predictions are celebrated as though they come from an objective, universal truth. Social media amplifies it all, turning these voices into the loudest ones in the room.

But louder doesn’t mean wiser.

In my book, I challenge this dynamic and explore the idea that we already have the compass we need. Each of us has an inner voice, a sense of right and wrong, that can guide these conversations. We owe it to ourselves—and to the generations that follow—to trust that voice more than the noise around us. After all, the future isn’t just about what we build; it’s about who we become while building it.

As business leaders deploy AI, as developers build it, as citizens interact with it, we need to think more deeply and assess critically about what we're really creating.

3. An Invitation to Think Differently

Next time you read about an AI breakthrough, try asking:

• What assumptions about intelligence and consciousness underlie this development?

• Whose interests does this narrative serve?

• What alternative approaches might we be overlooking?

I'd love to hear your thoughts: Which of our fundamental assumptions about AI deserves more scrutiny? What questions should we be asking but aren't?

With curiosity and caution,

-a

---

P.S. If these questions resonated with you, share them with someone who thinks differently about AI. The best insights often come from challenging our most basic assumptions.

P.P.S. Speaking of that, I have a super EXCITING announcement! Those of you who have read Honest AI for a while know how much of a fan I am of Erik J Larson’s book and Substack, Colligo.

After a long time, I managed to sit with Erik J Larson and record a one-hour conversation about these hot topics, the questions nobody dares to ask about AI. It will be the inaugural episode of the Honest AI podcast. I plan to launch it by early Feb, and for you, let me give you a sneak peak: