Current Software Engineers have no Deep Knowledge (Jonathan Blow)

Here are detailed notes from the transcript:

Detailed Notes: The Crisis of Knowledge in Programming

Main Topics Discussed:

• The fundamental problem of distinguishing between deep and shallow knowledge in programming.
• The rise and prevalence of shallow knowledge, especially since the mid-2000s.
• The role of API design (good vs. bad) in fostering or hindering deep understanding.
• The decline of "respectable engineering" standards in the software industry.
• The challenges and potential solutions for improving programming culture.

Key Points and Arguments:

1. The Deep vs. Shallow Knowledge Problem (0:03-0:23):

   • A serious, long-standing issue in programming culture (beyond games, everywhere).
   • People lack a clear picture of what constitutes deep versus shallow knowledge.
   • Deep Knowledge Example (0:25-0:37): Understanding algorithms like n log n vs. n squared. This is fundamental, applicable in many cases, and reveals something inherent about computers.
   • Shallow Knowledge Example (0:38-0:59): Learning a specific JavaScript API through a tutorial. This is about arbitrary decisions made by others, not understanding the underlying system or problem.
   • Rise of Shallow Knowledge (1:00-1:19): Became very popular since "around 2005" (the "second boom of programming") when many people entered the field primarily for job/money, learning only surface-level skills like specific JavaScript APIs. This led to a "deluge of shallow knowledge."

2. Learning Deep Knowledge from Shallow Things (1:23-1:03):

   • It is possible to learn deep patterns by looking at shallow implementations, especially through good APIs.
   • A well-designed API can implicitly reveal the problem and underlying mechanics, even without seeing the source code.
   • Example (0:50-1:03): A graphics API requiring vertex data to be put into a locked buffer and not touched suggests data transfer to a GPU, prompting deeper understanding.

3. The Problem of Bad or Cargo-Culted APIs (1:05-1:57):

   • Most APIs are "bad," full of "stupid" decisions, or result from "cargo culting" (copying features without understanding their original purpose).
   • Microsoft Direct2D Example (1:16-1:57): Microsoft copied Direct3D's pattern of "lockable resources" for simple ARGB colors in Direct2D. This was absurd, as a color value doesn't need to be locked and filled like graphics data. This demonstrates a lack of fundamental understanding in API design.

4. Decline of "Respectable Engineering" in Software (2:05-2:40):

   • There is no clear idea of what a "respectable engineer" is in software today.
   • Contrast with Civil Engineering (2:13-2:28): Civil engineers (e.g., building bridges) are seen as responsible, knowledgeable, and meticulous because correctness is paramount.
   • This standard is "completely dead" in software.
   • It's hard to tell "what's right" because the industry is like "a bunch of people arguing in a loud room," where "no real information gets through."

5. Challenges and Skepticism Regarding Solutions (2:45-3:22):

   • The speaker is unsure how to fix the problem, inviting ideas from others.
   • Speaker's Personal Approach (2:50-2:56): To "just do better in an obvious way."
   • Skepticism about Impact (2:56-3:22):
     • Even doing better "maybe doesn't win" (e.g., popular JavaScript frameworks are similar and often rely on shallow practices like excessive NPM dependencies).
     • Analogy: "the market can remain irrational longer than you could remain solvent."
     • The software industry culture "could remain stupid longer than you're alive."
     • Acknowledges a "limited ability to affect this giant thing."

Important Facts or Data Mentioned:

• Shallow knowledge became popular since 2005, coinciding with a "second boom of programming."
• n log n and n squared mentioned as examples of deep algorithmic knowledge.
• JavaScript API tutorials cited as examples of shallow knowledge.
• Microsoft Direct2D and Direct3D specifically named in the cargo-culting example.
• NPM mentioned as a current example of widespread shallow practice (over-reliance on packages for simple pathing).

Conclusions or Recommendations:

• Problem Diagnosis: The software industry suffers from a pervasive lack of clear distinction between deep and shallow knowledge, leading to poor engineering practices, bad API design, and a decayed sense of what constitutes a "respectable engineer."
• Speaker's Recommendation (Personal): To personally strive to "do better in an obvious way," demonstrating superior engineering.
• Outlook/Limitations: The speaker holds a pessimistic view on the industry's ability to self-correct quickly, believing that "stupidity" might persist for a very long time, and acknowledging their own limited capacity to change the entire culture.