AI Code Review: From Vibe Checks to Real QA

In this lesson: AI ships 1.7x more bugs · Plan before you prompt · Separate verifying from coding

Byron Mackay · 60 min · April 2026
Released April 1, 2026

Top 3 takeaways

01

AI ships 1.7x more bugs

AI-generated PRs carry measurably more performance, security, and logic issues, so faster output needs real quality gates to catch them.

02

Spell out the architecture yourself

Describe the path from button to service to database so the model follows your design. Talk it through with your team, turn the transcript into a spec, then build.

03

Make verification its own loop

Run tests on every change, keep PRs small, and use a separate reviewer model, because your ability to verify quality is the real bottleneck.

Byron Mackay

Byron Mackay

Director of Learning, Gauntlet AI

Director of Learning at Gauntlet AI, currently training hundreds of engineers to work AI-first. 16+ years as a mobile/iOS engineer before becoming an AI platform engineer (Savant, School AI), where he built eval platforms from scratch. Led curriculum development at BloomTech (a cohort of his saw nearly every graduate land an engineering role at Amazon) and ran the Amazon partnership/SDR program that moved non-traditional candidates into engineering roles at Amazon. Deep across platform engineering, AI, mobile, and learning.

Connect with Speaker

Lesson notes

A written walkthrough of the lecture, covering the patterns, the code, and the gotchas.

The quality drop shows up in the numbers

The lecture opens with a blunt premise that AI lets you build faster but not better, and the speaker backs it with concrete metrics on how AI-generated code degrades quality. AI-generated PRs contain 1.7x more issues overall than human-written ones. Performance regressions are 7x more likely, readability has 3x more issues, security vulnerabilities are 2.74x more common, error-handling gaps are 2x more common, and logic and incorrectness errors are 75% more common.

To frame it, the speaker notes that you wouldn't hire an engineer who made errors 1.7x more often than your team, unless they coded 100x faster, and that tradeoff is the world we now live in. The point is to reconsider how we use AI rather than to abandon it. The deeper issue is throughput, because before AI bugs arrived at "human speed" and could be reviewed, whereas now features ship faster than anyone can read the code.

Missing context is the first failure mode

LLMs have no inherent knowledge of your business rules, architectural constraints, or that useful helper function buried elsewhere. Claude Code will search through your code and do an impressive job, yet it still misses constraints. A recurring trap is context-window bloat. Even though Opus advertises a million-token window, that doesn't mean you should fill it. Stop around 100,000 tokens, consolidate, and start a fresh session, because models begin losing recall well before the window fills. Keeping a session alive too long "because it went so well last time" produces issues at a more frequent clip.

Design the path before you prompt

Rather than asking the LLM to "make this button," spell out the full path of how the button sends the API request, how that hits the service layer, and how that reaches the database. The model does a poor job when left to decide the architecture on its own. The concrete workflow for larger tasks looks like this:

  • Get the engineers on the task into a room and talk through the architecture for an hour or two, transcribing with Notion.
  • Hand the transcript to Claude and have it build out a spec list and a task list.
  • Go through those tasks meticulously, editing and fleshing them out.
  • Only then hand it to Claude Code or Codex to implement.

He calls this "pseudocode the logic first," meaning the planning, research, and engineer discussion that builds context before any code is written.

Guardrails that keep a team safe

  • Context files (CLAUDE.md, Codex's AGENTS.md) set rules, constraints, and patterns so the LLM doesn't re-derive your codebase each time, for example a rule that authentication logic may only live in one file.
  • Centralize risk with single ownership and triggers, so that when a sensitive file changes you require review from a specific person or team.
  • Smaller PRs, since above roughly 400 lines you should stop and ask why, and a 7-file-change rule of thumb works the same way. A 1,000-line PR should prompt a "what's really going on here" pause.
  • Naming, because LLMs throw in generic names and shorthand. Name critical things yourself, or remind the model to be explicit.
  • Logical checkpoint commits, plus the tool Graphite for stacked PRs where each builds on the last (database layer, then service layer, then API layer).

Tests, skills, and a separate verification loop

LLMs love writing tests, but those tests are often low value, and the favorite example is a test verifying that a static map is indeed that static map. Direct them toward edge cases, security, and parameter verification instead. Configure the agent to run all tests after every change so it catches regressions, encoded as a skill plus a CLAUDE.md instruction ("always run the test before committing") and backed by pre-commit hooks. The same approach applies to UI via Playwright.

Other guardrails include skills that automate recurring processes (scanning untrusted code, scaling ClickHouse nodes, generating docs), linters (static rather than AI, yet they still catch a wide range of issues and let the LLM self-correct), and model selection, where you use the smartest model at critical gates like pre-production code review.

Finally, separate the loops by keeping verification distinct from coding, and verification can start in the planning phase. This echoes segregation of duties (credited to Dex of HumanLayer), where "the bottleneck is your ability to verify quality." Slow down to a realistic 2x pace, never deploy anything touching payments or user data without review, and treat quality as a moat.

Transcript

Read transcript

AI Code Review: From Vibe Checks to Real QA

00:00:13.630Tom Babb: Byron, you can probably just introduce yourself and… Kill a couple minutes here.

00:00:18.730Bryon Mackay: Oh, sure. Okay.

00:00:20.750Bryon Mackay: Alright!

00:00:23.400Bryon Mackay: Alright, well, welcome, everybody.

00:00:25.510Bryon Mackay: People are coming in. Appreciate y'all coming, making time in your…

00:00:30.180Bryon Mackay: afternoons and evenings to come be here. My name is Byron. I'm gonna be your speaker tonight, and I am the Director of Learning here at Gauntlet, so I really focus a lot on the curriculum, making sure we're keeping things up to date.

00:00:47.580Bryon Mackay: Making sure you have the most pertinent information when you go out, and you have your job, and you're an AI engineer or an AI-first engineer. So, it's an interesting problem to solve, honestly. As many of you can probably attest, keeping up with AI is really hard.

00:01:05.860Bryon Mackay: There is a lot going on.

00:01:08.180Bryon Mackay: And, you know, the… this… this topic we're talking about tonight has really started to…

00:01:14.840Bryon Mackay: surface its ugly head in a way that we all kind of knew it was happening, and then it had just really, like, hit the wall hard. Where we start seeing, like, major issues with major companies on their apps, and everyone's trying to figure out what to do about this. Now.

00:01:33.860Bryon Mackay: You may not have felt this as much, maybe in your roles, or maybe in the apps that you've used, but I'm gonna show some metrics here in a second here that are…

00:01:43.400Bryon Mackay: hopefully gonna be a bit more eye-opening. It was definitely for me. And then we're gonna talk about how to, how to, you know, stop this from happening. It's interesting, though, I will say that…

00:01:55.730Bryon Mackay: There are a number of apps that I use. I won't name names, but they're all big companies, I'm sure you use a lot of the same apps, where I have just found the most random and annoying bugs that I have never seen before.

00:02:08.550Bryon Mackay: And I don't know, I don't think it's a configuration thing, but some of the most simplest of actions will cause something to break.

00:02:18.240Bryon Mackay: And I'm just… I just sit here, I sit back in my chair, and I go, this is an AI problem. This is a uniquely AI problem where we are seeing

00:02:26.180Bryon Mackay: So many more issues. But then the other shocker is these issues don't get resolved, right? They just kind of stick around for a while. So anyway, there's all sorts of big problems and little problems that we're seeing.

00:02:40.810Bryon Mackay: But our jobs as engineers is to fix it, yeah.

00:02:44.220Tom Babb: Quick question, sorry, we didn't address this, but how do you want questions? Some people, they're able to raise their hands. Do you want them to be able to ask questions along the way, or would you prefer that all questions are safe to the end?

00:02:55.840Bryon Mackay: You know what? Let's have him do it on the way. I'll find some spots to pause.

00:03:00.670Bryon Mackay: And then we can take some questions. I think that's tough. Yeah.

00:03:04.800Tom Babb: I'll say to the audience real quick, if you have a question, there's a little button that says, raise your hand, and that will put you in queue, and then when Byron's ready, we'll click allow to talk, and you can ask your question, and we'll just go through questions that way.

00:03:18.880Bryon Mackay: Awesome.

00:03:20.070Bryon Mackay: Okay.

00:03:22.990Bryon Mackay: Austin, we will not be covering that. We will not be covering how to bundle map files in our automated updates. That's not quite the scope of this one, it's more of a detail, but, you know, we'll, we'll get there. So, alright.

00:03:43.070Bryon Mackay: Let's go ahead and get started. Oh, let me give you one more bit of an intro here. So, Tom reminded me of this just a second ago, and I… it's an analogy that I love, and that I,

00:03:59.410Bryon Mackay: It's a little bit of a personal spot for me, so just so you all know, I spent 10 years as a mobile engineer, that's all I did. I did Android and I did iOS. And at the time, that was really rare, to do both.

00:04:10.560Bryon Mackay: I do have my preferences for one versus the other, but I did a lot of work in each, and I saw the mobile revolution happen as a developer, and so when the App Store came out, like.

00:04:23.490Bryon Mackay: we were all just going crazy trying to build these things, and… but the interesting thing is, what we're seeing today with AI is very similar to what we saw with the mobile revolution. And what I mean by that is, when apps first came out.

00:04:40.680Bryon Mackay: When apps first… when the app store opened up and everybody was building their apps, they all looked like mini web pages.

00:04:49.160Bryon Mackay: And so we all got these apps that looked like mini-web pages, because that's what we knew.

00:04:53.170Bryon Mackay: And all the apps that we had in the apps were similar to that, were like web pages. And then things started happening. I remember specific libraries coming out.

00:05:02.770Bryon Mackay: I remember when Facebook introduced the hamburger menu, and that was just, like, blew everybody's mind. We're like, what a hamburger menu? You gotta be kidding me! That's amazing! Apple had their human interface guidelines that really steered the way that, that… and still influences how mobile apps are today.

00:05:19.330Bryon Mackay: But over those years, then we started to identify there are applications that we can do that we've never done before because of a smartphone.

00:05:28.910Bryon Mackay: And so, for example, there was no web app to call a random stranger to come pick you up.

00:05:35.710Bryon Mackay: That didn't exist.

00:05:37.230Bryon Mackay: And it couldn't exist, because we didn't have a good way of doing it, and now we do, with phones.

00:05:42.750Bryon Mackay: We are at… the early mobile stage of AI right now.

00:05:49.520Bryon Mackay: There are a lot of things we are still trying to figure out, every single day, about how to use this, how to build with this, and I… it's my opinion that we are not thinking big enough.

00:06:00.700Bryon Mackay: That there is… at some point, someone's gonna hit it, and it's gonna nail a concept, and suddenly it's gonna just absolutely explode all over again. Or we're gonna realize the great power and capabilities that we have with these applications. And the application will probably change.

00:06:16.290Bryon Mackay: The way that we see them, the way that we use them, and right now, we're just using the paradigms that we know. We know websites, we know chatbots, right?

00:06:24.370Bryon Mackay: That's the current paradigm. But what if it wasn't chatbots? What if we figure out other ways to do it that are not… that don't look like chatbots?

00:06:31.560Bryon Mackay: I don't know the answers, I don't think anybody does quite yet, but I do feel like we're in those early stages, just like we were in mobile, where we're transitioning from what we can do right now

00:06:42.310Bryon Mackay: To what we can actually really do once we really harness this and think about it differently.

00:06:47.350Bryon Mackay: Part of this conversation today is going to be about that, about some of the lessons we're learning, and the hard lessons we're learning, so let's get into it.

00:06:54.620Bryon Mackay: Alright, who here is vibe-coded? I don't know if y'all have access to emojis, but I'm gonna assume all of you are raising your hand. We've all vibe-coded, right? It's a fun time. Wow, it makes programming so fun and so fast. You can just, with literally anything you can dream, you can see it.

00:07:10.540Bryon Mackay: And it's amazing.

00:07:12.090Bryon Mackay: So, there's… there's problems with that, right? Right? I… I have Vibecoded an app with my son, and I'm afraid to look at the code, right? I… I don't even want to know what's under… what's underneath that blanket. So we'll…

00:07:24.950Bryon Mackay: But anyway, so how do we fix this? So we can't do this in our jobs. We're engineers. We've got to make sure things are quality, that things are good.

00:07:32.680Bryon Mackay: Here's some, interesting stats that I think will, highlight just how prevalent these issues are that we're seeing. Ai-generated PRs contain 1.7 times more issues overall than if a human did it.

00:07:48.190Bryon Mackay: So, we're getting more issues. Are they sneaking through? You bet they are.

00:07:52.230Bryon Mackay: So, this isn't a reason to stop AI, this is a reason for us to stop and consider what we're doing with AI.

00:07:59.960Bryon Mackay: You know, I thought about this the other… last night as I was preparing for this. If somebody came up to you 10 years ago and said, here is somebody who has 1.7… who makes errors 1.7 more times than any of your engineers.

00:08:12.730Bryon Mackay: Would you hire this person? You'd be like, no. But then you'd say, what if they could code 100 times faster? You'd be like, well, maybe, you know? So it's this funny, like, what kind of world we're living in.

00:08:23.070Bryon Mackay: But yeah, a lot more issues. Here's some more issues.

00:08:25.480Bryon Mackay: Performance regressions are 7x more likely. Readability is 3X, has 3X more issues. Security vulnerabilities, that's a big one. 2.74.

00:08:36.710Bryon Mackay: I mean, we're talking… these are some serious numbers here of more issues that could be coming out, and we don't even know, what, you know, what's going on. Like, we don't even know why, initially, because we probably didn't read the, you know, I'm not saying all of you, but I'm imagining a lot of us probably didn't read the code thoroughly.

00:08:55.350Bryon Mackay: Error handling gaps, two times more common. Logic and correctness, 75% more common. Okay, this is the kind of stuff that's getting out there.

00:09:06.970Bryon Mackay: Okay? This is the kind of issues that we're seeing. AI and LLMs are not doing a better quality job. They're just doing it a whole lot faster. And that's what we're real… that's what we've been focused on for so long, right? 10x engineer!

00:09:22.510Bryon Mackay: Right? We hear that all the time.

00:09:24.810Bryon Mackay: What does that really mean? It's just, they're just coding faster, but they're actually coding worse.

00:09:30.370Bryon Mackay: So…

00:09:31.610Bryon Mackay: This is where we come in and go, okay, let's figure out what's going on. What can we do about this?

00:09:38.120Bryon Mackay: All right. The first point here is, I think, pretty, pretty evident to all of… to any of us who's tried to do this, especially at a production level.

00:09:47.010Bryon Mackay: Right? We need to make sure there's a lot of context. So…

00:09:52.200Bryon Mackay: They have no knowledge of your business rules, nowhere of architectural constraints. Sure, Cloud Code will go through your codebase and try to understand all these things. It will do its very best, and quite frankly, it does a really impressive job at that.

00:10:05.620Bryon Mackay: But it's still missing constraints. It still doesn't know about this function that's buried over here that could actually be really helpful in this situation, right? There's a lot of things like that that we as engineers for years and years and years prided ourselves on… ourselves on, on knowing, because we wrote the code by hand, and so we knew all the ins and outs.

00:10:25.260Bryon Mackay: No understanding configuration patterns. This last one, is something that I want to, highlight here, and that is that a lot of times, and I do this, I think we all probably do this to some extent.

00:10:37.330Bryon Mackay: we will keep a Claude session open, or a codec session open for so long that we are just… we just have exceeded the reasonable level of context in the context window.

00:10:51.010Bryon Mackay: So, context window currently is, what, a million for Opus, right?

00:10:54.860Bryon Mackay: Okay, a million does not mean we should use a million tokens.

00:10:58.580Bryon Mackay: Right?

00:10:59.660Bryon Mackay: I would stop at $100,000.

00:11:01.510Bryon Mackay: If we get to 100,000, stop, consolidate, do something, you know, start a new session, whatever it takes. But by that point, there are studies that are showing that it is starting to lose its ability to remember things at that point.

00:11:17.210Bryon Mackay: Okay, so that's something that we have to be, really cognizant of as well, is how much context are we using? And I'm guilty of this. I have context windows that are…

00:11:27.050Bryon Mackay: super bloated that I just want to keep going back to, because it went so well last time, but it inevitably, will cause more… well, not to say more issues, but it'll just… yeah, it will be more issues. It'll come at a more… at a more frequent clip, right?

00:11:41.530Bryon Mackay: And so, that's something we have to be, really keen on, really aware of, as we… as we work through our code.

00:11:49.240Bryon Mackay: But here's the bigger thing.

00:11:54.440Bryon Mackay: We do not have the time to review all the code that an AI produces. Now, I want to challenge you here for a second, because every company will say at some point.

00:12:06.290Bryon Mackay: We want more code. Look at these things, they go so fast. 10x engineer, yadda yadda yadda yadda, right?

00:12:13.030Bryon Mackay: What if we said no?

00:12:15.480Bryon Mackay: What if we said we're not doing that? What if we said we're actually going to go 2X faster, or something to that extent, right? Before AI, issues arrived at human speed, we could review them, right? You had your engineer sitting next to you, you'd write code, he'd write code, halfway through the week, you'd swap code and read each other's code.

00:12:33.930Bryon Mackay: You know, that was manageable, right?

00:12:36.560Bryon Mackay: Now, things are coming out so fast, oh, you can, you can, you know, essentially vibecode your way through a problem, and it's, like, you just keep pushing out features. I had…

00:12:50.010Bryon Mackay: I had a situation where I had a junior engineer sitting next to me, and I asked them to go do some task. And he came back with a 300-line PR.

00:12:59.120Bryon Mackay: And I looked at the PR, and I mean, about 20 lines in, I go, this could all be done in about 3 lines.

00:13:05.870Bryon Mackay: And so I went back to him and was like, hey.

00:13:08.400Bryon Mackay: you've… here's your solution. I'm just gonna… I'm just gonna give it to you and tell you what it is, because what you have is way too much code.

00:13:14.450Bryon Mackay: It's 3 lines, to do something that he had, essentially had an AI bill 300 for. So there's these checks, right, that we need to be aware of. I'll say this again later.

00:13:25.390Bryon Mackay: But…

00:13:26.570Bryon Mackay: the bottleneck that we should experience, we all… you know, what is the bottleneck? Well, it's reviews, right? No. The bottleneck is our ability to ensure quality.

00:13:37.410Bryon Mackay: And however that happens, happens.

00:13:39.530Bryon Mackay: But that should be the bottleneck.

00:13:42.420Bryon Mackay: Right? So, we have to be really, leery, and even push back when we're being pushed to do things to our… and that might be a hard thing for some leaders to hear, but as engineers, it needs to start with us.

00:13:57.490Bryon Mackay: So… Alright, so what do we do here?

00:14:02.780Bryon Mackay: Where do we start? So let's say we want to make sure that we talked about context, we've talked about quality, right? And assuring quality. So where do we start first?

00:14:12.970Bryon Mackay: Pseudocoding the logic first is a great way to go. Doesn't have to be pseudocode in your codebase, but it could be something where you are, where you talk it out, where you really think deeply about how is this going to work. I really find it valuable

00:14:28.040Bryon Mackay: To tell… to build out documents that… where I, like, go through the logic myself, and I go, this is how I expect it to be built. Like, that's kind of my practice, too, with a lot of PRs, like, I expect it to be built this way, let's see how they end up doing it.

00:14:42.360Bryon Mackay: But you provide that kind of guidance to the LLM, and suddenly it knows exactly where it should go, right? Us as engineers need to be aware of our codebases. We are aware of our codebases, and so there's a lot of guidance that we should be providing, as opposed to saying, hey, go make this button, and make it

00:15:00.740Bryon Mackay: You know, make this button so they can change their status.

00:15:03.740Bryon Mackay: go through the steps a little bit more. How do they… how does this button, you know, how do you want this button to send the API request? How do you want that API request to get to your service layer? How do you want that to go to the database? Should you make…

00:15:16.140Bryon Mackay: And just be a lot more, a lot more detailed in your context that you provide. Don't let the LLM decide, because heaven knows, as we've seen in those numbers, it will do a poor job.

00:15:29.070Bryon Mackay: Of course, this goes back to research,

00:15:33.100Bryon Mackay: And I'll go into this a little bit. I do have a practice that I really like doing when it comes to doing research. And it's not really research as much as it is having a team architecture conversation. So what we'll do is we will get the engineers in a room, the engineers that are working on said task.

00:15:50.000Bryon Mackay: we'll turn on Notion. Notion is my…

00:15:52.950Bryon Mackay: transcriber of choice, personally. I just… I like it. There's lots of great transcribers, but that's the one I like to use. Turn on Notion, and we'll just start talking for an hour or two.

00:16:02.650Bryon Mackay: About whatever the, whatever the, you know, task is at hand. We'll go into a lot of detail, like I was talking about. We'll go into a lot of detail of how to get

00:16:12.670Bryon Mackay: this task completed, and the changes that need to go into making it. Not saying to do this for everyone. Of course, there are small tasks that you can get by with without this much rigor, but for larger tasks, it's important for us on the team

00:16:27.770Bryon Mackay: to know all the innards of what's going on, right? I don't want to be an engineer on the island. I want to make sure my team's aware, and we have these conversations, so that we can build this context up.

00:16:41.880Bryon Mackay: When we're done with our conversation, and it's all transcribed in Notion, we go through the transcription… not the transcription, it usually… we then take that and say to Claude, hey, go take this transcription and build out, a spec list.

00:16:56.080Bryon Mackay: And filled out some tasks. And then we go through those meticulously.

00:17:00.590Bryon Mackay: And make sure that those align with what we're doing, and then we can go to the AI. Once we have those edited, fleshed out, and we're feeling really good about where we are, then we go to, you know, COLD Code, or Codex, or whatever you're using, and implement it. But it still just takes a lot of thought, and a lot of engineering know-how.

00:17:18.440Bryon Mackay: To get there. So, when I say pseudocode logic first, there's also a lot of planning involved. Of course, there's research involved if there's things you don't know. But there's this collaboration between engineers, too, that goes a long, long way.

00:17:33.400Bryon Mackay: Okay.

00:17:35.180Bryon Mackay: There's a couple other things you can do, and this is more on a team level.

00:17:39.690Bryon Mackay: In terms of what you can accomplish. So, context files are super helpful, especially when you have teams, when you have a repo that you have many, many people working on, and they all have different,

00:17:54.140Bryon Mackay: I'd say mastery, understanding of different areas of it, of your codebase.

00:18:00.360Bryon Mackay: So you have these CloudMD files. Codex has one as well, I think they're called Agents.md, I can't remember the exact name. But you basically can put these to set rules and constraints and patterns and understandings so that you're not relying on the LLM to go through your codebase and understand what's going on every single time.

00:18:18.570Bryon Mackay: Instead, it will rely on these files to go in and understand the rules and the constraints of a given file.

00:18:24.970Bryon Mackay: A good example of this would be, to, to say, don't, you know, you're not allowed… you're only allowed to change authentication logic in this one spot. You cannot put authentication logic anywhere else in the codebase, it has to go through this file.

00:18:42.440Bryon Mackay: Right? That will stop the LLM from going out and trying to do some things that are just not safe, or try to bypass some security measures that you've already established. So those are the kinds of things that you want to put inside those context files to give that LLM, that coding agent.

00:18:56.940Bryon Mackay: It's, it's guidance so that it does not go off the rails and start to cause those security issues or other performance issues that we were talking about earlier.

00:19:06.750Bryon Mackay: Okay?

00:19:07.860Bryon Mackay: Centralizing risk, something I just kind of mentioned in that example, but making sure there's, a single owner, mainly like the files, right? Everything's consolidated, it's in one spot, and anytime those change, that should raise a flag, right? You can set these triggers here and there, to say.

00:19:26.140Bryon Mackay: You know, when this file changes, we need to get a review from so-and-so, or we need to get a review from this guy, or that guy, or from platform, or from security.

00:19:34.070Bryon Mackay: You know, those things are all really, really good processes to have in place to make sure that the code that we're writing and the changes that we're making, even if it's a junior to the most senior, they're all getting reviewed by the same people. We all know what's going on.

00:19:47.790Bryon Mackay: Right?

00:19:49.260Bryon Mackay: Smaller PRs. Now, there's always give and take when there's a number involved here, especially something like lines of code, but…

00:19:59.290Bryon Mackay: It's more of a code smell, I'll admit, but if you're getting around the 400 line mark, or getting above it, especially significantly.

00:20:06.690Bryon Mackay: Think about it. Think about it for a second. Why is it taking so much code? More likely than not these days, the coding agent is bloating, right? They're usually going out there and they're just writing a whole lot of code to get something done, and it's a good time to stop and be like, wait a minute, what's really going on here?

00:20:22.680Bryon Mackay: Your PR reviewer will thank you too, as you can probably imagine.

00:20:28.230Bryon Mackay: The 7 file change, again, it's not really a hard number, it's a bit arbitrary.

00:20:33.790Bryon Mackay: But…

00:20:34.880Bryon Mackay: It's a… it's kind of that rule of, like a code smell. Again, if you're changing 15 files, you're going, wait, wait, why are we doing so many file changes? Now, sometimes you have to, right? Sometimes there are cases where you're actually just, you know… but you know what those are. I'm not trying to… I'm not trying to simplify… oversimplify this a bit.

00:20:53.260Bryon Mackay: But again, it's just good things to keep in mind. Like, LLMs are great at writing a whole lot of code.

00:21:00.430Bryon Mackay: And this is just one of those opportunities where you can sit there and go, when I see a PR with, like, a thousand line changes, I'm gonna stop and go, wait, what's really going on here? Why do we need all of these line changes? Let me understand this better.

00:21:13.010Bryon Mackay: Another thing that I've seen a lot in the code from our challengers, which is actually quite surprised me, is naming. There gets to be a lot of generic names that LLMs will throw into code, and that can be problematic when you try to read it.

00:21:31.210Bryon Mackay: Generic names, or using, you know, small shorthand.

00:21:37.700Bryon Mackay: There's just… there's just no room for that. And the way that we can remedy is quite simple, just tell… either… either you name it yourself.

00:21:45.800Bryon Mackay: Or just remind the LLM to be explicit and see how that goes, but I would lean into you naming it yourself when it's really critical, and when it's important, so that you can make sure that when you come back and read it, you know what's going on.

00:22:00.500Bryon Mackay: Right?

00:22:02.180Bryon Mackay: Finally, checkpoint commits, right? Commit at logical checkpoints, these are all things that are really useful.

00:22:10.640Bryon Mackay: And one tool that I… I'm not affiliated with them, but I'm just gonna throw it out there. If anyone has used Graphite.

00:22:17.680Bryon Mackay: Graphite is an excellent tool for this kind of thing.

00:22:22.390Bryon Mackay: Reason being is because what they… I'll just really quickly touch on it. It can sound a little confusing if you're not used to this kind of thing, but basically they'll make a stack of PRs.

00:22:32.110Bryon Mackay: Where every PR relies on the previous PR, right? It's built on top of itself. But they have some really great functionality around

00:22:40.120Bryon Mackay: If you have to make a change at, like, the bottom of the stack, that stack, that then gets propagated to the rest of the stack. Really, really helpful, when it comes to, when it comes to, like, building small PRs and building them in

00:22:54.210Bryon Mackay: And kind of some sort of scope, where it's like, hey, these are going to be my database changes, then it's gonna be my service layer, then it's gonna be my API layer. Really, really helpful, when it comes to putting all those together, so…

00:23:07.320Bryon Mackay: If you haven't looked into it, I recommend it. It's not for everybody, and it's not for every use case, but I think graphite and the process they have is, super helpful, so…

00:23:18.410Bryon Mackay: Okay.

00:23:20.190Bryon Mackay: This… this is another thing that I've seen so much of, and I'm totally guilty of this, and I'm learning better ways now.

00:23:28.850Bryon Mackay: It is really easy for LLMs to make tests, and we all feel so good about it, because we're like, look.

00:23:35.530Bryon Mackay: I'm running tests, and this might be the first time in years that I've really written this many tests. But the fact of the matter is, is that these tests are usually low value.

00:23:46.250Bryon Mackay: My favorite one is when they're verifying that a static map is indeed that static map, and it's like, no, we don't need to test that, like, this is a string, we don't need to spec that this value is here, because

00:24:00.730Bryon Mackay: That's just what it is. So, there's a lot of tests that it'll write that are low value. Now, I'm not saying there's no value in them, okay? There's value in them, and we should. We should be having them write tests.

00:24:14.570Bryon Mackay: But there's an opportunity here for us to consider what other tests we might want this, to verify that this is actually working.

00:24:24.460Bryon Mackay: And I'll give you a couple examples of categories. We've got edge cases, right? There's a lot of edge cases we can test. Security is another one of those areas where we can make sure that, yes, a test for this security thing is actually there. One thing that you have to be a little more mindful of sometimes with coding agents.

00:24:43.690Bryon Mackay: is verifying parameters, right? So, sometimes I've seen coding agents where they'll automatically do it, other times they do not, and they really should.

00:24:54.660Bryon Mackay: And then those are things, of course, that you would test. So, there's… I would almost… I would… I like to think that thinking through the tests are just as important, and this is a little… this has been said for many, many, many years, especially if you're in a TDD environment.

00:25:10.490Bryon Mackay: That we should also be thinking about the test just as much as we're thinking about the contact… the plan, the planning phase of… that we eventually hand to the AA agent.

00:25:20.830Bryon Mackay: It ensures not only that the, right

00:25:25.820Bryon Mackay: That the right functionality is there, but it also ensures that there's no regression later.

00:25:33.870Bryon Mackay: You may be doing this already, but something that I've almost forgotten to do, because I just have it… I have it drilled into our skill, is after every change, that it goes through and runs all the tests.

00:25:44.540Bryon Mackay: That it runs through them and checks for regression. And it's amazing, and kind of scary.

00:25:50.650Bryon Mackay: how often regression tests are failing because of something that the AI just wrote.

00:25:56.590Bryon Mackay: So, again, really important to have that set up, to have your tests set up. It's really important to make sure that your tests are covering the proper things, and the things that, are critical to your business.

00:26:09.700Bryon Mackay: Also important that you give the LLM, or I keep switching off between the two, but you know what I mean, the coding agent, the understanding and ability to go and run those tests every time.

00:26:22.660Bryon Mackay: So, I would just put a skill. Personally, what I do is I just… well, I put it in the… I think we have it in a skill, and we also have a CloudMD file, where we just say, after every change, when I say it's all done, like, always run the test before, you know, before committing.

00:26:39.170Bryon Mackay: And make sure that that's there. And of course, we have those things already also built in as pre-commit hooks.

00:26:45.770Bryon Mackay: But easier to have the LLM run it, and then go back and change it, and work through the problems themselves, and having to go back and forth.

00:26:52.840Bryon Mackay: That also is true for UI changes with, things like Playwright.

00:26:57.610Bryon Mackay: If you want to include that as well, it just is one less step that you have to do. Of course, now they all have their own… they're all coming out with their own,

00:27:05.450Bryon Mackay: Browser use so that you can go in there and… and simply say, hey, go spin up the app, go make sure it actually runs.

00:27:12.310Bryon Mackay: I would have loved to have had that, you know, a year ago, when I was doing some heavy UI changes to have it go figure that out, so I wouldn't have to be the one to do it, but… So anyway, a lot of great tools are coming out, and more still to come. Like I said, I still think we're in the early, early stages, and we haven't quite figured out where all this is going.

00:27:30.110Bryon Mackay: But when we do, it's gonna be… it's gonna be a heyday.

00:27:33.260Bryon Mackay: So,

00:27:36.480Bryon Mackay: Okay, I think we know this, we've been drilled with this in our heads as engineers for many, many years, so won't say any more on that.

00:27:44.070Bryon Mackay: Next, so we have, guard rails, are really…

00:27:51.210Bryon Mackay: it's… guardrails is such a funny term, I feel like, because it's… sometimes we say it to say, like, oh, you have to make sure your app has guardrails for all these things. I don't know, it just…

00:28:02.100Bryon Mackay: It's a very vague term in terms of what actually it is, but we're gonna… so we're gonna talk about what kind of guardrails you can put in place,

00:28:10.330Bryon Mackay: And we can… and then you can take that back to your work and see what, how you want to implement it.

00:28:18.260Bryon Mackay: But the first… is skills.

00:28:21.560Bryon Mackay: So, skills is an excellent, excellent way of defining a process that you want, the LLMs to do.

00:28:31.990Bryon Mackay: And they're super useful. So, for example, I, as part of my job, I pull down a lot of code, that I have no idea what's inside, right? I am pulling down code all the time and running it on my machine.

00:28:46.770Bryon Mackay: And so one thing I do is I have a set of skills that goes through the code and just makes sure that there's nothing nefarious in there, nothing that's gonna hose my machine.

00:28:55.600Bryon Mackay: And it's just… it's just part of the regular workflow. I've forgotten it's there, quite frankly. The only time I remember it's there is when I go, wow, this is taking a little bit longer than I thought, and then I look and it's, like, still looking through security concerns.

00:29:10.730Bryon Mackay: So… but it's great, because I never have to think about it again, and it just takes one more thing off my plate. So, if you have any of those, and it could be, let's see, I'll give you a couple examples here.

00:29:22.770Bryon Mackay: we had one that would check our Click House instance to make sure that things were… and then if there was an issue there, it would go and, update, or try to scale up our nodes, or scale down our nodes.

00:29:36.520Bryon Mackay: We had other ones that would do document generators, that was a really, really useful one, so whenever anything changed, we had a skill for that. But there's just all sorts of things where it's like, we always forget, right? So write a skill that just does it for you, and you don't have to remember as much, which is nice.

00:29:51.820Bryon Mackay: Okay.

00:29:53.820Bryon Mackay: So… Linters.

00:29:56.780Bryon Mackay: I still am a big advocate of lynching. It's static, yes. It's not AI, yes. But it's still really great, and it's a great thing for the LLMs, to be aware of. So, again, as part of your either pre-commit, or as part of your skills, or your CloudMD files, it's another great thing to put in there. It's like, hey, every now and then run this.

00:30:16.530Bryon Mackay: Make sure that, you know, when you change, when you're done changing, you know, 400 lines of code at a time.

00:30:22.350Bryon Mackay: go and run the linter and see where we're at. It'll catch a plethora of issues and mistakes well before, it hits production, and it just gives, again, the LLM a chance to self-correct.

00:30:35.360Bryon Mackay: So I'm still a huge, huge fan of lenders. I know there are… there's a lot of opinions out there about what lenders should do, but regardless, I think there's still a lot of value there.

00:30:46.080Bryon Mackay: Okay.

00:30:50.520Bryon Mackay: So, you can choose, as well, what models you're using for different tasks. Now, it's very easy for me to say, hey, I'm gonna use Opus, because Opus is the best, and it is.

00:31:02.590Bryon Mackay: Try to convince me otherwise.

00:31:04.860Bryon Mackay: But, sometimes we don't need it, right? However, when it comes to the critical gates that we have in our system.

00:31:14.440Bryon Mackay: That is where we want to use the best model possible.

00:31:17.590Bryon Mackay: Right? Before it goes out to production, when we have a coding agent go through and review our code, you better believe that we want the best thing possible, because we want to catch everything we can.

00:31:28.770Bryon Mackay: So…

00:31:30.110Bryon Mackay: Be mindful, right? We all have tokens, there's a cost, it's super easy for an LLM to become the most expensive thing in the engineering, budget.

00:31:41.770Bryon Mackay: But, so be mindful of that, but also when it comes to critical things, like, use the smartest model you can get your hands on for that verification.

00:31:51.830Bryon Mackay: Or, if you're using a lesser model, just make sure you give it enough context and enough guidance so that it can do a good job as well.

00:31:58.960Bryon Mackay: But I wouldn't leave that up to a weak one, and I think that's… I think that's pretty…

00:32:02.490Bryon Mackay: Pretty obvious, but worth saying.

00:32:06.390Bryon Mackay: Okay.

00:32:07.990Bryon Mackay: Finally, separate the loops. Keep the verification process distinct from the coding process. So,

00:32:17.730Bryon Mackay: this is… this is where you want to make sure that, you know, you still have other eyes on your code, right? You have other processes that are going through and verifying that, yes, what you have written is going to work in production. That can be QAing, that can be,

00:32:35.260Bryon Mackay: That can be, you know, other agents, that can be different… there's a whole lot of ways that we can go around this.

00:32:40.810Bryon Mackay: I would also want to say that verification does not happen always after the code's been pushed.

00:32:47.820Bryon Mackay: So, to give you an example.

00:32:50.990Bryon Mackay: We had, we had a case where we had to move data from a Postgres database to an OLAP database, because it was all analytical data that we wanted to aggregate. There's a lot of numbers that we wanted to crunch.

00:33:04.000Bryon Mackay: And, the verification process actually started in the planning phase.

00:33:08.760Bryon Mackay: So my co-worker, dutifully, did his research, went through the whole thing, and wrote this really great planning doc.

00:33:16.450Bryon Mackay: And it was my opportunity to then verify it, the plan. And so I went through it myself, I read it, I made some suggestions, most of it looked… it pretty much all looked great, there were just some additions I would have included, and then he went and built it, and it was… it was great. It was very, very smooth.

00:33:34.480Bryon Mackay: But again, we didn't rush it, right? He wrote it, and then I verified it, and then he wrote the code, and then I verified the code, and we had coding agents reviewing the code.

00:33:42.580Bryon Mackay: And it was a very, very smooth process. But you want to make sure that, of course, that you have

00:33:48.870Bryon Mackay: you know, multiple people look at that. If you've ever studied the, or looked into,

00:33:54.290Bryon Mackay: Oh, now I'm gonna forget the name. I wanna say Systems of Control, but I don't think that's quite the right term.

00:33:59.310Bryon Mackay: Where, basically, you can't… you don't give one person all the keys to handle finances, right? You have multiple people that have different keys, that have different gates that they're in charge of. Still want that, still need that. Does it slow things down? Sure. Sure, it does.

00:34:13.889Bryon Mackay: But we shouldn't let that speed be the determining factor on what we do.

00:34:20.110Bryon Mackay: Which leads me right to this thought.

00:34:23.540Bryon Mackay: If any of you know DEX, from Human Lair.

00:34:28.500Bryon Mackay: Segregation of duties. Jason. Yes, I think that's… I won't… I don't know if that's the exact right term, but that's really close to it.

00:34:36.429Bryon Mackay: It might even be the right one. It's been a while for me since I've had to use that term. Anyway…

00:34:42.889Bryon Mackay: This is from Dex, he's at Human Layer, you may know him, he's pretty… he's getting pretty big on… on X and other places. But he said this the other day, and I just… I think this is…

00:34:52.980Bryon Mackay: An excellent frame of mind to have as engineers.

00:34:56.870Bryon Mackay: He said the bottleneck is your ability to ensure quality.

00:35:01.970Bryon Mackay: I think that's so true.

00:35:03.870Bryon Mackay: I think that is absolutely what should be the bottleneck. I know we all want to push things fast, but for heaven's sakes, slow down.

00:35:13.190Bryon Mackay: And we should be pushing that up the leadership as well. That, look.

00:35:18.120Bryon Mackay: We're engineers, we're gonna make sure things work, we're gonna get more pushed out than ever before.

00:35:23.080Bryon Mackay: It may not just be the 100X that some people say.

00:35:26.710Bryon Mackay: It's not gonna be us open-clawing our way through, it's just not gonna work that way. And… if Anthropic's, little PR

00:35:37.840Bryon Mackay: Snafu that they had this last week.

00:35:41.500Bryon Mackay: Where we all got a peek a little bit into what Cloud Code actually looks like.

00:35:47.720Bryon Mackay: As well as that product works, there's some raised eyebrows. So, let's not be like that.

00:35:55.040Bryon Mackay: And let's make sure that our quality is good. It's really… it's almost becoming a bit of a moat, in a sense, these days. I don't know how long that will be.

00:36:04.730Bryon Mackay: But to have a quality application that's secure.

00:36:07.600Bryon Mackay: It's always been a mode, I suppose, but even more so today, with all the things that are going on, those metrics that we saw earlier kind of allude to that.

00:36:16.070Bryon Mackay: That we need to make sure that the quality of our code is still the most important thing that we can do with our jobs.

00:36:23.400Bryon Mackay: I do want to point out one other thing here, with this.

00:36:28.560Bryon Mackay: I saw an article the other day.

00:36:31.960Bryon Mackay: Excuse me. That I thought was actually… the headline was kind of funny, but also, said something about our jobs as engineers.

00:36:39.830Bryon Mackay: It said, CEO fires all lawyers and relies on ChatGPT for his lawsuit. Loses $250 million case.

00:36:52.140Bryon Mackay: And I just thought.

00:36:53.800Bryon Mackay: We as engineers are like those lawyers, and sometimes people feel like they are going to vibe their way out of a problem.

00:37:01.450Bryon Mackay: Because, look at me, I can build a website. And it's like, cool. Do you know what's going on in that website? Do you know what's going on under the hood? And there's a lot of opportunities for us as engineers to step in and be like, look.

00:37:15.420Bryon Mackay: We know what we need to do, we know what we need to put into this. We are going to ensure that quality does not lack.

00:37:25.240Bryon Mackay: We're going to ensure that your app is secure. But to do that, we need to slow down. We'll still build more, but probably at a 2X pace. And quite frankly, most companies need to be okay with that, in my opinion.

00:37:37.780Bryon Mackay: Yes, we can build feature after feature after feature, but as consumers, can we even consume that many features?

00:37:43.740Bryon Mackay: Not really.

00:37:45.390Bryon Mackay: Not really. We're creatures of habit, and if we're breaking the habit too much, what's going on?

00:37:52.240Bryon Mackay: But there's something to be said about quality. I feel like right now we're in a bit of a state where quality is a moat. If you can ensure quality.

00:38:00.170Bryon Mackay: You have a moat. You're gonna be just fine. You're protected, at least for the foreseeable future.

00:38:05.670Bryon Mackay: Call me back up when we have Opus 5.7, or whatever it is that comes out, and we'll see what happens then. But I really don't think it's gonna change too much.

00:38:14.910Bryon Mackay: Alright, this is a little bit for non-engineers. I don't know how many people in here are non-engineers.

00:38:20.450Bryon Mackay: But I just want to point this out.

00:38:24.000Bryon Mackay: that, as a non-engineer who's trying to build, their exposure is higher. They do not understand the ins and outs, as engineers do.

00:38:32.570Bryon Mackay: But engineers… so that when they build something, they're not aware of all the things. As one person likes to… as one phrase I like is the unconsciously incompetent.

00:38:44.590Bryon Mackay: Right?

00:38:45.650Bryon Mackay: Where they just don't know what they don't know.

00:38:48.350Bryon Mackay: So, it's very important.

00:38:50.910Bryon Mackay: to have an… at the very least, what I would recommend is sit down with an engineer and have them help you set up your workspace. They can put together a certain structure using… you can leverage CloudMD files, you could set rules and guardrails in there.

00:39:07.610Bryon Mackay: You can do… you can set up skills, and those kinds of things will help a non-engineer go a lot further and a lot safer. So, for example, if you have a PM that wants to do some changes, or you have a designer that wants to do some changes.

00:39:19.010Bryon Mackay: say, hey, let me set up your workspace, but let me only give you access to these files. Like, here are areas where… here are… we're gonna blacklist a whole bunch of this application, or whitelist some of it.

00:39:28.520Bryon Mackay: And so that they have the ability, right? They could very well do a lot of meaningful work.

00:39:34.040Bryon Mackay: But, we don't want to give them the keys to the kingdom either.

00:39:37.590Bryon Mackay: Which is what seems to be happening at some places.

00:39:42.070Bryon Mackay: So, anyway

00:39:43.450Bryon Mackay: Finally, and to that point, the next point here, never deploy anything touching auth payments or user data without a review, right? Think about blast radius, and just what awful things could happen, when people do… when people put those things together and, unknowingly, right? And then they release it, because it gets through some review process.

00:40:04.300Bryon Mackay: And suddenly we have… Bigger issues on our plate.

00:40:10.890Bryon Mackay: I want to just come back to this slide and leave this up here as I say one more… as I kind of talk a little bit about what Gauntlet's trying to do here. So, here at Gauntlet, like, this is… this is our… this is my job, this is our job, to help other engineers

00:40:25.710Bryon Mackay: know how to set up codebases, know how to be the AI leader that a lot of these companies need, right? We need people with good frames of mind where they think about these problems in terms of.

00:40:37.340Bryon Mackay: what's the worst that can happen, and how do I prevent that? There's also people… we also need to have good system designers, right? We have to understand the systems that we're in, that we're working in.

00:40:46.080Bryon Mackay: The, we have to understand the libraries we're leveraging.

00:40:49.650Bryon Mackay: And how those make for a better, you know, how we can leverage those to make a better application.

00:40:57.270Bryon Mackay: and avoid overloading our, you know, all of the dependencies that we may have. We understand as engineers these, these qual… the qualities of a good app.

00:41:08.570Bryon Mackay: it's getting harder and harder to do that when it's so easy to vibe. So what do you do?

00:41:15.320Bryon Mackay: Did we just stop vibing? Did we stop using AI? No, of course not. We're gonna keep using this. But we have to use it strategically, we have to use it well, and we're fearing, I'll just give you a little insight.

00:41:25.960Bryon Mackay: When we talk to the companies that hire our graduates, the things that I'm hearing from them, resonate there, where there are still some that want more features, and that's just kind of what… how they come off. I don't know if that's entirely what's in their mind.

00:41:42.450Bryon Mackay: But they come off as saying, like, yeah, I want these engineers to work faster and build more and more and more, and that's great, right? That's okay.

00:41:50.950Bryon Mackay: But, it also comes with a caveat. We're teaching them as much as we're teaching those that come to Gauntlet.

00:41:57.620Bryon Mackay: That there's a lot more nuance there that you are not… that they're not aware of, and we have… we'll supply… we'll teach our engineers how to identify that nuance, how to be a good communicator of that nuance, so that it can be addressed appropriately.

00:42:13.100Bryon Mackay: So, a lot of what we teach, and with every cohort, this changes, right? Every cohort gets a kind of a new flavor of it, because in 10 weeks, so many things can change, and so many things are becoming apparent.

00:42:28.400Bryon Mackay: That, so…

00:42:30.930Bryon Mackay: where was I going with that? The next time we do this cohort, the curriculum's gonna change a little bit. And we're gonna make sure we focus on some issues that we're seeing pop up all over the place. That's a big thing and part of my job, is to make sure that we're staying up to date.

00:42:44.010Bryon Mackay: And then, when it comes to… so, anyway, all that's… if you are interested in coming to Gauntlet and learning more about how to, be that kind of leader in a, in an…

00:42:56.850Bryon Mackay: in an environment where AI is working, it's thriving, but it's also very dangerous, and you want to be the voice of reason to help make sure that these companies actually

00:43:06.290Bryon Mackay: Produce quality output.

00:43:08.460Bryon Mackay: Come talk to us. It's a great opportunity. It's 10 weeks, fully paid for on our dime, not yours, and and you'll have the opportunity to interface with these companies and get jobs with them.

00:43:19.990Bryon Mackay: We have cohorts, starting throughout the year, and so we have another cohort starting in about three and a half weeks. So, if that's too fast for you, though, we have other cohorts starting later.

00:43:31.990Bryon Mackay: In the year. And we'd love to have you. We'd love to have you. It's such a great time. You get, there's kind of 3 things that I always point out to people that you get a lot of Gauntlet. You get, of course, the learning, and the opportunity to just work on your craft.

00:43:44.930Bryon Mackay: We always talk about 100-hour work weeks, that's not an exaggeration. There's a lot of work

00:43:49.340Bryon Mackay: Excuse me, that we're due every week.

00:43:51.420Bryon Mackay: And it's just a great time to hone your craft and really dig into this stuff without any other distractions. Two, you get to interface with a lot of great companies, so you get to see… you get to meet great people, you get to talk to them about what they're building, and decide if that's a good fit for you, and if that's something that you want to go do, in the future. And of course, all these companies are offering very high salaries, and so it's a great chance to kind of bring them all together, all these companies together, all these great

00:44:16.010Bryon Mackay: Engineers together, and, to make some, some great, some great jobs happen.

00:44:22.720Bryon Mackay: And then the third thing, which is a little bit in line with number two, is networking. You not only get to network, though, with the companies.

00:44:30.290Bryon Mackay: But you get to network with a great group of people that come in with you into the cohort. You're all like-minded, you're all engineers that want to learn more, you want to push the envelope, but you want to do it in a way that is… that produces value and quality.

00:44:42.870Bryon Mackay: And that's what you get when you come to Gauntlet. A great time to sharpen that saw. We'd love to have a lot of people out in Austin.

00:44:50.910Bryon Mackay: Where you get to be there for 7 weeks, and you just get to really work and grind. It's a fun, fun time, and a great way to get to know people, and also, and also just land that great job.

00:45:02.070Bryon Mackay: So… Alright.

00:45:04.880Bryon Mackay: I've been talking a lot, I'm gonna stop there. Haven't seen any hands pop up just yet, but if you have any questions, whether about the topic we've talked about, or, if you want to talk about, if you want to talk about Gauntlet, I'm here to… to answer. So, I'm going to…

00:45:23.760Bryon Mackay: Check real quick here… Let's see… Let's see here…

00:45:32.680Bryon Mackay: I had… let's see, guys, I had a document pulled up that I think I might need…

00:45:39.180Bryon Mackay: Oops, not that one. Sorry.

00:45:41.100Tom Babb: Also, we have someone with… that raised their hand, Christian, I'm gonna allow it to talk.

00:45:45.170Bryon Mackay: Oh, okay. Oh, I didn't see it. I'm sorry there, Christian. Go ahead.

00:45:47.890Tom Babb: I've just done it.

00:45:49.810Christian Jarjouhi: Hello.

00:45:51.270Christian Jarjouhi: Can you hear me?

00:45:52.240Bryon Mackay: Yep, I can hear you. Go ahead.

00:45:53.660Christian Jarjouhi: Yeah, thank you so much for the presentation. It resonates well. I've been doing vibe coding for the past few months.

00:46:01.760Christian Jarjouhi: And I realized the amplification of the errors. And, your point resonates well. I'm trying… for… my question is related to gauntlets. I tried to apply, and it looks that…

00:46:16.280Christian Jarjouhi: There's, somewhere,

00:46:21.370Christian Jarjouhi: that the review will… there will be a review, and there will be… somehow, a committee will be reaching out afterwards. I don't know if that, like, if there's any timeline on that, there wasn't.

00:46:34.110Christian Jarjouhi: And second question is, in terms of the teacher or instructors, like, what are the typical profiles, for the, for the instructors in, overall, and is there…

00:46:45.630Christian Jarjouhi: some kind of teacher assistance over there, or, just wanted to shed some light on that, on the content process. Thank you.

00:46:54.510Bryon Mackay: Yeah.

00:46:55.280Bryon Mackay: Yeah, so, first question about, having people, there's somebody… waiting for somebody from Gotland to reach out. I will… I can… we can reach out to our guy. There's a guy here named Drew. He's usually… he's the one in charge of that, and, he should be reaching out to… to see if…

00:47:13.400Bryon Mackay: Let's see, have you taken the CCAT already, Christian?

00:47:19.120Christian Jarjouhi: I'm sorry, yeah, yeah, I did that. Okay.

00:47:22.760Christian Jarjouhi: I believe I was in the 80s something? I didn't reach the 90s.

00:47:29.060Bryon Mackay: Oh, like the 90th percentile?

00:47:30.850Christian Jarjouhi: Yeah.

00:47:31.470Bryon Mackay: Okay, cool. Yeah, so,

00:47:34.660Bryon Mackay: I know there's a lot of people applying, so let me… I can… I can take your name and send it over to Drew, but yeah, you should be, you'll hear something from him, whether… either way, to… to see, what's going on there. So, if you haven't heard from him yet,

00:47:50.240Bryon Mackay: Hang tight.

00:47:51.300Bryon Mackay: But on the other side of it, with the instructors, yeah, all the instructors we have are all engineers. They're either actively working at a company, or they're working specifically at Gauntlet. They all have many, many years of experience. I'm trying to think of… I don't even know who has the least, per se, but…

00:48:09.860Bryon Mackay: 10, 15, 20 years is not… is typically what the people have that come and instruct here. So, so yeah, lots of… lots of really good engineers, a lot of the… I respect all of them.

00:48:22.100Bryon Mackay: And they are all building great stuff on, when they're not teaching. So, they're very much well-versed in what's going on, and how to leverage the tools to get their jobs done.

00:48:34.910Christian Jarjouhi: Thank you so much.

00:48:36.360Bryon Mackay: You bet, thank you.

00:48:40.930Bryon Mackay: I'm gonna read a few from, the comment, or the, yeah, the comment section here.

00:48:47.690Bryon Mackay: Let's see…

00:48:50.140Bryon Mackay: Looks like Austin is looking for anybody who wants to meet up at the next AI Camp event at Capital Factory, very cool, that'll be fun.

00:48:59.500Bryon Mackay: Igor asked, when do you guys stop taking applications for this cohort? Do you guys do stuff like fine-tuning or RAG? Yes and yes. I will say that RAG is one of those that is ever-revolving.

00:49:14.180Bryon Mackay: And it's used in many, many companies. Fine-tuning is, in my opinion, a really great one to know and understand.

00:49:23.490Bryon Mackay: But if there's only specific companies that will actually use that skill. More often than not, there's a lot more using of foundation models, and,

00:49:34.920Bryon Mackay: And just… and using, you know, kind of the top, top-tier things that are… the top-tier LLMs that are out there to… to get work done, so…

00:49:43.320Bryon Mackay: And as far as take… when we stop taking applications, I'm actually not sure, but I know that we accept people almost up to the day it starts.

00:49:51.500Bryon Mackay: So, I would… if you wanted to do this cohort, we have, you know, three and a half weeks, I would take… I would say take the time.

00:49:59.920Bryon Mackay: you know, think about it, think it through, you have time to get through it. Don't wait too long, though, because there are some things you gotta do. You gotta take a CCAT, and you have to go through a review process, as we've been talking about. So, just be aware that, there's a little bit of…

00:50:14.670Bryon Mackay: Just…

00:50:15.570Bryon Mackay: Housekeeping that we have to do before we accept you, but we do accept people right up to the date.

00:50:27.240Bryon Mackay: Let's see, I'm just looking…

00:50:34.590Bryon Mackay: Okay…

00:50:37.120Bryon Mackay: I have something else here… oh, you know what? There's a Q&A section that some of you have been putting stuff in, especially you, Jay Fuller, alright. Am I running daily security audits?

00:50:49.030Bryon Mackay: I don't know if they're daily, or for my stuff, personally.

00:50:53.490Bryon Mackay: moron, but there are certain things that I… there are certain actions I take where I do automatically have security audits happening.

00:51:01.470Bryon Mackay: Oh, then you raised your hand. So, there, there's your answer. Apologies. Anyway.

00:51:09.970Bryon Mackay: Okay.

00:51:14.420Bryon Mackay: Do you also encourage entrepreneurship tracks? Christian, that's a great question. And the answer is, yes, we do. We definitely push more, though, for getting the jobs, but there are definitely times where, if you have this great idea and you've vetted it out, come talk to us. You know, I…

00:51:34.110Bryon Mackay: Don't know if you know our CEO, Austin Allred, but he loves that kind of stuff.

00:51:39.000Bryon Mackay: And if there's an opportunity there, you… this is a great chance to rub shoulders with him. So, yeah, that's an opportunity as well.

00:51:51.830Bryon Mackay: Okay, Jeremy's asking, what kind of person is landing these 500K jobs, staff-level jobs? Creators of startups with 100K app downloads?

00:52:02.470Bryon Mackay: Not the creators, necessarily.

00:52:07.270Bryon Mackay: there are some people that we've had that had many years at Google, who are very well established already in their communities, in terms of maybe it's an ML or a data science kind of track, or,

00:52:21.860Bryon Mackay: What also happens a lot of times is we get some really incredible people that come in, and there are a lot of companies that want them, and then there becomes a bidding war for that person.

00:52:30.990Bryon Mackay: So, it… it's not everybody, right? But there's surprisingly more than I thought when I… when I started here, of how many people get up that high. It's… it's pretty… it's… it's impressive.

00:52:44.040Bryon Mackay: A lot of these companies we have are willing to pay top dollar for great, great people, so…

00:52:50.140Bryon Mackay: Yeah.

00:52:52.410Bryon Mackay: Let's see…

00:52:54.230Bryon Mackay: Idrisu asked, do you have to meet any eligibilities to apply, like level of experience? Can you apply while still in school?

00:53:04.410Bryon Mackay: We have started to consider people who are still in school. If you're taking a CS degree, go ahead and apply, and there's, there's some more consideration that goes into that one, just to make sure that, you're at a

00:53:19.110Bryon Mackay: At a spot where you can…

00:53:21.670Bryon Mackay: do the program and be successful in it. And, we have some of those happening already, which is so fun to watch.

00:53:29.130Bryon Mackay: But it is something that we kind of do on a case-by-case basis. Typically, we require 3 years of experience. That's our baseline requirement. But if you don't, still apply.

00:53:39.240Bryon Mackay: And if you're somebody that seems to have the experience, maybe you've coded a lot your whole life, or you've already had a lot of work experience, or whatever the case is, there's opportunities… well, I shouldn't say work experience, that's what we're talking about.

00:53:51.200Bryon Mackay: But, if you've had a lot of good experiences and you seem like the kind of person that, could handle the program and also land a job, we want to talk to you, too.

00:54:00.730Bryon Mackay: Jordan asks, what kind of salaries are you expecting for Gauntlet, for Gauntlet for America?

00:54:06.570Bryon Mackay: Yeah, that's a good question. I would… okay, so there's two here, right? There's salaries and there's whole comp.

00:54:12.610Bryon Mackay: So, I haven't done the calculations on the total comp, but the salaries are more typically in the $150 to $170 range, based on when… where… what level you get it at. There's a whole… I don't know if you've looked into this, but there's a whole structure of, like, different… of varying levels when you work for the U.S. government.

00:54:31.330Bryon Mackay: But the total comp is pretty high.

00:54:34.460Bryon Mackay: Because they do a couple of… for example, there's a pension that you're getting out of that. And and then there's all these other great benefits that they have included in there as well. So, there's a lot…

00:54:46.290Bryon Mackay: There's a lot to be said for Gauntlet for America that can go a long, long way.

00:54:51.360Bryon Mackay: So, the salaries may not reflect all the value themselves.

00:54:57.720Bryon Mackay: But, the perks that you get with that job?

00:55:01.010Bryon Mackay: I feel like make it a very viable and reasonable thing to… to look into. Definitely something if you're interested in that. Or if you live in the DC area already, that's where a lot of the jobs will be. That's an opportunity for you. Or if you have security clearances already, that's another thing I've been learning lately, is how important those can be. So…

00:55:19.570Bryon Mackay: Great opportunity there, great way to get your foot in the door, and just work on some pretty interesting stuff.

00:55:26.580Bryon Mackay: Ryan asked, is the application process different for Gotland for America? No. It's the same.

00:55:32.310Bryon Mackay: It's the same. You'll be, if you get a certain score on the CCAT, that's one of our main differentiators of whether you get into Gallant for America, whether you get into, the, the main, the flagship program, or if you don't make it at all. So,

00:55:51.300Bryon Mackay: So, yeah. Alper asks, does Gauntlet for America require a citizenship, or would a GC be sufficient? Citizenship is required for that one.

00:56:02.950Bryon Mackay: Yeah.

00:56:04.610Bryon Mackay: Yep.

00:56:06.940Bryon Mackay: Sure, you bet.

00:56:10.290Bryon Mackay: Great questions, everyone.

00:56:12.420Bryon Mackay: Keep them coming.

00:56:17.000Bryon Mackay: Let's see, Jordan asks, do people bring their own monitors to Austin? No need. Unless you really have a monitor you love. No, I'm kidding, don't even bring that. Not worth bringing it. Bring your computer, preferably a laptop, as opposed to a desktop if you can.

00:56:33.310Bryon Mackay: But we'll have monitors in Austin that, you get at your desk. So, you'll have plenty to work with there.

00:56:45.320Bryon Mackay: And Austin's just a really fun, unique time. Like I said, you get just… it's just time where it's just you and the rest of the cohort, and you're all just heads down, working and learning and pushing. It's… it's a… I mean, what other opportunity in your career do you get something like this, where you just…

00:57:04.680Bryon Mackay: you don't have to pay for anything. It's all covered. Your food, your housing, your laundry.

00:57:11.760Bryon Mackay: So that you can come and just sharpen the saw, like, for… for however… for 10 weeks. Well, 7 weeks in Austin, right? 10… 3 weeks remote. Then you just do that, and you get a network with great people, you get a high… you get to meet great companies. What an incredible opportunity that is.

00:57:27.260Bryon Mackay: So, it's quite the experience. I hope you all strongly consider it, and hopefully we'll see many, not all of you.

00:57:37.400Bryon Mackay: One of our cohorts soon.

00:57:55.400Bryon Mackay: Alright… Any other questions?

00:58:02.910Bryon Mackay: How large is the typical cohort? We typically aim for 100.

00:58:07.490Bryon Mackay: 100, 150? Somewhere in there?

00:58:10.800Bryon Mackay: We always try to push a little bit higher than that, but…

00:58:14.470Bryon Mackay: you know, if we can… if we can get to 200, the material I build is… and the processes I build out on the curriculum side, at the very least, are meant for 200. So we can push that number up if we have that number. But 100, 150 is where we like to start a lot of times.

00:58:32.050Bryon Mackay: How soon after graduation do you anticipate starting with a government job? asks Jordan.

00:58:37.250Bryon Mackay: That is a really good question, and one that I don't have a solid answer for, because, because this is actually the… we're running it right now for the first time.

00:58:48.570Bryon Mackay: And so, there are still some expectations that I haven't gotten clear guidance on. But…

00:58:54.640Bryon Mackay: Give me 3 or 4 weeks, and I will have a much better idea of what that looks like. But we are… let's see, we're currently in week 7, and we have got that ball rolling of getting everybody in the GFA program, or Gauntlet for America program.

00:59:09.910Bryon Mackay: into their system, so that we can get them, interviewing and getting the jobs they need. So, everything's in motion right now, and they're, being very…

00:59:20.180Bryon Mackay: very, easy to work with in terms of getting things moving and rolling. So, hopefully I'll know more concretely afterwards. Ideally, you're starting right after you graduate. Or maybe a week after, so you can get life in order if you need to move, or whatever the case is. But I'm sure they're flexible on that, too. They've got to understand that, right? So…

00:59:38.360Bryon Mackay: Idrisu asks, what's the acceptance rate?

00:59:42.740Bryon Mackay: That's a good question. I'm… I'm not sure what the exact acceptance rate is. It's pretty low, though.

00:59:52.720Bryon Mackay: The requirements we have are pretty high, and there's a lot of review we do with each person, so, it's not terribly high. I can't… I couldn't tell you the exact number, but I know it's… it's definitely sub…

01:00:07.480Bryon Mackay: 10%, I would say. Correct me if I'm way off there, Tom, but I believe that's around the number.

01:00:14.290Bryon Mackay: Jordan, you're already in cohort GFA5. Awesome. Oh.

01:00:19.930Bryon Mackay: You're gonna love it. You're gonna love it. I've been so… I've had so much fun developing this, that track.

01:00:26.990Bryon Mackay: I think it's really, really cool.

01:00:29.090Bryon Mackay: So, you'll be learning so much good stuff.

01:00:31.530Bryon Mackay: Christian, what is the ratio, then, for teachers to students? Good question. Let me… there's probably…

01:00:41.820Bryon Mackay: Oh… 10, I wanna say?

01:00:47.350Bryon Mackay: There's a lot of us that are just gonna be walking around, talking to you, so…

01:00:54.360Bryon Mackay: Yeah, and if there's… that's where we like to keep, and we like to send out teachers to talk to you, and see how you're doing, what questions you have. Lectures, of course, will be there, so…

01:01:04.710Bryon Mackay: Quite a… quite… I would say that's a pretty, pretty good ratio. And then don't forget, you have everybody else, too. You're all learning together, everybody comes in with different skills. A lot of times, we have,

01:01:15.460Bryon Mackay: within cohorts, there's, like, little meetups that they'll have, where there's, like, for different things. There's, like, a running group, a game group, there's groups that will do lightning talks, and we can help facilitate some of this, but we love it when the cohorts just get together and they're like, hey, let's do this together. It's like, oh, that's great. Because then you're really learning, more about

01:01:35.460Bryon Mackay: you know, specific topics that maybe we as teachers, like, don't have time to review or to go into. There's a lot of stuff out there, right? And so it's a great opportunity for all of you to work together and…

01:01:45.270Bryon Mackay: Build, you know, build up those relationships, and also just learn some really cool stuff.

01:01:50.670Bryon Mackay: Christian asks, and are projects mainly individual in nature, or do we have team projects? You do have a bit of mix, but a lot of it is individual work.

01:02:02.770Bryon Mackay: There are cases where you get together with groups to discuss the projects you're building, but a lot of the work has to be done individually so we can assess, your skills. At the very end, I will say there is a capstone where you will get with your team.

01:02:16.460Bryon Mackay: And, you'll get with a team and you'll build a product on your own. So, but yes, a lot of it just by nature of having to verify that everybody is capable and able to take on these jobs.

01:02:28.540Bryon Mackay: We do a lot of individual work as well, so…

01:02:33.250Bryon Mackay: Jason asks, what if I have a day job? It's remote, though.

01:02:38.580Bryon Mackay: So, Jason, this is… I'm glad you asked this question. This is, this is one of those situations where you've got to decide for yourself how much to put in. When you come to Austin, you're not going to be able to have that job. Some people are able to take some leaves, others just have to leave the job entirely.

01:02:56.740Bryon Mackay: That's typically what has to happen, is people have to leave the job because it's so rigorous and there's so much work to do. So,

01:03:04.490Bryon Mackay: Yeah, that's… it's… it's… that's not an easy decision to make, though. I fully understand and appreciate that, that decision, and… but it's… it's kind of one of those things where it's like.

01:03:15.850Bryon Mackay: it's… the opportunity is there, it's incredible. So I don't know where you are in your career, Jason, but it's something you gotta consider for yourself, but I… again, I would just emphasize what an incredible opportunity it is to… to go and learn, and just really spend a lot of time

01:03:34.170Bryon Mackay: To make yourself the best engineer possible.

01:03:39.160Bryon Mackay: Austin is saying, I'm putting in my vacation request as soon as I pass the CCAT. There you go. There you go.

01:03:47.410Bryon Mackay: Just… just saving up that time. Hopefully you have 10 weeks. I think the best company I ever worked at only allowed us to have 6, so… You can almost make that work, right? Squeeze it out? Sure, sure. Anyway.

01:04:03.250Bryon Mackay: Well, great. Well, everybody, we're, we're at time, but I want to say one more time, just thank you all for coming. Really excited to have all of you here tonight, and just get a chance to talk about

01:04:17.770Bryon Mackay: where we are in this crazy AI revolution that we're experiencing. But it's a lot of fun. It's a rollercoaster sometimes. New models come out, and suddenly it's like relearning how to ride a bike, but we love it, and there's so much opportunity out there for engineers like you.

01:04:33.690Bryon Mackay: to go and make a real difference and a real impact, especially as all this is happening. We are gonna be…

01:04:39.650Bryon Mackay: the… that wave of engineers that really understand and learn this stuff, and solve some really interesting new problems that come up. So…

01:04:47.980Bryon Mackay: Thank you all again for coming. I hope you have a… enjoy the rest of your evening, and we hope to hear from you more, and see you at Gauntlet.

Download transcript (.vtt)

FAQ

How do you QA AI-generated code? +
Replace gut-feel vibe checks with real quality gates, meaning automated tests, structured review workflows, and evaluation criteria applied before code reaches production. The goal is to catch AI mistakes systematically rather than by spot-checking.
What's wrong with vibe-checking AI code? +
Eyeballing output catches obvious errors but misses subtle logic, edge-case, and regression bugs, which is exactly where AI-generated code fails, and it doesn't scale as volume grows.
What quality gates should I put in place? +
Put in place tests that encode expected behavior, a review step against clear criteria, and CI checks that block merges when something fails.
How do evals fit into code quality? +
Evals turn "does this work?" into a repeatable, measurable check you run on every change, so quality stays verified continuously as the code keeps changing.
Can AI review AI-generated code? +
Yes, as a first-pass triage to flag issues, though it isn't ground truth, so keep a human gate for anything high-risk.
How do I test code when I'm shipping faster with AI? +
Generate tests alongside the implementation and make them part of the definition of done, because skipping them while moving fast just ships bugs faster.
Does this apply to prototypes too? +
You can stay lighter-weight for throwaway prototypes, but anything headed for users needs the gates, and the line is "will a real user touch this?"
What single habit improves AI code quality most? +
Write the check, a test or an eval, before or alongside the code, so correctness gets defined up front while you build.

What's next?

Keep building with the rest of Night School, or apply to Gauntlet — ten weeks of technical intensity with the best AI engineers we can find.