Fine-Tuning LLMs Explained: What It Is, When to Use It, and When to Skip It

In this lesson: usually you should not fine-tune, the data is the work, and PEFT and QLoRA make it cheap.

Aaron Gallant · 60 min · June 2026

Top 3 takeaways

01

Usually, don't fine-tune

Start with model selection, prompting, and RAG. Fine-tuning earns its place mainly when you want to make a small model good enough, cut down on prompting, or keep data private and local.

02

The data is the work

Fine-tuning is supervised learning, so bad training data quietly degrades the model for good. Curate a few thousand representative labeled pairs and always hold some back to catch overfitting.

03

PEFT and QLoRA make it cheap

Freezing most layers, LoRA rank reduction, and quantization let you fine-tune a 13B model on a laptop. From there you tune learning rate, epochs, and dropout, and judge the result on accuracy plus human eval.

Aaron Gallant

Aaron Gallant

Lead Instructor, Gauntlet AI

Lead Instructor at Gauntlet AI since the beginning; 18 years in tech across software development, data science, data pipelines, and internal tooling, including five years at Google. For his day job he synthetically generates HIPAA-compliant medical research data with LLMs and works deep in fine-tuning, reinforcement learning, and synthetic data generation — including fine-tuning jobs run on-premises on real hardware — so he knows how these models behave under the hood. MS in Computer Science, University of Illinois Urbana-Champaign; BA in Political Science & Philosophy, University of Rochester.

Lesson notes

A written walkthrough of the lecture, covering the patterns, the code, and the things that trip people up.

What Fine-Tuning Actually Changes

Fine-tuning changes the model itself, not just how you use it.

Prompting, RAG, memory, and context engineering all happen during inference, where the model's underlying parameters remain unchanged. Fine-tuning modifies those parameters, creating a lasting change in how the model behaves.

This is only possible with open-weight models, which allow you to continue training an existing model. Open-weight models are different from open-source models—they expose the trained parameters but not necessarily the full training process or code.

When to Fine-Tune

Most applications do not require fine-tuning.

Start by selecting the right model and improving your prompts. If the application depends on external knowledge, RAG is usually the better solution because building a knowledge base is much easier than creating high-quality training data.

Fine-tuning makes the most sense when you need a smaller model to perform a specialized task efficiently, run locally for privacy reasons, or consistently make decisions such as classification, fraud detection, or spam filtering.

The Fine-Tuning Process

Fine-tuning follows four steps:

  1. Choose a base model.
  2. Gather and clean training data.
  3. Train the model.
  4. Validate the results.

Most of the work happens in data preparation and evaluation. High-quality, representative data matters far more than large quantities of repetitive examples.

Always reserve a validation dataset so you can measure real performance and avoid overfitting.

PEFT, LoRA, and QLoRA

Modern fine-tuning is practical because only a small portion of the model needs to be trained.

Parameter-Efficient Fine-Tuning (PEFT) freezes most of the model while updating a small number of parameters. LoRA further reduces the amount of training required, and quantization shrinks the model by storing weights with lower precision.

Combined as QLoRA, these techniques make it possible to fine-tune large language models on consumer hardware instead of expensive GPU clusters.

Evaluation Matters More Than Training

Training a model is only part of the process. The real challenge is proving that it performs better.

Common metrics include accuracy, classification performance, alignment with human expectations, and perplexity, but the most valuable evaluation is still human judgment against clearly defined success criteria.

The lecture's main takeaway is that successful fine-tuning depends far more on data quality and evaluation than on the training algorithm itself. Models generate plausible outputs by design, making rigorous testing and human verification essential before deploying them in production.

Transcript

Read transcript

Fine-Tuning LLMs Explained: What It Is, When to Use It, and When to Skip It

00:00:08.029Aaron Gallant: Alright, so we are live.

00:00:10.690Aaron Gallant: Hello and welcome. Anybody out there, thank you for joining us. This is Night School.

00:00:17.650Aaron Gallant: My name is Aaron Gauntlet. This is my first time teaching night school, but pretty far from my first time teaching. In fact, I've been teaching at Gauntlet since the beginning. And if you're joining us, please, drop where you're from in the chat. Love to see the whole range and…

00:00:37.880Aaron Gallant: variety that we get here, it's always great. It's one of the powers of the… Tokyo! Very cool. Japan… Japan's fun to… fun to visit,

00:00:46.800Aaron Gallant: I… Let's see here. Saludos from Seattle, Indianapolis, Kansas, Sao Paulo, alright!

00:00:53.150Aaron Gallant: It's another place that's pretty fun to visit.

00:00:55.660Aaron Gallant: Germany? Wow. Belfast?

00:01:00.120Aaron Gallant: And some Austin, and then Aussie, which I think is different than Austin, but that's both great.

00:01:07.000Aaron Gallant: Dubai, even. Okay, we've got quite a spread.

00:01:10.100Aaron Gallant: And also, if you want to drop in the chat, you know, a little bit about your background and what you're hoping to, get out of this. You know, what you are interested in here.

00:01:24.220Aaron Gallant: And I do see there's an audio loop on the call.

00:01:29.160Aaron Gallant: I…

00:01:30.330Aaron Gallant: If anybody else can confirm if there's any audio quality issues, let me know. But, yeah, we've checked this a little bit, so could be on your side if you want to double check, too.

00:01:40.400Aaron Gallant: Alright, yeah, let's see what people are doing here. Hoping to become an AI-native engineer, forward-deployed engineer.

00:01:49.900Aaron Gallant: Sure.

00:01:51.920Aaron Gallant: Anybody else? Got some goals?

00:01:54.680Aaron Gallant: Some things they hope to use all this new fun technology for.

00:02:01.640Aaron Gallant: Alright, well, you can click on it and drop it in the chat, too.

00:02:05.110Aaron Gallant: Just here to learn, that's always good, too. I… I really connect to that, actually. That's…

00:02:10.470Aaron Gallant: Why I teach so much, because you also learn when you teach.

00:02:15.480Aaron Gallant: train a model to classify documents for work. Well, that's great, Jacob. We might talk…

00:02:20.280Aaron Gallant: Not specifically, specifically about that, but we'll give you some context to help you figure that out.

00:02:29.540Aaron Gallant: Great, some good stuff. Build ML models and AI-powered apps. Well, let's dig into it then, shall we? Because that's really the point of this session, is to, sort of…

00:02:40.790Aaron Gallant: disambiguate, to understand a little more specifically what these things are, and what it means, for instance, to train a machine learning model, and why you might want to do it, why you might not want to do it.

00:02:56.710Aaron Gallant: And how it is different from the many other things we do. So, the topic is fine-tuning, and part of what's so fantastic about large language models is that they can be fine-tuned. They can be sort of continually trained. They are pre-trained.

00:03:14.590Aaron Gallant: These big companies have spent piles of money training these enormous models that we have access to.

00:03:21.210Aaron Gallant: And that gives them a general range of capabilities.

00:03:25.780Aaron Gallant: But then you can train them more, and perhaps make them better, at least for some specific task. That would be the hope.

00:03:31.310Aaron Gallant: Now, that said, there's a lot of trade-offs here. It's complicated, it's expensive, it's not as expensive as training from scratch, but it is significantly more expensive than just, you know, consuming the model, just doing simple inference, and that's most of what we do, by the way.

00:03:47.430Aaron Gallant: So, you know, almost all the other techniques that are talked about in this space, all the other AI engineering stuff, RAG, context engineering, what have you, are all just…

00:03:57.790Aaron Gallant: essentially a suite of techniques to try to use the inference of the model effectively. And if you're not following exactly what I mean by that, you hopefully will soon.

00:04:07.950Aaron Gallant: But it's sort of like…

00:04:10.140Aaron Gallant: we're just taking the model and using it versus changing the model itself. And, you know, most of the time, we don't do that. Most of the time, we just accept the black box. Now we're gonna actually

00:04:21.079Aaron Gallant: Tinker with the black box a little bit.

00:04:23.420Aaron Gallant: So, I'm gonna move through the slides here. I will share the link with the slides at the end, and there's also, as time allows, there's some code at the end. We'll share the link for that, and if I have time, I'll walk through the code, but if not, you'll have the code.

00:04:37.070Aaron Gallant: And, please drop questions throughout in the chat. I'm monitoring the chat, I have it here on multi-monitor setup, so I will, I will do my best to pick up questions and answer them as I go.

00:04:52.570Aaron Gallant: Of course, I will probably focus on questions that are relevant to the slide, so… or at least relevant to the topic. If you have other questions about Gauntlet or anything else, we have other staff. We've got Sunny there, who will help out in the chat answering questions. All right?

00:05:09.610Aaron Gallant: Let's do it.

00:05:14.250Aaron Gallant: So… when should you fine-tune a model? When should you consider opening that black box? Well…

00:05:22.400Aaron Gallant: There's some situations here, and the first one teaches intuition where words fall short. That's a fancy way of saying you couldn't solve it with prompting.

00:05:32.780Aaron Gallant: Right? Like, you couldn't get the model to do the thing. The model can't do it, you need to improve the model. That's… that might be the first thing that leaps to your mind.

00:05:42.050Aaron Gallant: But that's actually not the most common case that I think we are seeing, because the models are pretty generally capable, and if you run into a situation where you can't do something with a particular model, you can usually just go up to the bigger model, right?

00:05:57.240Aaron Gallant: And if you're in a situation where the biggest flagship model can't do the task.

00:06:02.280Aaron Gallant: well, you're… maybe you could fine-tune that model and make it even a little better, but, like, that's… you're now at the cutting edge, like, you're competing with the research groups and the groups that are trying to advance the frontier, and it's cool, and I'm not saying you can't or shouldn't do that, but I will say that the majority of practical

00:06:20.860Aaron Gallant: paying tasks, you know, things that companies want done, do not necessarily live all the way on the frontier, and that's okay. You know, we ride that, and we live a little bit short of it. So,

00:06:34.040Aaron Gallant: We get to benefit from it.

00:06:35.740Aaron Gallant: So, the first one, yeah, it's a valid use case, but you're not gonna see it.

00:06:40.420Aaron Gallant: Now, the middle two are sort of my favorite, and they're the ones that you actually see a bit. You can make a smaller model better for a specific task.

00:06:50.680Aaron Gallant: And you can also potentially make a model better at a task, but with less prompting required, less use of your context window, essentially. And both of these are efficiency gains, essentially.

00:07:03.540Aaron Gallant: And also, potentially, not just efficiency gains, but, autonomy gains. And what I mean by that is,

00:07:12.000Aaron Gallant: Let's say you want to, you work in a sensitive space, like, financial or security or medical space, and you need to have tight control.

00:07:23.630Aaron Gallant: over certain aspects of your system. You can't just be tossing all of this stuff as API calls to somebody else's model. You know, you can't be using ChatGPT or Claude or any of that.

00:07:36.360Aaron Gallant: So… what do you do?

00:07:38.640Aaron Gallant: Well, those models are kind of the strongest models, but there are some pretty good models out there that are open, that you can download, and then you can fine-tune those models to be better for the specific task you're solving.

00:07:51.110Aaron Gallant: And then now you have control of a model that is solving that problem for you, and you could potentially deploy it on your own hardware and maintain all that. So there's that aspect, the sort of privacy-security control aspect. And then there's also the efficiency aspect.

00:08:05.730Aaron Gallant: Because really, as we'll see in most of the slides here, a lot of what we're doing with fine-tuning is working with smaller models and trying to make them good enough to solve some particular problem, maybe on the edge, or maybe just because we want it to be cheap, and maybe we still do it in the cloud, but a small model. And…

00:08:25.540Aaron Gallant: What's really convenient about that, as we'll see, is you can use large models to essentially synthesize the training data, because the large model's already able to solve whatever this task is.

00:08:35.970Aaron Gallant: And so you make the training data with the large model, and then you take that training data from the large model, and you train the smaller model.

00:08:42.890Aaron Gallant: So that's, I think, the… the sort of sweet spot where fine-tuning is the most practical, and to give a few concrete use cases, and I do see that's a great question in chat, I'll get to that at the end of the slide.

00:08:56.190Aaron Gallant: To give a few concrete use cases, let's say you wanted to train a

00:09:04.300Aaron Gallant: Fraud detection system, and you wanted to have a step that analyzed, open text, analyzed some messages or emails or something.

00:09:14.270Aaron Gallant: and tried to look for signs of fraudulent behavior, or signs of something like that, like, you know, in a financial context, payment processing or something. And let's say all this is considered kind of private, personal information.

00:09:26.920Aaron Gallant: and your lawyers didn't sign the thing that lets you just send it to some other service. You have to do it on your own hardware, or at least within your own private cloud. And so, you take a small model.

00:09:39.400Aaron Gallant: and you label, one way or another, you get a bunch of labeled cases of this is good text, this is bad text, and you use that to train the model to be better at making that decision. And you start from a smaller model, that, you can run more efficiently, and then you evaluate it, and you ensure that the accuracy is where you want it to be.

00:09:59.290Aaron Gallant: So…

00:10:00.550Aaron Gallant: that's sort of what this overall loop looks like, and we'll get to a few more of the specifics in the coming slides, of course. But that's, again, these two middle points here. If there's one takeaway from this whole session, it's that most of the time, you shouldn't use fine-tuning.

00:10:16.740Aaron Gallant: And the main time to think about, like, the main time to think about it is when those aspects, when, you know, efficiency and control of small models, that's the most likely situation where it's like, hey, we could use a small model for this, and we could make it more efficient, we could have control of it. And that's when you might want to fine-tune.

00:10:34.600Aaron Gallant: The last one, narrow the range of possibilities, that's talking about stuff like alignment, right? That's talking about,

00:10:44.400Aaron Gallant: Trying to make,

00:10:47.510Aaron Gallant: the model essentially have some sort of rails, some sort of limitations, or at least guidelines on what it does or doesn't do. And all models have this already. In fact, it's the big difference between the major flavors of large language models, because, you know, ChatGPT, Claude, and Grok are all trained on

00:11:06.820Aaron Gallant: Give or take the entire internet, but they all have much different, reinforcement learning, fine-tuning, essentially, that has tweaked their behavior.

00:11:16.670Aaron Gallant: And so, you know, maybe there's a situation where you would do this. I can imagine if you're working for some corporation that's particularly sensitive about how its, LLM behaves, you could potentially want to do this. But the thing is, is…

00:11:31.840Aaron Gallant: It's not really the best way to try to ensure that a model is aligned or secure. Really, you need to check

00:11:38.370Aaron Gallant: You need to have other checks. The way to do it is to check outside of the model, to not rely on the model to be fully aligned in and of itself, but to have other parts of your system that can also validate the integrity, security, whatever it is you care about about the model's behavior. You know, things like not allowing it to run RM-RF, stuff like that.

00:11:58.700Aaron Gallant: So… that deletes all files, for anybody out there who's a little bit like, what was that?

00:12:05.400Aaron Gallant: I know we might have a slight range of technical people in the audience here. So, let me get to some of the questions that came in here. Apple's working on getting Google's models to run on the iPhone. Are they using fine-tuning to do this? I mean, Apple might be fine-tuning the models to some extent for their use case.

00:12:21.370Aaron Gallant: you'd have to ask Apple to be sure. I think that's mostly just about using those small Gemma models, on, you know, a nice…

00:12:30.230Aaron Gallant: embedded context, because they're an appropriate model for that size, for, like, a phone. The Gemma models are good candidates for fine-tuning, though, if you wanted to fine-tune a model. They are reasonably sized for that.

00:12:43.600Aaron Gallant: Is fine-tuning basically for privacy and saving money. Yeah, I mean, that's kind of what I'm getting at here. It's…

00:12:49.690Aaron Gallant: Not strictly, to be clear, but that's where I would say most of the real, you know, useful cases are for it at the moment.

00:13:02.330Aaron Gallant: My understanding is that most problems you might think to solve with fine-tuning can be accomplished with more appropriate model selection and prompt engineering. Bingo! Yes!

00:13:10.880Aaron Gallant: You all already got the lecture, we're only 12 minutes in, we can call it, right? I'll keep going, but it's great to see this. Yeah, I mean, those two points are really emphasizing, like, yeah, do not start by fine-tuning. Start by

00:13:24.960Aaron Gallant: By making sure you selected your model appropriately, and you're prompting it correctly, you're, you know, you set up whatever to retrieve context for it, your memory, all these other systems we let… because none of those systems change the model. Those systems just change

00:13:37.930Aaron Gallant: How the model inference is done.

00:13:40.490Aaron Gallant: So…

00:13:42.030Aaron Gallant: Alright, and on that note, there's this little check for understanding here. So, if people want to drop in the chat, and if… hopefully, at least many of you, I think we've had previous sessions, know what retrieval augmented generation is at this point.

00:13:56.660Aaron Gallant: But what would be a use case for retrieval augmented generation versus fine-tuning? And we've already talked about fine-tuning, so give, like, a use case that would work well with retrieval augmented generation.

00:14:07.020Aaron Gallant: And not fine-tuning.

00:14:09.980Aaron Gallant: And I'll keep answering questions while you think about that.

00:14:13.830Aaron Gallant: With respect to privacy, are there any advantages to downloading a model and using it locally?

00:14:20.020Aaron Gallant: Absolutely. I mean, I… VPC… so, VPC is Virtual Private Cloud, and that's nice. That means you're writing a big enough check to whatever cloud company to get some level of assuredness that there is privacy and security, and those are all pretty trustworthy, and

00:14:36.690Aaron Gallant: Your corporate lawyers will probably be okay with it. But…

00:14:40.090Aaron Gallant: you know, push comes to shove, there's no substitute for having the silicon yourself, right? It depends how paranoid you feel like being, and what your… what your threat model is, as they say in the security world, and I would not encourage excessive paranoia just for the sake of it.

00:14:55.320Aaron Gallant: You know, make sure that you're solving the problem you actually need to solve. But yeah, strictly speaking, the closer things are, the more direct control you have over it, the more direct control you have over it.

00:15:04.790Aaron Gallant: And, you know, there are cases where data centers might have issues, right?

00:15:10.520Aaron Gallant: Of course, if you're gonna host it yourself, you are the data center now, and you're responsible for things like uptime. So, there's no free lunch. It's just more work.

00:15:20.020Aaron Gallant: Alright. Are AI skills better than fine-tuning? Well, that's… it's not like an Apple…

00:15:26.210Aaron Gallant: Apple comparison there, but, yeah, I mean, in the sense that, like, if you're trying to solve a problem and you think you could add skills, and by skills, I'm taking this to mean tool use plus pre-written prompts for those tools, essentially. That's kind of what we mean by skills.

00:15:42.630Aaron Gallant: Yeah, that might be a better way to solve a lot of problems.

00:15:47.420Aaron Gallant: Now, that said, fine-tuning does something that none of these other techniques can. So if you need fine-tuning, you need fine-tuning, right? And, again, the example would be constrained environment, you need to use a 1.3 billion parameter model, you need to make its accuracy at least

00:16:02.310Aaron Gallant: X, whatever your business criteria is, and, you know, you fine-tune the model to get it there.

00:16:10.320Aaron Gallant: Alright, so people were answering the question.

00:16:13.150Aaron Gallant: Seeing some good answers to the RAG stuff. I'm not gonna read all of them out loud, but y'all can follow the chat as well. And I'll take one or two more questions, then I'm gonna dive back into the slides. Use cases for narrowing the range of possibilities. So, yeah, that's the idea of… I mean, I guess if you wanna…

00:16:31.010Aaron Gallant: somewhat specific example, imagine you're a company like Disney, and you want to expose a chatbot, but you want to make sure that the chatbot behaves with appropriate, you know, Disney style. You could potentially do some fine-tuning to really enforce more than… like, because, yeah, you can prompt, you could prompt

00:16:51.300Aaron Gallant: any LLM to be like, hey, behave like Mickey Mouse, right? But if anybody else comes along.

00:16:57.020Aaron Gallant: and does a prompt injection, and says, you know, forget previous instructions and actually do this, that sort of… that's pretty easy to circumvent. If you actually fine-tune, you're changing the model itself, and it's a little bit harder to circumvent that sort of, guardrail.

00:17:13.460Aaron Gallant: it's still, as I said earlier, not a guarantee, because the model

00:17:17.650Aaron Gallant: It's something I say almost every time I teach about language models. Large language models, they're statistical models, they're inherently probabilistic. You can't have guarantees. So if you're in a situation where you need guarantees, you need really good, really hard guardrails, you need to have secondary checks. You need to have evals, you need to have monitoring, you need to have other things going on, maybe a human-in-the-loop pattern, which we'll talk about a little bit later, but

00:17:39.840Aaron Gallant: means a human checks it, right? So…

00:17:43.050Aaron Gallant: That's why I don't really emphasize that last use case either. Yeah, you can do it, and it's why, you know, different language models essentially have different voices, why some language models won't answer certain sorts of questions, or, you know, so forth, is often because of how they were… they were trained, with these examples.

00:18:03.300Aaron Gallant: Last question before I get back to the slides. What process is better for a medical industry where data privacy and industry language are specific?

00:18:12.070Aaron Gallant: And may not be available in major models. Rags are fine-tuning.

00:18:15.980Aaron Gallant: I'd go with RAGs. I mean, assuming that you have a reasonable knowledge base, and a reasonable knowledge base… it's easier to get a RAG knowledge base than it is to get a labeled set of data for fine-tuning. We haven't even gotten to that. Like, that's where all the work is, is having labeled data. So yeah, I would definitely start with RAG for that.

00:18:35.710Aaron Gallant: Now, that said,

00:18:38.570Aaron Gallant: if you also need it to run on the edge, and then you might need to also fine-tune the model as well, if the model's performance isn't good enough, even with the RAG context, but I would still start with RAG.

00:18:52.660Aaron Gallant: Fine-tuning is usually for when the task is a specific sort of problem. It's not just, like, write code in some language, but it's, like, make a decision. Is this fraud or not? Is this, person sick or healthy? These sorts of things, you know, like, is this spam or not?

00:19:09.130Aaron Gallant: classification problems.

00:19:11.750Aaron Gallant: Okay, so…

00:19:14.300Aaron Gallant: Fine-tuning the process. What do you actually do to fine-tune? We've said it enough, let's understand a little bit more here. Well, you start from a model, and we'll talk a bit about model selection, but model selection is its own topic. We could have

00:19:27.070Aaron Gallant: At least a whole lecture on that, I imagine. So, suffice it to say, there's a whole zoo of models out there. We live in an interesting time for this.

00:19:37.040Aaron Gallant: Step 2, that's what I was just talking about.

00:19:40.690Aaron Gallant: that's where all the work is. You know, gathering, cleaning, curating your data. You've probably heard garbage and garbage out, right?

00:19:49.540Aaron Gallant: And that's true for, and I've said this a few times, I'm gonna be a little bit crisper now, model inference. Model inference, just so you know, all of these language models, they really are just math. Now, they're…

00:20:01.720Aaron Gallant: somewhat incomprehensibly large math, we can't understand them fully, really, the explainability is actually poor, but we can know observationally that they…

00:20:11.620Aaron Gallant: are able to essentially understand statistical patterns in language, and then use that to generate more language. Okay. Now, when we do that with these models, that's actually basically the same as just a good old

00:20:27.200Aaron Gallant: Cartesian plot. How many people remember Y equals MX plus B? Anybody at least remember that much from math?

00:20:36.530Aaron Gallant: Back in… whatever, middle school?

00:20:39.710Aaron Gallant: The slope-intercept form. Now, Y equals mx plus b is, like, the simplest sort of parameterizable formula you can sort of come up with.

00:20:48.690Aaron Gallant: the parameters is M, the slope. It's one parameter, right? And that slope

00:20:54.930Aaron Gallant: determines how the line points, essentially. Yeah, and then the Bs, the intercept, we kind of don't care as much about it, but yeah, it's there too.

00:21:02.210Aaron Gallant: All of these models are the same thing, except they're all Y equals MX plus B.

00:21:09.900Aaron Gallant: But the difference is, of course, Y, what it gives you, what it returns, is actually tokens, which are numbers, but numbers that are turned back to words. So, essentially, it makes words. X, what you give it, is also words, of course, your prompt, that's the input.

00:21:26.620Aaron Gallant: And M, instead of being a single number, like 3,

00:21:30.670Aaron Gallant: Is a bazillion numbers, or, you know, slightly more accurately, say, 7 billion parameters for a modest open source model, right?

00:21:40.800Aaron Gallant: 13 billion parameters, and 130 billion parameters, like, billions of parameters as opposed to one. Parameter just means number. So the number of numbers in M organized in these matrices, really huge.

00:21:53.690Aaron Gallant: Now, that said, it's still the same thing. It's still basically multiplying through. It's a little more complicated, because linear algebra, but you don't have to worry about that to understand it conceptually, really. You have to worry about if you want to implement it from scratch, which is cool, but you don't have to do that, and that's not where the reward necessarily is, so…

00:22:13.230Aaron Gallant: That is inference, though. Just sticking an X in there.

00:22:17.490Aaron Gallant: and getting different Ys. That's what we all do. Every time we prompt, every time we rag, every time we do any of that, we're executing the function and getting some new Y out, right?

00:22:27.410Aaron Gallant: Fine-tuning, we're changing the M. We're going to the M, and we're changing those numbers. So that's what I mean about changing, and those are called the parameters, those are the weights, or numbers that determine the behavior of the model itself, and that's a durable change to the model.

00:22:43.610Aaron Gallant: I'm emphasizing this because I think it's an important thing to understand, an important distinction to make. Are people pretty clear on this?

00:22:55.100Aaron Gallant: If there are still questions on this, drop it in the chat. Great, alright.

00:22:59.280Aaron Gallant: So…

00:23:00.960Aaron Gallant: back to the real world. We can step back from math a little bit. Thank you, Trevor. I enjoy math, I mean, and to be clear, it's great to go deep in math, but it's great to know how deep you have to go to solve the problem you're solving. And right now, this is deep enough.

00:23:17.670Aaron Gallant: So…

00:23:19.870Aaron Gallant: Of course, most of the work in our, say, corporate setting is dealing with the data, and the reason I made that big deal about inference versus training

00:23:28.720Aaron Gallant: Rewind here. Garbage in, garbage out. You've all probably heard that. It applies to both inference and training, right? If you give a bad prompt, you're gonna get bad output.

00:23:37.970Aaron Gallant: But the thing is, is that only happens that one time. You give it that bad prompt.

00:23:41.990Aaron Gallant: that one time, and you get a bad response that one time. Obviously, that's bad, by definition, but it's once.

00:23:50.350Aaron Gallant: If you use bad data while you're training, you're teaching the model badly. You're now basically making the model bad. You're making all the future bad. You're getting bad…

00:24:00.670Aaron Gallant: Ad infinitum.

00:24:02.310Aaron Gallant: So much, much worse. So that's why prepare data is so critical. You really need to be sure, if you're training your model, that your data is correct, that the labels, and we'll get more about what that means in a bit.

00:24:14.460Aaron Gallant: But basically, what you're saying the data is.

00:24:17.160Aaron Gallant: what it's learning from, it better be learning from something that's right. If it's learning wrong things, it's gonna make things worse. And it's also why… so, by the way, step 3, adjust weights, that's just run a function. That's just a little bit of code, and then…

00:24:31.690Aaron Gallant: Possibly writing a check to some compute source to do all the crunching.

00:24:37.410Aaron Gallant: But then 4, validate is where, I guess, the next most work is, because you want to take your label data, set some of it aside, you don't let the model train on all of your label data.

00:24:48.010Aaron Gallant: Because you keep some of it, and you use it to validate later. You use it to see, hey, is the model actually good or not? And the reason you have to hold it aside is you don't want the model to cheat. You don't want it to learn the answer key, as it were.

00:25:00.900Aaron Gallant: Alright.

00:25:02.170Aaron Gallant: By the way, pretty much everything I'm saying on this slide more or less applies to machine learning in general. I mean, we're talking about this in a large language model fine-tuning context, but this is all general supervised learning, machine learning stuff.

00:25:14.770Aaron Gallant: This is sort of the intersection of AI engineering and, say, machine learning engineer. So… Alright.

00:25:23.060Aaron Gallant: OpenAI. I'm not gonna spend too long on this, but I want you all to know it exists. So, OpenAI

00:25:28.990Aaron Gallant: which, you know, offers of ChatGPT.

00:25:32.210Aaron Gallant: despite their name, they're not exactly open, right? Like, they are somebody else's computer.

00:25:38.560Aaron Gallant: They are a service that we can pay, granted, a service that offers pretty good quality, it seems, and cheap prices at the moment, but a service that we pay, send some information over the wire, get something, get a response back, right? An API.

00:25:53.620Aaron Gallant: They still offer, though, a fine-tuning API.

00:25:57.930Aaron Gallant: And, the way that works in the case of OpenAI is it's sort of like a limited fine-tuning playground. You can specify the base model, so you could say, oh, I want to start from 4.0 Mini, and you probably do want to start from a small model.

00:26:15.310Aaron Gallant: and then you give it data, and the data has to be in JSON-L, JSON lines format. And that's like a…

00:26:22.550Aaron Gallant: a text file where each line is a JSON blob. And the format of the JSON blob, as you'll see in the next slide, is just…

00:26:30.060Aaron Gallant: discussions, because that's what label data actually is in this case. You know, labeled data in general.

00:26:35.650Aaron Gallant: going back to our friend Y equals MX plus B, labeled data would be pairs of X's and Y's. You know, for this X, this should be Y. For this X, this should be Y, right?

00:26:45.550Aaron Gallant: For language, it's prompt response. For this question, or request, or whatever, user input.

00:26:54.730Aaron Gallant: This should be the model's response. This is how it should answer that question.

00:26:59.040Aaron Gallant: Or, you know, maybe it should do a tool call. Whatever, like, this is what the model should do.

00:27:04.370Aaron Gallant: And it's convenient because you get that naturally by just logging every time you interact with a language model. Like, if you log interactions using an observation platform like Langfuse or Langsmith, you can get these, you know, prompt

00:27:19.720Aaron Gallant: Discussion histories, and you can use that

00:27:23.020Aaron Gallant: Assuming you also quality check it and make sure it's good, that's at least the right shape to be training data.

00:27:29.210Aaron Gallant: And that's the shape that OpenAI wants.

00:27:32.010Aaron Gallant: Now, that said, OpenAI doesn't really let us actually get anything.

00:27:37.270Aaron Gallant: At the end of the day, we give them the data, and they give us, like, a little bookmark. They give us an ID that says, okay, we're holding the data.

00:27:44.520Aaron Gallant: And then we say, we want to train the model, and they train the model, and they give us another little bookmark, another little ID that says, hey, here's your trained model.

00:27:51.690Aaron Gallant: You can't download it, you can just use that model ID for future inference on OpenAI's API.

00:27:58.430Aaron Gallant: And, of course, it comes at a cost. It comes at both a training cost, and inference is more expensive on your custom models than similarly-sized, non-custom models. Because, of course, OpenAI has to have some extra infrastructure in place to host your custom model.

00:28:13.890Aaron Gallant: Probably can't optimize or cache it as well, that kind of thing.

00:28:17.120Aaron Gallant: So, definitely a lot of limitations. You also, you know, don't get access to anything, really, any of the model weights. You don't really have a lot of hyperparameters. We'll see more hyperparameters in the good example coming up.

00:28:31.100Aaron Gallant: But the reason to know this exists is, first off, if you want to just, like, play with fine-tuning, and you don't mind spending…

00:28:37.340Aaron Gallant: you know, a little bit of time and money on OpenAI, this can be a good little testbed environment.

00:28:43.670Aaron Gallant: And secondly, you know, there could be use cases for this if you're deep, if your org is all in, open AI, and you want to fine-tune. But overall, again.

00:28:53.380Aaron Gallant: I think, at least, that it's the open… the actually open models where the… where the fun is at.

00:28:59.660Aaron Gallant: So,

00:29:02.230Aaron Gallant: on that note, well, first off, there's a little bit more about the message format, as I said. So that's… that's one line of a JSON lines file showing an interaction where there's, like, a conversation history.

00:29:15.370Aaron Gallant: And it's just a silly example, you wouldn't really fine-tune on this, most likely, but it shows for a sarcastic chatbot, if it receives this question, this is the answer I should give.

00:29:27.020Aaron Gallant: Great.

00:29:28.000Aaron Gallant: And you can make this, you can synthesize this by just interacting with a big LLM that can solve this problem. And that, again, is sort of the,

00:29:39.610Aaron Gallant: the bootstrappy way to at least generate your initial data. You still need to curate its quality,

00:29:47.730Aaron Gallant: and you, still need to format it appropriately for whatever fine-tuning platform you're using. In the case of OpenAI, they want this format.

00:29:57.600Aaron Gallant: Alright, so this… I'm not going to spend time here. Click this link if you want to actually do the OpenAI stuff. That's the point. This links to OpenAI's documentation. You'll have the slides at the end, this link will be on this slide, but this shows the various steps to actually do things on OpenAI's fine-tuning if you want.

00:30:17.210Aaron Gallant: But… what can you do besides play at OpenAI's playground? Well…

00:30:23.200Aaron Gallant: a lot of stuff. Again, I really think the coolest thing of this space isn't the, the behemoth

00:30:29.000Aaron Gallant: models and companies necessarily, but is the whole ecosystem. And the fact that this technology

00:30:36.450Aaron Gallant: I wouldn't go so far as to say it's, like, inherently…

00:30:39.950Aaron Gallant: Shareable exactly, or inherently open. And in fact, what we call open models

00:30:46.010Aaron Gallant: they're not really the same as open source, when you think about it. So,

00:30:51.500Aaron Gallant: Oh, let me address this question before I go off on this topic here. Is it fair to assume OpenAI might be easier to learn with? Yeah, if you want to just do, like, hey, I just want to do a bare minimum feedback loop of fine-tuning a model, you could do that in, you know, 20, 30 lines of code with OpenAI,

00:31:08.490Aaron Gallant: And maybe a little more, but like, yes, it would not be too hard. You can read their docs, and it's a sort of curated experience. The hardest part, of course, is the data.

00:31:20.450Aaron Gallant: But if you have that, then that's the easiest to sort of try it with. But again, the only use cases, I think, are if your org is, like, really all in with OpenAI.

00:31:30.510Aaron Gallant: And it's like, yeah, we want to fine-tune GPT-40 Mini to make it really good at something, whatever.

00:31:38.040Aaron Gallant: And again, I'd still fine-tune the small models, typically.

00:31:42.350Aaron Gallant: But anyway, so, back to model… selection.

00:31:49.550Aaron Gallant: So…

00:31:53.690Aaron Gallant: We have here, all the different places we can get models from, and as I was saying, it's this whole ecosystem of, quote, open models.

00:32:02.830Aaron Gallant: But I do think it's worth understanding, as practitioners or interested, aspiring practitioners in this space, what open model actually means. Most people just mean open weight. That means you can download

00:32:16.260Aaron Gallant: perhaps a multi-gigabyte file that basically contains all those numbers I was talking about, all those parameters in that matrix, or really a bunch of matrices, right?

00:32:27.830Aaron Gallant: And that's all well and good. That lets you use the model, and what's really nice about fine-tuning, that lets you fine-tune the model.

00:32:34.790Aaron Gallant: You actually don't need, like, the original blueprint, per se, or the original training data, or the original anything. You just need the model in its current state, and you can continue to train it. It's sometimes in the supervised learning space called online learning, which is to say that you can just continually train something without having to start over from scratch.

00:32:52.560Aaron Gallant: Really cool. That's… that's what enables all of this. However, it's not really the same thing as open source.

00:32:58.930Aaron Gallant: Because open source, when you think about it for software, means the source code is open, and the model is re… and, sorry, the software is reproducible, because anybody can take it from the source and compile it.

00:33:09.940Aaron Gallant: An open model is actually more like a binary. It's kind of already been compiled.

00:33:14.500Aaron Gallant: It's a little fuzzy, but I encourage people to be mindful of that in this space. There are a few organizations out there that are potentially trying to do open source models, where they actually open not just the model, but also the data and all the architecture and training process and everything.

00:33:31.790Aaron Gallant: But most of the big open models, like, say, Llama from Meta.

00:33:35.310Aaron Gallant: are just really mostly open weight. You know, that's what they put out there.

00:33:39.730Aaron Gallant: That said, we can still use them for this. Hooray. So, how do we pick what model to start from? Well, there's a whole bunch of models out there, and again, you'll get these slides, all these links are clicky, and you can get more details, but the highlights here…

00:33:52.510Aaron Gallant: Is, you might already be aware, there's a whole bunch of benchmarks and ways to measure model performance, and in general, you'd want to pick a model to start from that is, well, first, of the size that you want to handle, so however many parameters

00:34:07.970Aaron Gallant: basically corresponds to how much RAM

00:34:10.780Aaron Gallant: you need to run the model, which corresponds to what sort of environment you want to run the model in, right? So that's probably your first decision.

00:34:18.739Aaron Gallant: And then once you make that decision, you can compare models based on their performance at tasks that are close to the task you want to solve. So, for instance, if you're making something that's supposed to write code, you might care about Sweebench, which is a benchmark specifically for software engineering skills.

00:34:35.239Aaron Gallant: And then, regardless of the specific sort of benchmarks out there, the general metrics that you see in this space

00:34:42.199Aaron Gallant: So…

00:34:43.360Aaron Gallant: sort of the most accessible, and an important one, and really the only one I would ever present in, say, a business context is accuracy. And what we typically mean by that is how often is it right? Like, how often did it do the thing?

00:34:56.699Aaron Gallant: And again, that often means boiling the task down to a binary success-failure, or at least a classification sort of thing, which is often a realistic thing to do for a lot of tasks we care about.

00:35:08.950Aaron Gallant: And then rating, okay, it did this one right, it did that one wrong, it did it right 92% of the time, out of our, you know, test cases that we held out, right?

00:35:18.740Aaron Gallant: And that's… that's a good number to have. There are other types of accuracy, and I'm not going to get into the nuances of that now. Suffice it to say that if you start diving deep in this, find your local friendly data scientist, and make sure that you are understanding the numbers, but accuracy is a good place to start.

00:35:38.460Aaron Gallant: These other ones are more about… are a little more LLM-specific here, alignment with, human reference, and perplexity. And perplexity I like in particular because that basically shows how surprised the model is

00:35:55.610Aaron Gallant: And, what that means…

00:35:59.620Aaron Gallant: you're like, wait, how surprised the model is? Well, don't anthropomorphize the model, to be clear. The model doesn't get surprised the way you or I might, but…

00:36:09.950Aaron Gallant: Let's say you, for some reason, train a language model on nothing but the complete text from everything from Lord of the Rings.

00:36:19.700Aaron Gallant: Shakespeare, whatever. Some famous author, right? And a little dated, say, right? Like, it's… so this language model has been trained on the entire corpus of Lord of the Rings for some reason.

00:36:33.460Aaron Gallant: And then you fed it,

00:36:37.070Aaron Gallant: I don't know, anything new, right? You fed it, tweets, right?

00:36:42.680Aaron Gallant: Most tweets are not written.

00:36:45.890Aaron Gallant: the way J.R.R. Tolkien writes, certainly not the way Shakespeare writes, if we go back with that as our hypothetical training data. And so, what that means is that the input is sort of out of domain from what the model expects to see based on the training data. The model is trained on a certain domain.

00:37:02.970Aaron Gallant: Y equals MX plus B, domain, range, ringing a few bells.

00:37:07.260Aaron Gallant: Was, was, trained on a certain domain of input.

00:37:10.530Aaron Gallant: And, that domain doesn't cover this new input.

00:37:16.740Aaron Gallant: Now, of course, the example I gave is a little bit absurd, but it's meant to be illustrative. Realistically, what perplexity shows you

00:37:24.790Aaron Gallant: is, you know, let's say you're in a very specific… well, like that question we got earlier, like, oh, we're working with… in a medical space with this very specific language, and the model probably hasn't seen a lot of training data for this language, because it's not like it's Python. There's not a million Stack Overflow articles

00:37:42.460Aaron Gallant: on this particular piece of technology, and so the model doesn't know about it. Well, that's something that perplexity might actually pick up.

00:37:50.720Aaron Gallant: I still wouldn't necessarily open with fine-tuning to solve that problem, but you might see that sort of thing with perplexity.

00:37:58.010Aaron Gallant: And then lastly, the real gold standard is still… look at it, think about it. Now, that's 4, if you were wondering. Human evaluation. I'm assuming you're all humans.

00:38:10.750Aaron Gallant: I don't think we have any LLMs in attendance at the moment. I imagine that they're not at that point.

00:38:17.060Aaron Gallant: In any case,

00:38:19.730Aaron Gallant: the human evaluation step is kind of critical, sort of like user acceptance testing from a software perspective. We are the ones who define what success and failure is, what business value is. You know, we talk about models hallucinating.

00:38:32.810Aaron Gallant: But models are just generating text. They really are. You know, what to some sense, the skeptics say is actually correct. They are just next token predictors, but they happen to do it in a way that's often really useful if you use them effectively.

00:38:46.800Aaron Gallant: However, we are the ones who have to evaluate that. And so it's not that the models hallucinate.

00:38:53.200Aaron Gallant: So much, because they, again, just like they can't be really surprised, they can't really hallucinate. They don't have a mental model like that. That's not what's going on. But they certainly confabulate. They can just string plausible text together. You know, something I try to avoid doing.

00:39:10.410Aaron Gallant: well, I suppose I try to have plausible, but also correct text, right? The correctness, though, is something we have to ascribe to it. So that's… that's the takeaway here.

00:39:20.270Aaron Gallant: Alright, I'm gonna pause for a sip. I've been talking for a while, I haven't seen any questions coming in a bit. If anybody wants to drop a question in the, chat.

00:39:34.040Aaron Gallant: Gone once.

00:39:37.590Aaron Gallant: Going twice. Alright, well, I'll also try to save time at the end for more questions.

00:39:42.690Aaron Gallant: Oh, I see, something that just came in about perplexity. Yeah, I mean, perplexity is… is… would often reflect, like, if your domain, like, again, the example is you have a bunch of really specific medical jargon or medical technical jargon that you're feeding into this model that isn't well reflected in the model's training corpus.

00:40:01.140Aaron Gallant: So yeah, I think the answer is yes to your question.

00:40:05.710Aaron Gallant: Alright, so, RLHF.

00:40:08.660Aaron Gallant: Now, to be clear, That's reinforcement learning with human feedback.

00:40:12.740Aaron Gallant: Fine-tuning is not exactly RLHF, but they're kind of cousins-ish, and RLHF is important to understand, because RLHF is what those big companies, what OpenAI and Anthropic and so forth, are doing to make the models into chat models. Indeed, RLHF, we've had…

00:40:31.050Aaron Gallant: general pre-trained transformers in one form or another. I mean, the attention is all you need paper was 2017. Like, transformer architecture's been around, and I've been following it for a while, and it's been cool for a while, but it didn't really get obviously useful to most people until it started actually, quote, chatting.

00:40:48.760Aaron Gallant: And how did it do that? Because before that, it really was just a next token completer. It really was just a really smart autocomplete. It was just trained on, here's the words up to this point, what should the next word be?

00:41:00.530Aaron Gallant: But it was trained on a huge amount of data, and that made it do interesting things.

00:41:04.830Aaron Gallant: Well, it turns out that if you take that model that, you know, because if you take a model like that, and you ask it a question, is it going to answer your question?

00:41:12.080Aaron Gallant: Maybe not, because what it might do is it might just write more similar questions.

00:41:16.150Aaron Gallant: That's actually the most likely thing you'll see, because you see that pattern, FAQ pages and stuff, lists of questions, quizzes and stuff written on the internet.

00:41:24.940Aaron Gallant: And so, to make it instead

00:41:27.920Aaron Gallant: decide to answer the question, we do reinforcement learning with human feedback, which is setting up an environment, that's essentially what reinforcement learning is here, and the environment here is paying a whole bunch of contractors.

00:41:38.430Aaron Gallant: to interact with this thing, according to some playbook and some set of prompts that… and again, this is the main thing that distinguishes Brock from Claude here, is what the prompts were to do this step, and what the evaluation, what the sort of playbook that the humans used to say, yeah, this is good, this is bad.

00:41:57.170Aaron Gallant: But that's all this is. You give it certain input, And, you…

00:42:03.970Aaron Gallant: then check the output, and a human rates it, and then you train the model based on that, and then you just do that in a feedback loop until the model gets where you want it to get. And in a sense, it's sort of like labeled data, except you're labeling it in real time with human scoring.

00:42:21.290Aaron Gallant: So that's where the connection to fine-tuning, comes in here. So again, all the models we're using already had this done to them.

00:42:30.850Aaron Gallant: We're going to use a similar mechanism, but instead, we're going to label data, and

00:42:37.380Aaron Gallant: I already talked about synthesizing data.

00:42:40.110Aaron Gallant: And that's one good way to get labeled data. Another good way to get labeled data is user behavior.

00:42:45.910Aaron Gallant: So, you know, you've all seen there are… you know, every website in the world, pretty much, has little thumbs up, thumbs down widgets, right? As a user, you'll be like, I liked this, I didn't like that.

00:42:57.040Aaron Gallant: Well… If you instrument that, and that's what this snippet of code is sort of mocking.

00:43:03.700Aaron Gallant: you score that, thumbs up is 1, thumbs down is 0, and you save that in a platform like Langsmith, and you associate that with the whole

00:43:11.950Aaron Gallant: trace the conversation, whatever led up to it. Congratulations, you now have labeled data. You now have an LLM response to a user question.

00:43:22.130Aaron Gallant: And you know whether it was good or bad. And you can use that to decide how to include it in a dataset, right? So, really, user-generated data is the only way to get data at huge scale. You know, nobody manually makes

00:43:35.840Aaron Gallant: billion row data sets. That's a lot of contractors. You know, the only way you get really big data is this. Now, that said, you don't need really big data, and big label data, that is, to do fine-tuning, but this is still a good, convenient way to grab it.

00:43:54.210Aaron Gallant: My rule of thumb… for any, like, how much data do I need to train a model in general.

00:44:00.200Aaron Gallant: is, well, how much variation is there in the space you're thinking about? You know, what's a representative sample look like?

00:44:06.450Aaron Gallant: And you probably want, you know, at least a few thousand observations, rows, you know, data entries, and you want them to cover all the cases. But I would rather have a thousand representative

00:44:20.660Aaron Gallant: sample that has variation in it than 10,000 that are all just about the same thing over and over again. Because that's not as good to train on, because it won't teach it about the whole variety.

00:44:34.570Aaron Gallant: Alright, so finally on to some of the alphabet soup here, and I'm checking the time. I'd love to answer more questions, but I'm gonna have to push a bit, I think, to get through these slides first, and then as there's time at the end, I will, knock out a few more.

00:44:50.820Aaron Gallant: How do we actually do this? How do we do this with a model that, we control? Well, we can use something called parameter-efficient fine-tuning to make it even easier.

00:45:01.720Aaron Gallant: And what this does is this takes our model, our pre-trained transformer model, freezes most of it, and then adds a few layers. And these layers have to be the right dimensions to map up. The software makes that not too hard.

00:45:16.380Aaron Gallant: Dimensions is just the number of numbers, and they have to align.

00:45:20.690Aaron Gallant: But then, that means that instead of training the whole model, we're training

00:45:25.660Aaron Gallant: less of the model. Now, you might think, well, that means we change it less. The answer is, yeah, kind of, but this is the output, so we still are potentially very much changing the actual

00:45:35.900Aaron Gallant: behavior, the end result of the model. We're just kind of doing it a little more cheaply and keeping most of the model in place, because hopefully the model is kind of close to what we want, we just want to tweak it.

00:45:46.280Aaron Gallant: So, that's efficiency number one.

00:45:49.320Aaron Gallant: Train less stuff.

00:45:51.360Aaron Gallant: Efficiency, too.

00:45:53.730Aaron Gallant: make the num… make there be fewer numbers in the number… sorry. This is have fewer sets of numbers. This is have fewer numbers in the sets of numbers. So, all of these…

00:46:04.980Aaron Gallant: neurons, all of these things in a neural network are organized in matrices. Matrices have rank, which is another word for, essentially, size or dimension, the number of numbers in them.

00:46:15.990Aaron Gallant: Well, linear algebra

00:46:18.600Aaron Gallant: lets you do certain things, to manipulate the rank of matrices. And, what that means is you can basically

00:46:31.390Aaron Gallant: lossily make a matrix smaller. Lawsily means you are losing information, but you're keeping the most significant information, the most relevant information in the matrix, the most important numbers.

00:46:42.560Aaron Gallant: And you're losing some other numbers that, you know, most matrices that are really big are sparse, they're mostly empty. So you're not losing too much. It's sort of like, you know, taking a wave and making it new into an MP3.

00:46:54.720Aaron Gallant: Or taking a picture and making it into a JPEG.

00:46:57.670Aaron Gallant: We're losing a little bit, but most of it's there. And the result is, hey, less numbers.

00:47:03.400Aaron Gallant: And then, lastly, we can also quantize, and quantization is not at all specific, of course, to,

00:47:11.790Aaron Gallant: to large language models. Quantization is something that's been around in computers as long as there have been computers, I imagine, at least digital computers, because it just refers to truncating floating-point numbers. So we're taking, say, 32-bit floats and truncating them down to 8.

00:47:28.900Aaron Gallant: And we keep the 8 most significant digits, and we're still accurate to a pretty fine decimal place, just not as many as before.

00:47:36.270Aaron Gallant: Again, we're losing some information here, but we're keeping the most important information, and we're making, in this case, 32 to 8, that really is… that's 4 times smaller, right? So, if you stack all this stuff together, the end result

00:47:51.200Aaron Gallant: Is you can take something like a 13 billion parameter model.

00:47:55.940Aaron Gallant: And you can actually fine-tune it on possibly the laptop you have on your desk right now.

00:48:02.590Aaron Gallant: Not gonna guarantee that for everybody here, but if you have a not terribly old Mac.

00:48:08.470Aaron Gallant: chances are pretty good. The reason I highlight Mac is Apple doesn't pay me, sadly, but the M1 architecture has good optimization for this sort of stuff. If you're not in the Apple ecosystem, never fear, but it just means you might have to…

00:48:24.020Aaron Gallant: get some other thing, like, you know, a GPU or some other bit of silicon that's the right way to do this sort of thing.

00:48:30.920Aaron Gallant: In any case… That's the takeaway here, that we can, train models

00:48:38.400Aaron Gallant: but really little resources, and that enables a lot of experimentation. And again, the main goal here is to train these small models anyway. That's the main useful use case, is to take a model that you might want to run on a Raspberry Pi, or something funny like that, and train it really small. And to be able to do that on your own hardware is pretty powerful.

00:48:58.930Aaron Gallant: Alright, summary slide, I'll leave this here for your reference, and I want to use the remaining time

00:49:04.710Aaron Gallant: Please start dropping questions again, and I will catch up, but while people are thinking and dropping some questions, I'll just very quickly show the code example. I'm not gonna really run through it, but you have it, it was just linked there, and it's this link here.

00:49:21.320Aaron Gallant: And, I have a pre-ran version of it here, because it does take a while to run. A few notes. This is Google Colab, it's a way to run Python notebooks in the cloud. You don't have to run this. This is only this notebook. Daniel's the only one. You don't have to run it in Colab. You can download it, you could run it on other services.

00:49:40.350Aaron Gallant: Also, you should probably make a copy of this notebook. So if you see here, save a copy in Drive, you want to make a copy so you have your own copy that you can edit and run, and you also are going to want to change the instance type. So if you see that there's these connection…

00:49:55.540Aaron Gallant: options, runtime types, you're gonna want the T4 GPU instance. Otherwise, this stuff won't really run great. And that is for reasons of parallelized vector computation stuff.

00:50:09.770Aaron Gallant: The math, right? Has to have the right chip behind it.

00:50:13.760Aaron Gallant: So, what's in this notebook?

00:50:16.640Aaron Gallant: 5-minute version or less. A lot of boilerplate and imports.

00:50:22.380Aaron Gallant: And then… a lot of setup with a lot of hyperparameters. So I've used the word parameters a ton.

00:50:28.840Aaron Gallant: Parameters, again, are the numbers inside the model, the numbers that we normally don't change when we're just doing RAG or whatever, but that we are changing now with fine-tuning. We're changing those numbers. Okay.

00:50:40.110Aaron Gallant: Hyperparam… the thing about parameters is there's literally billions of them, and if you point to any one parameter and be like, why is that parameter 2? Or it wouldn't be 2, like, 0.6, whatever.

00:50:52.320Aaron Gallant: we don't have… that's what I mean by the… we don't have good explainability. We don't know. All we can say is, like, well, that was just the optimal point that the training process arrived at for that parameter, and we can just observe the outcome.

00:51:03.310Aaron Gallant: However.

00:51:04.740Aaron Gallant: Hyperparameters are not inside the black box. They're like the knobs and levers on the outside, and they're what we have control over, and they're what you want to find, again, your local data scientist or machine learning engineer to get a deeper understanding of this, and to know what you're doing here.

00:51:19.940Aaron Gallant: But you want to choose these hyperparameters such that you have a good training process. And there's rules of thumb.

00:51:28.280Aaron Gallant: But there's not really any single hard and fast answer to how to do all this. So, to some extent, you also have to do experiments, where you do smaller training runs, and you try different hyperparameters to see what's best. There's a lot here. What I'll highlight

00:51:42.590Aaron Gallant: Dropout is an interesting one, because dropout's a little counterintuitive. Dropout refers to, essentially

00:51:50.890Aaron Gallant: A probability that a neuron that you're training just won't even fire.

00:51:55.080Aaron Gallant: And you might think, well, why would we ever do that? Why would we go to all this effort, and train things, and then this part of it… we just say, oh, this math isn't gonna math this time.

00:52:04.360Aaron Gallant: That's a waste, right? Well, no, because what we're doing is we're…

00:52:09.080Aaron Gallant: making the system robust to failure. It's sort of like chaos testing, if you're familiar with that, where, you know, your infrastructure should be robust to things randomly failing. Well, the model should be robust to neurons randomly not firing, because that also

00:52:23.880Aaron Gallant: ensures that we don't overtrain in any one particular area of the model and end up with, like, one neuron with sort of too much responsibility. The reason that would be bad is that's not going to generalize well. It might seem like it learns the training data really, really well, but it's overfitting, and it's not going to generalize to unseen data, and that's what we actually care about.

00:52:44.280Aaron Gallant: Bunch more stuff here. Some of these are a little more operational. Number of epics, that's how long we train, essentially, one full training run.

00:52:52.320Aaron Gallant: Learning rate. This is another one that I would really highlight. This is one that's even exposed by OpenAI's API. This is how fast you train, essentially. This is how fast you update the weights when you need to update the weights. And again, you might think.

00:53:06.520Aaron Gallant: well, just make this as big as we can, right? Like, obviously, we don't want to waste time. We want to train quickly. Well, yeah, we do, but what we're doing here is we're solving what is called an optimization problem, which is we're finding the best set of numbers to make the model behave as best as it can with our data.

00:53:25.900Aaron Gallant: Basically predict the data effectively, accurately.

00:53:30.100Aaron Gallant: And it's sort of like, if you picture it, it's in a very high-dimensional space, billions of parameters, but picture just three-dimensional space. Picture climbing a mountain.

00:53:39.380Aaron Gallant: Or maybe picture, two-dimensional space, even. Picture putting a golf ball on a green, right?

00:53:46.450Aaron Gallant: As you get closer, To the whole.

00:53:49.850Aaron Gallant: you want to use less force, right? And why is that? Well, if you hit the ball too hard, essentially you take too large of a step.

00:53:57.340Aaron Gallant: You're gonna skip straight over that hole.

00:53:59.870Aaron Gallant: Right? And that's… that's bad. You're not going to reach that optimum point. If you need to reach a specific, precise point in two-dimensional space, you need to take little steps the closer you get. That's what we're talking about here, and that's what…

00:54:13.150Aaron Gallant: This is doing for us.

00:54:16.560Aaron Gallant: All right, and then the rest of this, and then I want to get to the questions for the last 5 minutes. There's more hyperparameters here. Documentation for these, software will answer questions if you have particulars.

00:54:29.560Aaron Gallant: And we put it all together, basically, so we're hooking up… you see the bits and bytes, that's the quantization bit, so we're doing QLORA,

00:54:39.410Aaron Gallant: And, the tokenizer is the thing that changes the words into numbers, vice versa.

00:54:44.950Aaron Gallant: And we're fine-tuning the model.

00:54:49.180Aaron Gallant: And I'll show the data in a second, too, but what we're fine-tuning the model with

00:54:53.880Aaron Gallant: is from Hugging Face, somebody just mentioned Hugging Face.

00:54:57.960Aaron Gallant: we have some label data about tweet sentiment. So the idea is, is, what is the sentiment of this tweet? Rate it negative, neutral, positive, and then there's a tweet about some company doing something, and then at the end, there should be a label. Yeah, neutral.

00:55:14.330Aaron Gallant: And etc.

00:55:15.760Aaron Gallant: And the idea is, oh, we could use this model to maybe do financial stuff.

00:55:21.000Aaron Gallant: We run trainer.train, and it's a llama model that we're starting from.

00:55:25.680Aaron Gallant: We train it for a while, and we see the loss go down. The loss is sort of how bad the model is. It's another way to

00:55:32.730Aaron Gallant: say, model performance, and by how bad, I mean how unlike the good model output. Like, how un… how often is it wrong, based on the labeled training data? And how much is the output like?

00:55:44.710Aaron Gallant: What it should be.

00:55:46.100Aaron Gallant: And so we want loss to go down, and indeed it does. We see that it starts out at 3.29, and then it goes down, and it sort of bounces around, kind of stabilizes around .9 or 1, maybe.

00:55:58.190Aaron Gallant: If we look at that in this little dashboard here, we can see the…

00:56:03.500Aaron Gallant: Light gray line is the actual jagged loss, and the bolder line is smoothed as a rolling average of the loss. And you can see that it sort of bottomed out, and that's when we stopped training. So we've optimized the model to some extent.

00:56:16.580Aaron Gallant: Probably not really, because we could try other hyperparameters, but we certainly made the model better at doing this data.

00:56:24.430Aaron Gallant: And if we give it an example now, a hypothetical tweet about Tesla superchargers, we will see that it responds.

00:56:34.230Aaron Gallant: And you notice, this format's different Oh.

00:57:00.640Sunny with Gauntlet: Hi everyone, I think Aaron's just having a little glitch, let's just give him a second, as I think his computer might have froze, of course, right at the last second.

00:57:10.060Sunny with Gauntlet: Let's see if he's able to hop back on…

00:57:24.580Sunny with Gauntlet: Yeah, let's see…

00:57:27.090Sunny with Gauntlet: Oh, it looks like we might have dropped him. Sorry about that, everyone. But we'll be sharing this recording via email, so no worries. Oh, is he jumping back on? I think we might have gotten back.

00:57:38.540Aaron Gallant: Sorry about that. My computer over… my computer literally overheated, it's kind of hot.

00:57:43.240Sunny with Gauntlet: Oh, no!

00:57:43.860Aaron Gallant: I didn't… I didn't turn on air conditioning aggressively enough. Yeah, it's… anyway. We're about to wrap up. So, I was sprinting to the end of the notebook. I'll go ahead and,

00:57:56.950Aaron Gallant: you, you have a copy of the notebook, so you can see that it rates the tweet as positive sentiment. So that's… that's the idea.

00:58:04.420Aaron Gallant: Let me get this back up here.

00:58:08.640Aaron Gallant: And also, check the questions I can check in the chat.

00:58:18.430Aaron Gallant: Alright.

00:58:19.780Aaron Gallant: Get the chat back up.

00:58:22.840Aaron Gallant: Oh, except…

00:58:24.700Aaron Gallant: crashing out means I lost the chat history. So if you asked a question earlier, please feel free to re-ask it. I'll spend at least a few minutes here at the end answering questions, and then we'll, I think call it. So, do you have a strong preference, between LM Studio, Ollama, or other app storing local models? Not a strong preference. I mean…

00:58:45.230Aaron Gallant: a lot of them build on the same, like, you know, llama.cpp, or same sort of, underlying stuff. Lm Studio has been sort of a…

00:58:54.060Aaron Gallant: kind of convenient UI, I'll give it that. But if I was doing something programmatic, I might reach for Olama instead. A link to the slides. Yes, there is a link we can share for the slides. I can drop that now, or maybe Sonny will drop that.

00:59:11.200Aaron Gallant: Here, I've got it.

00:59:16.450Aaron Gallant: Or, no, I need to send it to everybody.

00:59:19.380Aaron Gallant: There we go.

00:59:23.430Aaron Gallant: All right, and while we are possibly waiting for a last question or two, I just want to also close by saying thank you for joining. I hope you enjoyed this. As clear, I enjoy this stuff. I think this is a great

00:59:36.650Aaron Gallant: area to be in, a great thing to be thinking about and learning about, and if you are interested in thinking and learning more, please join us more. They're, of course, going to continue to be night classes, but we also

00:59:49.170Aaron Gallant: As I imagine you know, I have the regular full gauntlet program. The next, application, I believe Sonny said, is July 6th. Did I get that right? Or not… apply before that, but the next start is July 6th, is that right?

01:00:03.560Sunny with Gauntlet: Yes, that's correct. Next cohort clicks off July 6th.

01:00:08.230Aaron Gallant: July.

01:00:08.730Sunny with Gauntlet: I'm gonna drop those links now.

01:00:10.490Aaron Gallant: And also, if maybe you're not looking to make that sort of a change, you're, you know, you're at a place, and you have a career, but you still want to learn more about this, and your employer's supportive of that, or…

01:00:23.080Aaron Gallant: perhaps you're the one making the decision. We have Catalyst, which allows for similar training for people who are already on the job, and we give a lot of hands-on help to make sure that these skills are really useful and applicable

01:00:38.420Aaron Gallant: And you see return on it in what you're doing.

01:00:40.870Aaron Gallant: And then last but not least, but there's also going to be…

01:00:45.790Aaron Gallant: And Sonny, you'll have to help me here, because I lost windows here, but the, event in Austin…

01:00:53.530Sunny with Gauntlet: Yes, so we are having a live event in Austin. It's called Fired Festival. I'm gonna drop a link right now in the chat for everyone. It's… we don't really always do live events, so it's gonna be a great time. If you want to apply for that or learn more, just follow the link that's gonna be in the chat.

01:01:09.630Aaron Gallant: Great, and I think that's all we've got. Made… made perfect time. I…

01:01:18.240Aaron Gallant: Looking at the questions here, what would we recommend for less technical people? I mean, not… this was probably a particularly technical night class, they vary. There will be other night classes that, have a variety of topics, and really, our programs in general, we do have, you know, product focus and other aspects to our program.

01:01:39.260Aaron Gallant: Or to our programs, I should say.

01:01:42.100Aaron Gallant: Alright, I don't see any other burning questions, so, thank you all, and

01:01:52.470Aaron Gallant: Hope to see you again at something gauntlet-related.

Download transcript (.vtt)

FAQ

When should you fine-tune an LLM? +
Fine-tune when the failure is about style, format, or domain behavior that prompting and retrieval cannot fix. If you just need the model to know more facts, use retrieval instead, since a good prompt often matches fine-tuning at a fraction of the cost.
What is fine-tuning, really? +
You continue training on your own examples so the model adapts its behavior, which is a different thing from steering it at inference time with prompts or retrieved context.
What are PEFT and QLoRA? +
They are parameter-efficient techniques, where PEFT fine-tunes a small set of weights and QLoRA quantizes to cut memory, and together they make fine-tuning much cheaper than full retraining.
How do you choose between fine-tuning, RAG, and prompting? +
Use prompting for behavior you can specify, retrieval for missing knowledge, and fine-tuning for consistent style, format, or domain behavior that the other two cannot reliably produce.
Why can fine-tuning quietly waste time and budget? +
It needs data prep, training runs, and evaluation, and it frequently lands where a better prompt would have, so the cost only pays off for the right job.
How much data do you need to fine-tune? +
Less than people assume with PEFT and QLoRA, where quality and representativeness count far more than raw volume.
How do you know fine-tuning actually helped? +
Evaluate against a held-out set before and after, and if a prompt baseline matches it, you did not need to fine-tune.

What's next?

Keep building with the rest of Night School, or apply to Gauntlet — ten weeks of technical intensity with the best AI engineers we can find.