The world of embedded systems (Interview)

32:00Something like this. That's a bad paraphrase. and I was just teaching my kids about budgeting this morning and was talking about how if you're not diligent that your lifestyle will expand to match the size of your income sometimes beyond. It's just kind of like a human nature thing and I think our software expands to match the size of the hardware that it's running on. To use every last bit and has allowed for many of us web developers like myself included to be lazy and do things the easy way, the fast way, the cheap way. But in that embedded world where every resource counts and it's heavily constrained, you must have to know all of the optimizations, all of the little tricks, the way to manage your memory so as not to bloat or leak.

32:54Or do you like, I mean, you must like that. I mean, it's literally a constraint. Puzzle. The challenge of, yeah, it's a puzzle. Yeah. And to many of us, that would be a huge bummer, you know, because it's hard and there's things to do. But you must like puzzle solving a lot. I do. I do. I really like puzzle solving. And I'm not super afraid of math. So if I need to add up where all of my memory is going or be able to look at a memory map file and see hex addresses, those don't really scare me. I know that they're just addresses. They just look funny. But I really enjoy the puzzle part. And like I said, I really enjoy the motor moving.

33:46And when the robot, for a personal project, I wanted to do some higher level software. So I got a Jetson, which is NVIDIA's AI embedded system. It's a Linux system. It's a computer. I used it as a computer for like two weeks. And I wanted to get a really janky arm, a robotic arm, that could go up and down. and grab things. And I wanted to make a typing robot. I wanted to have a camera that would see the keyboard and then the robot would hold a little, I don't know, pin tooth-like thing. And it would punch out hello or whatever I told it to punch out. And it kind of worked. And I used some machine learning and I did a lot of computer vision, which is an area I'm always interested in but then one day i'm telling it okay type hello and it types help and i'm like done i'm like this is the end of this personal project i cannot make it any better this is this is the best and i forgot why i started this story but i hope it was amusing well i'm wondering maybe if uh if it wasn't a fiona moment my friends who are big fans of a very awesome tv show called Silicon Valley will know what I'm talking about.

35:14Because in this episode, spoiler for those who haven't watched it yet, she asks for help. She is a robot. She needed help. I'm wondering maybe if your robot needed help. It might have. It certainly did. It was in the way of how much software can I throw at a problem to make the hardware that is very bad better. It turns out you can throw a lot of software, some bad hardware and it still is bad hardware but it can be really fun

36:01what's up friends i'm here with kurt mackie co-founder and ceo of fly as you know we love fly that is the home of changelog.com but kurt i want to know how you explain fly to developers do you tell them a story first how do you do it i kind of change how i explain it based on almost like the generation of developer i'm talking to so like for me i built and shipped apps on heroku which if you've never used heroku is roughly like building and shipping an app on versell today it's just it's 2024 instead of 2008 or whatever and what frustrated me about doing that was i I didn't, I got stuck. You can build and ship a Rails app with a Postgres on Heroku.

36:37The same way you can build and ship a Next.js app on Vercel. But as soon as you want to do something interesting, like as soon as you want to, at the time, I think one of the things I ran into is like, I wanted to add what used to be like kind of the basis for Elasticsearch. I want to do full text search in my applications. You kind of hit this wall with something like Heroku where you can't really do that. I think lately we've seen it with like people wanting to add LLMs kind of inference stuff to their applications. On Vercell or Heroku or Cloudflare or whoever these days, they've started like releasing abstractions that sort of let you do this.

37:09But I can't just run the model I'd run locally on these black box platforms that are very specialized. For the people my age, it's always like, oh, Heroku was great, but I outgrew it. And one of the things that I felt like I should be able to do when I was using Heroku was like run my app close to people in Tokyo for users that were in Tokyo. And that was never possible. For modern generation devs, it's a lot more Vercel based. It's a lot like Vercel is great right up until you hit one of their hard line boundaries. And then you're kind of stuck. There's the other one. We've had someone within the company.

37:39I can't remember the name of this game, but the tagline was like five minutes to start forever to master. It's sort of how our pitching fly is like you can get an app going in five minutes. But there's so much depth to the platform that you're never going to run out of things you can do with it. So unlike AWS or Heroku or Vercel, which are all great platforms, the cool thing we love here at ChangeLog most about Fly is that no matter what we want to do on the platform, we have primitives, we have abilities, and we as developers can charge our own mission on Fly. It is a no limits platform built for developers, and we think you should try it out.

38:15Go to fly.io to learn more. Launch your app in five minutes. Too easy. Once again, fly.io.

38:29I'm curious about the possibilities of embedded systems. Jared, maybe your question regarding the chasm is if I begin, how far is that dip, so to speak? How far is it from beginner to potentially professional? How big is this embedded world? Is it an underserved world? Is there a lot of opportunity? Is there one place to go to learn? Is it all in university or college? How do you get to be a pro? And if you do become a pro or someone who's professional, earning their money and career via this route, is it big? Is it expansive? Is it growing? You might have noticed that your home has more blinking lights than it used to.

39:13Yeah. I'm sad about that. Electrical tape. There you go. It is growing. Some things are becoming easier. Some things are becoming more straightforward. But I see more job openings than people looking for jobs. So I would say that's pretty good. How do you get there? There are a lot of paths because, like I said, what is an embedded system depends on your perspective. There are quite a few embedded Linux projects that still are embedded because they need to be very conscious of their constraints. And yet, to most software engineers, it wouldn't look that different. It might be boring because it's trying to figure out how to provision 100 small sensors with IP addresses.

40:08But the actual algorithms and such wouldn't be strange. It would be the interactions that were a little different, having to deal with actual boxes out there. But then when you start getting into higher, I guess it's lower tolerance, things where you have to last for five years or have to squeeze out every micro-amp possible or even things where firmware update is very hard and if you do it wrong, things go very bad for you. Like firmware update at the bottom of the ocean. Those, you don't start with those. I think that's what I need to say. There are better places to start. And there are a lot of things that software engineers know that embedded software engineers are still picking up.

41:02We are not really that good at CICD. somebody could just tell us how to use Jenkins instead of making us figure it out each time that would be great yeah I mean that's not that's not I use Jenkins but sure it's not some of the software engineers I work with Jenkins is just like obvious every part of it is obvious and to me it's like I can do what I need to do and when you ask me to set up a new project I just go home and, and like, I don't know, turn off my computer for the rest of the day. Design patterns was one of those things that I really wanted to bring to embedded systems. And I feel like I have kind of talked about it for the last 20 years about how important it is to not keep solving the problems in new ways, but to start solving the same problems in the same ways each time so that we don't have to rewrite all the code.

41:59And things like object-oriented programming, it's not new. And a lot of embedded people really understand it. And yet we mostly program in C still. I know. I know the gasps, the horrifying screams. Yes, we still mostly program in C. But we don't usually program in an assembly anymore. We're making progress. There you go. There you go. Yeah. So what about open source? Has that permeated your crew? Yes. There's open source. I mean, the vendors of our chips have figured out that they don't really want to keep giving us their crummy software in ways that only they can manage it. So like ST and Microchip and Nordic all have big repositories and they have examples of working with different sensors.

42:58They have all the drivers there. And then we have things like Zephyr, which is an operating system that is kind of like you took Linux and you made it much smaller. And then instead of anything you could run at runtime, you hard coded it at compile time. which has some interesting things. The device tree is one of those things that most people in their first three months of Zephyr experience will start cursing. And once you have, yeah, so Zephyr is huge, and FreeRTOS is now open source. Walkway is open source, most of it, I think. I think there's just a little bit that isn't. A lot of the sensor vendors, the IMU vendors, will give us code.

43:51And sometimes that's partially open source and sometimes it's not. But the easy stuff now, the Kalman filters that used to be so complicated, now that's code they just give you. They tell you how to do it. What's that? A Kalman filter is a way of, I could talk about this for like three hours. I'm trying not to. Okay. Accelerometers and gyroscopes. Accelerometers measure gravity. They measure acceleration too, that's what they say, but really they just measure gravity. Gyroscopes measure spinniness, how fast you're turning. And it doesn't really matter whether you're spinning around a certain thing or whether you're spinning around a part of a thing.

44:41So you can measure how your arm moves with a gyroscope. It's not like it has to go around and around. It's just how much is it turning. When you are in a car and you're going around a cloverleaf, you're spinning and you feel acceleration. You feel acceleration, you know, the outward push of you as you're spinning around. An inertial measurement unit can use both of these sensors and know which one is wrong. And it's kind of like your ears and your eyes. If you are spinning around in a circle and your eyes are closed, you're going to get dizzy much faster than if your eyes are open. a Kalman filter is a way to combine sensors with different error features, different error characteristics into a way to get a much better response because you can characterize the error when nothing's happening or you can characterize the error in certain conditions where you know one is good and one is bad, like on a cloverleaf.

45:49Gotcha. And so the vendors of the device would provide like a set of filters for you to use? They would provide the algorithm, yeah. And they would provide the base code for how good each of their sensors are. Okay, because they've done those tests and they understand the sensors and their error rates and these other things. Exactly. Better than you do. Better than I want to. Yeah, I mean, after a while you may understand them better than they do. but those used to be closed source or these used to be unavailable to you and now they they're just provided in 1990 it was a patent i worked on so you helped invent these things well no i helped implement them okay it was came out of a and and to be fair a lot of people started doing imus and Kalman filters in the early to mid 90s.

46:50And it exploded as to the applications. But a bunch of PhDs were done at that time. And so they became very, very popular because they solve the inertial measurement unit problem pretty well. And now they solve a lot of other problems that involve a bunch of sensors. Usually when we talk about self-driving cars, the GPS has one set of errors, mostly it's really slow. The LiDAR or camera can have another set of errors and it provides a different set of information and the different types of Kalmans can put all of these together. And the idea is just you put the errors here and you put the data here and you try to make the best out of the bad situation you have gotcha so you mentioned self-driving cars and you mentioned lidar so i have to ask your take as an embedded person on tesla's proposition that they can get to autonomous or whatever level you need to get to with cameras only and then waymo's using lidars other people are using LIDARs.

48:06How important is the LIDAR, do you think? Pure opinion. Versus having a bunch of cameras and letting that be your only input or your only data streams. I can't picture doing it only with cameras. That's what a lot of people say. I mean, the argument is humans do it only with cameras. Right. All we have is eyes, right? But we have a few other things. Such as? Our eyes and our inertial measurement units are very well connected. Okay. So there is no disconnect that if you jerk the wheel, you know what your vision's going to do. Okay, I'm tracking you. And as a human, cities and self-driving cars.

48:56I worked on autonomous vehicles, not just underwater, but self-driving large off-road vehicles. and we did it almost entirely with cameras. But part of our goal for the future was dealing with stop signs and pedestrians. And I spent a lot of time with the machine learning systems and the reinforcement learning systems. And it's a hard problem, and I don't know how they're going to solve city driving in a way that is truly safe and doesn't have systemic failures. That's one of the problems with medical devices. When we talk about doing embedded systems and medical devices, it's usually there's some element of, oh my goodness, there's so much documentation.

49:51There's so much documentation the FDA needs. And that's primarily to make us understand the ramifications of the points we're making. And the reason for that is because, oh, I've started this in the wrong place. Okay. Moving backwards. If you work on airplane stuff, there are different levels of concern. If you take down a small airplane, a Cessna, it may crash into something important like a school until you have a certain amount of damage you can do with a small plane. Sure. You have a much bigger amount of damage you can do with a larger plane. medical devices are a little confusing because, and safety critical devices, medical or vehicles, have a problem that you can have thousands of them in the field when they start to fail.

50:43And so your problem is not that you have hurt one person. Your problem is that your software has decided that the 29th of February is not a real day and has gone nuts and is now on its murdering spree. And that's what you need to avoid. And so the thing with cars is that we keep trying to make them better in small ways, but I feel like we've kind of reached a local minimum, that we're not going to get better with just cameras. I'm not even sure we're getting it better with just LIDAR. but I don't know how to build in that feeling of where the car is the proprioception that you get as an experienced driver I'm afraid that cars will drive like 16 year olds forever and I don't know that that's what we want yeah as an outsider who I'm not sure I would ever even venture into the world of what it would take to be on a team that creates autonomous cars i'm just i would yeah i'm a fan from afar let's just say of the idea me too i like the idea but i think it's this is layman this is from the outside it seems perpetually flawed because it's not a closed system there's so many there's just so many flaws in the system that you can almost never predict You can predict a lot of the various ones, but you can't predict all of them ever.

52:17And because it's not a closed system that you control, then it seems like you're perpetually going to be flawed. Like no matter what, there's always going to be something that changes the game, potentially catastrophically, that you can't circumvent. As a human, you can't do that either. Right. Yeah, I think your point about the horizontal scale, Alicia, if that's just the way I'll say it, is that, you know, we have crazily bad 16-year-old drivers, but it's not like at a specific moment in time, you know, this percentage of all cars on the road are going to turn into that. Like the error that happens spread across a fleet of cars all running the same software, for instance, that makes the same mistake all at once is catastrophic.

53:05Of course, you know, the famous or infamous, I think this was the one that the first Tesla that killed somebody that was in FSD mode was because it came over a hill and the sun was coming up and the sun shone into all the cameras at once. I'm just giving my depiction of what happened. It's roughly that. And the thing, I mean, it couldn't handle it. Well, of course, a human couldn't handle that either because your eyes are completely blinded. And so likely maybe the same result in that one instance. But if something like that happens across 50 ,000 cars at the same time, that is a catastrophe. Of course, it's a tragedy when one person dies.

53:45But when this fleet suddenly goes bad, I mean, the stakes are way higher, right? Yeah. I mean, I can, I've been that human, Jared, that's coming up over a hill with the sun. I mean, I was 16 and I can, well, not even at any age, really. I mean, period. As even a good driver, I would say a marginally good driver. I think I've had the most recent accident I had, which is like the one I didn't cause. The deer hit me. And this was like a year-ish ago. Some things can't be avoided. This deer wasn't like random at nighttime when you would think a deer would be out. It was 9 in the morning in town. Like where other cars were at.

54:27It wasn't the time when deer should be running necessarily. He was looking for your car. Ran right into me. And then the other time was, geez, probably 20 years ago. And it was just wet. And I just didn't stop in time. I stopped, but my car decided to skid because that's what happens. And sure, I could have been paying a little bit better attention, but I'm a pretty good driver is the point of the last 45 seconds. That's the point you're trying to get across? I'm a pretty decent driver. Right. But I've been in a scenario where the sun has completely overtaken my vision completely. I do recall in seconds beforehand what was in front of me.

55:09and so I went now from literal memory with confidence of my next driving maneuvers straight right and around the bend a little bit kind of kind of because I was going from my memory before my sight was taken away in the case of the cameras you know the thing freaking out I don't know what happened there maybe it was like a literal panic mode but you know like in my own body I had a panic version but i just didn't crash there was no adam bsd or bsod there's a it was just a continued driving was safe one of the reasons why i've been tentatively positive about this progress is because of how many accidents there are and because how distracted and bad we are at driving even though we are amazing alicia to your point about the actual physicality of observing the world that we're in and that connection between our hands and our eyes being just grounded in i don't know biology we're pretty amazing computers but i don't think these cars have to get so good to be better than us because we have so many failure rates already the failure rate's very high that being said it does seem fraught they've it's always been it's 18 months around the corner for the last 10 years.

56:29There's progress being made. I mean, Waymo is opening in new cities. It seems like maybe a city-by-city rollout. And I believe they have LiDAR involved as well, is working. They also, I think they have a call center, I believe. And I could be spreading fake news, but I think they have actual humans, not in the loop, but available in order to take over a car if need be. That seems like a good fail-safe. there's progress being made but it does seem like I don't want to call it an impossibility but at this point I just don't feel like we're close Can we go back to my idea of this closed system? Like, is that even a possibility?

57:09Like, to your point, Jared, like, can we create a closed system? Freeways and trucks. Yeah. Self-driving on freeways and trucks, I think, is a better possibility. Freeways have a much smaller number of things that can go wrong. And it is more of, it's not a closed system. But it is a differently constrained system. Right. And a smaller problem set. I mean, if they had their own roads, it would be much simpler than it would be truly closed. And in the meantime, I think we can get closer on freeways more than anything. I mean, there have been a lot of, most of the new cars now, we'll have lane keeping and keeping up with the person in front of you.

57:56all of the things that the initial full self-driving Tesla did, we can do now in my new Bolt. It is progressing, whether or not we call it self-drive or if we just call it new safety features. Yeah. I think it's getting there. And I like the idea of a closed system, but at some point you do go back to being a train. And so you're laying track. Or laying pavement that can only be used by this form of self-driving. Right. Some sort of guardrail, parameters, boundaries, something that says this is where you should be. And if you're not here, then you can no longer operate or something changes to no longer be the danger possibility, so to speak.

58:45We'll put sensors in the roads. We'll have it monitor the car in front of you. and humidity conditions. And then the sensors in the roads will talk to the cars. Wait, wait, let me scribble this down. I need to invest in a smart rock company because we're really selling a lot of smart rocks. Yeah. Well, as we're knee deep in this conversation, I was just thinking, you know, one of the most probably globally known, most people have bought one at least one point in their life or want one, an embeddable system is like a GoPro, for example. Like that dude who founded that company was just surfing. It's like, man, I would love to film this.

59:27And he went away and did his thing and now GoPro exists. And I'm just thinking like, wow, this must be a proving ground for not just innovation, but potential future founders. Like this seems like a good place to go and not so much get rich, but like change the world in dramatic ways. I think GoPro has changed the world in dramatic ways. is giving you, let's just say, a camera view, obviously, of something dangerous, but then you have that GoPro effect. Like, oh, that doesn't look that steep until you get there. You're like, that's pretty steep. You know, like that's, this embeddable system world seems like if you want to go and invent something, this is a place to go and invent something because it's real, it's physical.

1:00:13Yes. There are lots of times when it's like, I have a problem. I know how to fix it because I know how to wire things together. But I also have that same feeling in software that sometimes I can invent something just pure software. I do origami and I write Python scripts. I write very long and complicated NumPy scripts for doing origami. And I love that. That's crazy. What do the scripts do for you? They create crease patterns, which I can then use a laser cutter or Cricut or other device to pre-score paper. Then I can fold along wavy lines, which will give me origami that is not flat, that is patterned and weird and wavy and organic methods.

1:01:10Does this exist in the world? Can you do that? Can you go buy that product right now? No. Well, then you just invented something. Do you, I mean, there's curved origami out there, yes. Yes, there are other people who do curved origami. It's just a world that I've not been introduced to, so I don't know what exists. Yeah. But yours is custom curved origami that you want to make. Yes. And so your scripts give you the crease lines. The scripts give me the crease lines. and they let me combine straights and craces in order to get tucks and weaves. And I will go off the rails here. Go off the rails.

1:01:54No, you're allowed. We are with you. I'm listening. Adam was saying how you can invent things. And I think you can invent things in software too. And it's some of the same impetus that you invent them because you want them. I had a stuffed animal that would Wi-Fi through the cloud to another stuffed animal, and when you patted it, the other stuffed animal would light up. And you could have this sort of communication to someone else. What would the other animal do? I would pat mine, and theirs would light up. They would pat theirs, and mine would light up. I see. So there's like a. It was like a good night.

1:02:40Yeah. For maybe for like brothers, you know, they're in different rooms. Like, good night, brother. Kind of thing. Not a walkie talkie. It's more like I'm betting mine, you're betting yours. You just know that somebody else is there doing something as a connection. Because it's a two person system. Yeah. It's a peer to peer with two peers. Well, this actually went through the cloud. So it was more like you're in college and you're telling your parents you're going to bed. Yeah. That's cool. Yeah. All kinds of just, yeah. Two people connecting quietly. I just like, I think I'm liking this idea more now that we're talking about.

1:03:09I mean, I've always liked this idea, but this idea of software paired with hardware that creates these newer experiences that isn't software only. Like an iPhone or a computer screen or a browser software, which is like, maybe that's the kind of software I'm thinking of. But then I'm also thinking like, not just that GoPro, but I use. So if you didn't know this, Alicia, I like to barbecue. And so I own a Traeger. a trigger is a smoker but it's basically a computer it's an embedded system right somebody wrote software to have sensors that says how much how many pellets should i push through to maintain this temperature what temperature is it in the ambient what a temperature is it at the burner you know this this thing that i just push the button turn the dial set it to 275 and wait an hour or two that's a computer an embedded system that somebody invented because they were like, well, doing it with the old, you know, wood method was too cumbersome.

1:04:08It wasn't ubiquitous enough because too many people didn't want to tend to fire. And so now triggers exist as a smoker to give way more people access to push button oven light control of a barbecue system. Exactly. That's crazy. I never really thought about this trigger as this embedded system until this conversation. Obviously it is, but like I'd never thought about the fact that I could have made that. You could have made it. If the things were more accessible. And I think that's a lot of this conversation, like how accessible is this technology to people, to lay folks like Jared and I, or we're seasoned in software development, but not in embedded systems necessarily.

1:04:45But with an Arduino or one of these little MicroPython, CircuitPython boards, think about what you would need to do. You would, you need to, okay, so like mentally build the system architecture. You need to push pellets. You need to be able to tell time. And you need to monitor temperature. If you went online and looked up, how do I get a thermistor? How do I tell temperature? Really easy. That would take you three minutes. It's wiring. It's going to be easy. Uh, powering. It's easy doing the timing. Um, a little bit of research on that will probably turn up that, you know, you don't care whether you're doing two hours in one minute or one hour, 58 minutes.

1:05:42So you don't have a big, uh, a constraint between really good time versus kind of crummy time either way as long as you're around two hours. And now you have to figure out how to push the pellets in in a way that's probably metal because you don't want it to burn in plastic. You're not going to 3D print this. And now you need a motor subsystem. And that's where things get a little complicated, but motors are very common and you can buy the kit that just says, I want you to move here and I want you to move there. And so you push the pellets in. Maybe, maybe you have another door closer thing. Now you're getting mechanical and I do get intimidated by mechanical, but you have a unit that works.

1:06:29You can start to look at it and see, how does this work? What are the mechanical pieces, then I need to be able to push and close doors and scoop pellets. And it becomes, you want to build this. So you build each piece and then they all work together and then it's done. And then you want to do further more update and you call me. Yeah. Yes. Love it. It seems like the, you said you get intimidated by mechanical and it seems like the pusher would be the hardest part of that. Maybe you could utilize gravity to just have like, you know, an opener and shutter versus a pusher and then you're dropping the pellets in.

1:07:11I'm not sure how triggers work. Well, I know exactly how it works. I'll tell you. So there's a hopper. Yeah. Right. And the hopper, you fill it up. It uses gravity to, as you know, drops the pellets. Yeah. As the pellets fall down, it falls into this space that there's an auger and the auger just turns. And so it slowly just moves the pellets along the system. And so you fine tune the temperature within the firebox based on the speed of the auger. So if you slow down the auger, no pellets feed it. If you speed it up, it speeds it faster. And you probably have some, maybe an algorithm or machine learning of like how the speed of the auger versus temperature control.

1:07:51And you're obviously maintaining temperature, but it's just literally a turning mechanism. It's not even smart, aside from knowing speed and temperature and how speed affects temperature. And the hopper is just the pellets are falling based upon gravity. So gravity did the work there. Thank you. You know, no pusher necessary. And there is a sensor inside the hopper that will tell me if it's low, but I'm smart enough to know how much I put in there. I know when it's low. And my cook's never that long where I need to literally monitor my ever. I've never had to know if it's low. I think it's just a gimmick, really.

1:08:25in terms of time on the timer i mean trigger does have an iphone app and an android app that controls it and you can set time and stuff like that and you can tell it to turn off but that's a that's a rich person's feature like not everybody needs that you really just need the ability to maintain a temperature for a sustained amount of time like every any barbecue person will tell you all i'm teaching is time and temperature i'm not teaching you anything special about cooking food just time how much time do you need and at what temperature So that's the recipe for cooking is like, you know, time and temperature.

1:08:58And so the trigger just basically does those two things. It's time and temperature. And that mechanical is so much smarter than what I was coming up with.

1:09:14Okay, friends, have you ever been doxxed on the internet? Maybe you've been stalked. Maybe you've been harassed or you've had your identity taken from you. Privacy matters so much to everyone, obviously. But have you ever wondered how much of your personal data is really out there on the internet for anyone to see? There's more than you think. Your name, your contact info, social security number, what unlocks your credit history, your home address, or even information about your family. That's terrifying. All this data is being compiled by data brokers and is being sold online. So Delete Me is one of our sponsors and they gave me an account to use.

1:09:53I tried it and it's helping me remove my personal information from hundreds of data brokers out there. Here's how it works. You sign up, you provide them with exactly what information you want deleted and their experts take from there. They send you regular personalized privacy reports showing you what information they found, where they found it and what they removed. And Delete Me isn't just a one time service. it's always working for you, constantly monitoring and removing the personal information that you don't want on the internet. To put it simply, Delete Me does all the hard work. It wipes your information, your family's personal information from these data broker websites and gives you peace of mind.

1:10:35So take control of your personal data and keep your private life private by signing up for Delete Me now at a special discount for our listeners. Today, you can get 20 % off your Delete Me plan by texting CHANGELOG to 64000. Again, text CHANGELOG to 64000. Of course, message and data rates may apply. Check their terms for more details. Enjoy.

1:11:09so adam what do you want to build next uh geez let's see birthday cards birthday cards say more yes my family is a fan of singing birthday cards and my son loves to take them apart later i'll tell you a story so my son is curious like this like he loves to take things apart and figure out how they work. And so he took apart a birthday car. We got him and obviously you open it up and it starts to play the song or whatever it's supposed to do. Well, he decided to take it out of the back of the package, discover the sensor and discover this thing in this mechanism. And he decided to pull the wires out and put them in different places.

1:11:46He made this birthday car do things it wasn't intended to do for us. It was just meant to be open, do the thing. And it's a closed system in that regard. But he found out you can pull the sensor out or whatever the mechanism is and rewire it and he made it do something different. He's like, Dad, look at this. I'm like, what is that? He's like, it was my birthday card. You know, that's kind of cool. Okay. Chibitronics. C-H-I-B-I Tronics. This is away from the software, although they have little microcontrollers it works with. So we can get to the music and whatnot. But what they've done is they sell these kits that have copper tape and you copper tape around and you can make little buttons by having pieces of paper touch each other.

1:12:38You can make little sensors by having patterns of copper tape do different things. They sell stickers that are LEDs. So you put your copper tape up to one side and your copper tape to the other and then you put it to power and you have a little battery usually um the the clips um what are they called uh what would you call a clip like this nope that's binder clip oh yeah put a binder clip with the battery and you make your own card like that and it lights up or or you can have it do music. They have all of these different neat sensors. And this is more on the electronic side. This is how people understand how do you go from battery through a circuit.

1:13:30And you don't need to know Ohm's law and all of that, Kirchhoff's laws. That's not that important for this. It's really about you follow the copper tape to the different pieces and you make up, You make neat, lit things, and you can add a controller, which can then even add a little tiny motor, which can either vibrate or it can move like you do a paper engineering cutout of a butterfly, and it will gently wag the wings. Yes, Chibitronics. They're super fun. Chibitronics. They were directed towards women at one point, getting girls interested in storytelling through electronics. But I don't think they are as gender specific anymore because I've been seeing lots of their new sensors and new systems being a little louder.

1:14:30Okay, since we're here then, I have to ask you, I'll just say one brand and I want a reaction from you, Alicia. Crunch Labs. Nothing. I'm terrible. Oh, my gosh. Nothing. Well, then. That's great. So I don't know his name. I'm going to Google it real fast. Crunch Labs, the fellow that runs this, this is a very massive YouTube channel. But it's not a YouTuber, like a, I don't even know how to describe it. Oh, I've seen this guy. Oh, yeah. It's the coolest stuff ever. This guy used to work at NASA. And he is all in STEM and all kids focused. But he has his subscription box that you can get to, like, build different stuff.

1:15:09and as you're describing this how do I pronounce this Chibitronics I'm thinking like wow here's more kits that entertain and educate these yeah Mark Rober thank you Jared yeah everybody loves this guy yeah I love Mark Rober he's the coolest I mean he's I don't even know how he does what he does he's amazing and he's it's like he's like living his purpose for sure but my son loves Crunch Labs is so entertained by all this stuff has become smarter because of Mark Rober I mean like education educational curiosity stem embedded systems i'm sure electronics rube goldberg things etc like just all this cool stuff so i thought you might know crunch labs i don't but i'm gonna sign myself up i love the little robots and they had a crossbow how do i get the crossbow how do i get the crossbow that's that's great and these kits they come on this like frequently so i was thinking what made me think about crunch labs was is the monetary accessibility of of the uh chibitronics and and you know building these things that no one else seems to be doing to i mean at some point you you got you got to want more people in your field right you're probably lonely to some degree in your field.

1:16:32You want more folks who have the same drive and curiosity as you do. And the only way you get there is by inviting them in. Yes. And my field was so lucky to have Arduino because it really drew a lot of folks to be interested, both in electronics and embedded systems, but also in software. her. And there are Chibitronics and some of these other robotics kits. First Robotics is just wonderful. That's the competition for middle and high school students. And what the kids learn is just, I just want to hire all of the First Robotics engineers. Because once they get into the robotics and they start understanding how the mechanical and electrical and software all work together to make the robot move.

1:17:33They get a systems understanding that's really hard to teach. We've gotten so many amazing links from this conversation. Are there others? If we're like, Alicia, what's another cool thing you just showed us? It should be Tronix first. this first inspires.org is the first robots group people matt or matt adam pointed us to crunch labs of course we have micropython yeah adafruit the walk we what else you got um hackster.io instructables.com and hackaday.io oh hackaday i'm well aware of what was the first one you said Yeah. Hackster.io. Hackster. And instructables.com. And I'm grouping these all together.

1:18:22Even though they have very different flavors, they are all places you go and tell people about your projects. Nice. As an engineer, it's like, okay, I'm learning to document. This is why I tell people to use these sites because I want them to learn how to document and reproduce their projects. but as somebody who's doing it, as somebody who's excited about it, it's getting to show off what you did. We don't always get to go to these hackathons or to these hack meetups where you get to bring your project, but this lets you connect to people who are maybe in the same boat. Maybe there's a competition for Nordic DevKit and so everybody's looking at the same set of sensors and you get to see what other people's ideas are.

1:19:09But you also, you know, they all have community aspects where you can talk about what you're doing and talk about what you want to be doing. As well as demonstrations where other people have said, this is what I did and these are the steps so you can reproduce it in case they have a neat project. Love it. Where's this going? You know, like this kind of thing. Showing off is kind of cool, right? I'm looking at this one here on Instructables in the circus section because it lets you see things like crafting and cooking and living. So there's more than just simply this embedded world there. It's a sleep cycle lamp.

1:19:47And I'm seeing like this lamp with a common shade, probably a similar or sorry, a very small board beneath it that's powering this thing and probably basing it on time is doing different sleep cycles, colors, tones, etc. But I'm thinking like, man, where could this world go where you have infinitely accessible Lego-like things? Not so much Lego itself, but how you can take this part, pair with that part, and make a product. Or create something new or invent something new. Where can this go? Where's the future going for this? Like you have this idea and you want to market it? No, no, no. More like you.

1:20:25Where do you think, I mean, given your knowledge of embedded systems and where you've seen it go, what's next for, I suppose, this world of, from the simple hobby project to underwater things with sensors looking at dirt moving around? You know what I mean? Like, it's such a chasm, but like, where is this going? It's a big question, I'm sure. I mean, it's going everywhere. I fought against the term internet of things for so long. But that's what it is. Like everything wants to be on the internet, wants to help you be better or do something different. And I don't love all that. I mean, I don't need everything I have to be on the internet.

1:21:11but like having two stuffed animals essentially on the internet talking to each other over distance it turns out to be kind of cool the applications turn out to be really interesting the ability of my doctor to see if I'm using some medical device they've prescribed okay that gives them more tools if it's not okay with me then i make it not okay but it gives other people the opportunity to engage with us if we want to or to monitor elderly folks or to teach kids to read or adam i have to say it's going everywhere and I'm not sure that's the best thing but it's also going into all kinds of science.

1:22:13One of the projects that I worked on a year ago two years ago that's going out this year is they're just going to chuck these boxes into various rivers around the country and be able to monitor the environmental DNA so they would know if a river moved into or if an otter moved into this river before anybody could see if they were actually there yet and they can also monitor things going wrong like human DNA being in the water too much is bad I the internet of things that's where we're headed and it may not be great but some things will be great so embrace it right Right. You know, you've you've lamented your your fight against IoT, but just embrace it, basically.

1:23:09Be aware of your security. Be smart. You don't let strangers into your house just willy nilly. Yeah. Maybe look at the names of the companies you're letting into your house and make sure that they have, you know, that they have nice friends and good references and all of that. because you don't want to create bot networks or allow people to spy on you. So when you do engage with the Internet of Things, do so wisely. And it's not just people in their houses. It's companies and UAVs. You need to know where you get your UAV from because some of them report back to their home countries. And that's kind of scary, depending on what you're doing.

1:24:01If you're using it to take pictures of the beach, who cares? If you're using it to map your new mining prospect, I don't know if you care. But you should think about it. Security is, unfortunately, the downside of IoT. It's also the S in IoT, which is to say, you really need to think about it for yourself. And privacy too, which those two are, you know, kind of eternally linked, aren't they now? Because being able to throw a smart rock into a river and monitor things is awesome. But being able to throw it into my pool and monitor my swimming, maybe not so much, just for instance, you know? Yes, there are a lot of downsides to this.

1:24:47the internet of things and also for IOT makers out there you know just leave us physical buttons you know like give us let us let us opt out every once in a while and just use the dishwasher without having to launch an app for instance for instance yes I take your dishwasher requires you to use an application or something Jared? no that was just a for instance I'm sure somebody's does yeah well i i have lg washer and dryers and i can tell you when my washer is done and when dryer is done and i can i can tell it to start drying something right now if it's on so that one safety security and safety maybe the similar s's is that i can't control it from afar or remotely if it's not on currently i can't power it on and power it off i can only instruct it if it's been previously human touched like recently kind of thing and would you have paid for those features apparently i did i know we did too but i would never meant to pay for those features no i never want almost any like sometimes you're like oh this actually is kind of nice once you start using the feature you're like yeah i kind of appreciate this but it was never the dryer's off it's kind of nice yeah it never has sold me on any piece of appliance so the one thing i would say about that that I've appreciated because I try to be not the person who only complains.

1:26:12I try to be the person who appreciates pretty much any scenario. I do my best. I'm always the greatest at it. It's a different interface to a simple interface that doesn't make any sense. And so the iOS application actually makes a bit more sense to use its advanced features like the dry clean version. There's a dry clean setting on the dryer, for example. It hooks to the water and it can humidify the air kind of thing. It can do some cool stuff. That feature is not easily accessible on the device itself. I mean, you can. It's just harder to map your brain to like, okay, push button go. Whereas the iOS app is a bit more easier to use.

1:26:53I think of like a different interface to this thing. The notification is, I turn that off in most cases. I actually don't have it on unless I'm washing clothes personally. And when I say that, I'm like, I don't usually wash our clothes. So I'm that person who gets to just sit back and into life, for the most part. You better not be complaining. Until I have no more underwear, and then I'm like, oh my gosh, I have to wash my own clothes now. And you're checking the status at all times. That's right. Notifications on. Thank you. I mean, the app does give you more information than could possibly fit on the front of your washing machine.

1:27:28It does. Yeah, so that's how I look at it. It's like, I wouldn't have paid for that feature. I think I did because I bought it, obviously. I didn't buy it because of that feature. And I'd say that it's a nice to have, not a need to have, but it is nice to actually have because I've used some of the advanced features via the iOS app. And it's just a different way to interface with the device that's actually better and easier. But, you know, those bits of information that they're showing to you can also just seep out of your local area network and into the hands of somebody. Adam is washing his clothes again.

1:28:03Get him. you know someday you're gonna get an email from a hacker or anything like adam you're out of underwear yeah listen hey listen you probably should wash your clothes it's been about two months since your dryer's drying thing yeah clearly you're out of underwear at this point i've checked your your walmart checklist for your underwear or wherever you buy your i don't buy my underwear from walmart but i just said that tmi dude just so you just so you know okay oh all right alicia i assume the website is embedded.fm where else can folks connect with you you can get embedded from any of your regular podcast sites um and if you are faced with one that is embed in all black and looks scary don't do the one with the robot look for the one with a little robot head robot head slash radio old-timey radio so that that's us nice you can get my book.

1:28:59It's called Making Embedded Systems. And it's in second edition. It's from O 'Reilly. It's on normal Amazon and bookshop sorts of places. If you're interested in making embedded systems, the goal is for hardware engineers to understand more about software and for software engineers to understand more about hardware and how to design and make embedded systems. If you want to contact me for consulting, I think the answer to that is please don't. Please don't. Please don't? Is that right? We're booked. We're booked for like a year. It's great. And yet I always love to hear about neat things people are making.

1:29:44So if you want to tell me about your embedded system, hit the contact link on embedded.fm or email me at show at embedded.fm. And I'm just fascinated by the technologies coming out. Very cool. Any conferences or like IRL places? I know you mentioned several digital ones, which we've definitely collected and we'll put in the show notes. But is there anywhere that's like common? like is there an embedded conf or something like that where it's you know international or accessible to folks like the one you can't miss kind of event yeah like you must go there if you're really getting into it you must go to this conference kind of thing this is another one of those areas where there is a good hacker subset and there's a professional subset and they don't quite overlap I will be keynoting the embedded online conference in May of 2025.

1:30:44Please don't ask me what my talk is about. I don't know yet, but I will really soon. Embedded things. Probably sensors. And for the hobbyist or... See, it isn't just hobbyist. I've gone to Supercon from Hackaday. I would happily go to Teardown. There's open Hardware Summit. There are a ton of really fun places where people will show you how to get into embedded systems or hacking devices or making cool things. There aren't as many professional conferences unless they're about certain topics like Zephyr or security. Hardware.io is one of the international conferences that's pretty popular and more security based.

1:31:44I can't think of any other ones, but I know I will when we get off. So I'll send you email and you can provide links. For sure. I know IRL is important to folks. We can obviously shell out lots of digital locations to hang, but IRL, can't beat that, right? Real humans, real people, real fun. Really? I think so. Really? Maybe not. Maybe not. To each their own, as they say. I'm kind of an indoor cat. We'll have to get some sensors for that and maybe a motor to judge it. There you go. Or one of those things you can touch that we know, like, Alicia is still there. She can just send a sensor to the conf.

1:32:26Still here. Like light up a light. You know, blinking lights. We hear you like those. Yeah, for sure. As long as they're blinking for other people. Well, Lisa, thank you for your curiosity too. I know that like sometimes you're going alone, right? Sometimes you're just in the dark, but now you're not. Obviously, you're book solid with your business. You've written books. You've come on this podcast and you're doing cool stuff. So thank you for sharing all that with us. I appreciate it. Thank you for having me. Your podcast is a lot of fun. Well, thank you. Well, the world of embedded software is going everywhere.

1:33:05Traggers, GoPros, birthday cards, smart rocks. I mean, the possibilities seem to be literally endless. And so maybe this is for you. Maybe this is your new field. Maybe this is your invitation into the world of embedded. And on that note, there's a ton of links in the show notes. So make sure you go check those out. Wookie, Crunch Labs, Chibitronics, so much to explore. And I think you should do it. So there's your push. And if you haven't yet, make sure you check out Alicia's podcast, Embedded.fm. I'm a listener and you should check it out. Of course, a big thank you to our friends and our sponsors for this episode today, Augment Code.

1:33:45Check them out, augmentcode.com. This is the developer AI for teams, large code bases, the ask me anything out there. So level up your copilot and go with augmentcode. Check them out, augmentcode.com. And to our friends over at Fly, you know we love Fly. Fly.io, the developer cloud for everyone, for those who ship like you, like me, fly.io. And of course to our friends over at Delete Me, text changelog to 64000. Get 20 % off and delete your information off the internet. Again, text changelog to 64000. And a big thank you, of course, to Breakmaster Cylinder for those awesome beats, those banging beats.

1:34:36We got some new beats planned for this year, 2025. Stay tuned, of course. but in the meantime thank you bmc you're awesome okay we are now shipping full-length episodes chaptered and all to youtube so if you've ever want to see the behind the scenes the the never saw before the full-length not a clip but a full-length show on youtube our new workflow is video first and we have a lot that we're exploring this year on youtube and through this video first flow we have going on. So explore with us. Join us. YouTube.com slash changelog. And yes, CPU.fm is in full effect. We have things happening behind the scenes.

1:35:23They're not on the.fm yet, but soon. Very, very soon. We'll have a brand new super feed for you to subscribe to. And there you go. That's all I'm going to say. That's it. Okay. This show's done. We'll see you on Friday.

1:35:53Have you been tempted to build any sort of systems for your pets or your animals? To know where they're at or to warm them up in their beds kind of thing? Have you ever done anything like that? Like personal projects for your... Not for the pets. She's got an air tag on her, even though she's a tiny dog and the air tag is like half her weight.

1:36:18Heaters, no, because I'm really neurotic about setting the house on fire. But what if it was water cooled or water warmed? Warm water. Yeah. Smart rocks. I'm just, I never got over that. I'm like, he just said smart rock. And I'm like, that's new to me. You're dumb as a rock. No, you're as smart as a rock. I know. It's like a whole new world where rocks are smart. Cats and dogs living together. I'm inspired by this. I mean, I feel like we should quit podcasting, Jordan. Just build some smart rocks, man. Or something. I don't know. See well enough. I'm still entertained by that desire from you.

1:36:57You wanted to put a sensor in your freezer. Remember that? Yes. And I think you ended up doing that. Did you end up doing that? No. No. I forgot about it. But that'd be so cool. Oh, yeah. I have the, I still have the pie. And I was using it for something. I was using it as an, what's that thing called? The blocker pie? The pie hole. Yeah, I was using it as a pie hole. And then I quit using that. And I'm using NextDNS instead. And so the pie is just sitting down there. And it's relatively close to the freezer. I just want to make sure, Alicia, we have, we buy like half a cow once a year kind of a thing.

1:37:32And like, that's very.

1:37:36Thank you.