Securing ecommerce: "It's complicated" (Interview)

30:41And we provide a predefined set of UI elements that fit into the checkout UI and work with all the branding primitives. But we do that work on your behalf. And the critical part is, because we control the bridge between the web worker and the top-level page, we have tight control over what kind of mutations can be pushed between the parent and the isolator worker. So you can't just arbitrarily inject JavaScript or perform unsafe operations on the parent page. So in that way, we can take any third-party script, put it into a sandbox and say, you know what? You can do whatever the heck you want in that environment because you can load a transit of chain of other dependencies.

31:25We don't particularly care because all we know that the operations that you can pass back to the parent page are safe and approved set that we will allow. And we also control what data is exposed to you. So for example, if you have an extension that wants access to some sensitive buyer data, first that application and then the worker itself needs to have the right consent. So a worker that has not been granted the right consent by the merchant or the buyer will just not have access to that data. So that is our solution for extensibility and allows us to partition the problem of first-party and third-party content.

32:07It's based on remote DOM. And then we use the same technology for our pixels or analytics as well, where we define an event bus, we emit all the events, analytics providers are executed in the sandbox as well. Is that a compromise in terms of functionality? Do you get 100 % of what you could do before in terms of what you all are providing? Or is it like a, are you constraining people through and losing features along the way? Yeah, you asked exactly the right question. So So the answer is we've had to rebuild a lot of stuff because a web worker, if you're familiar, is not the same thing as working in the top level page.

32:44It doesn't give you access to the DOM. It doesn't expose all the same events. So the reason it took us as long as it did to layer all of this infrastructure is because we had to work with partners and replicate. So what do you actually need? Instead of raw access to the DOM tree, what are you looking for? For example, if you were building a heat map solution, as an example. Some of our merchants are really keen on having very visual, clear understanding of how users are behaving on their checkout page. You need access to a lot of different events and elements. Okay, well, let's work through that and figure out what is the right subset that we can expose via this bridge.

33:21So over time, we've built up a collection of these APIs and primitives, some of which effectively replicate what is available on the parent page. one of the challenges here by the way is like if you ever work with web workers is they use asynchronous communication right so you have to post message between a web worker and the top level page whereas a lot of the dom apis are synchronous apis so if you're just naively writing code expecting to be executed on top level page you would use synchronous apis so we've had to shim some of that and like in places we try to keep it as close as we can to what you would expect as a developer because we don't want to impose additional friction.

34:01But in certain places, we had to provide replacement APIs where we said, look, you're building for Shopify. It will operate across scale of millions of merchants if you're building an application. It is worth for you to do this extra step because then you have all these insurances in place. So a lot of handholding with partners and getting the developers to adopt all of those APIs. But the benefit of all of that work today is, I'm not going to say we're done because there's still more things to build, but we're in a really good place because now all of our merchants are running on the sandboxed primitive that I've described.

34:37And what we can provide is, first of all, upgrade safety. We can safely roll forward our capabilities in checkout, knowing that customizations that you've deployed will not break as you move forward, right? Because we control the bridge, we control the API interface. So if we change the underlying API on our site, we can still provide guarantees about that. We have reliability. We know that, for example, we saw examples where merchants would inject scripts, where a partner would just time out. So they would have some logic and for some reason their service goes down and then the checkout is broken because, well, it's just waiting to render, right?

35:16Like Shopify, you've broken the checkout. It's like, actually, it's your part. It's your script that you injected of a partner that failed to scale to your flash sale. Right? So now we have assurances about that. And then finally, performance and security. Another benefit of putting work into the sandbox is it moves all the work off the main thread. So you can't have code that monopolizes the main thread and renders the UI unresponsive, which gets back to our web vitals conversation, right? Like we can make better performance guarantees about how the page is loaded, how responsive it is, and the rest.

35:51And finally, there's a security bit, which is we know that you can't inject arbitrary content in top-level page and exfiltrate data. And then finally you have PCI compliance, because now we have a clean partition where we say, as Shopify, as a platform, we will provide all of the inventory, authorization, and integrity checks for the first-party scripts that are executed on top-level page. And, oh, by the way, you can totally bring third-party content, but we will execute it in this isolated context that allows us to punt that problem and not have to worry about all of the integrity problems that happen when you just include it on top-level page.

36:32so did you did i hear you right that you said all your merchants are already using this you're able to to deploy that without or did you not say all yes yeah yeah yes all yeah so this has been a long journey to move all of our merchants onto this new platform but as of earlier this year like 99.9 percent of our merchants are on this platform there may be like one or two exceptions but effectively any shopify checkout that you shopify power checkout that you visit today as consumer is running on this infrastructure. Hmm. And that was something that they had to opt into or that you just did on it.

37:07Like, how did you all roll that out? You said it took a long time, but what was, what did it look like? Well, it took a long time because of the right question that you asked, which is, Hey, did you, what, what did you have to take away? Right. And the answer is we had to rebuild a lot of the capabilities because we've created this isolated environment. We've had to recreate a lot of APIs. So a lot of our work was working with other developers, partners who provide capabilities that merchants want in checkout to make sure that they can bring the same capabilities into this new world of sandbox execution.

37:41That was the long haul. And then for some merchants that had ability to manipulate content in the top level page, it was a combination of documentation, handholding, consulting, and just getting them to move to the new world so they can benefit from all of these capabilities. But we're there and the time is right because now you have PCI v4 compliance effectively taking care for you. And do you think that PCI v4 compliance means you cannot be skimmed in the way that you could prior? Or do you think it could still happen in new and exciting ways? Right, right. So I think this actually is another layer that we should add here.

38:29What I've described is runtime compliance or runtime guarantees. So the thing that we've built actually allows us to provide assurance or extend some guarantees over, we just know that it's not possible to inject third-party content. So if you have a supply chain attack on that, it's isolated into a thing that doesn't matter. In practice, I think what a lot of other players and e-commerce providers will end up using or how they will provide compliance is retroactive monitoring. So PCI does not enforce a requirement that you have to have runtime guarantees. What it says is, hey, you should have a process that provides an inventory, make sure that scripts are authorized and you have the integrity.

39:18It doesn't specify that it needs to be guaranteed. So practically, how could you implement this? and how to most, like if you go and search for PCI compliance security products, you will find plenty that will basically say, hey, I know a great solution for your PCI problem. You know what it is? Deploy my JavaScript into your page. Right. Because more JavaScript is always a solution, right? And I will instrument the page and listen for all the things that are happening. I will observe all the other scripts. I will build an inventory. I will monitor if it changes. I will try to provide hashes and effectively I'll like, you can delegate this problem to me.

39:59Now you can see a flaw in that reasoning, right? It's like, how do you know that your script is not going to get compromised either? Who watches the watchers? Well, there's that. And how do you know that the malicious thing doesn't come up with a clever way to obfuscate itself from you, right? It's the antivirus problem, like virus hiding. Exactly. Virus hiding from the antivirus problem. But that is likely a solution that many will adopt as a retroactive solution. So effectively, you observe if anything has changed. It's like, oh, well, that's odd. I'm seeing a set of reports for a script that I did not expect relative to my inventory as I defined.

40:42Does that indicate that I have a problem on my site? Probably, right? So there's some guardrails that PCI sets for how long that period can be and how you need to react to it. but it is strictly lesser and less secure experience, which gets back to your question. If you have these assurances, does it mean that the class of attacks is eliminated? The answer is it depends on how you implement it. So in our case as Shopify, I would feel pretty strongly about extending a promise of like, yeah, unless our content, first party content is compromised, it would be very hard to compromise this page. Now, what we can control as Shopify is the buyer has installed a browser extension that injects arbitrary scripts into the page.

41:30That is outside of our control because that operates at a higher layer. Or maybe you even have malware on your computer that does things and injects content into the page or otherwise intercepts what you're typing. Those things are still possible. It is not a completely eliminated this type of attack. but it certainly makes it a heck of a lot harder because now it means that at least there at a minimum there's a way the merchants are required to detect these changes or these attacks and remediate so they can't just go unnoticed so this all sounds like a lot of really good work you all have done at shopify for shopify and Shopify's customers, thinking bigger, it would be great if your hard work and years of rethinking this runtime and sandboxing and actually providing the security that PCI v4 wants everybody to have, whether or not they do or not to be compliant, can't some of that get into the browser?

42:37Like, couldn't we just build it? Could your work extend beyond Shopify's borders? and help other people too. Yeah, absolutely. And I think this is key. This is not just about Shopify. It's about improving the buyer experience on the web holistically. So two things to answer that. First of all, the remote DOM library that I mentioned, it's an open source project that we've built and have open source. So if you go to github.com slash Shopify slash remote DOM, you'll find that there. Take a look at it, use it. That's the technology that powers Shopify checkout. Other large companies have already adopted it.

43:12I believe Stripe is using it for their apps. Actually, fun story. When we built the project, I think Stripe beat us to using it in a production product. Really? Even though we were the ones developing it for checkout. Good for them. But it is used at Shopify and by other big players to provide this form of isolation. And the general pattern is, hey, I have a trusted first-party surface into which I want to bring in third-party content. And I don't want to compromise integrity of my first party top level service. Well, remote DOM is one of the technical solutions for that. So please take a look at that.

43:51That's answer number one. Second, though, and coming back to the browser conversation, absolutely. The primitives that we have in browsers today, content security policy and SRI, we can make better. And we've actually done a bunch of work on exactly that at Shopify. We don't want to do work in JavaScript that we could push into the browser because the browser is just much more efficient and it has capabilities that otherwise would be very hard for us to replicate. So first, let's enumerate some trivial examples of gaps. Script integrity. So sub-resource integrity. For those not familiar, on your script tag, you can pass in effectively a hash.

44:31So when you inject the tag into your HTML, you can pass in a hash that is a fingerprint. And when the browser loads the script before it executes it, it can compare the hash of the thing that it fetched versus what you've defined and say, hey, if those two things match, great, I will execute the script. Otherwise, I'm going to raise a violation and not report this. That's a big capability that exists in browsers today. It's not simple to deploy, but it is doable, right? Because you need to figure out how do I get these hashes and how do I inject them at the right place? but then one of the gaps that existed for a long time was module imports so sri worked for top level scripts but if you're building a javascript application and you're using an you know import you just could not pass in an integrity hash why well because module imports came after sub-resource integrity was designed so it was just never it was never a thing that was a pain point for us because we use module imports at shopify so we worked with Chrome and Safari to upstream some patches to get that supported for module imports.

45:38So the good news is that's now baked in, I believe, as of May of 2024, I think when Safari shipped it in their release, both Chrome and Safari support SRI for module imports. So that's one. Another thing that came up in our thinking when we were exploring CSP compliance and how do we make our own life simpler is this idea of require SRI for. So what if you could express a content security policy that says, hey, all scripts must have an SRI or integrity hash. Gotcha. Right. And why is that useful? Well, then you can make a strong claim that if you have that policy being enforced by the browser, then if for some reason you sneak through by accident or malicious act, a script that doesn't have it, they would just be rejected, right?

46:32Which today it would just execute normally without any questions. And even though that might be hard to deploy in an enforcement mode, it could totally work and be really useful in a report-only mode. So for those not familiar with content security policy, you have an enforcement mode and a report-only mode where you can get violations, which is incredibly useful because you could say, hey, this is a policy I would like to enforce. What are the violations? So with the require SRI 4, you could deploy this in report-only mode and say, great, now I'm going to get reliable reports from the browser, from the wild, for any time a browser detects that a script is missing an SRI capability, right?

47:17This is great because sophisticated attackers would not emit these scripts on every single page load. They might target specific users or a class of user, or maybe they target a specific browser, or maybe if it's an extension, it'll apply some sort of other heuristic, right? It's very hard to, this kind of mirrors our conversation on why ROM is important, real user measurement. Gathering violation reports from real users gives you a much better and reliable signal for where the problems are. so require sri 4 is another capability that we've shipped into chrome and that allows you to get violations on missing sri attributes which allows you to build an inventory of like this is the list for me to burn down and figure out why right and if anything changed how should i react to it yeah another example is okay great now we have these reports coming in wouldn't it be nice if we could also get the hash of the content right it's today you would just get a report saying hey i detected script from example.com slash xyz.js but what was the content of that you don't know right wouldn't it be nice if you could also get a hash in the report such that you could audit it and say oh well maybe that's totally okay because the partner revved their version and it just happens to be the v2 i just put that i put that into my approved list and everything's fine versus i have no idea if that was a compromised version or a legitimate version of the script interesting so pardon my ignorance for a moment but where does the reporting take place or post to the browsers doing the reporting is it correct who gets the report and how is it the browser sends it off somewhere or yep so on the wire um you would when you emit a page uh you can define a content security policy or csp policy in a header and you would define you know for script source list for example a list of origins from which you're allowed to fetch for images and all the rest you also have a report to target gotcha and a separate report to header that provides a specification for you specify the endpoint to where you want the report violation report to be reported and you know it's good hygiene that reporting endpoint should ideally be like a distinct origin um in all the rest but you provide a destination so you can find services that will do this for you right they'll just say point your report to to us and we will provide a dashboard which you can drill down reports we will aggregate we'll give metrics and all the rest.

50:04That's something that we do in-house at Shopify. And I think many other large providers will do on their own, but you could outsource that problem. But just having the ability to even get the report with, Hey, a report has been admitted because the script is missing an integrity hash is by itself a really useful capability because otherwise you'd probably have to set up some sort of a crawling infrastructure that periodically checks your page and says you know i i accessed this page from five different points on the globe right every 24 hours and i observed that nothing has changed well that's good but we could do much better by just actually observing what the real users are seeing and getting the actual reports of violations gotcha so this new one require sri 4 would work in like manner as the csv violations in terms of reporting.

50:59The CSP policy is require SRI for scripts, right? So you're saying all of my script resources must have a hash. And then you can configure that to be a report-only policy such that it would still execute if the script is missing the hash, but you would get the violation fired in the background. And the browser has its own logic for prioritizing batching delivery and doing all of that to get you the report. Now, do you deploy this one in Shopify? Yep. And do you use it in report mode or do you let lockdown mode or how do you use it? So for this one, it would be report mode, but it depends on the shape of your checkout and how much control you have for your first party or third party content.

51:50Just to double back on that, for Shopify, for our checkout, but we enforce a CSP policy. Actually, let me run through the whole list. For our first-party content, we have a well-defined process for vetting all the dependencies and a process for updates, auditing, to make sure that we provide some guarantees over if the library that we depend on has been compromised, how can we detect that? We have change management process for it. So this is the reviews, testing, CI, all the things that you would expect. that allows us to create the inventory we know from where it's served which means that we can define a strict csp policy that says you should only fetch from these sub origins that we trust in our build step we can inject the hashes the sri hashes for known content and we can also emit the require sri 4 policy to ensure that if anything else for some reason if we omit a missed some script that we would get a violation on that because we don't want to break checkout but we want to be notified immediately if those things are detected then we can react to it and we have our own reporting endpoint which we aggregate we look at the reports this is a thing that merchants don't have to worry about because we do this work on their behalf right and we can provide this guarantee over overall integrity and then finally we've protected the parent page but the payment credentials page or the payment form itself is also isolated into its own iframe just as it was before so this is a defense in depth right we protected the parent but we also have our own implementation of the iframe and like the full pci compliance behind that particular form well that's a lot of stuff for pci compliance ilia what happens with v5 how many years are you going to put into that one i don't know that that's a great question i'm pretty sure that v4 will keep us busy for a long while yeah because this is only section 6.4.3 right that's all we're talking about right here that's right there's there's all the others okay so interesting stuff it sounds like you've solved some really difficult technical challenges in order to to do this in a way that's uh not just compliant but actually in the spirit of the compliance as well, like trying to actually make it more secure.

54:18What are some takeaways for listeners out there? You know, maybe they're doing their own checkout, maybe they have compliance they need to do, maybe they just want some more secure websites. Like, what do you think they could be thinking walking away from this if they're not in the actual situation that Shopify is in and having to implement this stuff? What could they learn from this conversation? Yeah, I think the meta pattern and message takeaway here is broadly the integrity and security of first party versus third party content. We mix first party and third party in most contexts, but even outside of checkout, there are many surfaces.

54:59Let's say you have an admin surface or a privileged surface that you only want certain users to access. And you want some sort of extensibility in there. So you want to bring in third-party content or customization in all the rest. The pattern that we're describing with isolating third-party content is a generic pattern that you can deploy there. So we use the same sandboxing technology in checkout. We use the same technology in our admin. So for merchants, we allow customizations and third-party developers to bring in their custom UI and other aspects. As you can imagine, that's a very sensitive surface.

55:38Order data is there. Customer data is there. You don't want to just open up a Pandora's box of injector arbitrary JavaScript because who knows where the data might travel. So the isolation primitive, it may be a remote DOM. It may be something else. But this way of thinking of isolating into either an iframe or a worker, I think, is a pattern that we should be adopting more widely. And it has these additional benefits. You have better assurances about security. Yes. Performance as well, because you're isolating content and moving it off the main thread. You get to define the API contract. So you have better upgradability if you need to maintain that.

56:24And I think that's just something that we need to get better at on the web. Now, the challenge, I think, for all of us and kind of as industry practitioners is to think through. boy the worker is kind of this like naked environment we can probably figure out we should think about how do we figure out some better set of apis that where we don't have to reinvent the entire wheel just as we did with you know at shopify for great now i want to build a heat map thing what does that mean how do i mirror the the entire stream of events from top level page into this isolated environment i think we can do some thinking and innovation there.

57:04Very cool. Anything else that's on your mind that we haven't discussed in this context or honestly in any developer context? I always love to hear your opinions on stuff. Anything else on your mind? I think one really interesting topic coming back to the world of checkout and commerce is of course agents and how agents will interact or how they might affect any of these behaviors. Oh yes. MCP, are you done with MCP? That's the newest acronym, Model Context Protocol. It's burgeoning. Yep, yep. MCP is definitely top of mind, and we're looking at it intently. We're using it for a number of tools and internal services at Shopify.

57:41We're also considering if and how we should be exposing MCP as a protocol and endpoint as a service on behalf of merchants. So imagine you could have a merchant storefront as a remote MCP endpoint. but more broadly like if you think of let's imagine you interacting with an agent asking it hey i'd like a pair of white sneakers size 10 15 to or 50 to 100 range please go find me a pair and check out under the hood the agent might crawl the web find the storefront add to cart head to checkout and what does it do then as it's looking at a payment form is it responsibility of the agent to hold on to your payment credentials?

58:23And what are the implications of that for entering? How does it enter those credentials? Are there any security and compliance problems or challenges in that? I think that's a wide open question that we as an industry are yet to figure out an answer. Is the human required in that loop? What if it's an accelerated checkout where maybe information is vaulted, right? I think there's a range of questions and answers that we need to figure out in the space. What's your personal thought on is the human required in the loop? How do you feel confidence-wise on removing the human from that loop? I think it's context-dependent.

59:03I think there's definitely a class of commerce in certain types of transactions where I know exactly what I want. There's very low risk and it's kind of a predefined flow where I just say, look, I need another carton of milk. you know exactly what i'm looking for you know where to shop and please go finish it and i just want it my front door and then there is other types of experiences where maybe you're this is your first time engaging with a merchant maybe a merchant has a set of rules where they actually require you or require the agent to decelerate because hey for compliance reasons i may need to verify your age or i need you to read this disclaimer on this product before you purchase it You can't just have the agent blindly ignore that context or click approve and then proceed with the transaction.

59:55So I think we'll need to define some protocol or shared mechanism to signal to agents that like, hey, in this particular case, I need you to pause and ask for human to either confirm or take over control and complete the transaction. There's so many questions there. I just don't feel like I even have the brain right now to analyze all the things that have to be considered. I'm glad that you're, are you going to be working on this for Shopify? Are you going to stay in the NPCI land? What's next for you inside? Is this an active thing that you're thinking about for Shopify? It is definitely an active area of exploration for us.

1:00:39It is one of the things I'm looking with our team and many of our partners who are building these agents. we're trying to figure out what is the future of checkout where agents drive some meaningful portion of that experience. What does a good experience even look like in that context? So I think those are all very interesting and pertinent questions given where we are today. Well, I'll have to have you come back in a year or two and let us know what you end up building as you've figured it all out. You seem to have figured it out. At least this hairy technical problem has come with this new PCI stuff.

1:01:13so I'm sure you'll figure out something. Yeah, we'd love to be back. And at the rate that we're moving in AI world in a year or two for now, who knows what will be there? Yes, I'm trying to think of the most recent person who said six to nine months and LLM will be writing 100 % of code. So, I mean, who knows, man? Maybe you and I will be out on the street corner talking about this stuff. I doubt that is the case, but who knows? Yeah, me too. But not a week goes by that somebody doesn't declare software engineering is dead or dying. How to squeeze that one in there. I think what we're actually saying is the definition of what software engineering is is changing.

1:01:58I am constantly amazed by what AI is capable of doing in terms of software development, but I'm also constantly surprised by the silly and stupid mistakes that it makes. And oftentimes those mistakes are actually due to

1:02:15misunderstanding or lack of poor definition of the problem that's being solved. It's kind of putting the mirror back to yourself, right? Because oftentimes I'll find that like, actually, you know what, you did exactly the right thing, the way I expressed it, but that's not what I meant. And I didn't even know what I meant when I typed it, because now that I've seen the mistake, I understand what I was actually trying to get to. So it is this like art of defining the problem and rubber deck programming. And I think we're heading more and more towards a world where we're actively collaborating instead of hands-on keyboard typing if statements.

1:02:46Yeah, the best rubber duck programmers might be the best programmers of the future. The ones who can just talk it out the best, you know, figure it out as you go. All right, Ilya, appreciate you coming on the show and chatting with us and looking forward to having you back soon. Thank you, Jared.

1:03:04Okay, so it turns out securing e-commerce checkouts has never been more complicated. But thankfully, brilliant engineers like Ilya and his team at Shopify are putting in the work, and some of that work is making its way back into the web platform. I love when that happens. And when you think about it, the complicated nature of it all makes sense. The stakes have never been higher. I read the other day that last year, e-commerce sales soared to a record$1.2 trillion. That's a lot of moolah being transferred. And if you can hack it, well, you can jack it. so yeah it's complicated for a good reason let's give one more thanks to our sponsors of this episode retool augment code and of course fly.io check out their wares to support their work which supports our work which we appreciate thanks also to our beat master in residence break master cylinder did you know our next full-length album is almost ready and i'll tell you right now it's called after party and i'll also tell you right now that i've been bumping all week.

1:04:09I dig it. Hopefully you will too. Soon. So soon. All right, that's all from me, but we'll talk to you again on ChangeLoginFriends on Friday. Bye, y 'all.

1:04:53Game on.