Do you want to know what your phone is sending & receiving? Nowadays, that means you need to control who it trusts. In modern connections everything sent & received is encrypted by TLS, and the only practical way to intercept traffic on Android is to generate a certificate, make the phone or app trust it as a certificate authority (CA), and then use this to act as a fully trusted proxy to the real server. Being able to view & modify this traffic is important. It's the basis of a huge amount of security & privacy research, allowing researchers to confirm what an app is actually sharing, and…
Dictionary compression could completely change how applications send data over the web. It's recently gained broad support, and offers absurd real-world traffic reductions: initial testing shows YouTube JS download size for returning desktop users shrinking up to 90% (!!!) compared to existing best-practice compression, while the Google search results HTML (arguably the most optimized content on the internet) shrinks nearly 50%. This works by initializing the (de)compression algorithm with a dictionary of data known in advance to both compressor & decompressor, so that the compressed data can…
HTTP Toolkit, like effectively all software businesses, depends on a huge quantity of open-source code for much of its fundamental functionality & infrastructure. Most of this is tirelessly maintained by volunteers, completely for free! This honestly is a great deal for the businesses, but it would be even better if these maintainers were actually rewarded for their hard work. As part of HTTP Toolkit's commitment to giving back to open source under the Open Source Pledge, a substantial chunk of revenue goes back into these open-source projects, to keep them healthy, reward maintainers for…
HTTP/3 has been in development since at least 2016, while QUIC (the protocol beneath it) was first introduced by Google way back in 2013. Both are now standardized, supported in 95% of users' browsers, already used in 32% of HTTP requests to Cloudflare, and support is advertised by 35% of websites (through alt-svc or DNS) in the HTTP Archive dataset. We've developed a totally new version of HTTP, and we're on track to migrate more than 1/3 of web traffic to it already! This is astonishing progress. At the same time, neither QUIC nor HTTP/3 are included in the standard libraries of any major…
The Open Source Pledge is a new push to make companies commit to funding the maintainers of the open-source software they depend on, and to publicly recognize the ones that do. HTTP Toolkit has donated back to maintainers for a few years now, but joining the Open Source Pledge today means formally committing to that, and to doing so publicly with a sustainable minimum level ($2000 per full-time developer, or higher) indefinitely into the future. What is the Open Source Pledge? HTTP Toolkit (and effectively 100% of other software businesses) depends on a huge quantity of open-source code for…
If you're using a local debugging proxy tool like HTTP Toolkit, you might run into the dreaded ERR_PROXY_CONNECTION_FAILED error in Chrome and other similar apps. This can be a very frustrating and unhelpful error! There's only a few possible causes though, and it's usually easy to fix. The Simple Case The simplest explanation is exactly what it says: the browser can't connect to your proxy. In the simple case this may be caused by a basic connection issue: you have the address, port or some authentication details wrong, or the details are correct but the proxy is just not reachable on your…
When everything goes smoothly with an API, life is pretty straightforward: you request a resource, and voilà, you get it. You trigger a procedure, and the API politely informs you it’s all gone to plan. But what happens when something goes pear-shaped? Well, that’s where things can get a bit tricky. HTTP Status Codes HTTP status codes are like a first aid kit: they’re handy, but they won’t fix everything. They give you a broad idea of what’s gone wrong, which can help plenty of tools and developers make reasonable assumptions, like: 400 Bad Request: Report error to developers, something is…
As of February 14th 2024, RFC 9512 formally registers application/yaml as the media type for all YAML content, and adds +yaml as a standard structured suffix for all YAML-based more specific media types. With this registration, it's now included in the official media types list maintained by the IANA. Media types like this (also known as the MIME types, from their original invention for email attachment metadata) are heavily used particularly in HTTP Content-Type headers for both requests & responses, and in all sorts of file metadata and processing logic elsewhere. These names give…
The X-Forwarded-For (XFF) HTTP header provides crucial insight into the origin of web requests. The header works as a mechanism for conveying the original source IP addresses of clients, and not just across one hop, but through chains of multiple intermediaries. This list of IPv4 and IPv6 addresses is helpful to understand where requests have really come from in scenarios where they traverse several servers, proxies, or load balancers. A typical HTTP request goes on a bit of a journey, traversing multiple layers of infrastructure before reaching its destination. Without the X-Forwarded-For…
Idempotency is when doing an operation multiple times is guaranteed to have the same effect as doing it just once. When working with APIs this is exceptionally helpful on slow or unreliable internet connections, or when dealing with particularly sensitive actions such as payments, because it makes retrying operations safe and reliable. This is why most payment gateways like Stripe and Adyen support 'idempotency keys' as a key feature of their APIs. Recently, the IETF have gone further, and created a draft RFC standard for this useful common pattern, as part of the 'Building Blocks for HTTP…
It's hard to work on APIs without hearing about OpenAPI. OpenAPI is an API description format, which is essentially metadata that describes an HTTP API: where it lives, how it works, what data is available, and how it's authenticated. Additional keywords can be used to provide all sorts of validation information, adding a type system to what would otherwise just be arbitrary JSON flying around the internet. OpenAPI has been around for donkeys years, previously known as Swagger but renamed to OpenAPI in 2016. It's powered by JSON Schema, which is also pretty popular in certain circles, but…
Everything crashes. Sometimes things crash when they're running inside a Docker container though, and then all of a sudden it can get much more difficult to work out why, or what the hell to do next. Docker's great, but it's an extra layer of complexity that means you can't always easily poke your app up close any more, and that can really hinder debugging when your container fails to start or breaks in unusual ways. If you're stuck in that situation, here are my goto debugging commands to help you get a bit more information on exactly what's up: docker logs Hopefully you've already tried…
A couple of weeks ago I published a post about changes in Android 14 that fundamentally break existing approaches to installing system-level CA certificates, even with root access. This has triggered some fascinating discussion! I highly recommend a skim through the debate on Mastodon and Hacker News. Since that was posted, quite a few people have talked to me about possible solutions, going beyond the previous approaches with new mechanisms that make it practical to do this in Android 14+, and there are some good options here. While direct root access to change these certificates by simply…
Update: This post sparked a lot of excellent discussion and debate on workarounds, and there are now multiple working solutions to allow certificate injection on Android 14, despite the restrictions discussed here. See the update post for more details. When Android was initially announced in 2007 by the Open Handset Alliance (headed by Google) their flagship project was billed as an "open platform", "providing developers a new level of openness", and giving them "complete access to handset capabilities and tools". We've come a long way since then, steadily retreating from openness & user…
There's been a lot of concern recently about the Web Environment Integrity proposal, developed by a selection of authors from Google, and apparently being prototyped in Chromium. There's good reason for anger here (though I'm not sure yelling at people on GitHub is necessarily the best outlet). This proposal amounts to attestation on the web, limiting access to features or entire sites based on whether the client is approved by a trusted issuer. In practice, that will mean Apple, Microsoft & Google. Of course, Google isn't the first to think of this, but in fact they're not even the first to…
Caching is hard. Unfortunately though, caching is quite important. Hosted caching & CDNs offer incredible powers that can provide amazing performance boosts, cost savings & downtime protection, essential for most modern sites with any serious volume of users. Unfortunately, while there are strict standards for how caching is supposed to work with HTTP on the web, many cache providers do not quite follow these, instead giving their customers free reign over all kinds of invalid caching behaviour, and providing their own default configurations that often don't closely follow these standards to…
This week, at long last, GitHub announced granular access tokens for npm. This is a big deal! It's great for security generally, but also particularly useful if you maintain any npm packages, as it removes the main downside of automating npm publishing, by allowing you to give CI jobs only a very limited token instead of full 2FA-free access to your account. In the past, I've wished for this, because I maintain a fair few npm packages including some very widely used ones. The previous solution of "just disable 2FA on your account, create an all-powerful access token with global access to…
As you may have seen, Docker Hub made a dramatic shift in policy this week, and effectively gave a 30 day eviction notice to almost all community-run images. They've now made an apology to 'clarify' a few details, and helpfully take some of the hard edges off, but this still highlights a big problem. Fortunately, there are solutions. As initially described, this would've been catastrophic: Docker Hub has been used as the default host in tooling, tutorials, demos, blog posts, scripts, deployment definitions, CI builds and more for many years, and all those references were going to break - a…
HTTP Toolkit has been selected to receive another round of open-source funding from the EU! This aims to improve interception of HTTPS traffic from mobile apps, making it easier for both security/privacy researchers and normal technical users to inspect & manipulate the data that any app they use sends & receives. This funding will directly support work to improve the precision & usability of mobile app interception, on both iOS and Android. In a couple of clicks, you'll be able to target any app installed on a connected device (on either platform) and inspect all its traffic. That means no…
HTTP is important on the web, but as other alternative protocols grow popular in networked applications, it's often important to be able to capture, debug and mock those too. I've been working on expanding HTTP Toolkit's support for this over the past year (as one part of a project funded by EU Horizon's Next Generation Internet initiative), to extend HTTP Toolkit to cover three additional rising protocols that are often used alongside simple HTTP in decentralized web applications: WebRTC, IPFS & Ethereum. This is now live! If you're using HTTP Toolkit to intercept browsers, and a web…
Read at the source
Your visit, your choice.
Optional Google Analytics helps us understand visits. Microsoft Clarity records masked interactions to improve the site. Optional tools stay off unless you choose them. Privacy details.