Understanding the Internet from First Principles with cURL

You have done this thousands of times. You type an address, something happens for about a second, and a page appears. Almost none of that is visible, and almost all of it is simple. Underneath there are only two machines and two messages. One machine wants something. That is this one, and we call it the client. The other machine has the thing and is waiting to be asked for it. That is the server: a computer in a building somewhere, switched on all the time, doing nothing else. The client sends a request. One short piece of text saying what it wants, and a few facts about itself. The server sends back a reply. A short piece of text saying how it went, and then the thing itself. That is the whole shape of it, and everything else is detail about those two arrows. So let us do exactly that, using a program called curl. It plays the part of the client, once, and prints whatever comes back. Here is the whole command. The first line that came back is the status line. It says which version of the language the server spoke, and then a number. Two hundred means it worked. Then a few lines about the reply itself. This one says the thing coming back is a web page. Then one blank line, and then the page, which is what a browser would draw for you. Here it simply gets printed. Four separate things had to happen before any of that text existed, and they happen in the same order every single time. First the name in the address had to become a number. Then a connection had to be opened between the two machines. Then the request had to be written out in a very particular form. And then the reply came back, carrying a code. None of those four is mysterious, and by the end you will have watched each one happen on your own screen. Let us begin with the number.

The name you typed is for you. It is a convenience for humans, and the machines in between do not use it at all. What they use is a number. Every machine that can be reached from the internet has one of these addresses. Here are four of them: four machines, somewhere in the world. One of them is the one we are after, the machine holding the page we asked for. It is this one, in green. Written down, an address is four small numbers with dots between them. Each one runs from zero up to two hundred and fifty five, which is as much as eight bits can hold. That number is the destination, and it is the only part of the address the network in between cares about. The name itself never travels anywhere. Which leaves us a problem. You typed a name. The wire wants a number. Something has to do the translation, and it has to happen before anything else. That something is called the domain name system. Picture it as one enormous table with two columns: names on the left, addresses on the right. No single machine holds that table. It is cut into pieces, spread over thousands of machines, and each piece is kept by whoever owns those particular names. So your computer asks a machine nearby, called a resolver. If the resolver does not know the answer, it asks the machines that do, and passes back what it gets. And the answer is remembered for a while, both on your machine and on the resolver. That is why the first visit to a site is a little slower than the second. You do not have to take any of this on faith. Add one flag to the command, dash v for verbose, and the program narrates its own work as it goes. The first thing it says is the number it looked up, and the door it is knocking on at that address. Four four three is called a port, and it is the one kept for encrypted web traffic. Then it says connected. From that moment the name is finished with, and everything that follows happens between two numbers.

We know where to send things now. That still is not a conversation, because the network in between does not carry conversations. It carries small pieces, one at a time, and it promises nothing about them. Send four pieces in this order. Each one is passed from machine to machine along whatever path happens to be free, so they arrive in whatever order they manage. Here the third piece overtook the second. And sometimes a piece never arrives at all, and nothing along the way even notices. So something has to number the pieces, keep track of which ones landed, ask again for the missing ones, and hand the whole thing over in order. That something is TCP, and it is the reason you have never had to think about any of this. Setting one of those connections up takes three short messages, and nothing useful is said in any of them. Time runs downward here. The client sends the first one. It means: I would like to talk to you. It is called SYN, short for synchronise. The server sends the second, which means: I heard you, and I would like to talk to you as well. And the client sends the third: I heard that. Now both ends know the other one can hear them, and both know where to start counting. That is the three way handshake. It costs one full round trip before a single useful byte moves. That is why a server on the far side of the world feels slow even when it is not busy at all. One more thing happens when the address begins with https. Before any useful words are exchanged, the two ends agree on how to encrypt everything that follows. That agreement is a short exchange of its own, and it gets reported when it finishes. From here on, anybody watching the wire in between sees only noise. The server also hands over a certificate: a document signed by an organisation your machine already trusts, saying that this really is the machine for that name. It gets checked, and here it passed. If that check fails, the program stops and tells you, and your request is never sent. So now we have an address, and an open, private, orderly line. Time to say something.

The connection is open, and both ends can now send plain text at each other. What the client sends first is called a request, and it always begins with a single line like this. There are three things on it, always in that order: what to do, which thing to do it to, and which version of the rules we are speaking. Get is the method, and it means send me this thing. The path is a single slash, which means the front page. The version is one point one, which almost everything still speaks. Under that first line come the headers. Each one is a name, a colon, and a value, and each is one extra fact about the request. Host says which site you meant, because one machine often answers for hundreds of them. User agent says who is asking. Accept says what kinds of thing you are willing to receive. Then one blank line. That is how the far end knows the headers are finished. After it comes the body, if the request has one, and a get request usually does not. The method is the interesting part, because it is a promise about what the request is going to do. Get asks for something and changes nothing. You can send it a hundred times over and the server is the same afterwards. Post carries something with it and expects the server to be different afterwards. Put replaces a thing with what you sent. Delete removes it. Those meanings are a convention rather than a law. Nothing stops a server doing something else entirely. Almost none of them do, and everything from your browser to a search engine is built on that. Now the other direction. The reply has exactly the same shape: one first line, then headers, then a blank line, then the body. But that first line carries a number where the request had a method, and the number is the whole point. Two hundred means it worked. The numbers come in families, and the family is the first digit. Two hundreds mean it worked. Three hundreds mean the thing is somewhere else. Four hundreds mean the request was wrong. Five hundreds mean the server itself broke. So four zero four is not a mysterious error message. It sits in the four hundreds family, and it means you asked for something that is not there. Then come the reply's own headers: what kind of thing this is, and how long it is. And after the blank line, the thing itself: the page, the picture, the data. That is the entire protocol. A first line, some headers, a blank line, a body, in both directions.

All of this is invisible by default. One flag makes every bit of it visible: dash v, for verbose. Now every line that gets printed is marked, and the mark tells you who said it. A star means the program talking about its own work. A greater than sign means the line went out to the server. A less than sign means it came back. So read it from the top. A star: the connection is open, which is the handshake we just watched, all three messages of it. Two lines going out: the request line we took apart, and the host header underneath it. Two lines coming back: the status line with its two hundred, and one of the reply's own headers. That is the whole conversation, in order, on your own screen. The lookup, the connection, the request, the code. Each thing we have talked about is one line of this. Three more commands are worth knowing, and each one is the same request with one thing changed. Dash capital I asks for the headers and nothing else. It sends the HEAD method, so the server answers with the status line and the headers but no body. It is the quick way to ask whether something is there at all. Dash d sends a body. Adding it changes the method to POST all by itself, and whatever you write after the flag is what goes into the body. And dash capital L follows a redirect. When the reply comes back in the three hundreds, the new address is sitting in a header called location, and this makes the program go there and ask again. That last one is a status code doing real work, rather than being an error you read about. So, back to the four steps we started with, and every one of them has now been on the screen. The name became a number, and you watched the lookup. A connection was opened, and you watched the handshake. A request went out, and you read it line by line. A reply came back carrying a code, and you read that too. None of it is magic and none of it is hidden. It is four steps and a great deal of plain text, and one command will print that text for you. That is the internet, from the bottom up. Go and run the command on something you use every day, and read what comes back.