Backend from the Beginning, Pt 1: Introduction, TCP, DNS, HTTP

A software article by Efron Licht

September 2023

software articles by efron licht

This article is part of a series on backend web development in Go. It will cover the basics of backend web development, from the ground up, without using a framework. It will cover the basics of TCP, DNS, HTTP, the net/http and encoding/json packages, middleware and routing. It should give you everything you need to get started writing professional backend web services in Go.

Source code for this article (& the entire blog) is publically available on my gitlab.

1. Table of Contents

2. Introduction: when I hear the word ‘framework’ I reach for my gun

One of the most common questions I get from new developers starting with Go is “what web framework should I use?” The answer I always give is “you don’t need a framework”, but the problem is, backend devs are used to frameworks.

Thinking about it, the motivation is understandable: engineers are under a lot of pressure, and the internet seems really complicated, the idea of learning all these layers of abstraction (tcp, http, etc) is daunting, and everyone else seems to use a framework - in most languages (javascript, python, etc), it’s practically required. There’s only one problem with this: it means you never learn how things actually work. Constantly relying on suites of specialized tools rather than learning the basics is the equivalent of being a senior chef who can’t use a knife. Sure, you can argue that your fancy food processor chops faster, but the second you need to do something for which your pre-packaged tools aren’t designed, you’re screwed; you have no idea how to do it yourself and no time to learn.

This may sound like an exaggeration, but I have now met four different senior software engineers who couldn’t tell me how to make a HTTP request to google without a framework.

For the record, you send this message to 142.250.189.14:80:

1GET / HTTP/1.1
2Host: google.com

It’s five words.

If you don’t know what stuff means or how I got that IP address, don’t worry; we’ll get to that. The point is, it’s not actually that hard; hard is ten thousand layers of callbacks and libraries and frameworks and tools and languages and abstractions and indirections and wrappers around wrappers. The problem is, most software engineers are so used to having so many layers of abstraction between them and the network that ‘getting to the bottom’ seems impossible. Now, some may argue that this is OK, because the increased power & flexibility of frameworks leads to faster, better, software development. The only problem is, software isn’t getting better; it’s getting measurably worse. Both desktop software and web pages are measurably slower year over year. Software is getting slower faster than computers are speeding up. With this in mind, it’s no surprise that software is drowning in complexity. Every year, we add more layers of abstraction, more libraries, more frameworks, more tools, more languages, more everything, but even our ‘experts’ don’t understand the basics of the stuff they’re building. It’s no wonder that software is so slow and buggy and impossible to maintain.

“An idiot admires complexity, a genius admires simplicity”

Of course, knowing that things are done badly doesn’t help you learn how to do it well, so I’m writing this series of articles to try and fill the gap by teaching the basics of backend web development in Go. Each article will be filled with real programs you can run on your computer, not cherry-picked code samples that don’t even compile.

This series will not be enough to teach you everything. At best it will expose you to enough of how things really work that you can start seeing the edges of your knowledge and begin filling in the gaps yourself. It will also necessarially have to simplify or omit some details or tell “white lies” to make things easier to understand; there’s no substitute for experience (or for reading the source code & documentation of the standard library). It will also largely omit databases; I hope to make those the subject of a future series.

That said, I hope it will help.

3. Series overview

3.1. backend basics, part 1: TCP, DNS, & HTTP

3.2. Practical backend: net/http and encoding/json

In the second article, we’ll graduate to using the net/http and encoding/json packages to build basic web clients and servers that can handle most day-to-day backend workloads. We’ll start diving into Go’s standard library and the show how it provides everything you need for basic client/server HTTP communication; using net/http and net/url to send and receive HTTP requests and responses, encoding/json to manage our API payloads, and context to manage timeouts and cancellation.

3.3. Finishing touches: middlewares, routing, and basic database access

In the third article, we’ll cover middleware and routing, the two ‘missing pieces’ of the net/http package. These are the bits that usually make people reach for a framework, but they’re actually pretty simple to implement yourself. We’ll also cover basic database access using the database/sql package.

4. What is backend?

‘Backend’ is connecting together computers via the internet. You know what a computer is, so…

4.1. What’s the internet anyways?

What is the internet? No, I’m serious. What is the problem the internet solves? The internet is a network of computers that can reliably communicate with each other even if some of the computers ‘in the middle’ are down. It allows you to reliably send messages (that is, text or binary data) to other computers, even if you don’t know where those computers are or how they’re connected to you. You can send a message to another computer, so long as there is a path of computers from you (the LOCALADDR) to the destination computer (the REMOTEADDR).

To do this, the internet must solve two problems:

4.2. TCP: how do I make sure that when I send a message through a network all of it gets through, in order?

The details of TCP are out of scope for this article, but at a high level it looks like this:

4.3. IP: how do I make sure that when I send a message through a network it gets to the right place?

IP is more complicated. The following explanation is wrong at pretty much every level if you zoom in enough, but it’s a good enough approximation for our purposes:

4.4. Addresses & Ports

OK, so how do we actually send a message to another computer? We need to know two things: the address of the computer we want to send a message to, and the port of the service we want to send a message to.

4.5. IP Addresses

IP Addresses come in two forms: ipv4, a 32-bit number; or ipv6, a 128-bit number. They look like this:

IPV4 looks like this: DDD.DDD.DDD.DDD, where DDD is a number between 0 and 255.

IPV6 looks like this: XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX, where XXXX is a 16-bit hexadecimal number; that is, each X is one of 0..=9 or a..=f

IP Address Type Note
192.168.000.001 ipv4 localhost; refers to hosting computer
192.168.0.1 ipv4 same as above; you can omit leading zeroes
0000:0000:0000:0000:0000:ffff:c0a8:0001 ipv6 refers to same computer as above; ipv4 addresses can be embedded in ipv6 addresses by prefixing them with ::ffff:
::ffff:c0a8:0001 ipv6 same as above; you can omit leading zeroes
2a09:8280:1::a:791 ipv6 fly.io

4.6. Ports

It’s common for a computer to want to host multiple internet services that behave in different ways. For example, we could want to host a game server (like starcraft), a web server (like this website), and a database (like postgresql) all on the same computer. Since they’re all on the same physical computer, they’ll share an IP address, so we’ll need some way to tell apart requests to the file server from requests to the game server. We do this by assigning a PORT to each service. A port is just a number between 0 and 65535. Even if we’re only hosting one service, each service needs (at least one) port.

eblog is hosted at port 6483. The following table lists default ports for some common services:

Service Port
HTTP 80
HTTPS 443
SSH 22
SMTP 25
DNS 53
FTP 21
Postgres 5432

5. Building a basic TCP/IP server and client

Let’s build a basic TCP/IP server and client to demonstrate how this works. We’ll build a server that listens on port 6483, and a client that connects to it. Anything sent on stdin on the client (that is, typed into the terminal) will be sent to the server, a line at a time. Anything line received on the server will be uppercased and sent back to the client.

That is, an example session might look like this:

1SERVER: (starts listening on port 6483)
2CLIENT: (connects to server)
3CLIENT: "hello, world!"
4SERVER: "HELLO, WORLD!"
5CLIENT: "goodbye, world!"
6SERVER: "GOODBYE, WORLD!"
7CLIENT: (disconnects)

To briefly review, the following functions and types are relevant to our examples:

function/struct description implements
net.Listen listens for connections on a port
net.Dial connects to a server at an IP address and port
net.TCPConn bidirectional TCP connection io.Reader, io.Writer, net.Conn
net.Conn bidirectional network connection io.Reader, io.Writer
bufio.Scanner reads lines from a io.Reader
fmt.Fprintf as fmt.Printf, but writes to a io.Writer
flag.Int register an integer command line flag
flag.Parse parse previously registered command line flags
log.Printf as fmt.Fprintf(os.Stderr, ...), but with a timestamp and newline
log.Fatalf as log.Printf, but calls os.Exit(1) after printing

5.1. client

 1Let's write the client first: we'll call it `writetcp`.
 2
 3```go
 4// writetcp connects to a TCP server at at localhost with the specified port (8080 by default) and forwards stdin to the server,
 5// line-by-line, until EOF is reached.
 6// received lines from the server are printed to stdout.
 7package main
 8
 9import (
10    "bufio"
11    "flag"
12    "fmt"
13    "log"
14    "net"
15    "os"
16)
17
18func main() {
19    const name = "writetcp"
20    log.SetPrefix(name + "\t")
21
22    // register the command-line flags: -p specifies the port to connect to
23    port := flag.Int("p", 8080, "port to connect to")
24    flag.Parse() // parse registered flags
25
26    conn, err := net.DialTCP("tcp", nil, &net.TCPAddr{Port: *port})
27    if err != nil {
28        log.Fatalf("error connecting to localhost:%d: %v", *port, err)
29    }
30    log.Printf("connected to %s: will forward stdin", conn.RemoteAddr())
31
32    defer conn.Close()
33    go func() { // spawn a goroutine to read incoming lines from the server and print them to stdout.
34        // TCP is full-duplex, so we can read and write at the same time; we just need to spawn a goroutine to do the reading.
35
36        for connScanner := bufio.NewScanner(conn); connScanner.Scan(); {
37
38            fmt.Printf("%s\n", connScanner.Text()) // note: printf doesn't add a newline, so we need to add it ourselves
39
40            if err := connScanner.Err(); err != nil {
41                log.Fatalf("error reading from %s: %v", conn.RemoteAddr(), err)
42            }
43        }
44    }()
45
46    // read incoming lines from stdin and forward them to the server.
47    for stdinScanner := bufio.NewScanner(os.Stdin); stdinScanner.Scan(); { // find the next newline in stdin
48        log.Printf("sent: %s\n", stdinScanner.Text())
49        if _, err := conn.Write(stdinScanner.Bytes()); err != nil { // scanner.Bytes() returns a slice of bytes up to but not including the next newline
50            log.Fatalf("error writing to %s: %v", conn.RemoteAddr(), err)
51        }
52        if _, err := conn.Write([]byte("\n")); err != nil { // we need to add the newline back in
53            log.Fatalf("error writing to %s: %v", conn.RemoteAddr(), err)
54        }
55        if stdinScanner.Err() != nil {
56            log.Fatalf("error reading from %s: %v", conn.RemoteAddr(), err)
57        }
58    }
59
60}
61```

5.2. server

 1Now let's put together the server; since it echoes back whatever it receives, in uppercase, we'll call it `tcpupperecho`.
 2
 3Usually when working in backend, we want to separate your 'business logic' from the networking code. Since all of go's networking APIs use the [net.Conn](https://golang.org/pkg/net/#Conn) interface, which implements both [io.Reader](https://golang.org/pkg/io/#Reader) and [io.Writer](https://golang.org/pkg/io/#Writer), we can write our business logic using standard text-handling functions and structs like [fmt.Fprintf](https://golang.org/pkg/fmt/#Fprintf) and [bufio.Scanner](https://golang.org/pkg/bufio/#Scanner).
 4
 5Our server's 'business logic' will look like this:
 6
 7```go
 8// echoUpper reads lines from r, uppercases them, and writes them to w.
 9func echoUpper(w io.Writer, r io.Reader) {
10    scanner := bufio.NewScanner(r)
11    for scanner.Scan() {
12        line := scanner.Text()
13        // note that scanner.Text() strips the newline character from the end of the line,
14        // so we need to add it back in when we write to w.
15        fmt.Fprintf(w, "%s\n", strings.ToUpper(line))
16    }
17    if err := scanner.Err(); err != nil {
18        log.Printf("error: %s", err)
19    }
20}
21```
22
23Which we can then use in our server like this:
24
25```go
26
27// tcpupperecho serves tcp connections on port 8080, reading from each connection line-by-line and writing the upper-case version of each line back to the client.
28
29package main
30
31import (
32    "bufio"
33    "flag"
34    "fmt"
35    "io"
36    "log"
37    "net"
38    "strings"
39)
40
41func main() {
42    const name = "tcpupperecho"
43    log.SetPrefix(name + "\t")
44
45    // build the command-line interface; see https://golang.org/pkg/flag/ for details.
46    port := flag.Int("p", 8080, "port to listen on")
47    flag.Parse()
48
49    // ListenTCP creates a TCP listener accepting connections on the given address.
50    // TCPAddr represents the address of a TCP end point; it has an IP, Port, and Zone, all of which are optional.
51    // Zone only matters for IPv6; we'll ignore it for now.
52    // If we omit the IP, it means we are listening on all available IP addresses; if we omit the Port, it means we are listening on a random port.
53    // We want to listen on a port specified by the user on the command-line.
54    // see https://golang.org/pkg/net/#ListenTCP and https://golang.org/pkg/net/#Dial for details.
55    listener, err := net.ListenTCP("tcp", &net.TCPAddr{Port: *port})
56    if err != nil {
57        panic(err)
58    }
59    defer listener.Close() // close the listener when we exit main()
60    log.Printf("listening at localhost: %s", listener.Addr())
61    for { // loop forever, accepting connections one at a time
62
63        // Accept() blocks until a connection is made, then returns a Conn representing the connection.
64        conn, err := listener.Accept()
65        if err != nil {
66            panic(err)
67        }
68        go echoUpper(conn, conn) // spawn a goroutine to handle the connection
69    }
70}
71```

5.3. Demo

 1Let's try it out.  In one terminal, we'll run the server:
 2
 3IN
 4
 5```sh
 6    go build -o tcpupperecho ./tcpupperecho.go
 7    ./tcpupperecho -p 8080 # run the server, listening on port 8080
 8    ```
 9```
10
11OUT:
12
13```text
14    tcpupperecho    2023/09/07 10:13:13 listening at localhost: [::]:8080
15```
16
17Let's run the client in another terminal and send it a message:
18
19```sh
20    $ go build -o writetcp ./writetcp.go
21    $ ./writetcp -p 8080 # run the client, connecting to localhost:8080
22    > writetcp 2023/09/07 10:20:32 connected to 127.0.0.1:8080: will forward stdin
23    hello
24    writetcp        2023/09/07 10:20:49 sent: hello
25    HELLO
26```
27
28And checking in back in the server terminal, we see:
29
30```sh
31tcpupperecho    2023/09/07 10:20:49 received: hello
32```

6. Connecting to a server on the internet

This works fine for local addresses, but what if we want to connect to a server on the internet? Most of the time, we don’t know the IP address of the server we want to connect to; we only know its domain name, like google.com or eblog.fly.dev. How do we connect to a server at a domain name?

6.1. DNS

Domain Name Service, or DNS, is a service that maps domain names to IP addresses. It’s essentially a big table that looks like this:

domain last known ipv4 last known ipv6
google.com 142.250.217.142 2607:f8b0:4007:801::200e
eblog.fly.dev 66.241.125.53 2a09:8280:1::37:6bbc

There are multiple DNS providers. Your ISP usually provides one, and there are public ones like Google’s, available at both 8.8.8.8 and 4.4.4.4. (Since you can’t resolve a domain name without knowing the IP address of a DNS server, you need to know at least one IP ‘by heart’ to get started.)

Browsers and other clients use DNS service to look up the IP address of a domain name.

6.2. Finding the IP address of a server

2021/08/18 16:00:00 tcpupperecho listening at localhost:
OK, so we want to connect to a server at a web address: say, https://eblog.fly.dev. How do we do that? Well, first we need to get the IP address of the server. The domain name service, or DNS, is a service that maps domain names to IP addresses. You can use the built-in nslookup command to look up the IP address of a domain name from your command-line on windows, mac, or linux.

IN:

1nslookup eblog.fly.dev

OUT:

1Server:  UnKnown
2Address:  192.168.1.1
3
4Non-authoritative answer:
5Name:    eblog.fly.dev
6Addresses:  2a09:8280:1::37:6bbc
7          66.241.125.53

Within a Go program, use net.LookupIP to look up the IP address or addresses of a domain name. The following full program duplicates the functionality of nslookup:

 1// dns is a simple command line tool to lookup the ip address of a host;
 2// it prints the first ipv4 and ipv6 addresses it finds, or "none" if none are found.
 3package main
 4
 5import (
 6    "fmt"
 7    "log"
 8    "net"
 9    "os"
10)
11
12func main() {
13    if len(os.Args) != 2 {
14        log.Printf("%s: usage: <host>", os.Args[0])
15        log.Fatalf("expected exactly one argument; got %d", len(os.Args)-1)
16    }
17    host := os.Args[1]
18    ips, err := net.LookupIP(host)
19    if err != nil {
20        log.Fatalf("lookup ip: %s: %v", host, err)
21    }
22    if len(ips) == 0 {
23        log.Fatalf("no ips found for %s", host) // this should never happen, but just in case
24    }
25    // print the first ipv4 we find
26    for _, ip := range ips {
27        if ip.To4() != nil {
28            fmt.Println(ip)
29            goto IPV6 // goto considered awesome
30        }
31    }
32    fmt.Printf("none\n") // only print "none" if we don't find any ipv4 addresses
33
34IPV6: // print the first ipv6 we find
35    for _, ip := range ips {
36        if ip.To4() == nil {
37            fmt.Println(ip) // we don't need to check for nil here, since we know we have at least one ip address
38            return
39        }
40    }
41    fmt.Printf("none\n")
42}
43

IN:

1go build -o dns ./dns.go # build the dns command
2./dns eblog.fly.dev # run the dns command

OUT:

166.241.125.53
22a09:8280:1::37:6bbc

6.3. putting it together: DNS & HTTP

We now have everything we need to for the basics of internet browsing: we can look up the IP address of a domain name, and we can connect to a server at an IP address and port.

When you type a URL into your browser, it does the following:

But wait, what’s HTTP? The HyperText Transfer Protocol is a text-based protocol for sending messages over the internet.

HTTP is not nearly as scary as it might appear. Let’s start with requests.

7. HTTP Requests

A HTTP Request is plain text, and looks like this:

1<METHOD>  <PATH>  <PROTOCOL/VERSION>
2Host: <HOST>
3[<HEADER>: <VALUE>]
4[<HEADER>: <VALUE>]
5[<HEADER>: <VALUE>] (these guys are optional)
6
7[<REQUEST BODY>] (this is also optional).

To give a more concrete example, the most basic HTTP request you could send to get this webpage would look like this:

1GET /backendbasics.html HTTP/1.1
2Host: eblog.fly.dev

(A couple of gotchas here: the line breaks are windows-style \r\n, not unix-style \n; and the request must end with a blank line.)

Let’s break this down. We can read this as

The first line is the REQUEST LINE. It has three parts:

The REQUEST LINE is followed by one or more HEADERS.

7.1. Headers

A HEADER is a key-value pair, separated by a colon (:). The key should be formatted in Title-Case, and the value should be formatted in lower-case; for example, Content-Type: application/json. A few headers have official meanings in the HTTP spec, but most are just suggestions to the server about how to handle the request. Technically, headers are MIME headers, but we have enough acronyms to deal with already; we’ll just call them headers for now.

The HOST header is required; it tells the server which domain name you’re trying to access. For this article, the HOST header is Host: eblog.fly.dev Other headers are optional, and can be used to send additional information to the server. Some common headers include:

Common Headers

header description example(s)
Accept-Encoding I can accept responses encoded with these encodings gzip, deflate
Accept the types of responses the client can accept text/html
Cache-Control how the client wants the server to cache the response no-cache
Content-Encoding my response body is encoded using: gzip, deflate
Content-Length my body is N bytes long 47
Content-Type the type of the request body application/json
Date the date and time of the request Tue, 17 Aug 2021 23:00:00 GMT
Host the domain name of the server you’re trying to access eblog.fly.dev
User-Agent the name and version of the client making the request curl/7.64.1, Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW)

Your browser sends a lot more headers than this: you can see them by opening the developer tools and looking at the network tab.

Here’s what chrome sent when I opened this page on the devtools network tab (that is, when I sent a GET request to https://eblog.fly.dev/backendbasics.html):

 1GET / HTTP/1.1
 2Host: eblog.fly.dev
 3Accept-Encoding: gzip, deflate, br
 4Accept-Language: en-US,en;q=0.9
 5Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7
 6Cache-Control: no-cache
 7Pragma: no-cache
 8Sec-Ch-Ua-Mobile: ?1
 9Sec-Ch-Ua-Platform: "Android"
10Sec-Ch-Ua: "Chromium";v="116", "Not)A;Brand";v="24", "Google Chrome";v="116"
11Sec-Fetch-Dest: document
12Sec-Fetch-Mode: navigate
13Sec-Fetch-Site: none
14Sec-Fetch-User: ?1
15Upgrade-Insecure-Requests: 1
16User-Agent: Mozilla/5.0 (Linux; Android 8.0.0; SM-G955U Build/R16NW) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.141 Mobile Safari/537.36

It’s OK to have multiple headers with the same key; for example, you might have multiple Accept-Encoding headers, each with a different encoding. Alternatively, you can separate multiple values with a comma.
That is, these two SHOULD be equivalent:

1Accept-Encoding: gzip
2Accept-Encoding: deflate

and

1Accept-Encoding: gzip, deflate

The server will usually pick the first one it understands. Servers are supposed to treat the keys as case-insensitive, but in practice this is not always the case; similarly, some web servers don’t properly handle multiple headers with the same key.

7.2. URL-encoding

Note that the HTTP request separates it’s sections using the following characters: ' ', '\r', '\n', :. That means we couldn’t use these characters in the request line or in headers without confusing the server; it wouldn’t know if we were trying to separate a section or give it literal text.

As such, URL paths and headers can’t contain these characters; we must ‘escape’ them using URL %-encoding before sending them to the server. URL-encoding is actually pretty simple: take any ASCII character, and turn it into it’s value in hexadecimal, prefixed with a %. For example, the space character is 0x20 in hexadecimal, so we encode it as %20. The percent character itself is 0x25, so we encode it as %25.

The following characters can ALWAYS be used in a URL path or header without escaping:

category characters
lowercase ascii letters abcdefghijklmnopqrstuvwxyz
uppercase ascii letters ABCDEFGHIJKLMNOPQRSTUVWXYZ
digits 0123456789
unreserved characters -._~
escaped % followed by two hexadecimal digits

But some characters can only be used unescaped in certain contexts:

character set context note
:/?#[]@ path i’ve never seen []; @ is for authentication
& query parameter separates query parameters
+ query parameter used to encode spaces in query parameters
= query parameter separates keys from values in query parameters
; path separates path segments; rarely used
$ path rarely used

Everything else must be escaped. For example, the following request path is valid:

1GET /backendbasics.html HTTP/1.1
2Host: eblog.fly.dev

but this one isn’t:

1GET /backend basics.html HTTP/1.1
2Host: eblog.fly.dev

And should be encoded as:

1GET /backend%20basics.html HTTP/1.1
2Host: eblog.fly.dev

The url.PathEscape and url.PathUnescape functions in the standard library can be used to escape and unescape a string for use in a URL path or header; we’ll cover that package in more detail in a later article.

7.3. Query Parameters

The PATH can also contain query parameters; these are key-value pairs in the form key=value that come after that path. You end the ‘normal’ part of the path with a ?, and then add the query parameters, separating each with a &.

If I want to make a google search for “backend_basics”, I would send the following request:

1GET /search?q=backend_basics HTTP/1.1
2Host: google.com

This has a single query parameter, with KEY q and VALUE backend_basics. I could add additional query parameters by separating them with &:

The scryfall API allows you to search for magic cards using a variety of query parameters: if I wanted to search for cards with the word “ice” in their name, ordered by their release date, I would send the following request:

1GET /search?q=ice&order=released&dir=asc HTTP/1.1

This would have three query parameters: “q=ice”, “order=released”, and “dir=asc”. Note that the = and & characters are not escaped in the query parameters.

That’s pretty much all there is to HTTP requests. Let’s try sending a HTTP request to eblog.fly.dev using TCP. The following complete program, sendreq, sends a HTTP request to a server at a given host, port, and path, and prints the response to stdout.

7.4. sending a HTTP request with sendreq.go

 1// sendreq sends a request to the specified host, port, and path, and prints the response to stdout.
 2// flags: -host, -port, -path, -method
 3package main
 4
 5import (
 6    "bufio"
 7    "flag"
 8    "fmt"
 9    "log"
10    "net"
11    "os"
12    "strings"
13)
14
15// define flags
16var (
17    host, path, method string
18    port               int
19)
20
21func main() {
22    // initialize & parse flags
23    flag.StringVar(&method, "method", "GET", "HTTP method to use")
24    flag.StringVar(&host, "host", "localhost", "host to connect to")
25    flag.IntVar(&port, "port", 8080, "port to connect to")
26    flag.StringVar(&path, "path", "/", "path to request")
27    flag.Parse()
28
29    // ResolveTCPAddr is a slightly more convenient way of creating a TCPAddr.
30    // now that we know how to do it by hand using net.LookupIP, we can use this instead.
31    ip, err := net.ResolveTCPAddr("tcp", fmt.Sprintf("%s:%d", host, port))
32    if err != nil {
33        panic(err)
34    }
35
36    // dial the remote host using the TCPAddr we just created...
37    conn, err := net.DialTCP("tcp", nil, ip)
38    if err != nil {
39        panic(err)
40    }
41
42    log.Printf("connected to %s (@ %s)", host, conn.RemoteAddr())
43
44    defer conn.Close()
45
46    var reqfields = []string{
47        fmt.Sprintf("%s %s HTTP/1.1", method, path),
48        "Host: " + host,
49        "User-Agent: httpget",
50        "", // empty line to terminate the headers
51
52        // body would go here, if we had one
53    }
54    // e.g, for a request to http://eblog.fly.dev/
55    // GET / HTTP/1.1
56    // Host: eblog.fly.dev
57    // User-Agent: httpget
58    //
59
60    request := strings.Join(reqfields, "\r\n") + "\r\n" // note windows-style line endings
61
62    conn.Write([]byte(request))
63    log.Printf("sent request:\n%s", request)
64
65    for scanner := bufio.NewScanner(conn); scanner.Scan(); {
66        line := scanner.Bytes()
67        if _, err := fmt.Fprintf(os.Stdout, "%s\n", line); err != nil {
68            log.Printf("error writing to connection: %s", err)
69        }
70        if scanner.Err() != nil {
71            log.Printf("error reading from connection: %s", err)
72            return
73        }
74    }
75
76}

Let’s try it out on the index page of this blog (running on localhost:8080):

1go build -o sendreq ./sendreq.go
2./sendreq -host eblog.fly.dev -port 8080

12023/09/07 13:59:19 connected to localhost (@ 127.0.0.1:8080)
22023/09/07 13:59:19 sent request:

1GET / HTTP/1.1
2Host: localhost
3User-Agent: httpget

And we get back a response, a redirect to /index.html:

1HTTP/1.1 308 Permanent Redirect
2Content-Type: text/html; charset=utf-8
3E-Req-Id: b641130b240142ae82ae8b122c35c80f
4E-Trace-Id: 086e9e55-364b-4cfd-b8fe-6497214af367
5Location: /index.html
6Date: Thu, 07 Sep 2023 20:59:19 GMT
7Content-Length: 47
8
9<a href="/index.html">Permanent Redirect</a>.

We’ll examine HTTP responses in the next section.

8. HTTP Responses

A HTTP response is also plain text, and looks like this:

1<PROTOCOL/VERSION> <STATUS CODE> <STATUS MESSAGE>
2[<HEADER>: <VALUE>] (these guys are optional)
3[<HEADER>: <VALUE>]
4[<HEADER>: <VALUE>]
5
6[<RESPONSE BODY>] (this is optional).

The first line is the STATUS LINE. It has three parts:

Headers work the same way as in requests: they’re key-value pairs separated by a colon (:), and the key should be formatted in Title-Case and the value should be formatted in lower-case. Headers of the response are usually ‘symmetric’ to the headers of the request; if you send an Accept-Encoding: gzip header, you’ll usually get a Content-Encoding: gzip header back.

The final section is the response body. Here, it’s some HTML that tells the browser to redirect to /index.html. Don’t worry, I’m not going to cover HTML: this is a backend article, not a frontend article.

Let’s follow the redirect and request /index.html:

1./sendreq -host eblog.fly.dev -port 8080 -path /index.html

We get back a 200 OK response and the (very sparse) contents of the index page:

 1HTTP/1.1 200 OK
 2E-Req-Id: 47cf0abba4fd4629a9a926769649f653
 3E-Trace-Id: dc2c9528-0322-4a16-8688-8ce760fff374
 4Date: Thu, 07 Sep 2023 21:04:28 GMT
 5Content-Length: 1300
 6Content-Type: text/html; charset=utf-8
 7
 8<!DOCTYPE html><html><head>
 9        <title>index.html</title>
10        <meta charset="utf-8"/>
11        <link rel="stylesheet" type="text/css" href="/dark.css"/>
12        </head>
13        <body>
14        <h1> articles </h1>
15<h4><a href="/performanceanxiety.html">performanceanxiety.html</a>
16</h4><h4><a href="/onoff.html">onoff.html</a>
17</h4><h4><a href="/fastdocker.html">fastdocker.html</a>
18</h4><h4><a href="/README.html">README.html</a>
19</h4><h4><a href="/mermaid_test.html">mermaid_test.html</a>
20</h4><h4><a href="/quirks3.html">quirks3.html</a>
21</h4><h4><a href="/console-autocomplete.html">console-autocomplete.html</a>
22</h4><h4><a href="/console.html">console.html</a>
23</h4><h4><a href="/cheatsheet.html">cheatsheet.html</a>
24</h4><h4><a href="/testfast.html">testfast.html</a>
25</h4><h4><a href="/quirks2.html">quirks2.html</a>
26</h4><h4><a href="/bytehacking.html">bytehacking.html</a>
27</h4><h4><a href="/benchmark_results.html">benchmark_results.html</a>
28</h4><h4><a href="/index.html">index.html</a>
29</h4><h4><a href="/noframework.html">noframework.html</a>
30</h4><h4><a href="/faststack.html">faststack.html</a>
31</h4><h4><a href="/backendbasics.html">backendbasics.html</a>
32</h4><h4><a href="/startfast.html">startfast.html</a>
33</h4><h4><a href="/quirks.html">quirks.html</a>
34</h4><h4><a href="/reflect.html">reflect.html</a>

This is OK, but dealing with raw HTTP requests and Responses is kind of a pain. Before we dive into Go’s net/http package, let’s think about how we might implement a HTTP library ourselves.

9. Building a HTTP library

We’d like a way to write our requests and responses without having to worry about the details of the protocol, like making sure our newlines are windows-style \r\n instead of unix-style \n or title-casing our headers.

That is, we’ll need four things, roughly in order of difficulty (easiest first):

9.1. The Request and Response structs

Restricting ourselves for now to HTTP 1.1, we can think of a HTTP request as a struct with the following fields:

 1
 2// Header represents a HTTP header. A HTTP header is a key-value pair, separated by a colon (:);
 3// the key should be formatted in Title-Case.
 4// Use Request.AddHeader() or Response.AddHeader() to add headers to a request or response and guarantee title-casing of the key.
 5type Header struct {Key, Value string}
 6// Request represents a HTTP 1.1 request.
 7type Request struct {
 8    Method string // e.g, GET, POST, PUT, DELETE
 9    Path string // e.g, /index.html
10    Headers []struct {Key, Value string}  // e.g, Host: eblog.fly.dev
11    Body string // e.g, <html><body><h1>hello, world!</h1></body></html>
12}

and a HTTP response as a struct with the following fields:

1type Response struct {
2    StatusCode int // e.g, 200
3    Headers []struct {Key, Value string}  // e.g, Content-Type: text/html
4    Body string // e.g, <html><body><h1>hello, world!</h1></body></html>
5}

The following functions will build a request or response for us:

 1func NewRequest(method, path, host, body string) (*Request, error) {
 2    switch {
 3    case method == "":
 4        return nil, errors.New("missing required argument: method")
 5    case path == "":
 6        return nil, errors.New("missing required argument: path")
 7    case !strings.HasPrefix(path, "/"):
 8        return nil, errors.New("path must start with /")
 9    case host == "":
10        return nil, errors.New("missing required argument: host")
11    default:
12        headers := make([]Header, 2)
13        headers[0] = Header{"Host", host}
14        if body != ""  {
15            headers = append(headers, Header{"Content-Length", fmt.Sprintf("%d", len(body))})
16        }
17        return &Request{Method: method, Path: path, Headers: headers, Body: body}, nil
18    }
19}
20
21func NewResponse(status int, body string) (*Response, error) {
22    switch {
23    case status < 100 || status > 599:
24        return nil, errors.New("invalid status code")
25    default:
26        if body == "" {
27            body = http.StatusText(status)
28        }
29        headers := []Header {"Content-Length", fmt.Sprintf("%d", len(body))}
30        return &Response{
31            StatusCode: status,
32            Headers: headers,
33            Body: body,
34        }, nil
35    }
36}

9.2. Headers

We’d like to be able to add headers to a request or response without worrying about casing of the keys. We’ll do this with a ‘builder’ method on *Request and *Response:

1func (resp *Response) WithHeader(key, value string) *Response {
2    resp.Headers = append(resp.Headers, Header{AsTitle(key), value})
3    return resp
4}
5func (r *Request) WithHeader(key, value string) *Request {
6    r.Headers = append(r.Headers, Header{AsTitle(key), value})
7    return r
8}

We can use these to build a request a header at a time:

1req, err := NewRequest("POST", "/api/v1/users", "eblog.fly.dev", `{"name": "eblog", "email": "efron.dev@gmail.com"}`)
2if err != nil {
3    panic(err)
4}
5req = req.WithHeader("Content-Type", "application/json").
6    WithHeader("Accept", "application/json").
7    WithHeader("User-Agent", "httpget")

But how is AsTitle implemented? Let’s write a quick test first to make sure we understand the requirements:

 1func TestTitleCaseKey(t *testing.T) {
 2    for input, want := range map[string]string{
 3        "foo-bar":      "Foo-Bar",
 4        "cONTEnt-tYPE": "Content-Type",
 5        "host":         "Host",
 6        "host-":        "Host-",
 7        "ha22-o3st":    "Ha22-O3st",
 8    } {
 9        if got := AsTitle(input); got != want {
10            t.Errorf("TitleCaseKey(%q) = %q, want %q", input, got, want)
11        }
12    }
13}

MIME headers are assumed to be ASCII-only, so we don’t need to worry about unicode here.

 1// AsTitle returns the given header key as title case; e.g. "content-type" -> "Content-Type"
 2// It will panic if the key is empty.
 3func AsTitle(key string) string {
 4    /* design note --- an empty string could be considered 'in title case',
 5    but in practice it's probably programmer error. rather than guess, we'll panic.
 6    */
 7    if key == "" {
 8        panic("empty header key")
 9    }
10    if isTitleCase(key) {
11        return key
12    }
13    /* ---- design note: allocation is very expensive, while iteration through strings is very cheap.
14    in general, better to check twice rather than allocate once. ----
15    */
16    return newTitleCase(key)
17}
18
19
20
21// newTitleCase returns the given header key as title case; e.g. "content-type" -> "Content-Type";
22// it always allocates a new string.
23func newTitleCase(key string) string {
24    var b strings.Builder
25    b.Grow(len(key))
26    for i := range key {
27
28        if i == 0 || key[i-1] == '-' {
29            b.WriteByte(upper(key[i]))
30        } else {
31            b.WriteByte(lower(key[i]))
32        }
33    }
34    return b.String()
35}
36
37
38// straight from K&R C, 2nd edition, page 43. some classics never go out of style.
39func lower(c byte) byte {
40    /* if you're having trouble understanding this:
41        the idea is as follows: A..=Z are 65..=90, and a..=z are 97..=122.
42        so upper-case letters are 32 less than their lower-case counterparts (or 'a'-'A' == 32).
43        rather than using the 'magic' number 32, we use 'a'-'A' to get the same result.
44    */
45    if c >= 'A' && c <= 'Z' {
46        return c + 'a' - 'A'
47    }
48    return c
49}
50func upper(c byte) byte {
51    if c >= 'a' && c <= 'z' {
52        return c + 'A' - 'a'
53    }
54    return c
55}
56
57
58
59// isTitleCase returns true if the given header key is already title case; i.e, it is of the form "Content-Type" or "Content-Length", "Some-Odd-Header", etc.
60func isTitleCase(key string) bool {
61    // check if this is already title case.
62    for i := range key {
63        if i == 0 || key[i-1] == '-' {
64            if key[i] >= 'a' && key[i] <= 'z' {
65                return false
66            }
67        } else if key[i] >= 'A' && key[i] <= 'Z' {
68            return false
69        }
70    }
71    return true
72}
73

We run the test and it passes, so we’re good to go. Compare to the actual standard library’s implementation of textproto.CanonicalMIMEHeaderKey; ours is essentially the same but doesn’t handle some corner cases and optimizations for common headers.

9.3. Writing the request to the wire

We’ll implement the io.WriterTo interface on both of these structs so we can efficiently write them to a net.Conn or other io.Writer.

 1// Write writes the Request to the given io.Writer.
 2func (r *Request) WriteTo(w io.Writer) (n int64, err error) {
 3    // write & count bytes written.
 4    // using small closures like this to cut down on repetition
 5    // can be nice; but you sometimes pay a performance penalty.
 6    printf := func(format string, args ...any) error {
 7        m, err := fmt.Fprintf(w, format, args...)
 8        n += int64(m)
 9        return err
10    }
11    // remember, a HTTP request looks like this:
12    // <METHOD>  <PATH>  <PROTOCOL/VERSION>
13    // <HEADER>: <VALUE>
14    // <HEADER>: <VALUE>
15    //
16    // <REQUEST BODY>
17
18    // write the request line: like "GET /index.html HTTP/1.1"
19    if err := printf("%s %s HTTP/1.1\r\n", r.Method, r.Path); err != nil {
20        return n, err
21    }
22
23    // write the headers. we don't do anything to order them or combine/merge duplicate headers; this is just an example.
24    for _, h := range r.Headers {
25        if err := printf("%s: %s\r\n", h.Key, h.Value); err != nil {
26            return n, err
27        }
28    }
29    printf("\r\n") // write the empty line that separates the headers from the body
30    err = printf("%s\r\n", r.Body) // write the body and terminate with a newline
31    return n, err
32}

Response has a nearly identical implementation:

 1func (resp *Response) WriteTo(w io.Writer) (n int64, err error) {
 2    printf := func(format string, args ...any) error {
 3        m, err := fmt.Fprintf(w, format, args...)
 4        n += int64(m)
 5        return err
 6    }
 7    if err := printf("HTTP/1.1 %d %s\r\n", resp.StatusCode, http.StatusText(resp.StatusCode)); err != nil {
 8        return n, err
 9    }
10    for _, h := range resp.Headers {
11        if err := printf("%s: %s\r\n", h.Key, h.Value); err != nil {
12            return n, err
13        }
14
15    }
16    if err := printf("\r\n%s\r\n", resp.Body); err != nil {
17        return n, err
18    }
19    return n, nil
20}

Sidenote: Go’s standard interfaces

Go has a number of standard interfaces that are used throughout the standard library. You’ve probably already seen io.Reader and io.Writer, but there are a lot more. Many functions in the standard library work better with types that implement these interfaces; for example, io.Copy will copy from an io.Reader to an io.Writer, but if the src implements io.WriterTo or the dst implements io.ReaderFrom, it will use those methods instead, which can be more efficient.

Similarly, fmt.Stringer is used to get a string representation of a type, and encoding.TextMarshaler is used to get a byte slice representation of a type in order to serialize it out across the network or to disk.

We’ll implement both of those interfaces on our Request and Response types for convenience and to make our tests easier to write.

All we need to do is call WriteTo and return the result:

1var _, _ fmt.Stringer = (*Request)(nil), (*Response)(nil) // compile-time check that Request and Response implement fmt.Stringer
2var _, _ encoding.TextMarshaler = (*Request)(nil), (*Response)(nil)
3func (r *Request) String() string { b := new(strings.Builder); r.WriteTo(b); return b.String() }
4func (resp *Response) String() string { b := new(strings.Builder); resp.WriteTo(b); return b.String() }
5func (r *Request) MarshalText() ([]byte, error) { b := new(bytes.Buffer); r.WriteTo(b); return b.Bytes(), nil }
6func (resp *Response) MarshalText() ([]byte, error) { b := new(bytes.Buffer); resp.WriteTo(b); return b.Bytes(), nil }

9.4. Parsing HTTP Requests and Responses

One last thing: we’d like to be able to parse HTTP requests and responses from text. This is a bit more complicated than writing them, but given what we’ve done so far, it should be relatively straightforward.

  1// ParseRequest parses a HTTP request from the given text.
  2func ParseRequest(raw string) (r Request, err error) {
  3    // request has three parts:
  4    // 1. Request linedd
  5    // 2. Headers
  6    // 3. Body (optional)
  7    lines := splitLines(raw)
  8
  9    log.Println(lines)
 10    if len(lines) < 3 {
 11        return Request{}, fmt.Errorf("malformed request: should have at least 3 lines")
 12    }
 13    // The first line is special.
 14    first := strings.Fields(lines[0])
 15    r.Method, r.Path = first[0], first[1]
 16    if !strings.HasPrefix(r.Path, "/") {
 17        return Request{}, fmt.Errorf("malformed request: path should start with /")
 18    }
 19    if !strings.Contains(first[2], "HTTP") {
 20        return Request{}, fmt.Errorf("malformed request: first line should contain HTTP version")
 21    }
 22    var foundhost bool
 23    var bodyStart int
 24    // then we have headers, up until an empty line.
 25    for i := 1; i < len(lines); i++ {
 26        if lines[i] == "" { // empty line
 27            bodyStart = i + 1
 28            break
 29        }
 30        key, val, ok := strings.Cut(lines[i], ": ")
 31        if !ok {
 32            return Request{}, fmt.Errorf("malformed request: header %q should be of form 'key: value'", lines[i])
 33        }
 34        if key == "Host" { // special case: host header is required.
 35            foundhost = true
 36        }
 37        key = AsTitle(key)
 38
 39        r.Headers = append(r.Headers, Header{key, val})
 40    }
 41    end := len(lines) - 1 // recombine the body using normal newlines; skip the last empty line.
 42    r.Body = strings.Join(lines[bodyStart:end], "\r\n")
 43    if !foundhost {
 44        return Request{}, fmt.Errorf("malformed request: missing Host header")
 45    }
 46    return r, nil
 47}
 48
 49
 50// ParseResponse parses the given HTTP/1.1 response string into the Response. It returns an error if the Response is invalid,
 51// - not a valid integer
 52// - invalid status code
 53// - missing status text
 54// - invalid headers
 55// it doesn't properly handle multi-line headers, headers with multiple values, or html-encoding, etc.
 56func ParseResponse(raw string) (resp *Response, err error) {
 57    // response has three parts:
 58    // 1. Response line
 59    // 2. Headers
 60    // 3. Body (optional)
 61    lines := splitLines(raw)
 62    log.Println(lines)
 63
 64    // The first line is special.
 65    first := strings.SplitN(lines[0], " ", 3)
 66    if !strings.Contains(first[0], "HTTP") {
 67        return nil, fmt.Errorf("malformed response: first line should contain HTTP version")
 68    }
 69    resp = new(Response)
 70    resp.StatusCode, err = strconv.Atoi(first[1])
 71    if err != nil {
 72        return nil, fmt.Errorf("malformed response: expected status code to be an integer, got %q", first[1])
 73    }
 74    if first[2] == "" || http.StatusText(resp.StatusCode) != first[2] {
 75        log.Printf("missing or incorrect status text for status code %d: expected %q, but got %q", resp.StatusCode, http.StatusText(resp.StatusCode), first[2])
 76    }
 77    var bodyStart int
 78    // then we have headers, up until an empty line.
 79    for i := 1; i < len(lines); i++ {
 80        log.Println(i, lines[i])
 81        if lines[i] == "" { // empty line
 82            bodyStart = i + 1
 83            break
 84        }
 85        key, val, ok := strings.Cut(lines[i], ": ")
 86        if !ok {
 87            return nil, fmt.Errorf("malformed response: header %q should be of form 'key: value'", lines[i])
 88        }
 89        key = AsTitle(key)
 90        resp.Headers = append(resp.Headers, Header{key, val})
 91    }
 92    resp.Body = strings.TrimSpace(strings.Join(lines[bodyStart:], "\r\n")) // recombine the body using normal newlines.
 93    return resp, nil
 94}
 95// splitLines on the "\r\n" sequence; multiple separators in a row are NOT collapsed.
 96func splitLines(s string) []string {
 97    if s == "" {
 98        return nil
 99    }
100    var lines []string
101    i := 0
102    for {
103        j := strings.Index(s[i:], "\r\n")
104        if j == -1 {
105            lines = append(lines, s[i:])
106            return lines
107        }
108        lines = append(lines, s[i:i+j]) // up to but not including the \r\n
109        i += j + 2 // skip the \r\n
110    }
111}

As before, let’s write a few quick tests to make sure we understand the requirements.

I’m omitting the error cases for brevity here; this article is more than long enough already.

 1func TestHTTPResponse(t *testing.T) {
 2    for name, tt := range map[string]struct {
 3        input string
 4        want  *Response
 5    }{
 6        "200 OK (no body)": {
 7            input: "HTTP/1.1 200 OK\r\nContent-Length: 0\r\n\r\n",
 8            want: &Response{
 9                StatusCode: 200,
10                Headers: []Header{
11                    {"Content-Length", "0"},
12                },
13            },
14        },
15        "404 Not Found (w/ body)": {
16            input: "HTTP/1.1 404 Not Found\r\nContent-Length: 11\r\n\r\nHello World\r\n",
17            want: &Response{
18                StatusCode: 404,
19                Headers: []Header{
20                    {"Content-Length", "11"},
21                },
22                Body: "Hello World",
23            },
24        },
25    } {
26        t.Run(name, func(t *testing.T) {
27            got, err := ParseResponse(tt.input)
28            if err != nil {
29                t.Errorf("ParseResponse(%q) returned error: %v", tt.input, err)
30            }
31            if !reflect.DeepEqual(got, tt.want) {
32                t.Errorf("ParseResponse(%q) = %#+v, want %#+v", tt.input, got, tt.want)
33            }
34
35            if got2, err := ParseResponse(got.String()); err != nil {
36                t.Errorf("ParseResponse(%q) returned error: %v", got.String(), err)
37            } else if !reflect.DeepEqual(got2, got) {
38                t.Errorf("ParseResponse(%q) = %#+v, want %#+v", got.String(), got2, got)
39            }
40
41        })
42    }
43}
44
45func TestHTTPRequest(t *testing.T) {
46    for name, tt := range map[string]struct {
47        input string
48        want  Request
49    }{
50        "GET (no body)": {
51            input: "GET / HTTP/1.1\r\nHost: www.example.com\r\n\r\n",
52            want: Request{
53                Method: "GET",
54                Path:   "/",
55                Headers: []Header{
56                    {"Host", "www.example.com"},
57                },
58            },
59        },
60        "POST (w/ body)": {
61            input: "POST / HTTP/1.1\r\nHost: www.example.com\r\nContent-Length: 11\r\n\r\nHello World\r\n",
62            want: Request{
63                Method: "POST",
64                Path:   "/",
65                Headers: []Header{
66                    {"Host", "www.example.com"},
67                    {"Content-Length", "11"},
68                },
69                Body: "Hello World",
70            },
71        },
72    } {
73        t.Run(name, func(t *testing.T) {
74            got, err := ParseRequest(tt.input)
75            if err != nil {
76                t.Errorf("ParseRequest(%q) returned error: %v", tt.input, err)
77            }
78            if !reflect.DeepEqual(got, tt.want) {
79                t.Errorf("ParseRequest(%q) = %#+v, want %#+v", tt.input, got, tt.want)
80            }
81            // test that the request can be written to a string and parsed back into the same request.
82            got2, err := ParseRequest(got.String())
83            if err != nil {
84                t.Errorf("ParseRequest(%q) returned error: %v", got.String(), err)
85            }
86            if !reflect.DeepEqual(got, got2) {
87                t.Errorf("ParseRequest(%q) = %+v, want %+v", got.String(), got2, got)
88            }
89
90        })
91    }
92}

We run the tests and they pass, so we’re good to go. This should give you a pretty good idea of how HTTP works under the hood.

10. Conclusion

You’re rarely going to directly parse HTTP, but when things go wrong it’s important to know how they actually work. The relative simplicity of the protocol should raise some eyebrows when you compare it to the incredibly overengineered complexity of the modern web. In the next article, we’ll start diving in to how to deal with HTTP ‘the real way’ and dive into the standard library’s net/http package.

Like this article? Need help making great software, or just want to save a couple hundred thousand dollars on your cloud bill? Hire me, or bring me in to consult. Professional enquiries at
efron.dev@gmail.com or linkedin