# A5 - HTTP Server

{% hint style="info" %}

### Run the server (your implementation)

```bash
python3 -m a5_http_server
```

{% endhint %}

How many websites do you visit per day? Do you need a specialized program for every one of them? Probably not. It turns out that it has not always been the same. Before the invention of HTTP protocol (the backbone of the web, by Tim Berners Lee) people used different programs to access different parts of the web. What made it possible was the unification of application-layer web protocols.

However, the complexities of dealing with HTTP are usually hidden behind the abstraction of web server frameworks like Node.js, Flask, Django, and others. *Not today!* In this assignment, you will explore what happens behind the curtains by implementing a simplified HTTP server.

***

## Assignment (350 points)

Your job is to implement a simplified HTTP 1.1 server supporting `GET` methods. The website, based on HTML files and images we provide, should be loadable from your server and be visible in any browser. Please note, that while most modern browsers will render you an HTML document even if was sent by a protocol other than HTTP (for example FTP), for this assignment you are required to use HTTP for **all** client-server communication.

We provide a minimal webpage that your implementation must be able to statically serve. Your server must support `GET` requests and be able to serve different types of files typically found on a webpage, such as jpeg images, HTML, and CSS files. Aditionally, your server shall support different status codes, such as 200, 201, 400, and 404 and serve the correct HTML file depending on the response status code.

***

## Specification

The main document you will be working with is [RFC 2616](https://datatracker.ietf.org/doc/html/rfc2616), the specification for the HTTP/1.1 protocol. You will not have to implement the entire HTTP/1.1 specification, however some sections of interest are:

* **Section 1 - Introduction**\
  A general overview of the HTTP/1.1 protocol and its purpose.&#x20;
* **Section 5.1 & 5.2 - Request Line**\
  Learn how to parse requests sent by browsers, the message format and its standard contents.
* **Section 9.3 - The** `GET` **method**
* **Section 3.2 - HTTP Version**
* **Section 7.2.1 - Type**\
  Specify the content type served, i.e., jpeg, html, and so on.
* **Section 7.2.2 - Entity Length**\
  Specify the content length, required for determining the end of the transmitted data.
* **Sections 10.2.1 - 10.4.1 - Status Codes**\
  Descriptions on status codes you will have to supoort. Note that we will only require a subset of the presented status codes.
* **Section 8 - Connections**\
  **S**upport for persistent connections (sk yourself: why were those important to add?). Without the persistent connections support, your browser will not likely load your HTTP 1.1 page.
* **Section 8.2.3 & 8.2.4 - Use of the 100 (Continue) Status & Client Behavior if Server Prematurely Closes Connection**\
  Learn about the ways you should deal with connections based on their state.<br>

{% hint style="warning" %}
RFCs may seem overly complex, holding trumendous amounts of information that one cannot process in one sitting. The key to using RFCs to their whole potential is treating them as a tool rather than a book to read. Use RFCs to guide your development process rather than a prerequisite read to starting to write code.

That being said, we recommend reading at least once the Introduction section to familiarize yourself with the protocol.
{% endhint %}

***

## Requirements

Your assignment must implement a subset of the features described in [RFC 2616](https://datatracker.ietf.org/doc/html/rfc2616). In short, your server should implement functionality that enables statically serving any webpage. We define the following requirements as a smaller specification based on the RFC to guide your development process.

{% tabs %}
{% tab title="Technical" %}
The following requirements indicate the protocol specification the server should adhere to, as well as technical requirements regarding implementation.

1. The server must support HTTP 1.1.
2. The server must support the `GET` method.
3. The server must support HTTP 1.1 persistent connections.
4. The implementation must not use either of the `exit()` or `sendall()` Python functions.
5. The server must support multiple concurrent connections.
   {% endtab %}

{% tab title="Responses" %}
The following requirements describe the subset of features that should be implemented from RFC 2616.

1. The server has to support the 200, 400, and 404 status codes.
2. The server must respond, indicating an appropriate response type in the header.
3. The server must indicate the appropriate response length in the header.
4. The server must indicate the appropriate protocol type (HTTP 1.1) in the header.
5. The server must use UTF-8 encoding and indicate it in every response header.
6. If a page is not found, the server must serve the file `404.html`.
7. When requested with `/`, the server must respond with the `index.html` page.
8. The server must be able to load any file from the filesystem in response to a `GET` request, including images, PDF documents, etc.
9. The server must support valid content types for the following file extensions: `.pdf`, `.jpeg`, `.html`, `.js`, `.css`, `.png`.
   {% endtab %}
   {% endtabs %}

***

## Bonus (250 points)

As HTTP is rapidly being adopted into more and more application-layer software, we start noticing the need for performant HTTP implementations. Namely, we want to be able to handle hundreds of thousands of concurrent connections and serve all of them with high throughput and low latency. Many web servers are currently competing for the [number one spot in performance](https://www.techempower.com/benchmarks/#section=data-r23), but can we do the same?&#x20;

Your task is to adapt your HTTP/1.1 server to serve a dynamic endpoint that gets updated through a `POST` request form. In doing so, consider performance, specifically handling many concurrent connections efficiently. Practice shows that handling each connection in a new thread will not get us far.

### Part 1: Dynamic endpoint & `POST` requests

The `personal_cats.html`  file contains a `<table>`  entry, which is currently being statically served. You will have to update the file to include a special placeholder that you will then replace with the personal cat **image links** of each user. Each connection may submit different pictures to be included in the table, therefore, each connection should see a different version of the `personal_cats.html` file upon requesting it.&#x20;

Submitting the image links should be done through a `POST /data` request, using a `application/x-www-form-urlencoded`  encoding with the following fields:

<table><thead><tr><th width="184.142822265625">Field Name</th><th width="249.99993896484375">Description</th></tr></thead><tbody><tr><td>cat_url</td><td>An url (link) of cat picture.</td></tr><tr><td>description</td><td>The description of the picture.</td></tr></tbody></table>

The table entry in the `personal_cats.html` file should be replaced as shown below. Your implementation should replace `TABLE_PLACEHOLDER`  with the `<tr><td>...</td></tr>`  entries containing the images stored by each user.&#x20;

{% tabs %}
{% tab title="Before" %}
{% code lineNumbers="true" %}

```html
        <table style="text-align: center; width: 100%">
<!--            The contents of this table should be dynamically generated by the server.-->
<!--            The following are just examples of how the table should look like.-->
<!--            You will have to remove the contents of this table and dynamically generate the rows.-->
            <tr>
                <td>
                    <img src ="Here should be your source url" width="300"/>
                </td>
            </tr>
            <tr>
                <td>Cat name 1</td>
            </tr>
            <tr>
                <td>
                    <img src ="Here should be your source url" width="300"/>
                </td>
            </tr>
            <tr>
                <td>Cat name 2</td>
            </tr>
        </table>
```

{% endcode %}
{% endtab %}

{% tab title="After" %}

```html
 <table style="text-align: center; width: 100%">TABLE_PLACEHOLDER</table>
```

{% endtab %}
{% endtabs %}

{% hint style="success" %}
The state should not be persistently stored. It is ok if the images are gone if the user restarts their browser or the server is restarted.
{% endhint %}

### Part 2: Handling concurrent connections efficiently

To handle many connections concurrently without losing performance, you will have to either implement a [thread pool](https://en.wikipedia.org/wiki/Thread_pool) consumer algorithm or implement your HTTP server without using any threads at all, instead using the [epoll](https://docs.python.org/3/library/select.html#select.epoll) interface. Epoll is a high-performance polling interface used in most top-tier web servers. You may choose which implementation you want to use, however, we expect a thread pool system to be cleverly designed to reuse threads after a connection closes.

{% hint style="warning" %}
If you are on MacOS, the epoll interface is not available to you, therefore, you will have to resort to implementing a thread pool.
{% endhint %}

Your implementation will be judged by the grading teaching assistant. If the implementation is not considered to meet the performance requirements of the bonus assignment, the submission may be rejected. There are **no automated tests** for performance testing on CodeGrade.&#x20;

### Bonus specification

{% tabs %}
{% tab title="Part 1" %}
The following requirements describe the expected behavior of the dynamic endpoints for the first part of the bonus. Please note that the `personal_cats.html` file will be treated as an endpoint following these requirements.

1. The server must support `POST` requests on the `/data` endpoint.
2. The /data endpoint must properly implement `application/x-www-form-urlencoded` using the prescribed fields.
3. The server must support dynamic `GET` requests on the `/personal_cats.html` endpoint. Namely, each connection should be able to see their images but not others'.
4. The server must implement the 201 status code and serve the `success.html` page with it upon successful access of the `/data` endpoint.
5. The server must support the status code 400 and serve the `400.html` page with it upon a failed access of the `/data` endpoint.
6. The `/data` endpoint must check if the fields are empty and display an appropriate error if erroneous fields are inputted.
7. The `/personal_cats.html` endpoint must support replacing the `TABLE_PLACEHOLDER`  placeholder, regardless of the file or context it is found in.
   {% endtab %}

{% tab title="Part 2" %}
The following requirements describe the expected behavior of the server regarding its performance in handling many concurrent connections.

1. The server should either implement a thread pool or use the epoll interface for handling multiple connections.
2. If using thread pools, the server should reuse threads that are no longer in use after a client disconnects.
3. If using epoll, the server should not spawn any new threads apart from the main thread.&#x20;
4. The server should be able to handle 1000 concurrent `curl` requests gracefully.
   {% endtab %}
   {% endtabs %}

To qualify for the **50 bonus points for code quality**, your implementation must be easily scalable to an indefinite number of endpoints, as well as being able to demonstrate its performance capabilities with several hundreds of parallel `curl` requests (at least as performant as Python can get).

### Testing performance

To test the performance of your HTTP server you want to create several thousand connections in parallel  and time your server's response time. If your server cannot handle many connections in parallel the wait time will be longer. The `time` command will help us determine the execution time of all `NUM_CONNECTIONS`  parallel requests. We will run the [curl](https://curl.se/) command in parallel by using the `&` control operator. &#x20;

Execute the script below to test your server's performance. Adjust the `NUM_CONNECTIONS`  variable according to your test's magnitude. For comparison, our sample implementation can handle 3000 connections in 6.37s user 12.62s system 839% cpu 2.263 total.

```bash
export NUM_CONNECTIONS=3000
time (
    for i in $(seq 1 $NUM_CONNECTIONS); do 
        curl -s http://localhost:8000 >/dev/null & 
    done
)
```

{% hint style="warning" %}
The test mentioned above is **not integrated** in the automated testing environment. To pass the performance criteria, you must demonstrate during the sign-off that your implementation can serve the required number of parallel connections.
{% endhint %}
