Introduction

Have you ever wanted to download something? Of course you have. But have you wanted to download anything? HTTP files, FTP archives, BitTorrent swarms, IPFS content hashes, S3 buckets—all through one tool?

That's what we're building. A download manager that doesn't care what protocol you throw at it.

What This Book Is

This is not a Rust tutorial. The Rust Book exists for that. This is a book about building a complex, multi-protocol system in Rust while learning the hard parts of async programming, protocol implementation, and production architecture along the way.

I'm writing this as I build it. You'll see the decisions, the mistakes, the refactors. My mom always said the best way to learn is to write, so here we are.

What We're Building

dlm - a download manager daemon that speaks:

• HTTP/HTTPS - The foundation (concurrent chunks, resume, range requests)
• FTP/FTPS - The 1971 protocol that refuses to die
• BitTorrent - P2P file sharing with tracker and DHT support
• IPFS - Content-addressed downloads
• S3 - Cloud object storage (and S3-compatible services)

All coordinated through a daemon architecture with REST APIs, worker pools, job queues, and production-grade observability.

The Journey

Chapter 1 starts simple: download a file over HTTP. By the end, you'll have concurrent chunk downloads, progress tracking, pause/resume, and SHA256 verification.

Chapter 2 transforms the CLI tool into a daemon. We build the architecture that everything else plugs into: the protocol abstraction layer, worker pools, state management, APIs, and logging infrastructure.

Chapters 3-6 implement protocols. Each chapter adds a new protocol to the daemon, showing how the abstraction handles wildly different download mechanisms.

Chapter 7 reflects on what worked, what didn't, and what I'd do differently.

What You'll Learn

• Async Rust patterns with tokio
• Protocol implementation (from simple HTTP to complex P2P)
• Production architecture (daemons, APIs, observability)
• State management and concurrency
• Real-world trade-offs and decisions

Prerequisites

You should know Rust basics. If you've read the Rust Book and written some async code, you're ready. If not, go do that first.

You should have cargo and a terminal. We're building a CLI/daemon tool, not a GUI.

Before You Start

This is going to be messy. Protocols have sharp edges. Async Rust has footguns. BitTorrent will kick your ass. But by the end, you'll have built something genuinely complex and impressive.

Ready?

Let's write a download manager, damn it.

Meet Mack

Throughout this book, Mack will guide you through different concepts using various expressions. Here are all 12 unique ways Mack can help you:

1. Note

2. Abstract/Summary

3. Info

4. Tip

5. Success

6. Question

7. Warning

8. Failure

9. Danger

10. Bug

11. Example

12. Quote

Now that you've met Mack and all his 12 unique expressions, let's dive into building that download manager!

Chapter 1 - What is a download anyway?

It's 2006, you're downloading a file from the internet. Let's assume that you're downloading a Linux ISO, you know, legal stuff.

You get the link, which looks like this.

https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86_64/alpine-standard-3.23.0-x86_64.iso

This is a static download, a file that's served over HTTP, downloadable using your browser, curl or wget. Downloading it is simply sending a HTTP request and receiving a response with some headers and a body, where said body is a stream of bytes representing a file.

Now, how would you write something in Rust that does just that? It should download this file and write to disk. And since downloads are rather annoying, it should also verify said file, using sha256sum.

A Simple Download CLI

First Steps

First, let's just write the "core" functionality for the above URL.

use reqwest::blocking::get;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    const URL: &str =
        "https://dl-cdn.alpinelinux.org/alpine/v3.23/releases/x86_64/alpine-standard-3.23.0-x86_64.iso";

    let response = get(URL)?;
    let bytes = response.bytes()?;
    std::fs::write("alpine-standard-3.23.0-x86_64.iso", bytes)?;

    Ok(())
}

We're using the reqwest crate, with the reqwuest::blocking::get method. For now, let's just download this file using a single-threaded approach, it is fairly inefficient though. We'll update it later, but for now, this is sufficient.

CLI With clap

That's fairly simple. Now, let's add a CLI interface so we don't have to hardcode the URL. Here’s the same logic, but driven by a tiny clap CLI so the URL (as well as a target directory and overwrite flag) come from the user.

use clap::Parser;
use reqwest::blocking::get;

#[derive(Parser)]
struct Cli {
    /// URL to download
    url: String,

}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let response = get(&cli.url)?;
    let bytes = response.bytes()?;
    let filename = cli.url.split('/').last().unwrap_or("download.bin");
    std::fs::write(&filename, bytes)?;
    println!("Downloaded to {}", destination.display());
    Ok(())
}

You'll notice the cli.url.split('/').last().unwrap_or("download.bin"); section immediately. This only falls back when the url string is empty. The CLI requires a non-empty URL argument, so this almost never triggers. but this prevents against an empty string, like when invoking the binary with --url "" perhaps.

Additional Arguments

Now, let's add some niceties to this command line interface. We could have an argument that sets the directory you'd download said file to, and perhaps one to choose to overwrite the file if it exists.

use clap::Parser;
use reqwest::blocking::get;

#[derive(Parser)]
struct Cli {
    /// URL to download
    url: String,

    /// Target directory for the download
    #[arg(long, default_value = ".download")]
    target_dir: String,

    /// overwrite existing file
    #[arg(long)]
    overwrite: bool,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let response = get(&cli.url)?;
    let bytes = response.bytes()?;
    let filename = cli.url.split('/').last().unwrap_or("download.bin");

    let target_dir = std::path::Path::new(&cli.target_dir);
    std::fs::create_dir_all(target_dir)?;
    let destination = target_dir.join(filename);

    if destination.exists() && !cli.overwrite {
        return Err("file already exists; add --overwrite to replace".into());
    }

    std::fs::write(&destination, bytes)?;
    println!("Downloaded to {}", destination.display());
    Ok(())
}

While this works and we could stop here, it'd be great to add a progress bar to see what's being downloaded.

Reporting Progress

The indicatif crate is really useful if you want to play with progress bars. To make the download manager a little more user-friendly, let's add a small spinner that shows off how much of the file is downloaded and what's pending.

use clap::Parser;
use indicatif::{ProgressBar, ProgressStyle};
use reqwest::blocking::get;
use std::fs::File;
use std::io::{Read, Write};
use std::path::Path;

#[derive(Parser)]
struct Cli {
    /// URL to download
    url: String,

    /// Target directory for the download
    #[arg(long, default_value = ".download")]
    target_dir: String,

    /// Overwrite existing file
    #[arg(long)]
    overwrite: bool,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    let response = get(&cli.url)?;
    let total_bytes = response.content_length().unwrap_or(0);
    let bar = ProgressBar::new(total_bytes);
    bar.set_style(
        ProgressStyle::default_bar()
            .template("{spinner} [{bar:40.cyan/blue}] {bytes}/{total_bytes} ({bytes_per_sec})")
            .unwrap(),
    );

    let filename = cli.url.split('/').last().unwrap_or("download.bin");
    let target_dir = Path::new(&cli.target_dir);
    std::fs::create_dir_all(target_dir)?;
    let destination = target_dir.join(filename);

    if destination.exists() && !cli.overwrite {
        let message = format!("File exists at: '{}'", destination.display());
        return Err(message.into());
    }

    let mut file = File::create(&destination)?;
    let mut stream = response;
    let mut buffer = [0_u8; 32 * 1024];

    while let Ok(bytes_read) = stream.read(&mut buffer) {
        if bytes_read == 0 {
            break;
        }
        file.write_all(&buffer[..bytes_read])?;
        bar.inc(bytes_read as u64);
    }

    bar.finish_with_message("download complete");
    Ok(())
}

There are still some improvements we can make here.

Bringing it All Together

use hex;
use indicatif::{HumanBytes, ProgressBar};
use sha2::{Digest, Sha256};
use std::fs::{self, OpenOptions};
use std::io::{Read, Write};
use std::path::Path;
use std::sync::{
    Arc,
    atomic::{AtomicBool, Ordering},
};
use std::time::Duration;
use std::time::Instant;
use clap::Parser;

/// Download manager application.
#[derive(Parser)]
#[command(version, about, long_about=None)]
struct Cli {
    /// URL to a file to download
    url: String,

    /// Target directory
    #[arg(short, long, default_value = ".download")]
    target_directory: String,

    /// Download chunk size
    #[arg(short, long, default_value_t = 65_536)]
    chunk_size: usize,

    /// Resume if the file already exists and isn't complete
    #[arg(short, long)]
    resume: bool,

    /// Overwrite existing file
    #[arg(short, long)]
    overwrite: bool,
}

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();

    let target_file = cli.url;
    let target_dir = Path::new(&cli.target_directory);
    fs::create_dir_all(target_dir)?;

    let fname = target_file.split('/').last().unwrap_or("tmp.bin");
    let fname = target_dir.join(fname);
    println!("File to download: '{}'.", fname.to_str().unwrap());
    let mut dest = if fname.exists() && fname.is_file() {
        if cli.overwrite {
            // Overwrite is on, so need to truncate the file.
            let message = format!("File exists at: '{}' overwriting.", fname.to_str().unwrap());
            println!("{}", message);

            OpenOptions::new()
                .read(true)
                .write(true)
                .truncate(true)
                .open(&fname)?
        } else if cli.resume {
            let message = format!(
                "File exists at: '{}', attempting to resume.",
                fname.to_str().unwrap()
            );
            println!("{}", message);
            OpenOptions::new().read(true).append(true).open(&fname)?
        } else {
            let message = format!("File exists at: '{}'", fname.to_str().unwrap());
            return Err(message.into());
        }
    } else {
        // File doesn't exist yet.
        OpenOptions::new()
            .read(true)
            .write(true)
            .create(true)
            .open(&fname)?
    };

    println!("File will be downloaded to: '{}'.", fname.to_str().unwrap());
    let resume_from = if fname.exists() {
        fs::metadata(&fname)?.len() as usize
    } else {
        0
    };

    let mut response = if resume_from > 0 {
        println!(
            "Resuming downloading from {}.",
            HumanBytes(resume_from as u64)
        );
        let resp = reqwest::blocking::Client::new()
            .get(&target_file)
            .header("Range", format!("bytes={}-", resume_from))
            .send()?;
        match resp.status().as_u16() {
            206 => {
                // partial content, resume working
                resp
            }
            416 => {
                // Range not satisfiable, file is likely complete.
                println!("File appears to be complete.");
                return Err("File already complete".into());
            }
            200 => {
                eprintln!("Server doesn't support resume for this file. Try `--overwrite`");
                return Err("Cannot resume - server sent full file.".into());
            }
            _ => {
                return Err(format!("Unexpected status: {}", resp.status()).into());
            }
        }
    } else {
        reqwest::blocking::get(&target_file)?
    };

    let mut hasher = Sha256::new();
    let chunk_size = cli.chunk_size;
    let content_length = response.content_length();
    let mut downloaded = resume_from;
    if resume_from > 0 {
        let mut existing_file = fs::File::open(&fname)?;
        let mut buffer = vec![0; chunk_size];
        loop {
            let bytes_read = existing_file.read(&mut buffer)?;
            if bytes_read == 0 {
                break;
            }
            hasher.update(&buffer[..bytes_read]);
        }
    };
    let bar = ProgressBar::new_spinner();
    let interrupted = Arc::new(AtomicBool::new(false));
    let interrupted_clone = interrupted.clone();
    ctrlc::set_handler(move || {
        interrupted_clone.store(true, Ordering::SeqCst);
    })
    .expect("Could not set ctrlc handler");

    bar.enable_steady_tick(Duration::from_millis(100));
    let mut last_update = Instant::now();
    let start_time = Instant::now();
    loop {
        let mut buffer = vec![0; chunk_size];
        let data = response.read(&mut buffer[..])?;
        if data == 0 {
            break;
        }
        downloaded += data;
        if interrupted.load(Ordering::SeqCst) {
            break;
        }
        if last_update.elapsed() >= Duration::from_secs(1) {
            let speed = (downloaded - resume_from) as u64 / start_time.elapsed().as_secs().max(1);
            match content_length {
                Some(len) => bar.set_message(format!(
                    "Downloaded {}/{}. Speed: {} per second.",
                    HumanBytes(downloaded as u64),
                    HumanBytes(len),
                    HumanBytes(speed),
                )),
                None => bar.set_message(format!(
                    "Downloaded {}. Speed: {} per second.",
                    HumanBytes(downloaded as u64),
                    HumanBytes(speed)
                )),
            };
            last_update = Instant::now();
        }
        hasher.update(&buffer[..data]);
        dest.write_all(&mut buffer[..data])?;
    }
    dest.sync_all()?;
    if interrupted.load(Ordering::SeqCst) {
        match content_length {
            Some(len) => bar.abandon_with_message(format!(
                "Download interrupted at {}/{}.",
                HumanBytes(downloaded as u64),
                HumanBytes(len),
            )),
            None => bar.abandon_with_message(format!(
                "Download interrupted at {}",
                HumanBytes(downloaded as u64)
            )),
        }
        eprintln!("Download cancelled!");
        return Err("Download cancelled by user.".into());
    }
    let speed = downloaded as u64 / start_time.elapsed().as_secs().max(1);
    bar.finish_with_message(format!(
        "Downloaded {} at {} per second.",
        HumanBytes(downloaded as u64),
        HumanBytes(speed)
    ));
    let file_metadata = fs::metadata(&fname)?;
    assert_eq!(file_metadata.len(), downloaded as u64);

    let result = hex::encode(hasher.finalize());
    println!("Sha256sum: {:?}", result);
    Ok(())
}

Downloading in Parallel

Now, let's rethink this problem. I said earlier that the reqwest::blocking::get is inefficient and we need to rethink it.

Gone, gone the form of man, rise the Daemon

Chapter 3 - FTP or Bust

BitTorrent: 2000 Lines of Controlled Chaos

IPFS: Files Have Fingerprints Now, Apparently

S3 and Friends: The Cloud Wants Your Files

What I Learned (And What I'd Do Differently)