Handle large file upload with Drogon web framework

=> home

Sometimes, you just need to handle large file uploads in your web application. Let's say you are writing a web drive. Dealing witch chunked upload can be a pain in most web frameworks. Not in Drogon - not well-known since the maintainers doesn't advertise much - Drogon supports processing large request bodies out of the box, transparently. Potentally much larger than the physical memory of the server.

No BS. I'll just show you the code.

Server side

Drogon supports uploading large files by storing overly large requests in a temporary fil. Then memory map the file thus content can be used directly like usual. This is controlled by a few options.

app().setUploadPath("/path/to/writable/directory") 
    // temporary file will be stored in /path/to/writable/directory/tmp
    .setClientMaxBodySize(256*1024*1024)  // Max 256MiB body size
    .setClientMaxMemoryBodySize(1024*1024) // 1MiB before writing to disk

The name is a bit confusing. MaxMemoryBodySize is the maximum size of the body we allow to store in memory. After that we dump contents into a temporary file, up to MaxBodySize. The temporary file is deleted when the request is released. You'll have to adjust these two settings to fit your needs. Too small then small API calls will also write to disk, slowing down your server. Too large then you use too much memory.

Then in your endpoint. Use MultiPartParser to parse the request.

MultiPartParser fileUpload;
bool ok = fileUpload.parse(req);
if (!ok) {
    // Handle error
    return;
}

auto files = fileUpload.getFiles();
for(auto& file : files) {
    // Do something with file, ex: save them
    file.saveAs("/path/to/save/to/filename"); // save to specified path
    file.saveAs("filename"); // save with custom name
    file.save(); // Save with name provided by uploader
}

• I can't stress this enough*. Large upload files are stored as a temporary file on disk. And the maximum memory a request consumes is MaxMemoryBodySize plus some overhead for header, internal buffers, etc..

Compressed body

As of writing this post. Dorgon's master branch supports transparent compressed requests. If a compressed request is sent to the server, it will decompress the body and store it in memory or a temporary file. The same rule of MaxMemoryBodySize and MaxBodySize applies. To enable this, set enableCompressedRequest to true (default is not enabled for compatiablity).

app().enableCompressedRequest(true);

Upload files as client

Drogon also supports uploading files to remote server. To upload efficiently, use UploadFile and HttpRequest::newFileUploadRequest(). This enables the HTTP client to send the file in chunks, without having to load the entire file into memory.

UploadFile file("/path/to/file");
// You can upload more than one file
auto req = HttpRequest::newFileUploadRequest({file});

co_await client->sendRequestCoro(req);

That's it. It's super easy.