In the last log, we successfully sent metadata of the file to the server. Now we will proceed with chunking the file and sending over these chunks to the server. But first let’s explore some questions I had while researching this topic.
What is chunking?
Chunking is the very simple process by which we read the file as bits and pieces or “chunks” in relatively smaller sizes, in our case 1048 bytes or 1 KB. These chunks are then sent to the server one after the other, allowing the receiver to reconstruct the original file.
Why chunking?
There are a few reasons why chunking is a good idea, instead of sending the whole file through all at once. The first very obvious reason being, if the network connection was interrupted, chunking makes it easier to resume the transfer from the last chunk that was received successfully. We can also use the number of chunks received to make calculations to display a progress bar, furthering user experience. Another reason we use chunking is that we can parallelize the process and send multiple chunks at the same time. We will not be implementing any parallelizing soon, maybe after the completion of the first version, we could implement something along those lines. So let’s get started!
Throwing the chunks
Let’s start by calculating the number of chunks on both the client and the server side. Starting with the client side, let’s first define a constant CHUNK_SIZE: usize = 1048. We can use the constant, along with the file length to calculate the variables,
let partial_chunk_size = file_length % CHUNK_SIZE as u64;
let chunk_count = file_length / CHUNK_SIZE as u64 + (partial_chunk_size > 0) as u64;With these variables, we can starting reading the chunks and sending them over to the server side like so
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
// Snip Snip
loop {
// Snip Snip
let mut file = File::open(file_path)?;
let mut buffer = vec![0; CHUNK_SIZE];
for count in 0..chunk_count {
let bytes_read = file.read(&mut buffer)?;
if bytes_read == 0 {
break;
}
stream.write_all(&buffer).await?;
println!("Sent chunk {}/{}", count, chunk_count);
}
}
Ok(())
}Catching the chunks
Over at the server, we can write a function handle_client which does exactly what it says it does.
async fn handle_client(socket: &mut TcpStream) -> Result<(), Box<dyn std::error::Error>> {
let mut buffer = vec![0; CHUNK_SIZE];
// Read metadata (file name and size)
let bytes_read = socket.read(&mut buffer).await?;
if bytes_read == 0 {
return Ok(()); // Client disconnected
}
// Extract metadata
let (file_name, file_size) = {
let metadata = String::from_utf8_lossy(&buffer[..bytes_read]);
let parts: Vec<&str> = metadata.split(':').collect();
if parts.len() != 2 {
return Err("Invalid metadata format".into());
}
let file_name = parts[0].trim().to_string();
let file_size: u64 = parts[1].trim().parse()?;
(file_name, file_size)
};
println!("Receiving file: {} ({} bytes)", file_name, file_size);
// Create a file to save the incoming data
let mut file = tokio::fs::File::create("new_".to_owned() + &file_name).await?;
// Receive chunks and write to file
let mut total_bytes_received = 0;
while total_bytes_received < file_size {
let bytes_read = socket.read(&mut buffer).await?;
if bytes_read == 0 {
println!("Client disconnected unexpectedly");
break;
}
file.write_all(&buffer[..bytes_read]).await?;
total_bytes_received += bytes_read as u64;
println!(
"Progress: {}/{} bytes ({:.2}%)",
total_bytes_received,
file_size,
total_bytes_received as f64 / file_size as f64 * 100.0
);
}
println!("File transfer completed: {}", file_name);
Ok(())
}To summarize, after getting the metadata from the client, we can read bytes from the socket, until total_bytes_recieved is equal to file_size, and write it to the new file we created. And running the client and server, and giving the input, we see that our input file gets copied by the server! Woohoo!
See whole commit on GitHub.
In the next log, we can work on encrypting the data we send, and maybe even making it work as an actual app with which I can transfer files between two machines, instead of running the server and client in the same machine lol. Anyways, this has been a great experience so far and I’m so excited to keep going! Seeya!