Tasks and Metatasks
do I need to bake a cake, or build a cake factory?
A task is simple. It’s something you have to do. A job. “Bake a cake.”
A metatask is the job of making a task much easier, or automating it completely. The original task is trivialized by completing a metatask. You might have new things to do, but you’re not worrying about baking cakes any more. Now you’re training other people to bake cakes for you, or maybe you’re even performing maintenance on your cake factory.
In software, the line between tasks and metatasks gets blurry. It feels easy to go from doing a task to doing a metatask. As a brief, contrived example, here’s how you roll a six-sided die in Python:
import random
print(random.randint(1,6))And here’s how you roll a million six-sided dice:
import random
for i in range(1000000):
print(random.randint(1,6))I wrote one extra line of code and I went from baking one cake to baking a million cakes. If I add another zero it’ll be ten million. Once I’ve given the recipe to the computer, it’s trivial to tell the computer to follow the recipe as many times as I like.
In code, building a cake factory is as easy as baking a cake.
Or at least that’s how it can feel when you’re high on the power of software.
Complications
When your recipe is as simple as printing a random number, it’s easy to repeat it. Real recipes that bake useful cakes are rarely this simple. I’ll use “send an email” as a brief, contrived example of a recipe that introduces complications when repeated.
Here’s some psuedo-Python that sends an email from me@example.com to you@example.com. I’m omitting details you would need if you actually wanted to send emails, but the point is that we have to choose an address to send email to:
import smtplib
smtplib.sendmail("me@example.com", "you@example.com", "hello there!")You might be able to imagine what will happen if we try to kick off our million-email marketing campaign with the same line of code from earlier:
import smtplib
for i in range(1000000):
smtplib.sendmail("me@example.com", "you@example.com", "hello there!")We just sent a million emails to the same person! Whoops.
Coding a solution to this is conceptually simple; we iterate through a list of recipient email addresses and send an email to each recipient in the list:
import smtplib
for recipient_address in list_of_email_addresses:
smtplib.sendmail("me@example.com", recipient_address, "hello there!")But our solution requires us to assemble a list of all one million recipients’ email addresses. That’s a lot of work. Where are you going to find legitimate email addresses? Are you going to type them in by hand? Are you sure you don’t have any typos? There are entire companies built on solving this problem.
The task (“send an email”) is simple, but the metatask (“automate email-sending”) has turned out to be enormous! And there are still further subtleties to encounter, such as giving each of a million people the ability to unsubscribe from your mailing list, or tracking who you’ve emailed recently so you don’t accidentally spam them.
The metatask is spawning additional tasks, and each of those may be upgraded to a new metatask, and so on and so forth, until one day you realize you’ve spent your career building mailchimp.com.
Even if it looks like a metatask will be a trivial extension of a task, it often isn’t!
A normal bakery
More on cakes. Let’s say you’re an aspiring baker, and you get a job at Alice’s Bakery. On your first day, Alice says “I need you to bake five cakes a week.”
On Monday, you bake a cake. On Tuesday, you bake a cake. On Wednesday, you say “I know how to do this. I bet I can automate it by next week!” You go behind the bakery to start building a cake factory. The other bakers look on in confusion.
A month later, Alice comes running up to you. She asks, “What the heck have you been doing?! You’re weeks behind on your orders. Everyone else is working overtime because you’re not in the kitchen. You’re a terrible baker!” You raise your head from a mess of conveyor belts, put down your soldering iron, and say, “I’m so close to a breakthrough! This thing will be able to bake a thousand cakes a day once I set up a machine-learning-powered defect detector and install a halon fire suppression system. Give me another few months, and it’ll be transformative!”
Alice fires you on the spot.
This happens, regularly, on every single software engineering team in the world. The distinction between “bake a cake” and “build a cake factory” often looks like it’s just a few extra lines of code, so it’s easy for ambitious engineers to attempt the metatask and neglect the task. And solving the metatask is heavily rewarded. But if the metatask turns out to be too hard and you didn’t earn trust by baking enough cakes beforehand, you’re in trouble.
A bakery, but with venture capital
After getting fired from Alice’s bakery, you walk up the street and apply to Bob’s Bakery. Bob recently landed $50 million in Silicon Valley venture capital funding and the expectations for his business are sky-high. The team needs to scale to baking millions of cakes a year or else there’s no way the finances make sense.
Bob explains his plan to you. “We’ve never baked a cake before. But we know there’s no way we can hit scale if we waste any time at all, so we’re focused on putting together our cake factory as fast as possible.”
You ask, “Shouldn’t you bake a few test cakes by hand, to make sure they taste good?”
Bob responds, “I’m telling you, we can’t waste any time on that. It’s automation or bust around here. Now grab a wrench, baker!” You shrug and get to work on the cake factory. After all, this is the work you were excited about!
A year later the first cakes roll off the assembly line:
The cakes are inedible. The bakery folds. You’re out of a job again.
This also happens regularly across the tech industry. Teams attempt to build a scalable solution (a metatask) without understanding how to provide a good customer experience to one single customer (the task). This tendency among software nerds is so strong that Y Combinator feels the need to tell founders “do things that don’t scale.”
A normal bakery, but with more customers
After getting laid off from Bob’s Bakery, you crawl back to ask Alice’s forgiveness. Alice hires you back immediately. She’s looking frazzled.
“I need you to bake twenty cakes a week. Customers love our cakes and I can’t bear to turn them away! We’re overbooked like you wouldn’t believe. I’ve been working sixteen hours a day for the past six months straight!” she says.
You say, “Last time it was five cakes a week. How am I supposed to quadruple that?”
She responds, “I have no idea. Things are so crazy now. One of the ovens broke and we haven’t been able to get the repairman in to fix it because we’re open 24/7. There’s no more space in my daily planner so I’m starting to lose track of orders. And every time I hire someone new, they’re useless because I don’t have time to train them!”
Alice is underwater. Computer people refer to this situation as “too much tech debt.” It’s what happens when you spend too much time doing tasks and not enough time doing metatasks.
Alice will only be able to make progress if she drops some of her cake-baking commitments and starts working on a metatask. Maybe that metatask is your half-finished cake factory out back! You suggest it.
She scoffs. “No thanks, I saw what happened over at Bob’s. I don’t need to be in the business of industrial engineering yet. But I have been meaning to open a second location… hold down the fort here, I need some time.”
Coping with freedom
Software affords engineers leeway to slide around on the task-metatask scale. Working at the wrong level on the scale can cause big problems. Too far on one side, and you might introduce absurd scope creep when working on simple jobs; too far on the other, and you might fail to keep pace with the growth of the business.
My most straightforward heuristic for identifying where I should be working on the task-multitask spectrum is whether I’m working on a greenfield or a brownfield project.
On greenfield tech projects, especially prior to launch, people have a strong tendency toward pursuing metatasks and neglecting tasks. Engineers want to “do it right this time,” and they remember all the scaling pains they ran into with their previous project. Solving a scaling problem is always a metatask, and it’s dangerous to focus on metatasks when you still haven’t figured out the root task: how to make one customer happy.
With brownfield projects, you have established ways of doing things. People like to stick with what they know will work, even when it’s driving them crazy. And if the customer base has been growing without the business’s hiring keeping pace, there will be only just enough staff to keep the system running. They won’t have spare time to take on tricky metatasks. So there’s a strong bias toward pushing the same buttons that keep the system working, over and over again. Making progress requires a team to consciously fight that bias.
Finally, a good software engineer should always remember to ask whether anyone ever managed to perform a given task using landline phones and a decent filing system. You don’t always need to get a distributed systems PhD to scale a business. I hear Alice’s Bakery #2 is doing great, and Alice is already working on opening #3 and #4.
Great article.