Product and Design

Test Big or Go Home Dec 7, 2020 12:00:00 AM Running lots of small split-tests made my team feel productive and gave us plenty to talk about. But it failed to answer the questions that mattered most. On a bright winter afternoon last year, I looked out the conference room window. Clean, blue sky stretched over the wall of mid rise buildings across Manhattan’s 23rd Street. I listened passively as the hair-splitting arguments set in. “Well…I’m not sure we can come to a conclusion on this test—it’s only been running for 12 days.” “Well, I’m not sure it’s the placement of the button that was bad—it might just be the copy we used in this version of it.” “Well, I’m not sure this new page design is what worked—it might just be the image we used in this version of it.” I was the product manager on the growth team responsible for getting visitors to Betterment’s website to sign up and deposit money with us.  We’d host a meeting every Friday afternoon to review split-tests in progress in the hope of calling them—that is, to determine whether recent changes we’d made to the website or signup flows were driving more users to open accounts and make deposits. Fancying ourselves data-driven, we sought to guard against flawed observations that might lead us to a wrong conclusion. Our approach was rigorous but exhausting. As I zoned out and looked at the sky, I thought about something Katherine Kornas, a product leader who had joined the company a few months earlier—and to whom I reported at the time—shared at a recent product team meeting. “It’s not a failure when your split test loses,” she’d said, “it’s a failure when your split-test is inconclusive.” When you change your product in the hope of spurring signups and it actually drives signups down, she shared, you’ve at least learned what doesn’t work. When you change your product and nothing happens, you’ve learned almost nothing about users’ needs or behaviors. At that moment, learning about what users wanted was more important than ever. We were still figuring out how best to draw site visitors into a new type of product. Smart Saver was an account that was filled with money market and government bond funds and designed to earn a modest but reliable return. Having spent years making our name as the place to invest for the long-term, Betterment had developed a product built for the kind of short-term dollars that one might otherwise store in a savings account. Smart Saver was the precursor to the cash management products we’d go on to launch in the ensuing year: Cash Reserve and Checking. My team and I wondered how to feature Smart Saver in our signup flows and on our website. Should it sit front-and-center on the homepage? That would be great for those who arrived in search of that product, but it might confuse visitors who arrived in the hope of rolling over a 401(k). How could we talk about this particular product without obscuring the larger story of what Betterment was all about? No one knew for sure, but Katherine had the expertise to guide us toward the answer. “We need to think beyond the page,” she told me in one of our weekly one-on-one meetings. Katherine could sense that the split-testing approach I’d managed—with its small tests that changed only a single thing at a time—would never crack the bigger questions around how best to sell our burgeoning cash products. She urged me to think bigger, about entire flows, or even the broader swath of the user experience. That was tough to hear. I was proud of running dozens of tests and reviewing them ruthlessly. It made me feel like the data-driven product manager I thought I was supposed to be. Even if running numerous tests led to a lot of arguments in our tedious review meetings, it also gave me a lot to talk about—in release notes, at team meetings, and in reports to the broader growth team. As I re-read my notes from the time, though, I can see that for all my talking, I shared little in the way of strategy-shaping insights. Katherine was right to call for fewer, but better, tests. Instead of starting with an idea for a feature or design change, she said, I should start with a big question: what's the thing we’re trying to learn? What change to the user’s experience—ideally across multiple stages of their journey—would teach it to us? How might our strategy for the coming months change if the test wins? What about if it loses? The big test We went on to test a new, vastly different experience for users who arrived at our site with an interest in Smart Saver. We sent them down a signup flow that was unique for its silence on our investing accounts. We wanted to avoid mentioning any product other than the one the user came for. When a customer finished signing up and arrived at their dashboard, we affirmed to them that they had opened a Smart Saver account and invited them to fund it. This new approach earned strong results. Strong enough, in-fact, to beat back the kind of hair-splitting questions that had arisen in test review meetings of the past. It established a pattern that Betterment still uses today. If a site visitor seems intent on opening a certain kind of account—say, Cash Reserve, an IRA or a general investing account—we draw into a flow that focuses tightly on getting them settled in. We wait until later to tell them what else about Betterment they might enjoy. A test is likely big “enough” if it earns buy-in from stakeholders that X result should indicate Y change in your approach. Imagine you’ve replaced your homepage hero image with a picture of a dog. As the test gets underway, you tell your colleagues from marketing, design and engineering that if the dog picture drives signups, you’ll display only pictures of dogs throughout your public-facing pages, your signup flows and even in-app. It would be easy for anyone in the room to raise doubts. “Well, maybe dogs just work on the homepage, but not on other pages.” “Well, maybe it’s just because this dog is a Labrador, people might not like German shepherds.” “Well, maybe people just like four-legged animals. Can we try a cheetah?” You’ll stand a better chance of winning the room if you suggest a more comprehensive test: placing pictures of various dogs throughout the user journey—not just in one spot. You might propose that, if the test drives not only sign-ups but also deeper levels of engagement, it will show that dogs are key to acquisition and retention. The key is to strike an agreement with your stakeholders in advance about what result you’re looking for and, more importantly, how it would change your strategy for the months to come. In thinking bigger about split-tests, my team and I learned how best to draw users into our service. Personally, I learned something even more important. While one should certainly be afraid of drawing the wrong conclusion from a change to the user experience, one must also beware of making changes that are so small as to yield no conclusion at all.

A Comprehensive Guide to Accessible Data Visualization Nov 2, 2020 12:00:00 AM When you design for accessibility, you benefit everyone. Your charts and graphs (and users) will thank you. For many organizations, accessibility is often overlooked. It’s rarely prioritized because it’s not perceived to benefit a majority of users. The work required to fix existing issues is too great. It’s simply a “nice-to-have.” I get it. In the tech world, we want to ship the most impactful stuff and we want to do it quickly. We swear by the 80/20 rule. But according to the Centers for Disease Control and Prevention (CDC), 1 in 4 adults in the U.S. live with a disability. That’s a huge population—61 million people, in fact. Yet if you focus on the flip side, the vast majority are still nondisabled. So how do we make the case for accessibility? It starts with your company’s values. At Betterment, we’re working to make both our company and our product more diverse, equitable, and inclusive. We’re no longer striving for the MVP, but the MLP (Minimum Lovable Product, except we’re taking it a step further and aiming for Maximum). This means no population should be excluded or feel unwelcome on our platform. Accessibility is an important part of that mission, and we’ve made a choice that everything we ship must be accessible. Does it require additional effort and forethought? Sometimes. But we’ve found that starting with accessibility as our baseline actually creates a better experience for everyone. Digital Accessibility Disability comes in many forms, like auditory, cognitive, neurological, physical, speech and vision.1 Thus, what it means to be accessible is an expansive question. It boils down to building products that all people can use, inclusive of those with disabilities or situational limitations. The World Wide Web Consortium (W3C) developed a set of accessibility benchmarks called the Web Content Accessibility Guidelines (WCAG) that is widely-accepted as the standard for building, well, accessibly. We can lean on this list of rules for all projects as it covers everything from information architecture to video. So what—and who—should we take into account when creating charts or graphs? Users With Visual Impairments While a wide range of disabilities are important to consider when building digital products, visual impairments or deficiencies are the most relevant when we consider data visualization. A major use case to consider is low vision or blindness. These users often rely on assistive technology, like a screen reader, to navigate websites or apps. Screen readers are software that scan the content of a page and output it into speech or braille, often operated using keyboard commands. The video below is a firsthand look at how they work. Another common use case is color blindness, or color vision deficiency. To support these users, design for clarity and follow proper color contrast guidelines (more on that below). There are four types of color blindness:2 Deuteranomaly/Deuteranopia: green appears to look red Protanomaly/Protanopia: red appears to look green and duller Tritanomaly/Tritanopia: blue and green look alike; yellow and red look alike Achromatopsia: unable to perceive color Other visual impairments to be aware of include tunnel vision (loss of peripheral vision) or light sensitivity (trouble with screen brightness or light-colored backgrounds). Learn more about visual impairments from the W3C. Once we understand the users that may interact with our chart or graph, we can begin to accommodate them. Let’s get to it! Start with a strong foundation. The following is true for any data visualization, but is especially important when you’re focused on accessibility. A concise and coherent concept ensures users of all abilities can engage with your visualization. Put the numbers aside for a moment and ask yourself these questions: What story does the data tell? This helps you decide which aspect(s) to focus on. Imagine a chart that shows the populations of 10 major cities. If your goal is to compare San Francisco and New York, let that story shine through. This means Chicago’s population isn’t as relevant, and you might choose to minimize it. Likewise, if you have a particularly complex data set, you can break it into smaller and simpler chunks. How should the reader benefit from the visualization? Consider how your graph will help the user understand insights from the data. Like in the previous example, layer in context to pull those learnings to the surface. Can you understand it in 5 seconds? If the visual adds more complexity than your written word would otherwise, think about whether you need it in the first place. Avoid “well, it technically passes…” When teams work towards accessibility, there are several benchmarks they can use to measure success—for instance, this method to mathematically determine whether the contrast of text color on its background is sufficient for users of varied visual abilities to comprehend. For color contrast, the answer is clear whether we pass or fail. However, not all aspects of accessibility can—or should—be measured this way. Our benchmark should instead be an equitable experience for users of all circumstances and abilities. It’s simple to follow a checklist of standards (the aforementioned WCAG, for example) and walk away feeling like you’ve successfully cleared the accessibility hurdle. We must go further and strive not for a passing experience, but a great one. Our benchmark should instead be an equitable experience for users of all circumstances and abilities. Let’s hear an example related to data visualization. One quick method to help your graph “pass” accessibility guidelines is to ignore how it looks or functions, but stuff behind it the table of data it references. The table is invisible on the page, but screen readers can still access it and thus, read out the rows and columns of information. You might think, “well, the table lets the user access all the information in the graph. That’s accessible.” While this approach technically does make the data accessible, it fails to provide a high quality experience for someone using assistive technology because it’s missing the story or insight that the graph illustrates. Check out this video from the University of Colorado for a deeper look at what it’s like to navigate a table using a screen reader. When we revisit the questions in the prior section, we learn that the table experience misses the mark. Anatomy Of An Accessible Graph The example below is a graph that’s accessible in its own right—it doesn’t rely on a table to communicate the data to those using assistive technologies.   Let’s break down the design: Title: Give the visualization a descriptive “takeaway” title that illustrates the major insight. Fine: “My ice cream consumption compared to the national average” Great: “A graph titled: My yearly ice cream consumption has bested the national average since 2010. From the years 2000 to 2010, the yearly ice cream consumption levels of myself and the average American are similar. Around 2010, my consumption level spikes much higher and remains that way to the present day. The summary of the data is as follows…” Summary: In text, summarize in real, human words the story the information seeks to convey. This text can be discoverable only by assistive technology. If it’s also visible on screen without those tools, even better. Ask yourself: How would you describe the graph and its insights to a friend? Better yet, ask a friend to analyze it and summarize what they find. Axis labels (when applicable): Label each axis and their tick marks. This should be visible to the user and accessible through assistive technology. Data labels: Label each data point directly—use a connecting line, if you need to—rather than using a separate color-coded key. Users who cannot discern certain colors will still be able to interpret the data. Take the following into account during implementation: Alt text: Ensure your visual has descriptive alt text. Bad: “ice_cream_consumption_chart” Great: “Graph showing my yearly ice cream consumption is much larger than the national average” SVG format: If you can go the extra mile, build the graph using SVG rather than a basic HTML image. It provides a better experience to those using screen readers, as it allows access to the individual elements inside the graph. Read more about SVG graphs. Visual Best Practices Once all the right elements of your data visualization are in place, follow the guidelines below for its appearance. Here, we need to take into account both users with assistive technology and color vision deficiency. To support colorblind users: The following example images feature visual dos and don'ts. For the images in this section, I’ve duplicated them and applied a filter on the second version. The filter is meant to simulate how a person with colorblindness might perceive the original example. Follow color contrast guidelines. This might be a given, but it’s always worth repeating since it ensures legibility for all users. Here’s a deep dive into color contrast guidelines and how they work for more information. Below is the same example with a filter applied to simulate how a person with Protanopia would see it. The information in the “Don’t” version is even more difficult to make out. Label data points directly. It’s quicker to read, and ensures the graph can be understood without reliance on color. Below is the same example with a filter applied to simulate how a person with Tritanopia would see it. It’s nearly impossible to make sense of the color-relient key. Separate elements with whitespace or pattern. This means distinguishing between data isn’t entirely reliant on contrast between colors. Add whitespace between adjacent sections in a pie chart, or use different patterns in a line or bar graph. Below is the same example with a filter applied to simulate how a person with Deuteranopia would see it. Notice how the blue and purple sections nearly blend in the “Don’t” version. To support assistive technology: Avoid complex tooling. Stay away from sliders or other intricate tools when allowing users to manipulate the graph. Instead, try dropdown inputs—they can be used by screen readers. Avoid hover overlays. Don’t require a user to hover over a graph to see the data output. This is often difficult or impossible to do when using assistive technology. Instead, show the output at all times. Remain curious. It’s easy to write accessibility off as restricting creative license. In fact, these constraints are an incredible tool. When you design for accessibility, you’re not only assisting folks with disabilities, but embracing a vast range of life experiences. You serve the person recovering from a brain injury who has trouble concentrating. Or the parent juggling their newborn while managing their finances. Or someone at a sunny park, reading on their glare-filled phone screen. A clear and straightforward experience benefits everyone. The learnings above are by no means an exhaustive framework for accessible data visualization, but I hope they are helpful. The conversation about digital accessibility has come far, but it also has a long way to go. As designers, we can use our curiosity and empathy for users to drive it even further. Citations 1 World Wide Web Consortium (W3C), Introduction to Web Accessibility 2 National Eye Institute, Types of Color Blindness Any links provided to other websites are offered as a matter of convenience and are not intended to imply that Betterment or its authors endorse, sponsor, promote, and/or are affiliated with the owners of or participants in those sites, or endorses any information contained on those sites, unless expressly stated otherwise.

Can You Be Creative In A Regulated Industry? Sep 18, 2020 12:00:00 AM Regulated industries turn out plenty of bad and boring creative. But it doesn’t have to be that way. Here’s how we think about creativity. From finance to pharma, regulated industries catch more than a few (well-deserved) bad raps, including one close to my heart: bad creative. “It’s boring.” “It’s wordy.” “It all looks the same.” Blame for those sentiments often lands on those enforcing the regulations or the regulators themselves, like in-house Legal and Compliance Teams and governmental agencies. But it’s up to us—writers, designers, and our adjacent partners—to create and advocate for great work that meets everyone’s expectations. Can you be creative in a regulated industry? Of course! For me, the joy of working in fintech is in the challenge itself: fostering beautiful, badass, engaging creative ideas and executions in an unlikely place. Although regulated industries turn out their share of bad creative, there’s plenty of boring, wordy work that all looks the same in unregulated environments as well. That's why we have to reframe the problem. Compliance isn't the enemy, complacency is. Regulations aren’t arbitrary rules designed to transmogrify our otherworldly concepts into bland copy and design. They’re built to protect and inform consumers. Working within the guidelines of what is and isn’t legally and ethically compliant isn’t antithetical to great creative—it’s what sharpens our ideas. That isn’t to say that creativity in a regulated world is easy. Like anything worthwhile, it takes hard work and flexible thinking. The value of a great creative team (and great creative partners) is not in the raw conception of an idea, but in the building up, tearing down, twisting, turning, and wringing of the idea until it works and works well. Broaden and elevate your creative inspiration. Within a regulated industry, it can be easy to feel like you’re in an echo chamber of boring work. Change that. If everything looks and sounds the same, break away from your norms for inspiration: Find companies and people doing great creative work outside your industry and think through ways to translate that brilliance to your world. Consider work that comes from people who don’t look, sound, or think like you, from companies that aren’t even meant for you. Look beyond traditional advertising and brands. Restaurants create incredible feelings and experiences. Cities craft unique identities for themselves. Inspiration doesn’t have to come in the form of an ad. It’s easy enough to think that unlimited creative freedom equals better work. “If only I could do, make, say, think whatever I wanted...” But where’s the excitement in that, and what’s the challenge? If it were easy, it wouldn’t be great, and it wouldn’t be fun. After all, what makes us stronger? Pushing against resistance. The same goes for creative work. In fintech, the resistance is built in. The joy comes in meeting that resistance with our best ideas.

Why Product Managers Must Resist “Quick Wins” Aug 13, 2020 12:00:00 AM Tackling unplanned tasks from stakeholders made me feel helpful, but it bogged my team down. Here’s how I narrowed our focus to the most impactful tasks. “How hard would it be... …to add a section for notes?” …to make the button a little bigger?” …to add a banner to the homepage?” For a long time, I loved this question. My quick response of, “Oh, it shouldn’t be too hard!” felt like a cheap ticket out of a feeling I’ve often faced as a product manager: that I was failing to do enough for the teams that depend on me. I sat in a small, fluorescent-lit room during a one-on-one with a colleague, also named Joe, who was a leader in our company’s customer service team. Building tools for customer service was one of my team’s responsibilities, so I’d listen as Joe looked at his laptop and read from a long list of indignities his team faced–—disorganized screens, drop-down menus that lacked the one item they always seemed to need, and more. As I gazed down at the foot-long log that, for some reason, sat sideways between the two chairs in the small meeting room, a knot formed in my stomach. What kind of job was I doing if Joe’s team had to put up with so many menial, seemingly fixable problems? I snapped out of it when Joe asked a question: How hard would it be to redesign the instructions a customer sees when transferring a retirement account to Betterment? “Oh, that? Um, that should be pretty easy!” I said. It continued. At UX review sessions and in chat messages, stakeholders would ask how hard it would be to add just one more thing. I would smile and add the task to our backlog or to the upcoming milestone of a project. I was helping! At least, I thought I was. It’s not about difficulty. Meanwhile, projects were taking longer than we’d expect. I’d feel woozy looking down from the swaying height of our backlog. Small things had gained a stubborn habit of turning into one big thing. I realized my love of quick wins, of feeling helpful, was doing no one any favors. I’ve since changed my tune. When someone asks, “How hard would it be…” I respond with a question of my own. “How big would it be?” How much of an impact would it make? How much time would it save? How much revenue would it bring? What I’ve begun to share with stakeholders is that our backlog is home to dozens of tasks that we’ll never get to, simple as they might be. They linger there because they would fail to make a big difference in my team’s pursuit of our higher goals. It’s no use (anymore) to advocate for a bit of work solely on the basis of how easy it would be to complete. That cuts both ways. Our roadmap is full of projects that we expect will make a big impact: projects that will help not only our customers, but also those of our colleagues who serve them. These are projects we eagerly pursue—even though they are hard.