Ask Women in Product: When and how do I add Machine Learning to a product roadmap?

Photo by Bea Rose via Twenty20

Answer from Kenlyn Terai

Table of Contents


What is Machine Learning?

Image credit:

Artificial Intelligence > Machine Learning > Deep Learning

Image credit:

When is a Machine Learning-based solution appropriate?

Problems that are well suited for ML-based solutions

  • Requires complex logic that’s impractical to solve with human-defined rules, or heuristics. For example, search engines often have multiple phases of ranking that happen in series, such as initial retrieval, primary ranking, contextual ranking, and personalized ranking. This is a great application for Machine Learning.
  • The problem will be scaling up very fast. If you expect that your problem will scale to thousands of users or more, then it could be a good use case for ML. Let’s say you have an online retail platform and expect to have thousands of customers avail of your new offer within three to six months. Retail customer expectations being what they are, you need a personalized experience and could give a good justification for ML.
  • Requires personalization at scale. Unless you can reduce the complexity of the problem space by, for example, creating solutions for a particular segment or category rather than for each individual, you’re better off using ML to define the rules you use.
  • Require rules that change quickly over time. If your rules generally remain static year after year, then a heuristic solution is preferred. However, if your business’s success depends upon quick adaptation and rule changes, then ML is a good route. For example, if you have a search product and Ed Sheeran drops a new album, your algorithm needs to adapt in real time and is amenable to an ML solution.
  • Has a known, pre-defined end result. For example, in online retail, you want your model to provide recommendations which result in a sale. In search, typing “shirt” should return results with lists of shirts that are most likely to lead to a purchase.
  • Does not require 100% accuracy. If business success can be achieved with a high probability of accuracy rather than with perfection, then ML is a good option. For example, recommendation systems will not be considered faulty if users don’t always want what is served. Users can still have a great experience and the program can learn from the lack of sales to deliver improved recommendations in the future.

Problems that don’t need ML-based solutions

A successful heuristic-based solution (Image by Google/Gmail)

What do I need to succeed?

  • A clearly defined business problem. Machine Learning shifts engineering work from a deterministic process to an experimental one, so there is an even greater need to know what you want to achieve upfront.
  • The right team. You’ll need a team with skills in both Data Science and Engineering. In Data Science, roles can include an ML Scientist, Applied Scientist, Data Scientist, and/or Research Scientist. In Engineering, roles can include a Business Intelligence Engineer, Data Engineer, Software Engineer, Development Manager, and/or Technical Program Manager.
  • A grasp of the project’s potential risks and returns. Even if a project is feasible from an ML perspective, the level of effort needed to develop and maintain an ML-based solution may make such an approach impractical. For example, if your model needs to be updated very frequently in production, to the point where it requires a lot of maintenance, it may not be worth it.
  • Enough data. You need at least thousands of rows of data for linear models and hundreds of thousands of rows for a neural network. If you don’t have the data, you’ll need strategies to acquire it, and may need to stick to heuristic-based approaches until you do.
  • Data with a clear pattern. Since algorithms require patterns to learn from, you need to have a sense of what patterns exist in your data, even if you don’t know the precise pattern before you get started. You should be able to articulate these patterns qualitatively or have a gut feeling at the minimum.
  • High-quality data. As the adage goes, “garbage in, garbage out.” Lean on your Data Scientists to help you determine your data’s status and/or how to acquire data that are:
    a. sufficient in completeness and simplicity
    b. relevant to the problem that you are solving
    c. recent, reflecting users’ current behaviors
    d. representative of the segment and timeframe you‘re addressing
    e. unbiased
    f. respects user privacy and is secure
  • The right technologies. There are several open-source tools and platforms, such as Amazon AI, TensorFlow (originally developed by Google), and many others that make machine learning accessible to virtually any company today.

How do I fit Machine Learning into my roadmap?

1. Understand the business need

2. Formulate the problem hypothesis

  • A change that you are testing (“Improving the search ranking with ML will…”)
  • A desired outcome (“… allow our customers to find the correct product…”)
  • Success metrics (“…in 15% less time.”)
  • A description of the Model’s output (“The model will score each possible product…”)
  • Predictors (“…by using products recently viewed by the customer, the types of products previously purchased, the monetary value of previous purchases, and our own understanding of what products are frequently brought together …”)
  • Target (“…to predict the product that the customer ultimately selects for purchase.”)

3. Define clear measures of success

4. Assess your candidate projects

Left: Impact/Effort Matrix (created by the author). Right: User Impact/ML Impact matrix (Image credit: Google)

5. Confirm that you have the data you need to succeed

6. Establish healthy data collection practices

  • Guide user input when you can
  • Use auto-suggest fields
  • Validate user inputs, emails
  • Collect user tags, votes, and ratings
  • Track impressions, queries, and clicks
  • Sessionize logs
  • Disambiguate and annotate entities (company names, locations, etc.)

7. Tackle the risks and challenges

  • Be aware of the types of bias that might impact your model. Mitigate these biases by allowing your model to measure them, then take steps to counteract the biases.
  • Be aware that seemingly small UI changes may result in significant backend ML engineering work that may put the overall project at risk. For example, if the wording of a question is changed in your website or app, it can change the response from users and render your historical data useless. Be fully aware of the impact of any recommended changes, A/B Test, and get user feedback!
  • Even with the right data, you may still not end up with a working model. For example, if your model overfits the training data, over-learning the data’s details and noise, the model will fail to make correct predictions on new data. Data scientists can address this by “regularizing” the model.
  • Asking for too much user data without any visible benefit can cause the user to abandon your product. Provide value to your users as early as possible before asking them for more data.
  • You need to have the appropriate security and privacy precautions in place any time your model includes or relies on personal data. Proactively consult with your company’s legal, safety, and security teams for advice in this area.

8. Expect to iterate


About the Author




A global community of women working in Product Management.

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

How the learning happens…

Variations of SSD — Understanding Deconvolutional Single-Shot Detectors

A healthy optimism and critique of AlphaFold2

NLP: Morphological and Spelling Similarity

Neuromation Story: From Synthetic Data to Knowledge Mining

7 Best Machine Learning and Deep Learning Courses

Easy way to understand Principal component analysis(PCA) and how it is used in Machine learning.

How to Save Model Training Time Using Callbacks

How to Save Model Training Time Using Callbacks

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Women in Product

Women in Product

A global community of women working in Product Management.

More from Medium

Design for Understanding: COVID Data Sprint

From the vision of a product to the MVP backlog

Parallels in Recruitment Process and Product Management

Transition from Risk to Product