← back

The future of ML Scientists

A colleague asked me the other day: what happens to ML Scientists now that the zero-shot capabilities of AI models are getting better and better? Is there still a role for ML Scientists - or will (prompt) engineers just have to build a product, add the right prompt and be done with it? At a surface level, it seems like the role of ML Scientists might not be needed anymore in an industrial setting, where the goal is to build a product that works well and is reliable. That is, in this world there seems to be no need for custom models that are trained on a specific dataset, as general APIs will match the performance of these custom models. What is the role of ML Scientists in this world?

The role of ML Scientists in industry

The goal of ML Scientists in industry is to improve the product that the company is building with the help of AI. From a practical (rather than strategic) perspective, this can be broken down into a few steps:
  • Dataset creation
    Scientists might build a dataset creation pipeline that ensures that the data that is used to train models is of high quality and diverse. Often times, this is one of the most important things a scientist can do: given that the data is usually the most important part of training a model, focussing on this part of the pipeline can have a large impact on the performance of the model, and be more important than the model itself.
  • Model training and iteration
    Once a dataset is created, scientists will train models on this dataset, and iterate on the model to improve performance. This can be done in a number of ways, including:
    • Hyperparameter tuning
    • Trying new (custom) model architectures. This can be the bulk of the work in cases where data is available. Is supervised learning the right approach? Or should we use unsupervised learning? What is the right pre-training task? How to deal with the variety of data? Is there a way to speed up inference time?
    • Making sure the model is trained correctly, scales, trains fast
  • Model evaluation
    Now that a model is trained, it needs to be evaluated to see if it works as expected. This means that the model needs to be compared to a baseline, tested on a test set, and measured across different metrics. Getting closer to production, this might also mean that the model needs to be tested on real-world data with an A/B test. After deployment, there might be a need to monitor the model's performance and retrain it if necessary.

What will change

Now, with increasing capabilities of Big Model APIs, we might see a shift in the role of ML Scientists in industry. The question is: what will this shift look like? Let's go over each component and ask: what will change in the future?

  • Dataset creation
    Assuming zero-shot performance of models improves, we might not need as much custom training data anymore. However, we will still need high quality and diverse datasets for evaluation purposes. This means that the role of ML Scientists in dataset creation might shift from creating training data to creating evaluation data.
    Depending on the problem, custom training might still be required, because the data is too specific to be seen in the training data of the model API. We can expect that e.g. fine-tuning a voice assistant to have a particular tone of voice will not be necessary anymore, as we can expect that different tones of voice are widely available online already, and can be specified in the prompt. However, for safety-critical applications, where the data is very specific (e.g. anomaly detection of rare objects/images), custom training might still be necessary.
    The question to ask here is how good the few-shot performance of big models will be, that is: given a few examples of a specific data modality, how well can the model perform on this data? So far, we have seen that models are not incredibly good at generalising beyond the training data, although this is likely to change somewhat in the future. At the moment, I am not sure few-shot performance of models is good enough to replace custom training data where performance is critical, and the task is specific enough that it is not seen in the training data of the model API. That is, simple object detection or classification might be solved with zero-shot or few-shot learning, but more complex tasks might still require custom training data and custom models.
  • Model training and iteration
    My expectation is that training models from scratch will become pretty much obsolete in the future. This is because the difference in the amount of pre-training data that a company can use and the amount of data that is available to train a model from scratch is so large that it is unlikely that training a model from scratch will be beneficial. There is a reason transfer learning works well: the models have seen so much data that they have a good understanding of the world already, that only needs to be tuned to the specific task at hand. I doubt many ML Scientists in industry today train models from scratch, and I expect this to become even less common in the future.

“The only framework that works is you can either build a business that bets against the next model being really good, or that bets on that being really good and benefits from that happening.
So if you’re doing a lot of work to make something work that was just on the ability of GPT-4, then GPT-5 comes out and it does that and everything else really well, you’re sad about the effort you put into it. But if you have something that just kind of works and you didn’t put a lot of work into it, to make this one thing possible, and then GPT-5 comes along and it’s amazing, you have a rising tide lift on all your boats.
What I would suggest: you’re not building an AI business in most cases, you’re building a business. In the early days of the App store there were a lot of obvious things that were there, that Apple then fixed. And then there were things like Uber, that were being enabled by smartphones.”

**Dataset creation** Dataset creation is somewhat manual right now. The question is: do we still need large, high quality and diverse datasets to do most tasks? We probably don’t - zero-shot or few-shot learning will continue to i

Model evaluation seems to be something that is difficult right now. How to know that when you are updating from one API version to another, your business still works, and is better? - Other than tracking just speed and costs, measuring the quality of the work might be somewhat of an art form for many businesses. For example, with a new model, available, how do you know:

  • If a summary of a conversation is better?
  • If your AI doctor app now gives more reliable advice?
  • If your tutoring app is now better at tutoring students?
I think that this is an area where AI Scientists can come in and contribute: build systems that create reliable test datasets, that ensure some sort of robustness across the spectrum of an AI use cases in a business. - Maybe building systems that use other models to evaluate outputs — this feels like the next *Weights and Biases,* where you can have quite a few “AI Scientists” work on this

Model training in its current form will probably disappear somewhat for most use cases. Assuming in-context learning will be very powerful, training models will probably be of less importance. That is, all of these following use cases will disappear:

  • Fine-tuning a text-to-speech model to have a certain tone of voice
  • Fine-tuning a text-to-text chatbot to have a certain tone of voice

At medium risk

  • Fine-tuning a chatbot on customer-specific data. Here, it matters how specific customer data is, as in: how frequent can we assume such data modality to be present in the training data?
  • Customer support messages, Slack messages, are likely to be very similar to regular chats online, and you do not need to have specific models for this

However, there might be some use cases that will still require AI Scientists to train models:

  • Safety-critical applications requiring an

Soon to be replaced by an AI model
What are the ‘cracks’ that need filling right now in the OpenAI landscape?

  • Object detection with reliable bounding box results: OpenAI does not offer anything in this direction right now. But it should be possible to do this, e.g. some attention over an image might work fine (see e.g. the CLIP anomaly localisation work)
  • I would think that these models probably have an incredible precision here - they have seen so, so much image and video data that localisation is a trivial task. We already see that SAM has amazing zero-shot capabilities.
  • Low-latency tasks