At Widget Brain, we help customers daily with all sorts of questions around the use of algorithms in Workforce. One of the main areas we focus on is improving employee satisfaction and happiness by using algorithms which better predict when they are required and algorithms to make fair schedules that determine who works when.
In our daily discussions we see a number of things that we think lead to suboptimal or even bad results and therefore we decided to create a series of articles that each highlight one of the many best practices when it comes to the use of algorithms for labour demand forecasting and the creation of schedules.
Today, we present: Best Practices in Workforce Optimisation Part 1 – Why using heuristics in demand forecasting and rostering can lead to suboptimal results
We often are asked why a customer should pick one of our algorithms and how much better they are compared to algorithms provided by for example other large software providers. The answer is that it all depends on whether the algorithm provided by the other party is a heuristic and whether it applies any domain knowledge.
In the drive to generate a result which applies to as many domains as possible, vendors opt to create logic which can be generally applied without taking any domain knowledge into consideration. Every industry is different and there is no such thing as one solution fits all. A solution for one industry, will not work in another industry because the business logic is different and different rules apply to make that industry successful. Think about business rules such as planning horizon, number of shifts, employees, labour rules, etc.
Customers using that kind of logic will find that their domain-specific business rules are not considered and that the results from such an approach are useless as they cannot be used operationally. Examples are rules which only apply to the healthcare sector, retail, maritime, etc. Therefore we recommend customers ask their suppliers always for proof that a result can be generated based on their rules.
When it comes to demand forecasting or the creation of rosters, a lot of ISVs are selling so-called ‘heuristics’ as algorithms. A heuristic is basically a set of fixed rules and procedures which are applied to the input data, without any mathematical modelling. Yes, that sounds vague so an example will help here: Let’s use a demand forecasting heuristic which we see a lot: to forecast the next week, take the data of the last six weeks and apply the average value as next week’s forecast. It might lead to a good result in some cases but if demand ramps up or down due to seasonal changes this approach will fall flat.
A typical resolution to this by vendors is then to provide additional tweaks to the rules or new procedures to say: you can also use last year’s week from the same month. This again works in some cases but not in other cases like for example where, due to a change in business like a new promotion, the demand has completely changed. Customers are then forced to make another manual change to ensure that this is taken into consideration, which might work for some locations, but as you might have guessed by this stage: not for others.
The more of these rules and procedures are introduced, the more users of these heuristics end up in what we call ‘tuning hell’. Adding a rule might fix the result for one situation, but it can also generate problems in other situations which were not problems before. Tuning the situation which stopped working might then create a problem for another location again. It’s modern whack-a-mole really.
This is not to say that heuristics don’t have a place in algorithms. Heuristics are an excellent way to create a start solution for a mathematical based algorithm to further improve on the result, however, heuristics alone will not provide the best results for a business in a large number of cases especially if they ignore domain specific rules.
Mathematical-based approaches either use a large set of historical data or mathematical modeling techniques to come to the best possible answer, compared to a heuristic which will come to an answer but you have no guarantee that it is a good answer. The type of problem and the amount of data will determine which type of technique is most suitable to come to the best result. The general idea is that the result is generated automatically with minimal manual interference or requirement to tweak settings and/or rules.
Mathematical approaches are also ideal to include domain-specific knowledge to make sure the results are applicable and realistic for the chosen domain. This does mean you can end up with a specific algorithm per domain, but that is a better situation than having a one-size-fits-all algorithm which generates results not realistic for any domain.
In the case of demand forecasting, for example, you are better off using machine learning, statistical or deep learning methods which are capable of understanding and projecting previous trends and thus get to a better result. They use all the historical data, industry-specific factors and other circumstantial knowledge to create a prediction model of not only the trends but also the expected demand per 15 minutes or hour. Or in the example of the creation of rosters, having specific rules for supermarkets where people move from area to area, adhering to break rules specific to the domain, etc. A mathematical approach would smartly create all the possible combinations, the objective to improve (optimise service level) and all the rules it has to adhere to after which it will return the best possible answer.
When making an algorithm for a new domain, heuristics are perfect to get a quick answer and to encounter the complexity of such domain with all its logic and rules. In the long term, however, more structured approaches will always win in result and maintainability. Don’t get me wrong, I do like a good game of whack-a-mole but when it comes to algorithms: I’d rather not have to play it at all.