Thanks to Toby Culshaw, Carlos Politi, and Alex Soltani for their thoughtful edits and collaboration on this article.

“Best places to work” rankings suggest there is a single definition of a great employer. There isn’t. Value depends on perspective. Employees evaluate companies based on what matters to them, while employers evaluate value through business needs and priorities. In this article, I break the employee value proposition into measurable components and introduce a model that allows those priorities to be adjusted, showing how company rankings shift as you redefine what matters.

Some people care most about compensation. Others prioritize flexibility, strong leadership, meaningful work, or the chance to build valuable skills. Because priorities vary so widely, most “best places to work” rankings only tell part of the story.

That got me curious about what would happen if you broke the employee experience into its underlying components and let people decide which ones matter most to them.

To explore that idea, I analyzed Fortune 500 companies across twelve dimensions of the employee value proposition and built an interactive dashboard that allows users to adjust the importance of each factor. The rankings change quickly once priorities shift. The dataset used in this analysis is a modeled sample designed to illustrate the framework rather than a definitive ranking of employer quality.

In other words, instead of declaring a single list of the “best” employers, the goal is to make the tradeoffs visible and let people explore how companies compare depending on what they personally value in a workplace.

You can explore the interactive dashboard, sampled in Figure 1, here.

One clear outcome of the model is how dramatically company rankings shift when priorities change. Some companies that appear near the middle of the pack overall move close to the top when flexibility or leadership quality receive more weight, while others known for strong compensation fall quickly when career development or culture are emphasized.

This highlights an important point: the “best employer” often depends more on what an individual employee values most. A company that is an ideal environment for someone seeking rapid skill development might feel very different to someone prioritizing flexibility or stability.

It’s also worth noting that what employees prioritize shifts over time. During COVID, safety, health, and remote work dominated. Post-pandemic priorities have shifted. While flexibility and remote work remain high on the employee list, compensation has re-emerged as high priority for workers due to persistent inflation and cost-of-living pressures.

In some cases, external market factors can directly influence EVP dimensions in ways that are easy to overlook. At companies with heavily stock-based compensation models, share price can have a meaningful impact on both employee sentiment and candidate demand. When stock performs well, total compensation increases and employee perception often improves alongside it. When it declines, the opposite can occur, even if underlying job characteristics remain unchanged. This creates a dynamic where elements of EVP are partially shaped by forces outside the organization’s immediate control.

Cultural context shapes EVP allocation preferences. Individualistic employees prefer task performance-based equity (rewarding quality and productivity), while collectivistic employees prefer extra-role performance-based equity (rewarding effort and loyalty). Power distance orientation further influences preferences: low power distance employees value equal distribution, while high power distance employees accept status-based and hierarchical allocation criteria.

A demonstration of a tailored EVP view is shown in Figure 2. In this scenario, the dashboard is filtered to Financial Services firms within the Fortune 100, and the weighting is adjusted heavily toward compensation (60%) and employer brand (25%), with the remaining weight placed on learning and development (10%), benefits (5%), and career mobility (5%).

The main takeaway is that some lagging metrics matter very little when they carry little or no weight in the evaluation. In this example, several dimensions shown in orange represent weaker performance areas, but because those dimensions are not prioritized in the weighting scheme, they have minimal influence on the final ranking.

As priorities change, the same company can move significantly up or down in the rankings. A firm that appears average under equal weighting can quickly become a top performer when the dimensions most important to a particular employee are emphasized.

This illustrates why static “best employer” rankings often fail to capture individual preferences. What matters most is not necessarily how a company performs on every dimension, but how it performs on the dimensions that matter most to the individual evaluating it.

I would be curious to hear how others would weight these dimensions. If you could adjust the sliders yourself, which factors would matter most to you when evaluating an employer?

What Is an Employee Value Proposition?

An Employee Value Proposition, often shortened to EVP, describes the overall value an organization offers employees in exchange for their work. It is the combination of compensation, benefits, leadership, culture, flexibility, development opportunities, and mission that together shape the employee experience.

For many years, conversations about EVP were heavily centered on pay and benefits. Those factors still matter, but the modern workforce evaluates employers across a much broader set of dimensions.

Flexibility, leadership quality, learning opportunities, and work life balance increasingly influence how people evaluate potential employers. In many cases, these factors shape day-to-day satisfaction more than compensation alone.

The competitive landscape of EVP itself evolves over time. Consider how some companies introduced office perks (e.g., cafeterias, fitness classes, laundry services) that became industry expectations. Years later, many of these firms reduced these ‘table stakes’ benefits to cut costs. This illustrates that dimensions become baseline, then get cut. Organizations compete not just on dimension performance, but on which dimensions are even available.

Research backs this up. LinkedIn’s Global Talent Trends research found that 94 percent of employees say they would stay longer at a company that invests in their career development. Gallup has also found that roughly 70 percent of the variance in employee engagement can be attributed to the manager, highlighting how strongly leadership quality influences the employee experience.

Because of this, two companies offering similar pay can feel completely different as places to work. Differences in leadership, culture, flexibility, or growth opportunities often determine whether employees stay and thrive or eventually decide to leave.

For organizations competing for talent, understanding and shaping these factors has become increasingly important.

Why EVP Matters From the Employee Perspective

Choosing where to work is one of the most consequential decisions most people make during their careers. According to the U.S. Bureau of Labor Statistics, the average full-time worker spends more than 90,000 hours working over the course of their lifetime. That means the quality of the work environment can have an enormous impact on overall well-being.

When people choose the wrong environment, the effects can go well beyond compensation. Research from MIT Sloan analyzing millions of employee reviews found that workplace culture is one of the strongest predictors of voluntary turnover, often outweighing compensation in employees’ decisions to stay or leave.

Because of this, many professionals now look at potential employers through a much broader lens. Some prioritize flexibility and autonomy, especially as hybrid and remote work have become more common. Others focus on career mobility and the chance to build skills that increase long-term earning potential. For some, leadership quality or meaningful work are the most important factors.

The challenge is that these priorities vary widely from person to person.

A company that ranks highly in a traditional “best employer” list might perform well on compensation but poorly on flexibility or career development. For someone who values flexibility above all else, that organization might not actually be a great fit.

This is where a more flexible EVP framework becomes useful. Instead of assuming everyone values the same things, the model allows individuals to adjust the importance of each dimension and see how rankings change. Someone focused on flexibility can weight that dimension more heavily. Someone else focused on career growth or compensation can do the same.

The result is a more personalized way to evaluate employers.

Why EVP Matters From the Employer Perspective

From an employer’s perspective, this talent proposition becomes the organization’s competitive strategy in the labor market.

Most companies devote enormous analytical effort to understanding how they compete for customers. Far fewer take the same structured approach when it comes to competing for talent.

Yet the financial stakes are substantial. Research from the Society for Human Resource Management estimates that replacing an employee can cost between 50 percent and 200 percent of that employee’s annual salary, depending on the complexity of the role. These costs include recruiting expenses, onboarding time, lost productivity, and the loss of institutional knowledge.

Employee engagement also has measurable effects on business performance. Gallup’s analysis of organizations worldwide has shown that companies with highly engaged employees experience 23 percent higher profitability, 18 percent higher productivity in sales, and significantly lower turnover compared with those where engagement is low.

Because of these impacts, understanding how an organization compares with competitors across dimensions such as compensation, flexibility, leadership, and development opportunities is not simply a cultural exercise. It is a strategic capability that influences an organization’s ability to attract and retain talent.

Looking at EVP through a structured framework helps organizations answer practical questions. How does our talent proposition compare with companies competing for the same skills? Where do we lead peers, and where do we lag behind? Which improvements might meaningfully strengthen our ability to recruit and retain great people?

These insights can help guide investments in the employee experience more effectively. In practice, this type of framework can also help organizations understand how their talent proposition compares with competitors targeting the same skills.

How the Data Was Built

To explore these questions, I constructed a dataset evaluating Fortune 500 companies across twelve dimensions of the employee value proposition. These dimensions include factors such as compensation competitiveness, workplace flexibility, leadership quality, career mobility, learning opportunities, mission alignment, and employee sentiment.

Large language models were used to assist with gathering and structuring information across these categories, drawing on a combination of publicly available sources and modeled signals.

First, employee sentiment indicators from publicly available sources were used to estimate how employees perceive aspects of the workplace experience such as culture, leadership quality, and development opportunities. While no single dataset perfectly captures employee sentiment, combining multiple signals helps approximate how organizations are perceived by their workforce.

Second, structural factors such as workplace flexibility were modeled using known company policies and industry norms. For example, organizations with strict five-day in-office requirements generally receive lower flexibility scores than companies with hybrid or remote work models.

Third, all metrics were normalized to a common scale so companies could be compared consistently across industries. This allows organizations in sectors such as technology, finance, manufacturing, and healthcare to be evaluated within a common framework.

The default weighting in the dashboard was designed to represent a balanced view of the employee value proposition rather than a definitive ranking formula. Each dimension was evaluated based on how strongly it tends to influence employee decision-making, how frequently it appears in workforce research and employee surveys, and how directly it affects the day-to-day employee experience.

Insights from workforce research published by organizations such as Gallup, LinkedIn, and MIT Sloan were used as directional signals when assigning baseline weights. Dimensions consistently associated with employee engagement and retention, such as leadership quality, career development, and compensation, received slightly higher weights, while other factors such as mission alignment or employer brand received more moderate weighting.

The goal is not to claim a single correct formula for evaluating employers. Instead, the default weighting provides a reasonable starting point that users can adjust to reflect their own priorities.

It is important to note that this dataset should be viewed as a structured demonstration rather than a definitive measurement of employer quality. Large language models are useful for synthesizing publicly available information and generating structured datasets, but they cannot replace a fully engineered data pipeline built from primary data sources.

The LLM-assisted approach provides several strengths. It allows the framework to be constructed quickly, creates a consistent structure across companies, and enables experimentation with weighting different aspects of the employee experience. For exploratory analysis or prototyping dashboards, this approach can be surprisingly effective.

At the same time, the model relies on approximated signals rather than direct measurements for many companies. Employee sentiment, leadership quality, and development opportunities were inferred from a combination of public sentiment indicators, company reputation signals, and industry context rather than a single comprehensive dataset. Timing of content also lags. For instance, Fortune 500 rankings of the companies in the study are not for the current year.

A more robust implementation would incorporate additional structured data sources. These could include large-scale employee review datasets, compensation benchmarks, workforce mobility indicators, and company-level policy data such as remote work practices. Building a pipeline that integrates these sources would require significantly more engineering effort but would produce a more precise and continuously updated model of employer comparisons.

In other words, the goal of this project is not to produce a definitive ranking of Fortune 500 employers, but to demonstrate how employee value propositions can be analyzed as a multi-dimensional system where tradeoffs become visible when priorities shift.

Reproducing the Analysis

One of the goals of this project was transparency and reproducibility.

The dataset and scoring approach can be recreated using modern large language models by following a structured prompt that defines how companies should be evaluated, how scores should be normalized, and how EVP dimensions should be represented in the final dataset.

For those interested in replicating or experimenting with the analysis, the prompt used to generate the dataset is available here.

Because LLM-generated datasets may vary depending on the model and prompt used, reproducing the analysis with different models can also serve as a useful experiment in how generative systems interpret and structure workforce data.

Providing the prompt allows others to reproduce the analysis in their own environment, compare results across models, or experiment with alternative weighting schemes and datasets.

Closing Thoughts

The idea of an Employee Value Proposition has been around for many years, but the ability to analyze it with data is still evolving.

By translating aspects of the employee experience into measurable signals, even imperfect ones, it becomes possible to compare employers in a more transparent and flexible way. For employees, this approach can help identify organizations that align with their priorities and career goals. For employers, it offers insight into how their talent proposition compares with competitors.

Ultimately, what makes a great employer depends heavily on what individuals value most. Some prioritize compensation, others flexibility, leadership quality, career growth, or meaningful work. A single ranking cannot capture all of these perspectives.

Allowing those priorities to change makes the tradeoffs visible and highlights how different definitions of a great employer lead to different conclusions.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Introduction: When the Same Question Produces Different Truths

In the first article, the failure was directional inaccuracy caused by the absence of a real data layer. In the second, it was bias sensitivity, where framing alone reshaped quantitative reasoning even when the data remained fixed. The third and final failure is more fundamental. Large language models cannot reliably reproduce their own outputs. Any system that produces numbers for decisions must satisfy a basic requirement: repeatability.

Use Case: The Model Melee Consistency Study

To evaluate repeatability, the same structured prompt was executed more than thirty times across four major LLMs: ChatGPT 4o, Claude Sonnet 4, Gemini 2.5 Flash, and Microsoft Copilot, as documented in the Model Melee study on TalStrat. The task was simple and fully controlled. The dataset, instructions, and output format all remained constant. The only variable was the model itself, yet what emerged was not minor variation but systemic inconsistency.

Within the same model, under the same conditions, outputs changed. Some runs introduced new metrics that were not requested. Others removed existing ones. In several cases, the weights failed to sum to one hundred percent even though the prompt required it, and in multiple runs, the model replaced part of the analytical output with a Python script. Those issues are notable, but the more important signal was the magnitude of variation. That variability is captured by the coefficient of variation, where lower values indicate higher consistency.

Claude Sonnet 4 emerged as the most consistent model, producing the lowest variability with a coefficient of variation of 0.12. ChatGPT 4o (0.24) and Gemini 2.5 Flash (0.23) both produced moderate inconsistency, while Microsoft Copilot exhibited the greatest instability at 0.32. These numbers are not abstract. They determine whether two analysts running the same workflow will arrive at the same conclusion.

Triangulation as a Risk Mitigation Technique

One implication of these results is that, if an organization chooses to rely on LLM outputs despite this variability, single-run answers should not be treated as authoritative. In high-variance models, a single response is effectively a random draw from a distribution, not a stable estimate.

In those cases, a more defensible approach is to run the same prompt multiple times and examine the distribution of outputs rather than selecting one result. Aggregation methods such as averaging or median selection can reduce volatility, but they do not eliminate underlying instability. They merely smooth it.

This matters because the degree of mitigation required depends on the model. In this study, repeated runs were far more important for Microsoft Copilot, which showed the highest variability, than for Claude Sonnet 4, which demonstrated substantially stronger consistency. Even so, triangulation should be viewed as a risk-reduction technique, not a substitute for repeatability.

Why Repeatability Is Non Negotiable

Once AI outputs begin feeding real workflows, inconsistency becomes operational risk. If a ranking, forecast, or prioritization score changes depending on which run is used, decisions drift. There is no visibility into which answer is correct because the system itself cannot establish a stable reference point. This becomes especially problematic inside automated or semi automated pipelines, where the same input can produce different actions on different iterations with no explanation and no accountability.

One Unexpected Constant

Across every model, every scenario, and every run, one result never changed: Michael Jordan was ranked as the greatest NBA player of all time in every test.

The GOAT...and his goat.

The stability of that outcome does not cancel the surrounding instability. It highlights it. Sometimes truth survives noise, and sometimes a broken clock is still right twice a day.

Series Conclusion: Confidence Without Foundations

LLMs are extraordinary accelerants for thinking, planning, and framing complex problems. They shorten the distance between question and workable structure, but they are not, by default, trustworthy quantitative engines. The first failure showed what happens when models generate synthetic data. The second showed how framing reshapes reasoning and results in biased outputs. The third shows that even when both the data and the prompt are fixed, the model itself does not provide stable answers. Directional inaccuracy, bias sensitivity, and repeatability failure are not edge cases. They are structural characteristics of how these systems operate today.

The responsible path forward is not to slow AI adoption. Rather, it is to anchor it, grounding models in verifiable data.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Introduction: When Data Stays the Same but the Answer Changes

In the first article, the central risk was directional inaccuracy caused by the absence of a real data layer. Even when outputs looked analytical, the rankings were fundamentally wrong because the model was inventing the numbers it needed. The second failure mode is more subtle and just as troubling.

This second risk is bias sensitivity. In practice, the model’s quantitative reasoning changes based on how a question is framed, even when the underlying data remains identical. In other words, the phrasing of a prompt becomes a hidden control system over the outcome.

To surface this behavior, a controlled experiment was designed around a familiar problem: determining the greatest NBA player of all time.

Use Case: The NBA GOAT Bias Study

The experiment focused on whether a large language model could assign appropriate weights to a fixed dataset of performance metrics to determine the greatest NBA player of all time. The task itself was intentionally neutral, and both the dataset and instructions remained fully controlled. The model’s job was to determine how much each metric should contribute to a final score.

This analysis was originally published as a two-part study. Part 1 on Directionally Correct, written with Cole Napper, covered how subtle prompt framing alters quantitative weighting logic, with Part 2 on TalStrat.

On the surface, this resembles many real business use cases. A dataset is provided, a scoring framework is defined, and the model is asked to reason quantitatively. The resulting output appears logical, methodical, and well justified.

This experiment was designed specifically to test for anchoring bias. A control prompt was compared with multiple variations that introduced subtle cues. Some prompts suggested exaggerated numeric values, such as assists per game accounting for ninety percent of the total weighting. Others implied narrative preferences, such as hoping the result would finally establish LeBron James as the greatest player.

The dataset never changed, but the outputs did.

The strongest effects appeared in the numerical anchoring scenarios. When the prompt stated that assists per game should represent ninety percent of the decision, ChatGPT assigned significantly higher assist weightings compared to the control group. A similar result occurred when the prompt suggested that assists should matter only ten percent. These results demonstrate that introducing any explicit number, whether large or small, distorts how the model interprets importance. Please see the two articles referenced above for a more detailed review of the findings.

Narrative cues also influenced the model. A prompt favoring Michael Jordan produced a statistically significant decline in assist weighting as well. Prompts favoring LeBron James or older eras produced weaker and less consistent effects, but the presence of measurable shifts confirmed that phrasing alone could influence quantitative output.

The model was not simply analyzing the dataset. It was susceptible to influence, and the result was biased output.

Cross Model Validation: Is This Just One System?

To determine whether this behavior was isolated to one model, the experiment was repeated across four major systems: ChatGPT 4o, Claude Sonnet 4, Gemini 2.5 Flash, and Microsoft Copilot. The dataset, instructions, and evaluation framework remained constant. Only the model changed. This experiment is covered in more detail in “Model Melee 1: Claude, ChatGPT, Gemini, and CoPilot Walk into a Prompt...” on TalStrat HERE.

The number of NBA championships was used as the test metric because it carries strong historical importance and was expected to magnify variance.

The pattern persisted across all four LLMs, but to varying degrees.

Claude Sonnet 4 showed the most resistance. It reacted only to the ten percent condition and remained stable under both the ninety percent and narrative prompts. It also produced the lowest coefficient of variation at 0.12.

Gemini 2.5 Flash demonstrated selective sensitivity. It reacted strongly to the ninety percent anchor with a Cohen’s d of 2.16 but resisted the remaining conditions. Its consistency matched ChatGPT 4o with a coefficient of variation of 0.23.

ChatGPT 4o exhibited the strongest susceptibility. It reacted to all anchor conditions and produced very large effect sizes, including a Cohen’s d of 3.20 under the ninety percent prompt. Its outputs also showed moderate inconsistency across runs with a coefficient of variation of 0.23.

Microsoft Copilot showed the least predictability. It reacted to both numerical anchors and produced the highest variability across runs, with a coefficient of variation of 0.32.

Why This Matters

Organizations increasingly rely on LLMs to accelerate decision frameworks. The problem is not that the model produces an answer. The problem is that small changes in wording produce different answers while maintaining the same level of confidence and apparent rigor.

If two analysts frame the same business question differently, the model may produce two materially different conclusions. Both will appear correct.

When those conclusions flow into real decisions, bias becomes operational.

The Second Pillar of the Reliability Gap

The first failure mode showed what happens when models generate synthetic data. The second shows what happens even when the data is correct. Without explicit controls, prompt design becomes a silent decision lever.

The next article examines the third and final pillar of the reliability gap. Even when the dataset is fixed and the prompt is identical, models cannot reliably reproduce their own outputs. That failure of repeatability undermines the basic requirements of any system treated as data.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Introduction: The Reliability Gap

Many organizations are already using AI outputs as inputs to real decisions. That is where the problem begins. You cannot trust AI-generated analytics unless they are grounded in a real data layer. That is my opinion, but one based on applied testing across real use cases. When large language models generate numbers instead of retrieving them from verifiable datasets, the outputs may appear reliable, but they rest on unstable foundations.

Across a series of experiments, a consistent reliability gap emerged. The failures that will be discussed fall into three categories: 1/ directional inaccuracy, 2/ bias sensitivity, and 3/ repeatability and consistency failure. Each of these weaknesses becomes more concerning as organizations increasingly use generative AI for workforce intelligence, benchmarking, prioritization, and automated decision support. This first article focuses on the data layer problem and the consequences of directional inaccuracy, illustrated through a talent pipeline study using validated labor market data.

The Illusion of Knowledge

Large language models are increasingly producing outputs that look reliable and accurate, and many people are beginning to treat those outputs as though they are grounded in real data. They generate ranked lists, scoring models, talent benchmarks, and similarity analyses that appear structured, legitimate, and ready for usage in decisions. The presentation creates an illusion of credibility, and it is easy to mistake that polish for reliability.

Three separate analyses were conducted which highlight this pattern of failures:

1. In a workforce sourcing test, the model generated a credible-looking ranking of companies that produce top Data Science talent, but when compared to validated labor market data, the ordering was directionally incorrect.

2. In an NBA GOAT scoring experiment, small changes in prompt wording produced statistically significant anchoring bias even when the structured task and dataset remained identical.

3. In a multi-model consistency benchmark, repeating the same structured request across multiple runs and multiple LLMs produced different answers, demonstrating a lack of repeatability.

LLMs are impressive tools for accelerating thinking, structuring analytical frameworks, and reducing time between question and approach. But without grounding in real, verifiable datasets, they do not yet operate as trustworthy quantitative engines.

This article focuses on the first failure mode: directional inaccuracy driven by the absence of a reliable data layer.

What a Data Layer Means in an LLM Context

When I refer to a “data layer” in the context of large language models, I am not talking about the model’s training data or its general knowledge. I am referring to a live, explicit, and verifiable source of truth that the model retrieves from at the moment it generates an answer. A proper data layer consists of structured datasets, maintained by a trusted system of record, that the model is forced to reference rather than invent. This can include labor market databases, financial systems, operational metrics, transactional records, or any domain-specific repository that provides authoritative values for the task at hand. Without such a layer, the model is not querying facts. It is estimating them.

This distinction matters because large language models are fundamentally probabilistic reasoning systems. When asked for numbers, rankings, or distributions without access to a real data layer, the model will either produce a null response or it generates values that are statistically plausible within the structure of the content from which it was trained. The output may appear accurate, but it is not grounded in reality unless an external data source is injected into the reasoning process. Without a data layer, even well-formatted quantitative outputs are potentially structured hallucinations.

Use Case: The Talent Pipeline Study

The first experiment explored whether a large language model could identify which companies would be the strongest sources of Data Scientist talent for a major tech employer. This type of question is becoming increasingly relevant as organizations explore whether generative AI may assist with workforce planning, talent intelligence, and sourcing strategy. If these systems are ever trusted to inform decisions, their accuracy matters.

To assess reliability, the same request was evaluated using two different inputs:

1. One relied on a verified labor market dataset from Lightcast containing real skill distributions.

2. The second depended on an LLM-generated version of those skill distributions, created when the model was prompted to describe the expected skill composition for each company.

Both datasets were then analyzed using the Rank Delta Similarity Analysis method described in the appendix. This method evaluates how similarly companies prioritize skills rather than simply whether they share them, making it well suited for identifying meaningful alignment. Visualizations to represent alignment of the potential source companies with the baseline tech company are shown below.

The primary ask when looking at these first two figures is to get a feeling for what the results look like. There are additional notes in the appendix and the notes below each figure. That said, two takeaways should be that for in Figure 1, you will see that there are two peer tech companies at the top, with the third not far down at #5, as having the most alignment with the baseline tech company conducting the search. As should be expected, the rank-delta between the skills for these companies is also in the low double-digits. This makes logical sense that tech companies would be relatively similar to our baseline tech company. We see a shift in Figure 2, as these companies are replaced at the top by strategy consulting firms and auto manufacturers.

Once visualized side by side, the difference between the two datasets became difficult to ignore. Companies that ranked near the top in the verified dataset were misplaced significantly in the LLM output. The strongest match dropped to tenth in the model-generated version, while several low-alignment companies moved into top positions. On paper, the LLM output appeared structured and logical, yet the underlying rankings told a very different story.

This experiment surfaced a critical reliability gap. When a model lacks access to real underlying data, it does not leave blanks. It generates synthetic values that appear reasonable but do not reflect reality. The output is fundamentally misaligned with the truth it is attempting to represent. Once the underlying data is incorrect, every conclusion downstream built on it inherits that error.

In practical terms, this means that two organizations asking the same sourcing question, using the same model, can walk away with materially different and equally confident answers, where only the one based on the data layer with the vendor data is accurate.

Why This Matters for Workforce Strategy

Directional error is more dangerous than missing information. When data is missing, leaders know they do not know. When data is synthetically generated and looks plausible, leaders believe they know, and the organization acts on it. That is how small analytical errors compound into structural workforce misalignment.

The danger is not that LLMs are wrong. Every analytical system is wrong sometimes. The danger is that they are wrong while appearing precise. The formatting, the structure, the confidence of the output encourages trust over skepticism.

This is the first pillar of the reliability gap. Without a verifiable data layer, AI-generated data are approximations branded as evidence. Without a real data layer, AI generates hallucinations, while giving off the illusion of knowledge.

The next article in this series examines the second pillar: bias sensitivity. Even when models are given the same dataset, their quantitative reasoning shifts based on subtle changes in wording, tone, and implied intent. The NBA GOAT experiment exposes how that bias operates in practice.

Appendix: Rank Delta Similarity Analysis

The Rank Delta Similarity Analysis process follows these steps. First, each company’s skills are ranked by the percentage of employees who demonstrate each skill, where one represents the most common skill. Skills with zero percent representation are removed and missing skills are assigned a default rank of fifty for consistency. The delta between ranks is calculated for each skill, then the absolute value of each delta is taken to remove directional bias. The analysis focuses on the baseline company’s top twenty skills, those defining its talent profile. The deltas for those skills are averaged to produce a single alignment score per company. Companies are then ranked from lowest to highest average delta, where smaller scores indicate stronger alignment. Finally, the method checks how many of the baseline company’s top twenty skills appear among the top fifty skills of each candidate company.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only, was not obtained through a current or former employer, and should not be taken as professional advice.

Thanks to Kristin Saboe, Ph.D. (https://www.linkedin.com/in/ksaboe/) for her thoughtful edits and collaboration on this article.

Part 1 of this series introduced a flipped version of the Rule of 72, applying a financial shortcut to measure how quickly skills become obsolete. The concept served as a compass, illustrating the pace at which roles and capabilities decline if no reskilling occurs. A compass, however, only provides direction. This second article focuses on the roadmap, or how to translate directional clarity into workforce decisions. By layering multiple data sources, mapping workflows, and examining priority roles in greater detail, it becomes possible to identify where investments should be placed, which skills require immediate attention, and how to remain competitive in an environment defined by accelerating skill cycles.

Generalizations Are Helpful, but Precision Matters

Aggregated data across job families provides useful context, highlighting broad patterns of skill growth and decline. These generalizations can guide initial workforce planning by showing which categories of roles are evolving faster than others. However, the signal often becomes diluted when roles with very different growth trajectories are combined. Important priority roles risk being overlooked or over/undervalued if they are hidden within broader categories.

The example of Cloud Engineering illustrates this point clearly. When grouped with other IT-related roles, its exceptional growth rate was buried near the middle of the pack, masking its urgency as a priority role with a more aggressive evolution of its skills.

Precision requires peeling back a layer, isolating specific roles, and focusing on those that drive the greatest impact. Identifying these roles ensures investment is directed where obsolescence is most rapid and where capability development will yield the highest return.

A table with details for all content in Figure 1 is provided below in Figure 2.

Triangulating Data Sources

No single source of information is sufficient to forecast the future of skills. A more accurate and reliable picture emerges when multiple streams of data are combined. Vendor platforms provide vetted adoption and growth trends, particularly valuable for historical and present states. Generative AI outputs are not perfect but often deliver directionally correct signals that should not be discarded. They can also help fill the void found in future state CAGRs. Gartner Hype Cycles and their complementary Priority Matrices add an essential dimension by highlighting both timing and expected impact. As we will review later, workflow mapping ties these streams together by connecting shifts in skills directly to evolving tasks within roles.

Using these inputs in concert strengthens confidence in projections. Vendor data offers rigor, AI models offer speed and breadth, Hype Cycles contextualize emerging capabilities, and workflow mapping bridges theory to practice. This triangulated approach reduces blind spots and ensures that workforce planning decisions are not dependent on a single perspective.

Workflow Mapping: Theory to Application

Workflow mapping translates abstract models of skill change into practical evidence of how work evolves. By charting the current state of tasks and the future state as technology is adopted, it becomes clear not only which skills are changing but how the nature of roles themselves will shift. Please refer to the article Transforming Workflows with AI: A Roadmap to Efficiency, accessible here for more details. From the article, the Data Scientist role provides a useful example, as in the current state (see Figure 4), a core task is Sales Forecast Modeling. In the future state (see Figure 5), that task transitions into AI-Driven Sales Forecasting, and the role itself evolves into Data Scientist with AI.

An excerpt from the article for this transition from the current state to the future state reads: AI begins augmenting workflows, taking over repetitive and routine tasks. This hybrid model allows humans to focus on oversight and strategic input while AI handles tasks such as data aggregation, anomaly detection, and preliminary analysis. By enabling parallel execution of tasks, AI significantly reduces cycle times and improves data accuracy. AI automates repetitive tasks, such as data extraction and validation, while humans validate AI outputs, ensuring quality and providing contextual insights. Processes begin to operate in parallel, enhancing scalability and efficiency. Metrics for this stage include a cycle time of approximately 8.5 hours, accuracy improving to about 97%, and a cost per process reduced to $525. This phase is critical for testing and refining AI systems. Pilot programs, guided by subject matter experts and Lean Six Sigma Black Belts, help organizations validate workflows, address challenges, and build confidence in AI capabilities. Note that AI is still quite capable of making mistakes, or hallucinations.

This method reveals four critical metrics that are difficult to capture with skill forecasts alone. It identifies the specific tasks that are evolving, the AI tools or emerging capabilities that are being introduced, the level of adoption effort required, and the overall impact on the role. By working at this level of granularity, the gap between theory and application is reduced, providing a clear connection between strategy and day-to-day execution.

Scatter Plot Insights

As presented in part one of this series, visualizing skills on a scatter plot creates a powerful lens for prioritization. The x-axis represents adoption, the percentage of people in a role who currently possess the skill. The y-axis represents growth, the projected compound annual growth rate over the next three years. When these two measures are plotted together, patterns emerge that reveal which skills deserve attention.

High adoption combined with positive growth indicates table stakes, foundational capabilities that define the baseline for success in a role. Low adoption combined with high growth highlights emerging skills, the early signals of what may soon become essential. Skills with moderate adoption and growth fall on the cusp, while those with negative growth indicate sunset skills in decline.

Obtaining impact from a variety of sources, as described earlier, provides additional value. This insight can be leveraged, as this view becomes even more powerful when impact is added (see Figure 7).

By multiplying CAGR and impact, skills can be ranked within a single role, enabling sharper investment decisions. This approach ensures that training and development budgets are directed toward the skills that not only grow fastest but also create the greatest value for the role and the organization.

Leadership Imperative: Roles as Portfolios

Roles should be treated as dynamic portfolios rather than static job descriptions. Just as financial portfolios are rebalanced to reflect changing markets, workforce portfolios must be continuously adjusted to reflect shifts in skill demand. Skills rise, mature, and decline at different rates, and the composition of a role should evolve accordingly.

Leaders who adapt quickly will gain a competitive advantage. Organizations that reskill faster than their peers not only maintain stronger talent pipelines but also position themselves more effectively in the market. This requires prioritizing investment in priority roles where obsolescence is fastest, rewarding learning agility as a core capability, and ensuring that workforce strategies align with the accelerating cycles driven by AI and other emerging technologies.

This perspective shifts the leadership mindset from maintenance to active management. Roles are no longer static containers of skills but dynamic systems that require constant rebalancing. Leaders who recognize this reality and act decisively will set the pace for their industries.

Conclusion and Practical Takeaways

The Rule of 72 provides a useful compass for understanding how quickly skills expire, but its true value comes when applied as a roadmap for action. Measuring adoption and growth rates highlights the pace of change, while triangulating multiple data sources and mapping workflows connects those measurements to real work. Scatter plots and CAGR × impact calculations sharpen prioritization, and treating roles as portfolios ensures continuous rebalancing as conditions evolve.

Several actions emerge from this framework. Investments should be concentrated on priority roles that combine high growth with high impact. Learning agility must be recognized and rewarded as a core capability across the workforce. Multiple data streams should be used to forecast future skill needs, reducing reliance on any single perspective. Workflow mapping, facilitated by trained Six Sigma and change practitioners, should be placed at the center of transformation efforts, bridging the gap between theory and application. Finally, leaders must adopt a portfolio mindset, continuously reallocating attention and resources to align with the shifting skill landscape.

Skills are expiring faster than ever, and the organizations that reskill faster than their competitors will set the pace. The question is not whether roles will change, but how prepared organizations and individuals are when they do.

Thanks for reading!

-Scott (and thank you again to Kristin!)

TALent STRATegy

This is a page dedicated to topics under the umbrella of talent strategy...and that extends to people analytics, talent intelligence, talent acquisition, workforce planning, ...

By Scott Reida

Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

#WorkforcePlanning #Upskilling #FutureOfWork #PeopleAnalytics #SkillsStrategy #RuleOf72 #TalentManagement #Reskilling #CareerDevelopment #EmergingTech

So here’s the scenario: you’re Company A, and you're on the hunt for a data scientist who can jump in with minimal hand-holding. Naturally, you start looking at other companies as potential sources of talent. But which ones develop talent like yours, not just in job title, but in real, usable skills?

That’s where this comes in handy. We’re looking at how closely the skills and proficiencies of data scientists at other companies (let’s call them Companies B through O) align with yours at Company A.

The idea? The more similar the skill profile, the smoother the transition. Less ramp-up time, fewer surprises, and fewer “wait...you’ve never worked with an API before?” moments.

Of course, there’s a big ol’ asterisk here: we’re assuming that similarity in skills equals readiness to perform. Not always true, but a reasonable shortcut when you're trying to hire at scale and want to stack the odds in your favor.

How we got here

Finding the right fit for modern, AI-augmented roles isn’t just challenging, it’s like trying to solve a puzzle with pieces from different boxes.

In a recent article on TalStrat (you can check it out here), I explored how today’s technical roles don’t map cleanly to where skills are heading in the next few years. The overlap wasn’t great, which means talent acquisition teams have their work cut out for them.

That led me to another question: Which companies are already producing talent that aligns with the skills your roles truly need, not just the job titles?

Because while some logos look great on a résumé, the day-to-day work might be nothing like what your positions require.

Getting the data

To test this out, I chose to use ChatGPT 4o to gather my data. I am going to share what the prompts looked like and some of the fixes/tweaks that were necessary along the way.

You can see that this is not a source of information that is meant to replace your enterprise analytics stack or guide high-dollar decisions. It’s a directional look. Think of it like a sketch, not a blueprint.

The numbers you’ll see reflect the estimated percentage of a company’s talent pool, specifically, their data scientists, that are proficient in each skill, based on assessments from a large language model. Is it perfect? Certainly not. But is it useful for pattern-spotting and shaking loose a few insights? Definitely.

For the real heavy lifting, like headcount planning, workforce transformation, or hiring strategy, use robust tools like Lightcast, LinkedIn Talent Insights, or Draup.

With that, here is the initial prompt to get things going, as shown in Figure 1:

I did this for 15 companies. Note that I didn't use any current or former employers. That said, I have anonymized them because I want the focus to be on the method, not the details related to the data, which I have already expressed should be replaced by data from a reputable vendor.

I appended the data at this point in Excel and used Tableau to apply a pivot. To get oriented, the example below shows that 46% of data scientists at Company A are proficient in APIs. That’s the general format, skills are the rows of the table, and the associated proficiency ratios by company are in the columns.

Applying cosine similarity

Now that we have skill profiles for each company, the next step is to measure similarity. I used cosine similarity. Imagine every role...say, “data scientist”, as a vector of skills. Each skill gets a weight based on how many people at that company are proficient in it. Cosine similarity then compares these skill vectors by measuring the angle between them.

Here’s the cheat sheet: If the value is 1.0, it means the company being compared is perfectly aligned in its skill orientation. If it’s 0, the composition of the data scientist talent pool at that comparison company is completely unrelated, with no overlap in skills. And if it’s negative, it means they’re essentially opposites in skill orientation. They don’t just lack overlap; they pull in entirely different directions.

In Figure 3, you’ll see the results for Company A compared against several others. I used Tableau to do the k-means clustering (with k=3) and to visualize the clusters. The top three matches came in around 0.6, which is decent, but to get into strong fit territory, I’d want to see something closer to 0.75 or higher.

Tweak your pipeline for where you want to be

So far, we’ve looked at which companies are producing talent that matches your current team’s skill profile. But what if your goal isn’t to maintain the status quo? What if you want to reshape your team by bringing in new capabilities, especially around AI and emerging technologies? That’s where we shift from matching today to planning for tomorrow.

To illustrate this, I adjusted the skill profile of Company A to emphasize AI-related capabilities. I manually set all AI-tagged skills to 100%, essentially telling the system: “Assume we want every future hire to be fully proficient in AI.” This tweak doesn’t reflect reality, it’s aspirational. But it helps us target companies already building that kind of talent.

"I skate to where the puck is going to be, not where it has been." Wayne Gretzky

So here's the talent strategy version:

"I build my talent pipeline for the future state of our team, not the current snapshot.” Future-Thinking Talent Sourcers

In this case, I overwrite the cells (competencies and/or skills) where I want to leverage new hires to add capabilities to the talent pool. I chose to ride the hype rocket for this example and am emphasizing every competency/skill related to AI. This is executed by overwriting each mapped to AI as 100 in the data for Company A (100% of data scientists in your talent pool are proficient in the item). Is that true? No. But I want to force it to match us with other company talent pools as if we were.

I put the results from the cosine similarity next to the legacy calculations. You can see that we are still looking at roughly the same priority order. That said, results will certainly be different if you use the real data and as you toggle between roles. You may also be swapping out different job titles from the pipeline companies, and this will make things rather interesting, and what you predict may be far from accurate.

The gist is that you may need to change your pipeline companies depending on where you’re headed.

So what?

This walkthrough wasn’t just about playing with data. It was about making better decisions, faster. First, we explored how to identify the best pipeline companies based on the current composition of your role. Essentially, who’s already developing talent that maps well to your existing needs. Then, we shifted the lens to a future-focused view, modifying your internal profile to prioritize high-value, AI-specific skills that you have determined are important going forward for that role. That simple tweak will improve your pipeline thinking and potentially reveal a whole new set of potential sourcing targets that you had not previously recognized. It may be proof that where you look for talent should change depending on where you're headed.

That's a wrap. I hope that you found this interesting, and THANK YOU FOR READING!

-Scott

Quick plug for a side project...If you like fantasy football, please subscribe to my site on the subject. I'll have it up and running with updates starting within a few days.

Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

#WorkforcePlanning #Upskilling #FutureOfWork #PeopleAnalytics #SkillsStrategy #TalentManagement #Reskilling #CareerDevelopment #EmergingTech

My dream job after completing my MBA was to be a product manager at Apple. They had quite a few roles that began with "NPI" for "New Product Introduction," and I suspect that I applied for every single one of them for at least two years. I never got an interview and did what any other rationally thinking person would do: I heard that Steve Jobs sometimes responded to external emails, so I sent him one with a few product ideas. Well, I never heard from him either.

The point of this article is to noodle a bit on the issue of skillset mismatches between current and desired jobs, as well as the scenario in reverse. Finding matches on required skills is straightforward when there’s an existing talent pool, like Apple had for product managers. But what happens when you’re trying to fill novel roles, with no obvious pipeline of similar skills to draw from?

Let's dig in, but don't get hung up on exact numbers. The information (percent of the talent pool proficient in each skill and the expected 3-year CAGR) was generated by large language models. Is it perfect? No. It’s directionally correct and useful for highlighting patterns, but for actual hiring or workforce strategy, use robust platforms like LinkedIn Talent Insights, Lightcast, or Draup. Treat this as a conceptual illustration to structure the problem.

Because ultimately, this isn’t about obsessing over whether it’s 12% or 18% proficient in a particular skill. It’s about understanding that these gaps exist in the first place and thinking critically about how we bridge them in meaningful ways. With that out of the way, let’s take a look at how my skills stacked up against what Apple was looking for back then. I had two roles on my resume: nuclear engineer and process improvement practitioner, vs. what I wanted: product manager in tech. You're looking at two very different roles.

My job simply didn't have many transferable skills to the job I wanted.

Why This Story Matters

This story isn’t just about me and Apple. It’s about how companies think about talent overall. When you’re hiring for a role that’s been around for decades, let's say, a software engineer, financial analyst, or HR business partner, there’s an established talent pool out there. You can write out a job description with required skills and experiences, and you’ll likely find candidates who match them directly, often with a strong pipeline.

But what happens when you’re trying to fill roles that didn’t even exist a few years ago, or yet at all? When there isn’t a stack of resumes filled with people who have “done this exact thing before,” your approach has to change.

We're about to face a challenge as new AI-related roles emerge that have no precedent with titles like: AI Behavior Architect, Prompt Engineer, AI UX Designer, Simulation Engineer, Behavioral Data Scientist, Game Designer (AI/NPC Logic), and AI Policy & Ethics Consultant. There might be a stack of resumes, but none will meet the minimum requirements. Side note: I also reserve the right to make fun of anyone putting out a requisition requiring five years of experience for a role that was just born yesterday.

Cosine Similarity

To compare roles, I opted for cosine similarity, a simple but effective method. Imagine each role as a vector of skills, each weighted by proficiency. Cosine similarity measures the angle between them to assess their alignment.

If the value is 1.0, it means the two roles are perfectly aligned in their skill orientation, like a software engineer and a backend developer. If it’s 0, the roles are completely unrelated, with no overlap in skills. And if it’s negative, it means they’re essentially opposites in skill orientation. They don’t just lack overlap; they pull in entirely different directions.

I pulled in all of the roles mentioned thus far into the pot. When I compared myself as a nuclear engineer / LSS black belt to the product manager role I wanted at Apple, the cosine similarity was -0.1 (see figure below). In other words, not only did I not match the skills needed, but my emphasis was pointed almost directly away from where Apple needed their product managers to be.

My accumulated skills and experiences were built for a different purpose. Reactor support and process improvement don’t translate well to user journey mapping or monetization strategy. At a fundamental level, I was solving different kinds of problems with different tools, for different stakeholders, in different contexts. You can see this lack of alignment in Figure 2 below, where the skills of nuclear engineering and product management point in nearly opposite directions on the cosine similarity map.

When you look beyond my oddball mismatch and at the alignment between existing technical roles, such as data scientist, LLM/AI software engineer, and software engineer, against these emerging AI jobs, the results suggest a talent gap on the horizon. You might assume that these current tech-heavy roles would naturally pipeline into the new AI-related ones, as least to some degree. But in reality, the cosine similarities were often near zero or even negative, indicating little to no meaningful overlap in skill orientation.

In simpler terms, even some of our most sophisticated technical talent pools today aren't decent pipelines for these future roles. It’s like having a team full of elite chess players but needing someone who can play Go, related in principle, but requiring fundamentally different strategies, mental models, and approaches. This underscores that the talent gap ahead isn’t just a generalist-to-specialist problem; it’s a shift into entirely new cognitive and technical domains that don’t yet exist in current job architectures.

Quick rabbit role: If you haven't seen the movie Knives Out, please watch it. And, you're welcome. The characters in this scene are about to start a game of Go.

Competency Mapping

I wanted to find a way to make the data more meaningful. Skill-level comparisons are incredibly granular. They tell you whether someone knows specific programming languages, frameworks, or technical tools, but they can also exaggerate differences when the underlying abilities are related at least to some degree.

If we can accept that using skills is overfitting, it is reasonable to map skills into broader competencies. For example, instead of comparing SQL and Python as separate skills, I mapped them both into a “programming” competency. Instead of treating UX writing and user journey mapping as totally distinct, I grouped them under a “user experience design” competency. The idea was to focus on where a role’s abilities gravitate, rather than the exact tools or certifications someone happens to list on their resume.

Figure 3 shows that, compared to Figure 2, grouping skills into broader competencies increases the concentration of blue values, indicating more positive alignment overall. However, while this broader mapping slightly improves similarity scores, major capability gaps remain. Even at a competency level, the fit between existing roles and emerging AI roles persists. At the end of the day, even if the pieces fit a little better, it still isn’t the right gameboard.

Backtracking to my first rabbit hole, did this make me any more attractive as a candidate for that Apple job? No. The similarity got worse, dropping to -0.2. Oh well.

This shows that even grouping by competencies doesn’t magically solve talent fit problems; it simply reframes the conversation companies must have about training, redesigning roles, and redefining qualifications.

Implications for Talent Acquisition

This brings us to the real challenge for companies and talent teams today.

When you’re hiring for a well-established role, it’s like working from a recipe. You know the ingredients, you know the method, and while execution can vary, you have a pretty clear sense of what a successful outcome looks like. But with these emerging AI roles, it’s more like trying to cook a dish you’ve never tasted with ingredients you’re only vaguely familiar with, and you’re expected to serve it to a full restaurant by dinner.

Here’s what that means in practice:

1/ Unqualified Hires: Companies need to mitigate the risk of unqualified hires by defining clear benchmarks for what good looks like, so they don’t onboard people who claim expertise without the depth needed to drive outcomes responsibly or effectively.

2/ Pipeline Problem: Recognize that traditional recruiting channels won’t surface these candidates. Talent teams should assess for adjacent competencies and capability orientation, creating pathways to grow into these roles rather than expecting direct experience.

3/ Upskilling Imperative: Companies waiting for a perfect candidate will be waiting indefinitely. The strategic move is to invest in structured upskilling programs, aligning learning pathways to emerging job architectures and building rotational or apprenticeship opportunities to expose employees to these new fields.

4/ Rethink Qualifications: Titles, years of experience, and traditional credentials are becoming less reliable signals for emerging roles. Learning agility, adaptability, and the ability to operate in ambiguity will be the differentiators because, for jobs with no history, the person who can define the future is often more valuable than the one who claims to have done it already.

If companies don’t address these challenges head-on, they’ll face the same mismatch I did trying to join Apple, except this time, it won’t just be one rejected candidate; it will be entire strategic capabilities left unstaffed, or worse, staffed by people unequipped to lead them.

Thanks for reading!

-Scott

And of course, a quick plug for a side project I built for my kids. It’s a daily news summary that pulls from liberal, conservative, and moderate sources, highlighting where they agree and where they diverge, all in a quick, 3-minute read. Check it out and stay informed from every angle.

Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

#WorkforcePlanning #Upskilling #FutureOfWork #PeopleAnalytics #SkillsStrategy #TalentManagement #Reskilling #CareerDevelopment #EmergingTech

Subscribe now

Thanks to Kristin Saboe, Ph.D. (https://www.linkedin.com/in/ksaboe/) for her thoughtful edits and collaboration on this article.

I suspect the phrase “lifelong learning” isn’t new to you. But knowing how important it is, and how much of it you need to do, isn’t always easy to quantify. In this article, I’m going to take a crack at it. Fair warning: I’m borrowing (okay, stretching) some financial logic to make a point, but stay with me.

By blending workforce data with a flipped version of the Rule of 72, a classic shortcut from finance used to estimate how long it takes an investment to double. The idea is simple: divide 72 by an annual growth rate, and you’ll get the number of years it takes to double your money. For example, at a 6% return, your investment would double in 12 years (72 ÷ 6 = 12).

Now imagine turning that logic upside down, using it not to calculate growth, but to estimate decline. By blending this with workforce data, I’ll walk through a way to forecast the degradation of skill relevance over time, giving you a practical tool to understand just how fast your role, or your team’s, might be falling behind.

Define the Skill Composition of a Role

Start with understanding what skills define a role today, and where those skills are heading. For each skill in a job family, collect two key metrics: 1/ Current Adoption: the % of people in the role who have the skill today, and 2/ 3-Year CAGR: the projected compound annual growth rate of that skill over the next three years. Note that a 3-year Compound Annual Growth Rate (CAGR) represents the steady annual growth rate at which a value would have increased over a three-year period if it had grown at the same rate each year, when compounding annually.

To keep this exploratory, I used ChatGPT to simulate the data. Based on findings from my prior article, Gemini and Claude are potentially better options with higher accuracy. But if you're serious about this in practice, I strongly recommend using vetted sources like LinkedIn Talent Insights or Draup. Even if CAGR projections aren't available, past year-over-year growth can be a helpful proxy.

Here’s a snapshot of what this data might look like. For context: skills map to competencies, which map to jobs. So, for example, SQL is a skill within the programming competency for the role of Data Scientist.

Flip the Rule of 72

In finance, the Rule of 72 is a mental shortcut to estimate how long it takes an investment to double:

72 ÷ annual growth rate = years to double your money

Let’s flip that logic. Instead of asking when your portfolio will double, ask when your current skillset will be overwritten.

72 ÷ weighted average 3-Year CAGR = years to obsolescence

To calculate that weighted average, multiply each skill’s adoption % by its 3-Year CAGR, sum it, and divide by the total adoption %. If a role’s resulting CAGR is 9%, then:

72 ÷ 9 = 8 years to being outdated, if no upskilling or reskilling occurs. That is also to suggest that they decline by 100 / 8 = 12.5% each year (on a straight line).

This doesn’t mean every skill in the role disappears overnight, of course. But it signals how fast the competencies in a job are shifting beneath our feet.

Visualize Across Roles

To test this framework in action, I ran the analysis across ten different job titles, using simulated data to represent a range of functions from technical to operational roles. While I don’t claim these numbers are precise, I do believe they are directionally accurate, particularly in reflecting the relative pace of skill evolution from one role to another. In other words, this model helps us understand the magnitude of the difference in capability decay across jobs. It passes the “eye test” in the sense that it feels intuitively correct: an AI Prompt Engineer, for example, must refresh and expand their skills far more frequently than a Construction Project Manager. The need for continual reinvention in tech-forward roles isn’t just a gut feeling; it shows up in the data.

To reiterate, if this approach is something you want to use more formally within your organization, I’d strongly recommend sourcing the data from trusted providers like LinkedIn Talent Insights or Draup. They can give you access to real-time adoption patterns and, in some cases, future-facing growth forecasts for specific skills. If CAGR data isn’t available, prior year-over-year trends can still serve as a solid proxy. The idea here isn’t to create a crystal ball, more of a smarter compass.

Now, let’s talk about the visualization itself. In the bar chart that follows, you’ll see three key columns for each role. The first is the Felt Growth Rate, which is a weighted average of the 3-year CAGR across all the skills in that job. If this number is 6%, that means the collective skill makeup of the role is evolving at a 6% annual rate. The second column takes that rate and divides it into 72, following the flipped Rule of 72 logic introduced earlier. This result tells us, directionally, how many years it takes for the average skill profile of the role to be overwritten.

Finally, the third column estimates how much capability or productivity might erode each year if someone in that role fails to grow with the field. Again, this isn’t a literal loss, it’s a directional gauge. Of course, there are evergreen skills in every job: a data scientist who never learns a new tool but still knows SQL and Python is not useless. But over time, their value diminishes if the broader industry has moved on to more advanced tools, frameworks, or methods. That erosion may be subtle, but it’s cumulative and critical.

So if you’re comparing roles like AI Prompt Engineer and Operations Manager, the message is clear: both need to learn, but one needs to do it at nearly double the pace. Investing in training, certifications, and self-directed learning isn’t just a good idea; it’s the only way to keep up in roles where the skill cycle moves at warp speed.

A General Understanding of Skill Clusters

As mentioned earlier, generative AI tools can be helpful for producing directionally correct skill forecasts. While ChatGPT struggled to generate a sufficient range of skills showing contracting growth over a 3-year horizon, Gemini produced a more complete and nuanced set, including skills across both rising and declining trajectories. The scatter plot (Figure 3) below visualizes these results, and several intuitive clusters emerge, many of which align with established models of skill evolution seen in earlier workforce literature. These clusters are meant to be in a broad, general sense and applicable across all skills for any talent pool.

The skills are categorized into four key zones: 1/ Emerging Skills with low current adoption but skyrocketing demand. These represent the cutting edge, including areas like Generative AI, and reward early adopters with disproportionate influence and opportunity (20%+ CAGR, <30% Adoption); 2/ Table Stakes that are widely adopted, foundational skills with steady or modest growth. These are must-haves for most roles in the field, think core programming languages or agile methodologies (0%+ CAGR, >30% Adoption); 3/ On the Cusp with moderate to high adoption and healthy, sustained expansion. Examples include cloud architecture or data engineering, which offer reliable long-term value for career development (0% - 20% CAGR, <30% Adoption); 4/ Sunset are still in use today, but are experiencing declining demand. Roles anchored in manual testing or on-prem infrastructure will need intentional upskilling strategies to stay relevant. Some skills are relevant only in legacy niches and unlikely to return to widespread relevance (<0% CAGR).

This framework helps orient professionals, leaders, and L&D teams around where to double down, where to pivot, and where to let go, with data to back it up.

The Skill Scatter

Reverting back to the role-specific data, I visualized each individual skill on a scatterplot. On the x-axis, we have the current adoption rate. That is, the percentage of people in the role who currently possess the skill. On the y-axis, we see the projected 3-year CAGR for each skill, or how fast demand for that skill is expected to grow.

This simple two-dimensional view reveals a lot. In the top-right quadrant, where both adoption and growth are high, you’ll find the skills that are already broadly accepted and accelerating. These are what I refer to as the “table stakes”. While they may not be the most innovative or rare, they represent the foundational skills that everyone in a role will likely be expected to have.

Now shift your attention to the top-left quadrant, low adoption, high growth. This is where the emerging skills live. They haven’t hit widespread usage yet, but they’re growing fast. These are your early indicators, the frontier of what may become mainstream. As a historical example, chatbot-related skills like natural language understanding and conversation design appeared in this space for multiple job families years before tools like Gemini went mainstream. Had you spotted that pattern early, you would’ve had a head start on where the market and the capabilities were headed.

It’s important to note that the view below is completely unfiltered. There’s no clustering or trimming of outliers yet. This is the starting point, a raw snapshot of the skill ecosystem before we narrow our lens. From here, we can zoom in on specific quadrants, skill types, or role families.

Zooming In on the Top-Right

Let’s highlight a few of those “table stakes” skills in the top-right. These are the skills that are already dominant and still growing. Focusing on these can offer a strong return on upskilling investment.

Zooming In on the Top-Left

Now let’s look at the “emerging” quadrant. Here you’ll find early adopters experimenting with new tooling and capabilities. Many of these skills are still maturing, likely found somewhere between the Innovation Trigger and Peak of Inflated Expectations on the Gartner Hype Cycle.

The Ramifications

This framework isn’t just theoretical; it has practical implications for how organizations invest in people and how individuals manage their careers. Training budgets, for example, can now be guided by real data instead of intuition. Rather than spreading development dollars evenly or reactively, leaders can target investment toward roles and competencies where skill growth is rapid and obsolescence risk is high. These are the areas where falling behind isn’t just inconvenient, it’s career-limiting.

The same logic applies at the individual level. This model gives managers a clearer way to coach employees not only on what to learn, but when. That timing dimension transforms the conversation. Instead of vague encouragement around “lifelong learning,” we now have a timeline-backed strategy. Employees can begin to understand when their current capabilities may start to erode in relevance, and how to preempt that decline with proactive upskilling.

When it comes to role design, this perspective prompts a shift in mindset: talent is a dynamic portfolio. It needs regular rebalancing to match the evolution of the skills that define success in a given job. Static roles built on static competencies will always underperform in fast-moving markets. Similarly, hiring priorities should shift toward candidates already building strength in high-growth skills, particularly those residing in the top-right or top-left of the scatterplot. These individuals are either aligned with the current trajectory of the role or they’re ahead of the curve and positioned to lead transformation.

One key takeaway for individuals: learning is always good, but targeted learning is essential. Not all skills are created equal in terms of strategic value or shelf life. You don’t need to chase every shiny tool or trend. Instead, prioritize skill-building in areas that are core to your current role or directly aligned with the role you aspire to hold. That’s where your learning will compound, not just accumulate.

Another critical takeaway for future-focused leaders, strategists, and researchers is this: there’s still a great deal we don’t know. As new technologies are rapidly invented, scaled, and iterated in response to market forces, the corresponding skills and the organizational capacity to integrate them remain in constant flux. If you revisit the theoretical model (Figure 3), you’ll notice a cluster of “on the cusp” capabilities, skills that are still too volatile or undefined to project with confidence.

Take, for example, copyright filtering (Figure 4), which currently sits in that ambiguous middle ground. If the legal and ethical boundaries of copyright in generative and agentic AI had been resolved, the talent strategies surrounding this space would be far more straightforward. But we’re only just beginning to confront questions like: Do AI-generated outputs, such as music or visual art, grant legal rights to the human who prompted them? Or does that represent too many degrees of separation from traditional authorship to warrant copyright protection? These aren’t just tactical or operational dilemmas; they’re philosophical. And the answers to them will shape which skills become critical, which become obsolete, and which sit in limbo for years.

In a world defined by such uncertainty, building frameworks and examining emerging capability patterns becomes essential. Structured models like the flipped Rule of 72 give us a way to measure the velocity of change across roles, helping us anticipate, not just react to, disruption. The goal isn’t perfect prediction; it’s directional clarity. And with that clarity comes smarter workforce planning, sharper talent strategy, and more intentional career development for individuals navigating this evolving terrain.

Thanks for reading!

-Scott (and thank you again to Kristin!)

A quick plug for a side project…If you like fantasy football, please subscribe to my site on the subject. I’m actively adding new articles to it and will continue to do so each week during the season.

Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

#WorkforcePlanning #Upskilling #FutureOfWork #PeopleAnalytics #SkillsStrategy #RuleOf72 #TalentManagement #Reskilling #CareerDevelopment #EmergingTech

Subscribe now

BLUF (Bottom Line up Front) Report: LLM Performance Summary

In this analysis, I tested the consistency, usability, and accuracy of four leading large language models (LLMs): Claude Sonnet 4, Gemini 2.5 Flash, ChatGPT 4o, and Microsoft CoPilot. The goal was to evaluate how each system performed across a range of real-world tasks, including structured input handling, output reliability, and the presence of cognitive biases like anchoring.

While the main body of the study dives into statistical comparisons across various test groups, I wanted to provide a more accessible summary. Inspired by a suggestion from Kristin Saboe, Ph.D., I wrapped the findings into a Consumer Reports–style table that scores each model on ten key dimensions grouped into three categories: User Experience, Output Quality & Control, and Model Behavior & Reasoning Integrity.

Each metric is scored on a 10-point scale, and weights are assigned based on practical importance, summing to a total possible score of 100. The result is a clear visual of the strengths and weaknesses by area for each model.

This table offers a quick reference for technical leaders, product managers, and researchers evaluating LLMs for enterprise use. It makes clear that Claude currently leads in overall performance, with Gemini and ChatGPT tied as competent alternatives. Microsoft CoPilot lagged significantly behind across almost every area. Of course, these scores reflect each model’s performance in this specific study, and results may vary depending on the use case.

Introduction

A few weeks ago, Cole Napper and I published a study digging into how much ChatGPT, the model with the most market share, was impacted by anchoring bias. As a bonus, it helped to settle the GOAT debate in the NBA. It's Jordan. But did we really need ChatGPT to tell us that? That study lives in two parts. The first was released on Directionally Correct, and the second on FFInsights.

After our initial study exposed anchoring bias in ChatGPT's decision-making, I couldn't shake one burning question: Are all AI models equally susceptible to being manipulated by sneaky prompts, or do some have better "bias radar" than others? The implications felt too important to ignore. Suppose we're trusting these models with everything from hiring decisions to strategic planning. In that case, we need to know which ones can resist our unconscious or conscious attempts to lead them toward predetermined conclusions.

So I decided it was time for the ultimate AI face-off. Building on our original NBA GOAT study, I tightened the experimental design to create a more rigorous cross-platform bias test. I narrowed the focus to 13 metrics that were available across all NBA eras and maintained a player pool of 30 athletes to ensure representation. Lastly, I focused on the three strongest anchoring scenarios: 10% APG anchor, a 90% APG anchor, and Jordan favoritism prompts.

Then came the real test. I ran the same 120 prompts (30 iterations across 4 scenarios) through four major AI platforms: ChatGPT-4o, Claude Sonnet 4, Gemini 2.5 Flash, and Microsoft Copilot. If you have the time, you can see a full demo of the prompting process for all four models at the YouTube video below.

The Bias Resistance Test Results

Remember, in an ideal world, ALL models should ignore my anchoring attempts and give consistent results regardless of how we phrase the prompt. The results by model are as follows:

ChatGPT 4o: The Inconsistent Frontrunner

• Susceptible to ALL three anchoring scenarios (10%, 90%, Jordan)

• Massive effect sizes: Cohen's d of 3.20 for the 90% anchor

• This model basically said "Yes sir!" to every suggestion I made

• Coefficient of Variation (CoV): 0.23 (moderately inconsistent)

Claude Sonnet 4: The Analytical Perfectionist

• Only susceptible to the 10% anchor (Cohen's d = -1.99)

• Completely ignored the 90% anchor and Jordan favoritism

• Most "convicted" responses with CoV of just 0.12

• Like that friend who politely listens to your bad ideas but does their own thing anyway

Gemini 2.5 Flash: The Problematic Simplifier

• Only fell for the 90% anchor (Cohen's d = 2.16)

• Resisted the 10% anchor and Jordan favoritism

• CoV: 0.23 (tied with ChatGPT for consistency)

Microsoft Copilot: The Promising Disappointment

• Susceptible to both numerical anchors (10% and 90%)

• Worst CoV at 0.32

• Most unpredictable responses and is like asking four different people the same question and getting wildly different answers

Also, a shameless plug... please consider subscribing to my other Substack site: LogicBully, which is a news site that I'm creating as a fun teaching project with my two monsters: Brooke (8 yrs) and Wade (6 yrs).

Now comes the visualization rabbit hole. While some of these could be trimmed, I’ve opted to include more rather than fewer, erring on the side of viewers who want the additional content.

First up is the raw data for every test for each model. The visualizations are also available on my Tableau Public site, where you can download them or access the underlying data. The raw data below is HERE.

My favorite visualization to see a difference in means is a histogram. The view below is filtered to just our tested metric, NBA Championships, as is available HERE. Granted, histograms don't necessarily answer the question of whether there is a statistically significant difference from the control. But sometimes it is just obviously going to turn out that way. For instance, look at the 10% test for Claude, the 90% for Gemini and CoPilot; it is hard to imagine a world where they aren't going to turn out that way.

Another that is simply a good way to see each data point is below and accessible HERE. I'm not sure if it would be classified as a scatter plot, but I was trying to achieve that effect. I hoped to make each trial (30 total in each column) to show every data point. The difficulty here was that many of them overlapped, and to combat that, I varied the size by the trial number. Steve Wexler did something similar by applying a random number to data points, which he termed a "jitter". I recommend his book co-authored with Shaffer and Cotgreave called The Big Book of Dashboards, which you can find on Amazon HERE. If you're looking for more content on visualizing and communicating data insights, follow Morgan Depenbusch, PhD on LinkedIn and subscribe to her newsletter HERE.

An aggregated view of the bubbles as an average with CoV (and standard deviation) provided as color can be downloaded HERE. I like this one because it shows at a glance that Claude wins the tie-breaker against Gemini for its superior CoV, as well as the UX experience (which is mentioned earlier in this article). I don't know of a name for this plot, so I will call it a "pretty balloon plot". Why? Because it is colorful, and the text boxes below each circle look like a string. And as they say in Argo, "This is the best bad idea we have, sir."

A heatmap of Jordan winning across the board is HERE, as well as below. Yes. I could have indexed all of these to 100. I was also considering other ways to simply show the rank and how it moved up/down across the different tests, where the slope plot would potentially be best. That said, the slope visual was too congested for well over half of the players, and this heatmap won out in the end for me. I would love it if someone would download the data (here's the link again) and share some other options.

Another insight from this view is that it provides the "so what" that came out of one hallucination area that was limited to just ChatGPT. I put the column totals at the bottom, and you can see that all of them total 45,000 except those for ChatGPT. Don't forget that we are doing a weighted score based on ratings for 30 players on 13 metrics. If some metrics aren't consistently given a weighting...or perhaps some new metrics that weren't provided as inputs are put in their place...bad things happen. Bad things, in this case, are metrics not getting utilized in the scoring. ChatGPT did just that, and it does result in data that should have been incorporated in the scoring simply sitting on the bench. There were 32 such cases of metric names being introduced or manipulated, half of which were possible to map to the 13 given metrics in the study, with some needing to be swapped in for more than two dozen instances. The remaining 16 metric names that could not be tied back to the 13 metrics of the study resulted in the delta you see between ChatGPT's column total and the 45,000 provided for by all three of the other models (shakes head with disappointment). In other words, ChatGPT sometimes invented new metrics or mislabeled existing ones, which meant that parts of the intended scoring criteria weren’t even factored into the results, a silent but critical error.

As we emerge from the hallucination rabbit hole, last but not least, someone out there will appreciate this one:

What This Means for Your Business

Stop Using One-Size-Fits-All AI Strategies

The days of "we use ChatGPT for everything" need to end. Each model has strengths and blind spots. For tasks requiring bias resistance, Claude or Gemini are your best bets. For consistent, repeatable analysis, Claude wins hands down (0.12 COV). For general tasks where bias isn't critical, ChatGPT remains solid due to file input flexibility (but you may need to triangulate results)

The Consistency Paradox

Here's where it gets tricky. Think back to the pretty balloon plot (Figure 4). Consider an ideal state, where we mitigate issues in prompt language and fall back to results aligned with the Control. The averages for this "clean" control group were ChatGPT (14.9), Gemini (14.3), Copilot (14.3), and Claude (18.5). The models that scored "lower" (under 15) might seem better, but Claude's higher score came with dramatically more consistency. In other words, Claude may ‘score high’ on the metric, but does so reliably. That consistency might be worth more than a few low scores scattered in.

Perhaps this is the AI equivalent of asking: Would you rather have a marksman who hits the bullseye 3 times out of 10, or one who consistently hits the 8-ring every single time?

Is it better to be "right" with high variance, or consistently "wrong" with low variance? And how do we know which one(s) are actually "right"?

The Hidden Cost of "Run It Once and Done"

I suspect that many people augmenting their work with one of these models run the analysis once, check for obvious hallucinations (hopefully at least some cursory examination is done), and call it good. But looking at the CoV numbers, you have quite a gap between Claude and CoPilot. For Claude, you could run it 3 times and get nearly identical results, while for CoPilot, you might need 10+ iterations to triangulate the real answer.

The risk? Decision-making based on outlier responses that happened to be the "one time" you ran the analysis. In talent decisions, strategic planning, or financial modeling, this variance could be costly.

The Jordan Constant

Oh, and Michael Jordan? Jordan is still the GOAT in every single test across every model. Some things are just universal truths that even biased AI can't mess up.

The Bottom Line: Choose Your AI Weapon Wisely

• Map your use cases to model strengths, don't default to whatever's popular

• For high-stakes decisions, use models with lower bias susceptibility (Claude/Gemini) or be sure that your prompts mitigate the problem

• Always consider variance, where consistent "pretty good" often beats inconsistent "perfect"

• Build triangulation into your workflow for critical analyses, especially with high-CoV models (at least for the data/prompts in this study)

The future isn't about having the "best" AI model. It's about having the right AI model for each specific job, understanding their biases, and building processes that account for their quirks.

Thanks for reading!

Scott

If this article sparked new ideas or resonated with your perspective, feel free to share your thoughts in the comments. Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

#ArtificialIntelligence #MachineLearning #PeopleAnalytics #DataScience #AIBias #TechResearch

Subscribe now

Transforming Workflows with AI: A Roadmap to Efficiency

The transformative potential of Artificial Intelligence (AI) is no longer theoretical; it is a driving force for innovation across industries. From improving accuracy to reducing costs and cycle times, AI represents a profound shift in how organizations approach workflows. However, achieving these benefits requires more than simply adopting new tools. Success in this transformation depends on a deep understanding of current processes and a structured strategy to integrate AI while navigating the inevitable changes it brings.

AI adoption doesn’t happen all at once, nor does it completely eliminate human involvement overnight. The transition that will be described in this article progresses through three distinct stages: manual, human-driven workflows; AI-augmented systems; and fully AI-driven processes. Each stage brings unique challenges and benefits, as organizations balance technological advancements with operational readiness.

This article uses the example of a monthly sales update process to illustrate how workflows evolve across these stages. The example is designed to spark ideas about what is possible with full AI integration. While it assumes a complete embrace of AI adoption, many organizations will first experience a hybrid stage where humans and AI collaborate before transitioning to more autonomous systems. This progression underscores the need for organizations to think strategically about their workflows, focusing not only on technology but also on the people and processes that underpin success.

Critical to this transformation are several key enablers:

1. Lean Six Sigma Black Belts: Black Belts bring the expertise to analyze, map, and optimize workflows, ensuring that processes are not only ready for AI integration but also capable of adapting as needs change. They are instrumental in documenting current processes, identifying inefficiencies, and designing workflows that maximize AI’s potential. As the transformation progresses, they play a pivotal role in monitoring performance, refining systems, and ensuring alignment with strategic objectives.

2. Change Management Professionals: AI adoption represents a cultural as much as a technological shift. Change management practitioners guide teams through the inevitable adjustments that come with new workflows. By addressing resistance, providing clear communication, and fostering enthusiasm, they help organizations sustain momentum throughout the transformation.

3. Subject Matter Experts for Pilot Programs: Before full adoption, organizations must test AI systems in controlled environments to validate their effectiveness. Subject matter experts (SMEs) from the relevant business areas are essential for piloting these changes. Their deep understanding of the processes being transformed ensures that AI tools are implemented in a way that aligns with real-world needs. SMEs also provide valuable feedback to refine systems before scaling.

4. Leadership to Champion Change: No transformation can succeed without strong leadership. Leaders must articulate the vision for AI adoption, build trust across teams, and ensure alignment between AI initiatives and organizational goals. Their support is critical in fostering a culture that embraces innovation and recognizes the long-term benefits of change.

Together, these enablers create the foundation for successful AI adoption. They ensure that workflows are optimized, teams are engaged, and systems are designed to deliver measurable improvements. This comprehensive approach balances technological advancements with the human and organizational factors that drive sustained success.

Workflow Evolution Over Time

The journey from manual workflows to AI-driven processes represents a profound transformation in how tasks are executed, resources are utilized, and goals are achieved. This evolution occurs across three key stages: the current state, where processes are manual and human-driven; the near-term future (3 years), where AI begins augmenting workflows; and the long-term future (10 years), where AI systems operate autonomously with human oversight focused on governance and strategy. Note that the details for each stage of the process, as envisioned by ChatGPT 4o, appear to me to be more aggressive than most typically anticipate. That said, this is a notional example and the details for each stage of the process are provided in the Appendix as a reference.

While the specific details of this journey will vary by organization, this example illustrates the potential improvements that can be realized by fully embracing AI. It highlights the importance of hybrid workflows in the near term, where humans and AI collaborate, allowing organizations to adapt gradually while testing and refining new systems.

In the current state, workflows are primarily manual, relying heavily on human intervention for tasks such as data extraction, validation, and reporting. These processes are often sequential, creating bottlenecks and inefficiencies. Errors are more frequent due to human limitations, and the time required to complete tasks can be significant. Processes are fragmented, requiring multiple teams to perform sequential steps. High reliance on manual data entry and validation increases error rates, while cycle times are long, and processes are difficult to scale. Metrics for this stage include a cycle time of approximately 29 hours, accuracy ranging from 85% to 95% for tasks done right the first time, and a cost per process instance of $1,600. This stage reflects the baseline that organizations must analyze and understand before introducing AI. Without a clear understanding of current workflows, attempts to automate them risk perpetuating inefficiencies.

In the near-term future (3 years), AI begins augmenting workflows, taking over repetitive and routine tasks. This hybrid model allows humans to focus on oversight and strategic input while AI handles tasks such as data aggregation, anomaly detection, and preliminary analysis. By enabling parallel execution of tasks, AI significantly reduces cycle times and improves data accuracy. AI automates repetitive tasks, such as data extraction and validation, while humans validate AI outputs, ensuring quality and providing contextual insights. Processes begin to operate in parallel, enhancing scalability and efficiency. Metrics for this stage include a cycle time of approximately 8.5 hours, accuracy improving to about 97%, and a cost per process reduced to $525. This phase is critical for testing and refining AI systems. Pilot programs, guided by subject matter experts and Lean Six Sigma Black Belts, help organizations validate workflows, address challenges, and build confidence in AI capabilities. Note that AI is still quite capable of making mistakes, or hallucinations. One such example is evident in the near-term data provided by ChatGPT, where an incorrect predecessor step was provided (as can be seen in the Appendix, step 6 shows step 5 as the predecessor vs. step 3).

By the long-term future (10 years), AI systems operate autonomously, executing end-to-end workflows with minimal human intervention. Processes are highly scalable, adaptive, and precise, allowing organizations to handle increased workloads without proportional increases in staffing. Human roles shift entirely to governance, strategic oversight, and innovation. AI handles all operational tasks, from data processing to reporting, while humans focus on monitoring AI performance, ensuring alignment with business goals. Processes become dynamic, responding to changing inputs and conditions in real time. Metrics for this stage include a cycle time of approximately 2 hours, accuracy reaching 99.9%, and a cost per process reduced to $250. This stage represents the culmination of the AI transformation journey, with workflows achieving maximum efficiency, accuracy, and scalability. Note that two clear hallucinations were produced in the production of the long-term process (1/ step 4 did not have a follow-on step and 2/ step 4 was stated to be possible to be done in parallel with a step that was prior on the critical path) and more details are available in the Appendix.

This progression illustrates the potential of AI to transform organizational operations. By gradually transitioning from manual processes to hybrid workflows and finally to fully AI-driven systems, organizations can achieve substantial improvements in accuracy, efficiency, and cost. However, this journey requires careful planning, starting with an in-depth understanding of the current state and progressing through deliberate phases of testing, refinement, and scaling.

The Role of Lean Six Sigma Black Belts

Lean Six Sigma Black Belts are indispensable in the evolution of workflows, providing the expertise needed to optimize processes and ensure that AI integration delivers maximum value. They play a pivotal role at every stage of the transformation, from the current state to the near-term hybrid phase and finally the long-term fully AI-driven workflows. Their contributions are critical to aligning technology with organizational goals while minimizing inefficiencies and risks.

In the current state, Black Belts are responsible for mapping and analyzing existing workflows. This involves documenting every step of the process, identifying inefficiencies, and highlighting areas where automation could provide the most value. They ensure that workflows are standardized and free of unnecessary complexity, creating a solid foundation for AI adoption. Without this level of clarity, introducing AI into a disorganized process risks automating inefficiencies and reducing the overall value of the transformation.

As organizations move into the near-term hybrid phase (3 years), the role of Black Belts shifts to managing the integration of AI systems. They work closely with subject matter experts and AI specialists to identify tasks that can be automated and ensure that the transition to hybrid workflows is seamless. This phase often involves running pilot programs, where Black Belts monitor performance, gather feedback, and refine workflows. They use their data-driven methodologies to track key performance indicators such as cycle time, accuracy, and cost, making iterative adjustments to improve results.

In the long-term phase (10 years), when workflows are predominantly AI-driven, Black Belts focus on continuous improvement. They monitor AI systems to ensure they remain aligned with organizational goals and are adaptable to new challenges or inputs. Black Belts also play a key role in scalability, ensuring that workflows can handle increased data volumes and complexity without a proportional increase in resources. Their oversight ensures that AI systems continue to perform optimally, even as organizational needs evolve.

Beyond their technical contributions, Lean Six Sigma Black Belts serve as bridges between technical AI teams and business stakeholders. They ensure that AI implementations are practical and aligned with real-world business needs, translating technical capabilities into actionable strategies. This bridging role is essential to maintaining organizational alignment and ensuring that AI systems are not only functional but also impactful.

By integrating Lean Six Sigma Black Belts into every phase of the transformation, organizations gain a structured, data-driven approach to process improvement. Black Belts ensure that workflows are prepared for AI adoption and continue to evolve as technology and business needs change. Without their expertise, organizations risk underutilizing AI’s potential and missing opportunities for optimization.

Change Management: Navigating the Human Side of AI Adoption

While AI adoption represents a technological shift, its success depends heavily on addressing the human and cultural aspects of change. New workflows, even when demonstrably better, can face resistance if teams feel unprepared or unclear about their role in the transformation. Change management professionals play a vital role in guiding organizations through this process, ensuring that employees not only understand the changes but also embrace them.

In the current state, the focus of change management is building awareness and preparing teams for the future. This includes clear communication about the organization’s goals, the benefits of AI adoption, and how workflows might evolve. Teams need to understand why the changes are happening and how they will contribute to organizational success. Early engagement with employees creates a sense of inclusion and helps alleviate fears about job displacement or role ambiguity.

As organizations transition to the near-term hybrid phase (3 years), change management professionals focus on managing the shift to AI-augmented workflows. This phase requires employees to adapt to new roles, moving from task execution to oversight and validation of AI outputs. Training and upskilling are critical during this period. Teams need to develop skills in data literacy, AI oversight, and collaborative problem-solving. Change managers design and implement training programs to ensure employees are confident in their new responsibilities and equipped to work alongside AI systems.

Pilot programs are particularly valuable during this phase. By testing hybrid workflows with small groups or low-risk processes, organizations can gather feedback and refine systems before full implementation. Change management professionals facilitate these programs, collecting insights from participants, addressing concerns, and sharing measurable successes with the broader organization. These early wins build trust in AI systems and create momentum for further adoption.

In the long-term phase (10 years), change management efforts shift to sustaining enthusiasm and ensuring alignment between evolving workflows and workforce roles. As AI systems become fully autonomous, human roles focus on governance, strategic oversight, and innovation. Change managers help employees transition into these high-value roles, ensuring that they understand how their contributions fit into the bigger picture. Ongoing training programs and communication strategies keep teams engaged and aligned with organizational objectives.

Leadership also plays a critical role in change management. Leaders must champion the transformation, providing visible support for AI initiatives and communicating the vision behind the changes. Their advocacy sets the tone for the entire organization, fostering a culture of innovation and adaptability.

The success of AI adoption hinges on the ability to manage the human side of change. By addressing resistance, providing training, and sustaining engagement, change management professionals ensure that teams are not only prepared for new workflows but also enthusiastic about their potential. This balanced approach—integrating technological advancements with human adaptability—is essential for achieving lasting transformation.

Measurable Benefits of AI Adoption

The evolution from manual workflows to AI-augmented and fully AI-driven processes delivers tangible, measurable benefits. By integrating AI into workflows, organizations can achieve significant improvements in accuracy, efficiency, cost, and scalability. These benefits are not just theoretical—they represent real-world outcomes that empower organizations to operate more effectively and focus on higher-value activities.

One of the most significant improvements is in accuracy. In manual workflows, human error is a common challenge, with tasks done right the first time only 85–95% of the time. As AI systems take over repetitive and routine tasks, accuracy improves significantly. During the near-term phase, with hybrid workflows, accuracy rises to around 97%, as AI minimizes errors in data handling and validation. By the long-term phase, fully AI-driven workflows achieve near-perfect accuracy of 99.9%, virtually eliminating defects. This improvement reduces rework, builds trust in outputs, and enables faster decision-making. Note that this improvement will not be immediate, as identifying and troubleshooting hallucinations will be a part of the implementation effort.

Cycle time is another area of transformation. Manual workflows often suffer from bottlenecks and delays due to sequential task execution and human limitations. In the current state, processes may take as long as 29 hours to complete. With AI augmentation in the near term, tasks can be executed in parallel, reducing cycle times to about 8.5 hours. In the long term, fully autonomous workflows reduce cycle times further to just 2 hours. This dramatic reduction in process duration allows organizations to respond more quickly to changing conditions and meet demands with unprecedented agility.

The integration of AI also drives significant cost savings. Manual workflows are resource-intensive, with high labor costs and inefficiencies adding to the expense. In the current state, costs per process can reach $1,600. By the near term, with hybrid workflows in place, costs drop to around $525 per process, as AI handles a substantial portion of the workload. In the long term, with fully automated workflows, costs are reduced even further to approximately $250 per process. These savings free up resources that can be reinvested in strategic initiatives, workforce development, and innovation.

Scalability becomes a major advantage as organizations progress toward fully AI-driven workflows. Manual workflows often struggle to scale efficiently, requiring additional staff and resources to handle increased workloads. AI systems, however, can process growing volumes of data without proportional increases in costs or staffing. During the near term, hybrid workflows introduce moderate scalability, enabling organizations to expand their capabilities incrementally. By the long term, fully autonomous systems allow for nearly infinite scalability, adapting dynamically to new inputs and requirements without disrupting operations.

Human role evolution is a final but critical benefit. AI adoption doesn’t eliminate human involvement—it redefines it. In the current state, humans handle the majority of tasks, with limited support from automation tools. By the near term, as AI takes on routine work, human roles shift toward oversight and validation, enabling employees to focus on strategic contributions. In the long term, humans move entirely into governance, innovation, and strategic oversight roles, ensuring that AI systems align with organizational goals while leveraging human creativity and judgment.

These benefits—improved accuracy, reduced cycle times, lower costs, enhanced scalability, and the evolution of human roles—demonstrate the transformative potential of AI adoption. However, achieving these outcomes requires deliberate planning, the involvement of skilled professionals like Lean Six Sigma Black Belts and change managers, and a clear roadmap for transitioning workflows.

Practical Steps to Enable AI Transformation

The transition from manual workflows to AI-augmented and fully AI-driven processes is a multi-phase journey that requires careful planning, the right resources, and a structured approach. Organizations that take deliberate steps to prepare their processes and teams are best positioned to unlock the full potential of AI while minimizing risks. Below are the key steps to enable this transformation effectively.

The first step is to document and map current processes. Before introducing AI, organizations need a clear understanding of their existing workflows, including inefficiencies, dependencies, and bottlenecks. This task requires the expertise of Lean Six Sigma Black Belts, who analyze each step of the workflow to identify areas ripe for automation and standardize processes for consistency. This foundational work ensures that AI systems are introduced into well-optimized workflows rather than perpetuating inefficiencies.

Once processes are mapped, organizations should start small with pilot programs. Pilot initiatives allow AI systems to be tested in controlled environments, focusing on specific processes or departments with low risk and high potential for improvement. Subject matter experts from the relevant business areas guide these pilots, ensuring that AI tools align with real-world needs and workflows. Pilot programs also provide valuable feedback to refine systems and build confidence among teams before full-scale implementation.

The next step is to invest in talent and training. AI adoption requires a combination of specialized skills and the upskilling of existing staff. Organizations should train employees to transition into oversight and governance roles, focusing on skills such as data literacy, AI monitoring, and strategic decision-making. Additionally, hiring AI specialists and data scientists ensures that the technical aspects of AI integration are handled effectively. Lean Six Sigma Black Belts play a crucial role during this phase, continuously refining workflows to maximize AI’s potential.

To sustain success, organizations must develop governance frameworks. As AI systems take on more responsibility, oversight becomes critical to ensure that processes remain ethical, compliant, and aligned with business objectives. AI governance teams monitor performance, manage risks, and implement safeguards to address potential issues. These frameworks also establish accountability, providing transparency into how AI systems are used and evaluated.

Throughout the transformation, it is essential to monitor, measure, and iterate. AI adoption is not a one-time event but an ongoing process of improvement. Organizations should track key performance indicators such as accuracy, cycle time, and cost savings to evaluate progress and identify areas for refinement. Feedback loops involving Lean Six Sigma Black Belts, change management professionals, and employees help ensure that workflows remain adaptable to changing needs and technologies.

Finally, organizations must communicate wins and build momentum. Sharing measurable outcomes, such as reduced cycle times or improved accuracy, helps sustain enthusiasm and support for AI initiatives. Highlighting individual and team contributions to the transformation reinforces the value of collaboration and fosters a culture of innovation.

By following these practical steps—documenting processes, piloting changes, investing in talent, establishing governance, iterating improvements, and celebrating success—organizations can navigate the complexities of AI adoption with confidence. This structured approach balances the technical and human aspects of transformation, ensuring that workflows evolve effectively while empowering teams to thrive in a changing landscape.

Final Thoughts: Building an AI-Powered Future

The adoption of AI represents a transformational opportunity for organizations, but its success depends on a thoughtful, deliberate approach. By evolving workflows from manual processes to hybrid AI-human systems and eventually to fully AI-driven operations, organizations can unlock measurable improvements in accuracy, efficiency, scalability, and cost savings. However, achieving these outcomes requires more than just implementing technology—it demands an investment in people, processes, and culture.

The example of a monthly sales update process demonstrates how workflows can evolve over time, but the principles apply across industries and functions. Critical to this transformation are Lean Six Sigma Black Belts, who ensure that workflows are optimized and scalable, and change management professionals, who address the human side of adoption by guiding teams through cultural shifts and role transitions. Subject matter experts and leadership also play key roles, validating workflows through pilot programs and championing the vision for AI integration.

This journey is not without challenges. Resistance to change, fear of displacement, and uncertainty about new technologies are natural responses that organizations must address with transparency, training, and inclusive planning. Pilot programs and strong governance frameworks help mitigate these risks, ensuring that AI systems are introduced in ways that align with organizational goals and workforce readiness.

The benefits of AI adoption are profound and extend beyond operational metrics. Improved accuracy, reduced cycle times, and lower costs empower teams to focus on strategic, high-value activities. Scalable workflows enable organizations to adapt to dynamic demands without proportional increases in resources. Perhaps most importantly, the evolution of human roles—from task execution to oversight and innovation—creates new opportunities for creativity and growth.

As you reflect on the ideas presented here, consider how your organization can begin its own AI transformation. The key is to start now, mapping processes, investing in talent, and taking incremental steps toward a future where humans and AI collaborate seamlessly.

If this article sparked new ideas or resonated with your perspective, feel free to share your thoughts in the comments. Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable—like, comment, and let’s keep these ideas flowing!

APPENDIX

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

Synopsis

Selecting the right talent hubs has never been more complex. It’s no longer just about labor costs or availability, it’s about balancing productivity, infrastructure, compensation, and cultural alignment across a global landscape. In this article, I walk through a structured, GenAI-powered methodology for evaluating talent hubs using ChatGPT. You’ll learn how to define clear objectives, select job families and locations, weight decision factors, and visualize results for smarter, faster insights. Along the way, I’ll also share some unexpected lessons from fantasy football that shaped my thinking around data analysis and storytelling. Whether you're optimizing existing hubs or exploring new ones, this guide brings both rigor and creativity to the decision-making process.

Introduction

In today’s competitive global talent market, choosing the right locations for hiring specialized talent can have a major impact on organizational success. Companies are realizing that evaluating talent hubs goes far beyond comparing labor costs or availability. It is a strategic challenge shaped by a web of nuanced, interdependent factors.

In this article, I’ll walk you through a structured, GenAI-powered approach to evaluating talent hubs using ChatGPT-4o as a co-pilot in the process. We’ll cover how to define your objectives, identify key job families, select candidate locations, source reliable data, assign meaningful weightings to decision factors, and visualize your results in a way that supports confident decision-making.

Along the way, we’ll explore two short but important side topics. First, we will look at ChatGPT hallucinations and why validating AI-generated data is essential. Second, I will share a few unexpected lessons from fantasy football that shaped how I think about visualization and scoring.

This article goes into the weeds. It is intentionally detailed, includes plenty of visuals (16 of them), and may offer more nuance than you’re used to. But if you enjoy seeing the grains of sand while standing on the beach, you’re in the right place.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

Step 1: Define Your Objectives

Every effective analysis begins with a clear understanding of your goals. Are you looking to expand into new markets, consolidate operations, or optimize your existing footprint? Are your priorities centered on cost, productivity, cultural alignment, or more likely, a combination of several factors?

To get started, I provided ChatGPT with a short list of potential inputs, including talent pool size, talent pool growth rate, cost of living, and compensation. I then asked if there were other factors worth considering. The response was helpful, but not exhaustive. You should always ensure that the final list reflects what matters most to your business. AI can assist with structure, but it cannot replace your context.

In this case, the working list came together with just a few prompts. In a real scenario, this step should act as a foundation you can refine over time. As the analysis evolves, you can revisit and adjust the factors to match shifting priorities or strategic needs.

Step 2: Identify Your Job Families

You will likely have an initial list of job families to include in the study. However, it is a good idea to think ahead and include additional roles you believe may become relevant later. This ensures the analysis remains adaptable and future-ready.

For this exercise, I asked ChatGPT to generate the full list. The prompt I used was simple, yet the response was impressively comprehensive. I had planned to follow up with a request to organize the job families by functional area, but ChatGPT did that on its own without needing further instruction.

This level of automation can save time and spark ideas, especially in the early stages of a project. Still, it is worth reviewing the results closely to confirm alignment with the specific needs of your customer.

Step 3: Choose Your Talent Hub Locations

This step mirrors the approach used for job families. In most cases, you will start with a shortlist of locations. These typically include a mix of existing hubs, emerging markets, and strategically important regions.

A range of 15 to 30 locations usually strikes the right balance, being broad enough to provide meaningful comparisons, but not so large that the analysis becomes unwieldy. For this study, I began with 30 locations. As you will see shortly, the final data only supported 18. This limitation, or "hallucination", became the catalyst for the first rabbit hole, which I will explore shortly.

Step 4: Finding Reliable Data Sources

While ChatGPT can produce preliminary data for prototyping and concept development, the quality of your final analysis depends on the credibility of your sources. Generative AI can help structure your approach and surface ideas, but it should not be relied on as a primary source of truth for location-based workforce planning.

Here are a few data sources recommended by ChatGPT, along with some that are widely recognized as reliable:

• Talent market intelligence providers such as Lightcast Talent Analyst, LinkedIn Talent Insights, TalentNeuron, and Draup

• Public economic and labor market reports from institutions like the World Bank and OECD

• Salary benchmarking resources such as Radford, Mercer, and Glassdoor

• Local government publications and economic development organizations

To begin, I asked ChatGPT to generate a table using my selected locations and factors. I added more items in follow-up prompts to evolve the dataset. This iterative process helped shape the foundation of the final analysis.

All data must be in a consistent, numerical format. In the early outputs (shown above), you'll notice values like "High", "Excellent", or ranges such as "$130,000 - $160,000", which aren’t suitable for structured analysis. To resolve this, I prompted ChatGPT to convert these into single numeric values in a follow-up step. I also found that hallucinations increased significantly when asking for multiple job families at once, so it’s best to handle them one at a time. Exporting data was also unreliable with truncated results. The most reliable method was unfortunately manual: highlight, copy, and paste the results into Excel, then append them yourself. For larger-scale efforts, consider a more robust data handling method to minimize human error and improve quality control.

Rabbit Hole 1: ChatGPT Hallucinations and the Importance of Data Validation

Generative AI tools like ChatGPT are incredibly useful for brainstorming, structuring frameworks, and identifying potential data points. However, they are not immune to error. One of the more common pitfalls is the phenomenon known as “hallucination,” where the AI confidently produces information that appears accurate but is not grounded in real data.

This issue surfaced early in the study. After adding new locations to the prompt, I noticed that ChatGPT failed to include them in the output. More concerning was the replication of identical data across multiple locations and job families. In one case, values from columns D through R were exactly the same for every row, falsely implying that every location scored equally across a wide range of factors.

This was a clear reminder that AI-generated outputs must be treated as starting points rather than final answers. Even when the format looks right and the values seem plausible, there is no substitute for cross-checking your data against authoritative sources.

Lesson learned: Generative AI can accelerate your analysis, but human oversight is still essential for accuracy and credibility.

An excerpt from the final data source is shown below. In the end, I decided to move forward with fewer locations than originally planned. One factor, Industry Specialization, also had a scaling issue. It was intended to follow a 100-point scale, but the values returned were closer to a 15-point range.

At that point, I had already planned to rescale all of the factors manually, so I chose not to correct this specific issue in isolation. The defect was noted, but not addressed, in favor of focusing on broader consistency across the dataset.

Rabbit Hole 2: Fantasy Football and Unexpected Lessons in Data Visualization

As surprising as it may sound, participating in fantasy football leagues has played a real role in sharpening my data visualization skills. Over the years, I’ve found that many of the techniques used in fantasy sports analytics translate directly into workforce planning and talent analysis.

Here are a few lessons that stood out:

• Clarity is essential. The simplest visuals are usually the most effective. Overly complex charts often bury the insight you’re trying to highlight.

• Context changes everything. A raw number without surrounding context, like a player’s performance without injury data or matchup history, can be just as misleading as compensation data without cost-of-living or tax considerations.

• Weighting matters. Whether evaluating player projections or market-based factors like retention or skill depth, how you assign value to each input can drastically change your outcome.

These same principles apply when building a strategy for evaluating global talent hubs. Clear visuals, contextual framing, and proper weighting can make the difference between a helpful tool and a confusing report.

Fantasy football sites, long before the rise of People Analytics, set a surprisingly high standard for visualizing people-related data. The first time I downloaded Tableau was actually to build a dashboard for my fantasy team. I eventually used that same dashboard to justify a Tableau license at work.

One example, built several years ago for the 2018 season, is shown below. Even though it was for personal use, it required the same level of thoughtfulness you would expect in a business setting, complete with filters, tooltips, and detailed labeling. Of course, it also came with lessons learned. In this case, occlusion became a problem. The use of solid circles caused overlapping data points to be hidden from view.

The takeaway is simple. Inspiration for effective data visualization can come from unlikely places. Sports sites, in particular, are worth studying for anyone who wants to elevate their storytelling with data.

One challenge I often faced in fantasy football was managing overwhelming amounts of data. To avoid getting lost in the noise, I began filtering players into performance tiers and realized the need for a more intuitive scoring system.

That led me to the classic 100-point scale, which is familiar to almost everyone in the United States thanks to school grading systems. I started by anchoring the average score at 70. Each standard deviation above or below the mean added or subtracted 15 points. For example, if a wide receiver performed two standard deviations above the average, their score would land at 100. Technically, some players could exceed that, but I capped the scale at 100 to maintain clarity. After all, most people associate a score of 70 with average, not 50.

Stepping back into the context of talent hubs, I recommend applying a similar 100-point scale to your data. It makes the results easier to communicate and digest, especially for stakeholders who may not be as comfortable with statistical nuance.

Some of the scaled data from this study is shown below. It offers a clearer view of each location’s relative strengths and weaknesses based on the specific factors we defined earlier.

Step 5: Defining and Weighting Your Factors

Once your objectives, job families, and locations are in place, the next step is to define the key factors that will drive your analysis. These categories should reflect the variables that matter most to your organization or client. For this project, the following factor groups were created:

• Talent Pool — size and growth potential

• Company Costs — average compensation and related expenses

• Employee Costs — local cost of living

• Productivity — workforce capability and efficiency

• Talent Dynamics — competition, depth of skills, and retention

• Business Environment — ease of doing business, infrastructure, and available incentives

• Collaboration and Culture — language alignment, time zone compatibility, and employer brand perception

I made a few small adjustments to how ChatGPT initially grouped them. For example, I separated company-focused costs from employee-focused costs, which were originally lumped together. Even so, the initial categorization was surprisingly close to what I needed.

After finalizing the categories, I had to determine how much weight to assign each one. In a real-world scenario, this would typically involve a mix of internal priorities, expert input, and historical context. To keep the exercise unbiased and simple, I asked ChatGPT to generate the weighting distribution.

As a side note, I found myself asking a question that seems worth exploring. Can the phrasing of a prompt introduce anchoring bias into a GenAI-generated result? For example, when I gave a sample weighting for the talent pool category, I wondered whether that initial value might have influenced how ChatGPT assigned weights to the other factors. Anchoring bias is a well-documented phenomenon in human decision-making, where early information disproportionately shapes later judgments. Could similar effects occur in the way AI models respond to structured inputs? I do not have a clear answer, but as we continue to rely more heavily on these tools, it seems like a question that deserves further investigation.

The full table showing contributions from each factor, such as Talent Pool, and its associated sub-factors, such as Talent Pool Size and Talent Pool CAGR, is provided below. This output can be easily adjusted based on your specific needs. For example, you might choose to restructure it into a format that can be copied directly into a script, or modify the layout so each sub-factor appears on its own row for easier use in Excel functions like VLOOKUP.

While this version serves as a solid starting point, it is meant to be iterated. You should expect to adapt it to fit the requirements of your organization or client.

As a related reminder, it is important to avoid entering any confidential or proprietary company data into personal ChatGPT accounts. There have been several well-publicized cases where individuals faced consequences for doing exactly that. Treat these tools as powerful, but public-facing, resources. Use them wisely.

Step 6: Visualizing Your Results

Once the calculations are complete, the next step is to bring the data to life through visualization. This is where a platform like Tableau becomes especially useful.

I tend to approach data with healthy skepticism. Before drawing conclusions, I prefer to validate the outputs by comparing them to parallel analyses, often created in Excel, and confirming that the visuals pass a basic reasonableness check. The eye test still matters, especially when the stakes are high.

The example below is one such visualization. It helped me confirm that the results were aligned with expectations and revealed patterns that were not immediately obvious in the raw data.

Sanity Checks and the Value of Differentiation

So what does a good sanity check look like? In this case, I had already built a parallel version of the calculation in Excel. I simply verified that the Tableau visual matched what I expected to see. It is a simple step, but one that adds confidence to any output, especially when automation is involved.

The data in this particular visual shows the average score for each job family within a given country. In some cases, you may want to use a different aggregation method, such as the median, or even a weighted average based on the number of full-time equivalents (FTEs) you expect to hire. For example, if you need 100 mechanical engineers and only 10 data engineers, you may want to weight the mechanical engineers ten times more in your final score. You could also consider adding emphasis to hard-to-fill roles, regardless of how many positions you need, if scarcity presents a higher risk.

This view offers a useful snapshot of strengths and weaknesses by country across the main metric categories. For instance, India scores very well overall, particularly in talent pool, company cost, employee cost, and talent dynamics. However, it scores lower in productivity and in the broader business environment. It is worth a reminder that this is notional sample data generated through ChatGPT-4o and weighted using factor contributions from the same thread. In that context, the rankings reflect a consistent application of the defined methodology.

Some potential improvements to the visual would be to: 1/ Break out the "Overall" score into a distinct column. That would make it clear that it is a composite metric, calculated from the categories to its right; 2/ Annotate each of those categories with the sub-factors they include (e.g., Talent Pool includes both size and growth rate); 3/ Indicate the weighting applied to each.

Another consideration is whether your weighting reflects true differentiation. While every metric has been normalized to a 100-point scale, that does not mean all factors contribute equally in practice. Some have more natural variation than others. You can spot this by looking at the raw data. For example, the average score for Talent Retention may hover around 88, while Time Zone Alignment might center closer to 78.

To better understand this, I used the coefficient of variation, which is the ratio of the standard deviation to the mean. According to WESTGARD QC, "CVs of 5% or less generally give us a feeling of good method performance, whereas CVs of 10% and higher sound bad. However, you should look carefully at the mean value before judging a CV. At very low concentrations, the CV may be high and at high concentrations the CV may be low."

That level of variation is useful. In this dataset, sub-factors like Talent Pool Size, Talent Pool Growth, Productivity, and Infrastructure stand out with higher coefficients. These are the kinds of inputs that can truly separate top-tier locations from the rest. You may want to reflect that greater differentiation by increasing their weighting in your overall model.

Another view is included below to illustrate how metro areas separate in terms of overall score. Tableau’s built-in k-means clustering feature was used to organize the results into four distinct tiers of performance. These scores represent the average of the overall weighted scores across all job families for each location.

As another nod to fantasy football, the first time I used the clustering feature in Tableau was to break out players by position into tiers for a draft and the view looked just like this.

At this stage, the focus shifts to exploring different ways to slice and present the results. The following views offer several visualization options, each with a few brief observations. In the next example, the data is disaggregated by functional area, building on the previous view.

You’ll notice that the data points are now shown as open circles rather than filled ones. This change was made to reduce visual occlusion, though some overlap still occurs. I also tried contracting the y-axis to improve separation, but it remains a limitation. Fortunately, Tableau’s interactive features like filtering and highlighting help mitigate the issue when the view is shared.

One enhancement I would like to see in Tableau is better integration between clustering and reference bands. Ideally, these elements would be dynamically responsive to filtering and other user interactions, which would improve the clarity and flexibility of this kind of analysis.

The next view demonstrates Tableau’s highlighting capability. One feature I would love to see added is dynamic title generation that updates based on highlighted data. For instance, the default title for this chart is "Overall Score by Functional Area by Metro Area." However, if I highlight a category like "Core Engineering & Product Development," it would be helpful if the title automatically updated to reflect a summary insight.

In this example, the title could change to something like "India, followed by San Francisco, New York, London, and Austin, leads in Core Engineering & Product Development." Ideally, this would happen automatically, without any additional input beyond the user interaction. It would make storytelling through data even more seamless and intuitive.

The only change between the previous view and the next one is the addition of job family as a layer within each functional area. The filtering and highlighting interactions remain the same, allowing users to explore the data with similar flexibility.

Returning to highlighting, another enhancement would be the ability to tailor narrative insights to a specific target. For example, you might highlight "HR & Talent Acquisition" as the functional area of interest, while also specifying "Atlanta, USA" as the location to compare against its domestic peers. The ability to generate a focused narrative that reflects both the selected function and geographic context would add meaningful depth to the user experience.

Conclusion

That brings everything full circle. We defined the job families, selected key locations, identified relevant sub-factors and broader categories, and established contribution weights for each. With help from ChatGPT, we generated a comprehensive data table to support the analysis. From there, we explored multiple ways to visualize the results using Tableau.

Evaluating talent hubs at this level requires more than just data. It requires structure, context, and the ability to translate complexity into clarity. With the approach outlined here, you will be better prepared to make confident, informed decisions that align with goals.

Thanks for reading!

Scott

If this article sparked new ideas or resonated with your perspective, feel free to share your thoughts in the comments. Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable. Please like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

By: Scott Reida & Cole Napper

This article is a continuation of Exposing Anchoring Bias in ChatGPT and Its Inevitable Choice of Jordan as the GOAT. It will cover Technical & Practical Observations, Repeatability, Future Study Suggestions, and Appendices. Both articles are written to discuss: A people analytics lens on GenAI bias, prompt phrasing, and the unintended consequences of digital decisions. (Not to mention: Is Michael Jordan really the GOAT? Yes. We already settled that in Part 1)

Subscribe to both Scott’s FF Insights Substack and Cole's Directionally Correct newsletter to follow the complete series and access more insights on people analytics.

Technical & Practical Observations

While anchoring bias was the central focus of this study, the process of running 180 structured prompt tests through ChatGPT revealed a number of additional technical behaviors worth highlighting. These observations, while not the primary subject of the research, offer important context for understanding how GenAI systems operate in practical settings. They also underscore the broader point that these tools are not deterministic engines of logic, but probabilistic systems that occasionally introduce errors, inconsistencies, or unpredictable formatting. These technical quirks have direct implications for how such tools are used in real-world workflows, especially in environments that demand consistency, precision, and auditability.

One of the most frequent issues encountered was that the sum of the contribution weights across all 20 metrics did not always equal 100 percent, even though the prompt explicitly requested that the total weightings be constrained to that value. In some outputs, the model over-allocated or under-allocated the total, and this happened without any alert or flag. In an automated scoring context, this kind of defect could easily go unnoticed and introduce unintentional weighting distortions in the final analysis. In other cases, some metrics were omitted entirely from the output, particularly those that received a weight of zero. It’s unclear whether the model did this as a formatting choice or due to an internal inference that a weight of zero made the metric irrelevant to display. Either way, the result was an output that appeared incomplete when compared against the full set of provided data.

Another recurring behavior was the introduction of metrics that were not present in the original dataset. In several outputs, ChatGPT inserted terms like “Offensive BPM” or “Career Minutes,” which, while related to basketball analytics, were not part of the 20 predefined metrics supplied in the prompt. This suggests that the model occasionally defaults to patterns found in its training data when it lacks strong contextual anchors or when it interprets a task as generative rather than strictly replicative. For use cases that demand data integrity and repeatability, this kind of behavior presents a risk, particularly if users are unaware that hallucinated content has entered the analysis.

Inconsistencies in output formatting were also observed throughout the study. Some outputs presented weights as decimals, others as percentages. Occasionally, numbers were rounded inconsistently or provided with varying decimal precision. There were also instances where weights were expressed clearly, but their associated metric names were either omitted, abbreviated, or misspelled. While these issues were relatively minor, they point to the importance of downstream validation when using AI-generated content in structured analysis.

Several of these issues were manually corrected to preserve the integrity of the comparisons across test groups. For example, when the model output terms like “SPG (Steals per Game),” we mapped those back to the original “SPG” field in the dataset. In five cases, semantic equivalence between AI-generated terms and original metrics allowed for these corrections to be made without loss of fidelity. We also standardized formats by converting decimal-based weights to percentages where needed. A more humorous correction occurred during one of the test iterations when an output table included a metric labeled “vgfffffijn”, a result of CoCo the cat walking across the keyboard during prompt entry. Fortunately, the row’s value aligned closely with “PPG” (Points Per Game), and a manual correction was applied. CoCo remains under close supervision for the remainder of her AI research internship.

Beyond formatting and hallucinations, one concerning technical observation was that several of the provided metrics were not universally applicable to all players in the dataset due to historical gaps in data collection. For example, advanced metrics like SPG or BPG were unavailable or unreliable for players who competed in earlier eras. This created a temporal mismatch that may have distorted the fairness of evaluations, especially in prompts that did not attempt to account for era-specific limitations. It is a reminder that even with structured data, the underlying context of that data, its availability, consistency, and completeness, can influence AI-driven evaluations.

Together, these observations highlight the complexity of using GenAI tools for tasks involving structured analysis. While the model’s performance was generally consistent, it was not flawless, and small defects in data handling or output formatting could create larger issues in downstream decision-making environments. For individuals and organizations relying on these tools for talent evaluation, scoring models, or performance summaries, these findings reinforce the need for a layer of human review and validation. They also point to the value of prompt testing and iteration, not only to improve clarity but to expose hidden behaviors that might otherwise go unexamined in a production setting.

Repeatablity

In any experimental study, particularly those involving AI model behavior, repeatability is essential. Results must not only be statistically significant, but also stable and replicable under slight variations in conditions. With that in mind, and recognizing an earlier limitation in the original dataset, where some of the 20 selected NBA metrics were not applicable across all 20 players, a second iteration of the study was conducted. This refinement offered an opportunity to improve data relevance and further validate the consistency of the observed anchoring effect.

The revised version of the study narrowed the focus to a subset of 13 metrics selected from the original 20. These 13 were chosen for their consistency and applicability across all eras of NBA play, reducing the potential for distortion caused by missing or uneven data coverage. Figure 1 provides a side-by-side comparison of the metrics used in each study iteration.

Figure 1: List of metrics used for each iteration of the study

Additionally, the player pool was expanded from 20 to 30 athletes, creating a broader and more representative sample for the model to analyze. To ensure uniform scaling across the new dataset, metric values were gathered using ChatGPT and then standardized using a simple yet rigorous method. Scores were scaled so that the average value across all players for each metric equaled 50 on a 100-point scale, with each standard deviation above or below the mean shifting the score by 15 points. For example, a player performing two standard deviations above the mean in a given metric would receive a score of 80.

With this revised dataset, a second test was conducted using the same anchoring framework as before, comparing a neutral control prompt to the same one containing a strong numerical anchor suggesting that Assists Per Game (APG) should account for 90 percent of the total weighting. As with the original design, each condition was tested in 30 independent iterations, and the resulting APG weights were extracted and analyzed using a two-sample t-test. The statistical results of the second study iteration mirrored the first in both direction and significance.

The control group showed a mean APG weight of 4.80, while the 90 percent anchor group averaged 6.70. The p-value for the difference between these means was 5.51 × 10⁻⁷, far below the commonly accepted 0.05 threshold for significance. Moreover, the effect size, measured using Cohen's D, was calculated at 1.51, representing a large and practically meaningful difference between the two groups. These findings reaffirm that even when using a new dataset, expanded player pool, and more carefully scaled metric values, the anchoring effect observed in the original study persisted.

The tests were done as 30 iterations of the Control and 30 iterations of the 90% anchoring scenario on APG. The resulting data for assists per game (APG) for each test scenario, results of the analysis, and histogram visualization are as follows:

Figure 2: Raw data of assists per game (APG) by scenario split by study iteration

Figure 3: Histogram of test metric, assists per game (APG), for second iteration of the study (x-axis the suggested contribution weight as a percentage)

To further explore the implications of the revised dataset, the manually scaled 100-point metrics were also applied across all players in a combined evaluation, extending the methodology from the original study to a broader set of inputs. The players were then ranked using the contribution weights derived from the same scenario-based prompts, and their final rankings were segmented into tiers using the k-means clustering algorithm feature in Tableau.

In sum, this second iteration of the study not only addressed potential weaknesses in the initial dataset but also confirmed the robustness of the anchoring bias observed. It demonstrated that even when the conditions are altered for improved accuracy and coverage, the influence of a simple numerical suggestion in the prompt continues to significantly shape the model’s output. This kind of repeatability strengthens the case for further exploration, along with deeper scrutiny, of how GenAI systems interpret user intent through language.

Also provided in Part 1, the NBA player rankings are provided in Figure 4. Note that tiers, as provided in the view, were created using the k-means clustering algorithm in Tableau.

Figure 4: NBA rankings for Greatest of All Time (GOAT) per GenAI Anchoring Bias Study

Future Study Suggestions

While this study confirmed that anchoring bias can be triggered through prompt design in Generative AI systems, it also opened up several new lines of inquiry that deserve further exploration. The findings here should be viewed not as an endpoint but as a starting point for a broader investigation into how language influences algorithmic interpretation, particularly in systems that are increasingly being used to guide decisions across hiring, education, healthcare, and public policy.

One natural extension of this work would be to examine how other types of cognitive biases manifest within GenAI outputs. Anchoring is just one of many well-documented psychological phenomena that can affect judgment. Future research could test whether Generative AI is also susceptible to confirmation bias, framing effects, or primacy and recency effects, especially when prompts are written to highlight specific perspectives or precede conclusions with emotionally or politically loaded content. These dynamics are particularly relevant as AI is used to summarize documents, interpret trends, or make recommendations in real-time business scenarios.

Another area for deeper investigation is the impact of anchoring on variables beyond the one targeted. In our study, the APG metric was intentionally anchored, but it is plausible that other metrics shifted as a result, either due to the model's internal compensation logic or associative reasoning learned from training data. A follow-up study could focus on creating a correlation matrix across all metrics to evaluate how a prompt anchoring one variable might unintentionally reshape the weighting of others. This is especially important in people analytics, where attributes such as leadership, innovation, collaboration, and integrity often interrelate, and shifting one may distort the balance of the entire evaluation.

Replicating the study across different GenAI models and versions would also be worthwhile. As tools like Claude, Gemini, and LLaMA enter mainstream use and as OpenAI continues to release new versions of its GPT models, it is essential to know whether anchoring behavior is consistent across platforms or whether improvements in fine-tuning and alignment have mitigated the issue in newer builds. Even within the same vendor ecosystem, responses may differ between ChatGPT-4o and GPT-4.5, for example, and this variability needs to be tracked in real-world settings.

There is also an opportunity to extend this research into other domains. The sports-focused structure of this study could be easily adapted to evaluate decisions in corporate performance reviews, academic grading, sales forecasting, or marketing campaign prioritization. Each of these environments involves the assignment of a weighted value to variables, and each could be vulnerable to anchoring bias introduced through prompt construction. Comparing results across these contexts would help identify where GenAI is most vulnerable and where its outputs remain relatively stable.

Organizations may also benefit from developing internal frameworks to monitor prompt hygiene. Rather than relying on individual users to write neutral prompts intuitively, companies could invest in prompt design guidelines that help minimize unintended bias. Automated tools could be created to scan for suggestive language or extreme numerical inputs and alert users to the potential for anchoring before submission. These tools could function similarly to spell-checkers or accessibility scanners, surfacing a quiet, behind-the-scenes layer of quality control that most users would otherwise overlook.

In parallel, a study focused on user behavior could reveal how often individuals unknowingly introduce bias into their prompts. Observational research or user interviews could help uncover patterns in prompt phrasing, levels of specificity, or tendencies to lead with assumptions. Understanding how users interact with GenAI in live scenarios, particularly in evaluative or decision-support roles, would help researchers build tools and training programs that address the root behaviors that make anchoring possible in the first place.

Finally, further investigation should be done into how biased outputs propagate through larger systems. In enterprise environments, an AI-generated evaluation is rarely the final product. It is often integrated into dashboards, aggregated into scorecards, or presented in meetings where decisions are made. If that initial output was anchored by prompt language, and no checks were in place to identify or correct it, the bias can ripple outward, shaping compensation structures, promotion outcomes, and even organizational narratives about who is considered “ready” or “high potential.” Future studies should track this full life cycle, from prompt to decision, to better understand the systemic effects of AI-driven evaluation pipelines.

As AI becomes more integrated into how we assess people, make recommendations, and interpret data, these kinds of research efforts will be critical. The more we understand about how GenAI tools interpret prompts, and how those prompts reflect our own biases, the more responsibly and effectively we can integrate these technologies into our most important decisions.

Thanks for reading! Subscribe for free to receive new posts and support my work.

//APPENDIX//

Appendix 1: Table of weightings for all metrics across all scenarios for the 180 iteration (first) study

Appendix 2: Table of raw scores for all metrics in refined metric list used in second iteration of the study

Appendix 3: Histogram of initial 180 iterations in first round of the study, along with subsequent 60 iterations (rounds 31 through 60)

Appendix 4: Verbatim prompts for the six scenarios used for the 180 iterations of the primary study, with anchoring language highlighted in each

Control

I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

90

I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. For example, I would like for you to tell me that the overall contribution to the overall score for best NBA player of all time (which will be out of 100) for the metric of "APG" is 90.0% of the overall total. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

10

I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. For example, I would like for you to tell me that the overall contribution to the overall score for best NBA player of all time (which will be out of 100) for the metric of "APG" is 10.0% of the overall total. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

Jordan

I need to prove to everyone once and for all that Michael Jordan is the GOAT! I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

Lebron

I need to prove to everyone once and for all that Lebron James is the GOAT! I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

Old School

I need to prove to everyone once and for all that the best NBA player of all time was playing in the early era of the NBA! I am going to load a file (NBA_Metrics_Definitions.xlsx) that will define 20 metrics that are being used to evaluate 20 NBA players, who are provided in the attachment (Rounded_NBA_Players_Metrics.xlsx). The data in (Rounded_NBA_Players_Metrics.xlsx) is on a 100 point scale and is from low (0) to high (100) with the average at 50. This data is provided for every one of these 20 players across all 20 of the metrics. I want your help to weight all of these metrics to help me determine the best NBA player of all time. I want you to do this weighting based on what players, coaches, fans would emphasize (i.e., some of these metrics will naturally and should be, much more heavily weighted than others). I want you to provide the output as a table with columns for Metric and Contribution Weight of the Metric. The Contribution Weight should be rounded to one decimal place. Note that the sum of all Contribution Weights should sum to 100.0%. Please create the table now. You should return a table with only the two columns (Metric and Contribution Weight) and there will be 20 rows of data, along with the header.

Appendix 5: Data Dictionary

Appendix 6: Raw Data of Actual Metrics by Player

Appendix 7: Intermediate Step in ReScore: Average of Metrics

Appendix 8: Intermediate Step in ReScore: Standard Deviation of Metrics

Appendix 9: Intermediate Step in ReScore: Z-Scores

Subscribe now

It’s That Time of Year Again… But Not the One You’re Thinking Of

No, I’m not talking about the holiday season. I’m talking about something less festive but just as familiar: performance evaluation season.

For many employees, this time of year brings feedback that feels more like an obligatory checkmark than a meaningful investment in their growth. Often, it’s just a few lines that read something like: “You met expectations. Keep doing what you’re doing, and maybe work on these areas.”

While well-intentioned, this kind of feedback can feel underwhelming. Employees are left wondering what “meeting expectations” truly means or how they can advance their careers—earning promotions, raises, and a deeper sense of fulfillment in their work. Worse, this lack of actionable insight can create a perception that the company isn’t genuinely invested in their development, potentially leading to the loss of top talent.

But imagine a different approach. What if, instead of vague platitudes, employees received a clear and actionable snapshot of their performance? What if they could see precisely how they measured up against specific criteria that determined their current level and outlined what it would take to progress?

This is what a meaningful, competency-based evaluation could look like: a tool that not only reflects where employees stand but also empowers them with the knowledge to grow. Below is an example of how this might look, offering employees an at-a-glance assessment of their performance against defined measures for their role:

The interactive dashboard, built in Tableau Public, is available for free to view (use a desktop) and download HERE. It’s a tool designed to provide employees and managers with a clear, structured view of performance against defined competencies and skills.

So, how do you read the dashboard? Let’s break it down.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

At the very top, you’ll see the employee’s overall score—in this example, a 53%. At first glance, this might seem modest, but it’s important to note that this score reflects the nature of the role and the framework itself. With that in mind, expectations, or thresholds for “meets expectations” by role level are provided in the view. For this example:

• Entry-Level Data Scientist: 35–44%

• Mid-Level Data Scientist: 45–59%

• Senior Data Scientist: 60–74%

• Principal Data Scientist: 75–89%

In this case, the employee’s score of 53% places them squarely within the mid-level data scientist range, suggesting they are performing as expected for their role if they are in fact at the mid-level.

Moving down the dashboard, you’ll notice four core competencies that form the foundation of the evaluation:

1. Technical Skills and Analytical Knowledge

2. Business Acumen and Problem Solving

3. Communication and Influence

4. Innovation and Learning Agility

Each competency represents a broad area of capability. Within these competencies are specific skills, such as SQL Proficiency under the Technical Skills competency. This distinction is deliberate: skills are the building blocks of broader competencies, and understanding their relationship is key to interpreting the evaluation.

The dashboard further highlights that not all skills contribute equally to the overall score. Each skill is weighted according to its importance for the role. For example, Machine Learning and Predictive Modeling contributes 15% to the overall score, reflecting its critical role for data scientists.

This weighting ensures that evaluations focus on what truly matters, allowing for a nuanced and role-specific assessment.

To determine the score for each skill, the framework incorporates a detailed rubric accessible through the dashboard. The rubric provides specific performance criteria for each skill, making evaluations clear and consistent.

For example, the rubric for Machine Learning and Predictive Modeling includes the following performance levels (viewable within each dropdown on the dashboard by skill):

This structured approach ensures that scores reflect meaningful differences in performance, guiding both employees and managers toward actionable insights for growth.

The dashboard provides a transformative experience for employees, offering them a clear, actionable view of their performance rather than vague, generic feedback. At a glance, they can understand the following:

1. What they are being graded on, with a detailed breakdown of competencies and skills that eliminates ambiguity about expectations.

2. The relative importance of each skill, helping them prioritize areas that have the greatest impact on their overall score.

3. How they are performing in each area, with rubric scores that provide precise feedback on strengths and weaknesses.

4. Where they stand overall, with their total score contextualized against thresholds for their role level.

This detailed insight equips employees to take control of their development and advocate for promotions with confidence. For instance, in the current example, the employee is performing well within the mid-level range, scoring a total of 53%. However, the dashboard also highlights opportunities for growth. While they are already performing at a senior level in Technical Skills and Analytical Knowledge—scoring consistently above the 60% "meets expectations" threshold for a senior data scientist—they are underperforming in other areas such as Data Storytelling and Insight Communication and Innovation and Learning Agility, both of which are scored at 30%. These two skills, each contributing 12% to the overall score, represent the most significant opportunities for improvement. By focusing on these areas in the next evaluation cycle, the employee can make meaningful progress toward advancing to the senior level.

This approach isn’t just about highlighting gaps—it’s about providing a pathway to improvement. Organizations can amplify the effectiveness of these insights by offering targeted support, such as curated resources tailored to skill gaps. For example, the competency framework created with the help of ChatGPT4o includes an appendix of development resources. This appendix provides specific recommendations, direct links to learning materials, and even pricing information, ensuring employees have everything they need to take actionable steps toward growth. By providing such clear guidance, organizations not only empower their employees but also foster a culture of continuous learning and strategic alignment, driving both individual and organizational success.

Performance evaluations are most effective when supported by robust documentation that serves as a reference for employees and managers. This ensures that expectations are clear, criteria are consistently applied, and feedback is grounded in transparent guidelines. A sample of this type of documentation is available HERE, showcasing a structured approach to competency-based evaluations. To highlight repeatability, I followed the same process to make a guidebook for industrial/organizational psychologists HERE and software engineers HERE.

But how did this framework and its accompanying resources come to life? The data scientist role was selected as a notional example to demonstrate how competency frameworks can be constructed and operationalized. Generative AI, specifically ChatGPT4o, was instrumental in developing much of the content. The process involved using ChatGPT4o to iteratively refine competencies, skills, rubrics, and development pathways, creating a comprehensive and actionable framework.

The entire ChatGPT4o thread, accessible HERE, walks through every prompt used to produce the outputs necessary to build the dashboard and the guidebook. There is even a tangential discussion about the best key lime pie recipe to offset using "to" instead of "too". These touches of levity aside, the AI-assisted process highlights how technology can accelerate and enhance the development of detailed frameworks that align with real-world needs.

By combining thoughtful design with innovative tools, this example provides a blueprint for organizations to develop their own role-specific frameworks, empowering employees and ensuring evaluations are both meaningful and actionable.

Beyond the elements already discussed, several additional considerations and decisions shaped the design and development of these outputs. Each step of the process was guided by a commitment to practicality, transparency, and scalability, with ChatGPT4o serving as a valuable resource for accelerating and enhancing the work. Below is a comprehensive overview of the key aspects of the process, highlighting both strategic decisions and the technical execution behind them.

1. Prioritizing Scalability for Broader Application

One of the first considerations was ensuring that the competency framework and supporting tools could be adapted to other roles. While data scientists were the focal point, the goal was to create a replicable system that could be extended to other priority roles across the organization. ChatGPT4o was instrumental in:

• Drafting Modular Components: Competencies and skills were designed in a modular fashion, making it easy to swap or adjust them for different roles. For example, while SQL Proficiency is critical for data scientists, it could be replaced with role-specific skills like Java development or sales pipeline management for other roles.

• Testing Scalability: By prompting ChatGPT4o to generate draft competencies and skills for unrelated roles (e.g., software engineer, product manager), we validated the adaptability of the framework’s structure.

This scalability ensures the investment in developing the framework can deliver long-term value across various teams and functions.

2. Balancing Granularity with Usability

Creating detailed scoring rubrics required striking the right balance between specificity and usability. The aim was to provide enough granularity to capture meaningful differences in performance without overwhelming users or creating complexity that would hinder adoption.

• Iterative Drafting: ChatGPT4o was used to draft performance levels (e.g., novice to expert) for each skill, which were then iteratively refined to ensure clarity and alignment with real-world expectations.

• Clarity in Language: Each level was reviewed for precise, accessible language. For example, instead of saying “performs well under pressure,” criteria like “executes key deliverables within tight deadlines with minimal supervision” were used to provide actionable specificity.

• Avoiding Overlap: Particular care was taken to ensure that thresholds between levels (e.g., “meets expectations” vs. “exceeds expectations”) were distinct, avoiding ambiguity that could confuse employees or managers.

3. Designing for Equity and Consistency

One of the challenges in performance evaluation is mitigating bias and ensuring consistency across teams and managers. To address this, several measures were embedded into the process:

• Standardized Scoring: Every competency and skill was assigned explicit weights, ensuring that evaluations focused on the most critical aspects of performance.

• Clear Rubric Levels: Each skill’s rubric included detailed descriptions for each scoring tier (e.g., 10%, 30%, 50%, 70%, 90%, 100%), reducing the subjectivity of assessments.

• Cross-Functional Relevance: Competencies like Communication and Innovation were intentionally included to ensure the framework valued skills often overlooked in technical roles, promoting a more holistic view of performance.

These efforts were aimed at building trust in the framework’s fairness while ensuring evaluations were meaningful and actionable.

4. Emphasizing Development Pathways

Another priority was ensuring the framework supported not just evaluation but also growth. Employees need to understand not only where they stand but also how to progress. This focus on development was addressed through:

• Curated Resources: ChatGPT4o helped identify tailored learning opportunities for each skill, categorized by proficiency level (novice, intermediate, expert). This included free tutorials, industry-recognized certifications, and advanced workshops.

• Actionable Roadmaps: For each skill, employees are provided with specific recommendations for improvement. For example: For SQL Proficiency, resources ranged from free W3Schools tutorials for novices to advanced Udemy courses for experts. For Data Storytelling, suggestions included public speaking courses and interactive visualization workshops.

• Integrated Guidance: These resources were linked directly to the scoring rubrics, ensuring that feedback flowed naturally into development opportunities.

This focus shifted the narrative of performance evaluations from static assessments to dynamic tools for career growth.

5. Iterative Collaboration with ChatGPT4o

The role of ChatGPT4o in the process went beyond simple content generation. It served as a creative partner, facilitating rapid iteration and refinement. Key contributions included:

• Drafting and Refining Competencies: ChatGPT4o provided initial drafts for competencies and skills, which were then reviewed and adjusted for alignment with organizational goals.

• Generating Granular Rubrics: Performance levels were developed collaboratively, with ChatGPT4o suggesting nuanced criteria for each level, which were then validated for accuracy and applicability.

• Identifying Industry Trends: ChatGPT4o highlighted emerging skills and methodologies in data science, ensuring the framework remained forward-looking and relevant.

The AI-assisted process saved significant time while maintaining a high level of detail and precision.

• At-a-Glance Insights: The visual layout emphasized clarity, ensuring users could quickly identify key strengths and areas for improvement.

• Interactive Filtering: Filters allowed managers and employees to tailor views to specific competencies, skills, or thresholds, making the data relevant to their unique needs.

• Progress Tracking: The dashboard enabled real-time tracking of performance improvements, motivating employees to engage with their development plans.

These visual tools transformed the framework from a static document into a dynamic, user-friendly resource.

• User Testing: The framework was tested with hypothetical employee profiles to ensure the scoring rubrics and thresholds worked as intended.

• Manager Enablement: Sample scoring templates and evaluation guides were created to help managers apply the framework consistently.

• Employee Accessibility: The inclusion of user-friendly resources and straightforward rubrics ensured that employees could easily understand and engage with the framework.

This focus on practicality aimed to maximize adoption and impact across teams.

• Regular Updates: Competencies and rubrics will be reviewed periodically to reflect evolving industry standards and organizational priorities.

• Scalable Methodology: The framework’s modular design makes it easy to replicate for other roles, ensuring its long-term relevance and utility.

This future-focused approach ensures the framework remains a valuable resource for years to come.

In conclusion, the creation of these outputs represents a thoughtful blend of innovation, collaboration, and strategic foresight. By combining the capabilities of generative AI with human oversight, the framework provides a clear pathway for evaluating and developing high-impact roles. While the example focused on data scientists, the principles and processes behind this work offer valuable insights for organizations looking to empower their teams and drive long-term success.

Let’s keep the conversation going—feel free to share your thoughts in the comments or message me directly!

Note that the works cited label for the banner image isn't functioning. That is as follows: Abstract Visualization of Data Science Competencies." Created using DALL-E, a generative AI-powered image creation tool.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

1. Introduction: Setting the Stage

Roy Amara’s Law posits that while the immediate effects of emerging technologies often receive considerable attention, their broader and more enduring implications may only become fully evident over time. This principle applies to the evolution of generative AI (GenAI) and augmentation tools. Initially regarded as incremental enhancements, these technologies increasingly influence how tasks are distributed between humans and machines, restructure professional roles, and guide strategic decisions at an organizational level.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

Data science offers a particularly clear view of these transformations. As a discipline that depends on computational tools, iterative workflows, and analytical rigor, data science is well-positioned to reflect the shifts prompted by GenAI. Instead of merely expediting existing processes, augmentation encourages a fundamental reconsideration of which tasks merit human involvement, how career trajectories should be defined, and how machine outputs can be effectively integrated into strategic initiatives.

A scenario-based analysis, created using augmented content generation, offers a practical illustration. In other words, I used ChatGPT to generate the data, much like how others might utilize similar tools to enhance their work, as outlined in this article. In this scenario, a data science team in a mid-sized technology organization of 10K employees is examined from 2024 to 2035 under conditions of stable workloads and unchanged operational baselines. Within this controlled setting, several patterns emerge:

• Early-Career Roles: By 2035, routine, repetitive tasks increasingly shift to AI-driven augmentation, enabling human effort to concentrate on validation, refinement, and judgment rather than low-value activities.

• Mid- and Senior-Level Roles: By 2028, these roles consolidate into a single “practitioner” band, which will be referred to as “Levolution” (Level-Evolution), reflecting reduced reliance on hierarchical complexity as a growing array of tasks is absorbed by augmentation.

• Expert Roles: Principal-level professionals retain distinct responsibilities, focusing on strategic oversight, quality assurance, and ensuring that human insight complements automated analyses.

Under these conditions, cost avoidance approaches approximately $2.2 million annually by 2035, representing about 64% of a maintained $3.5 million budget. This outcome demonstrates how systematic augmentation can realign financial structures, professional roles, and the overall configuration of work over time.

Figure: 2024 Baseline Task and Compensation Table (The table below is an example of the data used in this analysis. It was generated with augmentation support and presents the initial distribution of tasks and compensation among Junior, Mid-Level, Senior, and Principal Data Scientists in 2024. It details annual task frequencies, hours per task, compensation rates, and the initial fraction of work partially supported by GenAI. Examining this pre-augmentation baseline provides a foundation for understanding how increasing adoption of GenAI technologies ultimately reshapes cost structures, role definitions, and strategic considerations. The data was produced every year up to 2035. You can access the data via the dashboards posted to Tableau Public HERE.)

2. Augmentation Trends Across All Levels

The progression of augmentation technologies from 2024 to 2035 reveals a consistent increase in the ratio of tasks completed with GenAI across all data science roles. Although each level adopts augmentation at a different pace and proportion, a clear pattern emerges: repetitive, mechanical tasks are absorbed by AI-driven systems, allowing human contributions to shift toward higher-value activities.

By 2035, the reliance on augmented workflows and tools spans all professional tiers:

• Early-Career Roles: Approximately 75% of tasks become augmented, reflecting a substantial transfer of repetitive workloads—such as data cleaning, feature engineering, and missing value imputation—into AI-driven processes.

• Practitioners (Consolidated Mid- and Senior-Level Roles): About 60% of tasks are augmented following the 2028 “Levolution.” This consolidation reduces hierarchical complexity and positions practitioners to focus on advanced modeling, performance benchmarking, and strategic problem-solving.

• Expert Roles: Approximately 35% of tasks rely on augmentation, indicating that while some routine aspects are automated, these senior-level professionals concentrate on nuanced decision-making, long-term planning, and innovation, as well as ensuring that AI-generated insights align with organizational objectives.

This graduated adoption of augmentation highlights the importance of integrating AI capabilities throughout the entire career progression. Professionals entering the field encounter GenAI from the outset, honing skills that involve interpreting, validating, and refining machine outputs. Practitioners benefit from augmented workflows by dedicating more effort to complex analytical tasks and collaborative initiatives. Experts remain critical for oversight, guiding strategic direction, and addressing the inherent limitations and potential errors of AI-driven systems.

Figure: Augmentation Ratios Over Time (The accompanying figure illustrates the evolving ratios of task augmentation for each role category from 2024 through 2035. The trend lines demonstrate the increasing portion of tasks completed via GenAI for Early-Career, Practitioner, and Expert levels. Notably, the figure identifies 2028 as the year when Mid-Level and Senior roles merge into the Practitioner band, reflecting structural changes linked to increasing augmentation. Captions and annotations clarify the nature of these augmented tasks and the significance of their growth rates, enabling a clearer understanding of how AI reshapes the division of labor over time.)

Beyond efficiency gains, this trend raises critical questions regarding workforce readiness and the adequacy of current educational and professional development frameworks. As augmentation becomes more pervasive, organizations and institutions may need to reevaluate training methodologies, performance metrics, and the overall alignment of human skills with AI-driven capabilities.

3. Understanding Augmentation by Role

The impact of GenAI on data science roles does not manifest uniformly. Instead, specific responsibilities and career stages experience augmentation in distinct ways. Analyzing these patterns at the role level—Early-Career, Practitioner (consolidated Mid- and Senior-level), and Expert—offers insights into how different categories of tasks evolve, and how professionals at each level adapt their competencies and contributions.

Early-Career Roles: By 2035, early-career professionals rely on GenAI for approximately 75% of their tasks. These roles traditionally encompass repetitive, rule-based activities such as:

• Feature Engineering, Data Cleaning, and Missing Value Imputation: Tasks once handled manually shift primarily to augmented systems, reducing the time required and allowing earlier-career employees to focus on validating AI outputs and ensuring data integrity.

• Cost Implications: Automating these routine tasks results in significant cost avoidance. In the illustrative scenario, early-career roles contribute approximately $915K in annual savings by 2035, representing about 41% of total cost avoidance across the team.

Practitioner Roles (Post-2028 Consolidation): Following the “Levolution” in 2028, Mid-Level and Senior-level positions merge into a unified Practitioner band. By 2035, approximately 60% of their tasks are augmented:

• Modeling, Validation, and Performance Benchmarking: Although still partially reliant on human expertise, many foundational activities become increasingly AI-driven. Practitioners direct more attention to strategic decision-making, advanced problem-solving, and interpreting complex results rather than performing routine analytical tasks.

• Financial Outcomes: This shift in role definition and task composition accounts for roughly $1.3 million in annual cost avoidance, about 57% of the total savings across the team. The data suggests that reducing hierarchical layers and embracing AI-supported workflows can enable more strategic resource allocation.

Expert Roles: Expert-level professionals, such as Principal Data Scientists, rely least on augmentation, at around 35% of tasks by 2035:

• Strategic Oversight and Quality Assurance: Experts focus on activities that demand human judgment and creativity, including ensuring alignment between AI outputs and organizational goals, mitigating potential errors or “hallucinations” in machine-generated insights, and guiding less experienced team members in effectively leveraging augmentation tools.

• Modest Direct Savings, High Strategic Value: Although Experts contribute about $55K in annual cost avoidance (approximately 2% of the total), their role remains critical. By validating and refining augmented workflows, they preserve quality standards, reduce risk, and maintain strategic coherence.

This differentiated pattern of augmentation underscores the necessity of role-specific strategies in workforce planning, training, and professional development. It suggests that while early-career staff must rapidly adapt to an environment where AI manages routine activities, practitioners and experts must refine their strategic, analytical, and oversight capabilities to excel in an augmented ecosystem.

4. The 2028 Turning Point: “Levolution”

The year 2028 represents a pivotal inflection point in the examined scenario, marking a fundamental reconfiguration of the data science workforce structure. Before this date, four distinct roles—Junior, Mid-Level, Senior, and Principal Data Scientist—characterize the team’s hierarchy. As GenAI augmentation intensifies, the need for multiple layers diminishes, prompting the consolidation of mid-level and senior positions into a single “Practitioner” category.

Several factors drive this “Levolution” (Level-Evolution):

• Reduction in Hierarchical Complexity: Augmentation technologies automate substantial portions of routine analytical and preparatory tasks. The overlapping responsibilities of Mid- and Senior-level roles become increasingly redundant, enabling a more streamlined configuration.

• Broader Scope of Practitioner Responsibilities: The consolidated Practitioner role covers activities once divided between Mid-Level and Senior positions, encompassing both technical duties (model validation, performance benchmarking) and more strategic engagements. With core repetitive tasks delegated to AI, practitioners allocate time and expertise toward higher-value problem-solving and collaboration.

• Financial and Operational Efficiency: By 2035, the Practitioner band’s reliance on AI-driven augmentation contributes to significant cost avoidance, totaling approximately $1.3 million annually (57% of overall savings). Consolidation also reduces headcount requirements, aligns compensation structures more closely with value-added responsibilities, and supports agile responses to evolving organizational priorities.

• Expert Oversight and Direction: Although Expert-level roles maintain distinct responsibilities, Levolution enhances their capacity to provide strategic guidance, quality assurance, and mentorship. With a single Practitioner tier absorbing once-divided tasks, Experts can focus on long-term planning, innovation, and ensuring that AI outputs align with organizational objectives.

Figure: Sankey Diagram of Role Consolidation (2024–2035) (The accompanying Sankey diagram visualizes the flow of tasks from the original four-tier structure into the three-level configuration realized post-2028. Each ribbon represents categories of work as they transition from Junior, Mid-Level, Senior, and Principal roles toward the Early-Career, Practitioner, and Expert bands. This visual demonstration clarifies the evolution of workforce composition and underscores the impact of GenAI on role distribution.)

This turning point in 2028 exemplifies the broader influence of augmentation technologies on organizational hierarchies and career pathways. By reducing complexity and enabling greater strategic specialization, Levolution sets the stage for improved adaptability, cost management, and alignment of human expertise with evolving technological capabilities.

5. Cost Avoidance and Financial Impact

A primary advantage realized through the broad integration of GenAI augmentation is the significant reduction in operational costs. Within the examined scenario—one characterized by stable conditions and no external growth factors—deploying augmentation technologies leads to considerable annual savings by strategically reallocating tasks and responsibilities.

Key findings include:

• Aggregate Savings: By 2035, the data science team will achieve approximately $2.2 million in annual cost avoidance, corresponding to roughly 64% of a maintained $3.5 million budget. This outcome confirms that the strategic adoption of augmentation tools can substantially lower operating costs without requiring increases in workload volume or adjustments for inflation.

• Early-Career Roles: About $915K in annual cost avoidance (41% of the total) stems from automating repetitive tasks such as data cleaning, feature engineering, and missing value imputation. By transferring these activities to AI systems, early-career professionals can focus on higher-order responsibilities like validating machine-generated outputs and refining analysis quality.

• Practitioners (Post-Consolidation): Approximately $1.3 million in annual cost savings (57% of the total) emerges from consolidating Mid-Level and Senior roles into a single Practitioner tier. With augmented workflows handling the bulk of modeling, validation, and benchmarking tasks, practitioners direct their efforts toward advanced problem-solving, complex decision-making, and strategic performance optimization.

• Expert Roles: Around $55K in annual cost avoidance (2% of the total) is attributed to Expert-level professionals. Although this figure is relatively small, Experts contribute immense strategic value by ensuring the alignment of AI-driven processes with organizational objectives, detecting and mitigating potential errors or “hallucinations” in AI outputs, and providing guidance to other tiers that increasingly rely on augmented methods.

Figure: Waterfall Diagram of Cost Savings by Role and Task (The accompanying waterfall diagram visually decomposes the total cost avoidance into its constituent sources. Each bar segment represents a category of tasks initially performed by humans in 2024 and progressively automated by 2035. By examining how each role’s responsibilities evolve, the figure offers a clear, data-driven narrative of how augmentation generates sustained financial benefits. Note that saving rates (e.g., 35% for Experts) are specific to the level, not the aggregate.)

Collectively, these savings highlight the complexity and nuance of implementing augmentation strategies. While early-career and practitioner roles deliver the most immediate financial returns through the automation of routine tasks and hierarchical consolidation, the Expert tier ensures that cost avoidance does not compromise analytical rigor, strategic insight, or long-term organizational resilience.

6. Conclusions and Future Considerations

The introduction of GenAI and augmentation technologies into the data science workforce offers profound insights that extend beyond immediate efficiency gains. Over time, these tools reconfigure not only the division of labor but also the strategic focus, skill sets, and hierarchical structures within organizations. While the scenario under examination—featuring stable operational conditions—represents a controlled environment, its findings suggest broader implications relevant to various technology-intensive domains.

Several key themes emerge:

• Substantial Operational and Financial Benefits: By 2035, cost avoidance will reach approximately $2.2 million annually, representing about 64% of a maintained $3.5 million budget. This substantial reduction in expenses, achieved by redirecting repetitive tasks to AI-driven workflows, illustrates how augmentation can deliver long-term resource efficiencies without requiring increased workloads or inflationary adjustments.

• Redefined Roles and Hierarchies: The consolidation of Mid-Level and Senior positions into a single Practitioner tier—referred to as “Levolution”—highlights how augmentation reshapes role structures and reporting lines. This realignment enables professionals at all levels to dedicate more time to strategic tasks, complex decision-making, and the interpretation of AI outputs, while Expert roles focus on quality assurance, oversight, and alignment with organizational goals.

• Shifts in Skill Requirements and Training Needs: As repetitive activities become increasingly automated, the ability to interpret, refine, and integrate AI-generated insights into strategic initiatives grows in importance. Early-career professionals must swiftly adapt to augmented environments, while practitioners and experts develop more advanced analytical, creative, and oversight capabilities. Educational and professional development programs may need to emphasize AI literacy and advanced analytical reasoning to ensure workforce readiness.

• Cross-Disciplinary Implications: Although data science provides a clear test case, similar patterns of augmentation may emerge in other fields that rely on structured, data-intensive tasks. Conversely, sectors demanding high levels of emotional intelligence, manual skill, or nuanced interaction may experience a more gradual or limited adoption of AI-driven augmentation. Understanding these distinctions can guide the development of tailored strategies that account for industry-specific dynamics and constraints.

• Strategic Imperatives and Risk Management: The complexity and power of AI-driven systems underscore the importance of considering augmentation as a strategic priority rather than a mere operational improvement. Policymakers, educators, and organizational leaders may need to establish guidelines, standards, and best practices that ensure ethical use, maintain data integrity, and mitigate risks such as algorithmic errors or over-reliance on automation.

The findings suggest that organizations prepared to invest in workforce readiness, recalibrate their role definitions, and embrace augmentation technologies responsibly stand to gain significant advantages in efficiency, adaptability, and innovative capacity. As GenAI continues to evolve, it will likely shape the future of work in ways not yet fully understood, demanding ongoing vigilance, strategic foresight, and a willingness to integrate human expertise with increasingly capable machine-based partners.

If this article sparked new ideas or resonated with your perspective, feel free to share your thoughts in the comments. Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable—like, comment, and let’s keep these ideas flowing!

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

In today's dynamic business landscape, workforce planning stands at the forefront of strategic decision-making. As companies navigate through ever-evolving market conditions, the traditional approach to workforce planning may have some organizations looking for a revised approach. Enter zero-based workforce planning—a methodology that can shape how businesses align their human capital with organizational goals and enable a more cost-effective solution that gets closer to having the right people at the right time.

This article will progress through the steps to create a dashboard that utilizes outputs from ChatGPT for demand and aligns them with the supply of FTEs, facilitating the understanding of capability gaps. This dashboard can be accessed and downloaded free from Tableau Public here (desktop recommended):

https://public.tableau.com/views/BottomsUp5YearWorkforcePlan/WorkforceDataDetailbyProject?:language=en-US&publish=yes&:display_count=n&:origin=viz_share_link

As a preview, the final output with gaps between supply and demand by role and experience level is shown below. The idea here is that it is an at-a-glance dashboard providing the ability for a user to quickly see every role (across) and level (down) where there are too many FTEs (blue) and vice versa (orange).

Zero-Based Workforce Planning

Zero-based workforce planning isn't just a buzzword; it's a strategic imperative. Unlike conventional methods that rely on historical data and incremental adjustments, zero-based planning starts from scratch, challenging preconceived notions about workforce needs. This approach demands a comprehensive understanding of every organizational role, function, and activity, fostering a leaner and more agile workforce structure.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

Leveraging ChatGPT for Strategic Insights

In the pursuit of redefining workforce strategies, innovative tools such as ChatGPT are invaluable. Leveraging its capabilities, ChatGPT can assist in crafting meticulous templates delineating roles, experience levels, and corresponding compensation required for diverse business activities. Organizations can gain a clearer understanding of their workforce landscape by tapping into its analytical prowess.

As part of a notional sample of a workforce plan with demand (what mix and number of colleagues are needed over time to execute the work forecasts) and supply (how many colleagues are currently staffed and projected into the future), I prompted ChatGPT for sample data related to demand. For this, I asked for software development projects in three different sizes. The full thread, from inception to generated tabular data, can be viewed at this link:

https://chat.openai.com/share/d7203bad-ad4c-424e-a296-3e0c0cb5f766

The thread from ChatGPT is on the left in the image below. The data provided by ChatGPT with needs for each project type are loaded into Tableau Public as shown at the right of the view. The idea is that an organization would have a repository of at least 20-30 activity types like these three examples. There are restrictions on how tools like ChatGPT can, or cannot be, used in the workplace, and the composition of the work by activity will potentially need to be created without ChatGPT in their entirety. Confidential information should be omitted from the public domain or a news article may be written about it, like this one:

https://aibusiness.com/nlp/samsung-to-fire-employees-caught-using-chatgpt

I mention refining the ChatGPT outputs, which should be considered mandatory and part of the process. The necessity for this can be found in OpenAI's testing of the accuracy, or inaccuracy of the outputs from GPT-4 and prior iterations. Their recently published visual of output accuracy is shown below and can be accessed here:

https://openai.com/research/gpt-4

The free version of ChatGPT (which I'm using) uses GPT-3.5 and hovers around an accuracy of 60% (dark blue bar). The remainder of these outputs may have some error as OpenAI defines the balance of outputs as not matching answers "judged to be in agreement with human ideal responses". For a future trivia night question: a term used for these inaccuracies is "hallucinations".

Demand: Collaborative Forecasting of Activities

The beauty of zero-based workforce planning lies in its collaborative nature. ChatGPT facilitates an initial bank of activities, which initiates downstream user interaction, receiving feedback on the frequency and magnitude of various business activities. This collaborative exchange fuels the process, enabling data-driven resource allocation and workforce distribution decisions.

The tab in the Tableau Public dashboard with the user-provided projections for each activity type is below. Users will enter the iterations of each activity type projected out to five years. Note that the upper left provides an aggregate of FTEs over time, followed by double-click views by experience level and role. The lower left follows this same path but for compensation. The right side provides this data in a tabular format.

Supply: Current and Forecasted Future Workforce

Supply in the dashboard is an internal-only data set and would be provided by role and experience level. For simplicity, there are eight roles with three levels (early/mid/late) and only two inputs for each: current number of FTEs and annual growth rate for years 2 through 5. The line graph provides a visual for what is entered in the parameters above it.

Visualizing Insights with Tableau

Turning raw data into actionable insights is where visualization plays a pivotal role. Integration with Tableau enables the transformation of workforce forecasts into visually compelling dashboards. These dynamic visuals provide a holistic view of workforce dynamics, aiding stakeholders in making informed decisions.

To this point, we have gathered sufficient information on both supply and demand. Referring back to the desire to have the right people at the right time, the goal is to see a perfect alignment of supply and demand. The visual below plots the demand (blue) and the supply (orange) on the top and they seem to be nearly perfectly aligned. The gap between the two is provided in FTEs on the bottom. The worst delta is in year two with 8.9 FTEs off of a base of around 250 FTEs. This essentially represents an "error" on the right people right time of less than 4%. That sounds good. What we will find though is that it gets quite a bit more ugly as we peel away a layer of the onion.

The dashboard allows two layers of "cuts" in this view for job family and experience level. They can be applied in isolation or in either order. A sample of this data dive is shown below.

The prior view becomes rather difficult to digest as cuts are applied and scrolling becomes excessive. A superior view would allow an at-a-glance view and allow one to see the big picture within seconds. This brings us back to the first image in this article. This view allows the viewer to digest quickly where excess colleagues exist (e.g., early career software engineers) and where shortages exist (e.g., mid/late career software engineers, early career UX/UI designers, and early career QA testers).

This example provided for only three theoretical activity types with eight roles. In reality, an organization will have quite a few more of each. Filters will be necessary for dashboards like this to allow for tailored views that are relevant and accessible (need-to-know permissions) for each user.

Conclusion: Empowering Decision-Making

Interpreting the data generated through zero-based workforce planning is the crux of its effectiveness. These insights are more than numbers; they're strategic intelligence guiding resource allocation, talent acquisition, and skill development. Understanding the nuances of workforce trends equips businesses to adapt swiftly to market shifts.

In conclusion, zero-based workforce planning, powered by cutting-edge AI tools like ChatGPT and visualized through platforms such as Tableau, heralds a new era in strategic workforce management. Embracing this methodology empowers businesses not just to react but proactively shape their workforce, ensuring alignment with organizational objectives. It's not merely about planning for the present but strategically envisioning and preparing for the future—a future where agility and adaptability are paramount for success.

I invite you again to please download and iterate on the dashboard that I have posted on Tableau Public. It is free! Thank you for sharing some of your busy schedules with me and again, that link for the dashboard is:

https://public.tableau.com/app/profile/scottreida/viz/BottomsUp5YearWorkforcePlan/WorkforceDataDetailbyProject?publish=yes

Best wishes, Scott

Note: The caption to give credit for the image at the top of this article was not working properly. It was sourced from DALL-E and I generated it with the following prompt: "picasso style painting with many people with an impressionist style blended in with lots of color" and can be viewed at the source here:

https://labs.openai.com/s/p4KS3POdNyCT2z9slvdp1vLD

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

This article will explain how ChatGPT4 can be queried to develop a draft of what is needed for an organization's priority roles in the present and the future. Note that I said draft and the output of ChatGPT should not be considered final. The logical next step is to augment the result with other sources, such as Gartner Hype Cycles and Priority Matrices, followed by iterations with internal subject matter experts. I will touch on this as well in the latter half of this article.

My objective was to get the following deliverables for the role of data scientist: 1/ A table showing a list of competencies described by their timeline for adoption and their impact on an organization and 2/ A dashboard showing every one of those competencies in (1) with a clear sight of how invested in human capital a company would need to be for each over ten years.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

I generated the content discussed using ChatGPT4 and the back-and-forth can be accessed HERE. The table (deliverable 1) is a direct output from ChatGPT4 and the dashboard (deliverable 2) was produced using the ChatGPT4 output in Tableau and is viewable/downloadable on Tableau Public HERE.

1/ A table showing a list of competencies described by their timeline for adoption and their impact on an organization

Note that the timeline (columns) is determined based on the period until at least 25% of the data scientist talent pool will be capable in the given competency area. I have provided additional details, including definitions for the three levels of impact below, which were also generated in the ChatGPT output.

2/ A dashboard showing every one of those competencies in (1) with a clear sight of how invested in human capital a company would need to be for each over ten years.

This is a lot to digest. With that in mind, the following image and explanations will hopefully help. Note that these FTE projections for the bins are based on a prompt to ChatGPT4 to provide estimates for a theoretical technology firm of 10K employees in a growth phase as an organization.

Note again that the dashboard is accessible HERE. An explanation of each section is as follows:

1/ FTE need is all competency areas over time. Each trend is left to right for time and in terms of a bin for the number of FTEs needed in that competency area each year. These bins are: 1 (0-10 FTEs), 2 (11-25 FTEs), 3 (26-50 FTEs), 4 (51-100 FTEs), and 5 (>100 FTEs). The intent is not necessarily to be able to digest all at once in this view (as too much overlap is present to differentiate them). Also, note that these FTE figures by competency should not be considered mutually exclusive and these can overlap (e.g., one FTE can be hired to touch on more than one competency at once)

2/ FTE need in detail presents a crosstab view of what is provided above in FTE need. Note that the color is deeper blue as more FTEs are required (note that all of these are trending up over time)

3/ Filters allow you to cut by the level of impact (high, moderate, or low), competency area, or current state (i.e., is the competency currently considered foundational for those in the talent pool, something that differentiates the best, or emerging that is rarely seen even by differentiators)

4/ Definitions provide an explanation for each per ChatGPT4 for each competency. Note that highlighting is active across all visuals and when applied, these definitions will present the starting/ending bins for the competency highlighted (e.g., 1 on the top and a 4 on the bottom indicates that the FTE needs moves from 1 (0-10 FTEs in year 1 to 4 (50-100 FTEs) by year 10).

An example filtered to Federated Learning is shown in Figure 5. The data shows a rapid increase in need from 0-10 FTEs in 2024 up to 25-50 FTEs in just one year. A view like this is available for every competency area and can later be used to start conversations such as: 1/ Do we upskill current employees, 2/ Do we focus on these in talent acquisition, 3/ Do we seek acquisitions of companies strong in these competencies, or 4/ Other?

I mentioned previously that this should not be considered final. My article on zero-based workforce planning expressed the need to refine ChatGPT outputs and I will restate that here as well. The necessity for this can be found in OpenAI's testing of the accuracy, or inaccuracy of the outputs from GPT-4 (utilized by ChatGPT4) and prior iterations. Their recently published visual of output accuracy is shown below and can be accessed HERE.

The free version of ChatGPT (which you may likely use) uses GPT-3.5 and hovers around an accuracy of 60% (dark blue bar). I am using the paid version of ChatGPT (uses GPT-4) as represented by the green bar. But as you can see, this still is limited to around an accuracy of 80%. The remainder of these outputs may have some error as OpenAI defines the balance of outputs as not matching answers "judged to be in agreement with human ideal responses". For a future trivia night question: a term used for these inaccuracies is "hallucinations".

I have seen three primary areas in this study with a need to address: 1/ Competencies are not exhaustive, 2/ The scaling of each competency over time is often suspect, and 3/ The full list of competencies gathered will not all be relevant to your organization. We will now review each of these three areas.

1/ Competencies are not exhaustive

I mentioned Gartner earlier and this is where they come into play. An example of a Priority Matrix is shown below. Note that it is in a similar format to that created using ChatGPT4. There will be some additional competencies to gain from their resources. Just so you know, you will need to search their content to get adjacent areas of scope. In other words, if you are searching for data science, gather content (Priority Matrices and Hype Cycles) for related areas. For instance, if data science is desired, it is best to query and gather content for data science, data engineering, machine learning, and artificial intelligence.

Figure 6:

https://twitter.com/Gartner_inc/status/1721887261124677743

Hype Cycles are typically the focus (from what I can tell at least) of Gartner's articles. The Priority Matrices, like the Gen AI one above, are more of a supplement to them. However, for this exercise, I believe that the Priority Matrices are superior, as they express the expected impact. They serve a different purpose as Hype Cycles are more focused on where each capability is along its lifetime and Priority Matrices lean to actual expectations (vs. hype).

2/ The scaling of each competency over time is often suspect

What do I mean by this? Well, I have seen outputs over time in many cases (especially from GPT-3.5) that are simply linear increments in growth each year. While ChatGPT may believe that the competency need will increase over time, simply stating a constant slope for the increase each year is overly simplistic in my opinion. An example of actual output from ChatGPT-3.5 of this subpar result is below in Figure 8.

While you take what you can on your initial efforts to gather data, you should be on the lookout for this to hopefully either: 1/ Consider purchasing ChatGPT4 for better results (I have found it to be superior over ChatGPT-3.5 for trend forecasts- at least they aren't all simply linear increases in slope as in Figure 8 or 2/ Reword the prompt to potentially get a better response (e.g., state the issue in the ChatGPT window and consider prompts of smaller outputs- I've found asking for less does at times result in fewer defects). While efforts may fail to improve results, all is not lost, as there is value in the insight of relative need for each (e.g., more are needed in cloud computing vs. data science for sustainability per Figure 8).

3/ The full list of competencies gathered will not all be relevant to your organization

Once all competency areas are gathered from sources we should consider this a starting point for discussions with subject matter experts in the organization. Note that this should not be the first they hear of the study- but this is when you start to put more time on calendars to review the findings. Several inputs and outputs should come out of these discussions.

For inputs, seek to add more competencies to the list. The research sources to this point likely fail to fully capture all of the foundational skills (capabilities you deem as table stakes for those in the role in the current state). Some of them over the next ten years will evolve (expand or contract)- or become obsolete (and knowledge of this is valuable to understand). Subject matter experts should be able to mitigate this gap. Once all sources (ChatGPT, Gartner, SMEs, etc.) are captured, lay out a complete list of everything in a spreadsheet like that in Figure 9.

At this point, you should work with the subject matter experts in the organization to label each row as: 1/ green (it is relevant to the organization), 2/ yellow (they aren't sure what the item is and more research/understanding is necessary before a decision), or 3/ red (it is not relevant to the organization). For the green items, validate/iterate the impact (referenced as benefit in Figure 9) and the FTEs (by way of bin requirement described previously). Do the same for yellow items deemed later to be in the green category.

Once complete, everything is in place to realize the objective, which was to get the following deliverables for the role of data scientist: 1/ A table showing a list of competencies described by their timeline for adoption and their impact on an organization and 2/ A dashboard showing every one of those competencies in (1) with a clear sight of how invested in human capital a company would need to be for each over ten years. Simply update Figures 1 and 3 with the surviving rows of data (competency areas) from Figure 9 and you're there.

THANK YOU for reading and please ping me with any questions and/or comments.

Best wishes, Scott

***One more thing***

Are you wondering what types of projects all of these FTEs would be undertaking? ChatGPT4 can even provide you with some ideas in that department as well. The remainder of this article is a copy/paste directly from the chat window where I asked for precisely that. From this point, you have the ability in the tool to dive into each project to gather details such as what other roles would support the project, FTEs for those roles, time to completion, etc. My prompt for this information is at the same link as previously noted.

Side note: If you visited the ChatGPT link that was provided over the course of this article you likely noticed that the majority of the narrative centered around the development of the data scientist role from early (0-3 years of experience) to mid (4-10 years of experience) to expert (more than 10 years of experience). I intend to expand on this in a future article (and utilize the same data already there). Hopefully, that is also interesting to see in terms of what is possible, as well as how the prompts needed to be tailored to get the results desired.

ChatGPT

Certainly! For high-impact competency areas expected to be adopted within less than 3 years, here are sample projects that a growth-phase technology firm with 10,000 employees might consider to leverage these competencies effectively.

Python Programming

1. Automated Data Processing System: Develop a system that automates the ingestion, cleaning, and initial analysis of large data sets using Python's extensive library ecosystem, enhancing data readiness for analytics.

2. Internal Tool Development: Create custom tools for various internal teams (e.g., HR, Finance, Operations) to automate routine tasks, such as report generation, data entry, and workflow management.

3. Machine Learning Model Deployment: Implement a series of machine learning models for predictive analytics, customer segmentation, or personalized recommendations, using Python’s machine learning libraries like TensorFlow or PyTorch.

R Programming

1. Statistical Analysis Platform: Build a comprehensive platform for performing complex statistical analyses, catering to data scientists and analysts for market trend analysis, A/B testing, and forecasting.

2. Custom Visualization Dashboard: Develop dashboards for real-time data monitoring, using R’s powerful ggplot2 and Shiny packages to visualize KPIs across business operations, finance, and customer analytics.

3. Risk Management Modeling: Create models to assess and manage risk, particularly in finance and insurance sectors, using R’s robust packages for statistical modeling.

Data Visualization

1. Interactive Customer Insights Dashboard: Develop an interactive dashboard that visualizes customer behavior, sales trends, and product performance, allowing for drill-down analysis and real-time decision-making.

2. Supply Chain Optimization Visuals: Create visualizations that map out the entire supply chain, identifying bottlenecks, predicting stockouts, and optimizing routes for efficiency improvements.

3. Market Analysis Tool: Implement a tool that aggregates market data from various sources and presents it through comprehensive visuals, aiding in competitive analysis and strategic planning.

SQL

1. Enterprise Data Warehouse Optimization: Enhance the company's data warehouse with advanced SQL queries for better data aggregation, summarization, and reporting to support decision-making processes.

2. Customer Data Platform Integration: Develop a project to integrate various customer data sources into a unified platform, using SQL for data extraction, transformation, and loading (ETL) processes.

3. Data Governance Framework: Establish a data governance framework with SQL-based auditing, monitoring, and compliance reporting features to ensure data quality and regulatory compliance.

Machine Learning

1. Customer Churn Prediction Model: Implement a machine learning model to predict customer churn based on historical data, helping the marketing team to develop retention strategies.

2. Fraud Detection System: Develop a system that uses machine learning to detect fraudulent transactions in real-time, enhancing security for financial operations.

3. Automated Support Ticket Classification: Create a machine learning model to automatically classify support tickets into categories, improving response times and operational efficiency.

Cloud Computing

1. Cloud Migration for Legacy Systems: Execute a project to migrate legacy systems and applications to the cloud, improving scalability, performance, and cost-efficiency.

2. Serverless Data Processing Pipelines: Implement serverless architectures for data processing pipelines that automatically scale with demand, optimizing costs and performance for big data analytics.

3. Disaster Recovery Plan: Develop a comprehensive disaster recovery plan utilizing cloud computing’s redundancy and data backup capabilities to ensure business continuity.

Deep Learning

1. Image Recognition for Quality Control: Deploy a deep learning-based image recognition system to automate the quality control process in manufacturing, identifying defects or anomalies in real-time.

2. Voice-Activated Customer Service: Implement a voice-activated system for customer service using deep learning models, enhancing customer interaction through natural language processing and speech recognition.

3. Predictive Maintenance System: Develop a predictive maintenance system for equipment and machinery using deep learning algorithms to analyze sensor data, predicting failures before they occur to minimize downtime.

Big Data Technologies

1. Real-Time Data Streaming for Market Insights: Utilize big data technologies to process and analyze real-time data streams from social media, news, and market feeds for actionable market insights and trends.

2. Logistics Optimization Platform: Create a platform that leverages big data analytics for logistics and supply chain optimization, using historical data and real-time inputs to improve routing, inventory management, and delivery schedules.

3. Customer Behavior Analysis System: Build a system to analyze large volumes of customer interaction data across multiple channels, employing big data technologies to uncover insights into customer preferences, behaviors, and trends.

Automated Machine Learning (AutoML)

1. Sales Forecasting Model: Implement an AutoML solution to develop and deploy sales forecasting models, enabling the sales team to predict future sales trends and make data-driven decisions.

2. Marketing Campaign Optimizer: Use AutoML to optimize marketing campaigns by automatically analyzing campaign performance data and suggesting adjustments for targeting, messaging, and budget allocation.

3. Risk Assessment Automation: Develop an AutoML-based system for automating risk assessments in financial services, enhancing accuracy and efficiency in credit scoring and loan approvals.

Edge Computing

1. Smart Retail Solutions: Deploy edge computing devices in retail stores for real-time inventory tracking, personalized customer notifications, and in-store analytics without the latency of cloud processing.

2. Industrial IoT for Manufacturing: Implement an Industrial Internet of Things (IIoT) solution using edge computing for real-time monitoring and control of manufacturing processes, improving efficiency and reducing waste.

3. Remote Monitoring in Healthcare: Develop a remote patient monitoring system utilizing edge computing for real-time data analysis, enabling quick responses to health alerts and improving patient outcomes.

Blockchain for Data Security

1. Secure Supply Chain Tracking: Implement a blockchain-based system to securely track the provenance and movement of products in the supply chain, enhancing transparency and reducing fraud.

2. Digital Identity Verification: Develop a blockchain framework for secure digital identity verification processes, improving privacy and security for online transactions and verifications.

3. Intellectual Property Protection: Use blockchain to create tamper-proof records of intellectual property (IP) creation and ownership, streamlining IP management and enforcement.

Quantum Computing (Emerging within 3 years but starting to have high-impact applications)

1. Cryptography Research Project: Explore quantum computing's potential to develop new cryptographic algorithms that are secure against quantum attacks, ensuring long-term data security.

2. Drug Discovery Acceleration: Leverage quantum computing to simulate molecular interactions at a quantum level, significantly accelerating the drug discovery process for new treatments.

3. Optimization Problems in Logistics: Investigate quantum computing's application in solving complex optimization problems in logistics and supply chain management, seeking to drastically reduce computation times and improve efficiency.

These projects exemplify how high-impact competency areas can be harnessed to drive innovation, efficiency, and competitiveness in a technology firm, aligning with strategic priorities and leveraging cutting-edge technologies to address business challenges.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Subscribe now

I. Introduction: The Imperative of Optimized Workforce Planning

In today’s ultra-competitive tech industry, finding the perfect balance between cost management and operational excellence is both an art and a science. Organizations are under increasing pressure to deliver value quickly, adopt evolving technologies, and respond to ever-shifting customer needs. Yet talent remains the single largest driver of development success—especially for product-centric teams where engineering, design, quality assurance, and operations must synchronize to meet deadlines and uphold quality. Every additional FTE hired is an investment in both payroll and benefits, not to mention the onboarding and management overhead that comes with growing headcount. Conversely, staffing too lean can create bottlenecks that delay critical milestones and risk losing market share to more agile competitors.

This article, developed in collaboration with ChatGPT o1, explores three distinct models of workforce planning: Zero-Based, Real World, and a Hybrid approach. Each model is examined through the lens of a notional product development team, covering roles such as Product Management, Engineering, UX/UI, QA, DevOps, Data, and Support. It’s important to note that I’ve explicitly excluded functions like marketing, sales, or finance from the analysis, as well as augmentation via GenAI and/or AgenticAI, to keep the focus on technical and product roles. By limiting the scope to these core development functions, we can more accurately assess the headcount fluctuations and associated costs of building and launching a product over a three-year timeline (12 quarters).

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

In the sections that follow, plots (with data and visuals available on Tableau Public HERE) will illustrate how headcount requirements change quarter by quarter for each of these three approaches. Specifically, we’ll see:

• Job Family by Level Trend by Quarter for each Scenario (Zero-Based, Real World, and Hybrid)

• Total FTE Requirements Trend by Quarter for Each Scenario (All Approached Combined)

• Compensation (with Cumulative) Trend by Quarter for Each Scenario (Combined)

These visual aids will help reveal how costs accumulate under different staffing philosophies, ranging from the lean but occasionally risky Zero-Based (deeper dive in my prior article HERE) approach, to the steady but costlier Real World approach, and finally the balanced Hybrid strategy that may better align with project needs, budget constraints, and the realities of hiring. The example data shows cumulative three-year costs of $50.9 million for Zero-Based, $58.4 million for Real World, and $52.9 million for the Hybrid model—figures that underscore just how vital it is to choose a staffing plan aligned with your organizational goals and risk tolerance. As a baseline for this example, picture a mid-sized technology firm creating a cloud-based collaboration platform over three years. The product integrates real-time data syncing, intuitive user interfaces, and secure file sharing. Development phases range from initial concept and prototyping to full-scale deployment and post-launch maintenance, requiring a cross-functional team of engineering, product management, QA, DevOps, and design roles at different intensities throughout the timeline.

Throughout this article, you’ll see how I’ve leveraged generative AI technology—specifically ChatGPT o1—to rapidly model different staffing outcomes and cost scenarios. While the models are hypothetical, they are grounded in realistic salary bands and quarterly ramps typically observed in mid-sized tech firms. I hope this provides a clear framework for decision-makers seeking to refine or reevaluate their own workforce strategies, whether that means trimming non-essential roles, bolstering vital functions, or shifting to a more fluid model that adapts quickly to changes in product scope.

Ultimately, I aim to demonstrate that an optimal staffing plan is more nuanced than simply slashing headcount or padding teams across the board. By weighing the operational continuity of Real World against the financial stringency of Zero-Based planning—and arriving at a Hybrid compromise—leaders can contain costs while also retaining critical expertise. In the next sections, I’ll dive into each approach individually, illustrate the peaks and valleys of resource needs, and show how total compensation accumulates over time.

II. The Zero-Based Approach: Efficiency in Theory

Zero-based workforce planning starts with a fundamental premise: assume zero headcount at the start of each quarter, then allocate only the roles and levels that are strictly necessary for that specific period. In practice, this can mean ramping up engineering, QA, and DevOps resources in the middle quarters—when product development is most intense—and scaling back or even zeroing out these teams in the early or final phases where the load is lighter. The logic is straightforward: pay for what you need, when you need it. By doing so, you minimize idle talent, reduce overhead costs, and keep a sharp focus on efficiency.

When visualizing this approach (the plot is provided at the end of the scenario summaries and includes all three), the graph typically shows sharp inclines in headcount for roles like Software Engineering and QA during the main build (often Quarters 5–8), reflecting the heavy coding, testing, and integration efforts. Outside of these main development windows, many job families retreat to minimal or even zero staff—particularly junior roles, which organizations often hire only when the workload justifies additional bandwidth. This on-demand style of staffing can yield a lower cumulative cost compared to other methods: in the three-year projection, the Zero-Based scenario totals about $50.9 million.

On paper, this figure is highly attractive for finance teams and C-level executives determined to stretch cost. However, it’s important to recognize that this level of optimization can come with significant challenges. First and foremost is hiring speed: if you don’t maintain a baseline team, you may need to recruit and onboard a large number of people in a short timespan. Any delay in finding the right talent—or getting them up to speed—risks pushing back your product’s development milestones. Additionally, knowledge transfer becomes more difficult if you frequently off-board staff; critical lessons learned in an earlier quarter might vanish when employees exit, forcing new hires to re-learn processes or repeat mistakes. This risk grows even more acute with specialized roles (e.g., Senior DevOps or Lead QA) where company-specific context is essential.

Despite these drawbacks, the Zero-Based approach remains popular as a thought experiment—or “theoretical ideal”—for those seeking to test how lean their organizations could become without operational chaos. It forces every department to justify headcount. It also provides a benchmark for comparing other approaches, such as the Real World or Hybrid scenarios I'll cover next. Think of Zero-Based planning like the strictest version of a diet: it’s lean, cost-controlled, and offers absolute clarity on what’s essential, but it can be hard to sustain—especially if you need consistent expertise or fast pivots.

In the upcoming sections, we’ll see how the Real World approach takes the opposite tactic—maintaining a more continuous set of roles—and then examine how a Hybrid model merges the best of both worlds. For now, it’s worth noting that while $50.9 million for a three-year project might seem like a dream come true (when compared to models of $58.4 million for Real World and $52.9 million for the Hybrid model) on a financial sheet, achieving that number in practice would require exceptional hiring agility, flawless organizational memory, and a strong willingness to ramp down staff whenever workload subsides. Few companies can orchestrate all these factors seamlessly, which is why we often see a gap between the Zero-Based concept and what’s ultimately feasible in reality.

III. The Real World Approach: Prioritizing Continuity

In contrast to the leanness of Zero-Based planning, the Real World Approach maintains a more continuous workforce. This model assumes that while certain phases of product development (e.g., discovery or post-launch stabilization) may not require heavy engineering efforts, completely ramping down staff can create more problems than it solves. By keeping a core of Product, Engineering, and QA resources consistently on board, organizations avoid the start-stop challenges of repeated hiring and offboarding. They also retain institutional knowledge—the nuanced understanding of decisions, trade-offs, and domain specifics acquired over time.

When visualized, the Real World approach typically shows smoother lines on the graph. For example, Software Engineers and QA Engineers remain on staff from the earliest quarters, working on prototypes, preliminary testing, or even cross-training and support tasks during quieter periods. By the time the product enters its heaviest development window, the team is already in place—reducing onboarding delays and preserving momentum. This consistency extends to roles like Senior DevOps or Lead Product Managers, who keep the technical and strategic fabric intact even as the product evolves.

The advantage of this stability is obvious: fewer surprises. Deadlines are less likely to slip because you’re not depending on last-minute hires, and the baseline team members are continuously building on past knowledge. They become deeply familiar with the product’s architecture, user persona needs, and organizational processes—making them more adept at tackling complex challenges that emerge throughout the lifecycle. You also minimize the “day one productivity gap” since fewer new hires need to be brought up to speed.

However, a major trade-off is cost. In the three-year modeling, this “Real World” method racks up a total of about $58.4 million, making it the highest-cost scenario among the three. That extra expense reflects the months or quarters where some staff may be underutilized but still on payroll. In financial terms, the Real World approach is easier on the operational side but heavier on the balance sheet—particularly if market conditions shift or product priorities change.

Another potential drawback is complacency: if your team is fully staffed year-round, you might lose the forcing function that a tighter budget can provide. Without the strict discipline of a Zero-Based or Hybrid approach, some roles can become less focused on pivotal tasks. Managers need to remain vigilant, ensuring that “cushion” time is spent on strategic initiatives like refactoring, cross-training, or future-proofing rather than idle or redundant work.

Despite these drawbacks, the Real World approach resonates strongly with many mid-sized companies. The risk of missing crucial deadlines or losing vital staff is often deemed more damaging than covering a few extra salaries. From a morale standpoint, employees also tend to feel more secure when they aren’t worried about being ramped down at the end of a phase. This sense of stability can foster better team cohesion and long-term commitment to the product vision.

In the next section, I’ll look at the Hybrid model, which attempts to blend the lean efficiencies of Zero-Based with the steadier resource allocation of the Real World approach. As you’ll see, the Hybrid strategy lands at about $52.9 million in total cost over three years—somewhere between the tight constraints of Zero-Based and the operational ease of Real World. This middle ground is where many organizations could find the sweet spot: controlling costs while still preserving the essential continuity required for a successful product launch.

IV. The Hybrid Model: Balancing Cost and Continuity

Somewhere between the strict efficiency of Zero-Based staffing and the continuity-driven Real World approach lies a Hybrid strategy. The Hybrid model recognizes that while it’s unwise to keep an army of idle staff on payroll, it’s equally risky to shed critical roles between phases and gamble on fast, efficient re-hiring. In other words, the Hybrid approach attempts to calibrate headcount needs from quarter to quarter—retaining a core layer of senior and mid-level experts throughout, while strategically adding or reducing associate/junior roles based on workload forecasts.

If you visualize this approach, you typically see modest peaks and dips in staffing numbers across Engineering, QA, and DevOps. For instance, a Senior or Lead Engineer might stay on from the earliest discovery sprints in Q1 through the final stabilization efforts in Q12, ensuring architectural consistency. Mid-level engineers could still ramp up significantly during the heaviest build periods (like Q5–Q8), but they wouldn’t be completely cut in earlier quarters. This avoids the steep hiring surges required by Zero-Based planning.

Financially, the Hybrid scenario in our three-year model comes in at about $52.9 million—a middle ground between the $50.9 million of Zero-Based and the $58.4 million of Real World. While it doesn’t match the razor-thin costs of Zero-Based, it preserves far more institutional knowledge and allows for an easier pivot if deadlines shift or new features need prioritization. Meanwhile, it’s not as expensive as the Real World approach, because the staffing isn’t as consistently high in the quieter phases of development.

By striking this balance, the Hybrid model provides flexibility without sacrificing core continuity. Senior-level staff remain present to guide architecture and process decisions, ensuring that momentum from one quarter carries smoothly into the next. Junior hires are brought on when needed—say, to implement bulk feature development or assist with large-scale testing—then either rolled off or reallocated when those tasks are complete. This can still pose operational challenges, like managing transitions for incoming and outgoing team members, but the impacts are less abrupt than in a purely Zero-Based scenario.

Another benefit is risk mitigation. If the product’s scope changes mid-project or an unexpected technical challenge arises, having a baseline of senior engineers and product experts on hand can be the difference between minor setbacks and major delays. In that sense, the Hybrid approach helps you hedge against uncertainty, ensuring the organization isn’t left scrambling to rehire or rebuild a team in the face of critical issues.

In summary, the Hybrid model offers a practical roadmap for cost containment without entirely relinquishing operational steadiness. It also resonates with the reality that many product strategies evolve; planning for an adaptable workforce can reduce friction when priorities change. In the next (and final) sections, we’ll look at comparative charts illustrating how all three approaches—Zero-Based, Real World, and Hybrid—stack up in terms of total FTE headcount and cumulative compensation over the twelve quarters. We’ll then delve into final recommendations on how to choose the best model for your organization’s risk tolerance, budget, and product goals.

V. Comparative Charts and Cumulative Costs

Having explored each approach, it’s helpful to zoom out and examine how all three models stack up when viewed side by side at the aggregate. While the job-family-specific graphs reveal the nuanced peaks and troughs for each role (e.g., Software Engineers vs. QA vs. DevOps), it’s equally important to see the big-picture data: total headcount over time and overall compensation as the months unfold. By visualizing these metrics in combined charts, decision-makers gain a more holistic sense of cost vs. continuity trade-offs.

This first combined chart shows the aggregate number of FTEs required each quarter. Notice how the Zero-Based line tends to exhibit steeper fluctuations, dropping to low levels in early or late quarters and spiking around the core development period. Meanwhile, the Real World line maintains a steadier high plateau, reflecting the philosophy of retaining staff to ensure continuity, even when workload temporarily dips. The Hybrid line sits between these extremes—suggesting a more measured approach that peaks when needed but still steps down modestly during less resource-intensive phases. This balanced pattern is often more reflective of real product roadmaps, where new features, testing cycles, and user feedback demands vary from quarter to quarter.

The second visualization focuses on compensation over the same three-year horizon (and the third visualization provides a running sum). In these charts, you’ll see how each model’s staffing decisions translate directly into dollars spent on payroll. The Zero-Based approach starts lower in earlier quarters but may experience sharp increases once heavy development kicks in; ultimately, it closes out at around $50.9 million across all twelve quarters. By contrast, the Real World scenario plots a consistently higher curve, leading to the largest overall outlay—approximately $58.4 million—by the end of the third year. Finally, the Hybrid curve generally remains above the Zero-Based line yet below the Real World line, culminating in a total of $52.9 million. This represents a controlled middle path, with some of the leanness of Zero-Based yet enough continuity to avoid the extreme ramp-ups and roll-offs.

From a risk management perspective, these combined charts underscore that cost is not the only factor at play. While the Zero-Based approach yields the lowest overall spending, it also poses the greatest risk of hiring lag or knowledge gaps. The Real World approach is the most robust but also the most expensive, reflecting the price tag of carrying staff steadily. Meanwhile, the Hybrid sits in between, attempting to balance continuity needs with budget discipline. Depending on your organization’s market conditions, cash flow, and urgency to reach the market, any one of these three curves could be preferable—or you might take a portion of one model and adapt it further.

Ultimately, these visuals help leaders answer a critical question: What do we get for each additional dollar spent on staff? In scenarios where timeline agility and knowledge retention are paramount, the Real World approach may justify its higher investment. Yet for organizations that can tolerate a bit of flux and manage onboarding swiftly, Zero-Based or Hybrid might conserve precious resources while still getting the job done.

VI. Conclusion & Recommendations

Each of the three models—Zero-Based, Real World, and Hybrid—offer distinct advantages and challenges in this example. The Zero-Based approach champions frugality and forces disciplined thinking but requires near-flawless hiring agility. By contrast, the Real World method provides stability and preserves institutional knowledge, albeit at a higher price point. Occupying the middle ground is the Hybrid model, which pragmatically balances cost controls with enough continuity to weather unexpected shifts in product scope or team dynamics.

While money is a critical factor, risk tolerance, corporate culture, and speed to market are equally important considerations when selecting or adapting a staffing model. A company under intense competitive pressure to ship a product first may prefer more continuous staffing—even at a premium—to reduce the odds of a crucial delay. Conversely, an organization in a more stable market might lean toward tighter budgetary control, relying on fast, targeted hiring if and when new features or pivots demand it.

1. Adapt, Don’t Adopt Blindly

None of the approaches presented here should be copied and pasted without adjustments. Consider the size of your existing team, regional labor market conditions, and organizational skill sets. If you already have a strong internal pool of talent familiar with your domain, a leaner Zero-Based or Hybrid model might pose less risk because you can reallocate experienced staff more flexibly. If you struggle to hire specialized roles or face a tight labor market, the Real World approach might be less risky in retaining those hard-to-replace skill sets.

2. Leverage Data & Tools

Scenario planning—like that performed here with ChatGPT o1—provides a fast, flexible way to test the impacts of various staffing decisions. Leaders can run “what-if” simulations on different assumptions about project scope, market shifts, or technology stacks. This data-driven insight helps align executives, product leaders, and finance teams around a shared perspective on how staffing impacts both timelines and the bottom line.

3. Maintain a Core Leadership Layer

Even in more aggressive approaches (whether Zero-Based or Hybrid), retaining a core group of senior specialists ensures continuity for architecture, product strategy, and knowledge transfer. This group can anchor the team through each phase of development while scaling up or down with junior or mid-level hires to meet shifting demands.

4. Budget for Upskilling & Internal Mobility

If you choose a leaner approach, consider incorporating training and development budgets that allow existing employees to learn new roles or fill gaps. Upskilling existing staff is often cheaper and faster than hiring from scratch—especially mid-project. Plus, investing in employee growth can boost morale and retention, mitigating turnover that might otherwise hamper timelines.

5. Plan for Post-Launch Phases

Product development doesn’t end the moment you release version 1.0. Whether you adopt Zero-Based, Real World, or Hybrid, ensure that support roles, maintenance engineering, and user feedback loops remain accounted for in your headcount plans. The last thing you want after a successful launch is to scramble for QA or DevOps resources when user bugs start rolling in.

In sum, workforce optimization is an ongoing exercise in trade-offs, forecasts, and organizational constraints. By analyzing and comparing models in a structured way—from the extremes of Zero-Based and Real World to a more nuanced Hybrid—companies can make informed decisions about where to allocate resources. Ultimately, the best approach is one that aligns with your product’s time-to-market goals, budget realities, and risk appetite. Whether you’re scaling up a new product team, right-sizing an established workforce, or transitioning to a new hiring philosophy, these insights and models offer a starting blueprint for more thoughtful, data-driven workforce planning.

If this article sparked new ideas or resonated with your perspective, feel free to share your thoughts in the comments. Let’s keep the conversation going and explore how we can collectively prepare for and shape the future of work. Your feedback and insights are invaluable—like, comment, and let’s keep these ideas flowing!

APPENDIX

While the main body of this article focused on the strategic implications, cost dynamics, and overarching trends of workforce augmentation across Zero-Based, Real World, and Hybrid scenarios, the finer details of job family mix and job family level mix tell an equally compelling story. These nuances reveal how different staffing strategies distribute roles across functional areas—like Engineering, QA, and Product Management—and how those distributions change when segmented by seniority levels.

I won’t delve too deeply into every variation or speculate on each micro-difference, but I believe these visualizations offer valuable context. Below, I’ve provided plots for each scenario, accompanied by brief narratives to highlight key observations and explain the most significant deltas in the job family mix view.

Each scenario tells its own story about how headcount is allocated across functions and levels, reflecting distinct strategic priorities and trade-offs. Whether it’s the leanness of the Zero-Based model, the continuity of the Real World approach, or the balanced adaptability of the Hybrid strategy, these distributions reveal the hidden gears turning behind each staffing philosophy.

Let’s briefly examine the patterns and trends that emerge in each scenario.

The detailed view can also be accessed on Tableau Public HERE.

The Job Family Mix visualization reveals how each workforce planning scenario—Zero-Based, Real World, and Hybrid—allocates headcount across key functional areas like Software Engineering, QA, Product Management, DevOps, Design, and Support over time. These distributions tell a story about when and where resources are prioritized, how teams scale throughout the project lifecycle, and the inherent trade-offs each approach carries.

Key Observations and Trade-offs

1. Software Engineering:

• In the early quarters, Real World starts with a relatively steady percentage of the organization dedicated to Software Engineering. This reflects an emphasis on baseline continuity and ongoing architecture work.

• Hybrid maintains a moderate presence, balancing early prototyping and foundational development with flexibility for future ramp-ups.

• Interestingly, Zero-Based starts lower but ramps up aggressively in the middle quarters (e.g., Q5–Q8), surpassing the other two models. This reflects a philosophy of resource intensification during critical build periods while keeping costs minimal in early and late phases.

• Trade-off: Zero-Based’s surge in engineering resources during peak quarters maximizes efficiency, but it comes with the risk of onboarding delays or knowledge gaps if hiring doesn’t align perfectly with project needs.

2. Product Management:

• In the early quarters, Zero-Based allocates a higher percentage of the workforce to Product Management—often close to a third of the organization. This reflects a focus on defining scope, setting strategic priorities, and aligning stakeholders before intensive development begins.

• As the project advances, Product Management’s share gradually normalizes across all scenarios, converging to similar levels in the middle and later quarters.

• Real World maintains steadier Product Management representation throughout, reflecting a belief in continuous oversight and strategic alignment.

• Trade-off: Zero-Based heavily frontloads Product Management effort, ensuring clear direction early but potentially risking gaps in sustained oversight if handoffs to execution teams aren’t seamless.

3. QA and Testing:

• Real World maintains consistent QA representation, prioritizing ongoing testing, stability, and bug management across all quarters.

• Zero-Based scales QA roles more dramatically, mirroring Software Engineering trends. Early quarters see minimal QA representation, but the percentage jumps significantly during core development phases, often surpassing both Real World and Hybrid scenarios.

• Hybrid balances these two extremes, maintaining moderate QA levels early while allowing for surges during heavy testing phases.

• Trade-off: Zero-Based optimizes QA for peak efficiency during critical build windows but risks missed early-phase validation opportunities without consistent coverage.

4. DevOps and Support:

• DevOps follows a pattern similar to Software Engineering, with Zero-Based showing dramatic peaks in the middle quarters as deployment and system scaling efforts intensify.

• Real World maintains steady DevOps staffing throughout, ensuring backend infrastructure remains stable at all times.

• Support roles remain relatively flat across all three scenarios, with only slight variations. This consistency reflects the ongoing need for maintenance and customer-facing support throughout the product lifecycle.

• Trade-off: Zero-Based emphasizes DevOps during high-intensity quarters, optimizing short-term efficiency but relying on flawless execution during ramp-ups to avoid infrastructure delays.

5. Design (UX/UI):

• Zero-Based allocates significant design resources early in the project timeline, reflecting a focus on early prototyping, interface design, and user experience mapping during the initial quarters.

• Design presence tapers off sharply in later quarters under the Zero-Based model, aligning with the logic that heavy design input is less critical during stabilization and post-launch phases.

• In contrast, Real World and Hybrid maintain more consistent Design staffing, allowing for iterative refinements and ongoing improvements even late into the project lifecycle.

• Trade-off: Zero-Based frontloads design investment for maximum early impact but risks reduced adaptability to late-stage user feedback without steady Design representation.

Strategic Trade-offs Between Scenarios:

Zero-Based:

• Prioritizes intense bursts of activity during critical development phases.

• Certain job families (e.g., Product Management, Design) are heavily frontloaded, while others (e.g., Engineering, QA, DevOps) peak in the middle quarters.

• While cost-efficient, this approach relies on precise timing, smooth onboarding, and minimal knowledge loss between staffing fluctuations.

Real World:

• Maintains steady-state staffing across all job families, ensuring operational continuity and knowledge retention.

• Provides stability across all functional areas but carries a higher cumulative cost due to sustained headcount, even during low-activity periods.

Hybrid:

• Balances the surges of Zero-Based with the continuity of Real World, offering adaptability without extreme volatility.

• Roles like Engineering and QA follow moderate peaks, while Product Management and Design roles are spread more evenly.

Summary Takeaway:

The Job Family Mix visualization highlights the contrasting philosophies behind each scenario:

• Zero-Based prioritizes focused bursts of specialized effort, especially in Engineering, QA, and DevOps, while heavily weighting Product Management early.

• Real World emphasizes consistency and stability, ensuring every functional area remains staffed at all times, reducing reliance on just-in-time hiring.

• Hybrid attempts to strike a balance, preserving a core layer of steady roles while scaling others strategically based on project needs.

These patterns suggest that the choice of staffing model hinges on an organization’s risk appetite, timeline urgency, and ability to flexibly scale teams during key project phases. Each scenario carries its own strengths and vulnerabilities, and the distribution of job family ratios offers valuable insights into the operational DNA of each approach.

The detailed view can also be accessed on Tableau Public HERE.

Disclaimer: The views and opinions expressed in this article are solely those of the author and do not reflect the official policy or position of any current or former employer. Any content provided in this article is for informational purposes only and should not be taken as professional advice.

Thanks for reading FFinsights! Subscribe for free to receive new posts and support my work.

Subscribe now