Without strong leaders, data science teams fail to thrive. Their projects are selected haphazardly. They aren’t accountable for results with positive business impact. Lack of leadership is a big part of why data science teams return in the 0X to 5X range instead of reaching their full potential of 25X to 100X.
Study data science teams at Google, Facebook, or Amazon. Even at companies that have figured out data science, team performance is inconsistent. The highest performing teams don’t always have the top talent, but they do have the best leaders.
Without a clear understanding of what does and doesn’t work, developing data science leaders can be a hit and miss process. The effectiveness of a business’s leadership development process is an overall predictor of employee success. That’s especially true for the data science team. However, leadership traits that are highly effective on other technical teams fall flat in data science teams.
I’m going to cover what has worked and go deeper than usually necessary into what doesn’t. I’m drawing from my experience leading traditional technical teams as well as data science teams. Though they seem similar on the surface, it takes different leadership behaviors and techniques to successfully oversee a data science team. I’m starting with what doesn’t work to highlight those differences.
The Challenges Of Leading A Data Science Team – What Doesn’t Work
I have friends who ask me to come work for them. I respond, “I like you too much to do that to you.” Data scientists are a challenge to lead because we deny leaders their traditional sources of authority. Add to that the challenges of leading high performers and the complexity of the work itself. This is a role few leaders are successful in.
Leaders who haven’t taken on a team like this don’t know where all the pitfalls are. Too many stumbles and the leader has lost the team’s trust. That’s the tightrope. For inexperienced leaders, it’s a setup for failure. However, there is a roadmap and with a few insights, leaders have a much better chance of succeeding in their new role.
Data science leaders must abandon the traditional sources of authority. Most data scientists don’t want to be leaders but that doesn’t mean they are quick to accept the premise that someone else can lead them. Data scientists question assumptions relentlessly. Traditional authority erodes here because the necessity of that leader isn’t accepted by the group based on title or hierarchy.
Formal authority is rooted in a rule structure. There are positive consequences for following and negative consequences for ignoring the leader’s authority. Data scientists have the leverage in this structure rather than the leader. Attrition is expensive. That makes firing or discipline that leads to data scientists departing nonstarters.
Competence is a common source of authority in technical teams. Often the best engineer is promoted to a leadership position because the team follows that person organically. This is also true in data science teams. This path fails because most data scientists don’t want to be leaders and many of those who take on the role spend their time as technical leaders instead of people leaders.
The first approach I tried combined charisma and competence. What had previously been highly effective tools of charismatic influence were quickly revealed as amateurish psychological parlor tricks. On many data science teams there is someone with a behavioral/psychological background. The worst takedown of my career happened about 5 years ago when one of these data scientists exposed my charisma trick by trick. I quickly abandoned charisma because analytical minds see through it and want something more substantial.
Even competence eventually fails because there’s always someone smarter than me. Data science is a field so complex that no one person is ever the most competent across every project. One of the cornerstones for leading a data science team is the leader must be able to relinquish technical leadership without losing their source of authority.
What’s Left? Two Sources Of Authority That Work
The first is alignment of the organization’s goals with the individuals’ goals. It’s basic game theory. Create an equilibrium where the activities of the data science team lead to success for both the business and the data scientist. Leaders need to take the time to understand each data scientist’s personal goals and motivators then build a structure that rewards the data scientist for activities that create business value. Alignment is a powerful source of authority because the data scientist is internally motivated by those layered goals.
There is a second piece to alignment, connecting team goals with individual goals. Data science should be a collaboration between all team members to avoid ‘Not My Project’ syndrome. It’s easy for data scientists to focus on their projects to the detriment of others’. Putting in place a system that rewards everyone for team success creates an incentive to jump in as needed. That framework creates accountability to each other instead of focusing accountability on the leader.
This is the setup for the second source of authority, trust. Aligned goals build the framework for the team to be a part of the larger company and for each data scientist to be part of the team. That allows each person to trust the team and larger business when they see both contributing to aligned goals. However, when they perceive, right or wrong, that a person or group isn’t fully supporting those goals, they’ll be just as committed to calling it out and insisting on a resolution.
I’ve learned that trust is a two-way street. A leader can only ask the team to follow her/him if that leader is willing to follow them. It sounds like a motivational poster but it’s a truism for leading data science teams. Data scientists expect to be listened to and have their thoughts acted on. That means a data science leader is often in the position to manage and translate up.
A data science leader needs to be capable of pushing back on deadlines, projects, a lack of funding or staffing, and any other scenario where what the team is being told to do doesn’t line up with their understanding of the goals. That happens a lot because these types of goals are often dictated to the team without team input. That disconnect requires a leader who can address the issue at whatever level is necessary.
The leader needs data science domain expertise to understand the details of the disconnect. They also need business acumen, so the leader can translate the disconnect into language decision makers understand. The team needs to trust the leader is not only willing to take their concerns up but capable of driving a sensible resolution.
In other cases, the leader uses those same skills to translate business cases to the data science team well enough that they build models to meet the need. The team must trust the leader as a conduit between them and external teams, leadership, and sometime users. That’s an important piece of the data science leader’s value proposition and goes a long way towards the team seeing the leader as an enabler rather than an extra, unnecessary step in the chain.
Trust and alignment replace technical competence with leadership competence. Once the team understands the need for and the benefits of leadership, they’re more willing to accept the leader. Acceptance is the end goal of alignment and trust-based leadership. While this is basic leadership theory, it’s instructive to understand why other fundamentals of leadership theory fail while this approach thrives.
Why Can’t Companies Hire Experienced Data Science Leaders?
In a perfect world, every data science manager would have 5+ years of experience and 2-3 years in a leadership role. Every director would be closer to 10 years with 5 in leadership. What I’ve just covered would be known to them through experience. In the real world, there are only a few thousand data scientists with those qualifications and waiting for one of them to show up isn’t an option.
Data science leaders are pressed into service out of necessity rather than readiness. Often, they must be convinced to take the role in the first place. Data scientists see leadership roles as gateways to more control over the projects they work on and methodologies used. They want to be technical leaders rather than people leaders.
I’ve used two approaches to provide data science teams with competent leadership.
Training Leaders Into Data Science
Pulling experienced leaders out of other teams is the easiest way to source data science leadership talent. Business analytics, software development, research, and advanced R&D leaders are perfect for these roles. Each functional area has a piece of the data science skillset and that’s a big part of integrating the leader into their new functional area.
Business acumen, software development experience, and advanced research capabilities are all value adds to a data science team. Data scientists are generalists in each area. Having someone who has extensive experience in one of these areas makes everyone on the team better.
These leaders have well established relationships with other teams and senior leadership. Most data scientists don’t see the big picture of how their work fits into the business. Established leaders from related teams have a broader picture. In their previous role, they’ve built most of the relationships the data science team needs to deliver products. That connection and collaboration with external groups is key to getting more out of the data science team.
How does the business make up for the knowledge gap between the leader and the data science team? Select a member of the data science team as a technical lead. Combining an experienced leader with a knowledgeable technical lead works to bridge the gap.
Training Data Scientists To Lead
To put it bluntly, this is difficult and unlikely. It’s worth mentioning because the data scientist leader is immensely valuable. From an employee lifetime value standpoint, the data scientist leader is close to or on par with members of the C-Suite. Andrew Ng and Kirk Borne are examples of data scientist leaders who’ve returned $100’s of millions to their companies. If you find a data scientist with leadership potential and aspirations, put golden handcuffs on them.
They need the same training as any other new leader. Partnering them with a strong engineering leader is the perfect mentorship. I was fortunate to have 3 amazing senior mentors during my early career. They taught me leadership at a level I couldn’t have gotten from books and training classes. There’s nothing like the day to day education from someone who’s been there before.
My mentors created an informal learning path for me using books and training classes that they reinforced during our talks. The process of learn, do, improve brought me a long way forward.
It’s a tough decision for business, spending that much to improve an employee, because of the age-old question, “What if they leave?” Data scientists are motivated by challenge and growth. Provide both to a data scientist leader and they won’t go anywhere because that opportunity is very rare. Most businesses don’t know they need to train data science leaders, let alone how.
From the data science leader’s perspective, this role is a career cornerstone. Few leaders in the tech space can say they’ve led teams with this level of productivity and complexity. I would argue that without taking a role in the C-Suite or starting their own company, there’s no such thing as a promotion for a data science leader. These leaders want the role as much as the company wants them to stay in it.