A Data Scientist’s Daily Work 3 – Building Data Team Culture and Programming Standards

Series Articles

〈A Data Scientist’s Daily Work 1 – Mining Business Value from Data and Code〉
〈A Data Scientist’s Daily Work 2 – Understanding Company Organizational Structure Before Job Hunting〉
〈A Data Scientist’s Daily Work 3 – Building Data Team Culture and Programming Standards〉

This is the third article in the “A Data Scientist’s Daily Work” series. The first article provided a general overview; the second discussed possible organizational structures for data teams and the pros and cons of each structure. This article will delve into more technical and practical aspects, such as what risks poor coding style might create, and how it could potentially harm the company.

Comprehensive Background Knowledge

For data scientists and data analysts, although they write code, their programming habits differ significantly from conventional engineers. Some even lack adequate background knowledge in coding. You might argue, “Since many people doing data analysis aren’t from computer science backgrounds, their theoretical knowledge naturally won’t be as solid.” I acknowledge this as a possible reason, but I don’t think it should be used as an excuse. A professional should naturally fill in their knowledge gaps. I know similar issues exist among software engineers, but since I’m more familiar with the data science field, I’ll only comment within my area of expertise.

Let me share an example I witnessed firsthand. Our data team had a server for processing larger datasets. Servers typically allow multiple accounts, with each person working under their own account. Occasionally, we’d create dedicated accounts for specific business purposes, storing all code and computation results under these accounts. Additionally, data teams often need to schedule certain computations, like predicting customers likely to churn at the beginning of each month and collaborating with marketing to run retention campaigns targeting these customers.

I once discovered a rarely-used account on the server. According to team documentation, this account would run certain checks daily and determine whether computations and data updates were needed. When I logged into this account, I found that its automated scheduling wasn’t using Linux’s crontab, but rather a for loop that checked at fixed times daily whether computations were needed, and if not, would wait 24 hours before checking again. When I discovered this mind-boggling code, it had been running non-stop for two years. The programmer had long since left the company, and this program was consuming 99.8% of one CPU’s performance – absolutely ridiculous.

Another example: I discovered someone actually using Chinese characters as variable names in their code. While this might be technically feasible, it carries certain risks because Chinese characters involve encoding issues. When the same file runs on another computer, it might fail due to different system settings or IDE configurations. If you must write in Chinese, I strongly recommend keeping it in comments only.

Modularization / Functionalization

Let me first explain what a function is. Functions are a common technique in programming where frequently reused code is written in a specific format. This code might span dozens or hundreds of lines, but when you need to use it, you only need a few lines to call the function. This achieves the same effect as copying and pasting those hundreds of lines multiple times, but the latter is obviously inferior in terms of maintainability.

I’ve noticed that in the data science community, whether using R or Python, many people dislike writing functions. However, the benefits of writing functions are obvious – they improve work efficiency, program maintainability, and data accuracy. This phenomenon is quite puzzling to me. My colleagues have also struggled to understand why I’m so obsessed with turning everything into functions. In fact, I’m even more obsessed with the idea that a codebase – a collection of functions – represents a data team’s assets, a toolkit that can be reused quickly and efficiently. These assets, to some extent, reflect the team’s capabilities. Each team member works on different projects with different objectives. While projects may end, the functions written during these projects get incorporated into the codebase, making the team increasingly powerful.

If you work at a consulting company, this need might not be as strong since each client’s data format could be different. Conversely, if you’re an in-house analyst, I strongly recommend building your own analytical framework, because internal data remains relatively consistent – there’s no need to reinvent the wheel every time.

High Efficiency

For frequently executed actions or commonly retrieved data, writing them as functions with adjustable parameters maintains flexibility without starting from scratch each time. I’ve seen many people write SQL from the first line when pulling data, or copy and paste old code. This not only reduces efficiency but also creates maintenance nightmares.

For example, if you need to calculate detailed order information for every company order and include product data, this would require joining at least two data tables. Functionalizing this code and setting date ranges as parameters creates an easily understandable, reusable function.

Accelerating Development with Parameters

Beyond convenience, this approach offers absolute advantages in development speed. Continuing the previous example, if you need to use order details and product data to calculate monthly revenue and gross profit for each product category over the past year, and compare it with the previous year to calculate growth rates, this would obviously be a substantial program in both scope and computational load. If you hard-code all parameters, it becomes quite inconvenient during development and debugging.

A better approach is to set dates as function variables, allowing you to use just one month or one week of data during development and debugging. Under normal circumstances, if a function can correctly calculate one month of data, it should naturally handle one year of data as well.

Reducing Errors

Data teams face two major challenges. First, ensuring that every piece of data provided is accurate. Warren Buffett once described derivatives as weapons of mass destruction – data teams possess equivalent destructive power, for better or worse. A simple slip of the finger, adding an extra zero, could produce vastly different results. If this error crashes the entire program, that’s unfortunate but fortunate. The real fear is results that look normal but are actually completely wrong. If users don’t catch these errors and use them to make major but incorrect business decisions, the data department might easily be seen as saboteurs sent by competitors. Such errors can occur even in top multinational companies, like Facebook’s 2016 incident, 〈【Advertisers Cried】Facebook Admits Algorithm Error, Video Performance Severely Overestimated〉.

The second challenge is making users understand your analytical process, even if they don’t understand statistics, can’t write code, or aren’t familiar with algorithms. One of the data department’s missions is to explain complex computational logic simply. Only when users approve of your analytical process will they confidently use your results, because no one will completely trust your reports, especially when wrong decisions might affect their personal performance. But fundamentally, I believe the first reason is the most critical.

Providing accurate numbers sounds simple but is a real challenge many data teams face. Without systematic, rigorous validation mechanisms, producing incorrect data damages users’ trust in the data team. Eventually, users might lose interest in collaborating with the data team, making it impossible to promote a data-driven culture within the company. Everyone reverts to old methods and habits.

To avoid data errors, I believe there are two solutions: first, start with the code by functionalizing common features with built-in checking mechanisms; second, establish a code review culture within the team. These can be implemented simultaneously, but the former should be prioritized because eliminating most errors during the programming stage effectively reduces subsequent code review time and labor costs.

Starting with Code

For example, before functionalization, colleagues A and B might calculate different revenue figures for the same period. Perhaps colleague A excluded returns (negative revenue), but should returns be excluded according to business logic and company conventions? Then everyone starts discussing heatedly, even consulting relevant departments. However, such situations occur frequently, and people might forget today’s discussion results within a month. The same errors and scenarios will repeat. Not to mention new team members who might calculate incorrect results due to unfamiliarity with company business.

Since we know this will happen, why not write definitions and filtering conditions as functions, leaving comments within the function? You could even document which department’s representative was consulted on which date, because business environments change rapidly, and today’s definitions might be revised in a few months. For the department making definition changes, no one knows the data team uses this data, so naturally, no one will notify you when definitions change.

Writing explanations directly in comments has another benefit: team members don’t need to spend much time writing documentation. Everyone knows documentation is important, but everyone claims they don’t have time to write it, or there’s too much documentation to find what they need, so they just ask directly. While independent documentation is sometimes necessary, in most cases, combining documentation with functions has positive impacts on both writing and retrieval.

Another major programming consideration is building validation mechanisms into the code, such as confirming that data counts haven’t exploded due to incorrect joins, or using alternative data sources to verify result accuracy. For error handling, you can stop execution directly, or if it’s automated scheduling on a server, send emails to team members notifying them which program encountered errors.

Starting with Team Culture

Team culture here refers to code review culture. For software engineers, code typically goes through code review before being officially deployed, usually with senior colleagues helping verify program logic and quality. But I think code review can be divided into two levels: software and cultural.

Common code review methods require software implementation. Due to different business natures, software companies usually invest in such software, but if you’re in finance, manufacturing, or retail – application-oriented industries – companies might not be willing to pay for licenses.

If your team lacks code review software, but you ask colleagues to check your code at your desk after writing it, this might sound foolish, but it embodies the essence of code review culture. Everyone should be accustomed to and willing to do this, believing that code review positively impacts both personal growth and team capability enhancement. Honestly, I’m still exploring how to build this culture. I recommend referencing Google’s documentation: Google’s Engineering Practices documentation.

Reducing Collaboration Difficulty

Let’s Talk About Coding Style

Many programmers have some perfectionist tendencies, whether software engineers or data scientists. Because everyone has different coding style preferences, when taking over someone else’s code, they often think, “I need to modify their work and understand their logic first – might as well start from scratch.”

Coding style isn’t black and white; there are many gray areas. For example, Python has official 〈PEP 8 — Style Guide for Python Code〉 and Google’s 〈Google Python Style Guide〉. I haven’t carefully compared their differences, but it’s reasonable that different team philosophies would develop different coding styles, like Facebook emphasizing rapid product iteration while Google emphasizes high-quality code.

Although coding styles have many schools, many fundamental principles are the same, like not using one variable throughout the entire program, constantly overwriting it. This is a common bad habit. Because normal people don’t write perfect code in one go – there’s always a process of modification and refinement. If you use one variable throughout, you can’t simultaneously print and compare computational results with original data, significantly extending development time.

People using this approach usually respond, “Multiple variables increase memory usage.” I admit this is a potential drawback, but I don’t believe every program they write processes such massive data volumes. Trading obvious development efficiency for preventing risks that may not occur doesn’t seem smart to me.

In every coder’s learning journey, I believe you should pick one representative coding style, imitate it, learn it, and first ensure you don’t write terrible code. When you eventually need to collaborate with people using different styles, you can stand your ground and confidently say, “Although our styles differ, that doesn’t mean mine is inferior.”

Using Functions as Collaboration Breakpoints

As mentioned earlier, due to different coding styles, many programmers dislike inheriting others’ code. Indeed, even when teams establish clear coding styles, members still maintain personal habits in small details, because people aren’t machines. This uncertainty is human nature and a fundamental recognition needed when building team culture.

However, this uncertainty also creates opportunities for function culture to shine. No matter how much you can’t tolerate others’ programming, as long as it’s accurate and stable enough, just use it confidently without worrying about how the function is written internally. This recognition is crucial – only by accepting this working model can data teams stand on predecessors’ shoulders and continue growing, which leads to the next key point.

Standing on Predecessors’ Shoulders

Two Types of Data Analysis Cases

I usually categorize data analysis cases into two types: finding conditions and finding causes, the latter being root cause analysis. The former is easier, the latter more difficult.

What does finding conditions mean? For example, if you’re IKEA’s data team tasked with identifying consumers likely to move into new homes soon and offering them specific promotions, this is finding conditions. Condition sources might include past purchase records, website browsing history, or wishlists saved on the website. Through various condition filters, you identify a group of people, then calculate purchase probability, estimated spending, ROI, etc., and based on marketing budget, decide how many potential consumers should receive promotional messages.

From a product perspective, if IKEA wants to phase out old products and introduce new ones, which old products need elimination and at what clearance prices to avoid losses? Or whether kitchen display space should be reduced and toy area expanded – these are all examples of finding conditions.

Additionally, finding conditions analysis can actually take shortcuts by directly applying the 80/20 rule, because it’s a well-known, easily understood, easily implemented, and widely accepted principle. Some companies apply this principle by allocating 80% of resources to retaining existing customers and 20% to developing new ones, or keeping products that account for 80% of revenue while eliminating the remaining 20%. I don’t think blindly applying the 80/20 rule is good practice, but in business environments where comprehensive analysis before every decision is difficult, the 80/20 rule still has practical value.

As for finding causes cases, if you’ve ever been asked, “Can you help analyze why sofa sales declined this quarter?” this is a typical finding causes example. Finding conditions usually works within existing company data frameworks, which is why I consider it less difficult. But finding causes might have unlimited possibilities and very likely requires combining external data – perhaps competitors launched massive price-cutting promotions, or continuously rising housing prices mean fewer people buy homes, reducing furniture demand, or last year’s poor product quality led to numerous online complaints, affecting this year’s sales. Each reason makes some sense, and each requires external data for analysis. Whether needed external quantitative data can be obtained is one issue – worst case, potential causes can only be studied qualitatively. The time needed to import external data is another issue, but usually when looking for causes, sales are declining, and bosses might not have patience for slow progress.

Using Functions to Accumulate Internal Variables

There’s an article on Toward Data Science’s Medium: 〈How to conduct a proper root cause analysis〉, which mentions that finding root causes requires collecting lots of data and analyzing each potential cause’s impact. I believe the final result would be a regression equation, using coefficient sizes to determine which are the real root causes. I say “which” because the real world is always complex and interconnected – sales performance can’t be caused by a single factor.

Such analytical frameworks, even when collecting company internal data as variables, represent massive undertakings. For instance, sales impact might be due to lack of promotions or using less effective marketing methods; it could be insufficient supply causing sales decline; or customers usually want consistent living room furniture styles, so they buy sofas and tables together – table shortages affecting sofa sales.

In an ideal framework, marketing method effectiveness would be one function, determining adequate inventory would be another function, and correlation between sofa and table sales would be yet another function. Only by continuously accumulating code and wisdom can teams face such analytical cases.

Conversely, if teams haven’t cultivated function accumulation habits, they’ll definitely be overwhelmed, thinking about potential causes while starting to write code from line one, or spending time searching through computers for similar programs from other projects – maybe not finding them, or finding them but discovering they’re unusable. If you’re a current data scientist, this scenario will feel very familiar.

Conclusion

These are some experiences and insights I’ve gained from work, hoping they help everyone improve work efficiency or build team culture.

One additional note: in object-oriented programming languages, besides functions, you can also write classes. Based on my current project experience, whether using R or Python, I don’t currently see compelling reasons to write classes. You certainly could if you wanted, but it might be overkill. If you have different thoughts, please leave comments to share with me.

Online Learning Resources

If you’re interested in data science, there are many quality online learning platforms both domestically and internationally, like Taiwan’s Hahow or America’s Udemy. While Udemy’s courses are primarily in English, they’re gradually adding more Chinese courses. There are also platforms specifically for programming education where users can practice coding directly on the website, like DataCamp. You can reference 〈Learning Python and R on DataCamp for Quick Data Science Entry〉.