Project Management Skills Learned from Shopee Data Analysis Competition

On November 21, 2020, I teamed up with friends to participate in Shopee’s data analysis competition “I’m the Best Coder! Challenge 2020”. The competition venue was at Nangang Exhibition Center, with a maximum team size of four people. According to the organizers’ verbal announcement that day, there were approximately one thousand participants in total, which matched my rough estimate of the seating capacity. The competition was conducted through Kaggle, while official communications were delivered via Telegram.

This competition was divided into student and professional categories. The professional category’s task was to predict high-value customers for August, using transaction and login data from February to July. The student category focused on logistics box analysis. Since we registered for the professional category, I’m not certain about the detailed content of the student category’s problems.

Here’s the key point: this competition lasted only two hours. Under such intense time pressure, every minute and second counted, which is why this article came to be. Honing your programming skills and filling gaps in statistical and machine learning knowledge are basic necessities that go without saying. Therefore, how to properly organize planning, work allocation, and risk management in advance became my biggest takeaway from this competition. Simply put, we needed a PM (Project Manager).

First Challenge: Slow Internet Speed

The professional category had five datasets, all hosted on Kaggle, with a total uncompressed size of about 1GB. Imagine this: the venue was packed with a thousand people, and when the competition started, everyone’s first task was to download files. Although Shopee had set up seven WiFi routers and delayed the start by about twenty minutes for network checks, the internet was still severely congested, with my file download time reaching around 30 minutes at one point.

Due to the crowd, even mobile data experienced slowdowns, though it seemed faster than the venue’s WiFi. For this first challenge, I believe the optimal solution is to divide team members into WiFi download groups and mobile data download groups, with everyone downloading simultaneously using a dual approach. Another crucial point: bring your USB drive and ensure it has sufficient available space beforehand. Based on combinatorial principles, the optimal number of USB drives is “number of team members minus one.” Don’t let file transfers cause team downtime, and don’t count on Google Drive, Dropbox, or other cloud storage to save the day.

Two side notes here, both credit goes to my teammate Ken.

First is a mobile network tip. Even when everyone uses 4G networks, different data plans may affect maximum download speeds. Therefore, ideally, assign the teammate with the highest-tier mobile plan to the mobile data download group. If your plan isn\t unlimited or is a cheaper option, you naturally belong to the WiFi download group.

Second is a suggestion for organizers. Unless the organizers intentionally wanted to test participants’ process and risk management capabilities by arranging network congestion scenarios, they could have easily put the datasets in compressed files with strong password encryption, allowing participants to download files in advance and releasing the decompression password via Telegram when the competition begins. Even if someone attempted to crack the password, humanity might be extinct by the time a strong password gets broken.

Team Structure

Usually in data analysis competitions, most data has been cleaned and the volume isn’t excessively large, but participants still need to understand the data content and check for NAs or anomalies. Therefore, under a four-person team structure, I would divide the team into three roles:

1. Data Handlers: Two members responsible for understanding dataset content, data processing, and organizing data into model-ready format.
\n\n
2. Utility Player: One member serving as PM, moving between Data Handlers and Model Builder, must be the person who best understands the overall process, with solid grasp of both data processing and machine learning techniques.
\n\n
3. Model Builder: One member focused on building machine learning models and listing modeling Plans A and B.

Phase One

Once team members begin downloading datasets simultaneously, they can execute tasks according to their assigned roles. First, the Model Builder doesn’t need to spend time understanding the datasets, only needs to confirm the final submission content and format, then can start planning model and algorithm approaches, listing Plans A, B, and even C. The reason for listing these plans is simple: you need both perfect scenarios and worst-case preparations, possibly submitting with just one algorithm. Especially under tight competition time constraints, being able to submit an answer means you’re already halfway to winning. Therefore, after listing plans, the Model Builder can start executing the most important, indispensable model, and wait for others to complete preliminary tasks before providing support to achieve the most perfect model solution.

Meanwhile, Data Handlers and PM must understand the data content, with Data Handlers executing data table merging and preliminary calculations, while PM confirms that Data Handlers’ output can seamlessly integrate with models. Since Data Handlers work with raw data, theoretically their computer performance cant be too poor, and their coding speed can’t be too slow, because this is the most crucial first step. If data processing isn’t done well upfront, there’s no point in continuing. This is why I assign two Data Handlers and arrange PM assistance, prioritizing resources on the most urgent phase.

After discussing with Data Handlers, the PM doesn’t need to personally participate in data processing. They have another important task: generating dummy data using the merged data format, so the Model Builder can have actual files for modeling after understanding the approach, avoiding psychic coding. Once Data Handlers complete their tasks, the Model Builder only needs to swap the data source.

Phase Two

After Phase One tasks are completed, finished members certainly can’t sit idle, so organizational restructuring is necessary to execute Phase Two tasks.

Theoretically, the PM will be the first person freed up, so after preparing dummy data, the PM must personally participate in the modeling process, discussing different model combinations with the Model Builder while ensuring Data Handlers’ work proceeds smoothly. Since the PM moves back and forth between groups (PMs always do this), the Model Builder handles the main model code, managing overall model control, and even deciding to cut losses and adopt fallback plans.

After Data Handlers complete data processing and verification, the two Data Handlers have different Phase Two tasks. Data Handler A generates dummy data for the final answer and tests uploading. Competitions usually allow multiple answer submissions, and uploads failing midway is common. Data Handler A’s task is to familiarize themselves with Kaggle’s upload process and confirm any answer restrictions. We encountered this situation in our competition: uploaded files needed exactly 58,000 data points, but our calculated data only had 40,000 points, causing upload failure.

Data Handler B’s second task is organizing data processing code, merging two files into one, preparing code for possible final review stages. Note that if the competition lasts two hours, complete code might need uploading within two and a half hours, giving an extra half hour of work time. In this case, code organization work might not need immediate execution; Data Handler B can participate in modeling, letting Data Handler A handle code organization after testing uploads.

Phase Three

Phase Three is the final wrap-up stage. After the Model Builder merges all models, they first pass the code to the Data Handler (A or B) responsible for code organization, letting them review all code and record variables that need renaming. The Model Builder can use this time to re-execute and check models, then happily upload the answer upon completion.

Opportunity Favors the Prepared

In fast-paced competitions, I believe the biggest crisis is failing to clarify each team member’s work content and lacking a clear leader, with everything being improvised, leading to imbalanced workload and resource allocation, or redundant work. Therefore, if you accept the above organizational structure, you can elect a PM candidate before the competition and understand everyone’s strengths to further allocate tasks. Many team participants may lack previous collaboration experience, making preliminary preparation work even more important, including tool differences—whether Python or R, Tableau or Power BI. Without clarifying these, competition day will surely be full of surprises.

Of course, the above organizational structure is just my personal thinking; practical implementation may require adjustments. For instance, dividing data processing between two people somewhat increases task complexity. If a teammate is capable enough to handle this work single-handedly, the other Data Handler can transition to Model Builder.

Additionally, I personally adhere to the principle of “code written at work should also be usable after work.” Under this principle, I write many commonly used small functions and package them into a file, similar to a library concept, which many data science friends don’t seem to practice. I brought my library to this competition, and it indeed helped a bit. For usage methods, refer to my other article “Python – Loading Self-Written Packages/Modules/Libraries”.

After writing three thousand words extensively, honestly, this was also my first competition, and we didn’t even manage to submit our answer and script in the end, so the latter half of the process is essentially war-gaming (gets hit). If you find any errors or have any thoughts, please feel free to comment and discuss. Related article: “Learning Python and R on DataCamp: Quick Start to Data Science”