VN1 Forecasting - Accuracy Challenge Phase 1
Share:
twitter social icon linkedin social icon facebook social icon
Finished
competition-bg

VN1 Forecasting - Accuracy Challenge Phase 1

Flieber, Syrup Tech, and SupChains Launch an AI-Driven Supply Chain Forecasting Competition

VN1 Forecasting - Accuracy Challenge
Machine Learning/AI
Enterprise
E-commerce/Retail
Total Prize 20,000
Join Datathon
Scroll To Top

27 Discussions

All other topics

Submission Error (Ordering)

Hi, so yesterday I was trying to make submissions for the phase 1. Unfortunately, I kept receiving errors in the submission page flagging nulls, blanks etc. I checked my datasets and these weren't present so I emailed the team but unfortunately, no response.

Since the phase 2 datasets were released, I was able to test using those datasets and it seems that have the datasets ordered so the index would match appeared to be causing the issue.

I don't expect to have any late submissions but thought I would make everyone aware in case of similar issues with phase 2.

Read more

Submission Error (Ordering)

Hi, so yesterday I was trying to make submissions for the phase 1. Unfortunately, I kept receiving errors in the submission page flagging nulls, blanks etc. I checked my datasets and these weren't present so I emailed the team but unfortunately, no response.

Since the phase 2 datasets were released, I was able to test using those datasets and it seems that have the datasets ordered so the index would match appeared to be causing the issue.

I don't expect to have any late submissions but thought I would make everyone aware in case of similar issues with phase 2.

Read more

About Competition Phases

I have not submitted any result yet. Reason: late joining and time management. 

Kindly tell me that I am eligible for phase 2 competition and if able to score good on leaderboard, then I will be eligible for price money or not.

Hey, you have to sign up to phase 2, it does seem like a seperate thing so when you submit you will go on a leaderboard, you just won't be able to see your ranking

Competition Evaluation Metric

I wanted to share a thought regarding the competition's evaluation metric and get your inputs. 
Since the competition's metric resembles MAE and 1) we are dealing with sparse, zero-inflated data, and 2) we are not evaluating models at a higher level of the hierarchy, could we end up selecting models (as the best model) that systematically under-forecast for non-zero, high-volume items?

MAE tends to select models that best estimate the median of the data, and in zero-inflated datasets, the median is often zero (or close to it). While we do have a bias term in the evaluation metric to help balance things, I’m curious if it’s strong enough to prevent this issue.

Typically, the dollar value of those minor, high-volume items is as significant, if not greater, than the majority of low-volume items, making under-forecasting them particularly costly.

Read more
There might be different ways to tackle the under-forecasting part but you need to process Client-Warehouse-Product one by one (can't vectorize all combinations with the same model), like using a segmentation approach to select the best forecasting model based on the time series characteristics. You can also optimize different kpis for each Client-Warehouse-Product combination based on the same "segmentation".
Also, you can improve the forecast probably if you don't select the full horizon for all the products. If there are new product introductions you don't need to send the full period of time to your forecasting model because it will under forecast.
Those were my 5 cents, at least that's how I structure the problem/solution. My score says my approach sucks though. 
Interesting, thanks. 
Evaluating Client-Warehouse-Product individually would be ideal, However it might not be feasible when dealing with millions of items, especially when using a global forecasting model. With using global models we need a loss function that delivers reasonable accuracy (RMSE) and bias for both slow-moving and fast-moving items, while minimizing the need for manual post-processing adjustments.
What if you do something in the middle? Like do some clustering of the Client-Warehouse-Product based on the time-series characteristics (seasonality periods, stationary, amount of periods with zero demand, average demand, trend, etc) so you can classify them in let's say 5 groups, and then you just run 5 models (one each of course).
I'm afraid clustering time series is even more complicated than forecasting them :). You barely can extract any meaningful features out of sparse time series.
Indeed all metrics based on AE will incentivize underforecasting for skewed datasets (when the median is lower than the average demand) 

That’s why we also look at the bias. 

But even when looking at both we might still slightly incentivize for low forecasts. 

For the sake of discussion, which would be your perfect kpi?
I tend to favor using RMSSE (or even scale-dependent RMSE) along with BIAS as two simultaneous evaluation terms. Ideally, we would also evaluate at higher levels of the hierarchy and differentiate between slow-moving and fast-moving items.
I think the best kpi depends on the characteristics of each Client-Warehouse-Product combination time series, that's why forecasting is artisanal.
I like the wasserstein distance for daily demand forecasts that need to sum up fairly well and can tolerate some flexibility on when the demand occurs (normally stores for stocking don't care too much if demand was off by a day, but they would care if it missed the demand entirely).
Thanks for sharing, was not aware of this measure. 

price == 0

what does price = 0 mean?. Is it free of charge?

Hello Adem, 
That's the data we got; I guess it might happen that some goods are given away for free or paid by a voucher/reimbursement. 
Thank you Nicolas

Submission upload broken? Or is it supposed to take 24 hours?

While the leaderboard does show some updated results, nothing I've uploaded works.  That includes the submission example provided here… which really should work.

I've also tested my submissions and they meet all the specified formatting requirements as per the attached test code. So what's up?

 

Read more
Hi Colin, Indeed Datasource.ai system is flagging your submission. The system is working without disruption as you can see from the volume of submissions. Some minor formatting issue in your file. You can send us an email your file support@datasource.ai and we will try to review. Please note that we have many messages, so we can't provide immediate feedback. 
So the issue is upload location, it is really weird. Does not accept files from my Windows computer, but the exact same file from my Mac is graded within seconds.
Working for me now, but geez
I tried a new account and uploaded just the sample and it is failing (never grading) too.

Large gap between submission and internal validation.

Hi all, 

 

I'm observing a large gap between my internal time-based cross validation, and the score shown on leaderboard.

I was expecting to find some bug in my code leading to leakage but I couldn't find anything relevant.

 

Are you also experiencing the same?

Read more
Hard to say since we know so little about the dataset, but one thing you could check is your holdout accuracy on the holiday period for 2022. The eval period is holiday season which is going to have very different sales patterns (presumably) than the previous parts of 2023.
Join our private community in Discord

Keep up to date by participating in our global community of data scientists and AI enthusiasts. We discuss the latest developments in data science competitions, new techniques for solving complex challenges, AI and machine learning models, and much more!

Join our Discord