Automatically re-run data tasks on a bad run#2061

Data tasks can run “successfully” but return bad data. In a case like this, the task should be automatically flagged by windsor and re-run. Currently we (customers) have to manually re-run these silently failed tasks on our side.

Please see recent support session:

Our Meta destination tasks ran successfully this morning but returned almost no spend for the recent 30 days and had to be manually re-run.
Task names:

pluto_meta_ads_daily ID 46334
jtv_meta_ads_daily ID 40013
stryker_meta_ads_daily ID 46961
span_meta_ads_daily ID 46319

Windsor.ai Support
Helloo
let me check

max
thanks!
I’d like to understand if the issue was with meta or with windsor, and if there is any way to detect and automate fixing it if it occurs again in the futre
for example we pull a fresh 30 day refresh every day, so if suddenly today’s refresh contains 0 spend for the last 30 days and yesterday’s refresh did not, there was probably an error and a re-run should trigger

Windsor.ai Support
Thanks Max. This is a known Meta issue: for some ad accounts, Meta returns zero spend on daily rows when conversion fields are in the same request. We know about it and we are working on it. Your four 10:00 UTC runs happened before our first fix. Your re-runs this evening completed. Can you check if spend is correct now in BigQuery? If a date is still low, send me the task and the date.

max
yes its correct now after the manual re-run a few minutes ago
thanks! glad to hear your team is aware of it
in the mean time is there any way for you (or us? not sure if there is a setting somewhere perhaps) to implement a failsafe check/rerun like I described?

Windsor.ai Support
unfortunately we don’t have such a feature right now , there is no current way of knowing when it fails because I understand the job does not fail
it just silently ingest zeros in your metrics

max
yes exactly
I’m going to eventually build in a layer of alerting in our warehouse to check for stuff like this, but it won’t solve the task needing to be re-run
Windsor.ai Support
yes you can build a query in your warehouse to identify anomalies or when metrics return 0 on several rows when it should not be the case
max
thats what i’m imagining potentially could be automated on your end instead? like i imagine you guys cache a snapshot of the most recent run
to compare against i mean

Windsor.ai Support
as mentioned we don’t currently support any failsafe capacity , it would need to be requested as a feature request here
https://feature-requests.windsor.ai/
max
i’ll do that, thanks