Implement a data hole warning (customizable)

[stastnypremysl]

Feature Request

Problem Statement

I would like to get the the data integrity verified during each run of any strategy. The data holes in data presents in my opinion large potential problem for a user and he should be notified immediately, if any is detected.

I also propose a warning to be fired at the beginning and the end of backtest run under same conditions, so the user is aware, that the time range filled doesn't contain all data.

Proposed Solution

I would like to have at least two parameters given to the NT high level API. One about max data hole size in bars and one about max data hole size of ticks in time. (more parameters might be more suitable as the different setting might be for quote ticks and trade ticks,...)

If max data hole in bars is exceeded, the warning will be fired (eg. if one bar is missing). If max time without a tick is exceed, the warning will be fired (e.g. if there was no tick for more than one minute).

Further note

This function might help to find potential bugs in the catalog, as the user will be aware of any data missing.

[stastnypremysl]

I also propose to add data hole exceptions time ranges for ticks. There might have been some outage in the venue, due to which no tick is really present.

The checks for ticks might be done by scheduled function, so there is almost none performance hit.

[stastnypremysl]

As i realized, there must also be data holes exceptions for weekends for Forex, and for similar situations.

[limx0]

so the user is aware, that the time range filled doesn't contain all data.

I don't think you can know this for sure.

I would like to get the the data integrity verified during each run of any strategy.

This sounds like something more appropriate to be handled by the user when they load data into the platform. It can be checked once rather than every time a strategy runs (which may happen many times during optimisation).

I don't believe there is much value adding these checks into nautilus itself as it is not specifically a core competency of the platform.

[stastnypremysl]

@limx0, yes. You can't know for sure. But better something then nothing in my opinion.

And you are right, this is not core competency of the platform. To explain the motivation, I will explain the general concept which would this be part of.

This is rather one of few validations, I am thinking about proposing, which might reveal some bugs during unexpected usage by user - therefore it is working in the same time as a test and enhancing the blackbox testing concept.

I am planning to make a RFC for validation levels, similar to optimization level in compiler (inverse). It would automatically enable validations, as the appropriate level is set. But I haven't thought about all details yet.

Basically, during the optimization user would set value 0 and no validation would be made. If set 3, all validations including validations of the platform would be made.

This would basically be next approach, how to deal with some kind of bugs. Let the blackbox tests fail on generic validation errors without repeating the assert code over and over. Let the user know by the checks that the backtest can't be trusted and let him write the issue.

I know the idea itself might seem unnatural, but it would be similar to some Linux kernel features, which runs validations on runtime.

The platform needs in my opinion enhance testing and this might be one of the ways, how to do it. And furthermore, the platform will be more trustworthy, if it contains validations of it own. And there won't be almost any performance hits as the user would enable and disable validation as he needs. Not speaking about the added value of validating the user mistakes.

The real problem I would see is the prioritization and slowing down development by too many tests. Until NT is a beta, I guess requirements on sufficiant testing aren't so high as when the NT stabilizes. It's up to discussion if/when these will be implemented.