Understanding AggregationSource.INTERNAL

[rsmb7z]

Hi team,
I am trying use the Nautilus internal bar aggreation source to get the bars of higher timeframe, let's say using input of 1-MINUTE-EXTERNAL. I understand this is possible and works but may I know the following:

• Assuming I subscribe 1-HOUR-INTERNAL, when the on_bar is triggered. Will it be when the hour complete?
• If I need the 1-HOUR-INTERNAL current incomplete bar, how can I get that in a Strategy?

Thank you in advance. Also I am working on IB integration and found some bugs and improvments, I have made my way through to make it work but can I post my feedback in here same way as this post? Pardon my ignorance as first experience on github.

[cjdsellers]

Hi @rsmb7zhx

Thanks for your interest, to cover your first questions about the bar aggregation. To confirm, the input would need to be quote or trade ticks, and so for backtesting you'd need to pre-process any bar data into ticks to be able to aggregate to higher timeframes.

• 1: Yes, the platform will use its own internal clock to trigger closing of the bar right on the hour (so not based on ts_event of any tick)
• 2: Right now there is no way to make partial bars available to other components on request, I'm interest to know what your use case for this would be? the only way to do it would be to aggregate the bar at the strategy also, or create a custom component / actor for it

Thanks so much for your work with the IB adapter. I think you could either open a discussion here if the feedback is more conversational. Otherwise if you believe you have identified definite bugs or issues, you could raise a bug report from the top line here
https://github.com/nautechsystems/nautilus_trader/issues/new/choose

[rsmb7z]

Hi @cjdsellers
Thanks for the prompt response and clarification that it uses quotes and trade ticks only as input.
The use case of partial bar is assuming the Indicator cross below -80 or above -20. The strategy would work as follows:

• Entry into trade when Indicator value crosses below -80 [In this case previous closed bar is used which is current/normal behavior of system]
• Exit the trade if Indicator value crosses above -80 [In this case I want this on current active bar]
• The reason for not waiting for the bar to close is price may move much further during that time, alternatively strategy can always enter again if it was just whipsaw. This usage would make more sense while signal is generated on Hourly and Daily bars, as the idea is to use indicator as monitor for open position.

[cjdsellers]

In your second case, you can just immediately close the position, you don't need to wait for a bar. i.e. it will be closed on the current active bar.

[rsmb7z]

So the Indicator will be giving the value generated on current bar? Because Indicator is subscribed on Hourly bar, unless only the on_bar is triggered at hour completion but the Indicator will keep getting the current/partial bar updated even and resulting in the value based inclusive of partial bar. Is that correct?

[rsmb7z]

Makes sense. I will give it a try. Thanks.

[rsmb7z]

In your second case, you can just immediately close the position, you don't need to wait for a bar. i.e. it will be closed on the current active bar.

Hi @cjdsellers
I have tested second case and unfortunately it didn't worked as I needed. Indicator will keep returning the value of last closed bar on which it is registered which means it doesn't have visibility of in-time bar updates. This behavior is more similar to the pandas.resample case (below function) used in different backtesting systems, whereas more real life scenario is that if you are in middle of timeframe until that time the bar and its respective indicators will always be up to date. If you can please add this in feature request that would be great by adding some sort of event trigger on_bar_update into Strategy.

Example timesampling function used in Vectorized systems [current behavior of nautilus is same].

    def resample(self, df, rule, label='right'):
        OHLCV_AGG = OrderedDict((
            ('Open', 'first'),
            ('High', 'max'),
            ('Low', 'min'),
            ('Close', 'last'),
            ('Volume', 'sum'),
        ))
        agg = OHLCV_AGG.get(getattr(df, 'name', ''), 'last')
        if isinstance(df, pd.DataFrame):
            agg = {column: OHLCV_AGG.get(column, 'last')
                   for column in df.columns}
        resampled = df.resample(rule, label=label).agg(agg).dropna()
        resampled = resampled.reindex(index=df.index.union(resampled.index), method='ffill').reindex(df.index)
        return resampled

[cjdsellers]

You're correct - the indicator doesn't have visibility into partial bar updates. This is actually by design though, the bar aggregators exist in the DataEngine, when a bar is closed as per the AggregationMethod then the bar is passed to the message bus as a data event, where the strategy that has subscribed for them (and all other components which have subscribed) will then receive it.

If your strategy needs to act on logic which is not part of the bar aggregation, or it needs updates and to act on those prior to the closing of a bar, then I think you'll need to subscribe to the ticks directly in the strategy, or change the indicator to instead subscribe to ticks rather than bars.

[santatic]

@rsmb7z I'm looking for a solution for the second case, how did you deal with it?

[rsmb7z]

Hi @santatic,
The INTERNAL aggregator doesn't support this. I have achieved using EXTERNAL bars. This has been resolved to some extend in #897 where is_revised flag is introduced for the Bar. So when the Bar is sent by adapter, in my case IB, it will set this flag to True for subsequent updates of same Bar. You can refer #964, where if handle_revised_bars=True will behave the keepUptoDate behavior. This way, in the Strategy, you will become aware that the Bar is new or update to previous one and perform actions accordingly. However beware that built-in Indicators still doesn't recognize the is_revised bar and will simply keep appending as new Bar causing wrong results - you can use with custom or modified indicators. Btw which adapter you are working with?

[santatic]

Hi @rsmb7z,
Thank you for your quick reply.
I working on a strategy working on M5 timeframe, but need to get current incomplete daily timeframe bar.

Btw which adapter you are working with?

I'm testing on backtest but wanna try it on binance.
Did you do any testing it on backtest? I'm trying to implement a is_revised on backtest but still didn't have any idea on indicators

[rsmb7z]

@santatic I have raised the feature request #1015. Followup there, hopefully we shall come across a solution. So if this is implemented will support both backtesting/live scnearios. As for live, at the moment only InteractiveBrokers adapter support to receive partial bars in Strategy on_bar.