Portfolio¶

Portfolio refers to any combination of financial assets held by a trader. In the world of vectorbt, "portfolio" is a multidimensional structure capable of simulating and tracking multiple independent but also dependent portfolio instances. The main function of a portfolio is to apply a trading logic on a set of inputs to simulate a realistic trading environment, also referred to as "simulation". The outputs of such a simulation are orders and other information that can be used by the user in assessing the portfolio's performance, also referred to as "reconstruction" or "post-analysis". Both phases are isolated in nature, which enables various interesting use cases for quantitative analysis and data science.

The main class concerned with simulating and analyzing portfolios (i.e., with the actual backtesting) is Portfolio, which is a regular Python class subclassing Analyzable and having a range of Numba-compiled functions at its disposal. It's built similarly to other analyzable classes in the way that it has diverse class methods for instantiation from a range of inputs (such as Portfolio.from_signals taking signals), it's a state-full class capable of wrapping and indexing any Pandas-like objects contained inside it, and it can compute metrics and display (sub-)plots for quick introspection of the stored data.

Simulation¶

So, what's a simulation? It's just a sophisticated loop!

A typical simulation in vectorbt takes some inputs (such as signals), gradually iterates over their rows (time steps in the real world) using a for-loop, and at each row, runs the trading logic by issuing and executing orders, and updating the current state of the trading environment such as the cash balance and position size. If we think about it, it's the exact same way we would approach algorithmic trading in reality: at each minute/hour/day (= row), decide what to do (= trading logic), and place an order if you decided to change your position in the market.

Now, let's talk about execution. The core of the vectorbt's backtesting engine is fully Numba-compiled for best performance. The functionality of the engine is distributed across many functions in an entire sub-package - portfolio.nb, ranging from core order execution commands to calculation of P&L in trade records. Remember that those functions aren't meant to be used directly (unless specifically desired) but are used by Python functions higher in the stack that know how to properly pre-process their input data and post-process the output data.

In the following parts, we'll discuss order execution and processing, and gradually implement a collection of simple pipelines to better illustrate various simulation concepts.

Primitive commands¶

Remember that vectorbt is an exceptionally raw backtester: it's primary commands are "buy" and "sell" This means that any strategy that can be translated into a set of those commands is also supported out of the box. This also means that more complex orders such as limit and SL orders must be implemented manually. In contrast to other backtesting frameworks where processing is monolothic and functionality is written in an object-oriented manner, Numba forces vectorbt to implement most of the functionality in a procedural manner.

Functions related to order execution are primarily located in portfolio.nb.core. The functions implementing our primary two commands are buy_nb and sell_nb. Among the requested size and price of an order, the primary input of each of these functions is the current account state of type AccountState, which contains the current cash balance, position size, and other information about the current environment. Whenever we buy or sell something, the function creates and returns a new state of the same type. Furthermore, it returns an order result of type OrderResult, which contains the filled size, price adjusted with slippage, transaction fee, order side, status information on whether the order succeeded or failed, and valuable information about why it failed.

Buying¶

The buy operation consists of two distinct operations: "long-buy" implemented by long_buy_nb and "short-buy" implemented by short_buy_nb. The first one opens or increases a long position, while the second one decreases a short position. By chaining these two operations, we can reverse a short position, which is done automatically by buy_nb: it checks the position we're in (if any), and calls the responsible function.

Let's say we have $100 available and want to buy 1 share at the price of $15:

>>> from vectorbtpro import *

>>> account_state = vbt.pf_enums.AccountState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,  # (1)!
...     locked_cash=0.0,  # (2)!
...     free_cash=100.0  # (3)!
... )
>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state,
...     size=1.0,
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=1.0,
    price=15.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=85.0,
    position=1.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=85.0
)

1. Debt is non-zero only when leveraging or shorting
2. Locked cash is non-zero only when leveraging or shorting
3. Free cash deviates from the regular cash balance only when leveraging or shorting

The returned state indicates that we spent $15 and increased our position by 1 share. The order result contains details about the executed order: we bought 1 share for $15, with no transaction fees. Since order side and status are of named tuple type OrderSide and OrderStatus respectively, we can query the meaning behind those numbers as follows:

>>> vbt.pf_enums.OrderSide._fields[order_result.side]
'Buy'

>>> vbt.pf_enums.OrderStatus._fields[order_result.status]
'Filled'

Now, based on the new state, let's execute a transaction that uses up the remaining cash:

>>> order_result, new_account_state2 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state,  # (1)!
...     size=np.inf,  # (2)!
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.666666666666667,
    price=15.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=0.0,
    position=6.666666666666667,
    debt=0.0,
    locked_cash=0.0,
    free_cash=0.0
)

1. Use the previous account state as input
2. Infinity means using up the entire balance

Since vectorbt was originally tailored to cryptocurrency and fractional shares, the default behavior is buying as much as possible (here 5.67), even if the amount is below of that requested. But what happens if we wanted to have the entire share instead? Let's specify the size granularity of 1, indicating that only integer amounts should be allowed:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state,
...     size=np.inf,
...     price=15.0,
...     size_granularity=1
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=6.0,
    price=15.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=10.0,
    position=6.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=10.0
)

This has bought exactly 6 shares. Given the new account state, let's run the same transaction again:

>>> order_result, new_account_state2 = vbt.pf_nb.buy_nb(
...     new_account_state,  # (1)!
...     size=np.inf,
...     price=15.0,
...     size_granularity=1
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=np.nan,
    price=np.nan,
    fees=np.nan,
    side=-1,
    status=1,
    status_info=5
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=10.0,
    position=6.0,
    debt=0.0,
    free_cash=10.0
)

1. Use the account state from the previous operation

The account state remains unchanged, while so many NaNs in the order result hint at a failed order. Let's query the meaning behind the status and status information numbers using OrderStatus and OrderStatusInfo named tuple respectively:

>>> vbt.pf_enums.OrderStatus._fields[order_result.status]
'Ignored'

>>> vbt.pf_enums.OrderStatusInfo._fields[order_result.status_info]
'SizeZero'

>>> vbt.pf_enums.status_info_desc[order_result.status_info]  # (1)!
'Size is zero'

1. There is a list with more elaborative descriptions of different statuses

Here, the status "Size is zero" means that by considering our cash balance and after applying the size granularity, the (potentially) filled order size is zero, thus the order should be ignored. Ignored orders have no effect on the trading environment and are simply, well, ignored. But sometimes, when the user has specific requirements and vectorbt cannot execute them, the status will become "Rejected", indicating that the request could not be fulfilled and an error can be thrown if wanted.

For example, let's try to buy more than possible:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=1000.0, 
...     price=15.0,
...     allow_partial=False
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=np.nan,
    price=np.nan,
    fees=np.nan,
    side=-1,
    status=2,
    status_info=12
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=100.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=100.0
)

>>> vbt.pf_enums.OrderStatus._fields[order_result.status]
'Rejected'

>>> vbt.pf_enums.status_info_desc[order_result.status_info]
'Final size is less than requested'

There are many other parameters to control the execution. Let's use 50% of cash, and apply 1% in fees and slippage:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=np.inf, 
...     price=15.0,
...     fees=0.01,  # (1)!
...     slippage=0.01,  # (2)!
...     percent=0.5  # (3)!
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=3.2676534980230696,
    price=15.15,
    fees=0.4950495049504937,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=50.0,
    position=3.2676534980230696,
    debt=0.0,
    locked_cash=0.0,
    free_cash=50.0
)

1. 0.01 = 1%. Paid always in cash. To specify fixed fees, use fixed_fees instead.
2. 0.01 = 1%. Applied on the price. By artificially increasing the price, you always put yourself at a disadvantage, but this might be wanted to make backtesting more realistic.
3. 0.01 = 1%. Applied on the resources used to open or increase a position.

The final fees and the price adjusted with the slippage are reflected in the order result.

Whenever we place an order, we can specify any price. Thus, it may sometimes happen that the provided price is (by mistake of the user) higher than the highest price of that bar or lower than the lowest price of that bar. Also, if the user wanted the price to be closing, and he specified a slippage, this would also be quite unrealistic. To avoid such mistakes, the function performs an OHLC check. For this, we need to specify the price_area of type PriceArea: with the price boundaries, and what should be done if a boundary violation happens via price_area_vio_mode of type PriceAreaVioMode:

>>> price_area = vbt.pf_enums.PriceArea(
...     open=10,
...     high=14,
...     low=8,
...     close=12
... )
>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state,
...     size=np.inf,
...     price=np.inf,
...     price_area=price_area,
...     price_area_vio_mode=vbt.pf_enums.PriceAreaVioMode.Error
... )
ValueError: Adjusted order price is above the highest price

Selling¶

The sell operation consists of two distinct operations: "long-sell" implemented by long_sell_nb and "short-sell" implemented by short_sell_nb. The first one decreases a long position, while the second one opens or increases a short position. By chaining these two operations, we can reverse a long position, which is done automatically by sell_nb: it checks the position we're in (if any), and calls the responsible function.

The function for selling takes the same arguments but uses them in the opposite direction. Let's remove 2 shares from a position of 10 shares:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=0.0,
...     position=10.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=0.0
... )
>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state,
...     size=2.0,
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=2.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=30.0,
    position=8.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=30.0
)

The size in the order result remains positive but the side has changed from 0 to 1:

>>> vbt.pf_enums.OrderSide._fields[order_result.side]
'Sell'

Shorting¶

Shorting is a regular sell operation with sell_nb, but with one exception: it now involves the debt as well as the locked cash balance. Whenever we short, we are borrowing shares and selling them to buyers willing to pay the market price. This operation increases the cash balance and turns the position size negative. It also registers the received cash amount as a debt, and subtracts it from the free cash balance. Whenever we buy some shares back, the debt decreases proportionally to the value of the shares bought back, while the free cash might increase depending upon whether the price was higher or lower than the average short-selling price. Whenever we cover the short position entirely, the debt becomes zero and the free cash balance returns to the same level as the regular cash balance.

To borrow any shares, we need a positive free cash balance to be used as a collateral. The exact amount of free cash needed for a shorting operation depends on margin; by default, we need the same amount of funds available in our margin account as the value of to-be-borrowed shares. For example, if we short a stock and the new position had a value of $100, we would be required to have the $100 that came from the short sale plus an additional $100 in cash, for a total of $200, which depending on the definition is either a 100% (before sale) or 200% (after sale) initial margin requirement. Maintenance margin and liquidation checks are the responsibility of the user (for now).

>>> account_state = vbt.pf_enums.AccountState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0
... )
>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=np.inf,  # (1)!
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=6.666666666666667,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=200.0,
    position=-6.666666666666667,
    debt=100.0,
    locked_cash=100.0,
    free_cash=0.0
)

1. Short-sell as much as possible

Here's what happened. First, we've converted all the available free cash ($100) into the locked one such that it becomes the collateral of our shorting operation. Because the default leverage is 1, we've borrowed the same value in shares ($100), which has been added to the regular cash balance as well registered as a debt. This value corresponds to (minus = borrowed) 6.67 shares. But here's the problem: since we've doubled the regular cash balance, it may be used by other assets. To avoid that, all operations are done strictly on the free cash balance! But it's zero right now, so how do we buy back the borrowed shares? Remember that the debt and locked cash represent the total amount of cash that we used in the first place; by adding both to the free cash, we get our cash limit for the current buy operation, which nicely matches the regular cash when dealing with only one asset.

To change the margin, we can use the argument leverage. For example, setting it to 2 will allow us to borrow twice as many shares as can be covered by the current free cash:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0
... )
>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=np.inf,
...     price=15.0,
...     leverage=2
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=13.333333333333334,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=300.0,
    position=-13.333333333333334,
    debt=200.0,
    locked_cash=100.0,
    free_cash=0.0
)

The debt-to-locked-cash ratio is now 2, which corresponds to the leverage that we specified.

Let's try to run the same operation again, but now on the new account state:

>>> order_result, new_account_state2 = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     size=np.inf,
...     price=15.0,
...     leverage=2
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=np.nan,
    price=np.nan,
    fees=np.nan,
    side=-1,
    status=2,
    status_info=6
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=300.0,
    position=-13.333333333333334,
    debt=200.0,
    locked_cash=100.0,
    free_cash=0.0
)

>>> vbt.pf_enums.OrderStatus._fields[order_result.status]
'Rejected'

>>> vbt.pf_enums.status_info_desc[order_result.status_info]
'Not enough cash'

We see that vectorbt prevents the free cash balance to become negative.

To order any quantity possible, we can use the unlimited leverage:

>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=1000, 
...     price=15.0, 
...     leverage=np.inf
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=1000.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=15100.0,
    position=-1000.0,
    debt=15000.0,
    locked_cash=100.0,
    free_cash=0.0
)

What's the effective leverage of this operation?

>>> new_account_state.debt / new_account_state.locked_cash
150.0

If we had to calculate the current portfolio value, it would still default to the initial cash since no transaction costs were involved and no additional trades were made:

>>> new_account_state.cash + new_account_state.position * order_result.price
100.0

As we see, the positive cash balance and the negative position size keep the total value in balance. Now, let's illustrate buying back some shares using buy_nb. First, we'll borrow 10 shares with 2x leverage and sell them for the price of $10 per share:

>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=10.0, 
...     price=15.0,
...     leverage=2
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=250.0,
    position=-10.0,
    debt=150.0,
    locked_cash=75.0,
    free_cash=25.0
)

Let's buy back 5 shares for the price of $30 per share (my condolences):

>>> order_result, new_account_state2 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state, 
...     size=5.0, 
...     price=30.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=30.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=100.0,
    position=-5.0,
    debt=75.0,
    locked_cash=37.5,
    free_cash=-12.5
)

We executed the order for $150, which have been deducted from the regular cash balance. The position has been reduced by half, to 5 borrowed shares. Along with the position, the debt and locked cash have also been reduced by half. Given the absolute amount of released debt ($75), we can then compute the P&L, which is the total spent cash subtracted from the total released debt, or -$75. But this operation has also released some locked cash - $37.5, such that the amount of cash added back to our free cash balance is -$37.5, which makes it negative. A negative free cash means that we won't be able to buy any other assets apart from reducing any short positions and thus potentially releasing additional funds (i.e., profits as well as losses are shared among all assets within the same group with cash sharing). Even though we've got a negative free cash, we can still buy back more shares because the sum of debt, locked_cash, and free_cash is greater than zero.

Not only the debt and locked cash can be used to compute the effective leverage, but we can also derive the average entry price of the entire position:

>>> new_account_state2.debt / abs(new_account_state2.position)
15.0

Let's say instead of jumping, the price has dipped to $10 per share (my congratulations!):

>>> order_result, new_account_state2 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state, 
...     size=5.0, 
...     price=10.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=10.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=200.0,
    position=-5.0,
    debt=75.0,
    locked_cash=37.5,
    free_cash=87.5
)

We see that the debt and locked cash have decreased to the same level as previously (because we've bought back the same relative amount of shares), but the free cash balance is now $200, netting $25 in profit! Again, the calculation is simple: we just take the total amount of spent cash ($5 * 10 = $50) and subtract it from the total released debt (0.5 * $150 = $75) to get the P&L. By adding the P&L, the released locked cash (0.5 * $75 = $37.5), and the current free cash ($25) together, we get the new free cash of $87.5 immediately available for all other assets to use.

Let's compute the equity to validate the profit:

>>> st0 = account_state
>>> st1 = new_account_state2
>>> avg_entry_price = st1.debt / abs(st1.position)  # (1)!
>>> new_equity = st1.cash + st1.position * avg_entry_price
>>> new_equity - st0.cash
25.0

1. The remaining shares are still valued at $15

Let's close out the open short position using the same price:

>>> order_result, new_account_state3 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state2, 
...     size=5.0, 
...     price=10.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=10.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state3)
AccountState(
    cash=150.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=150.0
)

The free cash balance equals to the regular cash balance, and we are debt-free! Additionally, the last two operations have brought us $50 in profit, or (15 - 10) * 10 = $50.

Finally, let's try to close out the position using an astronomically high price!

>>> order_result, new_account_state3 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state2, 
...     size=5.0, 
...     price=100.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=2.0,
    price=100.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state3)
AccountState(
    cash=0.0,
    position=-3.0,
    debt=45.0,
    locked_cash=22.5,
    free_cash=-67.5
)

We could buy back only 2 shares out of the remaining 5. If we try the same operation again, we would get the "Not enough cash" message because debt + locked_cash + free_cash is below or equal to zero. We also witness the regular cash balance going to zero, which means that we've exhausted all of our capital; but we shouldn't rely on it to make trading decisions! If any other asset buys shares with leverage, regular cash may go negative, which doesn't necessarily mean that we have no cash left - only free cash (with debt and locked cash when covering shorts) provide us with the right information.

Leverage¶

Even though vectorbt allows setting an arbitrary (even an infinite) cash and ordering as many shares as required by the user, this constellation is associated with some drawbacks: an infinite cash leads to an infinite portfolio value, which makes certain operations on that value impossible, such as the conversion from a target percentage into a target amount of shares; but also, the more cash we have, the smaller are potential contributions of the positions to the portfolio value, thus lowering the magnitude of the portfolio returns. What we need though is to multiply those contributions without inflating the cash balance, which can be effectively done using leverage.

Leverage involves borrowing additional funds to buy shares. In main contrast to shorting, leverage is applied to long positions and borrows cash instead of shares. The underlying mechanism is quite similar though. First, we multiply the available free cash by leverage. Then, we derive the order value and the fraction of it to be borrowed. Finally, we move the borrowed cash to debt while declaring a part of the free cash as the collateral of the operation and moving it to locked_cash. But since the locked cash must be spent to buy a part of the shares, it changes the way the effective leverage can be calculated from debt / locked_cash to debt / locked_cash + 1.

Let's say we have $100 in our margin account and want to buy $200 worth of shares. As we learned previously, we can specify an infinite leverage and vectorbt will derive the effective leverage internally for us:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0
... )
>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=20, 
...     price=10.0,
...     leverage=np.inf
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=20,
    price=10.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=-100.0,
    position=20.0,
    debt=100.0,
    locked_cash=100.0,
    free_cash=0.0
)

We can see that $100 were deducted from our free cash balance and additional $100 were borrowed, thus bringing the effective leverage to 2:

>>> new_account_state.debt / new_account_state.locked_cash + 1
2.0

Buying 10 shares instead would use no leverage since the transaction can be entirely covered by the free cash, even if the leverage is infinity:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=10, 
...     price=10.0,
...     leverage=np.inf
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10,
    price=10.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=0.0,
    position=10.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=0.0
)

Is there a way to use only a subset of our own free cash while borrowing the rest? Yes! The command buy_nb takes the argument leverage_mode of the type LeverageMode, which supports two modes: "lazy" and "eager" leveraging. The first mode is the default one and enables leverage only if the quantity to be bought cannot be fulfilled using own resources. The second mode enables leverage for any quantity and requires the leverage to be set explicitly, that is, an infinite leverage would raise an error.

Let's buy 10 shares with a 3x leverage:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=10, 
...     price=10.0,
...     leverage=3,
...     leverage_mode=vbt.pf_enums.LeverageMode.Eager
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10,
    price=10.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=0.0,
    position=10.0,
    debt=66.66666666666667,
    locked_cash=33.333333333333336,
    free_cash=66.66666666666666
)

We've used only $33.3 from our free cash balance as the collateral to borrow additional $66.6, making a total of $100 that were spent to buy the desired quantity.

Now, how do we repay the debt? When selling, the debt and locked cash balances decrease proportionally to the number of shares sold, which is exactly the same procedure as during (partially-) closing a short position. The main difference is in the calculation of the P&L: we take the total of the released debt and locked cash and subtract them from the cash retrieved from the operation.

First, we'll use a 2x leverage to buy 10 shares for the price of $20 per share:

>>> order_result, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=10, 
...     price=20.0,
...     leverage=2
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10,
    price=20.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=-100.0,
    position=10.0,
    debt=100.0,
    locked_cash=100.0,
    free_cash=0.0
)

Let's sell 5 shares for the price of $5 per share (my condolences):

>>> order_result, new_account_state2 = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     size=5.0, 
...     price=5.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=5.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=-75.0,
    position=5.0,
    debt=50.0,
    locked_cash=50.0,
    free_cash=-25.0
)

We've retrieved $25 from this operation, which have been added to the regular cash balance. The debt and locked cash have been cut in half because the half of the leveraged position has been closed out. The P&L of this operation is the gained cash ($25) minus the released debt ($50) and locked cash ($50), which makes it a loss of $75. By adding this number to the released locked cash, we arrive at the new free cash of -$25 - a change that is propagated to all other assets using the same cash balance and thus preventing them to open or increase their positions.

Another way of calculating the P&L using the equity:

>>> st0 = account_state
>>> st1 = new_account_state2
>>> avg_entry_price = (st1.debt + st1.locked_cash) / abs(st1.position)  # (1)!
>>> new_equity = st1.cash + st1.position * avg_entry_price
>>> new_equity - st0.cash
-75.0

1. The remaining shares are still valued at $20

Now, let's say instead of dipping, the price has jumped to $40 per share (my congratulations!):

>>> order_result, new_account_state2 = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     size=5.0, 
...     price=40.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=40.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=100.0,
    position=5.0,
    debt=50.0,
    locked_cash=50.0,
    free_cash=150.0
)

We've retrieved $200 from this operation, which have been added to the regular cash balance. The debt and locked cash have been cut in half because the half of the leveraged position has been closed out. The P&L of this operation is the gained cash ($200) minus the released debt ($50) and locked cash ($50), which makes it a profit of $100. By adding this number to the released locked cash, we arrive at the new free cash of $150, which can be now used by all other assets sharing the same balance.

Let's close out the remaining position using the same price:

>>> order_result, new_account_state2 = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     size=5.0, 
...     price=40.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=5.0,
    price=40.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=300.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=300.0
)

We've made a profit of $200, which is just the same as we had used our own cash:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=200.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=200.0
... )
>>> _, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     size=10, 
...     price=20.0
... )
>>> _, new_account_state2 = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     size=10.0, 
...     price=40.0
... )
>>> new_account_state2.free_cash - account_state.free_cash
200.0

Symmetry¶

Long and short positions behave symmetrically. For example, let's open two opposite positions using an infinite size and 10x leverage and close them out with a price difference of $5 per share that favors the current position:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0
... )

>>> _, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=account_state, 
...     direction=vbt.pf_enums.Direction.LongOnly,
...     size=np.inf, 
...     price=10.0,
...     leverage=10
... )
>>> _, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=new_account_state, 
...     direction=vbt.pf_enums.Direction.LongOnly,
...     size=np.inf, 
...     price=15.0
... )
>>> vbt.pprint(new_account_state)
AccountState(
    cash=600.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=600.0
)

>>> _, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     direction=vbt.pf_enums.Direction.ShortOnly,
...     size=np.inf, 
...     price=10.0,
...     leverage=10
... )
>>> _, new_account_state = vbt.pf_nb.buy_nb(
...     account_state=new_account_state, 
...     direction=vbt.pf_enums.Direction.ShortOnly,
...     size=np.inf, 
...     price=5.0
... )
>>> vbt.pprint(new_account_state)
AccountState(
    cash=600.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=600.0
)

Reversing¶

Positions in vectorbt can be reversed with a single order. To reverse a position, the direction argument should stay at its default - Direction.Both. Let's start with a position of 10 shares, reverse it to the maximum extent in the short direction, and then reverse it to the maximum extent again but in the opposite (long) direction:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=0.0,
...     position=10.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=0.0
... )
>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=np.inf, 
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=20.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=300.0,
    position=-10.0,
    debt=150.0,
    locked_cash=150.0,
    free_cash=0.0
)

>>> order_result, new_account_state2 = vbt.pf_nb.buy_nb(
...     account_state=new_account_state, 
...     size=np.inf, 
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=20.0,
    price=15.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state2)
AccountState(
    cash=0.0,
    position=10.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=0.0
)

Both operations are symmetric in nature and cancel each other out by a repetitive call, thus ultimately we've arrived at our initial state of the account.

Closing¶

To close out a position and to avoid its reversal, we can either specify the exact size, or the size of infinity and the current direction via the direction argument of the type Direction. For example, if we're in a long position and specified the long-only direction, the position won't be reversed:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=0.0,
...     position=10.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=0.0
... )
>>> order_result, new_account_state = vbt.pf_nb.sell_nb(
...     account_state=account_state, 
...     size=np.inf, 
...     price=15.0, 
...     direction=vbt.pf_enums.Direction.LongOnly
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=150.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=150.0
)

We can also use the commands that are guaranteed to execute within the current position and not open an opposite one: long_sell_nb for long positions and short_buy_nb for short positions. They don't require the argument direction at all, just the size of infinity:

>>> account_state = vbt.pf_enums.AccountState(
...     cash=0.0,
...     position=10.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=0.0
... )
>>> order_result, new_account_state = vbt.pf_nb.long_sell_nb(
...     account_state=account_state, 
...     size=np.inf, 
...     price=15.0
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=10.0,
    price=15.0,
    fees=0.0,
    side=1,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_account_state)
AccountState(
    cash=150.0,
    position=0.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=150.0
)

Pipeline/1¶

Even by using just these two essential commands, we can already build our own backtesting pipeline of arbitrary complexity and flexibility. As said before, a simulation is just a loop that iterates over timestamps. Let's create a simplified pipeline that puts $1 into Bitcoin each time it discovers a Golden Cross entry signal, and sells $1 otherwise. What we want is one number: the final value of the portfolio.

>>> @njit
... def pipeline_1_nb(close, entries, exits, init_cash=100):
...     account_state = vbt.pf_enums.AccountState(  # (1)!
...         cash=float(init_cash),
...         position=0.0,
...         debt=0.0,
...         locked_cash=0.0,
...         free_cash=float(init_cash)
...     )
...     for i in range(close.shape[0]):
...         if entries[i]:
...             _, account_state = vbt.pf_nb.buy_nb(  # (2)!
...                 account_state=account_state,
...                 size=1 / close[i],
...                 price=close[i]
...             )
...         if exits[i]:
...             _, account_state = vbt.pf_nb.sell_nb(
...                 account_state=account_state,
...                 size=1 / close[i],
...                 price=close[i]
...             )
...     return account_state.cash + account_state.position * close[-1]  # (3)!

>>> data = vbt.YFData.pull("BTC-USD", end="2022-01-01")
>>> sma_50 = vbt.talib("SMA").run(data.get("Close"), 50)
>>> sma_200 = vbt.talib("SMA").run(data.get("Close"), 200)
>>> entries = sma_50.real_crossed_above(sma_200)
>>> exits = sma_50.real_crossed_below(sma_200)

>>> pipeline_1_nb(
...     data.get("Close").values, 
...     entries.values, 
...     exits.values
... )
210.71073253390762

1. Initial account state
2. Execute the order and return a new account state
3. Calculate the final portfolio value

We can validate the pipeline using one of the preset simulation methods:

>>> vbt.Portfolio.from_orders(
...     data.get("Close"), 
...     size=entries.astype(int) - exits.astype(int), 
...     size_type="value"
... ).final_value
210.71073253390762

Order execution¶

Using the primitive commands is fun as long as we exactly know the direction of the order and are sure that the provided arguments make sense. But very often, we have to deal with more complex requirements such as target percentages, which change the order direction depending on the current value. In addition, the commands do not validate their arguments; for example, there won't be any error thrown in a case when the user accidentally passes a negative order price. But also, we need a better representation of an order - it's a bad practice of passing all the parameters such as slippage as keyword arguments.

All the checks and other pre-processing procedures are happening in the function execute_order_nb. The first input to this function is an order execution state of type ExecState, which contains the same information as an account state we saw above, but with additional information on the current valuation. The second input is a named tuple of type Order representing an order. The third argument is the price area, which we are also already familiar with.

Order¶

An order in vectorbt is represented by a named tuple. Named tuples are alternatives to data classes in both the Python and Numba world; they are very efficient and lightweight data structures that can be easily constructed and processed. Let's create an instance of an order:

>>> order = vbt.pf_enums.Order()
>>> vbt.pprint(order)
Order(
    size=np.inf,
    price=np.inf,
    size_type=0,
    direction=2,
    fees=0.0,
    fixed_fees=0.0,
    slippage=0.0,
    min_size=np.nan,
    max_size=np.nan,
    size_granularity=np.nan,
    leverage=1.0,
    leverage_mode=0,
    reject_prob=0.0,
    price_area_vio_mode=0,
    allow_partial=True,
    raise_reject=False,
    log=False
)

The tuple allows for attribute access through the dot notation:

>>> order.direction
2

Other than this, it behaves just like any other tuple in Python:

>>> order[3]
2

>>> tuple(order)  # (1)!
(inf,
 inf,
 0,
 2,
 0.0,
 0.0,
 0.0,
 nan,
 nan,
 nan,
 1.0,
 0,
 0.0,
 0,
 True,
 False,
 False)

1. Convert to a regular tuple

One issue that we still have to address when working with Numba are default arguments: although we can construct a new tuple solely with default arguments in Numba as we did above, if we want to override some values, they must be located strictly on the left in that tuple's definition. Otherwise, we must explicitly provide all the default arguments located before them:

>>> @njit
... def create_order_nb():
...     return vbt.pf_enums.Order()  # (1)!

>>> create_order_nb()
Order(size=inf, price=inf, ...)

>>> @njit
... def create_order_nb(size, price):
...     return vbt.pf_enums.Order(size=size, price=price)  # (2)!

>>> create_order_nb(1, 15)
Order(size=1, price=15, ...)

>>> @njit
... def create_order_nb(size, price, direction):
...     return vbt.pf_enums.Order(size=size, price=price, direction=direction)  # (3)!

>>> create_order_nb(1, 15, 2)
Failed in nopython mode pipeline (step: nopython frontend)

1. Using the default values only
2. Overriding the default values of arguments on the left side
3. Overriding the default values of arguments on different positions

Another issue are data types. In the example above where we provided integer size and price, Numba had no issues processing them. But as soon as we create such as order in a loop and one of the arguments is a float instead of an integer provided previously, Numba will throw an error because it cannot unify data types anymore. Thus, we should cast all arguments to their target data types before constructing an order.

Both issues are solved by using the function order_nb:

>>> @njit
... def create_order_nb(size, price, direction):
...     return vbt.pf_nb.order_nb(size=size, price=price, direction=direction)

>>> create_order_nb(1, 15, 2)
Order(size=1.0, price=15.0, ..., direction=2, ...)

Notice how the size and price arguments were automatically cast to floats.

To create an order that closes out the current position, we can conveniently use close_position_nb:

>>> vbt.pf_nb.close_position_nb(15)  # (1)!
Order(size=0.0, price=15.0, size_type=6, ...)

1. Uses size of zero and size type of TargetAmount

Validation¶

Having constructed the order, execute_order_nb will check the arguments of that order for correct data types and values. For example, let's try passing a negative price:

>>> exec_state = vbt.pf_enums.ExecState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0,
...     val_price=15.0,
...     value=100.0
... )
>>> vbt.pf_nb.execute_order_nb(
...     exec_state,
...     vbt.pf_nb.order_nb(price=-15)
... )
ValueError: order.price must be finite and 0 or greater

Price resolution¶

After validating the inputs, execute_order_nb uses them to decide which command to run: buy or sell. But first, it has to do some preprocessing.

Even though vectorbt isn't associated with any particular data schema and can run on tick data just as well as on bar data, it still gives us an option to provide the current candle (price_area) for validation and resolution reasons. In such a case, it will consider the passed order price as a price point located within four price bounds: the opening, high, low, and closing price. Since order execution must happen strictly within those bounds, setting order price to -np.inf and np.inf will replace it by the opening and closing price respectively. Hence, next time, when you see any default price being np.inf, just know that it means the close price

>>> price_area = vbt.pf_enums.PriceArea(
...     open=10,
...     high=14,
...     low=8,
...     close=12
... )
>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(  # (1)!
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=-np.inf),
...     price_area=price_area
... )
>>> order_result.price
10.0

>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(  # (2)!
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=np.inf),
...     price_area=price_area
... )
>>> order_result.price
12.0

>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(  # (3)!
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=np.inf)
... )
>>> order_result.price
nan

1. Price gets replaced by the open price. Order executed.
2. Price gets replaced by the close price (default). Order executed.
3. Price gets replaced by np.nan since the price area is not defined. Order ignored.

Size type conversion¶

Our primitive commands accept only a size in the number of shares, thus we have to convert any size type defined in SizeType to Amount. Different size types require different information for conversion; for example, TargetAmount requires to know the current position size, while Value also requires to know the current valuation price.

Let's execute an order such that the new position has 3 shares:

>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(
...         size=3, 
...         size_type=vbt.pf_enums.SizeType.TargetAmount
...     ),
...     price_area=price_area
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=3.0,
    price=12.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_exec_state)
ExecState(
    cash=64.0,
    position=3.0,
    debt=0.0,
    locked_cash=0.0,
    free_cash=64.0,
    val_price=15.0,
    value=100.0
)

Since we're not in the market, vectorbt used buy_nb to buy 3 shares. If we were in the market with 10 shares, it would have used sell_nb to sell 7 shares.

Direction¶

Function order_nb takes the argument direction for two reasons: resolve the direction of the order based on the sign of the argument size, and decide on whether to reverse the position or just to close out it. When the direction is LongOnly or Both, a positive size means buying and a negative size means selling. When the direction is ShortOnly though, the exact opposite happens: a positive size means selling and a negative size means buying. This is because a positive size is associated with increasing a position, which means buying to increase a long position and selling to increase a short position. For example, if the direction was ShortOnly and the size was a negative infinity, any short position would be closed out and any long position would be enlarged.

Valuation¶

Speaking about the valuation price, it's the latest available price at the time of decision-making, or the price used to calculate the portfolio value. In many simulation methods, valuation price defaults to the order price, but sometimes it makes more sense to use the open or previous close price for the conversion step. The separation of the valuation and order price enables us to introduce a time gap between order placement and its execution. This is important because, in reality, not always an order can be executed right away.

Let's order 100% of the portfolio value:

>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(
...         size=1.0, 
...         size_type=vbt.pf_enums.SizeType.TargetPercent
...     ),
...     price_area=price_area
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=6.666666666666667,
    price=12.0,
    fees=0.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_exec_state)
ExecState(
    cash=20.0,
    position=6.666666666666667,
    debt=0.0,
    locked_cash=0.0,
    free_cash=20.0,
    val_price=15.0,
    value=100.0
)

Why haven't we spent the entire cash? Because to convert the target percentage into the target amount of shares, vectorbt used the provided order execution state with val_price of $15 and value of $100, which produced 100 / 15 = 6.67. The closer the valuation price is to the order price, the closer the calculation result would be to the target requirement.

By default, if we want to place multiple orders within the same bar (for example, in pairs trading), vectorbt wouldn't adjust the portfolio value after each order. This is because it assumes that we made our trading decisions way before order execution and adjusting the value would affect those decisions. But also, an order has only a marginal immediate effect on the value, for example, because of a commission. To force vectorbt to update the valuation price and value itself, we can enable update_value:

>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(
...         size=1.0, 
...         size_type=vbt.pf_enums.SizeType.TargetPercent,
...         fixed_fees=10,
...         slippage=0.01
...     ),
...     price_area=price_area,
...     update_value=True
... )
>>> vbt.pprint(order_result)
OrderResult(
    size=6.666666666666667,
    price=12.120000000000001,
    fees=10.0,
    side=0,
    status=0,
    status_info=-1
)
>>> vbt.pprint(new_exec_state)
ExecState(
    cash=9.199999999999989,
    position=6.666666666666667,
    debt=0.0,
    locked_cash=0.0,
    free_cash=9.199999999999989,
    val_price=12.120000000000001,
    value=90.0
)

Notice how the new valuation price has been set to the close price adjusted with the slippage while the value has decreased by the fixed commission. Any new order placed after this one would use the updated value and thus probably produce a different outcome.

Pipeline/2¶

Let's create another simplified pipeline that orders given a target percentage array. In particular, we'll keep 50% of the portfolio value in shares, and rebalance monthly. We'll calculate the portfolio value based on the open price at the beginning of each bar, and order at the end of each bar (to keep things realistic). Also, we'll fill asset value and portfolio value arrays to later plot the allocation at each bar.

>>> @njit
... def pipeline_2_nb(open, close, target_pct, init_cash=100):
...     asset_value_out = np.empty(close.shape, dtype=np.float_)  # (1)!
...     value_out = np.empty(close.shape, dtype=np.float_)
...     exec_state = vbt.pf_enums.ExecState(  # (2)!
...         cash=float(init_cash),
...         position=0.0,
...         debt=0.0,
...         locked_cash=0.0,
...         free_cash=float(init_cash),
...         val_price=np.nan,
...         value=np.nan
...     )
...
...     for i in range(close.shape[0]):
...         if not np.isnan(target_pct[i]):  # (3)!
...             val_price = open[i]
...             value = exec_state.cash + val_price * exec_state.position  # (4)!
...
...             exec_state = vbt.pf_enums.ExecState(  # (5)!
...                 cash=exec_state.cash,
...                 position=exec_state.position,
...                 debt=exec_state.debt,
...                 locked_cash=exec_state.locked_cash,
...                 free_cash=exec_state.free_cash,
...                 val_price=val_price,
...                 value=value
...             )
...             order = vbt.pf_nb.order_nb(  # (6)!
...                 size=target_pct[i],
...                 price=close[i],
...                 size_type=vbt.pf_enums.SizeType.TargetPercent
...             )
...             _, exec_state = vbt.pf_nb.execute_order_nb(  # (7)!
...                 exec_state=exec_state,
...                 order=order
...             )
...
...         asset_value_out[i] = exec_state.position * close[i]  # (8)!
...         value_out[i] = exec_state.cash + exec_state.position * close[i]
...         
...     return asset_value_out, value_out

1. Create two empty arrays with a floating data type. Remember that creating an array with np.empty will produce an array with uninitialized (garbage) values that you should override.
2. Our initial order execution state
3. There is no need to run order execution when target percentage is np.nan (= do not rebalance)
4. Calculate the portfolio value at the beginning of the bar (= valuation)
5. Create a new existing order execution state after the valuation
6. Create a new order tuple using the close price as order price
7. Execute the order and return the order result and the new execution state
8. Fill the arrays (you should fill each single element!)

Let's run the pipeline on our Bitcoin data:

>>> symbol_wrapper = data.get_symbol_wrapper()  # (1)!
>>> target_pct = symbol_wrapper.fill()
>>> target_pct.vbt.set(0.5, every="M", inplace=True)  # (2)!

>>> asset_value, value = pipeline_2_nb(
...     data.get("Open").values, 
...     data.get("Close").values, 
...     target_pct.values
... )
>>> asset_value = symbol_wrapper.wrap(asset_value)  # (3)!
>>> value = symbol_wrapper.wrap(value)
>>> allocations = (asset_value / value).rename(None)  # (4)!
>>> allocations.vbt.scatterplot(trace_kwargs=dict(
...     marker=dict(
...         color=allocations, 
...         colorscale="Temps", 
...         size=3,
...         cmin=0.3,
...         cmid=0.5,
...         cmax=0.7
...     )
... )).show()

1. We cannot use data.wrapper because it contains OHLC as columns. What we need is a wrapper that has symbols as columns, to fill the array with target percentages.
2. Fill the array with NaNs and set all data points at the beginning of each month to 0.5.
3. Use the same wrapper to convert the NumPy array into Pandas Series
4. Divide the asset value by the portfolio value to derive the allocation

We see that allocations are being regularly pulled back to the target level of 50%.

Let's validate the pipeline using Portfolio.from_orders:

>>> pf = vbt.Portfolio.from_orders(
...     data, 
...     size=target_pct, 
...     size_type="targetpercent"
... )
>>> pf.allocations.vbt.scatterplot(trace_kwargs=dict(
...     marker=dict(
...         color=allocations, 
...         colorscale="Temps", 
...         size=3,
...         cmin=0.3,
...         cmid=0.5,
...         cmax=0.7
...     )
... )).show()

One of the biggest advantages of using vectorbt is that you can run your minimalistic trading environment in any Python function, even inside objective functions of machine learning models! There is no need to trigger the entire backtesting pipeline as a script or any other complex process like most backtesting frameworks force us to do

Order processing¶

Order execution takes an order instruction and translates it into a buy or sell operation. The responsibility of the user is to do something with the returned order execution state and result; mostly, we want to post-process and append each successful order to some list for later analysis - that's where order and log records come into play. Furthermore, we may want to raise an error if an order has been rejected and a certain flag in the requirements is present. All of this is ensured by process_order_nb.

Order records¶

Order records is a structured NumPy array of the data type order_dt containing information on each successful order. Each order in this array is assumed to be completed, that is, you should view an order as a trade in the vectorbt's world. Since we're dealing with Numba, we cannot and should not use lists and other inefficient data structures for storing such complex information. Given that orders have fields with variable data types, the best data structure is a record array, which is a regular NumPy array with a complex data type and that behaves similarly to a Pandas DataFrame.

Since any NumPy array is a non-appendable structure, we should initialize an empty array of a sufficient size, and gradually fill it with new information. For this, we need a counter - a simple integer - that points to an index of the next record to be written.

Let's create an array with two order records and a counter:

>>> order_records = np.empty(2, dtype=vbt.pf_enums.order_dt)
>>> order_count = 0

We shouldn't access this array just yet because it contains memory garbage, thus it requires the user to manually set all the values in the array, and should be used with caution.

>>> order_records
array([(4585679916398730403, ..., 4583100142070297783),
       (4582795628349012822, ..., 4576866499094039639)],
      dtype={'names':['id','col','idx','size','price','fees','side'], ...})

Let's execute an order using execute_order_nb at the 678th bar, and fill the first record in the array:

>>> exec_state = vbt.pf_enums.ExecState(
...     cash=100.0,
...     position=0.0,
...     debt=0.0,
...     locked_cash=0.0,
...     free_cash=100.0,
...     val_price=15.0,
...     value=100.0
... )
>>> order_result, new_exec_state = vbt.pf_nb.execute_order_nb(
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=15.0)
... )
>>> if order_result.status == vbt.pf_enums.OrderStatus.Filled:  # (1)!
...     order_records["id"][order_count] = order_count  # (2)!
...     order_records["col"][order_count] = 0
...     order_records["idx"][order_count] = 678  # (3)!
...     order_records["size"][order_count] = order_result.size
...     order_records["price"][order_count] = order_result.price
...     order_records["fees"][order_count] = order_result.fees
...     order_records["side"][order_count] = order_result.side
...     order_count += 1

>>> order_records[0]
(0, 0, 678, 6.66666667, 15., 0., 0)

>>> order_count
1

1. Check that the order has been filled
2. Order ids start with 0 and follow the counter
3. Index of the current bar

At the next bar, we'll reverse the position and fill the second record:

>>> order_result, new_exec_state2 = vbt.pf_nb.execute_order_nb(
...     exec_state=new_exec_state,
...     order=vbt.pf_nb.order_nb(size=-np.inf, price=16.0)
... )
>>> if order_result.status == vbt.pf_enums.OrderStatus.Filled:
...     order_records["id"][order_count] = order_count  # (1)!
...     order_records["col"][order_count] = 0
...     order_records["idx"][order_count] = 679
...     order_records["size"][order_count] = order_result.size
...     order_records["price"][order_count] = order_result.price
...     order_records["fees"][order_count] = order_result.fees
...     order_records["side"][order_count] = order_result.side
...     order_count += 1

>>> order_records[1]
(1, 0, 679, 13.33333333, 16., 0., 1)

>>> order_count
2

1. Don't forget to increment the order id

Here are the order records that we've populated:

>>> order_records
array([(0, 0, 678,  6.66666667, 15., 0., 0),
       (1, 0, 679, 13.33333333, 16., 0., 1)],
      dtype={'names':['id','col','idx','size','price','fees','side'], ...})

But instead of setting each of these records manually, we can use process_order_nb to do it for us! We just need to do one little adjustment: both the order records and the counter must be provided per column since vectorbt primarily works on multi-columnar data. This means that the order records array must become a two-dimensional array and the counter constant must become a one-dimensional array (both with only one column in our example):

>>> order_records = np.empty((2, 1), dtype=vbt.pf_enums.order_dt)
>>> order_counts = np.full(1, 0, dtype=np.int_)

>>> order_result1, new_exec_state1 = vbt.pf_nb.process_order_nb(
...     0, 0, 678,  # (1)!
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=15.0),
...     order_records=order_records,
...     order_counts=order_counts
... )
>>> order_result2, new_exec_state2 = vbt.pf_nb.process_order_nb(
...     0, 0, 679,
...     exec_state=new_exec_state,
...     order=vbt.pf_nb.order_nb(size=-np.inf, price=16.0),
...     order_records=order_records,
...     order_counts=order_counts
... )

>>> order_records
array([(0, 0, 678,  6.66666667, 15., 0., 0),
       (1, 0, 679, 13.33333333, 16., 0., 1)],
      dtype={'names':['id','col','idx','size','price','fees','side'], ...})

>>> order_counts
array([2])

1. Current group, column, and index (row)

Such filled order records will become the backbone of the post-analysis phase.

Log records¶

Log records have the data type log_dt and are similar to order records, but with a few key differences: they are saved irrespective of whether the order has been filled, and they also contain information on the current execution state, the order request, and the new execution state. This way, we can completely and post-factum track down issues related to order processing.

>>> order_records = np.empty((2, 1), dtype=vbt.pf_enums.order_dt)
>>> order_counts = np.full(1, 0, dtype=np.int_)
>>> log_records = np.empty((2, 1), dtype=vbt.pf_enums.log_dt)
>>> log_counts = np.full(1, 0, dtype=np.int_)

>>> order_result1, new_exec_state1 = vbt.pf_nb.process_order_nb(
...     0, 0, 678,
...     exec_state=exec_state,
...     order=vbt.pf_nb.order_nb(size=np.inf, price=15.0, log=True),  # (1)!
...     order_records=order_records,
...     order_counts=order_counts,
...     log_records=log_records,
...     log_counts=log_counts
... )
>>> order_result2, new_exec_state2 = vbt.pf_nb.process_order_nb(
...     0, 0, 679,
...     exec_state=new_exec_state,
...     order=vbt.pf_nb.order_nb(size=-np.inf, price=16.0, log=True),
...     order_records=order_records,
...     order_counts=order_counts,
...     log_records=log_records,
...     log_counts=log_counts
... )

>>> log_records
array([[(0, 0, 0, 678, ..., 0., 15., 100., 0)],
       [(1, 0, 0, 679, ..., 0., 15., 100., 1)]],
      dtype={'names':['id','group',...,'res_status_info','order_id'], ...})

1. Logging of each order must be explicitly enabled

Pipeline/3¶

Let's extend the last pipeline to independently process an arbitrary number of columns, and gradually fill order records. This way, we can backtest multiple parameter combinations by taking advantage of multidimensionality!

>>> @njit
... def pipeline_3_nb(open, close, target_pct, init_cash=100):
...     order_records = np.empty(close.shape, dtype=vbt.pf_enums.order_dt)  # (1)!
...     order_counts = np.full(close.shape[1], 0, dtype=np.int_)
...
...     for col in range(close.shape[1]):  # (2)!
...         exec_state = vbt.pf_enums.ExecState(
...             cash=float(init_cash),
...             position=0.0,
...             debt=0.0,
...             locked_cash=0.0,
...             free_cash=float(init_cash),
...             val_price=np.nan,
...             value=np.nan
...         )
...
...         for i in range(close.shape[0]):
...             if not np.isnan(target_pct[i, col]):  # (3)!
...                 val_price = open[i, col]
...                 value = exec_state.cash + val_price * exec_state.position
...
...                 exec_state = vbt.pf_enums.ExecState(
...                     cash=exec_state.cash,
...                     position=exec_state.position,
...                     debt=exec_state.debt,
...                     locked_cash=exec_state.locked_cash,
...                     free_cash=exec_state.free_cash,
...                     val_price=val_price,
...                     value=value
...                 )
...                 order = vbt.pf_nb.order_nb(
...                     size=target_pct[i, col],
...                     price=close[i, col],
...                     size_type=vbt.pf_enums.SizeType.TargetPercent
...                 )
...                 _, exec_state = vbt.pf_nb.process_order_nb(
...                     col, col, i,  # (4)!
...                     exec_state=exec_state,
...                     order=order,
...                     order_records=order_records,
...                     order_counts=order_counts
...                 )
...         
...     return vbt.nb.repartition_nb(order_records, order_counts)  # (4)!