Creating a trading system from scratch

How many lines of code you are comfortable with


  • Total voters
    61
Code:
#Import relevant libraries

import pandas as pd

import numpy as np

import pandas_datareader.data as web

import matplotlib.pyplot as plt

import scipy.optimize as spo

import pyfolio as pf

import math

#Fetch data from yahoo and save under DataFrame named 'data'

stock = ['ERIS.NS', 'CROMPTON.NS', 'GNFC.NS', 'GRAPHITE.NS','BEPL.NS','KMCSHIL.BO']

data = web.DataReader(stock,data_source="yahoo",start='07/01/2017',end='today')['Adj Close']

#Arrange the data in ascending order

data=data.iloc[::-1]

print(data.round(2))





#Compute stock returns and print the returns in percentage format

stock_ret = data.pct_change()*252

print (stock_ret.round(4)*100)

mybenchmark_return=web.DataReader('^NSEI',data_source="yahoo",start='07/01/2016',end='today')

#Calculate mean returns and covariances of all four the stocks

mean_returns = stock_ret.mean()

cov_matrix = stock_ret.cov()

print (mean_returns)

print (cov_matrix)







#Set the number of iterations to 10000 and define an array to hold the simulation results; initially set to all zeros

num_iterations = 1000000

simulation_res = np.zeros((4+len(stock)-1,num_iterations))







for i in range(num_iterations):

#Select random weights and normalize to set the sum to 1

        weights = np.array(np.random.random(6))

        weights /= np.sum(weights)

#Calculate the return and standard deviation for every step

        portfolio_return = np.sum(mean_returns * weights)

        portfolio_std_dev = np.sqrt(np.dot(weights.T,np.dot(cov_matrix, weights)))

#Store all the results in a defined array

        simulation_res[0,i] = portfolio_return

        simulation_res[1,i] = portfolio_std_dev

#Calculate Sharpe ratio and store it in the array

        simulation_res[2,i] = simulation_res[0,i] / simulation_res[1,i]

#Save the weights in the array

        for j in range(len(weights)):

                simulation_res[j+3,i] = weights[j]







sim_frame = pd.DataFrame(simulation_res.T,columns=['ret','stdev','sharpe',stock[0],stock[1],stock[2],stock[3],stock[4],

                                                   stock[5]])

print (sim_frame.head (5))

print (sim_frame.tail (5))





#Spot the position of the portfolio with highest Sharpe Ratio

max_sharpe = sim_frame.iloc[sim_frame['sharpe'].idxmax()]

#Spot the position of the portfolio with minimum Standard Deviation

min_std = sim_frame.iloc[sim_frame['stdev'].idxmin()]

print ("The portfolio for max Sharpe Ratio:\n", max_sharpe.round(3)*100)

print ("The portfolio for min risk:\n", min_std.round(3)*100)



#Create a scatter plot coloured by various Sharpe Ratios with standard deviation on the x-axis and returns on the y-axis

plt.scatter(sim_frame.stdev,sim_frame.ret,c=sim_frame.sharpe,cmap='RdYlBu')

plt.xlabel('Standard Deviation')

plt.ylabel('Returns')

plt.ylim(0,.003)

plt.xlim(0.0075,0.012)

#Plot a red star to highlight position of the portfolio with highest Sharpe Ratio

plt.scatter(max_sharpe[1],max_sharpe[0],marker=(5,1,0),color='r',s=600)

#Plot a blue star to highlight position of the portfolio with minimum Variance

plt.scatter(min_std[1],min_std[0],marker=(5,1,0),color='b',s=600)

plt.show()
i have this strategy for portfolio construction can you tell me how to do pyfolio for yahoo finance
 

ncube

Well-Known Member
Excellent observation regarding price sensitivity. :up: How did you discovered it?
Its a corner case and I have dealt with it by excluding the specific cases from the results so that the returns are much more rational. I have not included it in my original notebook but have described it in threads on how to make these adjustments.
Also, how did you generated these charts (what program do you use)
@UberMachine , Even though the backtest results are excellent, if I introduce a small slippage of even 0.1% at open price and close price the strategy expectancy turns negative. Also if you check the average returns its is about 1% and this is for 5 stocks, so if this return is reduced to zero due to slippage and brokerages it will turn the overall returns negative and introduce high volatility in the portfolio. One way to improve the results is to ensure the trade executions happen at a good price and the other is to have stop loss placed at about 3% (I have not optimized just random value)

I use python and these charts are coded using matplotlib/seaborn plotting library
 

ncube

Well-Known Member
Code:
#Import relevant libraries

import pandas as pd

import numpy as np

import pandas_datareader.data as web

import matplotlib.pyplot as plt

import scipy.optimize as spo

import pyfolio as pf

import math

#Fetch data from yahoo and save under DataFrame named 'data'

stock = ['ERIS.NS', 'CROMPTON.NS', 'GNFC.NS', 'GRAPHITE.NS','BEPL.NS','KMCSHIL.BO']

data = web.DataReader(stock,data_source="yahoo",start='07/01/2017',end='today')['Adj Close']

#Arrange the data in ascending order

data=data.iloc[::-1]

print(data.round(2))





#Compute stock returns and print the returns in percentage format

stock_ret = data.pct_change()*252

print (stock_ret.round(4)*100)

mybenchmark_return=web.DataReader('^NSEI',data_source="yahoo",start='07/01/2016',end='today')

#Calculate mean returns and covariances of all four the stocks

mean_returns = stock_ret.mean()

cov_matrix = stock_ret.cov()

print (mean_returns)

print (cov_matrix)







#Set the number of iterations to 10000 and define an array to hold the simulation results; initially set to all zeros

num_iterations = 1000000

simulation_res = np.zeros((4+len(stock)-1,num_iterations))







for i in range(num_iterations):

#Select random weights and normalize to set the sum to 1

        weights = np.array(np.random.random(6))

        weights /= np.sum(weights)

#Calculate the return and standard deviation for every step

        portfolio_return = np.sum(mean_returns * weights)

        portfolio_std_dev = np.sqrt(np.dot(weights.T,np.dot(cov_matrix, weights)))

#Store all the results in a defined array

        simulation_res[0,i] = portfolio_return

        simulation_res[1,i] = portfolio_std_dev

#Calculate Sharpe ratio and store it in the array

        simulation_res[2,i] = simulation_res[0,i] / simulation_res[1,i]

#Save the weights in the array

        for j in range(len(weights)):

                simulation_res[j+3,i] = weights[j]







sim_frame = pd.DataFrame(simulation_res.T,columns=['ret','stdev','sharpe',stock[0],stock[1],stock[2],stock[3],stock[4],

                                                   stock[5]])

print (sim_frame.head (5))

print (sim_frame.tail (5))





#Spot the position of the portfolio with highest Sharpe Ratio

max_sharpe = sim_frame.iloc[sim_frame['sharpe'].idxmax()]

#Spot the position of the portfolio with minimum Standard Deviation

min_std = sim_frame.iloc[sim_frame['stdev'].idxmin()]

print ("The portfolio for max Sharpe Ratio:\n", max_sharpe.round(3)*100)

print ("The portfolio for min risk:\n", min_std.round(3)*100)



#Create a scatter plot coloured by various Sharpe Ratios with standard deviation on the x-axis and returns on the y-axis

plt.scatter(sim_frame.stdev,sim_frame.ret,c=sim_frame.sharpe,cmap='RdYlBu')

plt.xlabel('Standard Deviation')

plt.ylabel('Returns')

plt.ylim(0,.003)

plt.xlim(0.0075,0.012)

#Plot a red star to highlight position of the portfolio with highest Sharpe Ratio

plt.scatter(max_sharpe[1],max_sharpe[0],marker=(5,1,0),color='r',s=600)

#Plot a blue star to highlight position of the portfolio with minimum Variance

plt.scatter(min_std[1],min_std[0],marker=(5,1,0),color='b',s=600)

plt.show()
i have this strategy for portfolio construction can you tell me how to do pyfolio for yahoo finance
Use this code to generate the NSE pyfolio tearsheet, Enjoy tweeking the code..!! :)

Code:
#Import relevant libraries

import pandas as pd

from pandas_datareader import data as pdr
import fix_yahoo_finance

import matplotlib.pyplot as plt

import pyfolio as pf
%matplotlib inline
# silence warnings
import warnings
warnings.filterwarnings('ignore')


#Fetch data from yahoo and save under DataFrame named 'data'
data = pd.DataFrame()

#stock = ['ERIS.NS', 'CROMPTON.NS', 'GNFC.NS', 'GRAPHITE.NS','BEPL.NS','KMCSHIL.BO','^NSEI']
stock = ['^NSEI']

for i in stock:
    data[i] = pdr.get_data_yahoo(i,start="2000-01-01")['Adj Close']
   

#print(data.tail())
#Compute stock returns and print the returns in percentage format
stock_ret = data.pct_change().fillna(0)
#print(stock_ret.tail())

pf.create_returns_tear_sheet(stock_ret['^NSEI'])
output:

1538375132743.png


1538375161987.png


1538375255828.png
 
Use this code to generate the NSE pyfolio tearsheet, Enjoy tweeking the code..!! :)

Code:
#Import relevant libraries

import pandas as pd

from pandas_datareader import data as pdr
import fix_yahoo_finance

import matplotlib.pyplot as plt

import pyfolio as pf
%matplotlib inline
# silence warnings
import warnings
warnings.filterwarnings('ignore')


#Fetch data from yahoo and save under DataFrame named 'data'
data = pd.DataFrame()

#stock = ['ERIS.NS', 'CROMPTON.NS', 'GNFC.NS', 'GRAPHITE.NS','BEPL.NS','KMCSHIL.BO','^NSEI']
stock = ['^NSEI']

for i in stock:
    data[i] = pdr.get_data_yahoo(i,start="2000-01-01")['Adj Close']
  

#print(data.tail())
#Compute stock returns and print the returns in percentage format
stock_ret = data.pct_change().fillna(0)
#print(stock_ret.tail())

pf.create_returns_tear_sheet(stock_ret['^NSEI'])
output:

View attachment 28963

View attachment 28964

View attachment 28965
Thanks alot it work...!!!:up::up::up:
 

UberMachine

Well-Known Member
Actually this error due to I checked otherthan nifty 50 only now in system...we checked it in backtests only before :).
so something error have in system from start :)
@UberMachine , Even though the backtest results are excellent, if I introduce a small slippage of even 0.1% at open price and close price the strategy expectancy turns negative. Also if you check the average returns its is about 1% and this is for 5 stocks, so if this return is reduced to zero due to slippage and brokerages it will turn the overall returns negative and introduce high volatility in the portfolio. One way to improve the results is to ensure the trade executions happen at a good price and the other is to have stop loss placed at about 3% (I have not optimized just random value)

I use python and these charts are coded using matplotlib/seaborn plotting library
i know i m pushing it but when i try it for full tear sheet it gives me this error View attachment 28968
Full tear sheet requires fama-french returns, which it tries to fetch from a website.
We need to null this variable to get this working.
 

UberMachine

Well-Known Member
@UberMachine , Even though the backtest results are excellent, if I introduce a small slippage of even 0.1% at open price and close price the strategy expectancy turns negative. Also if you check the average returns its is about 1% and this is for 5 stocks, so if this return is reduced to zero due to slippage and brokerages it will turn the overall returns negative and introduce high volatility in the portfolio. One way to improve the results is to ensure the trade executions happen at a good price and the other is to have stop loss placed at about 3% (I have not optimized just random value)

I use python and these charts are coded using matplotlib/seaborn plotting library
I have adjusted for slippage and it is quite sensitive to open price. Bang on. stop loss is at 3% and I have expanded the universe of stocks and covered corner cases to improve the results. One specific case open=low takes away 40% of the profit.
I have covered a few of them in the posts.
Also, could you share your results. I just wanted to check whether your recommendations and mine match off.
If you could share code base, it would be great since I have installed backtrader
 

ncube

Well-Known Member
I have adjusted for slippage and it is quite sensitive to open price. Bang on. stop loss is at 3% and I have expanded the universe of stocks and covered corner cases to improve the results. One specific case open=low takes away 40% of the profit.
I have covered a few of them in the posts.
Also, could you share your results. I just wanted to check whether your recommendations and mine match off.
If you could share code base, it would be great since I have installed backtrader
For the backtest I just used nifty50 stocks as universe, the backtest report I had posted in earlier post does not consider the slippage/stop-loss. The daily stock recommendation is just the top 5 stocks which have gained the most the previous day. I have calculated the daily return as just the difference between close & open price.

Todays recommendation was : ['AXISBANK', 'WIPRO', 'ITC', 'HDFCBANK', 'HDFC']

I am really sorry I will not be able to share my complete code base in an open forum. Hope you can understand. Though I am using the backtrader concept, over the months I have fully revamped it as per my needs, mainly support for pandas and have introduced many machine learning algorithms based on my research and proprietary libraries. Also it will not run directly on the backtrader current version. However I will try to share code snippets wherever possible.
 

ncube

Well-Known Member
Full tear sheet requires fama-french returns, which it tries to fetch from a website.
We need to null this variable to get this working.
You may try by passing the benchmark_ret variable, as @UberMachine mentioned it will through few errors but most of the charts are displayed.
example:
pf.create_full_tear_sheet(stock_ret['ERIS.NS'],benchmark_rets=stock_ret['^NSEI'])
 

Similar threads