Anyone use Python to analyze their finances?
Anyone use Python to analyze their finances?
I am in the midst of a 4-day "Python for Data Science" bootcamp where we are learning how to do data analysis through python scripts. It is very interesting and practical and makes me wonder if anyone has put this to use for finances....
I would love to be able write a script that would:
1. Log into misc finance websites, grab pieces of data (balance.. etc)
2. Take that data and compile it into a single document (spreadsheet or whatever)
3. Analyze the data and/or make predictive calculations
Has anyone done something similar using Python? If so, what have you done and how have you done it?
Just curious, I am sure there is a lot I can do with this
I would love to be able write a script that would:
1. Log into misc finance websites, grab pieces of data (balance.. etc)
2. Take that data and compile it into a single document (spreadsheet or whatever)
3. Analyze the data and/or make predictive calculations
Has anyone done something similar using Python? If so, what have you done and how have you done it?
Just curious, I am sure there is a lot I can do with this
-
- Posts: 22
- Joined: Tue Dec 30, 2014 2:26 pm
Re: Anyone use Python to analyze their finances?
I've used python a fair bit for running financial simulations and projections (#3 in your list) and for data cleaning and compiling (#2 in your list). I know enough to know writing the code to log into websites and grab data is possible, but it'd be a lot more work than either of the other two points so, thus far, I mostly export data out of existing aggregation websites like Mint, and have my own python scripts to parse the outputs.
Single biggest thing I've enjoyed about working with code instead of excel is how much easier it is to model uncertainly. For my own net worth projects I have a 50% confidence interval and a 90% confidence interval going forward (based on the shiller historical monthly stock market dataset).
Single biggest thing I've enjoyed about working with code instead of excel is how much easier it is to model uncertainly. For my own net worth projects I have a 50% confidence interval and a 90% confidence interval going forward (based on the shiller historical monthly stock market dataset).
Re: Anyone use Python to analyze their finances?
Good stuff. I use SQL at work all the time but that's the extent of my "programming" knowledge. I am looking forward to putting Python to some good use...soda_bandit wrote:I've used python a fair bit for running financial simulations and projections (#3 in your list) and for data cleaning and compiling (#2 in your list). I know enough to know writing the code to log into websites and grab data is possible, but it'd be a lot more work than either of the other two points so, thus far, I mostly export data out of existing aggregation websites like Mint, and have my own python scripts to parse the outputs.
Single biggest thing I've enjoyed about working with code instead of excel is how much easier it is to model uncertainly. For my own net worth projects I have a 50% confidence interval and a 90% confidence interval going forward (based on the shiller historical monthly stock market dataset).
Re: Anyone use Python to analyze their finances?
I have done OO for a long time in professional life - java/c++. Last year I dabbled with python for a financial analytics project and was very impressed by the numerous utilities/frameworks available - few notable ones: scipy/numpy, matplotlib.
However I have yet to do full fledged financial analysis as I had to abandon the project due to time constraints. If one of you can share some pointers to a working project, it would be good.
However I have yet to do full fledged financial analysis as I had to abandon the project due to time constraints. If one of you can share some pointers to a working project, it would be good.
Re: Anyone use Python to analyze their finances?
I use Python a lot. I'm not sure it is really needed to "analyse" finances, though A pretty simple spreadsheet and occasional use of online tools like cfiresim are good enough for 99.9% of people, I reckon.
That said...
Pandas is a great library for data analysis: http://pandas.pydata.org/pandas-docs/stable/index.html
I use it extensively in my own analysis.
It has a related project, pandas-datareader https://pandas-datareader.readthedocs.io/en/latest/ that provides simple & easy access to Yahoo Finance, Google Finance, FRED, Fama/French, the World Bank, OECD, the TSP fund data, Oanda historical currency exchange rates, and a few other things. The bit on "caching queries" -- which actually uses the requests_cache library -- is also useful.
I've mostly used seaborn https://seaborn.pydata.org/ for visualising things. Though I'm thinking about trying out plotly https://plot.ly/python/ since it offers interactivity, which seems like a near-mandatory thing nowadays.
That said...
Pandas is a great library for data analysis: http://pandas.pydata.org/pandas-docs/stable/index.html
I use it extensively in my own analysis.
It has a related project, pandas-datareader https://pandas-datareader.readthedocs.io/en/latest/ that provides simple & easy access to Yahoo Finance, Google Finance, FRED, Fama/French, the World Bank, OECD, the TSP fund data, Oanda historical currency exchange rates, and a few other things. The bit on "caching queries" -- which actually uses the requests_cache library -- is also useful.
I've mostly used seaborn https://seaborn.pydata.org/ for visualising things. Though I'm thinking about trying out plotly https://plot.ly/python/ since it offers interactivity, which seems like a near-mandatory thing nowadays.
- happenstance
- Posts: 306
- Joined: Sun Jul 26, 2015 11:24 am
- Location: NYC
Re: Anyone use Python to analyze their finances?
Yes, I use Python extensively for my financial understanding. I keep my data in Google Sheets and then connect it to a Python data platform for running analysis. More detailed, I have:
1) A Google Spreadsheet that contains every transaction, with one fund per tab. I just manually enter my transactions whenever I make a trade, and I add rows for dividends and capital gains when I receive my monthly account statements. This sheet also has a rollup tab to compute cost basis, number of outstanding shares, dividends, etc. for all my funds.
2) Another Google Spreadsheet where I track my overall allocation, rebalancing bands, and monthly investment calculator. This imports the transactions rollup sheet to get current positions and cost basis.
3) Jupyter/Pandas notebooks that use Google APIs to connect to my spreadsheets. Jupyter is a web-based interface to something that resembles Mathematica/Matlab. Pandas is a powerful Python data processing library that was originally created at AQR, so it's well-suited for handling timeseries financial data. These notebooks allow me to import my entire transaction history into a single "data frame", so that I can look at various things:
The initial setup to get all my transactions organized was a bit of work, though I'm lucky that I only started investing a few years ago. Learning Pandas and the world of other PyData projects (Seaborn, PyPlot, Bokeh) has been fun. But I am a software engineer by trade, so this does come rather easily for me.
1) A Google Spreadsheet that contains every transaction, with one fund per tab. I just manually enter my transactions whenever I make a trade, and I add rows for dividends and capital gains when I receive my monthly account statements. This sheet also has a rollup tab to compute cost basis, number of outstanding shares, dividends, etc. for all my funds.
2) Another Google Spreadsheet where I track my overall allocation, rebalancing bands, and monthly investment calculator. This imports the transactions rollup sheet to get current positions and cost basis.
3) Jupyter/Pandas notebooks that use Google APIs to connect to my spreadsheets. Jupyter is a web-based interface to something that resembles Mathematica/Matlab. Pandas is a powerful Python data processing library that was originally created at AQR, so it's well-suited for handling timeseries financial data. These notebooks allow me to import my entire transaction history into a single "data frame", so that I can look at various things:
- Rate of return on a monthly basis. With full transaction data, I can calculate this using the modified Dietz method, which is important for me since I'm in the accumulation stage and my inflows do cause other calculation methods to get biased.
- Rate of return of my portfolio against any benchmark I can import. Pandas can import CSV feeds easily, so it's trivial to compare against any arbitrary benchmark returns.
- Fama-French factor regressions against my precise portfolio, over time. Using the pandas.datareader module, you can import French's datasets, and then use the scikit Python module to run a linear regression. I tilt, so this is a rather significant tool for my portfolio management needs.
- Compound annual growth rate over any time window.
- Investment cash flows on a monthly basis.
- In my transactions notebook, I also record NAV when I buy ETFs. This allows me to calculate how much I'm losing in trading costs against NAV.
The initial setup to get all my transactions organized was a bit of work, though I'm lucky that I only started investing a few years ago. Learning Pandas and the world of other PyData projects (Seaborn, PyPlot, Bokeh) has been fun. But I am a software engineer by trade, so this does come rather easily for me.
-
- Posts: 50
- Joined: Tue Feb 18, 2014 1:26 pm
Re: Anyone use Python to analyze their finances?
I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.
When I combine both of these I can create very cool tools. I can keep track of all my transactions and display the transactions in very cool ways such as the sankey method as seen below.
It also automatically downloads all the investment price data using pandas datareader and calculates Betas and Alphas for my portfolio. You can see the image below. I also used to calculate the efficient frontier and display that information also. But I did not find that as useful for personal investing. Since I keep track of my transactions and portfolio prices, I can calculate my IRR and other items. I have a dashboard page that shows all that information in detail.
Finally, I can create different goals and create projections. The one below if for college savings.
I think python is a really easy tool to be able to accomplish all three items in the OP.
FamilyGuy
When I combine both of these I can create very cool tools. I can keep track of all my transactions and display the transactions in very cool ways such as the sankey method as seen below.
It also automatically downloads all the investment price data using pandas datareader and calculates Betas and Alphas for my portfolio. You can see the image below. I also used to calculate the efficient frontier and display that information also. But I did not find that as useful for personal investing. Since I keep track of my transactions and portfolio prices, I can calculate my IRR and other items. I have a dashboard page that shows all that information in detail.
Finally, I can create different goals and create projections. The one below if for college savings.
I think python is a really easy tool to be able to accomplish all three items in the OP.
FamilyGuy
-
- Posts: 958
- Joined: Wed Mar 11, 2009 4:10 pm
- Location: Philadelphia
Re: Anyone use Python to analyze their finances?
I've also been looking into Python, but for me it's been about building my career skill set. I don't know if I'll commit to learning Python in depth or not, but I just thought I'd share a link comparing Python to R. It seems that Python is used much more in general, but R is probably better for stat analysis tools. I have no horse in the race, so I will probably end up learning some of both. (My main language is SAS currently). Hope you find this useful:
https://www.datacamp.com/community/tuto ... gs.=DoSG9Y
https://www.datacamp.com/community/tuto ... gs.=DoSG9Y
After one has played a vast quantity of notes and more notes, it is simplicity that emerges as the crowning reward of art. Chopin
Re: Anyone use Python to analyze their finances?
Interesting. I have been going through Codecademy and 'Learn Python the Hard Way' just to get familiar with the language over the past few weeks. I don't really have any strong background in writing code, mainly VBA to facilitate spreadsheets.
Are there any books that you would recommend on Python that you have found to be invaluable? Any other resources that you'd recommend?
Thanks!
Yeti
Are there any books that you would recommend on Python that you have found to be invaluable? Any other resources that you'd recommend?
Thanks!
Yeti
- JupiterJones
- Posts: 3624
- Joined: Tue Aug 24, 2010 3:25 pm
- Location: Nashville, TN
Re: Anyone use Python to analyze their finances?
I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).
Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
"Stay on target! Stay on target!"
Re: Anyone use Python to analyze their finances?
So far, google sheets does everything I need. I use one set of sheets to track expenses and another to track portfolio holdings (for rebalancing).
I make heavy use of the GOOGLEFINANCE() function.
I do wish it was a bit more mobile friendly, however. But I think that's just the nature of spreadsheets.
I make heavy use of the GOOGLEFINANCE() function.
I do wish it was a bit more mobile friendly, however. But I think that's just the nature of spreadsheets.
"Buy-and-hold, long-term, all-market-index strategies, implemented at rock-bottom cost, are the surest of all routes to the accumulation of wealth" - John C. Bogle
Re: Anyone use Python to analyze their finances?
+1 to this. I see no big reason to use Python for personal finances, vs. any other general-purpose programming language, and there would be advantages to using a math/statistics-specialized language like R. But the real point is that doing custom software development for personal finances seems like major overkill (even for someone who is a professional or hobbyist programmer and I can't imagine it making sense for someone who is not).JupiterJones wrote:I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).
Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
If you want sexier charts than Excel / Google Sheets can do, maybe consider Tableau, see e.g. https://www.interworks.com/blog/skenned ... -too-often
-
- Posts: 50
- Joined: Tue Feb 18, 2014 1:26 pm
Re: Anyone use Python to analyze their finances?
freebeer wrote:+1 to this. I see no big reason to use Python for personal finances, vs. any other general-purpose programming language, and there would be advantages to using a math/statistics-specialized language like R. But the real point is that doing custom software development for personal finances seems like major overkill (even for someone who is a professional or hobbyist programmer and I can't imagine it making sense for someone who is not).JupiterJones wrote:I'm an R guy myself when it comes to doing any sort of statistical heavy lifting, although I have done some Python programming too (just not for stats).
Really though, for the type of math I do with my finances, a spreadsheet usually works fine. I'm not a chartist or anything.
If you want sexier charts than Excel / Google Sheets can do, maybe consider Tableau, see e.g. https://www.interworks.com/blog/skenned ... -too-often
I agree, you can do majority of the analysis using either excel or google sheets. I have used both prior to my python tool. The only time I found Excel/Google to be insufficient is when I wanted to integrate my personal budget with my investing and keep track of all my information in one place. It is doable with excel; however, it is cumbersome the more complex you make your tool.
So i guess if you are looking for something simple that gets the job done I would go with excel/google sheets. If you are looking for something complex I would lean towards python/R.
FamilyGuy
Re: Anyone use Python to analyze their finances?
I use python for some general purpose programming, but when it comes to finance and statistics, including optimizing my own portfolio, I use R.
Both are very flexible and powerful. Python's a nicer language overall, but R is designed for these types of problems and it's the language that's used by people who solve them.
Both are very flexible and powerful. Python's a nicer language overall, but R is designed for these types of problems and it's the language that's used by people who solve them.
Re: Anyone use Python to analyze their finances?
My first Python book (15ish years ago) was Learning Python by Mark Lutz, which I liked. I think I read either the first or second edition, which was a much shorter book than the current version. I think the edition I read was in the 350 - 400 pages range. Now, the book is over 1600 pages. In fact, it used to be that you read the Learning Python O'Reilly book for an intro and the big Python O'Reilly book for details. Now, they're both 1600 pages, so I don't know the difference anymore. In general, I like O'Reilly books. My one complaint is that they often seem to embody the mantra of "Why say in one paragraph what you could instead say in five." That said, that style does lend itself to easier reading for beginners.Yeti wrote:Interesting. I have been going through Codecademy and 'Learn Python the Hard Way' just to get familiar with the language over the past few weeks. I don't really have any strong background in writing code, mainly VBA to facilitate spreadsheets.
Are there any books that you would recommend on Python that you have found to be invaluable? Any other resources that you'd recommend?
Thanks!
Yeti
As to the OP's question, I do not use a programming language to handle any of my investing. I find that everything I need can be done in a spreadsheet, and I say that as someone who manufactures programming projects to do for fun. To those who don't get it, I equate it to other people doing crossword puzzles. That said, I have some fairly extensive experience with HTML screen scraping (necessary if the website doesn't provide a useful API to gather information) from several hobby projects (my favorite is my rental car price watching application). Anyway, unless you are doing this simply for fun, it's quite laborious and probably not worth your time. You have to manually examine the HTML and essentially write routines to parse/interact with it. My experience is in interacting with the DOM tree using webkit in a mixture of C++ and javascript. The biggest pain is that after your "done," your application will require updating forever to keep up with every little change the vendor makes to their HTML code. If you go that route, good luck.
Re: Anyone use Python to analyze their finances?
To this good list of resources I would add Statsmodels for econometric models to predict volatility and returns (volatility is somewhat predictable based on GARCH models, but returns predictability depends on the asset and is more controversial) and the book "Python for Data Analysis: Data Wrangling with Pandas, NumPy, and IPython". The low-cost brokerage firm Interactive Brokers has introduced a Python API for automated trading -- they have long had APIs for C++, Java, and C#.AlohaJoe wrote:I use Python a lot. I'm not sure it is really needed to "analyse" finances, though A pretty simple spreadsheet and occasional use of online tools like cfiresim are good enough for 99.9% of people, I reckon.
That said...
Pandas is a great library for data analysis: http://pandas.pydata.org/pandas-docs/stable/index.html
I use it extensively in my own analysis.
It has a related project, pandas-datareader https://pandas-datareader.readthedocs.io/en/latest/ that provides simple & easy access to Yahoo Finance, Google Finance, FRED, Fama/French, the World Bank, OECD, the TSP fund data, Oanda historical currency exchange rates, and a few other things. The bit on "caching queries" -- which actually uses the requests_cache library -- is also useful.
I've mostly used seaborn https://seaborn.pydata.org/ for visualising things. Though I'm thinking about trying out plotly https://plot.ly/python/ since it offers interactivity, which seems like a near-mandatory thing nowadays.
Someone interested in Python for finance should also consider R, which was discussed in a recent thread.
Last edited by Beliavsky on Wed May 10, 2017 9:57 pm, edited 1 time in total.
Re: Anyone use Python to analyze their finances?
If anyone would like to share their code, I can add it to the wiki: Using open source software for portfolio analysis
If there's a lot, upload the script to Google drive and share the link.
If there's a lot, upload the script to Google drive and share the link.
Re: Anyone use Python to analyze their finances?
I'm a software developer and am surprised at the extent some of you are tracking your finances with various scripts. To me, the Boggleheads wiki google sheets spreadsheet is good enough for most everything. If I want to track anything fancier, I can just use Mint or Personal Capital. But maybe I just don't like coding at home. Feels too much like work.
- happenstance
- Posts: 306
- Joined: Sun Jul 26, 2015 11:24 am
- Location: NYC
Re: Anyone use Python to analyze their finances?
Indeed, most of what I do in Jupyter/Pandas could be done in a spreadsheet, and it is what I used to do. But I think it's a lot easier to write a few short lines of Python where my data can have clear variable names, rather than several nested SUMIF()s and COUNTIF()s of anonymous ranges in a cramped formula bar. I also think it's a lot easier to create custom graphs using the various plotting libraries available, rather than being limited by specific spreadsheet template graphs. For me, this is more fun than work, but I definitely understand the not coding at home feeling (except for small automation tasks like this, I seldom do personal coding projects).Gadget wrote:I'm a software developer and am surprised at the extent some of you are tracking your finances with various scripts. To me, the Boggleheads wiki google sheets spreadsheet is good enough for most everything. If I want to track anything fancier, I can just use Mint or Personal Capital. But maybe I just don't like coding at home. Feels too much like work.
Regarding R, it is great for statistics, but I personally find that the language has a lot of unusual semantics that don't match most general purpose programming languages. With Numpy and Pandas, I can't think of anything that you can do in R that you couldn't do in Python. So if you're new to programming, I'd recommend a more general-purpose language like Python with data crunching libraries, where knowledge of Python could then be used to automate and program other tasks, rather than an a specialized language like R or Matlab.
- McGilicutty
- Posts: 349
- Joined: Tue Dec 13, 2016 4:24 pm
Re: Anyone use Python to analyze their finances?
I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.
Re: Anyone use Python to analyze their finances?
Q. Do you trust this stuff, or is it just another form of " crystal ball" gimmickry? I can see using the Bloomberg , in depth, company profile programs for educated guesswork, but a subscription is about $25k per month! According to CBS.McGilicutty wrote:I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.
SEEMoe..
"By gnawing through a dike, even a Rat can destroy a nation ." {Edmund Burke}
Re: Anyone use Python to analyze their finances?
For the OPs intended use, I would lean to using Excel for its easy input and short learning curve. I use R to explore financial data beyond my portfolio. With R or Python, we can do the work of the Portfolio Visualizer web site, Simba's spreadsheet or Morningstar's Growth of $10,000 charts, with the funds/stocks and weights of our choosing. If we want to compare the many Lazy Portfolios, Larry Portfolio, or the Permanent Portfolio with the Three Fund, Four Fund, or Eight Fund portfolios, we can in a few lines of code. I like to be able to translate something that I read into something I can see and manipulate.
IMO, Python requires a better programmer than R. R has over 6000 packages than contain functions for just about whatever we want to do. With Python, we may have to translate a formula into code ourselves. In the R packages, others have done much of the coding for us.
IMO, Python requires a better programmer than R. R has over 6000 packages than contain functions for just about whatever we want to do. With Python, we may have to translate a formula into code ourselves. In the R packages, others have done much of the coding for us.
- McGilicutty
- Posts: 349
- Joined: Tue Dec 13, 2016 4:24 pm
Re: Anyone use Python to analyze their finances?
From the checks I've done the data usually matches up pretty closely to Google Finance's and Morningstar's. In any event, my program is just an initial screener. It doesn't execute trades. Before I did any trades I would do some more in-depth research.SeeMoe wrote:Q. Do you trust this stuff, or is it just another form of " crystal ball" gimmickry? I can see using the Bloomberg , in depth, company profile programs for educated guesswork, but a subscription is about $25k per month! According to CBS.McGilicutty wrote:I use Python to get market cap, sector, industry, price quote, and after-hours price quote for tickers from the Yahoo! earning calendar and to put that data into a sortable Web form in an attempt to find stocks to invest in. So far, I haven't invested in anything turned up by the program, but I've only been doing it for a couple of days.
SEEMoe..
Like I said, I've only been running the program for a couple of days and it's really just an experiment so far.
Re: Anyone use Python to analyze their finances?
Would some of the individuals writing high-level code be willing to share their motivation for doing this and comment any obstacles they may have come across? Specific questions are:
1. What do see as the main deficiencies in the free portfolio analysis tools offered by analysis provided by Personal Capital, SigFig, Morningstar, and Vanguards own Portfolio Analysis tool. I purposefully left Mint off this list as I see thins too being more geered to analyzing how much you spend (it was pretty quick to tell me I spend an insane amount in coffee).
2. How much of the source data you obtain is available using API's? Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis? Can yo access any of your out accounts via APIs.
For anyone one unfamiliar with APIs (Application Program Interface) these are internal calls that are used by different cloud based programs to talk to each other. Sometimes they are given out for free; sometimes you need to pay for them. In the finance world they allow data aggregators like Yodlee to access data from whatever account you are willing to provide the password too - but they are used by travel apps to merge flight information, with hotel accommodation and car rentals. Any data in the cloud. If you have ever downloaded data from one program to your computer and then uploaded the same data to a different program you have performed an unnessecary time consuming step that can be avoided by an API. They also allow specific security measures to be put in place, controlling what data can be downloaded or transferred.
I'm interested in the 2 questions above as it seems there are a lot of concerns about different Portfolio Analysis tools providing different answers. Different answers can be due to errors in coding, correct coding but different assumptions or definitions, or differences in source data.
1. What do see as the main deficiencies in the free portfolio analysis tools offered by analysis provided by Personal Capital, SigFig, Morningstar, and Vanguards own Portfolio Analysis tool. I purposefully left Mint off this list as I see thins too being more geered to analyzing how much you spend (it was pretty quick to tell me I spend an insane amount in coffee).
2. How much of the source data you obtain is available using API's? Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis? Can yo access any of your out accounts via APIs.
For anyone one unfamiliar with APIs (Application Program Interface) these are internal calls that are used by different cloud based programs to talk to each other. Sometimes they are given out for free; sometimes you need to pay for them. In the finance world they allow data aggregators like Yodlee to access data from whatever account you are willing to provide the password too - but they are used by travel apps to merge flight information, with hotel accommodation and car rentals. Any data in the cloud. If you have ever downloaded data from one program to your computer and then uploaded the same data to a different program you have performed an unnessecary time consuming step that can be avoided by an API. They also allow specific security measures to be put in place, controlling what data can be downloaded or transferred.
I'm interested in the 2 questions above as it seems there are a lot of concerns about different Portfolio Analysis tools providing different answers. Different answers can be due to errors in coding, correct coding but different assumptions or definitions, or differences in source data.
- McGilicutty
- Posts: 349
- Joined: Tue Dec 13, 2016 4:24 pm
Re: Anyone use Python to analyze their finances?
Below is a code snippet for getting some data from Yahoo! I use the re and urlib.request libraries. Basically, for each ticker, the code downloads the associated HTML and then does some regular expression matching to get the data I want. I don't know if this is the best way to get the data, but it's the way I do it [Edit: When I copied and pasted my code it removed the indentations. I'll see if I can figure out how to get it back.]:RoboFan wrote:Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis?
import urllib.request
import re
p2 = re.compile('MARKET_CAP.*?><.*?>(.*?)</span>')
p3 = re.compile('reactid="36">(.*?)<')
p4 = re.compile('reactid="37">(.*?)<')
p5 = re.compile('reactid="41">(.*?)<')
listStrMarketCaps = []
listFltDayPrices = []
listStrDayChanges = []
listFltAHPrices = []
intCount = 1
for strGoodTicker in listStrGoodTickers:
text = ""
try:
local_filename, headers = urllib.request.urlretrieve('https://finance.yahoo.com/quote/' + strGoodTicker + '/?p=' + strGoodTicker)
html = open(local_filename, encoding="utf-8")
text = html.read()
except:
print("An error occurred at line 59")
matchMarketCap = p2.search(text)
if (matchMarketCap != None):
listStrMarketCaps.append(matchMarketCap[1])
print(str(intCount) + ' Market Cap = ' + matchMarketCap[1])
else:
listStrMarketCaps.append("0.0")
print(str(intCount) + 'Market Cap = 0.0')
listStrDayPrice = p3.findall(text)
fltDayPrice = 0.0
try:
strDayPrice = listStrDayPrice[2]
fltDayPrice = float(strDayPrice)
except:
print("Error getting day price")
listFltDayPrices.append(fltDayPrice)
listStrDayChange = p4.findall(text)
strDayChange = ""
try:
strDayChange = listStrDayChange[1]
except:
print("Error getting day price change")
listStrDayChanges.append(strDayChange)
fltAHPrice = fltDayPrice
if blnAH:
listStrAHPrice = p5.findall(text)
try:
strAHPrice = listStrAHPrice[2]
fltAHPrice = float(strAHPrice)
except:
print("Error getting AH price")
listFltAHPrices.append(fltAHPrice)
print("Day price = " + str(fltDayPrice))
print("Day change = " + strDayChange)
print("AH Price = " + str(fltAHPrice))
print("")
intCount += 1
Re: Anyone use Python to analyze their finances?
With these two lines of code in R:
library(tidyquant)
data <- tq_get("IBM", get = 'key.stats')
I can get all of the following metrics for IBM:
library(tidyquant)
data <- tq_get("IBM", get = 'key.stats')
I can get all of the following metrics for IBM:
Code: Select all
[1] "Ask" "Ask.Size"
[3] "Average.Daily.Volume" "Bid"
[5] "Bid.Size" "Book.Value"
[7] "Change" "Change.From.200.day.Moving.Average"
[9] "Change.From.50.day.Moving.Average" "Change.From.52.week.High"
[11] "Change.From.52.week.Low" "Change.in.Percent"
[13] "Currency" "Days.High"
[15] "Days.Low" "Days.Range"
[17] "Dividend.Pay.Date" "Dividend.per.Share"
[19] "Dividend.Yield" "EBITDA"
[21] "EPS" "EPS.Estimate.Current.Year"
[23] "EPS.Estimate.Next.Quarter" "EPS.Estimate.Next.Year"
[25] "Ex.Dividend.Date" "Float.Shares"
[27] "High.52.week" "Last.Trade.Date"
[29] "Last.Trade.Price.Only" "Last.Trade.Size"
[31] "Last.Trade.With.Time" "Low.52.week"
[33] "Market.Capitalization" "Moving.Average.200.day"
[35] "Moving.Average.50.day" "Name"
[37] "Open" "PE.Ratio"
[39] "PEG.Ratio" "Percent.Change.From.200.day.Moving.Average"
[41] "Percent.Change.From.50.day.Moving.Average" "Percent.Change.From.52.week.High"
[43] "Percent.Change.From.52.week.Low" "Previous.Close"
[45] "Price.to.Book" "Price.to.EPS.Estimate.Current.Year"
[47] "Price.to.EPS.Estimate.Next.Year" "Price.to.Sales"
[49] "Range.52.week" "Revenue"
[51] "Shares.Outstanding" "Short.Ratio"
[53] "Stock.Exchange" "Target.Price.1.yr."
[55] "Volume"
Re: Anyone use Python to analyze their finances?
Use the "Code" post edit button to surround your code, which is shown in the previous post. You can edit your post to fix the content, then bump the thread to let everyone know it was changed.McGilicutty wrote:[Edit: When I copied and pasted my code it removed the indentations. I'll see if I can figure out how to get it back.]:
I should also mention that you don't need to pay for MS Excel. LibreOffice Calc is a free and open source alternative that works just fine. There are some compatibility differences on charts and possibly macros, but it shouldn't stop you from using the software.
Re: Anyone use Python to analyze their finances?
Many here have mentioned that they use spreadsheets for their calculations. IMHO the use of a spreadsheet for anything more complex than adding a column of numbers is a serious mistake. The reason for this is that for anything more complex than that it is almost impossible to:
(a) validate the calculations to gain assurance that you are actually calculating what you want
(b) modify or extend the calculation to other cases.
I am not the only one that will not use a spreadsheet for serious computations. See this article by statistician Patrick Burns:
http://www.burns-stat.com/documents/tut ... addiction/
I would like to see more use of a high level language like Python in the computation software posted on Bogleheads. I am a Matlab man myself but I can also swing with Python or R. I will draw the line at Java or C++ but they are still better than Excel.
(a) validate the calculations to gain assurance that you are actually calculating what you want
(b) modify or extend the calculation to other cases.
I am not the only one that will not use a spreadsheet for serious computations. See this article by statistician Patrick Burns:
http://www.burns-stat.com/documents/tut ... addiction/
I would like to see more use of a high level language like Python in the computation software posted on Bogleheads. I am a Matlab man myself but I can also swing with Python or R. I will draw the line at Java or C++ but they are still better than Excel.
Re: Anyone use Python to analyze their finances?
Sure you can do the same thing in 2 lines of python. Most normal people don't do what the previous poster did when fetching Yahoo quote data There are libraries for most things under the sun and generally people use those libraries...just like you use a library in your R example.GLState wrote:With these two lines of code in R:
library(tidyquant)
data <- tq_get("IBM", get = 'key.stats')
I can get all of the following metrics for IBM
Code: Select all
import pandas_datareader.data as web
f = web.DataReader("F", 'yahoo')
Re: Anyone use Python to analyze their finances?
I wouldn't say that I do "portfolio analysis" since that's not what I'm interested in.RoboFan wrote:Would some of the individuals writing high-level code be willing to share their motivation for doing this and comment any obstacles they may have come across?
Here are some things that I've written in Python that would be either hard or impossible to do in Excel.
See if using CAPE10 to market time rebalancing makes a difference when using monthly data
Build a bond fund simulator that allows me to simulate historical performance of bonds
Extend that to do the same for Japanese government bonds, to see how bonds outside the US have performed
Compare 18 different withdrawal strategies in retirement
Compare 8 different "harvesting" strategies during retirement
Compare the relative importance of early "bad returns" and "high inflation" on retirement
Calculate safe withdrawal rates in Japan for various portfolio allocations
Compare various income smoothing strategies in retirement
Calculate how many months of gains are wiped out by various market corrections
What kinds of things are easier to do in Python?
1. I can run lag-1 autoregressions to simulate bond yields being low right and slowly drifting higher over the next several years...instead of assuming historical averages
2. I can calculate certainty-equivalent withdrawals, Gompertz mortality, and Sustainable Spending Rates -- all of which have slightly complicated math
3. I can calculate Milevsky's "risk of ruin" using the Gamma distribution CDF.
4. I can run 50 different Monte Carlo simulations -- varying lots of parameters -- in a simple for-loop instead of needing continually modify a spreadsheet (or have 50 tabs).
Re: Anyone use Python to analyze their finances?
This thread is now in the wiki: Using open source software for portfolio analysis
I have added AlohaJoe's github links to the Python section.
I have added AlohaJoe's github links to the Python section.
Re: Anyone use Python to analyze their finances?
I haven't used Python specifically, but I have written custom code in various languages over the years to analyze financial data, including C++ and JavaScript. I have also used Excel quite a bit, which is more capable than many people realize, but is slow and cumbersome for running simulations.
- masterofinvesting
- Posts: 101
- Joined: Sat Jun 22, 2013 9:40 am
Re: Anyone use Python to analyze their finances?
I have a related question. A lot of us have years (or decades) of financial data in Quicken, but we still rely on Quicken's reporting to extract and visualize and data that we need. Wouldn't it be nicer to have a scripting interface for Quicken? (just like Word or Excel - both of which can be scripted via VB macros or Python/Perl). Even a read only interface to Quicken's database would be quite useful. This way we can do some deeper and highly customized analysis than what Quicken reporting provides - of course, this isn't for everyone - probably for those (like me) who find this kind of analysis fun.
Couple of recent books I've read on intermediate Python that I found incredibly enlightening:
Couple of recent books I've read on intermediate Python that I found incredibly enlightening:
- Fluent Python: Clear, Concise, and Effective Programming - Luciano Ramalho
- Python Cookbook: Recipes for Mastering Python 3 - Brian K. Jones and David M. Beazley
- McGilicutty
- Posts: 349
- Joined: Tue Dec 13, 2016 4:24 pm
Re: Anyone use Python to analyze their finances?
Below is a code snippet for getting some data from Yahoo! I use the re and urlib.request libraries. Basically, for each ticker, the code downloads the associated HTML and then does some regular expression matching to get the data I want. I don't know if this is the best way to get the data, but it's the way I do it.McGilicutty wrote:RoboFan wrote:Can you automatically get data from Yahoo (prices, dividends, market cap, etc.) or are you downloading text data from these sites and then uploading the downloaded text files into your analysis?
Code: Select all
import urllib.request
import re
p2 = re.compile('MARKET_CAP.*?><.*?>(.*?)</span>')
p3 = re.compile('reactid="36">(.*?)<')
p4 = re.compile('reactid="37">(.*?)<')
p5 = re.compile('reactid="41">(.*?)<')
listStrMarketCaps = []
listFltDayPrices = []
listStrDayChanges = []
listFltAHPrices = []
intCount = 1
for strGoodTicker in listStrGoodTickers:
text = ""
try:
local_filename, headers = urllib.request.urlretrieve('https://finance.yahoo.com/quote/' + strGoodTicker + '/?p=' + strGoodTicker)
html = open(local_filename, encoding="utf-8")
text = html.read()
except:
print("An error occurred at line 59")
matchMarketCap = p2.search(text)
if (matchMarketCap != None):
listStrMarketCaps.append(matchMarketCap[1])
print(str(intCount) + ' Market Cap = ' + matchMarketCap[1])
else:
listStrMarketCaps.append("0.0")
print(str(intCount) + 'Market Cap = 0.0')
listStrDayPrice = p3.findall(text)
fltDayPrice = 0.0
try:
strDayPrice = listStrDayPrice[2]
fltDayPrice = float(strDayPrice)
except:
print("Error getting day price")
listFltDayPrices.append(fltDayPrice)
listStrDayChange = p4.findall(text)
strDayChange = ""
try:
strDayChange = listStrDayChange[1]
except:
print("Error getting day price change")
listStrDayChanges.append(strDayChange)
fltAHPrice = fltDayPrice
if blnAH:
listStrAHPrice = p5.findall(text)
try:
strAHPrice = listStrAHPrice[2]
fltAHPrice = float(strAHPrice)
except:
print("Error getting AH price")
listFltAHPrices.append(fltAHPrice)
print("Day price = " + str(fltDayPrice))
print("Day change = " + strDayChange)
print("AH Price = " + str(fltAHPrice))
print("")
intCount += 1
Re: Anyone use Python to analyze their finances?
I use Alteryx to do this. It works wonders and really takes reporting and data analytics based on info from quick to the next level.masterofinvesting wrote:I have a related question. A lot of us have years (or decades) of financial data in Quicken, but we still rely on Quicken's reporting to extract and visualize and data that we need. Wouldn't it be nicer to have a scripting interface for Quicken? (just like Word or Excel - both of which can be scripted via VB macros or Python/Perl). Even a read only interface to Quicken's database would be quite useful. This way we can do some deeper and highly customized analysis than what Quicken reporting provides - of course, this isn't for everyone - probably for those (like me) who find this kind of analysis fun.
Couple of recent books I've read on intermediate Python that I found incredibly enlightening:
- Fluent Python: Clear, Concise, and Effective Programming - Luciano Ramalho
- Python Cookbook: Recipes for Mastering Python 3 - Brian K. Jones and David M. Beazley
Re: Anyone use Python to analyze their finances?
I do not use Python for my finances yet (use it for a few other things like data validation etc).
I use a collection of data files, C programs, Unix shell scripts, Awk scripts and Excel spreadsheets to do the following:
Parse OFX transaction data from Vanguard. Download monthly closing price data from Yahoo.
Keep track of basis lots for all my mutual funds.
Produce monthly portfolio printouts with asset value / capital gain information.
Keep track of overall portfolio composition, monthly asset class / net worth totals over time in both nominal and real terms.
Compute internal rate of return of portfolio and each asset class.
Compute return / volatility data for each of my mutual funds.
Lately I have been looking into OFX file parsers - there is a github project for a python implementation that looks good. If I had the energy and time, today I might implement most of this with python, a SQL database and Excel. Most of what I have was coded over 20 years ago.
I suspect many of the more technically minded folks have needs (desires) that are so far from what the average person wants that most commercial financial portfolio tools / websites will simply not suffice.
I use a collection of data files, C programs, Unix shell scripts, Awk scripts and Excel spreadsheets to do the following:
Parse OFX transaction data from Vanguard. Download monthly closing price data from Yahoo.
Keep track of basis lots for all my mutual funds.
Produce monthly portfolio printouts with asset value / capital gain information.
Keep track of overall portfolio composition, monthly asset class / net worth totals over time in both nominal and real terms.
Compute internal rate of return of portfolio and each asset class.
Compute return / volatility data for each of my mutual funds.
Lately I have been looking into OFX file parsers - there is a github project for a python implementation that looks good. If I had the energy and time, today I might implement most of this with python, a SQL database and Excel. Most of what I have was coded over 20 years ago.
I suspect many of the more technically minded folks have needs (desires) that are so far from what the average person wants that most commercial financial portfolio tools / websites will simply not suffice.
Re: Anyone use Python to analyze their finances?
Lots of great answers to my question about why people are writing code (with example code!). I think the last poster captured the answers nicely - individuals want to do things well outside what the average investor want to do.
What I did not pick up was whether anyone is using API's per se. There is another thread "Vanguard versus interactive brokers: API?" that address some of what I was getting at:
viewtopic.php?t=174129
But the thread is a bit old and I'm not sure everyone on it recognized what an API is. Some posters did not believe Vanguard has API's and expressed security concerns suggesting they would be unhappy about it if they found outs such a thing exists (thing being a piece of code that allows another program to access their account information). To be clear - API's are everywhere. The fact that apps such as Personal Capital and SigFig can, if I provide a password, access my Vanguard account suggests Vanguard provides API support. This is not a bad thing. From a security perspective Vanguard is still responsible for your account, and they control what access they give via an API. This is different from read only - an API can limit what get reads, when it get read, whatever, the owner of the API get to specify what it does.
What I got out of the thread referenced above was that Vanguard does not provide API support to individual users but TDAmerica does. Not sure if this is true today. What I'm curious about is whether any of the "tech savvy" posters on this thread have tried to access their own Vanguard data via software. I see examples of accessessing Yahoo and other "public" data but no examples of accessing their own "secure" data.
Anyone tried to do this?
What I did not pick up was whether anyone is using API's per se. There is another thread "Vanguard versus interactive brokers: API?" that address some of what I was getting at:
viewtopic.php?t=174129
But the thread is a bit old and I'm not sure everyone on it recognized what an API is. Some posters did not believe Vanguard has API's and expressed security concerns suggesting they would be unhappy about it if they found outs such a thing exists (thing being a piece of code that allows another program to access their account information). To be clear - API's are everywhere. The fact that apps such as Personal Capital and SigFig can, if I provide a password, access my Vanguard account suggests Vanguard provides API support. This is not a bad thing. From a security perspective Vanguard is still responsible for your account, and they control what access they give via an API. This is different from read only - an API can limit what get reads, when it get read, whatever, the owner of the API get to specify what it does.
What I got out of the thread referenced above was that Vanguard does not provide API support to individual users but TDAmerica does. Not sure if this is true today. What I'm curious about is whether any of the "tech savvy" posters on this thread have tried to access their own Vanguard data via software. I see examples of accessessing Yahoo and other "public" data but no examples of accessing their own "secure" data.
Anyone tried to do this?
Re: Anyone use Python to analyze their finances?
Vanguard, along with most banks, do not provide any sort of API. Personal Capital and the like don't use an API. They screenscrape. Or go to a known URL to download a CSV or OFX. I haven't checked but they probably use Yodlee and outsource the screenscraping to them, it seems like everyone does.
Re: Anyone use Python to analyze their finances?
I can't see what use Python would be to track a portfolio when a simple Excel spreadsheet would do.
Sounds to me like more of a hobby.
Sounds to me like more of a hobby.
Re: Anyone use Python to analyze their finances?
Aren't those things contradict each other? One is that past performance not guarantee future results and in the the same time complex portfolio analysis trying to make predictions. If you cannot relay on the past for some reason no point trying to predict because prediction based on the past data analysis.
Re: Anyone use Python to analyze their finances?
Screenscraping, as I understand it, involves pulling data from the display screen once the account is accessed and the data are displayed. However, there have been reports on other threads that SigFig can access user information from Schwab despite the user have implemented 2-factor authentication. The SigFig thread - dating back to 2014 - never really resolved how SigFig managed to get around the 2-fact authentication. I just assumed they did so via an API.AlohaJoe wrote:Vanguard, along with most banks, do not provide any sort of API. Personal Capital and the like don't use an API. They screenscrape. Or go to a known URL to download a CSV or OFX. I haven't checked but they probably use Yodlee and outsource the screenscraping to them, it seems like everyone does.
For readers who are not following this so well, and it is confusing, an API allows one program to access a limited set of information and commands available from another program. Two-factor authentications should prevent any person or program from obtaining access to any data or commands until you enter a code sent to you via a second device you have with you (typically cell phone).
I'm thinking this issue might be worth a separate new thread.
-
- Posts: 283
- Joined: Sat Jul 23, 2011 8:06 pm
Re: Anyone use Python to analyze their finances?
Wondering if you're still doing this, and if so, how it's working. I've been pretty disappointed with most budget/portfolio apps and am looking to perhaps roll my own.FamilyGuy87 wrote: ↑Mon May 08, 2017 8:31 am I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.
- happenstance
- Posts: 306
- Joined: Sun Jul 26, 2015 11:24 am
- Location: NYC
Re: Anyone use Python to analyze their finances?
You might be interested in checking out Beancount. It's a Python text-based accounting system with a pretty nice web frontend called Fava.ScaledWheel wrote: ↑Mon Dec 09, 2019 3:51 pmWondering if you're still doing this, and if so, how it's working. I've been pretty disappointed with most budget/portfolio apps and am looking to perhaps roll my own.FamilyGuy87 wrote: ↑Mon May 08, 2017 8:31 am I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.
Re: Anyone use Python to analyze their finances?
The minor problem with beancount/fava is that you have to learn double entry bookkeeping, which is a little excessive for just monitoring investments. And while the front end is nice, the reports are limited for investment analysis.happenstance wrote: ↑Mon Dec 09, 2019 4:16 pmYou might be interested in checking out Beancount. It's a Python text-based accounting system with a pretty nice web frontend called Fava.ScaledWheel wrote: ↑Mon Dec 09, 2019 3:51 pmWondering if you're still doing this, and if so, how it's working. I've been pretty disappointed with most budget/portfolio apps and am looking to perhaps roll my own.FamilyGuy87 wrote: ↑Mon May 08, 2017 8:31 am I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.
For those reasons, I too ended up rolling my own inspired by beancount but with an investment friendly angle in mind, which happens to allow more rich reporting. I have it available on a website (pre-alpha version?). I won't post it here but PM me if anyone is interested.
-
- Posts: 283
- Joined: Sat Jul 23, 2011 8:06 pm
Re: Anyone use Python to analyze their finances?
I don't need fine-grained budgeting tools, what I'm hoping to do is aggregate spending data via an online tool that lets me download CSVs to use in the program, then of course do a bunch more analysis in my program. Have a background in probabilistic modeling so interested in playing around with portfolio information, but don't have any experience in front end development.glorat wrote: ↑Mon Dec 09, 2019 7:27 pmThe minor problem with beancount/fava is that you have to learn double entry bookkeeping, which is a little excessive for just monitoring investments. And while the front end is nice, the reports are limited for investment analysis.happenstance wrote: ↑Mon Dec 09, 2019 4:16 pmYou might be interested in checking out Beancount. It's a Python text-based accounting system with a pretty nice web frontend called Fava.ScaledWheel wrote: ↑Mon Dec 09, 2019 3:51 pmWondering if you're still doing this, and if so, how it's working. I've been pretty disappointed with most budget/portfolio apps and am looking to perhaps roll my own.FamilyGuy87 wrote: ↑Mon May 08, 2017 8:31 am I use Python for my personal budget and investing exclusively. I started with excel and over the years I have migrated to using only python. I find it's more robust and less prone to errors. My tool has a python backend which uses Pandas and Flask. For the frontend I use HTML based frameworks such as angular and dc.js.
For those reasons, I too ended up rolling my own inspired by beancount but with an investment friendly angle in mind, which happens to allow more rich reporting. I have it available on a website (pre-alpha version?). I won't post it here but PM me if anyone is interested.
Thanks for the offer, will PM.
- masterofinvesting
- Posts: 101
- Joined: Sat Jun 22, 2013 9:40 am
Re: Anyone use Python to analyze their finances?
So far, I've noticed no one has mentioned (GNUCash) yet. I used to track my finances using Microsoft Money first, then switched to Quicken, eventually gave up and switched to GNUCash. The key advantage of using GNUCash is the fact that your financial data is stored in an open SQL-LITE format (see here for a description of the SQL object model and schema).
With the open SQL format, you can do quite a few things:
If you are a functional programming junkie, GNUCash also comes with a in-build Scheme interpreter that you can use to create custom reports that you can open within the GUI.
With the open SQL format, you can do quite a few things:
- Use Piecash (which internally uses SQLAlchemy) to process your financial data in a very inituitive manner using Python classes.
- Use Pandas to read multiple SQL tables into dataframes and do fun-stuff with it.
- You can use Alphavantage to get time series data and directly update the "Prices" table in the GNUCash DB.
If you are a functional programming junkie, GNUCash also comes with a in-build Scheme interpreter that you can use to create custom reports that you can open within the GUI.
-
- Posts: 1000
- Joined: Wed Apr 26, 2017 12:03 am
Re: Anyone use Python to analyze their finances?
You Need A Budget will aggregate spending data for you, and said data can be exported as a nicely formatted CSV for further analysis. It will do the hard work of auto fetching days from providers etc.ScaledWheel wrote: ↑Mon Dec 09, 2019 8:27 pm I don't need fine-grained budgeting tools, what I'm hoping to do is aggregate spending data via an online tool that lets me download CSVs to use in the program, then of course do a bunch more analysis in my program. Have a background in probabilistic modeling so interested in playing around with portfolio information, but don't have any experience in front end development.
It’s not a good choice for tracking investments - at least I don’t use it for that, and i don’t consider it the right tool for that job. But I think it works great for tracking expenses. I myself also periodically export a CSV as a sort of backup.
-
- Posts: 283
- Joined: Sat Jul 23, 2011 8:06 pm
Re: Anyone use Python to analyze their finances?
I've settled on a way forward that works for my purposes. I can use Fidelity Full View as an aggregator for both investments and spending as it allows exporting to CSV. Then I can use Python/pandas to get it into the appropriate database format, and then use Dash [0] for visualization.
The main reason is that I don't do front end development, so keeping everything in Python is nice. Then Dash allows me to keep all my data/analysis local (though it could be hosted) and create an easy-to-navigate dashboard that I can share with my wife when we want to go over things. It also has the benefit that I will probably end up using Dash in a new job so I can use this small, fun project to practice putting together dashboards and visualizations.
[0] https://dash-gallery.plotly.host/Portal/
The main reason is that I don't do front end development, so keeping everything in Python is nice. Then Dash allows me to keep all my data/analysis local (though it could be hosted) and create an easy-to-navigate dashboard that I can share with my wife when we want to go over things. It also has the benefit that I will probably end up using Dash in a new job so I can use this small, fun project to practice putting together dashboards and visualizations.
[0] https://dash-gallery.plotly.host/Portal/
- nisiprius
- Advisory Board
- Posts: 52215
- Joined: Thu Jul 26, 2007 9:33 am
- Location: The terrestrial, globular, planetary hunk of matter, flattened at the poles, is my abode.--O. Henry
Re: Anyone use Python to analyze their finances?
Not Python. Although having worked professionally with MUMPS for several years... and having bought a OLPC XO on the basis of a promise, unfulfilled, that the OS was not only written entirely in Python but that it could be inspected and modified directly from within the device itself... I'm sympathetic to Python's style.
The comment I would make is, "don't look too much at the language, look at standard libraries customarily provided with it." That is, the decision factor should, not the intrinsic merits of the language itself, but whether the standard library has good web, network, and url access tools. The reason I emphasize "standard library" is not just cost, but also you can count on a standard library to be implemented reasonably well, or else the development system wouldn't be salable. I don't know how Python shapes up in this regard.
I'm using Xojo, a proprietary language and development system, once called "REALBasic." I like it from years of long familiarity, but I won't try to promote it particularly. I'm biased because I use a Mac and it began as a Mac-centric product, and have always treated Mac OS co-equally with Windows, whereas other systems tend to have fickle on-again off-again Mac support.
The comment I would make is, "don't look too much at the language, look at standard libraries customarily provided with it." That is, the decision factor should, not the intrinsic merits of the language itself, but whether the standard library has good web, network, and url access tools. The reason I emphasize "standard library" is not just cost, but also you can count on a standard library to be implemented reasonably well, or else the development system wouldn't be salable. I don't know how Python shapes up in this regard.
I'm using Xojo, a proprietary language and development system, once called "REALBasic." I like it from years of long familiarity, but I won't try to promote it particularly. I'm biased because I use a Mac and it began as a Mac-centric product, and have always treated Mac OS co-equally with Windows, whereas other systems tend to have fickle on-again off-again Mac support.
Annual income twenty pounds, annual expenditure nineteen nineteen and six, result happiness; Annual income twenty pounds, annual expenditure twenty pounds ought and six, result misery.
Re: Anyone use Python to analyze their finances?
That sounds really intriguing. One time I imported all my Quicken data into GnuCash to try it out, and was surprised at how well that went. The one weakness I felt I couldn't live with was the reports, so this sounds like a great way to custom enhancement. Always great to have that relational database behind it all.masterofinvesting wrote: ↑Mon Dec 09, 2019 8:53 pm So far, I've noticed no one has mentioned (GNUCash) yet. I used to track my finances using Microsoft Money first, then switched to Quicken, eventually gave up and switched to GNUCash. The key advantage of using GNUCash is the fact that your financial data is stored in an open SQL-LITE format (see here for a description of the SQL object model and schema).
With the open SQL format, you can do quite a few things:I've been using the combination of Python and D3.js to create some interesting plots out of my past (decades worth) of financial data.
- Use Piecash (which internally uses SQLAlchemy) to process your financial data in a very inituitive manner using Python classes.
- Use Pandas to read multiple SQL tables into dataframes and do fun-stuff with it.
- You can use Alphavantage to get time series data and directly update the "Prices" table in the GNUCash DB.
If you are a functional programming junkie, GNUCash also comes with a in-build Scheme interpreter that you can use to create custom reports that you can open within the GUI.
I'm another programmer by day who would rather save my time by getting something off the shelf that gives you 80% of your needs. Then sure, spend a little tinkering time trying to get that other 20%.