my system is ubuntu 12.04 at Digital Ocean, and i am going to install python3.3 and pip3.3

here are few simple steps with which you can do so!

$ apt-get install python-software-properties
$ add-apt-repository ppa:fkrull/deadsnakes
$ apt-get update
$ apt-get install python3.3
$ curl http://python-distribute.org/distribute_setup.py | python3.3
$ curl https://raw.github.com/pypa/pip/master/contrib/get-pip.py | python3.3

and yes. . don’t forget to apply “sudo” above!!!

enjoy. .. 

After appearing my app at xbrl challenge competition; this is going to be the first major update after that event!

I have updated the app to use mongodb. MongoDB promotes rapid development.

See the output of my recent test; its a html file, exported from ipython notebook.

Click here, to see it! (save the html file and then open it in browser to view the output!!!)

Well, further testing is going on, and lets expect something big from such tiny-tiny efforts!!!!

In this post i am going to explain type1 formula/array in details.

type1 formula/array is the most important, fundamental building block of xbrlfinapp.

It represent an xbrl element in terms of array.

To understand this; lets take an example:

Suppose we want to calculate “Total Assets” and in US GAAP taxonomy its id is “us-gaap_Assets”. So now we will find this xbrl element in our concerned xbrl filling. And if filler has tagged values with this xbrl element, then we will fetch that value and will use that value according to our need.

Anyways, This is the easiest part!

Now hardest part: How to calculate “Total Assets” is filler hasn’t provided?

So the solution is: Lets find out its part/components and return the total of all and call this total as our “Total Assets”.

This concept has been implemented in type1 array.

type1 array is considering first element as a primary element and rest in the list as a component of it.

Its algorithm will first search for 1st element of input array in the filling.

If it is there then simply take its values, and move to next elements in the list and will add into the values of the first element  if it is not the element wise duplicate of first or is not the part or component of it according to US GAAP taxonomy(with resolving context, unit, item type, balance type).

And if it is not there; then its algorithm will search for the components of “Total Assets”, and will sum them all, without any repetitions(means if current assets is there and its parts are also there; then algorithm will take current assets and forgot its part). After this algorithm will move to the next element of input array and will do the same for it and its returning value will be added to its previous one(with resolving everything: balance type, unit, item type).

When the iteration is over; type1 array will take the properties of first element and assign that to the final result.

Do you think that at this point our work is over? and we got what we have asked for?

My answer is: No

Up-to this, we have done nothing for extensions. But type1 array allows you to add them. Extensions are not algorithmically checked for part and its sub-part. They are filling specific. So you can add them in type1 array. algorithm will just add if it finds it to the final sum.

This was the main reason to use id’s of xbrl element. Because one extension’s id will differ from id other filler’s. So if it is there then it will add in its final sum or will forgot it and move to the other element.

By this way; you can add huge list of elements as formula/array where 1st element will be the US-GAAP’s xbrl element and others are from different fillings. If that id is present; then it will added otherwise forgot it and more to next element of input array.

Lets take real life example. I am going to consider filling that we have used in our previous post:

Its “Total Assets” is:

array2And when we have added multiple elements which are part of Total Assets, in that also the result is same as “Total Assets”. type1 knows how to avoid duplicate and how to avoid sub components.

array1

Similarly we can test “Total Assets” and “Liabilities and Stockholders Equity’s” total. If both are equal then returning total will be zero, because one is debit and other is credit and when we represent these two as a one; then both will eliminate each other and result will be zero.array3

In same way; We can find “Debt” as:array4

Above you can see that the debt’s component: short term and long term are not present; so will return its components.

In same way “Revenues” can be find. Here xbrl element for revenues “us-gaap_Revenues” is not provided by the filler, so its component are: us-gaap_InterestExpense, us-gaap_SalesRevenueNet were filtered by the app and its result is:array5

Hello friends,
New updated xbrlfinapp has been released.

Now It has more capabilities and more dynamic nature. So its going to be fun; and i am sure you will enjoy analysis of xbrl financial data.

Visit xbrlfinapp at: http://xbrlfinapp.pythonanywhere.com/

xbrl financial analysis home page

Some important updates are:

  • Formula Design: Now user can design custom formulas. For that they need to enter id of those xbrl elements. There are two types of formulas(type1 and type2). type1 formulas accepts xbrl elements in form of list/array of their ids. while type2 formulas are based on type1 formulas where you can write simple arithmetic formulas based on previously designed type1 formulas
  • login system: User has to register and login to get started with the app
  • Makeover of app: website’s look and feel has been update to make user comfortable with the site

Most important of them are formula building. User can build their formulas.  In mathematics; formulas are as simple as: x=3,y=2,z=x+y;

Like this; user can write their own financial formulas.

For example:

lets download sample xbrl: NUVASIVE INC’s 2012, 10-K, and upload this at xbrlfinapp.

UGT has xbrl element ‘us-gaap_AssetsCurrent‘ for Current Assets(or in simple manner, just assume, if x=30). So lets we first create type1 formula of this by giving it a name and in input list.

current asset's type1 formula

Similarly for current liabilities we have element ‘us-gaap_LiabilitiesCurrent‘, and we will create type1 formula(or in simple manner, just assume, if y=20) for it as:

liabilities type1 formula

So output of this operation at xbrlfinapp will be something like(if you have upload valid sec filled xbrl package) :

assets n lia

All that was for type1 formulas. Now type2. Say we want to find out working capital; then formula for it in general is current assets – current liabilities. So in XBRLFINAPP we can write such mathematical equations in type2 formulas.

Create type2 formula of working capital as:

working capital's type2 formula

We have just wrote “currentAssets – currnetlia” as a input and give it a name ‘workingCapital'(as simple as “z=x-y”). So output of this operation will be:

wc

Similarly type2 formula of current ratio can we written as:

current ratio's type2 formula

and its result will be:

currentratio

By this way; user is allowed to use “+”,”-” and “/” operator with “(” or “)” for writing equations.

Note:

  • type1 formulas will contain array or list of ids of xbrl elements(separated by comma) which you want to add. and these elements must have same item type. If user has added different item type elements; nothing will happen. you can also add ids of extension elements.
  • type2 formulas will contain previously formulated type1 and type2 formulas.
  • And you can’t delete type1 or type2 formulas which are used in formulas which were created after it.
  • You can’t modify the name or alias of formula after creating it. To modify it first delete it and then create it again with new name or alias.

One of the most important feature of xbrlfinapp is; its intelligent element discovery.

XBRLFINAPP is intelligent in terms of finding element. For example if the first xbrl element of input array of type1 formula is not present in instance then it will find its part; and return the total. for example; in this filling lets find “cash and cash equivalents”. But it is not present. So xbrlfinapp will search for its component and will return total of them(by resolving itemtype, balance type, context, unit, period in background), and if additional xbrl elements has been added as input; then rest of that type1 array will be added into this total if they are the not part of first element. Part means: are they duplicate or they are part of cash or cash equivalents. It means if user has created extension then we can  add extensions here.

cash equ type1 formula

Output of this is:

cashequ

Here; output has 4 values because XBRLFINAPP with search for element in complete filling. Also; look at the filtered elements list; it has two elements; and when you look at the balance sheet; here they are:

Screen shot 2013-03-12 at 11.18.59 PM

Such type of issue was the most challenging part for me; For example; hardest question were:

1. how will you calculate current ratio if filler has not provided current assets’s tag?

2. How will calculate Net Income if filler hasn’t tagged it??

But now;  this question has been resolved by xbrlfinapp :)

So my formulas are now constant for all companies. I don’t require to change them for companies to companies.

For more register and login xbrlfinapp; you will receive 35 formulas by defaults. you can learn from them how to create formulas based on these formulas.

Anyways; friends if you face any query/error/bug; please email me about that at “namitkewat@gmail.com”.

Hello friends,

Today i am releasing first alpha release of my xbrl analytics platform.

Its URL is: http://xbrlfinapp.pythonanywhere.com/

Its a web application, targeting users whose XBRL is on SEC/or not on SEC/or not in public domain yet/or anyone who is interested in XBRL data and want to do the financial analytics.

At present it is only working for US-GAAP TAXONOMY VERSIONS 09,11,12.

Website’s layout and design is simple; just upload valid zipped xbrl package and see the result on Homepage only. Upload file size has limit of 5MB. If you want to increase that limit then please show me that xbrl.

Limited numbers of financial formulas(Key performance Indicators) have been included; because current website design and hosting type is restricting them. But these formulas are using data from XBRL package only; no access  to market information(like; share price).

Well, all those improvements are in progress.

This app represents final analysed data in form of table;similar how arelle is loading xbrl file. But its not. It is doing complex algorithm of data finding.

For example; consider two cases as; 1st: ” Correctly determining what total equity is if an SEC filer does not provide total equity” 2nd: “Correctly determining what total revenues is when the filer does not provide total revenues or when they use obscure concepts to express revenues”

Lets take this challenge to my app; first download one xbrl filling from SEC from here and upload it to xbrlfinapp application.

case 1. Filler has not provided total equity concept(“us-gaap_StockholdersEquity”); and my application has find out its three components, they are: RetainedEarningsAccumulatedDeficit,  CommonStockValue,  AccumulatedOtherComprehensiveIncomeLossNetOfTax.

case 2. Filler has not provided ‘total revenues’ concept(‘us-gaap_Revenues’); and then in such case my application has find out its two components, they are; InterestExpense,  SalesRevenueNet

Similarly filler has not provided ‘Pre-Tax Income Loss’ concept(‘us-gaap_IncomeLossFromContinuingOperationsBeforeIncomeTaxesExtraordinaryItemsNoncontrollingInterest’);
and then my application has find out its two components, they are; IncomeLossFromEquityMethodInvestments,  IncomeLossFromContinuingOperationsBeforeIncomeTaxesMinorityInterestAndIncomeLossFromEquityMethodInvestments.
Other cases like; filer has not provided ‘Total Costs and Expenses’ concept (us-gaap_CostsAndExpenses) and then my application has find out its five components as; InterestExpense, CostOfGoodsSold, OtherPostretirementBenefitExpense, SellingGeneralAndAdministrativeExpense, PensionExpense.

 

but Current Assets(us-gaap_AssetsCurrent),Current liabilities(us-gaap_LiabilitiesCurrent), Total Assets(us-gaap_Assets) ,Operating Income Loss(us-gaap_OperatingIncomeLoss) are present; so they are not recalculated again.

By this way; you can upload any xbrl package from SEC(under UGT version 09,11,12).

 

Screen shot 2012-12-24 at 9.21.29 PM Screen shot 2012-12-24 at 9.22.10 PM

Also there might be some bug or error in doing analysis; if you find them then please inform me at ‘namitkewat@gmail.com’

Regarding my previous post that was Financial Analytics in XBRL was in pure-python with little bit ‘C’ touch (because of my base xml parsing lib is cElementTree). It was taking 20-21 mins to process 200 xbrl packages.

But thats not the end; I have optimized it with Cython. Its a C-extensions in python, giving multiple folds increase in the performance.
I have just converted my raw module to ‘.so’ and then imported it as usual; and i found 25% reduction in time. Its now near 14-15min for the same data set of 200 xbrl fillings.

See the screen-shot which i have just taken; 

Image
 

Here i have not implemented any type declaration or typed arguments; and yet i got 25% reduction in overall time. So you can imagine what will happen when i will implement those changes.
Anyway; ‘C’ rocks!!!!!

Hello all,

Just now i am completing my 8 months old experiment. It was to develop something with use of XBRL which can reduce the dependency on financial information which are processed by 3rd party.

I mean that the current dependency on Yahoo finance/Bloomberg/Google Finance/9W search/i-metrix_edgar-online/.. . etc for the financial information should not be there.

I am working as financial analyst doing XBRL tagging/verification/QC of financial information.

I thought if we can tag this information then we can use it for analytics purpose.

So lets build it. And now its almost done. I have just run my program on 200 companies and it took only 20-25 min to analyze them.
You can see the output in the link given below. Its for 200 SEC’s filling. (Not for entire quater; because it took me 3 hours to download them and just 20-25min to analyze them).

Download 200 filling’s analysed Data

I am covering 21 main financial ratios; and around 20-21 financial facts.

Ratios are also covered for dimension’s case.
For example company “OPLINK COMMUNICATIONS INC (0001022225)”‘s “Fixed turnover ratio” for “UnitedStatesMember” is “10.7756” while “ChinaMember” has “1.2279” for period of 2011-07-04/2012-07-01.

You can see the how much detail can be pulled out if XBRL document is well formed.

There was a time when it was taking more than 3 hours for 10% of current task. And at present it was just 20-25 mins on my Intel core2 duo/2GB RAM. Even then my instinct tells me that it can go down further.

Well to do that; i am really thankful of “Python”. It was great experience for me. Entire coding was done in python only.
I hope what python has given in field of Genome sequencing/Astronomy/ and Physics will be repeated in finance as well.

Scope of such program is in big data centers to small investors; like stock-exchange,financial information site like google-finance where authorities has to perform custom/personalized calculations on the financial data; also end user like you and me.

If you have more formulas to include that can be easily calculated from XBRL then please mention it as a comment below;

For more detailed study download this text file which contains the analysis of Google’s and 3M’s filling by my program: Google and 3M

Any query regarding my development; then email me at “namitkewat@gmail.com”

and any suggestion for current work??

I was just thinking that what i can do with my xbrlMapper module, so i thought lets try to make something  like arelle’s fact-list(3rd and last image of this post; ‘www.arelle.org”).

I have extended my present library; added new method “reportGen”; a methods with returns dicts of instance document’s facts; and i really love dicts; specially after watching pycon-2010 talk on dict “The Mighty Dictionary“.

See the snaps i am uploading to this post. Though the speed of process is fast but i am not satisfied because if it now takes 2.5sec for one xbrl package then how much time it will take for 20,000 xbrl packages!!!!

You know when i have started working on this , it has taken 71 seconds in first attempt(in evening of day one), it was like i am in stone-age  #$@#^@%#&.
So i have decided to improve it, then 25 sec, 20 sec(7 PM.. .of day one) then 35 sec- 40 sec(11 PM.. day 1).. .lollzz.. that evening was very funny, all time it was moving around 30 seconds!!! and then suddenly 9.5 sec(at 00:30AM of day2), and then 4.5 sec(aaa…5:30AM day 2), then finally its near 2-2.5 sec(..day 2..7:30AM.. then i left it because i have to go to my office), but my intuition is telling me that we can reduce it even further.

Here in fist image; its snap-shot of Arelle’s fact-list, in second  shows amount of python code i have to write to achieve this, and finally in last image… final output, its a sqlite db view in firefox plugin.

I got a log file of 131 GB from Mongo-DB that has been collected in last 6 months. So I just thought lets calculate the number of lines.

Here files size is greater than RAM, so after searching over internet; i got solution on http://stackoverflow.com.

I have modified it a little bit! But work is done.

It took around 75 min. and total number of lines was around 2.1 Billion.

Today in office or currently in office; (both are right…)

i am making python script that can automate calculation of decimal value from scale for various cases of xbrl instance facts.

Here below, i am giving my current output.

(scale, excel value) (decimal, instance value)

(None, ‘2345’) (‘INF’, ‘2345’)
(‘3′, ‘439.10’) (-1, ‘439100’)
(‘-4′, ‘1789000’) (4, ‘178.9’)
(‘-3′, ‘0.006’) (6, ‘0.000006’)
(‘0′, ‘3987’) (0, ‘3987’)
(‘0′, ‘0.0660’) (4, ‘0.066’)
(‘2′, ‘100’) (-2, ‘10000’)
(‘2′, ‘100.0’) (-1, ‘10000’)
(‘2′, ‘100.00’) (0, ‘10000’)
(‘3′, ‘100.660’) (0, ‘100660’)
(‘6′, ‘2.5’) (-5, ‘2500000’)
(‘6′, ‘2.50’) (-4, ‘2500000’)
(‘6′, ‘2.500’) (-3, ‘2500000’)
(‘6′, ‘2.540’) (-3, ‘2540000’)
(‘-1′, ‘0.1’) (2, ‘0.01’)
(‘-2′, ‘0.12’) (4, ‘0.0012’)
(‘-3′, ‘0.123’) (6, ‘0.000123’)
(‘9′, ‘10.5’) (-8, ‘10500000000’)
(‘9′, ‘105’) (-9, ‘105000000000’)
(‘0′, ‘0.02’) (2, ‘0.02’)
(None, ‘0.02’) (‘INF’, ‘0.02’)

 

Here in 1st tuple(bracket); 1st value is the scale i am applying on cell value on excel, and that value is on 2nd position of this tuple.

and in second bracket(tuple), its what the heading of this post.

1st value of second tuple is its decimal value, while 2nd value is the value which i will send in instance document.

Well here i have also tried on negative scales as well.

It will remove the headache for analyst to worry about decimals (or “precision”; what Bowne guys have on their “Bowne tagger”) ; he/she has to concern only about the scale value of fact, if it has any scale then apply it, program will intelligently calculate the decimal value, and if no scale is applied then it will put “INF” as a decimal for all numeric itemTypes.

But let me check on live files.
If you find any mistake please feel free to inform me about that.

 

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: