Skip to content

Nodejs creating large excel files

February 19, 2015
tags: ,

We wanted to created excel files that is having a few hundred thousand records – unfortunately the plugin msexcel builder did not work for us. other alternatives were to create a tabbed, or comma separated csv file and then open it in xls.

the msexcel builder plugin: had this very interesting concept of creating xls files using the concept of open office XML.

The only problem was that it was writing out all content at once, causing memory problem as data was big.  So we modified this to do appendFile instead of using writeFile. We also had to do away with the concept of sharedStrings, because when data is big this array itself would have costed us memory.

On testing with few thousand data, we also found traces of xml-builder objects lingering in memory – when we checked through ::findjsobjects in mdb(smartos) , So in the appending of content which was happening frequently we removed usage of xmlbuilder and stuck to plain old xml string formation. (Even though it looks less appealing it improved speed and reduced memory usage).

Next bottleneck that came was in usage of EasyZip, EasyZip – does full file read and writes out the binary content in the zip file which is actually the xlsx. So for this , the alternative we thought of was to use the OS specific zip command so that in memory usage reduced.

No comments yet

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: