Node CSV version 0.1 and future developments

David WORMS

By David WORMS

Jul 21, 2012

The Node CSV parser has just reach version 0.1 which close the 0.0.x releases. Started almost 2 years ago, the project has received a tremendous amount of participation in the form of bug reports, pull request and emails. It is used by a large population of the NodeJs community and could now be considered as stable.

Well, this isn’t the best news. The really awesome news is that I have now some time to start the development of the v0.2 branch. This paves the way to some exciting developments which I am going to present in more details.

Backward compatibility

We say in french that we can’t prepare scramble eggs without breaking some eggs. So yes, the API will change, for the best. Since the beginning of the project, it felt wrong and the incomplete support of the stream API makes it non standard at best and, to be honest, hard to understand and use properly.

If your project rely on the 0.0.x or 0.1.x versions, don’t feel nervous. We will maintain support and any bug that we found will be ported to those versions.

The new Stream API

This little schema illustrates the structure of the future code:

 |------------|        |------------|------------|         |------------|
 |            |        |            |            |         |            |
 |  csv.from  |        |           CSV           |         |   csv.to   |
 |            |        |            |            |         |            |
 |   Stream   |        |   Writer   |   Reader   |         |   Stream   |
 |   Reader   | .pipe( |    API     |    API     | ).pipe( |   Writer   | )
 |            |        |            |            |         |            |
 |            |        |            |            |         |            |
 |------------|        |------------|------------|         |------------|

A new CSV instance will be fully compliant with the stream API. It will be both an Stream Writer to send input data and a Stream Reader to access output data.

Example:

fs.createReadStream( './in' )
.pipe( csv() )
.pipe( fs.createWriteStream('./out') )

Convenient functionalities

Alternatively, it will come with convenient functions accessible by the from and to properties. Some of those functions were already present in the 0.1 release and are simply renamed. For exemple, the csv.fromPath() function is now csv.from.path(). New functions are planned such a the csv.to.string and the csv.to.array.

Example:

csv()
.from.path( './in' )
.to.path( './out' )

Code isolation

Until now, all the source code present inside a single file. From now on, the code layout will be:

  • ”./lib/from.js” Input convenient functions
  • ”./lib/to.js” Output convenient functions
  • ”./lib/csv.js” Core function implement the Stream API
  • ”./lib/options.js” Default input and output options
  • ”./lib/state.js” Default state object

This refactoring shall simplify the understanding of the source code and help new comers to prepare pull request which are particularly welcome.

Documentation

Like I have done in the past in many projects like Mecano (now Nikita), the readme content will be reduced to a minimum and the documentation will be generated directly from the source code. A small script will be written specifically from that purpose. The idea is to document each function with comments written in a markdown syntax. A simple regexp parser read each files, extract the comment and write markdown file inside a “./doc” folder. The doc folder is finally copied into the Jekyll directory of this website before being uploaded.

I find it much easier to document a project on a per function basis than by maintaining coherent a doc forlder or a large readme file.

CoffeeScript

This project was probably the last one I could claim to be written in JavaScript. I’m not here to start a war. I didn’t choose CoffeeScript to be cool, I use it for a year and a half now, and I find writting old JS to be a pain. The test have be translated some time ago but the production code was still in JavaScript. Starting from version 0.2.0, the code will be maitain in CoffeeScript. The source file will be located inside the “./src” folder and the transpiled JavaScript will be located inside the “./lib” folder.

Conclusion

It is an excellent time to watch the development of this new version and come with your own feedback. I’ll be happy to listen to your suggestions.

Canada - Morocco - France

International locations

10 rue de la Kasbah
2393 Rabbat
Canada

We are a team of Open Source enthusiasts doing consulting in Big Data, Cloud, DevOps, Data Engineering, Data Science…

We provide our customers with accurate insights on how to leverage technologies to convert their use cases to projects in production, how to reduce their costs and increase the time to market.

If you enjoy reading our publications and have an interest in what we do, contact us and we will be thrilled to cooperate with you.