EclairJS – Putting a Spark in Web Apps

Presentation by David Fallside from IBM, images extracted from the presentation.

Introduction

Web Apps development has moved from Java to NodeJS and Javascript. It provides a simple and rich environment with NPM.

EclairJS is a NodeJS library that provides bindings to a Spark application :

  • An RDD is bound to a JS object that is made immutable
  • Spark operators are transparently mapped to JS functions (ex: flatMap, filter, …)
  • Every Spark operator mapped returns a promise

The use of promises allows to emulate Spark’s use of the DAG :

  • Transformations return a new object and are added to the DAG
  • Actions executes the whole DAG to get a result

EclairJS - Code semantics

Architecture

EclairJS has two main components :

  • Client: JS API, installed with NPM
  • Server: JS providing Java mapping and able to run in the JVM using Oracle Nashorn, has to be run

The server also uses Jupyter Notebook to provide a WebSocket endpoint between client and server

EclairJS - Architecture

Performance

In terms of performances, Spark’s native Java API is way faster, however EclairJS is twice as fast as Spark’s PySpark API.

EclairJS - Performances

Conclusion

EclairJS seems to be a great project if you need to integrate Spark jobs into a web application.

By |2018-06-05T22:37:06+00:00July 17th, 2016|Categories: Events|0 Comments

About the Author:

César is a Big Data & Hadoop Solution Architect and Data Engineer with 2 years of hands-on experience in Hadoop and distributed systems. He has been designing, developing and maintaining data processing workflows and real-time services as well as bringing to clients a consistent vision on data management and workflows across their different data sources and business requirements. He steps in at all levels of the Data platforms, from planning, design and architecture to clusters deployment, administration, maintenance as well as prototyping and applications development in collaboration with business users, analysts, data scientists, engineering and operational teams. He enjoys discovering stuff and experimenting with new technologies in addition to his day to day work He also has a good experience as educator for knowledge transfer and training.

Leave A Comment