Apache Hadoop
Hadoop is massively scalable platform commonly used to process big data workloads. At its core, it is composed of a distributed file system (HDFS) and a resource manager (YARN).
Hadoop provides a high level of durability and availability while still being able to process computational analytical workloads in parallel. The combination of availability, durability, and scalability of processing makes Hadoop a natural fit for Big Data workloads.
Related articles
H2O in practice: a protocol combining AutoML with traditional modeling approaches
Categories: Data Science, Learning | Tags: On-premises, H2O, Automation, XGBoost, Cloud, Python, Machine Learning, Open source, MLOps
H20 comes with a lot of functionalities. The second part of the series H2O in practice proposes a protocol to combine AutoML modeling with traditional modeling and optimization approach. The objective…
Nov 12, 2021
Internship in Big Data infrastructure with TDP
Categories: Learning, Infrastructure | Tags: Hadoop, TDP, Cyber Security, Java, DevOps, IaC
Job Description Big Data and distributed computing is at Adaltas’ core. We support our partners in the deployment, maintenance and optimization of some of France’s largest clusters. Adaltas is also an…
By Daniel HARTY
Oct 25, 2021
Internship in Data Engineering
Categories: Learning, Front End | Tags: Monitoring, Metrics, Kubernetes, Delta Lake, Elasticsearch, Hive, Kafka, Streaming, IaC
Job Description Data is a valuable business asset. Some call it the new oil. The data engineer collects, transform and refine raw data into information that can be used by business analysts and data…
By David WORMS
Oct 25, 2021
H2O in practice: a Data Scientist feedback
Categories: Data Science, Learning | Tags: On-premises, H2O, Automation, Cloud, Python, Machine Learning, Open source, MLOps
Automated machine learning (AutoML) platforms are gaining popularity and becoming a new important tool in the data scientists’ toolbox. A few months ago, I introduced H2O, an open-source platform for…
Sep 29, 2021
Adaltas Summit 2021, 2nd edition in corsica
Categories: Learning, Adaltas Summit 2021 | Tags: Azure, Deep Learning, Hadoop, Ansible, Docker, Kubernetes, Spark, Terraform, Blockchain, Node.js
For its second edition, the whole Adaltas crew is gathering in Corsica for a whole week with 2 days dedicated to technology the 23rd and the 24th of september 2021. After a year and a half of sanitary…
By David WORMS
Sep 21, 2021
Storage size and generation time in popular file formats
Categories: Data Engineering, Data Science | Tags: Big Data, Avro, HDFS, JSON, ORC, Parquet, File Format, Hive, Data Lake
Choosing an appropriate file format is essential, whether your data transits on the wire or is stored at rest. Each file format comes with its own advantages and disadvantages. We covered them in a…
Mar 22, 2021
Build your open source Big Data distribution with Hadoop, HBase, Spark, Hive & Zeppelin
Categories: Infrastructure, Big Data | Tags: Unit tests, Hadoop, HBase, TDP, Release and features, Spark, Maven, Hive, Git
The Hadoop ecosystem gave birth to many popular projects including HBase, Spark and Hive. While technologies like Kubernetes and S3 compatible object storages are growing in popularity, HDFS and YARN…
Dec 18, 2020
Faster model development with H2O AutoML and Flow
Categories: Data Science, Learning | Tags: On-premises, H2O, Automation, Cloud, Python, Machine Learning, Open source, MLOps
Building Machine Learning (ML) models is a time-consuming process. It requires expertise in statistics, ML algorithms, and programming. On top of that, it also requires the ability to translate a…
Dec 10, 2020
Connecting to ADLS Gen2 from Hadoop (HDP) and Nifi (HDF)
Categories: Data Engineering, Big Data, Cloud Computing | Tags: Azure, Hadoop, Azure Data Lake Storage (ADLS), Authentication, HDFS, OAuth2, NiFi, Authorization
As data projects built in the Cloud are becoming more and more frequent, a common use case is to interact with Cloud storage from an existing on premise Big Data platform. Microsoft Azure recently…
Nov 5, 2020
Rebuilding HDP Hive: patch, test and build
Categories: Infrastructure, Big Data | Tags: Unit tests, TDP, Release and features, Java, Maven, Hive, GitHub, Git
The Hortonworks HDP distribution will soon be deprecated in favor of Cloudera’s CDP. One of our clients wanted a new Apache Hive feature backported into HDP 2.6.0. We thought it was a good opportunity…
Oct 6, 2020
Installing Hadoop from source: build, patch and run
Categories: Infrastructure, Big Data | Tags: Unit tests, Hadoop, Docker, TDP, HDFS, LXD, Java, Maven
Commercial Apache Hadoop distributions have come and gone. The two leaders, Cloudera and Hortonworks, have merged: HDP is no more and CDH is now CDP. MapR has been acquired by HP and IBM BigInsights…
Aug 4, 2020
Download datasets into HDFS and Hive
Categories: Data Engineering, Big Data | Tags: Hadoop, Big Data, Data structures, HDFS, Business intelligence, Data Engineering, Database, Hive, Data Analytics, Data Warehouse, Data Lake
Introduction Nowadays, the analysis of large amounts of data is becoming more and more possible thanks to Big data technology (Hadoop, Spark,…). This explains the explosion of the data volume and the…
By Aida NGOM
Jul 31, 2020
Comparaison of different file formats in Big Data
Categories: Data Engineering, Big Data | Tags: Protocol Buffers, Batch processing, Big Data, Avro, Data structures, Kubernetes, HDFS, JSON, Business intelligence, CSV, ORC, Parquet
In data processing, there are different types of files formats to store your data sets. Each format has its own pros and cons depending upon the use cases and exists to serve one or several purposes…
By Aida NGOM
Jul 23, 2020
Optimization of Spark applications in Hadoop YARN
Categories: Data Engineering, Learning | Tags: Hadoop, Spark, Python, Tuning
Apache Spark is an in-memory data processing tool widely used in companies to deal with Big Data issues. Running a Spark application in production requires user-defined resources. This article…
Mar 30, 2020
Should you move your Big Data and Data Lake to the Cloud
Categories: Big Data, Cloud Computing | Tags: Azure, AWS, Cloud, Databricks, GCP, CDP, DevOps
Should you follow the trend and migrate your data, workflows and infrastructure to GCP, AWS and Azure? During the Strata Data Conference in New-York, a general focus was put on moving customer’s Big…
Dec 9, 2019
Internship Data Science & Data Engineer - ML in production and streaming data ingestion
Categories: Data Engineering, Data Science | Tags: Hadoop, HBase, Kubernetes, Spark, Python, Flink, DevOps, Kafka
Context The exponential evolution of data has turned the industry upside down by redefining data storage, processing and data ingestion pipelines. Mastering these methods considerably facilitates…
By David WORMS
Nov 26, 2019
InfraOps & DevOps Internship - build a Big Data & Kubernetes PaaS
Categories: Big Data, Containers Orchestration | Tags: Hadoop, Kubernetes, Spark, LXD, NoSQL, DevOps, Kafka, Ceph
Context The acquisition of a high-capacity cluster is in line with Adaltas’ desire to build a PAAS-type offering to use and to provide Big Data and container orchestration platforms. The platforms are…
By David WORMS
Nov 26, 2019
Machine Learning model deployment
Categories: Data Engineering, Data Science, DevOps & SRE, Big Data | Tags: On-premises, AI, Cloud, Machine Learning, Operation, DevOps, MLOps, Schema
“Enterprise Machine Learning requires looking at the big picture […] from a data engineering and a data platform perspective,” lectured Justin Norman during the talk on the deployment of Machine…
Sep 30, 2019
Running Apache Hive 3, new features and tips and tricks
Categories: Big Data, DataWorks Summit 2019, Business Intelligence | Tags: Hadoop, LLAP, Release and features, JDBC, Hive, Druid, Kafka
Apache Hive 3 brings a bunch of new and nice features to the data warehouse. Unfortunately, like many major FOSS releases, it comes with a few bugs and not much documentation. It is available since…
Jul 25, 2019
Spark Streaming part 2: run Spark Structured Streaming pipelines in Hadoop
Categories: Data Engineering, Learning | Tags: Spark, Apache Spark Streaming, Python, Streaming
Spark can process streaming data on a multi-node Hadoop cluster relying on HDFS for the storage and YARN for the scheduling of jobs. Thus, Spark Structured Streaming integrates well with Big Data…
May 28, 2019
Publish Spark SQL DataFrame and RDD with Spark Thrift Server
Categories: Data Engineering | Tags: Hadoop, Spark, Thrift, JDBC, SQL, Hive
The distributed and in-memory nature of the Spark engine makes it an excellent candidate to expose data to clients which expect low latencies. Dashboards, notebooks, BI studios, KPIs-based reports…
Mar 25, 2019
Multihoming on Hadoop
Categories: Infrastructure | Tags: Hadoop, HDFS, Network, Kerberos
Multihoming, which means having multiple networks attached to one node, is one of the main components to manage the heterogeneous network usage of an Apache Hadoop cluster. This article is an…
Mar 5, 2019
Apache Knox made easy!
Categories: Cyber Security, Big Data, Adaltas Summit 2018 | Tags: Ranger, REST, Active Directory, LDAP, Kerberos, Knox
Apache Knox is the secure entry point of a Hadoop cluster, but can it also be the entry point for my REST applications? Apache Knox overview Apache Knox is an application gateway for interacting in a…
Feb 4, 2019
Monitoring a production Hadoop cluster with Kubernetes
Categories: DevOps & SRE | Tags: Hadoop, Docker, Kubernetes, Cluster, Shinken, Thrift, Python, Elasticsearch, Node, Prometheus, Knox, Node.js, Graphana
Monitoring a production grade Hadoop cluster is a real challenge and needs to be constantly evolving. The software we use today is based on Nagios. Very efficient when it comes to the simplest…
Dec 21, 2018
One week to discuss technology in a Moroccan riad
Categories: Learning, Adaltas Summit 2018 | Tags: Deep Learning, Hadoop, Data Science, Kubernetes, Gatsby, React.js, Flink, CDSW, Knox, Node.js
Adaltas organise the year its first conference between the 22 and 26 of October. On the agenda of these 5 days of conference: discuss technology in one of the most beautiful riad of Marrakech. Mix the…
By David WORMS
Oct 11, 2018
Clusters and workloads migration from Hadoop 2 to Hadoop 3
Categories: Infrastructure, Big Data | Tags: YARN, Docker, Erasure Coding, HDFS, Spark, Rolling Upgrade, Slider
Hadoop 2 to Hadoop 3 migration is a hot subject. How to upgrade your clusters, which features present in the new release may solve current problems and bring new opportunities, how are your current…
Jul 25, 2018
Deep learning on YARN: running Tensorflow and friends on Hadoop cluster
Categories: Data Science | Tags: Deep Learning, Hadoop, YARN, XGBoost, PyTorch, Spark, Spark MLlib, MXNet, GPU, TensorFlow
With the arrival of Hadoop 3, YARN offer more flexibility in resource management. It is now possible to perform Deep Learning analysis on GPUs with specific development environments, leveraging…
Jul 24, 2018
Apache Hadoop YARN 3.0 – State of the union
Categories: DataWorks Summit 2018, Big Data | Tags: Hadoop, YARN, MapReduce, Docker, Hortonworks, Data Science, Release and features, HDFS, Cloudera, GPU
This article covers the ”Apache Hadoop YARN: state of the union” talk held by Wangda Tan from Hortonworks during the Dataworks Summit 2018. What is Apache YARN? As a reminder, YARN is one of the two…
May 31, 2018
Running Enterprise Workloads in the Cloud with Cloudbreak
Categories: DataWorks Summit 2018, Big Data, Cloud Computing | Tags: Azure, Hadoop, AWS, Operation, HDP, GCP, OpenStack, Cloudbreak
This article is based on Peter Darvasi and Richard Doktorics’ talk Running Enterprise Workloads in the Cloud at the DataWorks Summit 2018 in Berlin. It presents Hortonworks’ automated deployment tool…
May 28, 2018
Present and future of Hadoop workflow scheduling: Oozie 5.x
Categories: DataWorks Summit 2018, Big Data | Tags: Hadoop, REST, Sqoop, Oozie, HDP, CDH, Hive
During the DataWorks Summit Europe 2018 in Berlin, I had the opportunity to attend a breakout session on Apache Oozie. It covers the new features released in Oozie 5.0, including future features of…
May 23, 2018
MariaDB integration with Hadoop
Categories: Infrastructure | Tags: Hadoop, Database, MariaDB, HA, Hive
During a workshop with one of our customers, Adaltas has identified a potential risk to use MariaDB’s High Availability (HA) strategy. Since the customer selected Cloudera’s CDH 5 distribution, the…
By David WORMS
Jul 31, 2017
Apache Apex: next gen Big Data analytics
Categories: Data Science, Tech Radar, Events | Tags: Hadoop, Storm, Data Science, Apex, Tools, Machine Learning, Flink, Kafka
Below is a compilation of my notes taken during the presentation of Apache Apex by Thomas Weise from DataTorrent, the company behind Apex. Introduction Apache Apex is an in-memory distributed parallel…
Jul 17, 2016
Apache Apex with Apache SAMOA
Categories: Data Science, Tech Radar, Events | Tags: Samoa, Hadoop, Storm, Apex, Tools, Machine Learning, Flink
Traditional Machine Learning Batch Oriented Supervised - most common Training and Scoring One time model building Data set Training: Model building Holdout: Paremeter tuning Test: Accuracy Online…
Jul 17, 2016
Hive, Calcite and Druid
Categories: Big Data | Tags: Hadoop, Business intelligence, Database, Hive, Druid
BI/OLAP requires interactive visualization of complex data streams: Real time bidding events User activity streams Voice call logs Network trafic flows Firewall events Application KPIs Traditionnal…
By David WORMS
Jul 14, 2016
Red Hat Storage Gluster and its integration with Hadoop
Categories: Big Data | Tags: Hadoop, HDFS, GlusterFS, Storage, Red Hat
I had the opportunity to be introduced to Red Hat Storage and Gluster in a joint presentation by Red Hat France and the company StartX. I have here recompiled my notes, at least partially. I will…
By David WORMS
Jul 3, 2015
Composants for CDH and HDP
Categories: Big Data | Tags: Hadoop, Zookeeper, Hortonworks, Sqoop, Oozie, Flume, Cloudera, HDP, CDH, Hive
I was interested to compare the different components distributed by Cloudera and HortonWorks. This also gives us an idea of the versions packaged by the two distributions. At the time of this writting…
By David WORMS
Sep 22, 2013
State of the Hadoop open-source ecosystem in early 2013
Categories: Big Data | Tags: Mesos, Hadoop, Pig, Data Science, Phoenix, Flume, Mahout, Kafka
Hadoop is already a large ecosystem and my guess is that 2013 will be the year where it grows even larger. There are some pieces that we no longer need to present. ZooKeeper, hbase, Hive, Pig, Flume…
By David WORMS
Jul 8, 2013
The state of Hadoop distributions
Categories: Big Data | Tags: Hadoop, Hortonworks, Intel, Oracle, Cloudera
Apache Hadoop is of course made available for download on its official webpage. However, downloading and installing the several components that make a Hadoop cluster is not an easy task and is a…
By David WORMS
May 11, 2013
Apache Hive Essentials How-to by Darren Lee
Categories: Learning, Business Intelligence | Tags: Hadoop, File Format, SQL, Hive, UDF
Recently, I’ve been ask to review a new book on Apache Hive called “Apache Hive Essentials How-to” (edit: the second edition is now available) written by Darren Lee and published by Packt Publishing…
By David WORMS
Apr 23, 2013
Definitions of machine learning algorithms present in Apache Mahout
Categories: Data Science | Tags: Hadoop, Machine Learning, Algorithm, Сlassification, Mahout, Clustering
Apache Mahout is a machine learning library built for scalability. Its core algorithms for clustering, classfication and batch based collaborative filtering are implemented on top of Apache Hadoop…
By David WORMS
Mar 8, 2013
Merging multiple files in Hadoop
Categories: Hack | Tags: Hadoop, HDFS, File system
This is a command I used to concatenate the files stored in Hadoop HDFS matching a globing expression into a single file. It uses the “getmerge” utility of but contrary to “getmerge”, the final…
By David WORMS
Jan 12, 2013
Hadoop and R with RHadoop
Categories: Data Science, Business Intelligence | Tags: Learning and tutorial, R, Hadoop, HBase, MapReduce, HDFS, Thrift, Data Analytics
RHadoop is a bridge between R, a language and environment to statistically explore data sets, and Hadoop, a framework that allows for the distributed processing of large data sets across clusters of…
By David WORMS
Jul 19, 2012
Timeseries storage in Hadoop and Hive
Categories: Data Engineering | Tags: Hadoop, CRM, timeseries, HDFS, File Format, Hive, Tuning
In the next few weeks, we will be exploring the storage and analytic of a large generated dataset. This dataset is composed of CRM tables associated to one timeserie table of about 7,000 billiard rows…
By David WORMS
Jan 10, 2012
Hadoop and HBase installation on OSX in pseudo-distributed mode
Categories: Learning, Big Data | Tags: Hadoop, HBase, Big Data, Deployment, Hue, Infrastructure
The operating system chosen is OSX but the procedure is not so different for any Unix environment because most of the software is downloaded from the Internet, uncompressed and set manually. Only a…
By David WORMS
Dec 1, 2010
Storage and massive processing with Hadoop
Categories: Big Data | Tags: Hadoop, HDFS, Storage
Apache Hadoop is a system for building shared storage and processing infrastructures for large volumes of data (multiple terabytes or petabytes). Hadoop clusters are used by a wide range of projects…
By David WORMS
Nov 26, 2010