Apache NiFi
Apache NiFi is a powerful and scalable open-source dataflow automation tool developed by Apache Software Foundation. It allows users to design complex workflow for collecting, transforming and delivering data between systems. Using a drag and drop interface, users create visual pipelines by connecting processors, each performing a specific task like filtering, enriching or routing data. This low-code approach enables teams to build flows without extensive programming.
NiFi supports multiple data formats such as JSON, Avro, XML etc. and communication portocols (HTTP, kafka, MQTT, etc.). It offers key features such as guaranteed delivery, back pressure control and detailed data provenance for end-to-end traceability.
It integrates seamlessly with cloud platforms, big data tools such as Hadoop and messaging systems, while also providing robust security through SSL, user authentification, and role-based control.
- Learn more
- Official website
Related articles
MiNiFi: Data at Scales & the Values of Starting Small
Categories: Big Data, DevOps & SRE, Infrastructure | Tags: MiNiFi, C++, HDF, NiFi, Cloudera, HDP, IOT
This conference presented rapidly Apache NiFi and explained where MiNiFi came from: basically it’s a NiFi minimal agent to deploy on small devices to bring data to a cluster’s NiFi pipeline (ex: IoT…
Jul 8, 2017
Apache Metron in the Real World
Categories: Cyber Security, DataWorks Summit 2018 | Tags: Algorithm, Solr, Storm, pcap, RDBMS, HDFS, Kafka, Metron, NiFi, Spark, Data Science, Elasticsearch, SQL
Apache Metron is a storage and analytic platform specialized in cyber security. This talk was about demonstrating the usages and capabilities of Apache Metron in the real world. The presentation was…
May 29, 2018
Data Lake ingestion best practices
Categories: Big Data, Data Engineering | Tags: Data Governance, HDF, Operation, Avro, Hive, NiFi, ORC, Spark, Data Lake, File Format, Protocol Buffers, Registry, Schema
Creating a Data Lake requires rigor and experience. Here are some good practices around data ingestion both for batch and stream architectures that we recommend and implement with our customers…
By David WORMS
Jun 18, 2018
Connecting to ADLS Gen2 from Hadoop (HDP) and Nifi (HDF)
Categories: Big Data, Cloud Computing, Data Engineering | Tags: Hadoop, HDFS, NiFi, Authentication, Authorization, Azure, Azure Data Lake Storage (ADLS), OAuth2
As data projects built in the Cloud are becoming more and more frequent, a common use case is to interact with Cloud storage from an existing on premise Big Data platform. Microsoft Azure recently…
Nov 5, 2020
CDP part 6: end-to-end data lakehouse ingestion pipeline with CDP
Categories: Big Data, Data Engineering, Learning | Tags: Business intelligence, Data Engineering, Iceberg, NiFi, Spark, Big Data, Cloudera, CDP, Data Analytics, Data Lake, Data Warehouse
In this hands-on lab session we demonstrate how to build an end-to-end big data solution with Cloudera Data Platform (CDP) Public Cloud, using the infrastructure we have deployed and configured over…
Jul 24, 2023
