What is Apache Slug?
Apache Slug continuously monitors microservices to provide real-time feedback on the inner workings of data movement, transformations, and advanced algorithmic computations. It is so thorough (monitoring at speeds anywhere from slow to really slow) that it’s the only monitoring system that gives you more time for coffee breaks.
Key properties of Apache Slug:
- Local storage and processes use Slime technology to smooth operations.
- Complete transparency into data processes, inching along and viewing each element of transformation that is being created.
- Uses the hermaphrodite design concept for maximum flexibility.
- Leverages a gastropodic data ingestion method (touch, ingest, digest in batch).
- Packaged with a full-set of RESTful API calls which integrates with other Apache projects such as Apache Ambari.
- Integration with farm-security projects for managing pens, e.g. Sentry or Ranger.
Apache Slug integration:
In a Slug-enabled environment, a new dataset generated using MapReduce on Hadoop or Scala on a Spark data frame will now give users the capability of executing this code in microbatches, even at the row-level.
- Return each microbatch of information.
- Return complete diagnostics.
- Return a prompt to the user before moving to the next step.
A simple diagram illustrating how Slug integrates in your data workflow: