Skip to main content

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)
The news is out, Apache MXNet has added a Java API. So as soon as I could I got my hands on the maven repo and an example program and got to work writing a new Apache NiFi processor for it.
I have run this on standalone Apache NiFi 1.8.0 and on HDF 3.3 - Apache NiFi 1.8.0 and both work. So anyone who wants to be an alpha tester, please download it and give it a try.
Apache MXNet SSD is a good example of a pretrained deep learning model that works pretty well for general images in a use cases especially around people and cars. You can fine-tune this with some more images and runs:
The nice thing is now we can start including Apache MXNet as part of Java applications such as Kafka Streams, Apache Storm, Apache Spark, Spring Boot and other use cases using Java. I could potentially inject this into a Hive UDF ( or Pig UDF. The performance may be fast enough. We now have four Java options for Deep Learning: DL4J, H2O, Tensorflow and Apache MXNet. Unfortunately, both TensorFlow and MXNet Java APIs are not quite production ready.
I may do some further research on running MXNet as a Hive UDF, it would be cool to have in a query.
For those who don't want to setup a development environment with JDK 8+, Maven 3.3+ and git, you can download a pre-built nar file here:
As part of the recent release of HDF 3.3, I have upgraded my OpenStack Centos 7 cluster.
Important CaveatsNotice, the Java API is in preview and so is this processor. Do not use this in production! This is in development and I am the only one working on it. The Java API from Apache MXNet is in flux and will be changing. See the POM as it is tied to the OSX/Mac version of the library. You will need to change that. You will need to download the pre-built MXNet model and place it in a directory accessible to Apache NiFi server/cluster. I am still cleaning up the rectangle code for identifying objects in the pictures.
As you will notice, my rectangle drawing is a bit off. I need to work on that.
Once you drop your built nar file and models in the nifi/lib directory and restart Apache NiFi, you can add it to your canvas.
We need to feed it some images. You can use my web cam processor, an image URL feed or local files.
To grab images from an HTTPS site, you need an SSL Context Service like this StandardSSLContextService below. You will need to point to the cacerts used by the JRE/JDK running your Apache NiFi node. The default password in Java is changeme. Hopefully you have changed it.
To configure my new processor, just put in the full path to the model directory and then "/resnet50_ssd_model" as that is the prefix for the model.
Our example flow with new processor being fed by traffic cameras, webcams, local files and local webcam.
Some output of our flow:
Our top 5 probabilities and labels
Example Data:

  1. {

  2. "ymin_1" : "456.01",

  3. "ymin_5" : "159.29",

  4. "ymin_4" : "235.83",

  5. "ymin_3" : "206.64",

  6. "ymin_2" : "383.84",

  7. "label_5" : "person",

  8. "xmax_5" : "121.14",

  9. "label_4" : "bicycle",

  10. "xmax_4" : "137.89",

  11. "label_3" : "dog",

  12. "xmax_3" : "179.14",

  13. "ymax_1" : "150.66",

  14. "ymax_2" : "418.95",

  15. "ymax_3" : "476.79",

  16. "label_2" : "bicycle",

  17. "label_1" : "car",

  18. "probability_4" : "0.22",

  19. "probability_5" : "0.13",

  20. "probability_2" : "0.90",

  21. "xmin_5" : "88.93",

  22. "probability_3" : "0.82",

  23. "ymax_4" : "413.43",

  24. "probability_1" : "1.00",

  25. "ymax_5" : "190.04",

  26. "xmax_2" : "149.96",

  27. "xmax_1" : "72.03",

  28. "xmin_3" : "83.82",

  29. "xmin_4" : "93.05",

  30. "xmin_1" : "312.21",

  31. "xmin_2" : "155.96"

  32. }

Maven POM (I used Java 8 and Maven 3.3.9)

  1. <?xml version="1.0" encoding="UTF-8"?>

  2. <!--

  3. Licensed to the Apache Software Foundation (ASF) under one or more

  4. contributor license agreements. See the NOTICE file distributed with

  5. this work for additional information regarding copyright ownership.

  6. The ASF licenses this file to You under the Apache License, Version 2.0

  7. (the "License"); you may not use this file except in compliance with

  8. the License. You may obtain a copy of the License at


  10. Unless required by applicable law or agreed to in writing, software

  11. distributed under the License is distributed on an "AS IS" BASIS,

  12. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  13. See the License for the specific language governing permissions and

  14. limitations under the License.

  15. -->

  16. <project xmlns="" xmlns:xsi="" xsi:schemaLocation="">

  17. <modelVersion>4.0.0</modelVersion>



  20. <parent>

  21. <groupId>com.dataflowdeveloper.mxnet</groupId>

  22. <artifactId>inference</artifactId>

  23. <version>1.0</version>

  24. </parent>



  27. <artifactId>nifi-mxnetinference-processors</artifactId>

  28. <packaging>jar</packaging>



  31. <dependencies>

  32. <dependency>

  33. <groupId>org.apache.nifi</groupId>

  34. <artifactId>nifi-api</artifactId>

  35. </dependency>

  36. <dependency>

  37. <groupId>org.apache.nifi</groupId>

  38. <artifactId>nifi-utils</artifactId>

  39. <version>1.8.0</version>

  40. </dependency>

  41. <dependency>

  42. <groupId>org.apache.nifi</groupId>

  43. <artifactId>nifi-mock</artifactId>

  44. <version>1.8.0</version>

  45. <scope>test</scope>

  46. </dependency>

  47. <dependency>

  48. <groupId>org.slf4j</groupId>

  49. <artifactId>slf4j-simple</artifactId>

  50. <scope>test</scope>

  51. </dependency>

  52. <dependency>

  53. <groupId>junit</groupId>

  54. <artifactId>junit</artifactId>

  55. <scope>test</scope>

  56. </dependency>



  59. <dependency>

  60. <groupId>org.apache.mxnet</groupId>

  61. <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>

  62. <version>1.3.1-SNAPSHOT</version>

  63. </dependency>



  66. </dependencies>

  67. </project>

I have moved from Eclipse to IntelliJ from my builds. I am looking at Apache Netbeans as well.

Popular posts from this blog

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice In Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP. In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi. This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones. Drone Live Communications for Sensor Readings and Drone Control You must connect t

NiFi on Cloudera Data Platform Upgrade - April 2021

CFM 2.1.1 on CDP 7.1.6 There is a new Cloudera release of Apache NiFi now with SAML support. Apache NiFi Apache NiFi Registry See:   For changes: Get your download on: To start researching for the future, take a look at some of the technical preview features around Easy Rules engine and handlers. Make sure you use the latest possible JDK 8 as there are some bugs out there.   Use a recent v

Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway

Using Apache NiFi in OpenShift and Anywhere Else to Act as Your Global Integration Gateway What does it look like? Where Can I Run This Magic Engine: Private Cloud, Public Cloud, Hybrid Cloud, VM, Bare Metal, Single Node, Laptop, Raspberry Pi or anywhere you have a 1GB of RAM and some CPU is a good place to run a powerful graphical integration and dataflow engine.   You can also run MiNiFi C++ or Java agents if you want it even smaller. Sounds Too Powerful and Expensive: Apache NiFi is Open Source and can be run freely anywhere. For What Use Cases: Microservices, Images, Deep Learning and Machine Learning Models, Structured Data, Unstructured Data, NLP, Sentiment Analysis, Semistructured Data, Hive, Hadoop, MongoDB, ElasticSearch, SOLR, ETL/ELT, MySQL CDC, MySQL Insert/Update/Delete/Query, Hosting Unlimited REST Services, Interactive with Websockets, Ingesting Any REST API, Natively Converting JSON/XML/CSV/TSV/Logs/Avro/Parquet, Excel, PDF, Word Documents, Syslog, Kafka, JMS, MQTT, TCP