Skip to main content

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)

Apache NiFi Processor for Apache MXNet SSD: Single Shot MultiBox Object Detector (Deep Learning)
The news is out, Apache MXNet has added a Java API. So as soon as I could I got my hands on the maven repo and an example program and got to work writing a new Apache NiFi processor for it.
I have run this on standalone Apache NiFi 1.8.0 and on HDF 3.3 - Apache NiFi 1.8.0 and both work. So anyone who wants to be an alpha tester, please download it and give it a try.
Apache MXNet SSD is a good example of a pretrained deep learning model that works pretty well for general images in a use cases especially around people and cars. You can fine-tune this with some more images and runs:
The nice thing is now we can start including Apache MXNet as part of Java applications such as Kafka Streams, Apache Storm, Apache Spark, Spring Boot and other use cases using Java. I could potentially inject this into a Hive UDF ( or Pig UDF. The performance may be fast enough. We now have four Java options for Deep Learning: DL4J, H2O, Tensorflow and Apache MXNet. Unfortunately, both TensorFlow and MXNet Java APIs are not quite production ready.
I may do some further research on running MXNet as a Hive UDF, it would be cool to have in a query.
For those who don't want to setup a development environment with JDK 8+, Maven 3.3+ and git, you can download a pre-built nar file here:
As part of the recent release of HDF 3.3, I have upgraded my OpenStack Centos 7 cluster.
Important CaveatsNotice, the Java API is in preview and so is this processor. Do not use this in production! This is in development and I am the only one working on it. The Java API from Apache MXNet is in flux and will be changing. See the POM as it is tied to the OSX/Mac version of the library. You will need to change that. You will need to download the pre-built MXNet model and place it in a directory accessible to Apache NiFi server/cluster. I am still cleaning up the rectangle code for identifying objects in the pictures.
As you will notice, my rectangle drawing is a bit off. I need to work on that.
Once you drop your built nar file and models in the nifi/lib directory and restart Apache NiFi, you can add it to your canvas.
We need to feed it some images. You can use my web cam processor, an image URL feed or local files.
To grab images from an HTTPS site, you need an SSL Context Service like this StandardSSLContextService below. You will need to point to the cacerts used by the JRE/JDK running your Apache NiFi node. The default password in Java is changeme. Hopefully you have changed it.
To configure my new processor, just put in the full path to the model directory and then "/resnet50_ssd_model" as that is the prefix for the model.
Our example flow with new processor being fed by traffic cameras, webcams, local files and local webcam.
Some output of our flow:
Our top 5 probabilities and labels
Example Data:

  1. {

  2. "ymin_1" : "456.01",

  3. "ymin_5" : "159.29",

  4. "ymin_4" : "235.83",

  5. "ymin_3" : "206.64",

  6. "ymin_2" : "383.84",

  7. "label_5" : "person",

  8. "xmax_5" : "121.14",

  9. "label_4" : "bicycle",

  10. "xmax_4" : "137.89",

  11. "label_3" : "dog",

  12. "xmax_3" : "179.14",

  13. "ymax_1" : "150.66",

  14. "ymax_2" : "418.95",

  15. "ymax_3" : "476.79",

  16. "label_2" : "bicycle",

  17. "label_1" : "car",

  18. "probability_4" : "0.22",

  19. "probability_5" : "0.13",

  20. "probability_2" : "0.90",

  21. "xmin_5" : "88.93",

  22. "probability_3" : "0.82",

  23. "ymax_4" : "413.43",

  24. "probability_1" : "1.00",

  25. "ymax_5" : "190.04",

  26. "xmax_2" : "149.96",

  27. "xmax_1" : "72.03",

  28. "xmin_3" : "83.82",

  29. "xmin_4" : "93.05",

  30. "xmin_1" : "312.21",

  31. "xmin_2" : "155.96"

  32. }

Maven POM (I used Java 8 and Maven 3.3.9)

  1. <?xml version="1.0" encoding="UTF-8"?>

  2. <!--

  3. Licensed to the Apache Software Foundation (ASF) under one or more

  4. contributor license agreements. See the NOTICE file distributed with

  5. this work for additional information regarding copyright ownership.

  6. The ASF licenses this file to You under the Apache License, Version 2.0

  7. (the "License"); you may not use this file except in compliance with

  8. the License. You may obtain a copy of the License at


  10. Unless required by applicable law or agreed to in writing, software

  11. distributed under the License is distributed on an "AS IS" BASIS,

  12. WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

  13. See the License for the specific language governing permissions and

  14. limitations under the License.

  15. -->

  16. <project xmlns="" xmlns:xsi="" xsi:schemaLocation="">

  17. <modelVersion>4.0.0</modelVersion>



  20. <parent>

  21. <groupId>com.dataflowdeveloper.mxnet</groupId>

  22. <artifactId>inference</artifactId>

  23. <version>1.0</version>

  24. </parent>



  27. <artifactId>nifi-mxnetinference-processors</artifactId>

  28. <packaging>jar</packaging>



  31. <dependencies>

  32. <dependency>

  33. <groupId>org.apache.nifi</groupId>

  34. <artifactId>nifi-api</artifactId>

  35. </dependency>

  36. <dependency>

  37. <groupId>org.apache.nifi</groupId>

  38. <artifactId>nifi-utils</artifactId>

  39. <version>1.8.0</version>

  40. </dependency>

  41. <dependency>

  42. <groupId>org.apache.nifi</groupId>

  43. <artifactId>nifi-mock</artifactId>

  44. <version>1.8.0</version>

  45. <scope>test</scope>

  46. </dependency>

  47. <dependency>

  48. <groupId>org.slf4j</groupId>

  49. <artifactId>slf4j-simple</artifactId>

  50. <scope>test</scope>

  51. </dependency>

  52. <dependency>

  53. <groupId>junit</groupId>

  54. <artifactId>junit</artifactId>

  55. <scope>test</scope>

  56. </dependency>



  59. <dependency>

  60. <groupId>org.apache.mxnet</groupId>

  61. <artifactId>mxnet-full_2.11-osx-x86_64-cpu</artifactId>

  62. <version>1.3.1-SNAPSHOT</version>

  63. </dependency>



  66. </dependencies>

  67. </project>

I have moved from Eclipse to IntelliJ from my builds. I am looking at Apache Netbeans as well.

Popular posts from this blog

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice

Ingesting Drone Data From DJII Ryze Tello Drones Part 1 - Setup and Practice In Part 1, we will setup our drone, our communication environment, capture the data and do initial analysis. We will eventually grab live video stream for object detection, real-time flight control and real-time data ingest of photos, videos and sensor readings. We will have Apache NiFi react to live situations facing the drone and have it issue flight commands via UDP. In this initial section, we will control the drone with Python which can be triggered by NiFi. Apache NiFi will ingest log data that is stored as CSV files on a NiFi node connected to the drone's WiFi. This will eventually move to a dedicated embedded device running MiniFi. This is a small personal drone with less than 13 minutes of flight time per battery. This is not a commercial drone, but gives you an idea of the what you can do with drones. Drone Live Communications for Sensor Readings and Drone Control You must connect t

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HDFS / Kudu / File / Hive

Migrating Apache Flume Flows to Apache NiFi: Kafka Source to HDFS / Kudu / File / Hive Article 7 - Article 6 - Article 5 -  Article 4 - Article 3 - Article 2 - Article 1 - Source Code: This is one possible simple, fast replacement for " Flafka ". Consume / Publish Kafka And Store to Files, HDFS, Hive 3.1, Kudu Consume Kafka Flow   Merge Records And Store As AVRO or ORC Consume Kafka, Upda

Advanced XML Processing with Apache NiFi 1.9.1

Advanced XML Processing with Apache NiFi 1.9.1 With the latest version of Apache NiFi, you can now directly convert XML to JSON or Apache AVRO, CSV or any other format supported by RecordWriters.   This is a great advancement.  To make it even easier, you don't even need to know the schema before hand.   There is a built-in option to Infer Schema. The results of an RSS (XML) feed converted to JSON and displayed in a slack channel. Besides just RSS feeds, we can grab regular XML data including XML data that is wrapped in a Zip file (or even in a Zipfile in an email, SFTP server or Google Docs). Get the Hourly Weather Observation for the United States Decompress That Zip  Unpack That Zip into Files One ZIP becomes many XML files of data. An example XML record from a NOAA weather station. Converted to JSON Automagically Let's Read Those Records With A Query and Convert the results to JSON Records