Why I don't use NIFI

First, I would like to be clear Nifi is a good tool. It meet it design initiative. It simplify development of what it designed for. But it is not a silver pullet.

In current software development, logic development is not the only thing. Finishing a business logic development does not means the project is finished. You must take testing, publishing, deploying, collaboration between different unit, security, change tracking etc into consideration. Unless the project is a one shot, and you will never touch it again.

So, here is the point that I cannot use Nifi in my production project.

• Testing I need all changes and all business logic and pipeline connection to be testable and also in the automatic way in flexible environment. So far Nifi does not have a good solution in this area. Someone (https://touk.pl/blog/2019/01/10/testing-nifi-flow-the-good-the-bad-and-the-ugly/) has put a great effort in it, but it is still not in the level that I expected.

• Security and access control

  Nifi having some basic access control feature, but as a multi-tenant platform it does not meet enterprise level requirement.

• Scalability and Domain isolation

  (Maybe not). Nifi support cluster, but how to prevent some user mis-use it and drained most of the resources. Probably multiple cluster and running NIFI in docker in the future. In our production nifi setup, once you opened a gate, and ppls stated to put their flow in, after a while, you will see hundred of squares laying around. To clearly isolate pipeline, dockernize is the best solution so far. Then goes to the operation problem.

• Operation

  In the Dev/Ops scenarios, I don't want to deploy things a based docker image which are 2GB large. In addition it also need additional effort to load pipeline login into NIFI. Template, variable, registry...

• Collaboration

  This area is not clear for my yet. What going to happen if different users need to modified the same components? Is it using a exclusive lock, or it trying to merge in the commit step?

• Change Tracking

  We have requirement to audit all changes. As I know it is impossible to easily and clearly see what have been changed between two environment.