Processing Files using Node.js

Quite often we come across situations where we want to create batch jobs on servers for processing files. We are required to monitor a folder or file for any changes and process the same as per our business requirement. One of the examples would be monthly attendance data for payroll processing.

Traditionally, a programmer has the following two options to implement such a scenario

1. Create a file processing program, schedule it as a task using any OS scheduler, and process the file at scheduled intervals. This approach may delay processing and you never know when you will receive a file and if your scheduler is in sync or totally out of sync with the frequency at which the file is received. You will also introduce the external dependency of a task scheduler.
   

2. You can write a never-ending program that continuously monitors your source folder using conventional loops, this will definitely pick your file as soon as it is received, however, this will unnecessarily consume a lot of processing power in folder monitoring along with dirty implementation logic.

My Experience

While trying to implement a similar kind of scenario for one of our customer projects, the requirement was that the file should be processed as soon as it is received, secondly, our job was going to share CPU with one of the existing live production application, so consuming unnecessary cycles was not at all advisable.

Out of the Box Solution

Node.js is one of the best platforms to implement a job. It is specially designed to make such a task super smooth. During my research, I came across a ready-made library available in a node called chokidar, initially, it sounded funny to me because of its not so serious name, and was skeptical to use it in my ready to go live customer project, however, thought of giving it a shot.

Node.js itself provides a built-in facility to perform functionality provided by this library, however, a clean wrapper always helps in increasing the overall productivity of the team.

The idea is to use chokidar to monitor a folder, as soon as a file is received chokidar will invoke a callback function with necessary file details. So this is how you should use chokidar in your project,

1. Install using npm

npm install chokidar --save

2. Usage

app.js

var chokidar = require('chokidar');

//--Set Chokidar to Watch your current folder--

chokidar.watch('.', { ignored: /[\/\\]\./ }).on('all', function (event, path) {
    //Perform Processing as soon as new file is added to root folder
	processFile(path);
});

functionprocessFile(path)
{
	//--Write File Processing Code for file in <path> variable--
}

You can execute the above application using the forever command

forever start app.js

That's it, chokidar will start monitoring your current folder (.) and throws a callback every time an IO operation is performed on this folder, if there are existing contents in the folder, chokidar will send callbacks right at the start of your program. Also, this will be executed in a multithreaded environment without any additional efforts.

No need to write a loop and schedule a job, clean simple code with live monitoring which is platform-independent too, we developed this application on mac and ported it on AWS without a single line of code change.

Now there are multiple file filter options available with chokidar, you can check it out on your own.

https://www.npmjs.com/package/chokidar

Overall I would say it's a pretty stable library and can be trusted for critical production deployments.