Revisiting gulp. Realtime JSX compilation and browserification.

Today I decided that it was worth investing some time in revisiting the build tools that are used across the Double Negative web properties.

We use gulp pretty much universally across our projects. In fact not only do most projects use gulp, but most projects use very similar build processes full stop.

Why? Most of our properties are built with and run on similar technologies - React, Browserify, v8js etc.

It thus seemed pretty stupid that each project maintained its own completely independent build process such that when I made 'big' changes and/or learned new things I would have to implement it across every project. I would inevitably forget, and then spend time down the line debugging bugs and problems that I had already fixed numerous times before.

What I wanted to achieve

I remember watching Bret Victors Inventing on Principle and thinking that the kind of development processes he demonstrates are what I want to have in place for Double Negative.

That is to say that I want to be able to make a change and see what it does in 'as realtime as possible'.

Now.. writing code in JSX, and compiling it to Javascript (giving consideration for ES6 transformations etc) is simply not instantaneous. That said, compiling a single JSX file is pretty darn quick. Milliseconds.

I want to have 'something' occurring in the background which upon saving a JSX file recompiles just that file (OK.. and any files that depend on it) to Javascript such that by the time I have switched to my browser window, and clicked refresh the updates are ready to test.

That distinction in itself is very important. Previous 'versions' of my build processes have simply recompiled all the JSX files after a single change simply because it was not that slow - a few seconds. But.. those few seconds really (and surprisingly) mess with your head and your workflow.

Execution

I was aware that Gulp 4 had functionality pertaining to sequential execution of tasks. Having however previously had issues with Gulp 4, I wanted a solution that worked with Gulp 3.

I stumbled upon the run-sequence package as outlined in this StackOverflow answer. It does what it says on the tin, but I had a few issues in that it was seemingly considering incomplete asynchronous tasks complete when they were not. My code was being browserified before it had all been compiled from JSX.

I investigated asynchronicity and noted from this answer that a return statement informs gulp that the task in question is over. This resolved my issue.

My next issue was what exactly to watch. I knew that when I edited a JSX file I wanted it to recompile, and then be browserified. I also wanted Javascript files to be browserified when edited directly (in the cases where they were not compiled from JSX).

This posed a problem in that each time a JSX file finished compiling it was considered a JS file edit, and as such the watch on JS files was triggered. If multiple JSX files (or files that depended on them changed) then the second watch task would be triggered multiple times.

I resolved this in a non-perfect, but clean and effective way. Two watch tasks, and a global boolean indicating the status of ongoing tasks.

My final code looked like this:

//Watches our .jsx files and our .js. If any change, it calls the build task.

const gulp        = require('gulp');
const runSequence = require('run-sequence');
const gutil       = require('gulp-util');

//Boolean to hold state of watch JSX watch task
let taskOngoing = false;

//This task simply sets the taskOngoing boolean to false
gulp.task('jsxdone', function(callback) {

	gutil.log('JSX done executed');

	taskOngoing = false; 

	callback();
});

//Does the 'watching'
gulp.task('watch', function() {

	//Watches .jsx files
	gulp.watch('../../public/assets/jsx/**/*.jsx', function(){ 

		//Indicate that we are running the JSX watch
		taskOngoing = true; 

		gutil.log('Running JSX watch');

		//Once the jsx task is executed we will execute jsxdone
		runSequence('jsx', 'jsxdone', 'browserify'); 

                //THIS line is discussed below
		gutil.log('Run sequence is NOT finished');
	}); 

	//Watches .js files
	gulp.watch('../../public/assets/js/**/*.js', function(){ 

		if (!taskOngoing) { 
		
			gutil.log('Running Javascript watch');

			runSequence('browserify'); 
		
		} else {

			//If our jsx - browserify watch task is executing then we will not allow this watch to execute

			gutil.log('SKIPPING Javascript watch. JSX task ongoing ' + (taskOngoing ? "true" : "false"));
		}

	}); 

	//Watches .css file for changes
	gulp.watch('../../public/assets/css/simple.css', ['cssconcat']); 
});

It works as expected.

Results

This is another case whereby I feel that had I read through the gulp documentation in advance, things would have progressed a lot quicker.

That said, I achieved what I wanted to achieve and noted some of the interesting intricacies of modern day Javascript in the process.

For example:

• I always forget that let is block scoped. That is why I define taskOngoing 'globally' outside of my 'watch' task body.

• Javascript is great because you can do asynchronous things easily. Javascript is annoying when you do not know which things are being executed asynchronously :)

And some of the intricacies of gulp:

• Tasks need a return statement to execute synchronously.

Take for example the line commented with 'THIS line is discussed below'. If you follow your log output (use gulp-util) this log will appear (in what might seem) the wrong place. Given that the tasks in my sequence are asynchronous this log may well be outputted prior to any logs contained within the sequenced tasks that only run asynchronously and after the previous task has completed.

Now..

Now when I begin a development session I simply make sure that my 'watch' task is running (gulp watch). If I change a JSX file then it is recompiled, as are any other files which depend on it. Once (and only once) all the aforementioned JSX files are recompiled our 'browserify' task is called which (again) only 'browserfies' the Javascript files which have changed.

As some of our products use raw unviolated Javascript (gosh !) we also watch them such that when such Javascript files are modified directly they are re-browserfied.

The initial problem

I noted that one part of the initial problem that I was trying to solve was that I used similar build processes across sites.

The final part of the puzzle was implementing this 'generic' build process as a git project in and of itself and including it within our projects as a git submodule.

The obvious down side to this is that it implies that my build process for all sites is exactly the same when it is in fact not. My resolution to this is simply to maintain a branch of the build project for each project. Generic changes to the build process can be merged into the individual branches, and the status of such merges can be easily and logically tracked from the repository.

Problem solved.

Shout outs

I skipped over some implementation details of the individual JSX/browserify tasks.

Some of the NPM package that I use are:

• gulp-dependencies-changed - for discerning which files have dependencies that have changed.

• gulp-newer - for discerning which JSX files are 'newer' than the JS files that they compile to.

• gulp-babel - transforms the latest Javascript syntax into syntax which will work now.

Take a look, and have a play. If you have any issues, or questions then let me know and I will do my best to point you in the right direction.