Can you run multiple instances of webpack at once? - parallel-processing

Let's say I have an array of components, that need to spit out their own source, as they wont always be included together, it needs to be it's own bundle. The idea is that there will eventually be 100's of components, and they can be cherry picked whenever.
However, when using webpack-stream with gulp, even though I'm dynamically registering the tasks, and that I can see they're running sequentially, it only runs one webpack stream by the looks of it, but seems to output it's bundle from the SECOND component, into the first components directory.
It's a pretty simple build process, it's an array of components like so:
var components = [
{'name : 'a', src : './foo/bar/entrya.js', dest : '/dir/a'},
{'name : 'b', src : './foo/bar/entryb.js', dest : '/dir/b'},
];
Relatively simple right? Then to register the tasks, it's something like this:
components.forEach(component => {
gulp.task(component.name, cb => {
function task(component) {
return gulp.src(component.src)
.pipe($.webpackStream(webpackConfig))
.pipe(component.dest(component.dest));
}
return task.apply(this, [component, cb]);
});
});
This is an incredibly dumbed down version of what I have, but it's pretty much the same thing, dynamically generates and then later on we run those tasks sequentially.

webpack-stream can handle multiple entry points and multiple builds per multiple entry points.
var gulp = require('gulp');
var webpack = require('webpack-stream');
gulp.task('build', function() {
return gulp.src(['src/entry.js']) // entry.js file doesn't need to exist
.pipe(webpack({
entry: {
a : __dirname + "/foo/bar/entrya.js",
b : __dirname + "/foo/bar/entryb.js"
},
output: {
filename: '[name].js'
}
}))
.pipe(gulp.dest('dir/'));
});

Your build is not running in parallel. you are just sequentially registering tasks. However you can run those tasks in parallel as child processes. One good option you can use is parallel-webpack.

Related

Multi-Module Task Dependencies

I want the outputs of one task to be available to an identical task in another submodule.
I'm trying to make yet-another plugin for compilation (of C/++, .hs, .coffee, .js et al) and source code generation.
So, I'm making a plugin and task/s that (so far) generate CMakeLists.txt, Android.mk, .vcxproj or whatever for each module to build the source code.
I have a multi-module build for this.
I can reach around and find the tasks from "other" submodules, but, I can't seem to enforce any execution order.
So, with ...
root project: RootModule
sub project: NativeCommandLine (requires SharedModule)
sub project: NativeGUI (requires SharedModule)
sub project: SharedModule
... I find that the NativeGUI tasks are executed before SharedModule which means that the SharedModule results aren't ready.
Bad.
Since the dependency { ... } stuff happens after plugins are installed (AFAIK) ... I'm guessing that the dependencies are connected after.
I need my tasks executed in order based on the dependency relations ... right? How can I do that?
I have created a (scala) TaskBag that lazily registers a collection of all participating Task instances.
I add instances of my task to this, along with a handler for when a new task appears.
During configure, any task can include logic in the lambda to filter and act on other tasks and it will be executed as soon as both tasks are participating.
package peterlavalle
import java.util
import org.gradle.api.Task
object TaskBag {
class AnchorExtension extends util.LinkedList[(Task, Task => Unit)]()
/**
* connect to the group of tasks
*/
def apply(task: Task)(react: Task => Unit): Unit =
synchronized {
// lazily create the central anchor ... thing ...
val anchor: AnchorExtension =
task.getProject.getRootProject.getExtensions.findByType(classOf[AnchorExtension]) match {
case null =>
task.getProject.getRootProject.getExtensions.create(classOf[AnchorExtension].getName, classOf[AnchorExtension])
case anchor: AnchorExtension =>
anchor
}
// show us off to the old ones
anchor.foreach {
case (otherTask, otherReact) =>
require(otherTask != task, "Don't double register a task!")
otherReact(task)
react(otherTask)
}
// add us to the list
anchor.add(task -> react)
}
}

Webpack 4.4.1 : performance issues with splitChunks

I'm working on an old project with a lot of code. This project uses Webpack 3.8.1 and I'm trying to update to 4.4.1, and it's a real obstacle course!
The main pain is that the projects uses the CommonsChunkPlugin:
new CommonsChunkPlugin({
name: 'common',
minChunks: 3,
chunks: _.without(_.keys(entry), 'ace-iframe', 'custom-theme-ace'),
}),
new CommonsChunkPlugin({
name: 'vendors',
minChunks(module, count) {
return isVendorModule(module) && count >= 2;
},
chunks: _.without(_.keys(entry), 'ace-iframe', 'custom-theme-ace'),
})
I know that Webpack 4 does not provide CommonsChunkPlugin anymore. A big thanks a lot to the below articles, they've saved hours of researches:
https://gist.github.com/sokra/1522d586b8e5c0f5072d7565c2bee693
https://medium.com/webpack/webpack-4-code-splitting-chunk-graph-and-the-splitchunks-optimization-be739a861366
Thanks to these amazing links, I've replaced CommonsChunkPlugin with these lines:
optimization: {
splitChunks: {
cacheGroups: {
vendors: {
priority: 50,
name: 'vendors',
chunks: 'async',
reuseExistingChunk: true,
minChunks: 2,
enforce: true,
test: /node_modules/,
},
common: {
name: 'common',
priority: 10,
chunks: 'async',
reuseExistingChunk: true,
minChunks: 2,
enforce: true,
},
},
},
},
},
Thanks to this config, the application is correctly building, chunks are created and the app is running as expected.
But the building time is really slow: more than 7 minutes!
Funny thing, if I totally remove the whole optimization.splitChunks configuration, the applications still works perfectly, and the building time is still around 7 minutes: it's totally like what I've done in optimization.splitChunks is useless.
I've tried to change the chunks properties: to be honest I don't really understand its role...
If I set them to all, the build is way quicker: around 1 minute.
But unfortunately, the generated files from my entries points are not running well: Webpack seems to wait that the chunks are loaded before executing my own code:
// Code from webpack
function checkDeferredModules() {
var result;
for(var i = 0; i < deferredModules.length; i++) {
var deferredModule = deferredModules[i];
var fulfilled = true;
for(var j = 1; j < deferredModule.length; j++) {
var depId = deferredModule[j];
if(installedChunks[depId] !== 0) fulfilled = false;
}
// If I understand well, Webpack checked that deferred modules are loaded
if(fulfilled) {
// If so, it runs the code of my entry point
deferredModules.splice(i--, 1);
result = __webpack_require__(__webpack_require__.s = deferredModule[0]);
}
}
return result;
}
Please tell me I am not wrong here: Webpack seems to wait deferred modules to be loaded, but it does not run the code which is actually loading them... How am I suppose to make this work?
In brief:
with chunks set to async: all is working well, but building time is not viable (more than 7 minutes)
with chunks set to all: building time is correct (around 1 minute), but my code is not running ¯\_(ツ)_/¯
Sorry for this long post, but if someone can help me to make all of this working with correct building time, it would be perfect.
Or at least helping me to understand how all of this is supposed to work, official documentation is not very helpful :(
Thanks in advance!
EDIT: I've tried to continue with chunks set to async, despite the 7mn building time.
I have 20 entries points, and if I add an import instruction importing jQuery and jQuery-UI in one of them, building time is doubling.
If I add it into 5 files, the build crashes:
<--- Last few GCs --->
[15623:0x103000000] 222145 ms: Mark-sweep 1405.0 (1717.4) ->
1405.2 (1717.4) MB, 671.3 / 0.0 ms allocation failure GC in old space requested [15623:0x103000000] 222807 ms: Mark-sweep 1405.2
(1717.4) -> 1405.0 (1667.9) MB, 662.4 / 0.0 ms last resort GC in old
space requested [15623:0x103000000] 223475 ms: Mark-sweep 1405.0
(1667.9) -> 1405.1 (1645.4) MB, 667.1 / 0.0 ms last resort GC in old
space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x1b6415c25ee1
1: fromString(aka fromString) [buffer.js:~298] [pc=0x1973a88756aa](this=0x1b6462b82311
,string=0x1b642d3fe779 ,encoding=0x1b6462b82311 )
3: from [buffer.js:177] [bytecode=0x1b6488c3b7c1 offset=11](this=0x1b644b936599 ,value=0x1b642d3fe779 ,encodingOrOffset=0x1b6462b82311
>> FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
Out of memory... I think setting chunks to async is not the correct way to solve this issue :/

Calling the same task multiple times in a single build.gradle file

I have a custom Gradle plugin that will generate Java files from a template file. I have several such template files in different locations, and I need to "compile" all of them to generate the Java files I need. Once I have the files, I want to package them into a .jar.
One way I thought I could do this was to call the "compile template" task multiple times from within the same build file. I'd call it once in a task that compiles template files in location A, again from a task that compiles template files from location B... etc., until I have all the Java files I need.
Something like this:
task compileFromLocationA <<{
compileTemplate.execute(A)...
}
task compileFromLocationB
compileTemplate.execute(B)...
...
packageJar(depends: compileFromLocationA, compileFromLocationB, ...)
...
However, you can't programmatically call a task from within another task. I suppose I could break each compileFromLocation_ task into it's own build.gradle file, but that seems like overkill. What's the "best practice" in a case like this?
This code seems to work in build.gradle by using tasks.register() - e.g. to perform multiple source code generating steps - in my case I needed to load different pairs of files (XML schema and generation options) in two different steps:
plugins {
id 'java'
id "com.plugin" version "1.0"
}
sourceSets.main.java.srcDirs += file("${buildDir}/genSrc")
sourceSets.test.java.srcDirs += file("${buildDir}/testGenSrc")
tasks.compileJava {
dependsOn tasks.named("genMessage")
}
genMessage {
codesFile = "${projectDir}/src/main/resources/file.xml"
}
def testGenModel1 = tasks.register("testGenModel1", com.plugin.TestGenModelTask.class) {
schema = "${projectDir}/src/test/resources/file.xsd"
options = "${projectDir}/src/test/resources/file.xml"
}
def testGenModel2 = tasks.register("testGenModel2", com.plugin.TestGenModelTask.class) {
schema = "${projectDir}/src/test/resources/file2.xsd"
options = "${projectDir}/src/test/resources/file2.xml"
}
tasks.compileTestJava {
dependsOn tasks.named("testGenModel1"), tasks.named("testGenModel2")
}

Mocha test failing using babel and webpack

So I am using webpack, babel, and mocha here. When I have code like this:
import userImage from '../../images/user.png';
and I build with webpack, userImage results in a string to the path of the file since I am using the file loader for images (requirements call for me not to embed images) however when I try to run my mocha tests using:
./node_modules/.bin/babel-node ./node_modules/.bin/babel-istanbul cover ./node_modules/.bin/_mocha
I get a syntax error:
SyntaxError: /repositories/react-seed/web/app/images/user.png: Unexpected character '�' (1:0)
> 1 | �PNG
| ^
2 |
3 |
I also get this error when removing istanbul. So it seems like it is trying to load the actually image file however can parse it as JavaScript since it is not.
Anyone know a way around this issue?
You can use the --compilers option which allows you to customize the nodejs require system in order to let it understand png files. So :
mocha --compilers png:./mochacfg.js
Or create a file 'test/mocha.opts' containing (better for your needs):
--compilers png:./mochacfg.js
With ./mochacfg.js:
require.extensions['.png'] = function(){ return null; }
This ignores png files (should be ok if you do nothing special with them).
If you want to do something with the image data:
var fs = require('fs');
require.extensions['.png'] = function(module, filepath) {
var src = fs.readFileSync(filepath).toString ('base64');
return module._compile('module.exports = "data:image/png;base64,' + src + '";');
}
Its quite late to answer this question but just for knowledge sharing purpose, I am answering another approach to do this.
Create a test-config.js file and use it while running the mocha test cases.
var jsdom = require('jsdom').jsdom;
process.env.NODE_ENV = 'test';
// -------------------------------
// Disable webpack-specific features for tests since
// Mocha doesn't know what to do with them.
['.css', '.scss', '.png', '.jpg'].forEach(ext => {
require.extensions[ext] = () => null;
});
and inside package.json use this test command to run the test cases
"test": "mocha ./test/test-setup.js './test/**/*.spec.js' --compilers js:babel-core/register",
I hope it helps someone.

Docpad: confused about extending template data

I'm totally confused about adding mongo data to template data. I haven't even started trying to get the data from a database as I can't get my templates to see test data (see below). This is in docpad.coffee for the moment, but ultimately g will be the output of mongoDB.
events:
extendTemplateData: (opts) ->
# {templateData} = opts
getGigsData: ->
g = { "date" : "3-4-2013", "location" : "Gent" }
return g
opts.templateData["getGigsData"] = getGigsData
And I hope to access it with <%= #getGigsData().date %>
Thanks so much for some guidance
I should add that this design is based on wanting to make it easy for the band to add gigs, without letting them edit the page content itself as I fear they would mess up the markup - if there are other ways to achieve this goal, I'd be pleased to hear.
Tried this locally. And hit the issue:
debug: Emitting the event: extendTemplateData
→ [2014-02-14 01:38:50.030] [/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/out/lib/docpad.js:1184] [DocPad.emitSerial]
error: Something went wrong with the action
→ [2014-02-14 01:38:50.037] [/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/out/lib/interfaces/console.js:107] [ConsoleInterface.destroyWithError]
error: An error occured:
ReferenceError: getGigsData is not defined
at Object.docpadConfig.events.extendTemplateData (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/docpad.coffee:42:44)
at ambi (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/ambi/out/lib/ambi.js:25:27)
at DocPad.<anonymous> (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/out/lib/docpad.js:995:25)
at ambi (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/ambi/out/lib/ambi.js:23:18)
at Task.<anonymous> (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/event-emitter-grouped/out/lib/event-emitter-grouped.js:45:23)
at ambi (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/ambi/out/lib/ambi.js:23:18)
at fire (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/taskgroup/out/lib/taskgroup.js:163:25)
at b (domain.js:183:18)
at Domain.run (domain.js:123:23)
at Task.fire (/Users/balupton/Projects/docpad-extras/skeletons/so-21747504/node_modules/docpad/node_modules/taskgroup/out/lib/taskgroup.js:173:25)
at processImmediate [as _immediateCallback] (timers.js:330:15)
Which indicates that the error is actually inside our event handler, rather than inside our code. That for some reason getGigsData is not being set, despite our:
getGigsData: ->
g = { "date" : "3-4-2013", "location" : "Gent" }
return g
Examining the code, as a CoffeeScript user, I found the issue. As a non-coffeescript user, you can use the coffeescript compiler on the coffeescript website http://coffeescript.org to see the compiled javascript, which is:
({
events: {
extendTemplateData: function(opts) {
({
getGigsData: function() {
var g;
g = {
"date": "3-4-2013",
"location": "Gent"
};
return g;
}
});
return opts.templateData["getGigsData"] = getGigsData;
}
}
});
As we can see that is definitely not what we expected. We are just defining getGigsData inside an object, then doing nothing with it.
The issue is that we used a colon instead of an equals sign, so getGigsData: -> instead of getGigsData = ->. This is not a coffeescript thing, but you would have run into the same issue if this was javascript too, albeit javascript may be a bit more obvious due to the necessary squiggly braces around object definitions.
As a sidenote, if you prefer to use JavaScript with DocPad for whatever reason, that is totally supported. You could use a docpad.json or docpad.js file for your docpad configuration file. Another option, is to continue using CoffeeScript then just wrap JavaScript code within the backtick, see: http://coffeescript.org/#embedded

Resources