When TypeScript project is configured to ESM, `projen` commands gets `ERR_UNKNOWN_FILE_EXTENSION` about projen HOT 21 OPEN

Thanks for reporting this! Sounds like an important thing to resolve one way or the other!

When the project is configured to use ESM, the projen command usually maps to "exec":

For purpose of reproducing this, how do you configure your project to use ESM?

tsc .projenrc.ts && node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts

For the fix, this seems to do quite a bit more than making ts-node use ESM? Is all of this needed? What am I missing?

from projen.

Ignoring ts-jest and esbundle configs, this is precisely what’s happening in the example above:

https://github.com/10mi2/tms-projen-projects/blob/9e552b1f75e40dd92994bc4f615db7aa619ff780/src/tms-typescript-app-project.ts#L376-L389

I believe the minimum to reproduce is the first of those lines:

this.package.addField("type", "module");

And what’s all happening: The ts-node command is a smart shortcut if you will to the second form, mostly. It does some other stuff to make the error messages clean that it’s easier for us to do by just running tsc with noEmit. In that “other stuff” is where the bug is.

# instead of
ts-node --project tsconfig.special.json src/index.ts

# use
tsc .projenrc.ts && \
  TS_NODE_PROJECT=tsconfig.special.json node --loader ts-node/esm --no-warnings=ExperimentalWarning src/index.ts

If there are any type errors, the node --loader ts-node/esm yields a difficult-to-read error message, so we run tsc first separately to get the type errors before running the node command.

The tsc command assumes the correct tsconfig file where the target is included has noEmit set to true. If not, add --noemit to the tsc command.

We don’t emit from things compiled with tsconfig.dev.json so that last bit is handled.

There are other options. We could use tsx or swc-loader - I’ve had great luck with tsx but haven’t tried swc-loader. Neither do type checking so the tsc pre-pass is still recommended.

from projen.

Okay, so it seems like ts-node --esm is what's needed.

This is the minimal change I did come up with:

const project = new typescript.TypeScriptProject({
  // ...
  tsconfigDev: { compilerOptions: { module: 'ESNext' } },
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('esm', true);

I don't really see how we can resolve this without addressing ESM in a wider context.

from projen.

That does indeed reproduce it for me, but it also breaks it more, sadly.

Last I looked, and it's a moving target, ts-node --esm and ts-node-esm both had the same issue as ts-node auto-detecting ESM mode.

It doesn't appear to fix it when I add --esm to the command manually or use ts-node-esm:

Running the command as follows gets past that but, but makes another:

TS_NODE_PROJECT=tsconfig.dev.json npx \
  node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts

To get it to work with ESM, minimally (not the project, just .projenrc.ts, for anyone reading this later):

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'node16' } }, // ← had to use 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true); // ← 'tsnode.esm'  vs 'esm'

So, as a workaround, add this before the synth:

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    'tsc .projenrc.ts && node --loader ts-node/esm --no-warnings=ExperimentalWarning .projenrc.ts',
  );
  project.defaultTask.env('TS_NODE_PROJECT', project.tsconfigDev.fileName);
  project.defaultTask.description =
        'Run projen with ts-node/esm (workaround for Node 18.19+ applied)';
}

And you can still run npx projen when in ESM mode.

And you're right, @mrgrain, more work is needed for ESM to work fully. But I thought I'd make smaller issues (with smaller PRs) that are easier to tackle to pave the way.

Another option, if we're willing to move away from ts-node, is tsx with tsc ran first (since esbuild that tsx uses doesn't do type checking):

const project = new typescript.TypeScriptAppProject({
  //...
  devDeps: ['tsx'],
  tsconfigDev: {
    compilerOptions: {
      module: 'node16',
    },
  },
});
project.package.addField('type', 'module');

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    `tsc .projenrc.ts && tsx --tsconfig ${project.tsconfigDev.fileName} .projenrc.ts`,
  );
}

from projen.

I don't really see how we can resolve this without addressing ESM in a wider context.

I just retested and the --loader ts-node/esm mechanism doesn't work with the module not in ESM mode. 😞

However, the tsx option does work in both modes:

const project = new typescript.TypeScriptAppProject({
  //...
  // No tsconfig overrides here
  devDeps: ['tsx'],
});
// optional - works with or without type: "module", and no other tsconfig changes are needed
project.package.addField('type', 'module');

if (
  project.defaultTask
) {
  project.defaultTask.reset(
    `tsc .projenrc.ts && tsx --tsconfig ${project.tsconfigDev.fileName} .projenrc.ts`,
  );
}

from projen.

Ha you are right, apologies! I think I misread the ts-node docs.

I can conform this works for me now:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'nodenext' } }, // ← or 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true);

  // No tsconfig overrides here
  devDeps: ['tsx'],

It feels odd to me to say that no tsconfig changes are required. You still need to change module for ESM to work, right? The override is only needed because the tsconfig types don't support ts-node as of now.

I'm not super keen on changing ts-node. I'm aware tsx is on the rise, but it feels to early. Have you looked at yet what it takes to create your own version ProjenRcTSX? 'Cause I'm not sure it's super easy to swap that out at the moment, but ultimately projen should allow that.

from projen.

It feels odd to me to say that no tsconfig changes are required. You still need to change module for ESM to work, right? The override is only needed because the tsconfig types don't support ts-node as of now.

I'm not sure I follow completely, but I'll take a swing at it: I was just saying that tsx works in my testing in both ESM (future) and non-ESM (current) contexts. In ESM context, you'd want to change tsconfig.dev,json somewhat, yes.

I'm not super keen on changing ts-node. I'm aware tsx is on the rise, but it feels to early. Have you looked at yet what it takes to create your own version ProjenRcTSX? 'Cause I'm not sure it's super easy to swap that out at the moment, but ultimately projen should allow that.

I've been using tsx and esbuild (that it's based on) for a while. I've only run into a couple of issues:

1. It doesn't type-check, so code that TypeScript wouldn't pass will pass - solution is to run tsc with noEmit first
2. It doesn't support experimental features like decorators - I don't expect anyone to be using those in the Projen tooling, but if they NEED to, the solution is to run tsc with emit enabled and then run the JS (not terribly pretty, since you have to put the temporary javascript somewhere)

That said, I don't think we necessarily need to switch - they do provide the same basic features.

Referencing Projenrc.addDefaultTask()

Here's an untested rough sketch to make tsx an opt-in:

export interface ProjenrcOptions {
//  ...
  /**
   * Whether to use `tsx` instead of `ts-node`.
   *
   * Ignored `swc` option if `tsx` is set to `true`.
   *
   * @default false
   */
  readonly tsx?: boolean;
}

// ...

export class Projenrc extends ProjenrcFile {
// ...
  private addDefaultTask() {
    // this is the task projen executes when running `projen` without a
    // specific task (if this task is not defined, projen falls back to
    // running "node .projenrc.js").

    // we use "tsconfig.dev.json" here to allow projen source files to reside
    // anywhere in the project tree.

    if (this._tsx) {
      // Use tsx
      this._tsProject.addDevDeps("tsx");
      this._tsProject.defaultTask?.exec(
        `tsc ${this.filePath} && tsx --tsconfig ${this._tsProject.tsconfigDev.fileName} ${this.filePath}`
      );
    } else {
      // Use ts-node
      const deps = [];
      if (this._swc) {
        deps.push("@swc/core");
      }
      deps.push("ts-node");
      this._tsProject.addDevDeps(...deps);

      const tsNode = this._swc ? "ts-node --swc" : "ts-node";

      this._tsProject.defaultTask?.exec(
        `${tsNode} --project ${this._tsProject.tsconfigDev.fileName} ${this.filePath}`
      );
    }
  }
//...
}

Note that this wouldn't be replacing or effecting other uses of ts-node, such as that inside the cdk projects.

from projen.

You can verify if tsx work easily with:

npx tsx --tsconfig tsconfig.dev.json .projenrc.ts

In fact, when ts-node hits an error, that's what I've been running to execute projen again and rebuild the package.json and tsconfig*.json files.

from projen.

I'm not sure I follow completely, but I'll take a swing at it: I was just saying that tsx works in my testing in both ESM (future) and non-ESM (current) contexts. In ESM context, you'd want to change tsconfig.dev,json somewhat, yes.

My point really just is that for ESM to work you need to make changes to your tsconfig. One change more doesn't seem like a big issue to me.

from projen.

Have you looked at yet what it takes to create your own version ProjenRcTSX?

That confused me for a while, since tsx (the tool) just executes TypeScript files by compiling them on the fly (with esbuild) and running them with node (same as ts-node, etc.). (I'm aware I greatly simplified what those tools do.)

Then I realized that there's confusion due to poor naming: *.tsx files (used by React, etc.) have nothing to do with tsx the tool I was referring to ☝️. (You can use them together, since *.tsx is effectively an extension of TypeScript, but otherwise they're unrelated.)

With the ProjenRcTSX reference were you thinking I was suggesting to make a .projenrc.tsx option, perhaps? If so, that's not the case.

On a slightly different note, how should I proceed? My eventual goal is to chip away at ESM support with several smaller Issue→PR steps. I have several projects that use ESM mode already (optionally), and projen itself works fine in/with ESM-based projects, so no changes are needed in projen itself, just how it configures the tsconfig, package, ts-node, and ts-jest.

Those projects are here: https://github.com/10mi2/tms-projen-projects

from projen.

Oh no, I see the confusion now. Sorry for being unspecific.

What I meant is:

• You are proposing to extend the existing typescript.Projenrc component with a new option (e.g. tsx: true)
• I am proposing you build a completely new tms.ProjenrcUsingTsx component that does exactly what you want.

Reason being that there is a in almost infinite permutation of ways how we can deal with .projenrc.ts. And even when we are taking out the clearly niche uses cases, that's still a lot of possible ways that we should support inside typescript.Projenrc.

Now saying that, I'm not sure if it's currently possible to write this tms.ProjenrcUsingTsx without using hacks. So I'm curious to find out where the gaps are and what paper cuts you encounter.

from projen.

On a slightly different note, how should I proceed? My eventual goal is to chip away at ESM support with several smaller Issue→PR steps. I have several projects that use ESM mode already (optionally), and projen itself works fine in/with ESM-based projects, so no changes are needed in projen itself, just how it configures the tsconfig, package, ts-node, and ts-jest.

Please start with writing an RFC issue on ESM support. It's really hard to judge smaller PRs without seeing the bigger pictures. Once we have Community agreement on the bigger picture, I'm happy to accept smaller PRs.

from projen.

I'm not sure how 'ProjenrcUsingTsx' would work. Or why, really. Nothing in the .projenrc.ts file would change, and nothing in the output would change other than the command in the default task, and the dev dependency going from ts-node to tsx.

I guess I could extend Projenrc class and override the addDefaultTask() (which is currently private). Is that what you had in mind?

On the ESM score, I'll work something up. Will likely take a few days to get the time.

from projen.

I'm not sure how 'ProjenrcUsingTsx' would work. Or why, really. Nothing in the .projenrc.ts file would change, and nothing in the output would change other than the command in the default task, and the dev dependency going from ts-node to tsx.

typescript.Projenrc and other Projenrc components are only partly about the contents of the rcfile. After the initial creation, the content is written by users after all. The Projenrc components are mostly about "how to execute a program, so that it produces the desired output" (i.e. project.synth()). But yes, they also take care of the initial generation.

I guess I could extend Projenrc class and override the addDefaultTask() (which is currently private). Is that what you had in mind?

That would probably be the best approach. You could write it from scratch, but like you've said some parts would end up being copy and paste.

On the ESM score, I'll work something up. Will likely take a few days to get the time.

Thank you!

from projen.

I can conform this works for me now:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: { module: 'nodenext' } }, // ← or 'node16'
});
project.package.addField('type', 'module');
project.tsconfigDev.file.addOverride('ts-node.esm', true);

Just wanted to add that we ran into the same issue and solved it very similar to what you did. This worked for us:

const project = new typescript.TypeScriptAppProject({
  //...
  tsconfigDev: { compilerOptions: {
    module: 'ES2022', // <-- starting from ES2015, everything works
    moduleResolution: TypeScriptModuleResolution.NODE,
  } },
});
project.tsconfigDev!.file.addOverride('ts-node', {
  esm: true,
  experimentalSpecifierResolution: 'node',
});

For us it didn't work without node as moduleResolution and epxerimentalSpecifierResolution. Not 100% sure why (I think it was something about file extensions in import statements or something like this), but I wanted to mention this.

from projen.

@dj-rabel This is very dependent on node version. Were you on node 18.18 or older, or something newer?

from projen.

@giseburt sure. I just wanted to contribute our experience to the discussion. For the project I'm referring to, we're on 18.14.

from projen.

@dj-rabel 👍. Watch out for upgrading to node 18.19 or newer, you'll run into the TypeError [ERR_UNKNOWN_FILE_EXTENSION] bug in ts-node

from projen.

@giseburt thanks for the hint. This error sounds very familiar to me. I'm not 100% sure, but this might be the reason why we ended up adding this experimentalSpecifierResolution setting 🤔🧐 will check with the colleges tomorrow.

from projen.

EDITED: I'm working on a new RFC for general ESM support, but for those running into this issue, here's the escape hatch I'm using ATM.

See edit in original comment above, I put the code there.

What's happening here is, since ts-node when run as --loader ts-node/esm has terrible error messages if there were type errors, and tsx doesn't do type checking at all, we want to call tsc before executing the code. We can't have it compile just one file and have it use settings from a tsconfig file (not that I've found, at least), so we have to have it compile a whole project.

If we use tsconfig.dev.conf it'll include the project source and the testing source code as well, which means if the code itself is malformed, it'll error and nor run .projenrc.ts. This happens in a case like where the code is referring to a module that hasn't been installed yet because it's in deps and you can't run projen because it keeps erroring on that missing dependency.

So, we make a new tsconfig.projenrc.json that extends tsconfig.dev.json and only includes ".projenrc.ts", "projenrc/**/*.ts" (the latter if you want to move some code out of .projenrc.ts into a library).

Also, tsx has far fewer issues than ts-node - it just doesn't do type-checking.

from projen.

@mrgrain:

Please start with writing an RFC issue on ESM support.

Here it is: #3447

from projen.