express instrumentation is not applied after webpack compilation

[AM1988]

What happened?

Steps to Reproduce

Hi there. Thank you for the awesome software.

I am trying to integrate the opentelemetry into the express application.

It has only 2 files: index.ts and tracer.ts.

When I run the applications using ts-node by script

nodemon --exec ts-node -r dotenv/config -r ./src/tracer.ts ./src/index.ts

all works perfectly, I see that express-instumentation library is injected and doing the job, I see traces in debug mode, middleware spans, etc.

[nodemon] starting `ts-node -r dotenv/config -r ./src/tracer.ts ./src/index.ts`
@opentelemetry/api: Registered a global for diag v1.4.1.leted in 3ms
@opentelemetry/api: Registered a global for trace v1.4.1.
@opentelemetry/api: Registered a global for context v1.4.1.
@opentelemetry/api: Registered a global for propagation v1.4.1.
@opentelemetry/instrumentation-http Applying patch for http@18.12.1s
Applying patch for express@4.17.3onfig:load:flatten Completed in 3ms
@opentelemetry/instrumentation-http Applying patch for https@18.12.1
{"level":"INFO","message":"Express server is listening on http://localhost:8080","traceId":"not_available","timestamp":"2023-05-07T14:10:52.050Z"}

I want to webpack the application using webpack.config.js

const path = require('path');
const webpack = require('webpack');
const CopyPlugin = require("copy-webpack-plugin");
module.exports = {
	entry: {
		index: path.resolve(__dirname, 'src/index.ts'),
		tracer: path.resolve(__dirname, 'src/tracer.ts'),
	},
	resolve: {
		extensions: ['.js', '.ts'],
	},
	output: {
		publicPath: '/public',
		path: path.join(__dirname, 'dist'),
		filename: '[name].js',
		libraryTarget: 'commonjs',
	},
	target: 'node',
	node: {
		// Need this for proper paths resolutions of static content
		__dirname: false,
		__filename: false,
	},
	module: {
		rules: [
			{
				// Transpiles ts into js
				test: /\.ts$/,
				use: [
					{
					  loader: 'ts-loader',
					  options: {
						transpileOnly: true
					  }
					}
				  ],
				exclude: [/node_modules/],
			}
		]
	},
	plugins: [
		// copy swagger ui static page files
		new CopyPlugin({
			patterns: [
				{ from: path.resolve(__dirname, 'node_modules/swagger-ui-dist/swagger-ui.css'), to: 'public/api' },
				{ from: path.resolve(__dirname, 'node_modules/swagger-ui-dist/swagger-ui-bundle.js'), to: 'public/api' },
				{ from: path.resolve(__dirname, 'node_modules/swagger-ui-dist/swagger-ui-standalone-preset.js'), to: 'public/api' },
		
			],
		}),
	],
}

It generates 2 files: index.js and tracer.js

Then I am running this as node (Node.js v18.12.1) process node -r "./tracer.js" index.js the express-instrumentation is not applied - I do not see any spans, etc.

@opentelemetry/api: Registered a global for diag v1.4.1.
@opentelemetry/api: Registered a global for trace v1.4.1.
@opentelemetry/api: Registered a global for context v1.4.1.
@opentelemetry/api: Registered a global for propagation v1.4.1.
@opentelemetry/instrumentation-http Applying patch for http@18.14.2
@opentelemetry/instrumentation-http Applying patch for https@18.14.2
{"level":"INFO","message":"Express server is listening on http://localhost:8080","traceId":"not_available","timestamp":"2023-05-07T14:26:58.428Z"}

There is no Applying patch for express@4.17.3onfig:load:flatten Completed in 3ms in that case as it was in the example above.

In index.js I see express-instumentation is included.

Could you please help me with this?

Expected Result

express instrumentation is working

Actual Result

express instrumentation is not working

Additional Details

OpenTelemetry Setup Code

/*tracer.ts*/
import { BatchSpanProcessor, ConsoleSpanExporter } from "@opentelemetry/sdk-trace-base";
import { Resource } from "@opentelemetry/resources";
import { SemanticResourceAttributes } from "@opentelemetry/semantic-conventions";
import { NodeTracerProvider } from "@opentelemetry/sdk-trace-node";
import { registerInstrumentations } from "@opentelemetry/instrumentation";
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { HttpInstrumentation } from '@opentelemetry/instrumentation-http';
import { ExpressInstrumentation } from '@opentelemetry/instrumentation-express';
import { diag, DiagConsoleLogger, DiagLogLevel } from '@opentelemetry/api';


diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.ALL),
// Optionally register instrumentation libraries
registerInstrumentations({
  instrumentations: [    new HttpInstrumentation(),
    new ExpressInstrumentation()],
});

const resource =
  Resource.default().merge(
    new Resource({
      [SemanticResourceAttributes.SERVICE_NAME]: "express-app",
    })
  );

const provider = new NodeTracerProvider({
    resource: resource,
});
const exporter = new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' });
const processor = new BatchSpanProcessor(exporter);
provider.addSpanProcessor(processor);

provider.register();

package.json

{
	"name": "express-app",
	"version": "1.0.0",
	"description": "express app",
	"scripts": {
		"start": "nodemon --exec ts-node -r dotenv/config -r ./src/tracer.ts ./src/index.ts",
		"start:build": "node -r ./dist/tracer.js ./dist/index.js",
		"build": "npm run build:clear && webpack --mode production --config webpack.config.js --stats-error-details",
		"build:clear": "shx rm -rf ./dist"
	},
	"nodemonConfig": {
		"ignore": [
			"./logs",
			"./dist",
			"src/**/*.spec.ts"
		]
	},
	"author": "",
	"license": "ISC",
	"dependencies": {
		"axios": "0.26.0",
		"express": "4.17.3",
		"follow-redirects": "^1.14.7",
		"helmet": "4.4.1",
		"jsonwebtoken": "9.0.0",
		"swagger-ui-express": "4.1.6"
	},
	"devDependencies": {
		"@opentelemetry/api": "1.4.1",
		"@opentelemetry/auto-instrumentations-node": "0.36.6",
		"@opentelemetry/exporter-trace-otlp-http": "0.38.0",
		"@opentelemetry/instrumentation-http": "0.38.0",
		"@opentelemetry/sdk-node": "0.38.0",
		"@opentelemetry/sdk-trace-base": "1.12.0",
		"@pact-foundation/pact": "^10.1.4",
		"@types/echarts": "^4.9.3",
		"@types/express": "^4.17.11",
		"@types/jest": "^26.0.20",
		"@types/node": "^14.14.22",
		"@types/puppeteer": "^5.4.2",
		"@types/supertest": "^2.0.10",
		"@types/swagger-ui-express": "^4.1.2",
		"@typescript-eslint/eslint-plugin": "^4.14.2",
		"@typescript-eslint/parser": "^4.14.2",
		"copy-webpack-plugin": "^8.1.1",
		"dotenv": "^8.2.0",
		"eslint": "^7.19.0",
		"eslint-config-prettier": "^6.15.0",
		"eslint-plugin-prettier": "^3.3.1",
		"jest": "^28.1.1",
		"jest-pact": "^0.10.1",
		"nodemon": "^2.0.7",
		"prettier": "^2.2.1",
		"shx": "^0.3.4",
		"supertest": "^6.1.3",
		"ts-jest": "29.1.0",
		"ts-loader": "^9.4.2",
		"ts-node": "10.9.1",
		"typescript": "^4.1.3",
		"webpack": "^5.74.0",
		"webpack-cli": "^4.10.0"
	}
}

Relevant log output

my tsconfig.json

{
	"compileOnSave": false,
	"compilerOptions": {
		"baseUrl": "./src",
		"outDir": "out",
		"sourceMap": false,
		"declaration": false,
		"module": "CommonJS",
		"moduleResolution": "node",
		"pretty": true,
		"target": "esnext",
		"allowSyntheticDefaultImports": true,
		"resolveJsonModule": true,
		"esModuleInterop": true,
		"emitDecoratorMetadata": true,
		"experimentalDecorators": true,
		"importHelpers": false,
		"typeRoots": [
			"node_modules/@types"
		],
		"types": [
			"node",
			"express",
			"axios",
			"jest"
		],
	},
	"include": [
		"src/**/*.ts"
	],
	"exclude": [
		"node_modules",
	]
}

No response

[Flarna]

instrumentations for modules like express depend on require which vanishes once a bundler is used.
I'm not an expert of bundlers but usually they have a possibility to exclude some packages from bundle.

Clearly this breaks one of the main goals of bundlers to get a single file so one has to decide what is more important monitoring or bundling.

[AM1988]

Hi @Flarna ,

I'm not an expert of bundlers but usually they have a possibility to exclude some packages from bundle.
could you suggest which packages should be excluded and in which file? Openetelemetry-related packages from index.js or tracer.js?

I am wondering how other people use opentelemery with express in the production environment if using typescript.

[Flarna]

I would assume they use it without bundler or with a bundler configured to not include express.
express is for node not for browser and server applications usually have no absolute need to get a single, minified file.

[AM1988]

Hm...the interesting thing is when I excluded express and running the application it did not start because of error

 
 Error: ENOENT: no such file or directory, open '/app/jaeger-idl/thrift/jaeger.thrift'
     at Object.openSync (node:fs:600:3)
     at Object.readFileSync (node:fs:468:35)
     at Function.loadJaegerThriftDefinition (/app/tracer.js:55805:27)
     at 4292 (/app/tracer.js:55949:23)
     at __webpack_require__ (/app/tracer.js:86117:42)
     at 96616 (/app/tracer.js:55560:15)
     at __webpack_require__ (/app/tracer.js:86117:42)
     at 83288 (/app/tracer.js:20877:21)
     at __webpack_require__ (/app/tracer.js:86117:42)
     at 1212 (/app/tracer.js:20726:17) {
   errno: -2,
   syscall: 'open',
   code: 'ENOENT',
   path: '/app/jaeger-idl/thrift/jaeger.thrift'
 }

[Flarna]

see here: NOTE: Bundling (with e.g. webpack, rollup, esbuild, ...) is not supported by this package. Please use @opentelemetry/exporter-trace-otlp-proto instead.

[AM1988]

I do not use @opentelemetry/exporter-jaeger directly, it comes as a dependency for the @opentelemetry/sdk-nod

[Flarna]

should be fixed by #3739

[AM1988]

Thank you @Flarna for your suggestions.

I managed to make it work by compiling typescript to javascript and I had to throw out webpack.

As a side note: with the newer approach I need to install the dependencies and keep node_modules, therefore my docker image size now is ~500Mb, and the previous version - ~60Mb.

[Flarna]

Thanks for the update!
Any improvements in the instrumentation area towards better bundler support are welcome. Maybe bundlers will be adopted at some time to allow hooking into their generated, bundled module loading code to instrument.