Skip to content

CrUX API request fails behind proxy #3874

Closed
Closed
CrUX API request fails behind proxy#3874
Labels
bug

Description

@diginote
diginote
opened on Jun 8, 2023

Have you read the documentation?

URL

https://www.google.de

What are you trying to accomplish

I tried to run the crux plugin behind a proxy in a docker container with this cli command:

docker run \
        -e NODE_DEBUG=net,http,tls \
        -e http_proxy=http://$HTTP_PROXY \
        -e https_proxy=http://$HTTP_PROXY \
        -e $TIMEZONE \
        --shm-size 2g \
        --rm \
        --network=$DOCKER_NETWORK \
        -v $RUN_VOLUME_MAP3 \
        sitespeedio/sitespeed.io:27.8.0-plus1 \
        --graphite.host=$GRAPHITE_HOST_NAME \
        --graphite.namespace sitespeed_io.desktop \
        --proxy.http=$HTTP_PROXY \
        --proxy.https=$HTTP_PROXY \
        --crux.key $CRUX_API_KEY \
        --crux.formFactor PHONE \
        --crux.formFactor DESKTOP \
        --crux.formFactor TABLET \
        --crux.formFactor ALL \
        --plugins.remove /gpsi/lib/index.js \
        --plugins.remove /lighthouse/index.js \
        --plugins.remove assets \
        --plugins.remove browsertime \
        --plugins.remove budget \
        --plugins.remove coach \
        --plugins.remove domains \
        --plugins.remove graphite \
        --plugins.remove harstorer \
        --plugins.remove html \
        --plugins.remove lateststorer \
        --plugins.remove metrics \
        --plugins.remove pagexray \
        --plugins.remove remove \
        --plugins.remove text \
        --plugins.remove thirdparty \
        --plugins.remove tracestorer \
        --browsertime.proxy.https=$HTTP_PROXY \
        --browsertime.proxy.http=$HTTP_PROXY \
        --v \
        sites.txt.crux

this is the result:

[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for url https://www....
HTTP 43: call onSocket 4 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 106 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 4,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 5 5
HTTP 43: write ret = true
HTTP 43: outgoing message end.
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for domain ...
HTTP 43: call onSocket 5 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 63 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 5,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 6 6
HTTP 43: write ret = true
HTTP 43: outgoing message end.
(node:43) Warning: Setting the NODE_DEBUG environment variable to 'http' can expose sensitive data (such as passwords, tokens and authentication headers) in the resulting log.
(Use `node --trace-warnings ...` to show where the warning was created)
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
NET 43: afterConnect
NET 43: destroy
NET 43: close
NET 43: close handle
HTTP 43: SOCKET ERROR: connect ECONNREFUSED 142.250.186.138:443 Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '142.250.186.138',
  port: 443
}

Node.js v18.16.0

this are two sections of the configuration:

"browsertime": {
      "proxy": {
        "https": "my_proxy:3128",
        "http": "my_proxy:3128"
      }
	}
	
"proxy": {
      "http": "my_proxy:3128",
      "https": "my_proxy:3128"
    }

Although the proxy was set correctly, it seems to be simply ignored during the request. I think the reason is that it is not respected here: https://github.com/sitespeedio/sitespeed.io/blob/main/lib/plugins/crux/send.js

When I try to curl the googleapis endpoint from within the container everythings works:

# start shell in container
docker run  -e NODE_DEBUG=net,http,tls -e HTTP_PROXY=http://my_proxy:3128 -e HTTPS_PROXY=http://my_proxy:3128 --rm -it --entrypoint bash sitespeedio/sitespeed.io:27.8.0-plus1



#curl endpoint
curl --location 'https://chromeuxreport.googleapis.com/v1/records:queryRecord?key=...' \
--header 'Content-Type: application/json' \
--data '{
  "origin": "https://www.../"
}'

Everything is fine when I run it on my localhost (without a proxy).
It may be possible to use axios here. It already respects the environment variables set for a proxy (see http_proxy or https_proxy on https://axios-http.com/docs/req_config)

What browser did you use?

Chrome

How to reproduce

run behind a proxy


docker run \
        -e NODE_DEBUG=net,http,tls \
        -e http_proxy=http://$HTTP_PROXY \
        -e https_proxy=http://$HTTP_PROXY \
        -e $TIMEZONE \
        --shm-size 2g \
        --rm \
        --network=$DOCKER_NETWORK \
        -v $RUN_VOLUME_MAP3 \
        sitespeedio/sitespeed.io:27.8.0-plus1 \
        --graphite.host=$GRAPHITE_HOST_NAME \
        --graphite.namespace sitespeed_io.desktop \
        --proxy.http=$HTTP_PROXY \
        --proxy.https=$HTTP_PROXY \
        --crux.key $CRUX_API_KEY \
        --crux.formFactor PHONE \
        --crux.formFactor DESKTOP \
        --crux.formFactor TABLET \
        --crux.formFactor ALL \
        --plugins.remove /gpsi/lib/index.js \
        --plugins.remove /lighthouse/index.js \
        --plugins.remove assets \
        --plugins.remove browsertime \
        --plugins.remove budget \
        --plugins.remove coach \
        --plugins.remove domains \
        --plugins.remove graphite \
        --plugins.remove harstorer \
        --plugins.remove html \
        --plugins.remove lateststorer \
        --plugins.remove metrics \
        --plugins.remove pagexray \
        --plugins.remove remove \
        --plugins.remove text \
        --plugins.remove thirdparty \
        --plugins.remove tracestorer \
        --browsertime.proxy.https=$HTTP_PROXY \
        --browsertime.proxy.http=$HTTP_PROXY \
        --v \
        sites.txt.crux

sites.txt.crux contains lines like

https://www.google.de 01-start-page google-com



### Relevant log output

```shell
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for url https://www....
HTTP 43: call onSocket 4 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 106 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 4,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 5 5
HTTP 43: write ret = true
HTTP 43: outgoing message end.
[2023-06-08 10:52:15] INFO: [plugin.crux] Get CrUx data for domain ...
HTTP 43: call onSocket 5 0
HTTP 43: createConnection chromeuxreport.googleapis.com:443::::::::::::::::::::: [Object: null prototype] {
  host: 'chromeuxreport.googleapis.com',
  port: 443,
  path: null,
  headers: { 'Content-Type': 'application/json', 'Content-Length': 63 },
  method: 'POST',
  _defaultAgent: Agent {
    _events: [Object: null prototype] {
      free: [Function (anonymous)],
      newListener: [Function: maybeEnableKeylog]
    },
    _eventsCount: 2,
    _maxListeners: undefined,
    defaultPort: 443,
    protocol: 'https:',
    options: [Object: null prototype] { noDelay: true, path: null },
    requests: [Object: null prototype] {},
    sockets: [Object: null prototype] {
      'chromeuxreport.googleapis.com:443:::::::::::::::::::::': [Array]
    },
    freeSockets: [Object: null prototype] {},
    keepAliveMsecs: 1000,
    keepAlive: false,
    maxSockets: Infinity,
    maxFreeSockets: 256,
    scheduling: 'lifo',
    maxTotalSockets: Infinity,
    totalSocketCount: 5,
    maxCachedSessions: 100,
    _sessionCache: { map: {}, list: [] },
    [Symbol(kCapture)]: false
  },
  noDelay: true,
  servername: 'chromeuxreport.googleapis.com',
  _agentKey: 'chromeuxreport.googleapis.com:443:::::::::::::::::::::'
}
TLS 43: client _init handle? true
NET 43: pipe false null
NET 43: connect: find host chromeuxreport.googleapis.com
NET 43: connect: dns options { family: undefined, hints: 32 }
HTTP 43: sockets chromeuxreport.googleapis.com:443::::::::::::::::::::: 6 6
HTTP 43: write ret = true
HTTP 43: outgoing message end.
(node:43) Warning: Setting the NODE_DEBUG environment variable to 'http' can expose sensitive data (such as passwords, tokens and authentication headers) in the resulting log.
(Use `node --trace-warnings ...` to show where the warning was created)
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
TLS 43: client initRead handle? true buffered? false
NET 43: _read
NET 43: _read wait for connection
NET 43: afterConnect
NET 43: destroy
NET 43: close
NET 43: close handle
HTTP 43: SOCKET ERROR: connect ECONNREFUSED 142.250.186.138:443 Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: connect ECONNREFUSED 142.250.186.138:443
    at TCPConnectWrap.afterConnect [as oncomplete] (node:net:1494:16)
Emitted 'error' event on ClientRequest instance at:
    at TLSSocket.socketErrorListener (node:_http_client:502:9)
    at TLSSocket.emit (node:events:513:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  errno: -111,
  code: 'ECONNREFUSED',
  syscall: 'connect',
  address: '142.250.186.138',
  port: 443
}

Node.js v18.16.0

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions