The integration does not throw an error, but does not send data to New Relic with some warning message

[gogoidaa101]

Description
I'm trying to set up the integration of New Relic with Salesforce, but the data is not sent to New Relic despite not generating any error logs.
Only the configuration file (config.yml) has been updated with customer ID, secret, username, password and token data. No other modifications were made to the rest of the source code.

Expected Behavior
SF log information sent to New Relic and visible through the standard Dashboard available in New Relic I/O

Relevant Logs / Console output
{"message": "Downloading log lines for log file: /services/data/v60.0/sobjects/EventLogFile/0ATJ4000001EbzXOAS/LogFile", "timestamp": 1725339696826, "level": "info"}
{"message": "Sent 1000 log messages.", "timestamp": 1725339697099, "level": "info"}

Warning message:
Retrying (Retry(total=2, connect=3, read=2, redirect=None, status=None)) after connection broken by 'ReadTimeoutError("HTTPSConnectionPool(host='tplcrm.my.salesforce.com', port=443): Read timed out. (read timeout=5)")': /services/data/v60.0/query?q=SELECT+Id,+Name,+BillingAddress+FROM+Account

Sharing below the config file. Let me know if I have missed anything on that:

`integration_name: com.newrelic.labs.sfdc.eventlogfiles
run_as_service: False
cron_interval_minutes: 60
service_schedule:
hour: "*"
minute: "0,15,30,45"
instances:

• name: sfdc-logs
  arguments:
  api_ver: "60.0"
  token_url: "https://login.salesforce.com/services/oauth2/token"
  auth:
  SF_grant_type: password
  SF_client_id: "3MVG9pe2TCoA1Pf7VL***********0XyJSgV2bWmm8Ere03"
  SF_client_secret: "AD767F9C363F15659FA7A64C2C45B3A"
  SF_username: a.g@tioup.com
  SF_password: "g****@09es$$"
  auth_env_prefix: SF_
  cache_enabled: False
  redis:
  host: my.redis.test
  port: 6379
  db_number: 0
  password: "MY_REDIS_PASSWORD"
  ssl: True
  expire_days: 2
  date_field: LogDate
  generation_interval: Hourly
  time_lag_minutes: 300
  queries:
  • query: "SELECT * FROM Account"
  • query: "SELECT * FROM SetupAuditTrail"
    limits:
    api_ver: "60.0"
    names:
    • ActiveScratchOrgs
    • DailyApiRequests
      logs_enabled: yes
      labels:
      environment: production
      queries:
• query: "SELECT Id,EventType,CreatedDate,LogDate,LogFile,Interval FROM EventLogFile WHERE CreatedDate>={from_timestamp} AND EventType='API' AND Interval='{log_interval_type}'"
• query: "SELECT Id,Action,CreatedDate,DelegateUser,Display FROM SetupAuditTrail WHERE CreatedDate>={from_timestamp}"
  timestamp_attr: CreatedDate
  rename_timestamp: actualTimestamp
  api_ver: "60.0"
• query: "SELECT Value, StartDate, EndDate FROM PlatformEventUsageMetric WHERE StartDate >= {start_date} AND EndDate <= {end_date}"
  env:
  end_date: "now()"
  start_date: "now(timedelta(minutes=-60))"
  api_ver: "60.0"
  timestamp_attr: StartDate
  newrelic:
  data_format: events
  api_endpoint: US
  account_id: "MY_NEW_RELIC_ACCOUNT_ID"
  license_key: "MY_NEW_RELIC_LICENSE_KEY"`

[sdewitt-newrelic]

Hi @gogoidaa101 , the warning message you are seeing is indicating that there is a read timeout when contacting your Salesforce instance. Does this error happen every time you run the integration or is it intermittent? Could you include the entire console output from a run please?

Also, I notice that you seem to be using the sample configuration. That configuration is not designed to be used as-is. It shows an example of every possible configuration parameter merely as an example. For example, the queries in the config_sample.yml are sample queries only. These values may not work in your environment but they are also not integral to the integration itself. In other words, there is no behavior in the integration that leverages a particular example query. Unless you have a need for the information from those queries, you can safely remove them without affecting the behavior of the integration. The purpose of the integration is to collect event logs and to allow you to run any SOQL query but it is not tied to a specific query nor is it tied to fields in the query. You might try starting with the following minimal configuration and see if you can get this working.

integration_name: com.newrelic.labs.sfdc.eventlogfiles
run_as_service: False
instances:
- name: sfdc-logs
  arguments:
      token_url: "MY_SALESFORCE_TOKEN_URL"
      auth:
        grant_type: password
        client_id: "MY_CLIENT_ID"
        client_secret: "MY_CLIENT_SECRET"
        username: "MY_SALESFORCE_USER"
        password: "MY_SALESFORCE_PASSWORD"
    date_field: CreatedDate
    generation_interval: Hourly
    time_lag_minutes: 0
  labels:
    environment: staging
newrelic:
  api_endpoint: US
  account_id: "MY_NEW_RELIC_ACCOUNT_ID"
  license_key: "MY_NEW_RELIC_LICENSE_KEY"

You will need to fill in all the places where you see MY_* with values appropriate for your New Relic and Salesforce accounts. When running with this minimal configuration, the integration will search for Hourly log files and post any log lines find in these files as Log events to New Relic Logs. Please give this a try.

[mca006]

Hey @sdewitt-newrelic , We are facing similar issue. I am trying to pass the values from the environment variable is side the docker container . The config.yml is exactly like above but the Auth values never picked up the environment variables during the execution.

arguments:
api_ver: "55.0"
token_url: $(SF_TOKEN_URL)
auth:
grant_type: password
client_id: "${SF_CLIENT_ID}"
client_secret: "${SF_CLIENT_SECRET}"
username: "#{SF_USERNAME}"
password: "#{SF_PASSWORD}"
auth_env_prefix: MY_

The way my environment variable looks like is MY_SF_TOKEN_URL ,MY_SF_CLIENT_ID and values as appropriate. but seems like process did not find the url or token

we get this error :
raise LoginException(f'salesforce token request failed. status-code:{resp.status_code}, reason: {resp.reason}')
newrelic_logging.LoginException: salesforce token request failed. status-code:400, reason: Bad Request

Scenario with by passing all hard coded values in the config.yml:

So Wanted to know does this newrelic exporter support sending the values to the config.yml from the environment variables either from system env or from application env can this understand or read? what would be the sytax if we have to use ?

in other words If we do use all hard coded value(which is not recommended in the container/application to keep sensitive values ) but still it fails to run the simple query
File "/usr/src/app/src/newrelic_logging/api.py", line 59, in get
raise SalesforceApiException(
newrelic_logging.SalesforceApiException: request error executing api operation: https://XXXX.my.salesforce.com/services/data/v55.0/query?q=SELECT+Id,name+FROM+Account

Can you please verify what we are missing here