Skip to content

Allow the use of the size parameter inside a query #142

Open
@Sergi-GC

Description

@Sergi-GC

For all general issues, please provide the following details for fast resolution:

  • Version: 7.9.1
  • Config File (if you have sensitive info, please remove it):
input {
  elasticsearch {
    cloud_id => "<deployment_endpoint>"
    cloud_auth => "<username>:<password>"
    index => "<filebeat-*>"
    schedule => "*/5 * * * *"
    query => '{ "size": 10, "query": { "bool": { "must_not": [ { "query_string": { "query": "(starting)", "default_field": "status" } } ], "filter": [ { "bool": { "must_not": { "exists": { "field": "event_ts" } } } } ] } } }'
    docinfo => true
  }
}
  • Use case: Replacement for the Elapsed filter plugin ingesting Kafka logs
  • Sample Data: Kafka logs
  • Workaround: Reducing the number of hits with a range filter or by limiting the number of indices to query...
  • Steps to Reproduce:

While using the Elasticsearch input plugin to query against a set of indices, it seems not possible to limit the number of documents returned by the query.

The size parameter of the Elasticsearch input plugin is used with scroll queries to limit the number of items per page in the scroll, not the total number of documents.

And the size parameter inside the query is ignored and the query always returns the default 10.000 documents.

This was reported in the discuss forum with a simple example:
https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html

I'd like to be able to use the size parameter inside the query field to limit the number of documents returned.

Related: #85

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions