Bug: Using pagination for non-existing page throws internal server error

[cudeso]

MISP-STIX usage

MISP core

Expected behavior

Query for stix2 results with pagination and return 0 results instead of an error. Or at least not an "An Internal Error Has Occurred"

Actual behavior

Not sure if this is a bug or by design, but when requesting results (stix2) for pages that do not exist the server returns a 500 error.

Use: to limit memory usage and not have the server return all STIX2 events/attributes in one set I query the server with a limit and page option (a simple counter, query 50 events at once). Unfortunately there's no way (?) in telling how many pages there are or if there is any page left in the query. So the solution is to request the results, until the server returns an empty result set, unfortunately this also throws an error.

Unknown error: the response is not in JSON.
Something is broken server-side, please send us everything that follows (careful with the auth key):
Request headers:
{'User-Agent': 'PyMISP 2.4.171 - Python 3.8', 'Accept-Encoding': 'gzip, deflate', 'Accept': 'application/json', 'Connection': 'keep-alive', 'Cookie': 'removed', 'Content-Length': '353', 'content-type': 'application/json'}
Request body:
{"returnFormat": "stix2", "page": 2, "limit": 50, "tags": ["tlp:clear"], "withAttachments": 0, "metadata": 0, "published": true, "enforceWarninglist": 0, "includeEventUuid": 0, "includeEventTags": 0, "sgReferenceOnly": 0, "includeContext": 0, "headerless": 0, "includeSightings": 0, "includeDecayScore": 0, "includeCorrelations": 0, "excludeDecayed": 0}
Response (if any):
{"name":"An Internal Error Has Occurred.","message":"An Internal Error Has Occurred.","url":"\/events\/restSearch"}

2023-07-03 14:57:25 Error: [Exception] Error while processing your query during STIX export: Arguments error, please check you entered a valid version and provided input file names.
Request URL: /events/restSearch
Stack Trace:
#0 /data/www/MISP/app/Model/Event.php(7277): StixExport->footer()
#1 /data/www/MISP/app/Controller/AppController.php(1314): Event->restSearch()
#2 [internal function]: AppController->restSearch()
#3 /data/www/MISP/app/Lib/cakephp/lib/Cake/Controller/Controller.php(499): ReflectionMethod->invokeArgs()
#4 /data/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(193): Controller->invokeAction()
#5 /data/www/MISP/app/Lib/cakephp/lib/Cake/Routing/Dispatcher.php(167): Dispatcher->_invoke()
#6 /data/www/MISP/app/webroot/index.php(99): Dispatcher->dispatch()

Steps to reproduce

misp_page = 0
while remaining_misp_pages:

     try:
            result = misp.search(controller='events', return_format='stix2', **config.misp_event_filters, limit=config.misp_event_limit_per_page, page=misp_page)
            misp_page += 1
     except Exception as e:
            remaining_misp_pages = False
            logger.info("Finished receiving MISP events.")

Version

2.4.171

Python version

3.8

Relevant log output

No response

Extra attachments

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[chrisr3d]

I had a look at it today, and identified the origin of the issue, which is I forgot to consider the case where the input is empty, in the Export lib in MISP - somewhere in this method.

I remembered the JSON export still returns {"response": []} event when the result is empty, because the handler method is the one handling the conversion of the actual content.
With the STIX export, the header and handler methods are preparing the input files that are passed to the conversion feature - i.e misp-stix - but the whole conversion process happens only when the footer method makes the call.

So with an if statement there to test whether there actually is an input or not, we can avoid the issue.
Now the question is do I want to return an error or an empty result like the JSON export does? Will discuss it and let you know. If you already have a preference don't hesitate to let me know

[chrisr3d]

So with an if statement there to test whether there actually is an input or not, we can avoid the issue. Now the question is do I want to return an error or an empty result like the JSON export does? Will discuss it and let you know. If you already have a preference don't hesitate to let me know

Our preference is an empty list, I'll check whether returning and empty Bundle makes sense / is valid somehow

[chrisr3d]

I've pushed the change mentioned above, that should avoid issues by returning an empty result.

It is on develop and will be merged with the next release.
I'll let you test it on your side to make sure it doesn't raise any other issue

[cudeso]

Ha! Thank you very much for the update. I'll test it soon.

To avoid the error from triggering, I switched the logic for the misp2sentinel integration to first query MISP in a "traditional" way (restsearch > JSON), and then locally do the transformation to STIX for each returned event (https://github.com/cudeso/misp2sentinel/blob/upload_indicators_api/script.py#L98 ) with the MISPtoSTIX21Parser().