Test potential race condition in topology

[Ktmi]

In kytos-ng/topology#281 an issue with a potential race condition, regarding updating/deleting switches was identified. These tests attempt to invoke such behaviour.

End-to-End Tests

In the current master branch of topology, these tests were able to invoke the race condition in the second case.

kytos-1  | Starting enhanced syslogd: rsyslogd.
kytos-1  | /etc/openvswitch/conf.db does not exist ... (warning).
kytos-1  | Creating empty database /etc/openvswitch/conf.db.
kytos-1  | Starting ovsdb-server.
kytos-1  | rsyslogd: error during config processing: omfile: chown for file '/var/log/syslog' failed: Operation not permitted [v8.2302.0 try https://www.rsyslog.com/e/2207 ]
kytos-1  | Configuring Open vSwitch system IDs.
kytos-1  | Starting ovs-vswitchd.
kytos-1  | Enabling remote OVSDB managers.
kytos-1  | There is no NAPPS_PATH specified. Default will be used.
kytos-1  | + '[' -z '' ']'
kytos-1  | + '[' -z '' ']'
kytos-1  | + echo 'There is no NAPPS_PATH specified. Default will be used.'
kytos-1  | + NAPPS_PATH=
kytos-1  | + sed -i 's/STATS_INTERVAL = 60/STATS_INTERVAL = 7/g' /var/lib/kytos/napps/kytos/of_core/settings.py
kytos-1  | + sed -i 's/CONSISTENCY_MIN_VERDICT_INTERVAL =.*/CONSISTENCY_MIN_VERDICT_INTERVAL = 60/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
kytos-1  | + sed -i 's/LINK_UP_TIMER = 10/LINK_UP_TIMER = 1/g' /var/lib/kytos/napps/kytos/topology/settings.py
kytos-1  | + sed -i 's/DEPLOY_EVCS_INTERVAL = 60/DEPLOY_EVCS_INTERVAL = 5/g' /var/lib/kytos/napps/kytos/mef_eline/settings.py
kytos-1  | + sed -i 's/LLDP_LOOP_ACTIONS = \["log"\]/LLDP_LOOP_ACTIONS = \["disable","log"\]/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos-1  | + sed -i 's/LLDP_IGNORED_LOOPS = {}/LLDP_IGNORED_LOOPS = {"00:00:00:00:00:00:00:01": \[\[4, 5\]\]}/' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos-1  | + sed -i 's/CONSISTENCY_COOKIE_IGNORED_RANGE =.*/CONSISTENCY_COOKIE_IGNORED_RANGE = [(0xdd00000000000000, 0xdd00000000000009)]/g' /var/lib/kytos/napps/kytos/flow_manager/settings.py
kytos-1  | + sed -i 's/LIVENESS_DEAD_MULTIPLIER =.*/LIVENESS_DEAD_MULTIPLIER = 3/g' /var/lib/kytos/napps/kytos/of_lldp/settings.py
kytos-1  | + kytosd --help
kytos-1  | + sed -i s/WARNING/INFO/g /etc/kytos/logging.ini
kytos-1  | + sed -i 's/keys: root,kytos,api_server,socket/keys: root,kytos,api_server,socket,aiokafka/' /etc/kytos/logging.ini
kytos-1  | + echo -e '\n\n[logger_aiokafka]\nlevel: INFO\nhandlers:\nqualname: aiokafka'
kytos-1  | + test -z ''
kytos-1  | + TESTS=tests/
kytos-1  | + test -z ''
kytos-1  | + RERUNS=2
kytos-1  | + python3 scripts/wait_for_mongo.py
kytos-1  | Trying to run hello command on MongoDB...
kytos-1  | Trying to run 'hello' command on MongoDB...
kytos-1  | Trying to run 'hello' command on MongoDB...
kytos-1  | Ran 'hello' command on MongoDB successfully. It's ready!
kytos-1  | + python3 scripts/setup_kafka.py
kytos-1  | Starting setup_kafka.py...
kytos-1  | Attempting to create an admin client at ['broker1:19092', 'broker2:19092', 'broker3:19092']...
kytos-1  | Admin client was successful! Attempting to validate cluster...
kytos-1  | Cluster info: {'throttle_time_ms': 0, 'brokers': [{'node_id': 1, 'host': 'broker1', 'port': 19092, 'rack': None}, {'node_id': 2, 'host': 'broker2', 'port': 19092, 'rack': None}, {'node_id': 3, 'host': 'broker3', 'port': 19092, 'rack': None}], 'cluster_id': '5L6g3nShT-eMCtK--X86sw', 'controller_id': 1}
kytos-1  | Cluster was successfully validated! Attempting to create topic 'event_logs'...
kytos-1  | Topic 'event_logs' was created! Attempting to close the admin client...
kytos-1  | Kafka admin client closed.
kytos-1  | + python3 -m pytest tests/test_e2e_07_topology.py
kytos-1  | ============================= test session starts ==============================
kytos-1  | platform linux -- Python 3.11.2, pytest-8.4.2, pluggy-1.6.0
kytos-1  | rootdir: /
kytos-1  | configfile: pytest.ini
kytos-1  | plugins: asyncio-1.1.0, rerunfailures-13.0, timeout-2.2.0, anyio-4.3.0
kytos-1  | asyncio: mode=Mode.AUTO, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
kytos-1  | collected 2 items
kytos-1  | 
kytos-1  | tests/test_e2e_07_topology.py .F                                         [100%]
kytos-1  | 
kytos-1  | =================================== FAILURES ===================================
kytos-1  | _______________ TestE2EMefEline.test_020_topology_race_condition _______________
kytos-1  | 
kytos-1  | self = <tests.test_e2e_07_topology.TestE2EMefEline object at 0x7648c480c110>
kytos-1  | 
kytos-1  |     async def test_020_topology_race_condition(self):
kytos-1  |     
kytos-1  |         self.net.stop_kytosd()
kytos-1  |     
kytos-1  |         link_count = 1000
kytos-1  |         attempt_count = 100
kytos-1  |     
kytos-1  |         # fake db entries
kytos-1  |         switch_entry = {
kytos-1  |             '_id': '00:00:00:00:00:00:00:01',
kytos-1  |             'connection': '127.0.0.1:50074',
kytos-1  |             'data_path': 's1',
kytos-1  |             'enabled': False,
kytos-1  |             'hardware': 'Open vSwitch',
kytos-1  |             'id': '00:00:00:00:00:00:00:01',
kytos-1  |             'inserted_at': datetime(
kytos-1  |                 2025,
kytos-1  |                 10,
kytos-1  |                 9,
kytos-1  |                 18,
kytos-1  |                 26,
kytos-1  |                 33,
kytos-1  |                 144000
kytos-1  |             ),
kytos-1  |             'interfaces': [
kytos-1  |                 {
kytos-1  |                     'id': '00:00:00:00:00:00:00:01:4294967294',
kytos-1  |                     'enabled': False,
kytos-1  |                     'mac': '6e:3c:76:aa:05:40',
kytos-1  |                     'speed': 0.0,
kytos-1  |                     'port_number': 4294967294,
kytos-1  |                     'name': 's1',
kytos-1  |                     'nni': False,
kytos-1  |                     'lldp': False,
kytos-1  |                     'switch': '00:00:00:00:00:00:00:01',
kytos-1  |                     'link': '',
kytos-1  |                     'link_side': None,
kytos-1  |                     'metadata': {},
kytos-1  |                     'updated_at': None
kytos-1  |                 },
kytos-1  |                 *[
kytos-1  |                     {
kytos-1  |                         'id': f'00:00:00:00:00:00:00:01:{i}',
kytos-1  |                         'enabled': False,
kytos-1  |                         'mac': 'be:2a:12:c4:c4:52',
kytos-1  |                         'speed': 1250000000.0,
kytos-1  |                         'port_number': i,
kytos-1  |                         'name': f's1-eth{i}',
kytos-1  |                         'nni': False,
kytos-1  |                         'lldp': False,
kytos-1  |                         'switch': '00:00:00:00:00:00:00:01',
kytos-1  |                         'link': '',
kytos-1  |                         'link_side': None,
kytos-1  |                         'metadata': {},
kytos-1  |                         'updated_at': None
kytos-1  |                     }
kytos-1  |                     for i in range(1, link_count + 1)
kytos-1  |                 ]
kytos-1  |             ],
kytos-1  |             'manufacturer': 'Nicira, Inc.',
kytos-1  |             'metadata': {},
kytos-1  |             'ofp_version': '0x04',
kytos-1  |             'serial': 'None',
kytos-1  |             'software': '3.3.0',
kytos-1  |             'updated_at': datetime(
kytos-1  |                 2025,
kytos-1  |                 10,
kytos-1  |                 9,
kytos-1  |                 18,
kytos-1  |                 26,
kytos-1  |                 33,
kytos-1  |                 172000
kytos-1  |             )
kytos-1  |         }
kytos-1  |         self.net.db["switches"].insert_one(switch_entry)
kytos-1  |     
kytos-1  |         self.net.start_controller()
kytos-1  |     
kytos-1  |         time.sleep(5)
kytos-1  |     
kytos-1  |         api_url = KYTOS_API + "/kytos/topology/v3/"
kytos-1  |     
kytos-1  |         response = httpx.get(api_url)
kytos-1  |     
kytos-1  |         assert response.status_code == 200, response.text
kytos-1  |     
kytos-1  |         data = response.json()
kytos-1  |     
kytos-1  |         assert "00:00:00:00:00:00:00:01" in data["topology"]["switches"]
kytos-1  |     
kytos-1  |         async def update_interfaces(client: httpx.AsyncClient, switch_id):
kytos-1  |             api_url = KYTOS_API + f"/kytos/topology/v3/interfaces/switch/{switch_id}/disable"
kytos-1  |             response = await client.post(api_url)
kytos-1  |             assert response.status_code < 500, response.text
kytos-1  |     
kytos-1  |         async def delete_switch(client: httpx.AsyncClient, switch_id):
kytos-1  |             api_url = KYTOS_API + f"/kytos/topology/v3/switches/{switch_id}"
kytos-1  |             response = await client.delete(api_url)
kytos-1  |             assert response.status_code == 200, response.text
kytos-1  |     
kytos-1  |         async with httpx.AsyncClient(timeout=120) as client:
kytos-1  |             await asyncio.gather(
kytos-1  |                 *[
kytos-1  |                     update_interfaces(client, '00:00:00:00:00:00:00:01')
kytos-1  |                     for i in range(1, attempt_count // 2)
kytos-1  |                 ],
kytos-1  |                 delete_switch(client, '00:00:00:00:00:00:00:01'),
kytos-1  |                 *[
kytos-1  |                     update_interfaces(client, '00:00:00:00:00:00:00:01:')
kytos-1  |                     for i in range(attempt_count // 2, attempt_count + 1)
kytos-1  |                 ],
kytos-1  |             )
kytos-1  |     
kytos-1  |             api_url = KYTOS_API + f"/kytos/topology/v3/switches"
kytos-1  |     
kytos-1  |             response = await client.get(api_url)
kytos-1  |     
kytos-1  |             assert response.status_code == 200, response.text
kytos-1  |     
kytos-1  |             data = response.json()
kytos-1  |     
kytos-1  |             assert not data["switches"]
kytos-1  |     
kytos-1  |         self.net.stop_kytosd()
kytos-1  |     
kytos-1  |         remaining_switches = list(self.net.db["switches"].find({}))
kytos-1  |     
kytos-1  | >       assert not remaining_switches
kytos-1  | E       AssertionError: assert not [{'_id': '00:00:00:00:00:00:00:01', 'connection': '', 'data_path': 's1', 'enabled': False, ...}]
kytos-1  | 
kytos-1  | tests/test_e2e_07_topology.py:281: AssertionError
kytos-1  | =============================== warnings summary ===============================
kytos-1  | tests/test_e2e_07_topology.py::TestE2EMefEline::test_020_topology_race_condition
kytos-1  |   /usr/lib/python3/dist-packages/mininet/node.py:1121: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
kytos-1  |     return ( StrictVersion( cls.OVSVersion ) <
kytos-1  | 
kytos-1  | tests/test_e2e_07_topology.py::TestE2EMefEline::test_020_topology_race_condition
kytos-1  |   /usr/lib/python3/dist-packages/mininet/node.py:1122: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
kytos-1  |     StrictVersion( '1.10' ) )
kytos-1  | 
kytos-1  | -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
kytos-1  | ------------------------------- start/stop times -------------------------------
kytos-1  | tests/test_e2e_07_topology.py::TestE2EMefEline::test_020_topology_race_condition: 2025-10-20,17:33:17.087958 - 2025-10-20,17:34:31.312142
kytos-1  | =========================== short test summary info ============================
kytos-1  | FAILED tests/test_e2e_07_topology.py::TestE2EMefEline::test_020_topology_race_condition
kytos-1  | ============= 1 failed, 1 passed, 2 warnings in 218.18s (0:03:38) ==============

�[Kkytos-1 exited with code 1

[viniarck]

Nicely done @Ktmi, great to have coverage for these cases that are being fixed, and nice that it caught also the race cond. Overall looks good, I left some comments to improve trivial code readability stuff

[viniarck]

Nicely done @Ktmi. Leaving it pre-approved, go ahead and merge it when its dependencies are ready too.