Pilot Migration

[Robin-Van-de-Merghel]

DIRAC Migration

We have to migrate some functionalities from DIRAC to DiracX:

Functions

These functions are only used in DIRAC Web, and won't be migrated to DiracX:

• countPilots(countDict, *older, *newer, *timestamp) ***
• getPilotStatistics(attribute, selectDict) ***
• getPilotMonitorSelectors(selectDict, sortList, startItem, maxItems) ***
• [ ] getPilotMonitorWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?) ***
• [ ] getPilotSummaryWeb(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?) ***
• [ ] getGroupedPilotSummary(selectDict, sortList, startItem, maxItems) (<-> getPilotSummary?) ***

Not distributed functions:

• getCurrentPilotCounters(*attrDict)
• getCounters(table, keys, condDict, *newer, *timestamp)

Function not adapted:

• killPilot(pilotRefList), see Moved killPilot to diracAdmin itself DIRAC#8203

In #550

Adapted, and modernized. Linked with DIRACGrid/Pilot#259

### In #511 : Pilot Logging

Refactored as a search function POST /api/pilots/logs. Need to see if getRemotePilotOutput is needed.

• getPilotLoggingInfo(pilotReference)
• storePilotOutPut(pilotReference, output, error)
• getRemotePilotOutput(pilotReference, pilotDict)
• getPilotOutput(pilotReference, pilotDict)

In #509 : feat: added definitions of summary tables (JobDB and PilotAgentsDB)

• getPilotSummary(*startdate, *enddate)

In #421 : Adding pilot registrations and authentification (Router)

• POST /api/pilots/ (add_pilot_stamps(pilot_stamps, vo, *grid_type, *pilot_references))
  • addPilotReferences(pilotRef, VO, *gridType, *pilotStampDict)
• DELETE /api/pilots/ (delete_pilots(pilot_stamps))
  • Needs tests
  • deletePilots
• DELETE /api/pilots/interval (clear_pilots(age_in_days, *delete_only_aborted))
  • Needs tests
  • clearPilots(*interval, *aborted_interval)
• PATCH /api/pilots/fields (update_pilot_fields(pilot_stamps_to_fields_mapping))
  • Needs tests and documentation
  • setPilotBenchmark
  • setAccountingFlag
  • setPilotStatus
• PATCH /api/pilots/fields/jobs (associate_pilot_with_jobs(pilot_stamp, pilot_jobs_ids))
  • setJobForPilot(jobID, pilotRef, *destination)
• GET /api/pilots/search (search(*page, *per_page, *SearchParams))
  • Needs documentation
  • getPilotInfo(pilotReference)
  • [x] getPilots (equiv: getPilotInfo)
  • [x] selectPilots(condDict) (equiv: getPilotInfo)

New Pilot Features in DiracX

• Pilot Auth
  • Secret
    • Multi-VO
    • Limited / Unlimited in time
    • Count Max
    • Retrieve secret in SiteDirector before sending the pilot (or vacuum)
  • Token
  • Refresh Token
  • Login (POST /api/auth/pilot-login)
  • Access Policies
    • Pilot Management
      • Basic Restrictions
      • [ ] Owner-Based Restrictions (No owner)
    • Job Access in Splitted the job endpoints to separate Pilots and Users #468 : Improving wmsjobpolicy to contain pilots
    • Log Access

Pilot Side

Associated PR: DIRACGrid/Pilot#248

• Login
• Refresh token
• Read DiracX CS (partially, not filtered)
• Jobs
• Logs
• More to be adapted...

[fstagni]

Partially related: DIRACGrid/DIRAC#8190 (moving few WebApp-only related RPC calls to separate handler)

[Robin-Van-de-Merghel]

All PRs for auth:

---
config:
  logLevel: 'debug'
  theme: 'base'
---
gitGraph
    commit
    commit
    branch "Pilot management"
    checkout "Pilot management"
    commit
    commit
    commit
    branch "Pilot auth"
    checkout "Pilot auth"
    commit
    commit
    commit
    branch "Pilot logging"
    checkout "Pilot logging"
    commit
    commit
    commit
    checkout "Pilot auth"
    branch "Splitting jobs endpoints"
    checkout "Splitting jobs endpoints"
    commit
    commit
    commit

Loading