possible regression in quadplane attitude controller?

[tridge]

I've had some strange and quite severe oscillations with a GriffinPro tilt quadplane recently when running master. The oscillations are bad enough that I am at risk of losing the aircraft. The odd thing is that running the same parameters on 4.5.7 I have not been able to reproduce the issue.
There are some complicating factors, so it isn't absolutely certain this is a regression, but I think it is worth looking into.
This is what happens on master:

notice the increasing amplitude of oscillation in QLOITER. Changing to QHOVER did allow it to slowly recover, but still huge overshoots.
This is the same parameters on 4.5.7:

there is still some overshoot, but not dangerous.
the complicating factors are:

• the tune is pretty bad. I don't know quite how I ended up with this tune, it is clearly a manual tune as it has round numbers for the gains. My past logs showed I changed to these gains sometime last year, but I can't remember why. Maybe some experiments I was doing at the time? The rate PID gains are clearly too low for the angle P gain of 10. I could retune it easily and make it fly a lot better, but my concern isn't getting a good tune, it is explaining why master seems much worse than 4.5.7 in case this impacts other users
• I recently increased the payload a bit, adding a RPi4 and a 4G radio, which I suspect is why it is worse now than it has been on previous flights, but as a percentage of total weight the extra payload is small (maybe 250g out of 10kg or so?)
• the ESCs are blheli32/dshot, but are going via a L431 DroneCAN adapter, so changes in the CAN code need to be considered as well as high level quadplane and attitude controller

Some things I've looked at:

• given the number of filtering changes in master compared to 4.5.7 I looked at the unfiltered vs filtered gyro data and I don't see any filter induced lag in the logs
• I'm intrigued by the flat bottom on the RATE.RDes graph above for master. That is probably the 75 deg/s limit from Q_A_RATE_R_MAX, but I don't see an equivalent flat bottom on the tests with 4.5.7. Some change in the rate limiting? Or just the different way of triggering the oscillation (I had to use the sticks to start the movement in 4.5.7 whereas with master it spontaneously oscillates when descending).
• i looked at the RPM feedback from the ESCs versus the demand in RCOU and I don't see any higher lag in master than in 4.5.7.

I haven't tested the 4.6.0 beta. To do that safely I think I'll need to retune first to get a safe set of gains, then use a script to switch to these gains to see if 4.6.0 is affected.

Logs are here:
http://uav.tridgell.net/tmp/GriffinProOscillation/

[tridge]

I chatted to @andyp1per and he thought this change is a possible cause?
7047e11

[lthall]

I chatted to @andyp1per and he thought this change is a possible cause? 7047e11

This is only used in copter guided so I can't see how this could be a factor.

Have I missed something?

[tridge]

Have I missed something?

I doubt you've missed anything, Andy and I only had a very quick look, sorry

[lthall]

Ok, so the difference in green desired rate lines where the top appears to be experiancing limiting but the lower isn't is due to the source of the rate command. In the first graph the rate command is coming from the attitude error and that is limited. In the second it comes from the commanded attitude and is not limited.

[lthall]

This is also a combination of roll and yaw:

It looks like the tune was on the edge and got into a corner of the flight evelope that was properly unstable with some cross compling between roll and yaw.

[timtuxworth]

I maybe have seen a similar problem today with my VTBird tilt rotor. As with yours I haven't flown it for a long time, and it had a "not great" (but flyable) tune on it, but I put a new build based on master on it for some other testing I'm I'm doing and just tried to do a basic hover a couple of meters off the ground. It was scary.
Then I put 4.5.7 Official on, the only change was to disable scripting and a minor tweak to one of the tilt rotor trims. Then I put it in the air (just a couple of meters in QLoiter) and it was much better.

Log of 4.7.0 (+ my "interlock" code) - https://www.dropbox.com/scl/fi/tvyeji5xwl80f6gxwz5bd/log_164_2025-1-15-11-35-08.bin?rlkey=o3nbav6p6t11h5gaeqr0lorf9&st=4oz9hglz&dl=0
Here's a video of part of that flight: https://youtu.be/aYb0AAqVtn0

Log of 4.5.7 (official from MP) - https://www.dropbox.com/scl/fi/amj418e15qbac4lgk5sib/log_166_2025-1-15-11-59-10.bin?rlkey=lp2mm44hwhc7jdmlk6ayj3y0t&st=8nz0lxat&dl=0
Here's a video of the much more stable flight: https://youtu.be/EGJ1jaxr5Ko

There is a slow oscillation on the yaw tune, which I've never been able to break on this plane.

[lthall]

I maybe have seen a similar problem today with my VTBird tilt rotor. As with yours I haven't flown it for a long time, and it had a "not great" (but flyable) tune on it, but I put a new build based on master on it for some other testing I'm I'm doing and just tried to do a basic hover a couple of meters off the ground. It was scary. Then I put 4.5.7 Official on, the only change was to disable scripting and a minor tweak to one of the tilt rotor trims. Then I put it in the air (just a couple of meters in QLoiter) and it was much better.

I have went over these logs too and it does look like the output is messier. It isn't night and day but the old code looks like it is generally softer in it's response.

It would be good to compare the Alt_Hold performance to seperate attitude control from position control.

Looking through the code changes there is a reasonably significant number of changes but I can't find anything that should cause a performance change:

Attitude controller is primarily naming but with a one loop movement of the update_attitude_target(); This should not do anything.
Position controller has a lot of offset work, some sysID work, but I can't see anything that should change the controller.
AC_Loiter looks fine.
I don't see anything in the quadplane.cpp code but I am moving out of my depth now.
Nothing in tiltrotor.cpp
There is a fair amount of work in servo.cpp

I got nothing, other than trying to break this down and comparing Alt_hold and Loiter to see if this is more closly related to attitude or position control would be interesting.

[timtuxworth]

The weather is looking nasty for the next week at least. I'll try to fly it in QHover/AltHold as soon as I can.

[timtuxworth]

There is a fair amount of work in servo.cpp

Given it's a tilt-rotor, this might be important.

[timtuxworth]

I flew this again today on 4.5.7 vs 4.6.0 beta 2. Two logs, one for each firmware. I flew Stabilize and QHover and QLoiter. This is on my VTBird Tilt-Rotor. Then I did a flight with a transition and back transition only on 4.6.0, it was very shaky on the transitions.

Subjectively, it feels to me like the plane is twitchier with 4.6.0, and more "solid" to the feel on 4.5.7. When flying QStabilize it felt like as if I had some high expo/rates on my transmitter in 4.5.7, and turned the expo/rates way down on 4.6.0. Of course I didn't actually do that, there was zero change on the transmitter settings between flights. When flying 4.6.0, I had to consciously be careful to not move the sticks as much as I would on 4.5.7.

So then I'd suggest that the AP code is reacting in the same "twitchier" way, regardless whether the input comes from the AP itself (e.g. when flying QLoiter, or when doing a QRTL), or if I'm providing the inputs on the sticks. Any input is more exaggerated as an output on 4.6.0.

Does that make sense?

Logs:
Comparative with QStabilize, QHover, QLoiter
4.5.7: https://www.dropbox.com/scl/fi/fv67sop6wh85yrbxa4htn/log_171_2025-1-25-11-38-42.bin?rlkey=kxq76j6vov5r3mx9mn9rndvls&dl=0
4.6.0: https://www.dropbox.com/scl/fi/3hkwkne5q1e175j57sbi0/log_172_2025-1-25-12-04-50.bin?rlkey=t59x3yf2rbbtr4urj3z2ht32m&dl=0

This log contains 2 flights running 4.6.0 beta 2 where under pilot control, I took off in QLoiter, switched to FBWA, flew a bit then hit RTL
https://www.dropbox.com/scl/fi/yzfvx3ultpevn2br0i63t/log_173_2025-1-25-12-26-42.bin?rlkey=lo3llig4tks1p5od2j0p8p62g&dl=0

[timtuxworth]

Following a suggestion from @tpwrules, I seem to have narrowed it down to this commit (75594dd), from this PR #28568. not sure why, but subjectively, if I fly this commit the plane feels very loose and floppy

If I fly the previous commit vs this one in QStabilize on RealFlight on a Griffin Pro, the plane is much more solid and locked on the previous commit (81a30e8).

[bnsgeyer]

@tridge @rmackay9 @lthall
I have been doing some investigation into this behavior. Initially Tridge and I worked to take the quadplane SITL model and give it some lightly damped characteristics (bad tune) that showed a roll oscillation around 1 hz. Tridge tried to excite the dynamics using the AHRS_TRIM param and see if he could get the same behavior as what he saw with his real aircraft. He reported that he didn't have much luck.

I saw that a user had posted something similar (or at least sounded similar) to what Tridge experienced with the quadplane but with a multirotor. So I decided to explore this in Copter using a state space model that I developed for the 3DR X8-M multirotor. I looked at two tune configurations, one with good tune and one with a poor tune that had very obvious oscillatory behavior around 1 hz.

I performed frequency sweeps of the attitude controller for both configurations in both 4.5.7 and 4.6-beta3. The frequency responses of the aircraft with a good tune looked nearly identical. There may have been a slight bit of phase loss in the 4.6 response but I would consider it negligible (equated to about 2 milliseconds of time delay). For the comparison of the poorly tuned aircraft, there was a noticeable difference in particular with the peak of the lightly damped mode.

Here is a comparison of the frequency responses for the poorly tuned aircraft. On the left is 4.6-beta3 and the right is 4.5.7. You can see the higher peak (indicating a lightly damped oscillation) with the 4.6-beta3 firmware with the difference being about 1.5 dB between the two firmware. There is also more phase loss in the 4.6-beta3 firmware. I don't know if this is enough to say 4.6 would crash an aircraft. I mean this just takes an aircraft that already badly oscillates about the roll axis in 4.5 and just makes it a little worse in 4.6. Certainly not a night and day difference.

I then flew the poorly tuned aircraft in loiter. The 4.6 firmware definitely exhibited the characteristics that Tridge saw with the quadplane. The oscillations grew pretty quickly and switching into stabilize resulted in the oscillations slowly dying out. You can even see the 75 deg/s clipping once in stabilize. I had my sticks centered so just the error correction was working and limiting the target rate to 75 deg/s

I flew the poorly tuned aircraft in loiter with 4.5.7. It was better behaved and the oscillations didn't grow. I had to increase the throttle quite a bit to get it to go unstable in loiter. Here is the plot for that

Overall, I think that the attitude controller changed slightly where poorly tuned aircraft my encounter oscillatory behavior especially if the position controller was not tuned for the poorly tuned attitude controller. which means that if they are using the default values of the position controller where the attitude controller can barely keep up with requests in 4.5 and then switch to 4.6, it is possible that they could see oscillations in position controller modes.

Here is a link to the log files I used to develop the plots above. I included a few extra and hopefully the file names are descriptive enough
https://drive.google.com/drive/folders/10LSl4x9tDfLGBrBKewM_XHD_I6d4i5aQ?usp=sharing