Skip to content

Major updates have arrived in 4.28.0 (mostly for UC Mode) #2865

Open
@mdmintz

Description

@mdmintz
Member

For anyone that hasn't been following #2842, CF pushed an update that prevented UC Mode from easily bypassing CAPTCHA Turnstiles on Linux servers. Additionally, uc_click() was rendered ineffective for clicking Turnstile CAPTCHA checkboxes when clicking the checkbox was required. I've been working on solutions to these situations.

As I mentioned earlier in #2842 (comment), if CF detects either Selenium in the browser or JavaScript involvement in clicking the CAPTCHA, then they don't let the click through. (The JS-detection part is new.) I read online that CF employees borrowed ideas from https://github.com/kaliiiiiiiiii/brotector (a Selenium detector) in order to improve their CAPTCHA. Naturally, I was skeptical at first, but I have confirmed that the two algorithms do appear to get similar results. (Brotector was released 6 weeks ago, while the Cloudflare update happened 2 weeks ago.)

The solution to bypassing the improved CAPTCHAs requires using pyautogui to stay undetected. There was also the matter of how to make pyautogui work well on headless Linux servers. (Thanks to some ideas by @EnmeiRyuuDev in #2842 (comment), that problem was overcome by setting pyautogui._pyautogui_x11._display to Xlib.display.Display(os.environ['DISPLAY']) on Linux in order to sync up pyautogui with the X11 virtual display.)

The improved SeleniumBase UC Mode will have these new methods:

driver.uc_gui_press_key(key)  # Use PyAutoGUI to press the keyboard key

driver.uc_gui_press_keys(keys)  # Use PyAutoGUI to press a list of keys

driver.uc_gui_write(text)  # Similar to uc_gui_press_keys(), but faster

driver.uc_gui_handle_cf(frame="iframe")  # PyAutoGUI click CF Turnstile

It'll probably be easier to understand how those work via examples. Here's one for uc_gui_handle_cf based on the example in #2842 (comment):

import sys
from seleniumbase import SB

agent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) Chrome/126.0.0.0"
if "linux" in sys.platform:
    agent = None  # Use the default UserAgent

with SB(uc=True, test=True, rtf=True, agent=agent) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    sb.uc_gui_handle_cf()  # Ready if needed!
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")

Above, I deliberately gave it an incomplete UserAgent so that CAPTCHA-clicking is required to advance. On macOS and Windows, the default UserAgent that SeleniumBase gives you is already enough to bypass the CAPTCHA screen entirely. The uc_gui_handle_cf() method is designed such that if there's no CAPTCHA that needs to be clicked on the page you're on, then nothing happens. Therefore, you can add the line whenever you think you'll encounter a CAPTCHA or not. In case there's more than one iframe on a website, you can specify the CSS Selector of the iframe as an arg when calling uc_gui_handle_cf(). There will be new examples in the SeleniumBase/examples/ folder for all the new UC Mode methods. To sum up, you may need to use the newer uc_gui_* methods in order to get past some CAPTCHAs on Linux where uc_click() worked previously.

On the topic of Brotector, (which is the open source bot-detector library that CF borrowed ideas from), there is a huge opportunity: Now that effective bot-detection software is available to the general public (all the code is open source!), anyone can now build their own CAPTCHA services (or just add CAPTCHAs to sites without the "service" part). I've already jumped on this with the Brotector CAPTCHA: https://seleniumbase.io/apps/brotector. I've also created a few test sites that utilize it:

I did make some improvements to the original Brotector algorithm in order to be suitable for CAPTCHAs: I needed a definite Allow/Block answer, rather than a number between 0 and 1 determining the likelihood of a bot, etc. I've been using these new test sites for testing the improved UC Mode.

That covers the major updates from 4.28.0 (with the exception of Brotector CAPTCHA test sites, which were already available to the public at the URLs listed above).

There will also be some other improvements:

  • More sb methods added directly into the driver.
  • An improvement to the Recorder to better handle autogenerated IDs in selectors.
  • Python dependency updates.
  • Some method simplifications.
  • Some timing updates.
  • Some updates to default settings.

Now, when using UC Mode on Linux, the default setting is NOT using headless mode. If for some reason you decide to use UC Mode and Headless Mode together, note that although Chrome will launch, you'll definitely be detected by anti-bots, and on top of that, pyautogui methods won't work. Use xvfb=True / --xvfb in order to be sure that the improved X11 virtual display on Linux activates. You'll need that for the uc_gui_* methods to work properly.

Much of that will get covered in the 3rd UC Mode video tutorial on YouTube (expected sometime in the near future).

In case anyone has forgotten, SeleniumBase is still a Test Automation Framework at heart, (which includes an extremely popular feature for stealth called "UC Mode"). UC Mode has gathered a lot of the attention, but SeleniumBase is more than just that.

Activity

self-assigned this
on Jun 23, 2024
changed the title [-]Major updates coming in `4.28.0` (mostly for UC Mode)[/-] [+]Major updates have arrived in `4.28.0` (mostly for UC Mode)[/+] on Jun 23, 2024
mdmintz

mdmintz commented on Jun 23, 2024

@mdmintz
MemberAuthor

4.28.0 has been released: https://github.com/seleniumbase/SeleniumBase/releases/tag/v4.28.0

The pyautogui example for a Cloudflare page with UC Mode:

Examples of bypassing the Brotector CAPTCHA with UC Mode:

Examples of how the Brotector CAPTCHA detects regular Selenium:

mdmintz

mdmintz commented on Jun 24, 2024

@mdmintz
MemberAuthor

Here's an example script for Linux to prove it's working:

from seleniumbase import SB

with SB(uc=True, test=True) as sb:
    url = "https://www.virtualmanager.com/en/login"
    sb.uc_open_with_reconnect(url, 4)
    print(sb.get_page_title())
    sb.uc_gui_handle_cf()  # Ready if needed!
    print(sb.get_page_title())
    sb.assert_element('input[name*="email"]')
    sb.assert_element('input[name*="login"]')
    sb.set_messenger_theme(location="bottom_center")
    sb.post_message("SeleniumBase wasn't detected!")
Screenshot 2024-06-23 at 8 18 24 PM

The second print() should show "Virtual Manager", which means that the automation was able to get past the Turnstile.

vmolostvov

vmolostvov commented on Jun 24, 2024

@vmolostvov

@mdmintz same problem here on linux vds (ubuntu without gpu), seleniumbase became unable to bypass the CloudFlare challenge. Using latest sb version. On local macos and windows keep working without any problem.

I can confirm that my issue on headless Linux Ubuntu was solved by 4.28.0

Снимок экрана 2567-06-24 в 11 50 54

Appreciate your work sir @mdmintz

SSujitX

SSujitX commented on Jun 25, 2024

@SSujitX
goldananas

goldananas commented on Jun 25, 2024

@goldananas

Hey @mdmintz , do you think you will be working on making the uc_gui_handle_cf method compatible with the returnable Driver ? Works just fine on headless Linux with SB, but I can't find way to do the same with the returnable Driver instead.

mdmintz

mdmintz commented on Jun 25, 2024

@mdmintz
MemberAuthor

@goldananas If using the Driver() format instead of SB(), you'll need need to spin up the special X11 virtual display yourself before launching the driver. (See #2842 (comment).)

With the SB() format, SB(uc=True, xvfb=True) does all that for you when running on Linux.

NCLnclNCL

NCLnclNCL commented on Jun 30, 2024

@NCLnclNCL

70 remaining items

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @bjornkarlsson@mdmintz@gabrielsim@enricodvn@chlwodud77

      Issue actions

        Major updates have arrived in `4.28.0` (mostly for UC Mode) · Issue #2865 · seleniumbase/SeleniumBase