Skip to content
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

Commit fec9a14

Browse files
authoredNov 8, 2021
Add more context in intro (#713)
* Add more context in intro * Take another pass * Add threadless in log section
1 parent fa18129 commit fec9a14

File tree

1 file changed

+155
-125
lines changed

1 file changed

+155
-125
lines changed
 

‎README.md

Lines changed: 155 additions & 125 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727

2828
- [Features](#features)
2929
- [Install](#install)
30+
- [Stable vs Develop](#stable-vs-develop)
3031
- [Using PIP](#using-pip)
3132
- [Stable version](#stable-version-with-pip)
3233
- [Development version](#development-version-with-pip)
@@ -119,7 +120,7 @@
119120
- Fast & Scalable
120121

121122
- Scales by using all available cores on the system
122-
- Threadless executions using coroutine
123+
- Threadless executions using asyncio
123124
- Made to handle `tens-of-thousands` connections / sec
124125

125126
```console
@@ -146,36 +147,56 @@
146147
- Uses only `~5-20MB` RAM
147148
- No external dependency other than standard Python library
148149
- Programmable
149-
- Optionally enable builtin Web Server
150-
- Customize proxy and http routing via [plugins](https://github.com/abhinavsingh/proxy.py/tree/develop/proxy/plugin)
151-
- Enable plugin using command line option e.g. `--plugins proxy.plugin.CacheResponsesPlugin`
152-
- Plugin API is currently in development phase, expect breaking changes.
150+
- Customize proxy behavior using [Proxy Server Plugins](#http-proxy-plugins). Example:
151+
- `--plugins proxy.plugin.ProxyPoolPlugin`
152+
- Optionally, enable builtin [Web Server Plugins](#http-web-server-plugins). Example:
153+
- `--plugins proxy.plugin.ReverseProxyPlugin`
154+
- Plugin API is currently in development phase, expect breaking changes
153155
- Realtime Dashboard
154-
- Optionally enable bundled dashboard.
156+
- Optionally, enable [proxy.py dashboard](#run-dashboard).
155157
- Available at `http://localhost:8899/dashboard`.
156-
- Inspect, Monitor, Control and Configure `proxy.py` at runtime.
157-
- Extend dashboard using plugins.
158-
- Dashboard is currently in development phase, expect breaking changes.
158+
- [Inspect, Monitor, Control and Configure](#inspect-traffic) `proxy.py` at runtime
159+
- [Chrome DevTools Protocol](#chrome-devtools-protocol) support
160+
- Extend dashboard using plugins
161+
- Dashboard is currently in development phase, expect breaking changes
159162
- Secure
160-
- Enable end-to-end encryption between clients and `proxy.py` using TLS
163+
- Enable end-to-end encryption between clients and `proxy.py`
161164
- See [End-to-End Encryption](#end-to-end-encryption)
165+
- Private
166+
- Everyone deserves privacy. Browse with malware and adult content protection
167+
- See [DNS-over-HTTPS](#cloudflarednsresolverplugin)
162168
- Man-In-The-Middle
163169
- Can decrypt TLS traffic between clients and upstream servers
164170
- See [TLS Interception](#tls-interception)
165171
- Supported proxy protocols
166172
- `http(s)`
167173
- `http1`
168-
- `http1.1` pipeline
174+
- `http1.1` with pipeline
169175
- `http2`
170176
- `websockets`
177+
- Static file server support
178+
- See `--enable-static-server` and `--static-server-dir` flags
171179
- Optimized for large file uploads and downloads
172-
- IPv4 and IPv6 support
180+
- See `--client-recvbuf-size` and `--server-recvbuf-size` flag
181+
- `IPv4` and `IPv6` support
182+
- See `--hostname` flag
183+
- Unix domain socket support
184+
- See `--unix-socket-path` flag
173185
- Basic authentication support
174-
- Can serve a [PAC (Proxy Auto-configuration)](https://en.wikipedia.org/wiki/Proxy_auto-config) file
186+
- See `--basic-auth` flag
187+
- PAC (Proxy Auto-configuration) support
175188
- See `--pac-file` and `--pac-file-url-path` flags
176189

177190
# Install
178191

192+
## Stable vs Develop
193+
194+
`master` branch contains latest stable code and is available via `PyPi` repository
195+
196+
`develop` branch contains cutting edge changes
197+
198+
Development branch is kept stable *(most of the times)*. But if you want 100% reliability and serving users in production environment, always use stable version from `PyPi` or `Docker` container from `hub.docker.com`.
199+
179200
## Using PIP
180201

181202
### Stable Version with PIP
@@ -200,18 +221,17 @@ or from GitHub `master` branch
200221

201222
## Using Docker
202223

203-
#### Stable Version from Docker Hub
224+
### Stable Version from Docker Hub
204225

205226
```console
206227
docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
207228
```
208229

209-
#### Build Development Version Locally
230+
### Build Development Version Locally
210231

211232
```console
212233
git clone https://github.com/abhinavsingh/proxy.py.git
213-
cd proxy.py
214-
make container
234+
cd proxy.py && make container
215235
docker run -it -p 8899:8899 --rm abhinavsingh/proxy.py:latest
216236
```
217237

@@ -239,38 +259,41 @@ or from GitHub `master` branch
239259
When `proxy.py` is installed using `pip`,
240260
an executable named `proxy` is placed under your `$PATH`.
241261

242-
#### Run it
262+
### Run it
243263

244-
Simply type `proxy` on command line to start it with default configuration.
264+
Simply type `proxy` on command line to start with default configuration.
245265

246266
```console
247267
proxy
248268
...[redacted]... - Loaded plugin proxy.http_proxy.HttpProxyPlugin
249-
...[redacted]... - Starting 8 workers
269+
...[redacted]... - Starting 8 threadless workers
250270
...[redacted]... - Started server on ::1:8899
251271
```
252272

253-
#### Understanding logs
273+
### Understanding logs
254274

255275
Things to notice from above logs:
256276

257-
- `Loaded plugin` - `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default.
258-
As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py`
277+
- `Loaded plugin`
278+
- `proxy.py` will load `proxy.http.proxy.HttpProxyPlugin` by default
279+
- As name suggests, this core plugin adds `http(s)` proxy server capabilities to `proxy.py` instance
259280

260-
- `Started N workers` - Use `--num-workers` flag to customize number of worker processes.
261-
By default, `proxy.py` will start as many workers as there are CPU cores on the machine.
281+
- `Started N threadless workers`
282+
- By default, `proxy.py` will start as many workers as there are CPU cores on the machine
283+
- Use `--num-workers` flag to customize number of worker processes
284+
- See [Threads vs Threadless](#threads-vs-threadless) to understand how to control execution mode
262285

263-
- `Started server on ::1:8899` - By default, `proxy.py` listens on IPv6 `::1`,
264-
which is equivalent of IPv4 `127.0.0.1`. If you want to access `proxy.py` externally,
265-
use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available
266-
on your machine. See [CustomNetworkInterface](#customnetworkinterface) for how to customize
267-
your system public IP as seen by the upstream servers.
286+
- `Started server on ::1:8899`
287+
- By default, `proxy.py` listens on IPv6 `::1`, which is equivalent of IPv4 `127.0.0.1`
288+
- If you want to access `proxy.py` from external host, use `--hostname ::` or `--hostname 0.0.0.0` or bind to any other interface available on your machine.
289+
- See [CustomNetworkInterface](#customnetworkinterface) for how to customize `proxy.py` *public IP seen by upstream servers*.
268290

269-
- `Port 8899` - Use `--port` flag to customize default TCP port.
291+
- `Port 8899`
292+
- Use `--port` flag to customize default TCP port.
270293

271-
#### Enable DEBUG logging
294+
### Enable DEBUG logging
272295

273-
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`.
296+
All the logs above are `INFO` level logs, default `--log-level` for `proxy.py`
274297

275298
Lets start `proxy.py` with `DEBUG` level logging:
276299

@@ -282,11 +305,18 @@ Lets start `proxy.py` with `DEBUG` level logging:
282305
...[redacted]... - Started server on ::1:8899
283306
```
284307

285-
As we can see, before starting up:
308+
You can use single letter to customize log level. Example:
309+
- `d = DEBUG`
310+
- `i = INFO`
311+
- `w = WARNING`
312+
- `e = ERROR`
313+
- `c = CRITICAL`
314+
315+
As we can see from the above logs, before starting up:
286316

287-
- `proxy.py` also tried to set open file limit `ulimit` on the system.
288-
- Default value for `--open-file-limit` used is `1024`.
289-
- `--open-file-limit` flag is a no-op on `Windows` operating systems.
317+
- `proxy.py` tried to set open file limit `ulimit` on the system
318+
- Default value for `--open-file-limit` used is `1024`
319+
- `--open-file-limit` flag is a no-op on `Windows` operating systems
290320

291321
See [flags](#flags) for full list of available configuration options.
292322

@@ -323,25 +353,24 @@ To start `proxy.py` from source code follow these instructions:
323353
❯ make
324354
```
325355

326-
- Run proxy.py
356+
- Run `proxy.py`
327357

328358
```console
329359
❯ python -m proxy
330360
```
331361

332-
Also see [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
362+
See [Plugin Developer and Contributor Guide](#plugin-developer-and-contributor-guide)
333363
if you plan to work with `proxy.py` source code.
334364

335365
## Docker image
336366

337-
#### Customize startup flags
367+
### Customize startup flags
338368

339369
By default `docker` binary is started with IPv4 networking flags:
340370

341371
--hostname 0.0.0.0 --port 8899
342372

343-
To override input flags, start docker image as follows.
344-
For example, to check `proxy.py` version within Docker image:
373+
You can override flag from command line when starting the docker container. For example, to check `proxy.py` version within the docker container, run:
345374

346375
❯ docker run -it \
347376
-p 8899:8899 \
@@ -1380,32 +1409,30 @@ As a decorator:
13801409

13811410
### build_http_request
13821411

1383-
#### Generate HTTP GET request
1412+
- Generate HTTP GET request
13841413

1385-
```python
1386-
>>> build_http_request(b'GET', b'/')
1387-
b'GET / HTTP/1.1\r\n\r\n'
1388-
>>>
1389-
```
1414+
```python
1415+
>>> build_http_request(b'GET', b'/')
1416+
b'GET / HTTP/1.1\r\n\r\n'
1417+
```
13901418

1391-
#### Generate HTTP GET request with headers
1419+
- Generate HTTP GET request with headers
13921420

1393-
```python
1394-
>>> build_http_request(b'GET', b'/',
1395-
headers={b'Connection': b'close'})
1396-
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
1397-
>>>
1398-
```
1421+
```python
1422+
>>> build_http_request(b'GET', b'/',
1423+
headers={b'Connection': b'close'})
1424+
b'GET / HTTP/1.1\r\nConnection: close\r\n\r\n'
1425+
```
13991426

1400-
#### Generate HTTP POST request with headers and body
1427+
- Generate HTTP POST request with headers and body
14011428

1402-
```python
1403-
>>> import json
1404-
>>> build_http_request(b'POST', b'/form',
1405-
headers={b'Content-type': b'application/json'},
1406-
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
1407-
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
1408-
```
1429+
```python
1430+
>>> import json
1431+
>>> build_http_request(b'POST', b'/form',
1432+
headers={b'Content-type': b'application/json'},
1433+
body=proxy.bytes_(json.dumps({'email': 'hello@world.com'})))
1434+
b'POST /form HTTP/1.1\r\nContent-type: application/json\r\n\r\n{"email": "hello@world.com"}'
1435+
```
14091436

14101437
### build_http_response
14111438

@@ -1422,66 +1449,66 @@ build_http_response(
14221449

14231450
### API Usage
14241451

1425-
#### gen_private_key
1452+
- gen_private_key
14261453

1427-
```python
1428-
gen_private_key(
1429-
key_path: str,
1430-
password: str,
1431-
bits: int = 2048,
1432-
timeout: int = 10) -> bool
1433-
```
1434-
1435-
#### gen_public_key
1436-
1437-
```python
1438-
gen_public_key(
1439-
public_key_path: str,
1440-
private_key_path: str,
1441-
private_key_password: str,
1442-
subject: str,
1443-
alt_subj_names: Optional[List[str]] = None,
1444-
extended_key_usage: Optional[str] = None,
1445-
validity_in_days: int = 365,
1446-
timeout: int = 10) -> bool
1447-
```
1454+
```python
1455+
gen_private_key(
1456+
key_path: str,
1457+
password: str,
1458+
bits: int = 2048,
1459+
timeout: int = 10) -> bool
1460+
```
14481461

1449-
#### remove_passphrase
1462+
- gen_public_key
1463+
1464+
```python
1465+
gen_public_key(
1466+
public_key_path: str,
1467+
private_key_path: str,
1468+
private_key_password: str,
1469+
subject: str,
1470+
alt_subj_names: Optional[List[str]] = None,
1471+
extended_key_usage: Optional[str] = None,
1472+
validity_in_days: int = 365,
1473+
timeout: int = 10) -> bool
1474+
```
14501475

1451-
```python
1452-
remove_passphrase(
1453-
key_in_path: str,
1454-
password: str,
1455-
key_out_path: str,
1456-
timeout: int = 10) -> bool
1457-
```
1476+
- remove_passphrase
14581477

1459-
#### gen_csr
1478+
```python
1479+
remove_passphrase(
1480+
key_in_path: str,
1481+
password: str,
1482+
key_out_path: str,
1483+
timeout: int = 10) -> bool
1484+
```
14601485

1461-
```python
1462-
gen_csr(
1463-
csr_path: str,
1464-
key_path: str,
1465-
password: str,
1466-
crt_path: str,
1467-
timeout: int = 10) -> bool
1468-
```
1486+
- gen_csr
14691487

1470-
#### sign_csr
1488+
```python
1489+
gen_csr(
1490+
csr_path: str,
1491+
key_path: str,
1492+
password: str,
1493+
crt_path: str,
1494+
timeout: int = 10) -> bool
1495+
```
14711496

1472-
```python
1473-
sign_csr(
1474-
csr_path: str,
1475-
crt_path: str,
1476-
ca_key_path: str,
1477-
ca_key_password: str,
1478-
ca_crt_path: str,
1479-
serial: str,
1480-
alt_subj_names: Optional[List[str]] = None,
1481-
extended_key_usage: Optional[str] = None,
1482-
validity_in_days: int = 365,
1483-
timeout: int = 10) -> bool
1484-
```
1497+
- sign_csr
1498+
1499+
```python
1500+
sign_csr(
1501+
csr_path: str,
1502+
crt_path: str,
1503+
ca_key_path: str,
1504+
ca_key_password: str,
1505+
ca_crt_path: str,
1506+
serial: str,
1507+
alt_subj_names: Optional[List[str]] = None,
1508+
extended_key_usage: Optional[str] = None,
1509+
validity_in_days: int = 365,
1510+
timeout: int = 10) -> bool
1511+
```
14851512

14861513
See [pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/proxy/common/pki.py) and
14871514
[test_pki.py](https://github.com/abhinavsingh/proxy.py/blob/develop/tests/common/test_pki.py)
@@ -1520,8 +1547,7 @@ optional arguments:
15201547

15211548
## Internal Documentation
15221549

1523-
Browse through internal class hierarchy and documentation using `pydoc3`.
1524-
Example:
1550+
Code is well documented. Browse through internal class hierarchy and documentation using `pydoc3`
15251551

15261552
```console
15271553
pydoc3 proxy
@@ -1539,8 +1565,6 @@ FILE
15391565

15401566
# Run Dashboard
15411567

1542-
**This is a WIP and may not work as documented**
1543-
15441568
Dashboard is currently under development and not yet bundled with `pip` packages.
15451569
To run dashboard, you must checkout the source.
15461570

@@ -1579,6 +1603,8 @@ Visit dashboard:
15791603

15801604
## Inspect Traffic
15811605

1606+
***This is a WIP and may not work as documented***
1607+
15821608
Wait for embedded `Chrome Dev Console` to load. Currently, detail about all traffic flowing
15831609
through `proxy.py` is pushed to the `Inspect Traffic` tab. However, received payloads are not
15841610
yet integrated with the embedded dev console.
@@ -1603,19 +1629,23 @@ Now point your CDT instance to `ws://localhost:8899/devtools`.
16031629

16041630
## Threads vs Threadless
16051631

1606-
### Pre v2.x
1632+
### `v1.x`
16071633

16081634
`proxy.py` used to spawn new threads for handling client requests.
16091635

1610-
### Starting v2.0
1636+
### `v2.0+`
16111637

16121638
`proxy.py` added support for threadless execution of client requests using `asyncio`.
16131639

1614-
### Starting v2.4.0
1640+
### `v2.4.0+`
1641+
1642+
Threadless execution was turned ON by default for `Python 3.8+` on `mac` and `linux` environments.
1643+
1644+
`proxy.py` threadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using `--threaded` flag.
16151645

1616-
Threadless execution was turned ON by default for `Python 3.8+` on `mac` and `linux` environments. `proxy.py` threadless execution has been reported safe on these environments by our users. If you are running into trouble, fallback to threaded mode using `--threaded` flag.
1646+
For `windows` and `Python < 3.8`, you can still try out threadless mode by starting `proxy.py` with `--threadless` flag.
16171647

1618-
For `windows` and `Python < 3.8`, you can still try out threadless mode by starting `proxy.py` with `--threadless` flag. If threadless works for you, consider sending a PR by editing `_env_threadless_compliant` method in the `proxy/common/constants.py` file.
1648+
If threadless works for you, consider sending a PR by editing `_env_threadless_compliant` method in the `proxy/common/constants.py` file.
16191649

16201650
## SyntaxError: invalid syntax
16211651

0 commit comments

Comments
 (0)
Please sign in to comment.