srs/trunk/3rdparty/srs-docs/doc/performance.md

---
title: Performance
sidebar_label: Performance 
hide_title: false
hide_table_of_contents: false
---

# Performance

There is a set of tools for performance improvement and detecting memory leaking.

> Note: All tools will hurts performance more or less, so never enable these tools unless you need to fix memory issue.

## RTC

RTC is delivering over UDP, so the first and most important configuration is for kernel network:

```bash
# Query the kernel configuration
sysctl net.core.rmem_max
sysctl net.core.rmem_default
sysctl net.core.wmem_max
sysctl net.core.wmem_default

# Set the UDP buffer to 16MB
sysctl net.core.rmem_max=16777216
sysctl net.core.rmem_default=16777216
sysctl net.core.wmem_max=16777216
sysctl net.core.wmem_default=16777216
```

> Note: For Docker, it read the configuration from host, so you only need to setup the host machine.

> Note：If need to set these configurations in docker, you must run with `--network=host`.

Or, you could also modify the file `/etc/sysctl.conf` to enalbe if when reboot:

```bash
# vi /etc/sysctl.conf
# For RTC
net.core.rmem_max=16777216
net.core.rmem_default=16777216
net.core.wmem_max=16777216
net.core.wmem_default=16777216
```

Query the network statistics and UDP packets dropping:

```bash
netstat -suna
netstat -suna && sleep 30 && netstat -suna
```

For Example:

* `224911319 packets received` The total received UDP packets.
* `65731106 receive buffer errors` The total dropped UDP packets before receiving
* `123534411 packets sent` The total sent UDP packets.
* `0 send buffer errors` The total dropped UDP packets before sending.

> Note: SRS also prints about the packets dropped in application level, for example `loss=(r:49,s:0)` which means dropped 49 packets before receiving.

> Note：Please note that you must run the command in docker container, not on host machine.

The length of UDP queue:

```bash
netstat -lpun
```

For example:

* `Recv-Q 427008` Established: The count of bytes not copied by the user program connected to this socket.
* `Send-Q 0` Established: The count of bytes not acknowledged by the remote host.

Other useful parameters of netstat:

* `--udp|-u` Filter by UDP protocol.
* `--numeric|-n` Show numerical addresses instead of trying to determine symbolic host, port or user names.
* `--statistics|-s` Show statistics.
* `--all|-a` Show  both  listening and non-listening sockets.  With the --interfaces option, show interfaces that are not up.
* `--listening|-l` Show only listening sockets.  (These are omitted by default.)
* `--program|-p` Show the PID and name of the program to which each socket belongs.

## PERF

PERF is Performance analysis tools for Linux.

Show performance bottleneck of SRS:

```
perf top -p $(pidof srs)
```

To record the data:

```
perf record -p $(pidof srs)

# Press CTRL+C after about 30s.

perf report
```

Show stack or backtrace:

```
perf record -a --call-graph fp -p $(pidof srs)
perf report --call-graph --stdio
```

> Note: Record to file by `perf report --call-graph --stdio >t.txt`。

> Remark: The stack(`-g`) does not work for SRS(ST), because ST modifies the SP.

## ASAN

[Asan](https://github.com/google/sanitizers/wiki/AddressSanitizer) is Google Address Sanitizer.

### ASAN: Usage

SRS5+ supports [ASAN](https://github.com/google/sanitizers/wiki/AddressSanitizer) by default.

If you want to disable it, please check bellow configure options:

```bash
./configure -h |grep asan
  --sanitizer=on|off        Whether build SRS with address sanitizer(asan). Default: on
  --sanitizer-static=on|off Whether build SRS with static libasan(asan). Default: off
  --sanitizer-log=on|off    Whether hijack the log for libasan(asan). Default: off
```

Enable leaks detection, see [halt_on_error](https://github.com/google/sanitizers/wiki/AddressSanitizerFlags) 
and [detect_leaks](https://github.com/google/sanitizers/wiki/SanitizerCommonFlags):

```bash
ASAN_OPTIONS=halt_on_error=1:detect_leaks=1 ./objs/srs -c conf/console.conf
```

> Note: SRS disable memory leak detection by default, because it will cause daemon to exit with error.

Highly recommend to enable ASAN because it works great.

### ASAN: Preload

If you encounter the following error:

```bash
==4181651==ASan runtime does not come first in initial library list; you should either link runtime 
to your application or manually preload it with LD_PRELOAD.
```

You should preload the ASAN library:

```bash
LD_PRELOAD=$(find /usr -name libasan.so.5 2>/dev/null) ./objs/srs -c conf/console.conf
```

> Note: Generally, the libasan.so file should be located at `/usr/lib64/libasan.so.5`

### ASAN: Options

By default, SRS use the following ASAN options:

```text
extern "C" const char *__asan_default_options() {
    return "halt_on_error=0:detect_leaks=0:alloc_dealloc_mismatch=0";
}
```

* `halt_on_error=0`: Disable halt on errors by halt_on_error, only print messages, note that it still quit for fatal errors, see [halt_on_error](https://github.com/google/sanitizers/wiki/AddressSanitizerFlags).
* `detect_leaks=0`: Disable the memory leaking detect for daemon by detect_leaks, see [detect_leaks](https://github.com/google/sanitizers/wiki/SanitizerCommonFlags).
* `alloc_dealloc_mismatch=0`: Also disable alloc_dealloc_mismatch for gdb.

You can override the options by `ASAN_OPTIONS`:

```bash
ASAN_OPTIONS=halt_on_error=1:detect_leaks=1:alloc_dealloc_mismatch=1 ./objs/srs -c conf/console.conf
```

Note that the `ASAN_OPTIONS` will be loaded before the `main()` function, so you can set it in the shell, 
but can not set in the `main()` function.

## GPROF

GPROF is a GNU tool, see [SRS GPROF](./gprof.md) and [GNU GPROF](http://www.cs.utah.edu/dept/old/texinfo/as/gprof.html).

Usage:
```
# Build SRS with GPROF
./configure --gprof=on && make

# Start SRS with GPROF
./objs/srs -c conf/console.conf

# Or CTRL+C to stop GPROF
killall -2 srs

# To analysis result.
gprof -b ./objs/srs gmon.out
```

## GPERF

GPERF is  [google tcmalloc](https://github.com/gperftools/gperftools), please see [GPERF](./gperf.md)。

### GPERF: GCP

GCP is for CPU performance analysis, see [GCP](https://gperftools.github.io/gperftools/cpuprofile.html).

Usage:

```
# Build SRS with GCP
./configure --gperf=on --gcp=on && make

# Start SRS with GCP
./objs/srs -c conf/console.conf

# Or CTRL+C to stop GCP
killall -2 srs

# To analysis cpu profile
./objs/pprof --text objs/srs gperf.srs.gcp*
```

> Note: For more details, please read [cpu-profiler](https://github.com/ossrs/srs/tree/4.0release/trunk/research/gperftools/cpu-profiler).

Install tool for graph:

```bash
yum install -y graphviz
```

Output svg graph to open by Chrome:

```bash
./objs/pprof --svg ./objs/srs gperf.srs.gcp >t.svg
```

### GPERF: GMD

GMD is for memory corrupt detecting, see [GMD](http://blog.csdn.net/win_lin/article/details/50461709).

Usage:
```
# Build SRS with GMD.
./configure --gperf=on --gmd=on && make

# Start SRS with GMD.
env TCMALLOC_PAGE_FENCE=1 ./objs/srs -c conf/console.conf
```

> Note: For more details, please read [heap-defense](https://github.com/ossrs/srs/tree/4.0release/trunk/research/gperftools/heap-defense).

> Note: Need link with `libtcmalloc_debug.a` and enable env `TCMALLOC_PAGE_FENCE`.

### GPERF: GMC

GMC is for memory leaking, see [GMC](https://gperftools.github.io/gperftools/heap_checker.html).

Usage:

```
# Build SRS with GMC
./configure --gperf=on --gmc=on && make

# Start SRS with GMC
env PPROF_PATH=./objs/pprof HEAPCHECK=normal ./objs/srs -c conf/console.conf 2>gmc.log 

# Or CTRL+C to stop gmc
killall -2 srs

# To analysis memory leak
cat gmc.log
```

> Note: For more details, please read [heap-checker](https://github.com/ossrs/srs/tree/4.0release/trunk/research/gperftools/heap-checker).

### GPERF: GMP

GMD is for memory performance, see [GMP](https://gperftools.github.io/gperftools/heapprofile.html).

Usage:
```
# Build SRS with GMP
./configure --gperf=on --gmp=on && make

# Start SRS with GMP
./objs/srs -c conf/console.conf

# Or CTRL+C to stop gmp
killall -2 srs 

# To analysis memory profile
./objs/pprof --text objs/srs gperf.srs.gmp*
```

> Note: For more details, please read [heap-profiler](https://github.com/ossrs/srs/tree/4.0release/trunk/research/gperftools/heap-profiler).

## VALGRIND

Valgrind is a powerful tool for memory leak and other issue.

### Valgrind: Memcheck

SRS3+ also supports valgrind.

```
valgrind --leak-check=full --show-leak-kinds=all ./objs/srs -c conf/console.conf
```

> Remark: For ST to support valgrind, see [state-threads](https://github.com/ossrs/state-threads#usage) and [ST#2](https://github.com/ossrs/state-threads/issues/2).

> Remark: For HTTP valgrind API, you should upgrade your SRS to required version, see [#4150](https://github.com/ossrs/srs/pull/4150).

### Valgrind: Incremental Memory Leak Detection

To use Valgrind to detect memory leaks in SRS, even though Valgrind hooks are supported in ST, there are 
still many false positives. A more reasonable approach is to have Valgrind report incremental memory leaks. 
This way, global and static variables can be avoided, and detection can be achieved without exiting the 
program. Follow these steps:

1. Compile SRS with Valgrind support: `./configure --valgrind=on && make`
1. Start SRS with memory leak detection enabled: `valgrind --leak-check=full --show-leak-kinds=all ./objs/srs -c conf/console.conf`
1. Trigger memory detection by using curl to access the API and generate calibration data. There will still be many false positives, but these can be ignored: `curl http://127.0.0.1:1985/api/v1/valgrind?check=added`
1. Retry memory detection, util the valgrind leak summary is stable, no any new lost blocks.
1. Perform load testing or test the suspected leaking functionality, such as RTMP streaming: `ffmpeg -re -i doc/source.flv -c copy -f flv rtmp://127.0.0.1/live/livestream`
1. Stop streaming and wait for SRS to clean up the Source memory, approximately 30 seconds.
1. Perform incremental memory leak detection. The reported leaks will be very accurate at this point: `curl http://127.0.0.1:1985/api/v1/valgrind?check=added`

```text
HTTP #0 11.176.19.95:42162 GET http://9.134.74.169:1985/api/v1/valgrind?check=added, content-length=-1
query check=added
==1481822== LEAK SUMMARY:
==1481822==    definitely lost: 0 (+0) bytes in 0 (+0) blocks
==1481822==    indirectly lost: 0 (+0) bytes in 0 (+0) blocks
==1481822==      possibly lost: 3,406,847 (+0) bytes in 138 (+0) blocks
==1481822==    still reachable: 18,591,709 (+0) bytes in 819 (+0) blocks
==1481822==                       of which reachable via heuristic:
==1481822==                         multipleinheritance: 536 (+0) bytes in 4 (+0) blocks
==1481822==         suppressed: 0 (+0) bytes in 0 (+0) blocks
==1481822== Reachable blocks (those to which a pointer was found) are not shown.
```

> Note: To avoid interference from the HTTP request itself on Valgrind, SRS uses a separate coroutine to perform periodic checks. Therefore, after accessing the API, you may need to wait a few seconds for the detection to be triggered.

### Valgrind: Still Reachable

Sometimes, you will receive the `still reachable` report for static or global variables, like this example:

```text
==3430715== 1,040 (+1,040) bytes in 1 (+1) blocks are still reachable in new loss record 797 of 836
==3430715==    at 0x4C3F963: calloc (vg_replace_malloc.c:1595)
==3430715==    by 0x7D8DB0: SrsConfig::get_hls_vcodec(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) (srs_app_config.cpp:7156)
==3430715==    by 0x781283: SrsHlsMuxer::segment_open() (srs_app_hls.cpp:418)

# It's caused by static variable:
string SrsConfig::get_hls_vcodec(string vhost) {
    SRS_STATIC string DEFAULT = "h264";
    SrsConfDirective* conf = get_hls(vhost);
    if (!conf) {
        return DEFAULT;
```

You can easily work around this by publishing the stream, stopping it, and then triggering the memory leak detection:

1. Publish stream to initialize the static and global variables: `ffmpeg -re -i doc/source.flv -c copy -f flv rtmp://127.0.0.1/live/livestream`
1. Trigger memory detection by using curl to access the API and generate calibration data. There will still be many false positives, but these can be ignored: `curl http://127.0.0.1:1985/api/v1/valgrind?check=added`
1. Retry memory detection, util the valgrind leak summary is stable, no any new lost blocks.
1. Perform load testing or test the suspected leaking functionality, such as RTMP streaming: `ffmpeg -re -i doc/source.flv -c copy -f flv rtmp://127.0.0.1/live/livestream`
1. Stop streaming and wait for SRS to clean up the Source memory, approximately 30 seconds.
1. Perform incremental memory leak detection. The reported leaks will be very accurate at this point: `curl http://127.0.0.1:1985/api/v1/valgrind?check=added`

With the variables initialized, the `still reachable` report will be gone.

## Syscall

Please use [strace -c -p PID](https://man7.org/linux/man-pages/man1/strace.1.html) for syscal performance issue.

## OSX

For macOS, please use [Instruments](https://stackoverflow.com/questions/11445619/profiling-c-on-mac-os-x)

```
instruments -l 30000 -t Time\ Profiler -p 72030
```

> Remark: You can also click `Sample` button in `Active Monitor`.

## Multiple Process and Softirq

You can run softirq(Kernel Network Transmission) on CPU0, so run SRS on other CPUs:

```bash
taskset -p 0xfe $(pidof srs)
```

Or run SRS on CPU1:

```bash
taskset -pc 1 $(pidof srs)
```

Then you can run `top` and press `1` to see each CPU statistics:

```bash
top # Press 1
#%Cpu0  :  1.8 us,  1.1 sy,  0.0 ni, 90.8 id,  0.0 wa,  0.0 hi,  6.2 si,  0.0 st
#%Cpu1  : 67.6 us, 17.6 sy,  0.0 ni, 14.9 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
```

Or use `mpstat -P ALL`

```bash
mpstat -P ALL
#01:23:14 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
#01:23:14 PM  all   33.33    0.00    8.61    0.04    0.00    3.00    0.00    0.00    0.00   55.02
#01:23:14 PM    0    2.46    0.00    1.32    0.06    0.00    6.27    0.00    0.00    0.00   89.88
#01:23:14 PM    1   61.65    0.00   15.29    0.02    0.00    0.00    0.00    0.00    0.00   23.03
```

> Note: Use `cat /proc/softirqs` to check softirq type, please see [Introduction to deferred interrupts (Softirq, Tasklets and Workqueues)](https://0xax.gitbooks.io/linux-insides/content/Interrupts/linux-interrupts-9.html)

> Note: If SRS run with softirq at CPU0, the total CPU will be larger than total of running on different CPUs.

If you got more CPUs, you can run softirq to multiple CPUs:

```bash
# grep virtio /proc/interrupts | grep -e in -e out
 29:   64580032          0          0          0   PCI-MSI-edge      virtio0-input.0
 30:          1         49          0          0   PCI-MSI-edge      virtio0-output.0
 31:   48663403          0   11845792          0   PCI-MSI-edge      virtio0-input.1
 32:          1          0          0         52   PCI-MSI-edge      virtio0-output.1

# cat /proc/irq/29/smp_affinity
1 # Bind softirq of virtio0 incoming to CPU0.
# cat /proc/irq/30/smp_affinity
2 # Bind softirq of virtio0 outgoing to CPU1.
# cat /proc/irq/31/smp_affinity
4 # Bind softirq of virtio1 incoming to CPU2.
# cat /proc/irq/32/smp_affinity
8 # Bind softirq of virtio1 outgoing to CPU3.
```

To disable softirq balance and force to run on CPU0, see [Linux: scaling softirq among many CPU cores](http://natsys-lab.blogspot.com/2012/09/linux-scaling-softirq-among-many-cpu.html) 
and [SMP IRQ affinity](https://www.kernel.org/doc/Documentation/IRQ-affinity.txt) by:

```bash
for irq in $(grep virtio /proc/interrupts | grep -e in -e out | cut -d: -f1); do 
    echo 1 > /proc/irq/$irq/smp_affinity
done
```

> Note：Run `echo 3 > /proc/irq/$irq/smp_affinity` if bind to CPU0 and CPU1.

Then run SRS on other CPUs except CPU0:

```bash
taskset -a -p 0xfe $(cat objs/srs.pid)
```

You can improve about 20% performance by bind softirq to CPU0.

You can also setup in the startup script.

## Process Priority

You can set SRS to run in higher priority:

```bash
renice -n -15 -p $(pidof srs)
```

> Note: The value of nice is `-20` to `19` and default is `0`.

To check the priority, which is the `NI` field of top:

```bash
top -n1 -p $(pidof srs)
#  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                
# 1505 root       5 -15  519920 421556   4376 S  66.7  5.3   4:41.12 srs
```

## Performance Banchmark

The performance benchmark for SRS, compare with nginx-rtmp single process.

Provides detail benchmark steps.

The latest data, read [performance](https://github.com/ossrs/srs/tree/develop#performance).

### Hardware

The client and server use lo net interface to test:

* Hardware: VirtualBox on ThinkPad T430
* OS: CentOS 6.0 x86_64 Linux 2.6.32-71.el6.x86_64
* CPU: 3 Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
* Memory: 2007MB

### OS

Login as root, set the fd limits:

* Set limit: `ulimit -HSn 10240`
* View the limit:

```bash
[root@dev6 ~]# ulimit -n
10240
```

* Restart SRS：`sudo /etc/init.d/srs restart`

### NGINX-RTMP

NGINX-RTMP version and build command.

* NGINX: nginx-1.5.7.tar.gz
* NGINX-RTMP: nginx-rtmp-module-1.0.4.tar.gz
* Read [nginx-rtmp](http://download.csdn.net/download/winlinvip/6795467)
* Build:

```bash
./configure --prefix=`pwd`/../_release \
--add-module=`pwd`/../nginx-rtmp-module-1.0.4 \
--with-http_ssl_module && make && make install
```

* Config nginx：`_release/conf/nginx.conf`

```bash
user  root;
worker_processes  1;
events {
    worker_connections  10240;
}
rtmp{
    server{
        listen 19350;
        application live{
            live on;
        }
    }
}
```

* The limit of fd:

```bash
[root@dev6 nginx-rtmp]# ulimit -n
10240
```

* Start: ``./_release/sbin/nginx``
* Check nginx started:

```bash
[root@dev6 nginx-rtmp]# netstat -anp|grep 19350
tcp        0      0 0.0.0.0:19350               0.0.0.0:*                   LISTEN      6486/nginx
```

### SRS

SRS version and build.

* SRS: [SRS 0.9](https://github.com/ossrs/srs/releases/tag/0.9)
* Build: ``./configure && make``
* Config SRS：`conf/srs.conf`

```bash
listen              1935;
max_connections     10240;
vhost __defaultVhost__ {
    gop_cache       on;
    forward         127.0.0.1:19350;
}
```

* Check limit fds:

```bash
[root@dev6 trunk]# ulimit -n
10240
```

* Start SRS: ``nohup ./objs/srs -c conf/srs.conf >/dev/null 2>&1 &``
* Check SRS started:

```bash
[root@dev6 trunk]# netstat -anp|grep "1935 "
tcp        0      0 0.0.0.0:1935                0.0.0.0:*                   LISTEN      6583/srs
```

### Publish and Play

Use centos to publish RTMP:

* Start FFMPEG:

```bash
for((;;)); do \
    ./objs/ffmpeg/bin/ffmpeg \
        -re -i doc/source.flv \
        -acodec copy -vcodec copy \
        -f flv rtmp://127.0.0.1:1935/live/livestream; \
    sleep 1; 
done
```

* SRS RTMP stream URL: `rtmp://192.168.2.101:1935/live/livestream`
* Nginx-RTMP stream URL: `rtmp://192.168.2.101:19350/live/livestream`

### Client

The RTMP load test tool, read [srs-bench](https://github.com/ossrs/srs-bench)

The sb_rtmp_load used to test RTMP load, support 800-3k concurrency for each process.

* Build: `./configure && make`
* Start: `./objs/sb_rtmp_load -c 800 -r <rtmp_url>`

### Record Data

Record data before test:

* Use top command：

```bash
srs_pid=$(pidof srs); \
nginx_pid=`ps aux|grep nginx|grep worker|awk '{print $2}'`; \
load_pids=`ps aux|grep objs|grep sb_rtmp_load|awk '{ORS=",";print $2}'`; \
top -p $load_pids$srs_pid,$nginx_pid
```

* The connections:

```bash
srs_connections=`netstat -anp|grep srs|grep ESTABLISHED|wc -l`; \
nginx_connections=`netstat -anp|grep nginx|grep ESTABLISHED|wc -l`; \
echo "srs_connections: $srs_connections"; \
echo "nginx_connections: $nginx_connections";
```

* The bandwidth in NBps:

```bash
[root@dev6 nginx-rtmp]# dstat -N lo 30
----total-cpu-usage---- -dsk/total- -net/lo- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0  96   0   0   3|   0     0 |1860B   58k|   0     0 |2996   465 
  0   1  96   0   0   3|   0     0 |1800B   56k|   0     0 |2989   463 
  0   0  97   0   0   2|   0     0 |1500B   46k|   0     0 |2979   461 
```

* The table

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 1.0% | 3MB | 3 | - | - | - | 0.8s |
| nginx-rtmp | 0.7% | 8MB | 2 | - | - | - | 0.8s |

Memory(Mem): The memory usage in MB.

Clients(Conn): The connections/clients to server.

ExpectNbps(ENbps): The expect network bandwidth in Xbps.

ActualNbps(ANBps): The actual network bandwidth in Xbps.

srs-bench(srs-bench/sb): The mock benchmark client tool.

Latency(Lat): The latency of client.

### Benchmark SRS

Let's start performance benchmark.

* Start 500 clients

```bash
./objs/sb_rtmp_load -c 500 -r rtmp://127.0.0.1:1935/live/livestream >/dev/null &
```

* The data:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 9.0% | 8MB | 503 | 100Mbps | 112Mbps | 12.6% | 0.8s |

* The data for 1000 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 23.6% | 13MB | 1003 | 200Mbps | 239Mbps | 16.6% | 0.8s |

* The data for 1500 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 38.6% | 20MB | 1503 | 300Mbps | 360Mbps | 17% | 0.8s |

* The data for 2000 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 65.2% | 34MB | 2003 | 400Mbps | 480Mbps | 22% | 0.8s |

* The data for 2500 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| SRS | 72.9% | 38MB | 2503 | 500Mbps | 613Mbps | 24% | 0.8s |

### Benchmark NginxRTMP

Let's start performance benchmark.

* Start 500 clients:

```bash
./objs/sb_rtmp_load -c 500 -r rtmp://127.0.0.1:19350/live/livestream >/dev/null &
```
* The data for 500 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 8.3% | 13MB | 502 | 100Mbps | 120Mbps | 16.3% | 0.8s |

* The data for 1000 clients:

| Server | CPU | Memory | Clients | ExpectNbps | ActualNbps | srs-bench | Latency|
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 27.3% | 19MB | 1002 | 200Mbps | 240Mbps | 30% | 0.8s |

* The data for 1500 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 42.3% | 25MB | 1502 | 300Mbps | 400Mbps | 31% | 0.8s |

* The data for 2000 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 48.9% | 31MB | 2002 | 400Mbps | 520Mbps | 33% | 0.8s |

* The data for 2500 clients:

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 74.2% | 37MB | 2502 | 500Mbps | 580Mbps | 35% | 0.8s |

### Performance Compare

| Server | CPU | Mem | Conn | ENbps | ANbps | sb | Lat |
| ------ | --- | ------- | ------ | ---------- | ---------- | ------ | -------- |
| nginx-rtmp | 8.3% | 13MB | 502 | 100Mbps | 120Mbps | 16.3% | 0.8s |
| SRS | 9.0% | 8MB | 503 | 100Mbps | 112Mbps | 12.6% | 0.8s |
| nginx-rtmp | 27.3% | 19MB | 1002 | 200Mbps | 240Mbps | 30% | 0.8s |
| SRS | 23.6% | 13MB | 1003 | 200Mbps | 239Mbps | 16.6% | 0.8s |
| nginx-rtmp | 42.3% | 25MB | 1502 | 300Mbps | 400Mbps | 31% | 0.8s |
| SRS | 38.6% | 20MB | 1503 | 300Mbps | 360Mbps | 17% | 0.8s |
| nginx-rtmp | 48.9% | 31MB | 2002 | 400Mbps | 520Mbps | 33% | 0.8s |
| SRS | 65.2% | 34MB | 2003 | 400Mbps | 480Mbps | 22% | 0.8s |
| nginx-rtmp | 74.2% | 37MB | 2502 | 500Mbps | 580Mbps | 35% | 0.8s |
| SRS | 72.9% | 38MB | 2503 | 500Mbps | 613Mbps | 24% | 0.8s |

### Performance Banchmark 4k

The performance is refined to support about 4k clients.

```
[winlin@dev6 srs]$ ./objs/srs -v
0.9.130
```

```
top - 19:52:35 up 1 day, 11:11,  8 users,  load average: 1.20, 1.05, 0.92
Tasks: 171 total,   4 running, 167 sleeping,   0 stopped,   0 zombie
Cpu0  : 26.0%us, 23.0%sy,  0.0%ni, 34.0%id,  0.3%wa,  0.0%hi, 16.7%si,  0.0%st
Cpu1  : 26.4%us, 20.4%sy,  0.0%ni, 34.1%id,  0.7%wa,  0.0%hi, 18.4%si,  0.0%st
Cpu2  : 22.5%us, 15.4%sy,  0.0%ni, 45.3%id,  1.0%wa,  0.0%hi, 15.8%si,  0.0%st
Mem:   2055440k total,  1972196k used,    83244k free,   136836k buffers
Swap:  2064376k total,     3184k used,  2061192k free,   926124k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                          
17034 root      20   0  415m 151m 2040 R 94.4  7.6  14:29.33 ./objs/srs -c console.conf                                                        
 1063 winlin    20   0  131m  68m 1336 S 17.9  3.4  54:05.77 ./objs/sb_rtmp_load -c 800 -r rtmp://127.0.0.1:1935/live/livestream               
 1011 winlin    20   0  132m  68m 1336 R 17.6  3.4  54:45.53 ./objs/sb_rtmp_load -c 800 -r rtmp://127.0.0.1:1935/live/livestream               
18736 winlin    20   0  113m  48m 1336 S 17.6  2.4   1:37.96 ./objs/sb_rtmp_load -c 800 -r rtmp://127.0.0.1:1935/live/livestream               
 1051 winlin    20   0  131m  68m 1336 S 16.9  3.4  53:25.04 ./objs/sb_rtmp_load -c 800 -r rtmp://127.0.0.1:1935/live/livestream               
18739 winlin    20   0  104m  39m 1336 R 15.6  2.0   1:25.71 ./objs/sb_rtmp_load -c 800 -r rtmp://127.0.0.1:1935/live/livestream   
```

```
[winlin@dev6 ~]$ dstat -N lo 30
----total-cpu-usage---- -dsk/total- ---net/lo-- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  3   2  92   0   0   3|  11k   27k|   0     0 |   1B   26B|3085   443 
 32  17  33   0   0  17| 273B   60k|  69M   69M|   0     0 |4878  6652 
 34  18  32   0   0  16|   0    38k|  89M   89M|   0     0 |4591  6102 
 35  19  30   0   0  17| 137B   41k|  91M   91M|   0     0 |4682  6064 
 33  17  33   0   0  17|   0    31k|  55M   55M|   0     0 |4920  7785 
 33  18  31   0   0  17|2867B   34k|  90M   90M|   0     0 |4742  6530 
 32  18  33   0   0  17|   0    31k|  66M   66M|   0     0 |4922  7666 
 33  17  32   0   0  17| 137B   39k|  65M   65M|   0     0 |4841  7299 
 35  18  30   0   0  17|   0    28k| 100M  100M|   0     0 |4754  6752 
 32  17  33   0   0  18|   0    41k|  44M   44M|   0     0 |5130  8251 
 34  18  32   0   0  16|   0    30k| 104M  104M|   0     0 |4456  5718 
```

![SRS 4k](/img/doc-advanced-guides-performance-001.png)

### Performance Banchmark 6k

SRS2.0.15, not SRS1.0, performance is refined to support 6k clients.
That is 4Gbps for 522kbps bitrate, for a single SRS process. Read https://github.com/ossrs/srs/issues/194

### Performance Banchmark 7.5k

SRS2.0.30 refined to support 7.5k clients, read https://github.com/ossrs/srs/issues/217

Winlin 2014.11

![](https://ossrs.io/gif/v1/sls.gif?site=ossrs.io&path=/lts/doc/en/v7/performance)