Buy commercial curl support. We
help you work out your issues, debug your libcurl applications, use the API,
port to new platforms, add new features and more. With a team lead by the
curl founder Daniel himself.
Re: Increase in CPU usage in 8.7.1 vs 8.6.0 for rate-limited downloads
- Contemporary messages sorted: [ by date ] [ by thread ] [ by subject ] [ by author ] [ by messages with attachments ]
From: Stefan Eissing via curl-library <curl-library_at_lists.haxx.se>
Date: Wed, 15 May 2024 09:56:37 +0200
> Am 15.05.2024 um 08:56 schrieb David Pfitzner via curl-library <curl-library_at_lists.haxx.se>:
>
> Hi,
>
> I've noticed that from curl 8.6.0 to 8.7.1, the CPU usage when doing rate-limited downloads with libcurl has increased significantly. This is most noticeable when making multiple (rate-limited) downloads in parallel, for large files. (I've seen this in a custom application using libcurl, and then reproduced using the 'curl' tool.)
>
> For example, running 'curl' with the following (where the file a32 contains 32 copies of " -o /dev/null http://127.0.0.1:9999/speed10", a test resource with size=46450490):
>
> #curl-8.6.0:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_6_0 --parallel --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; done
> 5.02 real 0.20 user 0.67 sys
> 4.95 real 0.15 user 0.68 sys
> 5.02 real 0.25 user 0.63 sys
> 5.00 real 0.21 user 0.63 sys
> 4.49 real 0.23 user 0.62 sys
>
> # curl-8.7.1:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_7_1 --parallel --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; done
> 4.51 real 0.52 user 0.65 sys
> 5.01 real 0.55 user 0.63 sys
> 5.02 real 0.56 user 0.59 sys
> 4.96 real 0.42 user 0.59 sys
> 5.01 real 0.37 user 0.71 sys
>
> Notice the increase in user-space CPU usage (almost doubled). (Of course the actual effect can vary depending on many factors - the number of parallel downloads, the specified rate limit, file sizes, network speed, etc.)
>
> From testing, this change appears to be due to this commit:
> https://github.com/curl/curl/commit/db5c9f4f9e0779b49624752b135281a0717b277b
>
> That change was to make rate limiting more accurate, and perhaps the increase in CPU usage is a necessary (and expected?) cost of doing so. But, in my use case the rate limiting accuracy in 8.6.0 was "good enough", and I would prefer not to incur the increase in CPU usage seen in 8.7.1. But, I'm not sure how to do that. Perhaps it would be useful for a user of libcurl to be able to (somehow) control this tradeoff between rate-limiting accuracy and CPU usage?
Thanks for the data. I believe there are certainly improvements possible here. We are much in favour of making transfers not only work correctly but also efficiently. It is more a matter of someone knowledgable finding the time to work on this.
Cheers,
Stefan
>
> Regards,
> David Pfitzner
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
Date: Wed, 15 May 2024 09:56:37 +0200
> Am 15.05.2024 um 08:56 schrieb David Pfitzner via curl-library <curl-library_at_lists.haxx.se>:
>
> Hi,
>
> I've noticed that from curl 8.6.0 to 8.7.1, the CPU usage when doing rate-limited downloads with libcurl has increased significantly. This is most noticeable when making multiple (rate-limited) downloads in parallel, for large files. (I've seen this in a custom application using libcurl, and then reproduced using the 'curl' tool.)
>
> For example, running 'curl' with the following (where the file a32 contains 32 copies of " -o /dev/null http://127.0.0.1:9999/speed10", a test resource with size=46450490):
>
> #curl-8.6.0:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_6_0 --parallel --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; done
> 5.02 real 0.20 user 0.67 sys
> 4.95 real 0.15 user 0.68 sys
> 5.02 real 0.25 user 0.63 sys
> 5.00 real 0.21 user 0.63 sys
> 4.49 real 0.23 user 0.62 sys
>
> # curl-8.7.1:
> for x in 1 2 3 4 5 ; do /usr/bin/time ../curl-8_7_1 --parallel --parallel-immediate --no-progress-meter --limit-rate 10485760 `cat ./a32` ; done
> 4.51 real 0.52 user 0.65 sys
> 5.01 real 0.55 user 0.63 sys
> 5.02 real 0.56 user 0.59 sys
> 4.96 real 0.42 user 0.59 sys
> 5.01 real 0.37 user 0.71 sys
>
> Notice the increase in user-space CPU usage (almost doubled). (Of course the actual effect can vary depending on many factors - the number of parallel downloads, the specified rate limit, file sizes, network speed, etc.)
>
> From testing, this change appears to be due to this commit:
> https://github.com/curl/curl/commit/db5c9f4f9e0779b49624752b135281a0717b277b
>
> That change was to make rate limiting more accurate, and perhaps the increase in CPU usage is a necessary (and expected?) cost of doing so. But, in my use case the rate limiting accuracy in 8.6.0 was "good enough", and I would prefer not to incur the increase in CPU usage seen in 8.7.1. But, I'm not sure how to do that. Perhaps it would be useful for a user of libcurl to be able to (somehow) control this tradeoff between rate-limiting accuracy and CPU usage?
Thanks for the data. I believe there are certainly improvements possible here. We are much in favour of making transfers not only work correctly but also efficiently. It is more a matter of someone knowledgable finding the time to work on this.
Cheers,
Stefan
>
> Regards,
> David Pfitzner
> --
> Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library
> Etiquette: https://curl.se/mail/etiquette.html
-- Unsubscribe: https://lists.haxx.se/mailman/listinfo/curl-library Etiquette: https://curl.se/mail/etiquette.htmlReceived on 2024-05-15