Root Zanli
Home
Console
Upload
information
Create File
Create Folder
About
Tools
:
/
usr
/
share
/
doc
/
curl
/
Filename :
CHANGES
back
Copy
_ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| Changelog Version 7.76.1 (14 Apr 2021) Daniel Stenberg (14 Apr 2021) - RELEASE-NOTES: synced curl 7.76.1 release - THANKS: add names from 7.76.1 - misc: update copyright year ranges to match latest updates - [Tatsuhiro Tsujikawa brought this change] ngtcp2: Use ALPN h3-29 for now Fixes #6864 Cloes #6886 Jay Satiro (11 Apr 2021) - TODO: remove 18.22 --fail-with-body --fail-with-body was added in 8a964cb (precedes curl-7_76_0). Daniel Stenberg (10 Apr 2021) - [Jürgen Gmach brought this change] src/tool_vms.c: remove duplicated word in comment Closes #6881 - configure: fix CURL_DARWIN_CFLAGS use The macro name change was not completely done. Follow-up to 5d2c384452543c Bug: https://github.com/curl/curl/commit/5d2c384452543c7b6c9fb02eaa0afc84fd5ab941#commitcomment-49315187 Reported-by: Marcel Raad Closes #6878 - [Anthony Shaw brought this change] github/workflow: add "security-extended" to codeql-analysis.yml Extends the CodeQL code scan. Closes #6815 - [Jochem Broekhoff brought this change] examples/hiperfifo.c: check event_initialized before delete If event_del is called with the event struct (still) zeroed out, a segmentation fault may occur. event_initialized checks whether the event struct is nonzero. Closes #6876 - [Patrick Monnerat brought this change] ntlm: fix negotiated flags usage According to Microsoft document MS-NLMP, current flags usage is not accurate: flag NTLMFLAG_NEGOTIATE_NTLM2_KEY controls the use of extended security in an NTLM authentication message and NTLM version 2 cannot be negotiated within the protocol. The solution implemented here is: if the extended security flag is set, prefer using NTLM version 2 (as a server featuring extended security should also support version 2). If version 2 has been disabled at compile time, use extended security. Tests involving NTLM are adjusted to this new behavior. Fixes #6813 Closes #6849 - [Patrick Monnerat brought this change] ntlm: support version 2 on 32-bit platforms Closes #6849 - [Patrick Monnerat brought this change] curl_ntlm_core.h: simplify conditionals for USE_NTLM2SESSION ... as !defined(CURL_DISABLE_CRYPTO_AUTH) is a prerequisite for the whole NTLM. Closes #6849 - lib: remove unused HAVE_INET_NTOA_R* defines Closes #6867 - [Michael Forney brought this change] configure: include <time.h> unconditionally In 2682e5f5, several instances of AC_HEADER_TIME were removed since it is a deprecated autoconf macro. However, this was the macro that defined TIME_WITH_SYS_TIME, which was used to indicate that <time.h> can be included alongside <sys/time.h>. TIME_WITH_SYS_TIME is still used in the configure test body and since it is no longer defined, <time.h> is *not* included on systems that have <sys/time.h>. In particular, at least on musl libc and glibc, <sys/time.h> does not implicitly include <time.h> and does not declare clock_gettime, gmtime_r, or localtime_r. This causes configure to fail to detect those functions. The AC_HEADER_TIME macro deprecation text says > All current systems provide time.h; it need not be checked for. > Not all systems provide sys/time.h, but those that do, all allow > you to include it and time.h simultaneously. So, to fix this issue, simply include <time.h> unconditionally when testing for time-related functions and in libcurl, and don't bother checking for it. Closes #6859 - [Michael Forney brought this change] configure: remove use of RETSIGTYPE This was previously defined by the obsolete AC_TYPE_SIGNAL macro, which was removed in 2682e5f5. The deprecation text says > Your code may safely assume C89 semantics that RETSIGTYPE is void. So, remove it and just use void instead. Closes #6861 - [Muhammed Yavuz Nuzumlalı brought this change] install: add instructions for Apple Darwin platforms Closes #6860 - [Muhammed Yavuz Nuzumlalı brought this change] configure: disable min version set for Darwin Fixes #6838 Closes #6860 - [David Hu brought this change] docs/HTTP3.md: update the build instruction using gnutls In ngtcp2 the `with-gnutls` option is disabled by default, which will cause `curl` unable to be `make` because of lacking the libraries needed. Closes #6857 - RELEASE-NOTES: synced - typecheck-gcc: make the ssl-ctx-cb check use SSL_CTX pointers ... and not values. Reported-by: locpyl-tidnyd on github Fixes #6818 Closes #6819 - ngtcp2+gnutls: clear credentials when freed ... to avoid double-free. Reported-by: Kenneth Davidson Fixes #6824 Closes #6856 Jay Satiro (5 Apr 2021) - [Cherish98 brought this change] tool_progress: Fix progress meter in parallel mode Make sure the total amount of DL/UL bytes are counted before the transfer finalizes. Otherwise if a transfer finishes too quick, its total numbers are not added, and results in a DL%/UL% that goes above 100%. Detail: progress_meter() is called periodically, and it may not catch a transfer's total bytes if the value was unknown during the last call, and the transfer is finished and deleted (i.e., lost) during the next call. Closes https://github.com/curl/curl/pull/6840 - [Emil Engler brought this change] libssh: get rid of PATH_MAX This removes the last occurrence of PATH_MAX inside our libssh implementation by calculating the path length from the string length of the two components. Closes #6829 Daniel Stenberg (5 Apr 2021) - http_proxy: only loop on 407 + close if we have credentials ... to fix the retry-loop. Add test 718 to verify. Reported-by: Daniel Kurečka Fixes #6828 Closes #6850 - h2: allow 100 streams by default instead of 13, before the server has told how many streams it accepts. The server can always reject new streams anyway if we go above what it accepts. Ref: #6826 Closes #6852 - [Luke Granger-Brown brought this change] file: support GETing directories again After 957bc1881e686f9714c4e6a01bf33535091f0e21, we no longer compute an expected_size for directories. This has the upshot that when we compare even an empty Range with the available size, we fail. This brings back the previous behaviour, which was to succeed, but with empty content. This also removes the "Accept-ranges: bytes" header, which is nonsensical on directories. Adds test 3016 Fixes #6845 Closes #6846 - RELEASE-NOTES: synced and bumped to 7.76.1 - TLS: fix HTTP/2 selection for GnuTLS, BearSSL, mbedTLS, NSS, SChannnel, Secure Transport and wolfSSL... Regression since 88dd1a8a115b1f5ece (shipped in 7.76.0) Reported-by: Kenneth Davidson Reported-by: romamik om github Fixes #6825 Closes #6827 Jay Satiro (2 Apr 2021) - hostip: Fix for builds that disable all asynchronous DNS - Define Curl_resolver_error function only when USE_CURL_ASYNC. Prior to this change building curl without an asynchronous resolver backend (c-ares or threaded) and without DoH (DNS-over-HTTPS, which is also asynchronous but independent of resolver backend) would cause a build error since Curl_resolver_error is called by and evaluates variables only available in asynchronous builds. Reported-by: Benbuck Nason Fixes https://github.com/curl/curl/issues/6831 Closes https://github.com/curl/curl/pull/6832 Daniel Stenberg (31 Mar 2021) - [Gilles Vollant brought this change] openssl: Fix CURLOPT_SSLCERT_BLOB without CURLOPT_SSLCERT_KEY Reported-by: Christian Schmitz Fixes #6816 Closes #6820 Version 7.76.0 (31 Mar 2021) Daniel Stenberg (31 Mar 2021) - RELEASE-NOTES: synced curl 7.76.0 release - THANKS: added names from 7.76.0 - CURLOPT_AUTOREFERER.3: clarify that it sets the full URL ... some users may not want that! - define: remove CURL_DISABLE_NTLM ifdefs It was never defined anywhere. Fixed disable-scan (test 1165) to also scan headers, which found this issue. Closes #6809 - vtls: fix addsessionid for non-proxy builds Follow-up to b09c8ee15771c61 Fixes #6812 Closes #6811 - [Li Xinwei brought this change] cmake: support WinIDN Closes #6807 - transfer: clear 'referer' in declaration To silence (false positive) compiler warnings about it. Follow-up to 7214288898f5625 Reviewed-by: Marcel Raad Closes #6810 - [Marc Hoersken brought this change] config: fix SSPI enabling NTLM if crypto auth is disabled Avoid enabling NTLM feature based upon Windows SSPI being enabled in case that crypto auth is disabled. Reported-by: Marcel Raad Follow-up to #6277 Fixes #6803 Closes #6808 - HISTORY: add two 2021 events - vtls: add 'isproxy' argument to Curl_ssl_get/addsessionid() To make sure we set and extract the correct session. Reported-by: Mingtao Yang Bug: https://curl.se/docs/CVE-2021-22890.html CVE-2021-22890 - [Viktor Szakats brought this change] transfer: strip credentials from the auto-referer header field Added test 2081 to verify. CVE-2021-22876 Bug: https://curl.se/docs/CVE-2021-22876.html - curl_sasl: fix compiler error with --disable-crypto-auth ... if libgsasl was found. Closes #6806 - [Patrick Monnerat brought this change] ldap: only set the callback ptr for TLS context when TLS is used Follow-up to a5eee22e594c2460f Fixes #6804 Closes #6805 - copyright: update copyright year ranges to 2021 Reviewed-by: Emil Engler Closes #6802 - send_speed: simplify the checks for if a speed limit is set ... as we know the value cannot be set to negative: enforced by setopt() - http: cap body data amount during send speed limiting By making sure never to send off more than the allowed number of bytes per second the speed limit logic is given more room to actually work. Reported-by: Fabian Keil Bug: https://curl.se/mail/lib-2021-03/0042.html Closes #6797 - urldata: merge "struct DynamicStatic" into "struct UrlState" Both were used for the same purposes and there was no logical separation between them. Combined, this also saves 16 bytes in less holes in my test build. Closes #6798 - tests/README.md: mentioned that en_US.UTF-8 is required Reported-by: Oumph on github Fixes #6768 - HISTORY: fixed the Mac OS X 10.1 release date Based on what Wikipedia says Jay Satiro (26 Mar 2021) - examples: Remove threaded-shared-conn.c due to bug Known bug 11.11 is the shared object's connection cache is not thread safe, so we should not have an example for it. Ref: https://github.com/curl/curl/issues/4915 Ref: https://curl.se/docs/knownbugs.html#A_shared_connection_cache_is_not Closes https://github.com/curl/curl/pull/6795 - KNOWN_BUGS: Update 11.9 - DoH option inheritance - Add description: Explain that some options aren't inherited because they are not relevant for the DoH SSL connections or may result in unexpected behavior. - Remove the reference to #4578 (SSL verify options not inherited) since that was fixed by #6597 (separate DoH-specific options for verify). - Explain that DoH-specific options (those created by #6597) are available: CURLOPT_DOH_SSL_VERIFYHOST, CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS. - Add a reference to #6605 and explain that the user's debug function is not inherited because it would be unexpected to pass internal handles (ie DoH handles) to the user's callback. Closes https://github.com/curl/curl/issues/6605 Daniel Stenberg (26 Mar 2021) - curl_easy_setopt.3: add curl_easy_option* functions to SEE ALSO - [Jean-Philippe Menil brought this change] openssl: ensure to check SSL_CTX_set_alpn_protos return values SSL_CTX_set_alpn_protos() return 0 on success, and non-0 on failure Signed-off-by: Jean-Philippe Menil <jpmenil@gmail.com> Closes #6794 - multi: close the connection when h2=>h1 downgrading Otherwise libcurl is likely to reuse the connection again in the next attempt since the connection reuse logic doesn't take downgrades into account. Reported-by: Anthony Ramine Fixes #6788 Closes #6793 - openssl: set the transfer pointer for logging early Otherwise, the transfer will be NULL in the trace function when the early handshake details arrive and then curl won't show them. Regresssion in 7.75.0 Reported-by: David Hu Fixes #6783 Closes #6792 - RELEASE-NOTES: synced - TODO: Custom progress meter update interval Ref: https://stackoverflow.com/q/66789977/93747 - docs/ABI: tighten up the language Make the promises more firm Closes #6786 - openldap: disconnect better Instead of clearing the callback argument in disconnect, set it to the (new) transfer to make sure the correct data is passed to the callbacks. Follow-up to e467ea3bd937f38 Assisted-by: Patrick Monnerat Closes #6787 - libssh2: kdb_callback: get the right struct pointer After the recent conn/data refactor in this source file, this function was mistakenly still getting the old struct pointer which would lead to crash on servers with keyboard-interactive auth enabled. Follow-up to a304051620b92e12b (shipped in 7.75.0) Reported-by: Christian Schmitz Fixes #6691 Closes #6782 - tftp: remove unused struct fields Follow-up to d3d90ad9c00530d Closes #6781 - openldap: avoid NULL pointer dereferences Follow-up to a59c33ceffb8f78 Reported-by: Patrick Monnerat Fixes #6676 Closes #6780 - http: strip default port from URL sent to proxy To make sure the Host: header and the URL provide the same authority portion when sent to the proxy, strip the default port number from the URL if one was provided. Reported-by: Michael Brown Fixes #6769 Closes #6778 - azure: disable test 433 on azure-ubuntu Something in that environment sets XDG_CONFIG_HOME for us in a way that breaks the test. Reported-by: Marc Hörsken Fixes #6739 Closes #6777 - tftp: remove the 3600 second default timeout ... it was never meant to be there. Reported-by: Tomas Berger Fixes #6774 Closes #6776 - docs: make gen.pl support *italic* and **bold** Remove some nroffisms from the cmdline doc files to simplify editing, and instead support this markdown style. Closes #6771 - ngtcp2: sync with recent API updates Closes #6770 - RELEASE-NOTES: synced - libssh2:ssh_connect: clear session pointer after free If libssh2_knownhost_init() returns NULL, like in an OOM situation, the ssh session was freed but the pointer wasn't cleared which made libcurl later call libssh2 to cleanup using the stale pointer. Fixes #6764 Closes #6766 - [Jacob Hoffman-Andrews brought this change] docs: document version of crustls dependency This also pins a specific release in the Travis test so future API-breaking changins in crustls won't break curl builds. Add RUSTLS documentation to release tarball. Enable running tests for rustls, minus FTP tests (require connect_blocking, which rustls doesn't implement) and 313 (requires CRL handling). Closes #6763 - [Jacob Hoffman-Andrews brought this change] rustls: Handle close_notify. If we get a close_notify, treat that as EOF. If we get an EOF from the TCP stream, treat that as an error (because we should have ended the connection earlier, when we got a close_notify). Closes #6763 - docs: clarify timeouts for queued transfers in multi API Closes #6758 - ftpserver: only load the preprocessed test file We always preprocess and tests are no longer sensible to load "raw" Closes #6738 - tests: use %TESTNUMBER instead of fixed number This makes the tests easier to copy and relocate to other test numbers without having to update content. Closes #6738 - KNOWN_BUGS: CURLOPT_OPENSOCKETPAIRFUNCTION is missing Closes #5747 - TODO: provide timing info for each redirect Closes #6743 Jay Satiro (17 Mar 2021) - docs: Add SSL backend names to CURL_SSL_BACKEND - Document the names that can be used with CURL_SSL_BACKEND: bearssl, gnutls, gskit, mbedtls, mesalink, nss, openssl, rustls, schannel, secure-transport, wolfssl Ref: https://github.com/curl/curl/issues/2209#issuecomment-360623286 Ref: https://github.com/curl/curl/issues/6717#issuecomment-800745201 Closes https://github.com/curl/curl/pull/6755 - docs: Explain DOH transfers inherit some SSL settings - Document in DOH that some SSL settings are inherited but DOH hostname and peer verification are not and are controlled separately. - Document that CURLOPT_SSL_CTX_FUNCTION is inherited by DOH handles but we're considering changing behavior to no longer inherit it. Request feedback. Closes https://github.com/curl/curl/pull/6688 Daniel Stenberg (17 Mar 2021) - http: make 416 not fail with resume + CURLOPT_FAILONERRROR When asked to resume a download, libcurl will convert that to HTTP logic and if then the entire file is already transferred it will result in a 416 response from the HTTP server. With CURLOPT_FAILONERRROR set in that scenario, it should *not* lead to an error return. Updated test 1156, added test 1273 Reported-by: Jonathan Watt Fixes #6740 Closes #6753 - Curl_timeleft: check both timeouts during connect The duration of a connect and the total transfer are calculated from two different time-stamps. It can end up with the total timeout triggering before the connect timeout expires and we should make sure to acknowledge whichever timeout that is reached first. This is especially notable when a transfer first sits in PENDING, as that time is counted in the total time but the connect timeout is based on the time since the handle changed to the CONNECT state. The CONNECTTIMEOUT is per connect attempt. The TIMEOUT is for the entire operation. Fixes #6744 Closes #6745 Reported-by: Andrei Bica Assisted-by: Jay Satiro - configure: remove use of deprecated macros AC_HEADER_TIME, AC_HEADER_STDC and AC_TYPE_SIGNAL - configure: make AC_TRY_* into AC_*_IFELSE ... as the former versions are deprecated. - configure: s/AC_HELP_STRING/AS_HELP_STRING AC_HELP_STRING is deprecated in 2.70+ and I believe AS_HELP_STRING works already since 2.59 so bump the minimum required version to that. Reported-by: Emil Engler Fixes #6647 Closes #6748 - RELEASE-NOTES: synced - travis: use ubuntu nghttp2 package instead of build our own Closes #6751 - travis: bump wolfssl to 4.7.0 - travis: only build wolfssl when needed Closes #6751 - [Jacob Hoffman-Andrews brought this change] rustls: allocate a buffer for TLS data. Previously, rustls was using an on-stack array for TLS data. However, crustls has an (unusual) requirement that buffers it deals with are initialized before writing to them. By using calloc, we can ensure the buffer is initialized once and then reuse it across calls. Closes #6742 - travis: add a rustls build ... that doesn't run any tests (yet) Closes #6750 - HTTP2: remove the outdated remark about multiplexing for the tool - [Robert Ronto brought this change] http2: don't set KEEP_SEND when there's no more data to be sent this should fix an issue where curl sometimes doesn't send out a request with authorization info after a 401 is received over http2 Closes #6747 Marc Hoersken (15 Mar 2021) - config: fix building SMB with configure using Win32 Crypto Align conditions for NTLM features between CMake and configure builds by differentiating between USE_NTLM and USE_CURL_NTLM_CORE, just like curl_setup.h does internally to detect support of: - USE_NTLM: required for NTLM crypto authentication feature - USE_CURL_NTLM_CORE: required for SMB protocol Implement USE_WIN32_CRYPTO detection by checking for Crypt functions in wincrypt.h which are not available in the Windows App environment. Link advapi32 and crypt32 for Crypto API and Schannel SSL backend. Fix condition of Schannel SSL backend in CMake build accordingly. Reviewed-by: Marcel Raad Closes #6277 - config: fix detection of restricted Windows App environment Move the detection of the restricted Windows App environment in curl_setup.h before the definition of USE_WIN32_CRYPTO via included config-win32.h in case no build system is used. Reviewed-by: Marcel Raad Part of #6277 Daniel Stenberg (15 Mar 2021) - HISTORY: curl 7.7.2 was the first version used in Mac OS X 10.1 - gen.pl: quote "bare" minuses in the nroff curl.1 Reported-by: Alejandro Colomar Fixes #6698 Closes #6722 Daniel Gustafsson (14 Mar 2021) - hsts: remove unused defines MAX_HSTS_SUBLEN and MAX_HSTS_SUBLENSTR were unused from the initial commit, and mostly likely leftovers from early development. Remove as they're not used for anything. Closes #6741 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (12 Mar 2021) - github: add torture-ftp for FTP-only torture testing and at 20% to try to keep the run-time reasonable Closes #6728 - travis: split "torture" into a separate "events" build as well Run torture without FTP and reducing coverage to 20% For some reason the torture tests now run a lot slower on travis and run into the 50 minute limit all the time. Closes #6728 - ftp: fix memory leak in ftp_done If after a transfer is complete Curl_GetFTPResponse() returns an error, curl would not free the ftp->pathalloc block. Found by torture-testing test 576 Closes #6737 - [oxalica brought this change] http2: fail if connection terminated without END_STREAM Closes #6736 - RELEASE-NOTES: synced - [Jacob Hoffman-Andrews brought this change] rustls: support CURLOPT_SSL_VERIFYPEER This requires the latest main branch of crustls, which provides rustls_client_config_builder_dangerous_set_certificate_verifier and rustls_client_config_builder_set_enable_sni. This refactors the session setup into its own function, and adds a new function cr_hostname_is_ip. Because crustls doesn't support verification of IP addresses, special handling is needed: We disable SNI and set a placeholder hostname (which never actually gets sent on the wire). Closes #6719 Daniel Gustafsson (12 Mar 2021) - cookies: Fix potential NULL pointer deref with PSL Curl_cookie_init can be called with data being NULL, and this can in turn be passed to Curl_cookie_add, meaning that both functions must be careful to only use data where it's checked for being a NULL pointer. The libpsl support code does however dereference data without checking, so if we are indeed having an unset data pointer we cannot PSL check the cookiedomain. This is currently not a reachable dereference, as the only caller with a NULL data isn't passing a file to initialize cookies from, but since the API has this contract let's ensure we hold it. Closes #6731 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (12 Mar 2021) - [Michael Hordijk brought this change] configure: only add OpenSSL paths if they are defined Add paths for OpenSSL compiling and linking only if they have been defined. If they haven't been defined, we'll assume that the paths are already available to the toolchain. Closes #6730 Jay Satiro (12 Mar 2021) - retry.d: Clarify transient 5xx HTTP response codes - Clarify the only 5xx response codes that are treated as transient are 500, 502, 503 and 504. Prior to this change it said it treated all 5xx as transient, but the code says otherwise. Ref: https://github.com/curl/curl/blob/curl-7_75_0/src/tool_operate.c#L462-L495 Closes https://github.com/curl/curl/pull/6724 - retry-all-errors.d: Explain curl errors versus HTTP response errors - Add a paragraph explaining that curl does not consider HTTP response errors as curl errors, and how that behavior can be modified by using --retry and --fail. The --retry-all-errors doc says "Retry on any error" which some users may find misleading without the added explanation. Ref: https://curl.se/docs/faq.html#Why_do_I_get_downloaded_data_eve Ref: https://curl.se/docs/faq.html#curl_doesn_t_return_error_for_HT Reported-by: Lawrence Gripper Fixes https://github.com/curl/curl/issues/6712 Closes https://github.com/curl/curl/pull/6720 Daniel Stenberg (11 Mar 2021) - travis: switch ngtcp2 build over to quictls The ngtcp2 project switched over to using the quictls OpenSSL fork instead of their own patched OpenSSL. We follow suit. Closes #6729 - test220/314: adjust to run with Hyper - c-hyper: support automatic content-encoding Closes #6727 - http: remove superfluous NULL assign Closes #6727 - tool_operate: bail if set CURLOPT_HTTP09_ALLOWED returns error Closes #6727 - setopt: error on CURLOPT_HTTP09_ALLOWED set true with Hyper Not supported. Closes #6727 - test306: make it not run with Hyper ... as it tests HTTP/0.9 which Hyper doesn't support. - test304: header CRLF cleanup to work with Hyper - FTP: allow SIZE to fail when doing (resumed) upload Added test 362 to verify. Reported-by: Jordan Brown Regression since 7ea2e1d0c5a7f (7.73.0) Fixes #6715 Closes #6725 - configure: provide Largefile feature for curl-config ... as cmake now does it correctly, and make test1014 check for it Closes #6702 - config: remove CURL_SIZEOF_CURL_OFF_T use only SIZEOF_CURL_OFF_T Make the code consistently use a single name for the size of the "curl_off_t" type. Closes #6702 Jay Satiro (10 Mar 2021) - [Jun-ya Kato brought this change] ngtcp2: Fix build error due to change in ngtcp2_addr_init ngtcp2/ngtcp2@b8d90a9 changed the function prototype. Closes https://github.com/curl/curl/pull/6716 Daniel Stenberg (10 Mar 2021) - [ejanchivdorj brought this change] multi: update pending list when removing handle when removing a handle, most of the lists are updated but pending list is not updated. Updating now. Closes #6713 - [kokke brought this change] lib1536: check ptr against NULL before dereferencing it Closes #6710 - [kokke brought this change] lib1537: check ptr against NULL before dereferencing it Fixes #6707 Closes #6708 - travis: make torture tests skip TLS-SRP tests ... as it seems to often hang. Also: skip the "normal" tests as they're already run by many other builds. Closes #6705 - openssl: adapt to v3's new const for a few API calls Closes #6703 - quiche: fix crash when failing to connect Reported-by: ウさん Fixes #6664 Closes #6701 - RELEASE-NOTES: synced Fixed the release counter and added a missing contributor - RELEASE-NOTES: synced - dynbuf: bump the max HTTP request to 1MB Raised from 128KB to allow longer request headers. Reported-by: Carl Zogheib Fixes #6681 Closes #6685 Jay Satiro (6 Mar 2021) - schannel: Evaluate CURLOPT_SSL_OPTIONS via SSL_SET_OPTION macro - Change use of those options from CURLOPT_SSL_OPTIONS that are not already evaluated via SSL_SET_OPTION in schannel and secure transport to use that instead of data->set.ssl.optname. Example: Evaluate SSL_SET_OPTION(no_revoke) instead of data->set.ssl.no_revoke. This change is because options set via CURLOPT_SSL_OPTIONS (data->set.ssl.optname) are separate from those set for HTTPS proxy via CURLOPT_PROXY_SSL_OPTIONS (data->set.proxy_ssl.optname). The SSL_SET_OPTION macro determines whether the connection is for HTTPS proxy and based on that which option to evaluate. Since neither Schannel nor Secure Transport backends currently support HTTPS proxy in libcurl, this change is for posterity and has no other effect. Closes https://github.com/curl/curl/pull/6690 - [kokke brought this change] c-hyper: Remove superfluous pointer check `n` pointer is never NULL once set. Found by static analysis. Ref: https://github.com/curl/curl/issues/6696 Closes https://github.com/curl/curl/pull/6697 - version.d: Add missing features to the features list - Add missing entries for gsasl, Kerberos, NTLM_WB, TrackMemory, Unicode and zstd. - Remove krb4 since it's no longer a feature. Reported-by: Ádler Jonas Gross Fixes https://github.com/curl/curl/issues/6677 Closes https://github.com/curl/curl/pull/6687 - [Vladimir Varlamov brought this change] docs: add missing Arg tag to --stderr Prior to this change the required argument was not shown. curl.1 before: --stderr curl.1 after: --stderr <file> curl --help before: --stderr Where to redirect stderr curl --help after: --stderr <file> Where to redirect stderr Closes https://github.com/curl/curl/pull/6692 - projects: Update VS projects for OpenSSL 1.1.x - Update VS project templates to use the OpenSSL lib names and include directories for OpenSSL 1.1.x. This change means the VS project files will now build only with OpenSSL 1.1.x when an OpenSSL configuration is chosen. Prior to this change the project files built only with OpenSSL 1.0.x (end-of-life) when an OpenSSL configuration was chosen. The template changes in this commit were made by script: libeay32.lib => libcrypto.lib ssleay32.lib => libssl.lib ..\..\..\..\..\openssl\inc32 => ..\..\..\..\..\openssl\include And since the output directory now contains the includes it's prepended: ..\..\..\..\..\openssl\build\Win{32,64}\VC{6..15}\{DLL,LIB} {Debug,Release}\include - Change build-openssl.bat to copy the build's include directory to the output directory (as seen above). Each build has its own opensslconf.h which is different so we can't just include the source include directory any longer. Note the include directory in the output directory is a full copy from the build so technically we don't need to include the OpenSSL source include directory in the template. However, I left it last in case the user made a custom OpenSSL build using the old method which would put opensslconf in the OpenSSL source include directory. - Change build-openssl.bat to use a temporary install directory that is different from the temporary build directory. For OpenSSL 1.1.x the temporary paths must be separate not a descendant of the other, otherwise pdb files will be lost between builds. Ref: https://curl.se/mail/lib-2018-10/0049.html Ref: https://gist.github.com/jay/125191c35bbeb894444eff827651f755 Ref; https://github.com/openssl/openssl/issues/10005 Fixes https://github.com/curl/curl/issues/984 Closes https://github.com/curl/curl/pull/6675 - doh: Inherit CURLOPT_STDERR from user's easy handle Prior to this change if the user set their easy handle's error stream to something other than stderr it was not inherited by the doh handles, which meant that they would still write to the default standard error stream (stderr) for verbose output. Bug: https://github.com/curl/curl/issues/6605 Reported-by: arvids-kokins-bidstack@users.noreply.github.com Closes https://github.com/curl/curl/pull/6661 Marc Hoersken (1 Mar 2021) - CI/azure: replace python-impacket with python3-impacket As of this month Azure DevOps uses Ubuntu 20.04 LTS which no longer supports Python 2 and instead ships Python 3. Closes #6678 - runtests.pl: kill processes locking test log files Introduce a new runtests.pl command option: -rm For now only required and implemented for Windows. Ignore stunnel logs due to long running processes. Requires Sysinternals handle[64].exe to be on PATH. Reviewed-by: Jay Satiro Ref: #6058 Closes #6179 - pathhelp.pm: fix use of pwd -L in Msys environment While Msys2 has a pwd binary which supports -L, Msys1 only has a shell built-in with that feature. Reviewed-by: Jay Satiro Part of #6179 Daniel Gustafsson (1 Mar 2021) - ldap: use correct memory free function unescaped is coming from Curl_urldecode and not a unicode conversion function, so reclaiming its memory should be performed with a normal call to free rather than curlx_unicodefree. In reality, this is the same thing as curlx_unicodefree is implemented as a call to free but that's not guaranteed to always hold. Using the curlx macro present issues with memory debugging as well. Closes #6671 Reviewed-by: Jay Satiro <raysatiro@yahoo.com> Reviewed-by: Daniel Stenberg <daniel@haxx.se> - url: fix typo in comment Correct a small typo which snuck in with a304051620. Jay Satiro (28 Feb 2021) - tool_help: Increase space between option and description - Increase the minimum number of spaces between the option and the description from 1 to 2. Before: ~~~ -u, --user <user:password> Server user and password -A, --user-agent <name> Send User-Agent <name> to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit -w, --write-out <format> Use output FORMAT after completion --xattr Store metadata in extended file attributes ~~~ After: ~~~ -u, --user <user:password> Server user and password -A, --user-agent <name> Send User-Agent <name> to server -v, --verbose Make the operation more talkative -V, --version Show version number and quit -w, --write-out <format> Use output FORMAT after completion --xattr Store metadata in extended file attributes ~~~ Closes https://github.com/curl/curl/pull/6674 Daniel Stenberg (27 Feb 2021) - curl: set CURLOPT_NEW_FILE_PERMS if requested The --create-file-mode code logic accepted the value but never actually passed it on to libcurl! Follow-up to a7696c73436f (shipped in 7.75.0) Reported-by: Johannes Lesr Fixes #6657 Closes #6666 - tool_operate: check argc before accessing argv[1] Follow-up to 09363500b Reported-by: Emil Engler Reviewed-by: Daniel Gustafsson Closes #6668 Daniel Gustafsson (26 Feb 2021) - [Jean-Philippe Menil brought this change] openssl: remove get_ssl_version_txt in favor of SSL_get_version openssl: use SSL_get_version to get connection protocol Replace our bespoke get_ssl_version_txt in favor of SSL_get_version. We can get rid of few lines of code, since SSL_get_version achieve the exact same thing Closes #6665 Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Signed-off-by: Jean-Philippe Menil <jpmenil@gmail.com> - gnutls: Fix nettle discovery Commit e06fa7462ac258c removed support for libgcrypt leaving only support for nettle which has been the default crypto library in GnuTLS for a long time. There were however a few conditionals on USE_GNUTLS_NETTLE which cause compilation errors in the metalink code (as it used the gcrypt fallback instead as a result). See the below autobuild for an example of the error: https://curl.se/dev/log.cgi?id=20210225123226-30704#prob1 This removes all uses of USE_GNUTLS_NETTLE and also removes the gcrypt support from the metalink code while at it. Closes #6656 Reviewed-by: Daniel Stenberg <daniel@haxx.se> - cookies: Support multiple -b parameters Previously only a single -b cookie parameter was supported with the last one winning. This adds support for supplying multiple -b params to have them serialized semicolon separated. Both cookiefiles and cookies can be entered multiple times. Closes #6649 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (25 Feb 2021) - build: remove all traces of USE_BLOCKING_SOCKETS libcurl doesn't behave properly with the define set Closes #6655 - RELEASE-NOTES: synced Daniel Gustafsson (25 Feb 2021) - docs: Fix typos Random typos spotted when skimming docs. - cookies: Use named parameters in header prototypes Align header with project style of using named parameters in the function prototypes to aid readability and self-documentation. Closes #6653 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (24 Feb 2021) - urldata: make 'actions[]' use unsigned char instead of int ... as it only needs a few bits per index anyway. Reviewed-by: Daniel Gustafsson Closes #6648 - configure: fail if --with-quiche is used and quiche isn't found Closes #6652 - [Gregor Jasny brought this change] cmake: use CMAKE_INSTALL_INCLUDEDIR indirection Reviewed-by: Sergei Nikulov Closes #6440 Viktor Szakats (23 Feb 2021) - mingw: enable using strcasecmp() This makes the 'Features:' list sorted case-insensitively, bringing output in-line with *nix builds. Reviewed-by: Jay Satiro Closes #6644 - build: delete unused feature guards - `HAVE_STRNCASECMP` - `HAVE_TCGETATTR` - `HAVE_TCSETATTR` Reviewed-by: Jay Satiro Reviewed-by: Daniel Stenberg Closes #6645 Jay Satiro (23 Feb 2021) - docs: add CURLOPT_CURLU to 'See also' in curl_url_ functions Closes https://github.com/curl/curl/pull/6639 Daniel Stenberg (23 Feb 2021) - [Jacob Hoffman-Andrews brought this change] configure: make hyper opt-in, and fail if missing Previously, configure would look for hyper by default, and use it if found; otherwise it would not use hyper, and not error. Now, configure will not look for hyper unless --with-hyper is passed. If configure looks for hyper and fails, it will error. Also, add -ld -lpthread -lm to Hyper's libs. I think they are required. Closes #6598 - multi: do once-per-transfer inits in before_perform in DID state ... since the state machine might go to RATELIMITING and then back to PERFORMING doing once-per-transfer inits in that function is wrong and it caused problems with receiving chunked HTTP and it set the PRETRANSFER time much too often... Regression from b68dc34af341805aeb7b3715 (shipped in 7.75.0) Reported-by: Amaury Denoyelle Fixes #6640 Closes #6641 - RELEASE-NOTES: synced - CODE_STYLE.md: fix broken link to INTERNALS ... the link would only work if browsed on GitHub, while this link now takes the user to the website instead and thus should work on either. Reported-by: David Demelier - curl_url_set.3: mention CURLU_PATH_AS_IS ... it has been supported since the URL API was added. Bug: https://curl.se/mail/lib-2021-02/0046.html Closes #6638 Viktor Szakats (21 Feb 2021) - time: enable 64-bit time_t in supported mingw environments (Unless 32-bit `time_t` is selected manually via the `_USE_32BIT_TIME_T` mingw macro.) Previously, 64-bit `time_t` was enabled on VS2005 and newer only, and 32-bit `time_t` was used on all other Windows builds. Assisted-by: Jay Satiro Closes #6636 Jay Satiro (20 Feb 2021) - test1188: Check for --fail HTTP status - Change the test to check for curl error on HTTP 404 Not Found. test1188 tests "--write-out with %{onerror} and %{urlnum} to stderr". Prior to this change it did that by specifying a non-existent host which would cause an error. ISPs may hijack DNS and resolve non-existent hosts so the test would not work if that was the case. Ref: https://en.wikipedia.org/wiki/DNS_hijacking#Manipulation_by_ISPs Ref: https://github.com/curl/curl/issues/6621 Ref: https://github.com/curl/curl/pull/6623 Closes https://github.com/curl/curl/pull/6637 - memdebug: close debug logfile explicitly on exit - Use atexit to register a dbg cleanup function that closes the logfile. LeakSantizier (LSAN) calls _exit() instead of exit() when a leak is detected on exit so the logfile must be closed explicitly or data could be lost. Though _exit() does not call atexit handlers such as this, LSAN's call to _exit() comes after the atexit handlers are called. Prior to this change the logfile was not explicitly closed so it was possible that if LSAN detected a leak and called _exit (which does not flush or close files like exit) then the logfile could be missing data. That could then cause curl's memanalyze to report false leaks (eg a malloc was recorded to the logfile but the corresponding free was discarded from the buffer instead of written to the logfile, then memanalyze reports that as a leak). Ref: https://github.com/google/sanitizers/issues/1374 Bug: https://github.com/curl/curl/pull/6591#issuecomment-780396541 Closes https://github.com/curl/curl/pull/6620 - curl_multibyte: always return a heap-allocated copy of string - Change the Windows char <-> UTF-8 conversion functions to return an allocated copy of the passed in string instead of the original. Prior to this change the curlx_convert_ functions would, as what I assume was an optimization, not make a copy of the passed in string if no conversion was required. No conversion is required in non-UNICODE Windows builds since our tchar strings are type char and remain in whatever the passed in encoding is, which is assumed to be UTF-8 but may be other encoding. In contrast the UNICODE Windows builds require conversion (wchar <-> char) and do return a copy. That inconsistency could lead to programming errors where the developer expects a copy, and does not realize that won't happen in all cases. Closes https://github.com/curl/curl/pull/6602 Viktor Szakats (19 Feb 2021) - http: add new files missed from referrer commit Ref: 44872aefc2d54f297caf2b0cc887df321bc9d791 Ref: #6591 - http: add support to read and store the referrer header - add CURLINFO_REFERER libcurl option - add --write-out '%{referer}' command-line option - extend --xattr command-line option to fill user.xdg.referrer.url extended attribute with the referrer (if there was any) Closes #6591 Daniel Stenberg (19 Feb 2021) - urldata: remove the _ORIG suffix from string names It doesn't provide any useful info but only makes the names longer. Closes #6624 - url: fix memory leak if OOM in the HSTS handling Reported-by: Viktor Szakats Bug: https://github.com/curl/curl/pull/6627#issuecomment-781626205 Closes #6628 - gnutls: assume nettle crypto support nettle has been the default crypto library with GnuTLS since 2010. By dropping support for the previous libcrypto, we simplify code. Closes #6625 - asyn-ares: use consistent resolve error message ... with the help of Curl_resolver_error() which now is moved from asyn-thead.c and is provided globally for this purpose. Follow-up to 35ca04ce1b77636 Makes test 1188 work for c-ares builds Closes #6626 Viktor Szakats (18 Feb 2021) - ci: stop building on freebsd-12-1 An updated freebsd-12-2 image was added a few months ago, and this older one is consistently failing to go past `pkginstall`: ``` Newer FreeBSD version for package py37-mlt: To ignore this error set IGNORE_OSVERSION=yes - package: 1202000 - running kernel: 1201000 Ignore the mismatch and continue? [Y/n]: pkg: repository FreeBSD contains packages for wrong OS version: FreeBSD:12:amd64 ``` FreeBSD thread suggests that 12.1 is EOL, and best to avoid. Ref: https://forums.freebsd.org/threads/78856/ Reviewed-by: Daniel Stenberg Closes #6622 Daniel Stenberg (18 Feb 2021) - test1188: change error from connect to resolve error Using the %NOLISTENPORT to trigger a connection failure is somewhat "risky" (since it isn't guaranteed to not be listened to) and caused occasional CI problems. This fix changes the infused error to be a more reliable one but still verifies the --write-out functionality properly - which is the purpose of this test. Reported-by: Jay Satiro Fixes #6621 Closes #6623 - url.c: use consistent error message for failed resolve - BUGS: language polish - wolfssl: don't store a NULL sessionid This caused a memory leak as the session id cache entry was still erroneously stored with a NULL sessionid and that would later be treated as not needed to get freed. Reported-by: Gisle Vanem Fixes #6616 Closes #6617 - parse_proxy: fix a memory leak in the OOM path Reported-by: Jay Satiro Reviewed-by: Jay Satiro Reviewed-by: Emil Engler Closes #6614 Bug: https://github.com/curl/curl/pull/6591#issuecomment-780396541 Jay Satiro (17 Feb 2021) - url: fix possible use-after-free in default protocol Prior to this change if the user specified a default protocol and a separately allocated non-absolute URL was used then it was freed prematurely, before it was then used to make the replacement URL. Bug: https://github.com/curl/curl/issues/6604#issuecomment-780138219 Reported-by: arvids-kokins-bidstack@users.noreply.github.com Closes https://github.com/curl/curl/pull/6613 Daniel Stenberg (16 Feb 2021) - multi: rename the multi transfer states While working on documenting the states it dawned on me that step one is to use more descriptive names on the states. This also changes prefix on the states to make them shorter in the source. State names NOT ending with *ing are transitional ones. Closes #6612 Viktor Szakats (16 Feb 2021) - http: do not add a referrer header with empty value Previously an empty 'Referer:' header was added to the HTTP request when passing `--referer ';auto'` or `--referer ''` on the command-line. This patch makes `--referer` work like `--header 'Referer:'` and will only add the header if it has a non-zero length value. Reviewed-by: Jay Satiro Closes #6610 Daniel Stenberg (16 Feb 2021) - lib: remove 'conn->data' completely The Curl_easy pointer struct entry in connectdata is now gone. Just before commit 215db086e0 landed on January 8, 2021 there were 919 references to conn->data. Closes #6608 - openldap: pass 'data' to the callbacks instead of 'conn' Jay Satiro (15 Feb 2021) - doh: Fix sharing user's resolve list with DOH handles - Share the shared object from the user's easy handle with the DOH handles. Prior to this change if the user had set a shared object with shared cached DNS (CURL_LOCK_DATA_DNS) for their easy handle then that wasn't used by any associated DOH handles, since they used the multi's default hostcache. This change means all the handles now use the same hostcache, which is either the shared hostcache from the user created shared object if it exists or if not then the multi's default hostcache. Reported-by: Manuj Bhatia Fixes https://github.com/curl/curl/issues/6589 Closes https://github.com/curl/curl/pull/6607 Daniel Stenberg (15 Feb 2021) - http2: remove conn->data use ... but instead use a private alternative that points to the "driving transfer" from the connection. We set the "user data" associated with the connection to be the connectdata struct, but when we drive transfers the code still needs to know the pointer to the transfer. We can change the user data to become the Curl_easy handle, but with older nghttp2 version we cannot dynamically update that pointer properly when different transfers are used over the same connection. Closes #6520 - openssl: remove conn->data use We still make the trace callback function get the connectdata struct passed to it, since the callback is anchored on the connection. Repeatedly updating the callback pointer to set 'data' with SSL_CTX_set_msg_callback_arg() doesn't seem to work, probably because there might already be messages in the queue with the old pointer. This code therefore makes sure to set the "logger" handle before using OpenSSL calls so that the right easy handle gets used for tracing. Closes #6522 - RELEASE-NOTES: synced Jay Satiro (14 Feb 2021) - doh: add options to disable ssl verification - New libcurl options CURLOPT_DOH_SSL_VERIFYHOST, CURLOPT_DOH_SSL_VERIFYPEER and CURLOPT_DOH_SSL_VERIFYSTATUS do the same as their respective counterparts. - New curl tool options --doh-insecure and --doh-cert-status do the same as their respective counterparts. Prior to this change DOH SSL certificate verification settings for verifyhost and verifypeer were supposed to be inherited respectively from CURLOPT_SSL_VERIFYHOST and CURLOPT_SSL_VERIFYPEER, but due to a bug were not. As a result DOH verification remained at the default, ie enabled, and it was not possible to disable. This commit changes behavior so that the DOH verification settings are independent and not inherited. Ref: https://github.com/curl/curl/pull/4579#issuecomment-554723676 Fixes https://github.com/curl/curl/issues/4578 Closes https://github.com/curl/curl/pull/6597 - hostip: fix crash in sync resolver builds that use DOH - Guard some Curl_async accesses with USE_CURL_ASYNC instead of !CURLRES_SYNCH. This is another follow-up to 8335c64 which moved the async struct from the connectdata struct into the Curl_easy struct. A previous follow-up 6cd167a fixed building for sync resolver by guarding some async struct accesses with !CURLRES_SYNCH. The problem is since DOH (DNS-over-HTTPS) is available as an asynchronous secondary resolver the async struct may be used even when libcurl is built for the sync resolver. That means that CURLRES_SYNCH and USE_CURL_ASYNC may be defined at the same time. Closes https://github.com/curl/curl/pull/6603 Daniel Stenberg (13 Feb 2021) - KNOWN_BUGS: cannot enable LDAPS on Windows with cmake Reported-by: Jack Boos Yu Closes #6284 - KNOWN_BUGS: Excessive HTTP/2 packets with TCP_NODELAY Reported-by: Alex Xu Closes #6363 - http: use credentials from transfer, not connection HTTP auth "accidentally" worked before this cleanup since the code would always overwrite the connection credentials with the credentials from the most recent transfer and since HTTP auth is typically done first thing, this has not been an issue. It was still wrong and subject to possible race conditions or future breakage if the sequence of functions would change. The data.set.str[] strings MUST remain unmodified exactly as set by the user, and the credentials to use internally are instead set/updated in state.aptr.* Added test 675 to verify different credentials used in two requests done over a reused HTTP connection, which previously behaved wrongly. Fixes #6542 Closes #6545 - test433: clear some home dir env variables Follow-up to bd6b54ba1f55b5 ... so that XDG_CONFIG_HOME is the only home dir variable set and thus used correctly in the test! Fixes #6599 Closes #6600 - RELEASE-NOTES: synced bumped the version to 7.76.0 - travis: install libgsasl-dev to add that to the builds Closes #6588 - urldata: don't touch data->set.httpversion at run-time Rename it to 'httpwant' and make a cloned field in the state struct as well for run-time updates. Also: refuse non-supported HTTP versions. Verified with test 129. Closes #6585 Viktor Szakats (11 Feb 2021) - tests: disable .curlrc in more environments by also setting CURL_HOME and XDG_CONFIG_HOME envvars to the local directory. Reviewed-by: Daniel Stenberg Fixes #6595 Closes #6596 - docs/Makefile.inc: format to be update-friendly - one source file per line - convert tabs to spaces - do not align line-continuation backslashes - sort source files alphabetically Reviewed-by: Daniel Stenberg Closes #6593 Daniel Stenberg (11 Feb 2021) - curl: provide libgsasl version and feature info in -V output Closes #6592 - gsasl: provide CURL_VERSION_GSASL if built-in To let applications know the feature is available. Closes #6592 - curl: add --fail-with-body Prevent both --fail and --fail-with-body on the same command line. Verify with test 349, 360 and 361. Closes #6449 - TODO: remove HSTS Provided now since commit 7385610d0c74 Jay Satiro (10 Feb 2021) - tests: Fix tests failing due to change in curl --help Follow-up to parent 3183217 which added add missing <mode> argument to --create-file-mode <mode>. Ref: https://github.com/curl/curl/issues/6590 - tool_help: add missing argument for --create-file-mode Prior to this change the required argument was not shown in curl --help. before: --create-file-mode File mode for created files after: --create-file-mode <mode> File mode (octal) for created files Reported-by: ZimCodes@users.noreply.github.com Fixes https://github.com/curl/curl/issues/6590 - create-file-mode.d: add missing Arg tag Prior to this change the required argument was not shown. curl.1 before: --create-file-mode curl.1 after: --create-file-mode <mode> Reported-by: ZimCodes@users.noreply.github.com Fixes https://github.com/curl/curl/issues/6590 Viktor Szakats (10 Feb 2021) - gsasl: fix errors/warnings building against libgsasl - also fix an indentation - make Curl_auth_gsasl_token() use CURLcode (by Daniel Stenberg) Ref: https://github.com/curl/curl/pull/6372#issuecomment-776118711 Ref: https://github.com/curl/curl/pull/6588 Reviewed-by: Jay Satiro Assisted-by: Daniel Stenberg Reviewed-by: Simon Josefsson Closes #6587 - Makefile.m32: add support for libgsasl dependency Reviewed-by: Marcel Raad Closes #6586 Marcel Raad (10 Feb 2021) - ngtcp2: clarify calculation precedence As suggested by Codacy/cppcheck. Closes https://github.com/curl/curl/pull/6576 - server: remove redundant condition `end` is always non-null here. Closes https://github.com/curl/curl/pull/6576 - lib: remove redundant code Closes https://github.com/curl/curl/pull/6576 - mqttd: remove unused variable Closes https://github.com/curl/curl/pull/6576 - tool_paramhlp: reduce variable scope Closes https://github.com/curl/curl/pull/6576 - tests: reduce variable scopes Closes https://github.com/curl/curl/pull/6576 - lib: reduce variable scopes Closes https://github.com/curl/curl/pull/6576 - ftp: fix Codacy/cppcheck warning about null pointer arithmetic Increment `bytes` only if it is non-null. Closes https://github.com/curl/curl/pull/6576 Daniel Stenberg (9 Feb 2021) - ngtcp2: adapt to the new recv_datagram callback - quiche: fix build error: use 'int' for port number Follow-up to cb2dc1ba8 - ftp: add 'list_only' to the transfer state struct and rename it from 'ftp_list_only' since it is also used for SSH and POP3. The state is updated internally for 'type=D' FTP URLs. Added test case 1570 to verify. Closes #6578 - ftp: add 'prefer_ascii' to the transfer state struct ... and make sure the code never updates 'set.prefer_ascii' as it breaks handle reuse which should use the setting as the user specified it. Added test 1569 to verify: it first makes an FTP transfer with ';type=A' and then another without type on the same handle and the second should then use binary. Previously, curl failed this. Closes #6578 - RELEASE-NOTES: synced - [Jacob Hoffman-Andrews brought this change] vtls: initial implementation of rustls backend This adds a new TLS backend, rustls. It uses the C-to-rustls bindings from https://github.com/abetterinternet/crustls. Rustls is at https://github.com/ctz/rustls/. There is still a fair bit to be done, like sending CloseNotify on connection shutdown, respecting CAPATH, and properly indicating features like "supports TLS 1.3 ciphersuites." But it works well enough to make requests and receive responses. Blog post for context: https://www.abetterinternet.org/post/memory-safe-curl/ Closes #6350 - [Simon Josefsson brought this change] sasl: support SCRAM-SHA-1 and SCRAM-SHA-256 via libgsasl Closes #6372 Jay Satiro (9 Feb 2021) - lib: use int type for more port variables This is a follow-up to 764c6bd. Prior to that change port variables were usually type long. Closes https://github.com/curl/curl/pull/6553 - tool_writeout: refactor write-out and write-out json - Deduplicate the logic used by write-out and write-out json. Rather than have separate writeLong, writeString, etc, logic for each of write-out and write-out json instead have respective shared functions that can output either format and a 'use_json' parameter to indicate whether it is json that is output. This will make it easier to maintain. Rather than have to go through two sets of logic now we only have to go through one. - Support write-out %{errormsg} and %{exitcode} in json. - Clarify in the doc that %{exitcode} is the exit code of the transfer. Prior to this change it just said "The numerical exitcode" which implies it's the exit code of the tool, and it's not necessarily that. Closes https://github.com/curl/curl/pull/6544 - lib: drop USE_SOCKETPAIR in favor of CURL_DISABLE_SOCKETPAIR .. since the former is undocumented and they both do the same thing. Closes https://github.com/curl/curl/pull/6517 - curl_multibyte: fall back to local code page stat/access on Windows If libcurl is built with Unicode support for Windows then it is assumed the filename string is Unicode in UTF-8 encoding and it is converted to UTF-16 to be passed to the wide character version of the respective function (eg wstat). However the filename string may actually be in the local encoding so, even if it successfully converted to UTF-16, if it could not be stat/accessed then try again using the local code page version of the function (eg wstat fails try stat). We already do this with fopen (ie wfopen fails try fopen), so I think it makes sense to extend it to stat and access functions. Closes https://github.com/curl/curl/pull/6514 - [Stephan Szabo brought this change] file: Support unicode urls on windows Closes https://github.com/curl/curl/pull/6501 - [Vincent Torri brought this change] cmake: fix import library name for non-MS compiler on Windows - Use _imp.lib suffix only for Microsoft's compiler (MSVC). Prior to this change library suffix _imp.lib was used for the import library on Windows regardless of compiler. With this change the other compilers should now use their default suffix which should be .dll.a. This change is motivated by the usage of pkg-config on MSYS2. Indeed, when 'pkg-config --libs libcurl' is used, -lcurl is passed to ld. The documentation of ld on Windows : https://sourceware.org/binutils/docs/ld/WIN32.html lists, in the 'direct linking to a dll' section, the pattern of the searched import library, and libcurl_imp.lib is not there. Closes https://github.com/curl/curl/pull/6225 Daniel Stenberg (9 Feb 2021) - urldata: move 'followlocation' to UrlState As this is a state variable it does not belong in UserDefined which is used to store values set by the user. Closes #6582 - [Ikko Ashimine brought this change] http_proxy: fix typo in http_proxy.c settting -> setting Closes #6583 - [Fabian Keil brought this change] tests/server: Bump MAX_TAG_LEN to 200 This is useful for tests containing HTML inside of <data> sections. For <img> tags it's not uncommon to be longer than the previous limit of 79 bytes. An example of a previously problem-causing tag is: <img src="http://config.privoxy.org/send-banner?type=auto" border="0" title="Killed-http://www.privoxy.org/images/privoxy.png-by-size" width="88" height="31"> which is needed for a Privoxy test for the banners-by-size filter. Previously it caused server failures like: 12:29:05.786961 ====> Client connect 12:29:05.787116 accept_connection 3 returned 4 12:29:05.787194 accept_connection 3 returned 0 12:29:05.787285 Read 119 bytes 12:29:05.787345 Process 119 bytes request 12:29:05.787407 Got request: GET /banners-by-size/9 HTTP/1.1 12:29:05.787464 Requested test number 9 part 0 12:29:05.787686 getpart() failed with error: -2 12:29:05.787744 - request found to be complete (9) 12:29:05.787912 getpart() failed with error: -2 12:29:05.788048 Wrote request (119 bytes) input to log/server.input 12:29:05.788157 Send response test9 section <data> 12:29:05.788443 getpart() failed with error: -2 12:29:05.788498 instructed to close connection after server-reply 12:29:05.788550 ====> Client disconnect 0 12:29:05.871448 exit_signal_handler: 15 12:29:05.871714 signalled to die 12:29:05.872040 ========> IPv4 sws (port 21108 pid: 51758) exits with signal (15) - [Fabian Keil brought this change] tests/badsymbols.pl: when opening '$incdir' fails include it in the error message - [Fabian Keil brought this change] runtests.1: document -o, -P, -L, and -E - [Fabian Keil brought this change] runtests.pl: add %TESTNUMBER variable to make copying tests more convenient - [Fabian Keil brought this change] runtests.pl: add an -o option to change internal variables runtests.pl has lots of internal variables one might want to change in certain situations, but adding a dedicated option for every single one of them isn't practical. Usage: ./runtests.pl -o TESTDIR=$privoxy_curl_test_dir -o HOSTIP=10.0.0.1 ... - [Fabian Keil brought this change] runtests.pl: cleanups - show the summarized test result in the last line of the report - do not use $_ after mapping it to a named variable Doing that makes the code harder to follow. - log the restraints sorted by the number of their occurrences - fix language when logging restraints that only occured once - let runhttpserver() use $TESTDIR instead of $srcdir ... so it works if a non-default $TESTDIR is being used. - [Fabian Keil brought this change] runtests.pl: add an -E option to specify an exclude file It can contain additional restraints for test numbers, keywords and tools. The idea is to let third parties like the Privoxy project distribute an exclude file with their tarballs that specifies which curl tests are not expected to work when using Privoxy as a proxy, without having to fork the whole curl test suite. The syntax could be changed to be extendable and maybe more closely reflect the "curl test" syntax. Currently it's a bunch of lines like these: test:$TESTNUMBER:Reason why this test with number $TESTNUMBER should be skipped keyword:$KEYWORD:Reason why tests whose keywords contain the $KEYWORD should be skipped tool:$TOOL:Reason why tests with tools that contain $TOOL should be skipped To specify multiple $TESTNUMBERs, $KEYWORDs and $TOOLs on a single line, split them with commas. - [Fabian Keil brought this change] runtests.pl: add -L parameter to require additional perl libraries This is useful to change the behaviour of the script without having to modify the file itself, for example to use a custom compareparts() function that ignores header differences that are expected to occur when an external proxy is being used. Such differences are proxy-specific and thus the modifications should be maintained together with the proxy. - [Fabian Keil brought this change] runtests.pl: add a -P option to specify an external proxy ... that should be used when executing the tests. The assumption is that the proxy is an HTTP proxy. This option should be used together with -L to provide a customized compareparts() version that knows which proxy-specific header differences should be ignored. This option doesn't work for all test types yet. - [Fabian Keil brought this change] tests: fixup several tests missing CRs and modified %hostip lib556/test556: use a real HTTP version to make test reuse more convenient make sure the weekday in Date headers matches the date test61: replace stray "^M" (5e 4d) at the end of a cookie with a '^M' (0d) Gets the test working with external proxies like Privoxy again. Closes #6463 - ftp: never set data->set.ftp_append outside setopt Since the set value then risks getting used like that when the easy handle is reused by the application. Also: renamed the struct field from 'ftp_append' to 'remote_append' since it is also used for SSH protocols. Closes #6579 - urldata: remove the 'rtspversion' field from struct connectdata and the corresponding code in http.c that set it. It was never used for anything! Closes #6581 - CURLOPT_QUOTE.3: clarify that libcurl doesn't parse what's sent ... so passed in commands may confuse libcurl's knowledge of state. Reported-by: Bodo Bergmann Fixes #6577 Closes #6580 - [Jacob Hoffman-Andrews brought this change] vtls: factor out Curl_ssl_getsock to field of Curl_ssl Closes #6558 - RELEASE-PROCEDURE: remove old release dates, add new - docs/SSL-PROBLEMS: enhanced Elaborate on the intermediate cert issue, and mention that anything below TLS 1.2 is generally considered insecure these days. Closes #6572 - THANKS: remove a Jon Rumsey dupe Daniel Gustafsson (5 Feb 2021) - [nimaje brought this change] docs: fix FILE example url in --metalink documentation In a url after <scheme>:// follows the possibly empty authority part till the next /, so that url missed a /. Closes #6573 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Reviewed-by: Daniel Gustafsson <daniel@yesql.se> Daniel Stenberg (5 Feb 2021) - hostip: fix build with sync resolver Reported-by: David Goerger Follow-up from 8335c6417 Fixes #6566 Closes #6568 - mailmap: Jon Rumsey - [Jon Rumsey brought this change] gskit: correct the gskit_send() prototype gskit_send() first paramater is a pointer to Curl_easy not connectdata struct. Closes #6570 Fixes #6569 - urldata: fix build without HTTP and MQTT Reported-by: Joseph Chen Fixes #6562 Closes #6563 - ftp: avoid SIZE when asking for a TYPE A file ... as we ignore it anyway because servers don't report the correct size and proftpd even blatantly returns a 550. Updates a set of tests accordingly. Reported-by: awesomenode on github Fixes #6564 Closes #6565 - pingpong: rename the curl_pp_transfer enum to use PP prefix Using an FTP prefix for PP provided functionality was misleading. - RELEASE-NOTES: synced ... and bump pending version to 7.75.1 (for now) Jay Satiro (4 Feb 2021) - build: fix --disable-http-auth Broken since 215db08 (precedes 7.75.0). Reported-by: Benbuck Nason Fixes https://github.com/curl/curl/issues/6567 - build: fix --disable-dateparse Broken since 215db08 (precedes 7.75.0). Bug: https://curl.se/mail/lib-2021-02/0008.html Reported-by: Firefox OS Daniel Stenberg (4 Feb 2021) - [Jon Rumsey brought this change] OS400: update for CURLOPT_AWS_SIGV4 chkstrings fails because a new string option that could require codepage conversion has been added. Closes #6561 Fixes #6560 - BUG-BOUNTY: removed the cooperation mention Version 7.75.0 (3 Feb 2021) Daniel Stenberg (3 Feb 2021) - RELEASE-NOTES: synced - THANKS: added contributors from 7.75.0 - copyright: fix year ranges in need of updates - TODO: remove items for next SONAME bump etc We want to avoid that completely, so we don't plan for things after such an event. - [Jay Satiro brought this change] ngtcp2: Fix build error due to change in ngtcp2_settings - Separate ngtcp2_transport_params. ngtcp2/ngtcp2@05d7adc made ngtcp2_transport_params separate from ngtcp2_settings. ngtcp2 master is required to build curl with http3 support. Closes #6554 - vtls: remove md5sum As it is not used anymore. Reported-by: Jacob Hoffman-Andrews Bug: https://curl.se/mail/lib-2021-02/0000.html Closes #6557 - [Alessandro Ghedini brought this change] quiche: don't use primary_ip / primary_port Closes #6555 Alessandro Ghedini (1 Feb 2021) - travis: enable quiche's FFI feature Daniel Stenberg (30 Jan 2021) - [Dmitry Wagin brought this change] http: improve AWS HTTP v4 Signature auth - Add support services without region and service prefixes in the URL endpoint (ex. Min.IO, GCP, Yandex Cloud, Mail.Ru Cloud Solutions, etc) by providing region and service parameters via aws-sigv4 option. - Add [:region[:service]] suffix to aws-sigv4 option; - Fix memory allocation errors. - Refactor memory management. - Use Curl_http_method instead() STRING_CUSTOMREQUEST. - Refactor canonical headers generating. - Remove repeated sha256_to_hex() usage. - Add some docs fixes. - Add some codestyle fixes. - Add overloaded strndup() for debug - curl_dbg_strndup(). - Update tests. Closes #6524 - hyper: fix CONNECT to set 'data' as userdata Follow-up to 14e075d1a7fd - [Layla brought this change] connect: fix compile errors in `Curl_conninfo_local` .. for the `#else` (`!HAVE_GETSOCKNAME`) case Fixes https://github.com/curl/curl/issues/6548 Closes #6549 Signed-off-by: Layla <layla@insightfulvr.com> - [Michał Antoniak brought this change] transfer: fix GCC 10 warning with flag '-Wint-in-bool-context' ... and return the error code from the Curl_mime_rewind call. Closes #6537 - [Michał Antoniak brought this change] avoid warning: enum constant in boolean context - copyright: fix missing year (range) updates - RELEASE-NOTES: synced - openssl: lowercase the hostname before using it for SNI ... because it turns out several servers out there don't actually behave correctly otherwise in spite of the fact that the SNI field is specifically said to be case insensitive in RFC 6066 section 3. Reported-by: David Earl Fixes #6540 Closes #6543 - KNOWN_BUGS: cmake: ExternalProject_Add does not set CURL_CA_PATH Closes #6313 - KNOWN_BUGS: Multi perform hangs waiting for threaded resolver Closes #4852 - KNOWN_BUGS: "pulseUI VPN client" is known to be buggy First entry in the new section "applications" for known problems in libcurl using applications. Closes #6306 - tool_writeout: make %{errormsg} blank for no errors Closes #6539 Jay Satiro (27 Jan 2021) - [Gisle Vanem brought this change] build: fix djgpp builds - Update build instructions in packages/DOS/README - Extend 'VPATH' with 'vquic' and 'vssh'. - Allow 'Makefile.dist' to build both 'lib' and 'src'. - Allow using the Windows hosted djgpp cross compiler to build for MSDOS under Windows. - 'USE_SSL' -> 'USE_OPENSSL' - Added a 'link_EXE' macro. Etc, etc. - Linking 'curl.exe' needs '$(CURLX_CFILES)' too. - Do not pick-up '../lib/djgpp/*.o' files. Recompile locally. - Generate a gzipped 'tool_hugehelp.c' if 'USE_ZLIB=1'. - Remove 'djgpp-clean' - Adapt to new C-ares directory structure - Use conditional variable assignments Clarify the 'conditional variable assignment' in 'common.dj'. Closes https://github.com/curl/curl/pull/6382 Daniel Stenberg (27 Jan 2021) - [Ikko Ashimine brought this change] hyper: fix typo in c-hyper.c settting -> setting Closes #6538 - libssh2: fix CURL_LIBSSH2_DEBUG-enabled build Follow-up to 2dcc940959772a Reported-by: Gisle Vanem Bug: https://github.com/curl/curl/commit/2dcc940959772a652f6813fb6bd3092095a4877b#commitcomment-46420088 Jay Satiro (27 Jan 2021) - asyn-thread: fix build for when getaddrinfo missing This is a follow-up to 8315343 which several days ago moved the resolver pointer into the async struct but did not update the code that uses it when getaddrinfo is not present. Closes https://github.com/curl/curl/pull/6536 Daniel Stenberg (27 Jan 2021) - urldata: move 'ints' to the end of 'connectdata' To optimize storage slightly. Closes #6534 - urldata: store ip version in a single byte Closes #6534 - urldata: remove duplicate 'upkeep_interval_ms' from connectdata ... and rely only on the value already set in Curl_easy. Closes #6534 - urldata: remove 'local_ip' from the connectdata struct As the info is already stored in the transfer handle anyway, there's no need to carry around a duplicate buffer for the life-time of the handle. Closes #6534 - urldata: remove duplicate port number storage ... and use 'int' for ports. We don't use 'unsigned short' since -1 is still often used internally to signify "unknown value" and 0 - 65535 are all valid port numbers. Closes #6534 - urldata: remove the duplicate 'ip_addr_str' field ... as the numerical IP address is already stored and kept in 'primary_ip'. Closes #6534 - select: convert Curl_select() to private static function The old function should not be used anywhere anymore (the only remaining gskit use has to be fixed to instead use Curl_poll or none at all). The static function version is now called our_select() and is only built if necessary. Closes #6531 - Curl_chunker: shrink the struct ... by removing a field, converting the hex index into a byte and rearranging the order. Cuts it down from 48 bytes to 32 on x86_64. Closes #6527 - curl: include the file name in --xattr/--remote-time error msgs - curl: s/config->global/global/ in single_transfer() - curl: move fprintf outputs to warnf For setting and getting time of the download. To make the outputs respect --silent etc. Reported-by: Viktor Szakats Fixes #6533 Closes #6535 - [Tatsuhiro Tsujikawa brought this change] ngtcp2: Fix http3 upload stall Closes #6521 - [Tatsuhiro Tsujikawa brought this change] ngtcp2: Fix stack buffer overflow Closes #6521 - warnless.h: remove the prototype for curlx_ultosi Follow-up to 217552503ff3 - warnless: remove curlx_ultosi ... not used anywhere Closes #6530 - [Patrick Monnerat brought this change] lib: remove conn->data uses Closes #6515 - pingpong: remove the 'conn' struct member ... as it's superfluous now when Curl_easy is passed in and we can derive the connection from that instead and avoid the duplicate copy. Closes #6525 - hostip/proxy: remove conn->data use Closes #6513 - url: reduce conn->data references ... there are a few left but let's keep them to last Closes #6512 - scripts/singleuse: add curl_easy_option* Jay Satiro (25 Jan 2021) - test410: fix for windows - Pass the very long request header via file instead of command line. Prior to this change the 49k very long request header string was passed via command line and on Windows that is too long so it was truncated and the test would fail (specifically msys CI). Closes https://github.com/curl/curl/pull/6516 Daniel Stenberg (25 Jan 2021) - libssh2: move data from connection object to transfer object Readdir data, filenames and attributes are strictly related to the transfer and not the connection. This also reduces the total size of the fixed connectdata struct. Closes #6519 - RELEASE-NOTES: synced - [Patrick Monnerat brought this change] lib: remove conn->data uses Closes #6499 - hyper: remove the conn->data references Closes #6508 - travis: build ngtcp2 --with-gnutls ... since they disable it by default since a few days back. Closes #6506 Fixes #6493 - hostip: remove conn->data from resolver functions This also moves the 'async' struct from the connectdata struct into the Curl_easy struct, which seems like a better home for it. Closes #6497 Jay Satiro (22 Jan 2021) - strerror: skip errnum >= 0 assertion on windows On Windows an error number may be greater than INT_MAX and negative once cast to int. The assertion is checked only in debug builds. Closes https://github.com/curl/curl/pull/6504 Daniel Stenberg (21 Jan 2021) - doh: make Curl_doh_is_resolved survive a NULL pointer ... if Curl_doh() returned a NULL, this function gets called anyway as in a asynch procedure. Then the doh struct pointer is NULL and signifies an OOM situation. Follow-up to 6246a1d8c6776 - wolfssh: remove conn->data references ... and repair recent build breakage Closes #6507 - http: empty reply connection are not left intact ... so mark the connection as closed in this condition to prevent that verbose message to wrongly appear. Reported-by: Matt Holt Bug: https://twitter.com/mholt6/status/1352130240265375744 Closes #6503 - chunk/encoding: remove conn->data references ... by anchoring more functions on Curl_easy instead of connectdata Closes #6498 Jay Satiro (20 Jan 2021) - [Erik Olsson brought this change] lib: save a bit of space with some structure packing - Reorder some internal struct members so that less padding is used. This is an attempt at saving a bit of space by packing some structs (using pahole to find the holes) where it might make sense to do so without losing readability. I.e., I tried to avoid separating fields that seem grouped together (like the cwd... fields in struct ftp_conn for instance). Also abstained from touching fields behind conditional macros as that quickly can get complicated. Closes https://github.com/curl/curl/pull/6483 Daniel Stenberg (20 Jan 2021) - INSTALL.md: fix typo Found-by: Marcel Raad - [Fabian Keil brought this change] http: get CURLOPT_REQUEST_TARGET working with a HTTP proxy Added test 1613 to verify. Closes #6490 - Merge branch 'bagder/curl_range-data-conn' - ftp: remove conn->data leftover - curl_range: remove conn->data Closes #6496 - INSTALL: now at 85 operating systems - quiche: fix unused parameter ‘conn’ Follow-up to 2bdec0b3 - transfer: fix ‘conn’ undeclared mistake for iconv build Follow-up to 219d9f8620d - doh: allocate state struct on demand ... instead of having it static within the Curl_easy struct. This takes away 1176 bytes (18%) from the Curl_easy struct that aren't used very often and instead makes the code allocate it when needed. Closes #6492 - socks: use the download buffer instead The SOCKS code now uses the generic download buffer for temporary storage during the connection procedure, instead of having its own private 600 byte buffer that adds to the connectdata struct size. This works fine because this point the buffer is allocated but is not use for download yet since the connection hasn't completed. This reduces the connection struct size by 22% on a 64bit arch! The SOCKS buffer needs to be at least 600 bytes, and the download buffer is guaranteed to never be smaller than 1000 bytes. Closes #6491 - urldata: make magic be the first struct field By making the `magic` identifier the same size and at the same place within the structs (easy, multi, share), libcurl will be able to more reliably detect and safely error out if an application passes in the wrong handle to APIs. Easier to detect and less likely to cause crashes if done. Such mixups can't be detected at compile-time due to them being typedefed void pointers - unless `CURL_STRICTER` is defined. Closes #6484 - http_chunks: correct and clarify a comment on hexnumber length ... and also rename the define for max length. Closes #6489 - curl_path: remove conn->data use Closes #6487 - transfer: remove conn->data use Closes #6486 - quic: remove conn->data use Closes #6485 - [Fabian Keil brought this change] Add test1181: Proxy request with --proxy-header "Connection: Keep-Alive" - [Fabian Keil brought this change] Add test1180: Proxy request with -H "Proxy-Connection: Keep-Alive" At the moment the test fails as curl sends two Proxy-Connection headers. - c-hyper: avoid duplicated Proxy-Connection headers - http: make providing Proxy-Connection header not cause duplicated headers Fixes test 1180 Bug: https://curl.se/mail/lib-2021-01/0095.html Reported-by: Fabian Keil Closes #6472 - runtests: preprocess DISABLED to allow conditionals ... with this function provided, we can disable tests for specific environments and setups directly within this file. Closes #6477 - runtests: turn preprocessing into a separate function ... and remove all other variable substitutions as they're now done once and for all in the preprocessor. - lib/Makefile.inc: convert to listing each file on its own line ... to make it diff friendlier and easier to read. Closes #6448 - ftplistparser: remove use of conn->data Closes #6482 - lib: more conn->data cleanups Closes #6479 - [Patrick Monnerat brought this change] vtls: reduce conn->data use Closes #6474 - hyper: deliver data to application with Curl_client_write ... just as the native code path does. Avoids sending too large data chunks in the callback and more. Reported-by: Gisle Vanem Fixes #6462 Closes #6473 - gopher: remove accidental conn->data leftover - libssh: avoid plain free() of libssh-memory Since curl's own memory debugging system redefines free() calls to track and fiddle with memory, it cannot be used on memory allocated by 3rd party libraries. Third party libraries SHOULD NOT require free() to release allocated resources for this reason - and libs can use separate healp allocators on some systems (like Windows) so free() doesn't necessarily work anyway. Filed as an issue with libssh: https://bugs.libssh.org/T268 Closes #6481 - send: assert that Curl_write_plain() has a ->conn when called To help catch bad invokes. Closes #6476 - test410: verify HTTPS GET with a 49K request header skip test 410 for mesalink in the CI as it otherwise hangs "forever" - lib: pass in 'struct Curl_easy *' to most functions ... in most cases instead of 'struct connectdata *' but in some cases in addition to. - We mostly operate on transfers and not connections. - We need the transfer handle to log, store data and more. Everything in libcurl is driven by a transfer (the CURL * in the public API). - This work clarifies and separates the transfers from the connections better. - We should avoid "conn->data". Since individual connections can be used by many transfers when multiplexing, making sure that conn->data points to the current and correct transfer at all times is difficult and has been notoriously error-prone over the years. The goal is to ultimately remove the conn->data pointer for this reason. Closes #6425 Emil Engler (17 Jan 2021) - docs: fix typos in NEW-PROTOCOL.md This fixes a misspelled "it" and a grammatically wrong "-ing" suffix. Closes #6471 Daniel Stenberg (16 Jan 2021) - RELEASE-NOTES: synced Jay Satiro (16 Jan 2021) - [Razvan Cojocaru brought this change] cmake: expose CURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG This does for cmake builds what --disable-openssl-auto-load-config does for autoconf builds. Closes https://github.com/curl/curl/pull/6435 Daniel Stenberg (15 Jan 2021) - test1918: verify curl_easy_option_by_name() and curl_easy_option_by_id() ... and as a practical side-effect, make sure that the Curl_easyopts_check() function is asserted in debug builds, which we want to detect mismatches between the options list in easyoptions.c and the options in curl.h Found-by: Gisle Vanem Bug: https://github.com/curl/curl/commit/08e8455dddc5e48e58a12ade3815c01ae3da3b64#commitcomment-45991815 Closes #6461 - [Gisle Vanem brought this change] easyoptions: add the missing AWS_SIGV4 Follow-up from AWS_SIGV4 - schannel_verify: fix safefree call typo Follow-up from e87ad71d1ba00519 Closes #6459 - mime: make sure setting MIMEPOST to NULL resets properly ... so that a function can first use MIMEPOST and then set it to NULL to reset it back to a blank POST. Added test 584 to verify the fix. Reported-by: Christoph M. Becker Fixes #6455 Closes #6456 - multi: set the PRETRANSFER time-stamp when we switch to PERFORM ... instead of at end of the DO state. This makes the timer more accurate for the protocols that use the DOING state (such as FTP), and simplifies how the function (now called init_perform) is called. The timer will then include the entire procedure up to PERFORM - including all instructions for getting the transfer started. Closes #6454 - CURLINFO_PRETRANSFER_TIME.3: clarify ... the timer *does* include the instructions for getting the remote file. Ref: #6452 Closes #6453 - [Gisle Vanem brought this change] schannel: plug a memory-leak ... when built without -DUNICODE. Closes #6457 Jay Satiro (14 Jan 2021) - gitattributes: Set batch files to CRLF line endings on checkout If a batch file is run without CRLF line endings (ie LF-only) then arbitrary behavior may occur. I consider that a bug in Windows, however the effects can be serious enough (eg unintended code executed) that we're fixing it in the repo by requiring CRLF line endings for batch files on checkout. Prior to this change the checked-out line endings of batch files were dependent on a user's git preferences. On Windows it is common for git users to have automatic CRLF conversion enabled (core.autocrlf true), but those users that don't would run into this behavior. For example a user has reported running the Visual Studio project generator batch file (projects/generate.bat) and it looped forever. Output showed that the Windows OS interpreter was occasionally jumping to arbitrary points in the batch file and executing commands. This resulted in unintended files being removed (a removal sequence called) and looping forever. Ref: https://serverfault.com/q/429594 Ref: https://stackoverflow.com/q/232651 Ref: https://www.dostips.com/forum/viewtopic.php?t=8988 Ref: https://git-scm.com/docs/gitattributes#_checking_out_and_checking_in Ref: https://git-scm.com/book/en/v2/Customizing-Git-Git-Configuration#_core_autocrlf Bug: https://github.com/curl/curl/discussions/6427 Reported-by: Ganesh Kamath Closes https://github.com/curl/curl/pull/6442 Daniel Stenberg (14 Jan 2021) - tool_operate: spellfix a comment - ROADMAP: refreshed o removed HSTS - already implemented o added HTTPS RR records o mention HTTP/3 completion - http_chunks: remove Curl_ prefix from static functions - transfer: remove Curl_ prefix from static functions - tftp: remove Curl_ prefix from static functions - multi: remove Curl_ prefix from static functions - ldap: remove Curl_ prefix from static functions - doh: remove Curl_ prefix from static functions - asyn-ares: remove Curl_ prefix from static functions - vtls: remove Curl_ prefix from static functions - bearssl: remove Curl_ prefix from static functions - mbedtls: remove Curl_ prefix from static functions - wolfssl: remove Curl_ prefix from static functions - nss: remove Curl_ prefix from static functions - gnutls: remove Curl_ prefix from static functions - openssl: remove Curl_ prefix from static functions ... as we reserve this prefix to library-wide functions. Closes #6443 - nss: get the run-time version instead of build-time Closes #6445 Jay Satiro (12 Jan 2021) - tool_doswin: Restore original console settings on CTRL signal - Move Windows terminal init code from tool_main to tool_doswin. - Restore the original console settings on CTRL+C and CTRL+BREAK. Background: On Windows the curl tool changes the console settings to enable virtual terminal processing (eg color output) if supported (ie Win 10). The original settings are restored on exit but prior to this change were not restored in the case of the CTRL signals. Windows VT behavior varies depending on console/powershell/terminal; refer to the discussion in #6226. Assisted-by: Rich Turner Closes https://github.com/curl/curl/pull/6226 Daniel Stenberg (12 Jan 2021) - gen.pl: fix perl syntax Follow-up to 324cf1d2e - [Emil Engler brought this change] help: update to current codebase This commit bumps the help to the current state of the project. Closes #6437 - [Emil Engler brought this change] docs: fix line length bug in gen.pl The script warns if the length of $opt and $desc is > 78. However, these two variables are on totally separate lines so the check makes no sense. Also the $bitmask field is totally forgotten. Currently this leads to two warnings within `--resolve` and `--aws-sigv4`. Closes #6438 - [Emil Engler brought this change] docs: fix wrong documentation in help.d curl does not list all categories when you invoke "--help" without any parameters. Closes #6436 - aws-sigv4.d: polish the wording Make it shorter and imperative form Closes #6439 - [Fabian Keil brought this change] misc: fix typos Bug: https://curl.se/mail/lib-2021-01/0063.html Closes #6434 - multi_runsingle: bail out early on data->conn == NULL As that's a significant error condition and scan-build warns for NULL pointer dereferences if we don't. Closes #6433 - multi: skip DONE state if there's no connection left for ftp wildcard ... to avoid running in that state with data->conn being NULL. - libssh2: fix "Value stored to 'readdir_len' is never read" Detected by scan-build - connect: mark intentional ignores of setsockopt return values Pointed out by Coverity Closes #6431 Jay Satiro (11 Jan 2021) - http_proxy: Fix CONNECT chunked encoding race condition - During the end-of-headers response phase do not mark the tunnel complete unless the response body was completely parsed/ignored. Prior to this change if the entirety of a CONNECT response with chunked encoding was not received by the time the final header was parsed then the connection would be marked done prematurely, before all the chunked data could be read in and ignored (since this is what we do with any CONNECT response body) and the connection could not be used. Bug: https://curl.se/mail/lib-2021-01/0033.html Reported-by: Fabian Keil Closes https://github.com/curl/curl/pull/6432 Daniel Stenberg (11 Jan 2021) - RELEASE-NOTES: synced - url: if IDNA conversion fails, fallback to Transitional This improves IDNA2003 compatiblity. Reported-by: Bubu on github Fixes #6423 Closes #6428 - travis: make the Hyper build from its master branch Closes #6430 - http: make 'authneg' also work for Hyper When doing a request with a request body expecting a 401/407 back, that initial request is sent with a zero content-length. Test 177 and more. Closes #6424 Jay Satiro (8 Jan 2021) - cmake: Add an option to disable libidn2 New option USE_LIBIDN2 defaults to ON for libidn2 detection. Prior to this change libidn2 detection could not be turned off in cmake builds. Reported-by: William A Rowe Jr Fixes https://github.com/curl/curl/issues/6361 Closes https://github.com/curl/curl/pull/6362 Daniel Stenberg (8 Jan 2021) - HYPER: no longer needs the special branch - test179: use consistent header line endings ... to make "Hyper mode" work better. - file: don't provide content-length for directories ... as it is misleading. Ref #6379 Closes #6421 - TODO: Directory listing for FILE: Ref #6379 - curl.h: add CURLPROTO_GOPHERS as own protocol identifier Follow-up to a1f06f32b860, to make sure it can be handled separately from plain gopher. Closes #6418 - http: have CURLOPT_FAILONERROR fail after all headers ... so that Retry-After and other meta-content can still be used. Added 1634 to verify. Adjusted test 194 and 281 since --fail now also includes the header-terminating CRLF in the output before it exits. Fixes #6408 Closes #6409 - global_init: debug builds allocates a byte in init ... to make build tools/valgrind warn if no curl_global_cleanup is called. This is conditionally only done for debug builds with the env variable CURL_GLOBAL_INIT set. Closes #6410 - lib/unit tests: add missing curl_global_cleanup() calls - travis: adapt to Hyper build change Closes #6419 - pretransfer: setup the User-Agent header here ... and not in the connection setup, as for multiplexed transfers the connection setup might be skipped and then the transfer would end up without the set user-agent! Reported-by: Flameborn on github Assisted-by: Andrey Gursky Assisted-by: Jay Satiro Assisted-by: Mike Gelfand Fixes #6312 Closes #6417 - test66: disable with Hyper ...as Hyper doesn't support HTTP/0.9 - c-hyper: poll the tasks until end correctly ... makes test 36 work. Closes #6412 - [Gergely Nagy brought this change] mk-ca-bundle.pl: deterministic output when using -t Printing trust purposes are now sorted, making the output deterministic when running on the same input certdata.txt. Closes #6413 - KNOWN_BUGS: fixed "wolfSSL lacks support for renegotiation" Fixed by #6411 - [Himanshu Gupta brought this change] wolfssl: add SECURE_RENEGOTIATION support Closes #6411 - RELEASE-NOTES: synced - wolfssl: update copyright year range Follow-up to 7de2e96535e9 - c-hyper: make CURLE_GOT_NOTHING work Test 30 Closes #6407 - http_proxy: make CONNECT work with the Hyper backend Makes test 80 run Closes #6406 - TODO: --fail-with-body perchance? Jay Satiro (4 Jan 2021) - tool_operate: fix the suppression logic of some error messages - Fix the failed truncation and failed writing body error messages to not be shown unless error messages are shown. (ie the user has specified -sS, or has not specified -s). - Also prefix same error messages with "curl: ", for example: curl: (23) Failed to truncate, exiting Prior to this change the failed truncation error messages would be shown if not -s, but did not account for -sS which should show. Prior to this change the failed writing body error messages would be shown always. Ref: https://curl.se/docs/manpage.html#-S Bug: https://curl.se/mail/archive-2020-12/0017.html Reported-by: Hongyi Zhao Closes https://github.com/curl/curl/pull/6402 - wolfssl: Support wolfSSL builds missing TLS 1.1 The wolfSSL TLS library defines NO_OLD_TLS in some of their build configurations and that causes the library to be built without TLS 1.1. For example if MD5 is explicitly disabled when building wolfSSL then that defines NO_OLD_TLS and the library is built without TLS 1.1 [1]. Prior to this change attempting to build curl with a wolfSSL that was built with NO_OLD_TLS would cause a build link error undefined reference to wolfTLSv1_client_method. [1]: https://github.com/wolfSSL/wolfssl/blob/v4.5.0-stable/configure.ac#L2366 Bug: https://curl.se/mail/lib-2020-12/0121.html Reported-by: Julian Montes Closes https://github.com/curl/curl/pull/6388 Daniel Stenberg (4 Jan 2021) - test1633: set appropriate name "--retry with a 429 response and Retry-After:" - travis: limit the tests with quiche builds to HTTPS and FTPS only ... since it runs into the 50 minute time limit too often otherwise. Closes #6403 - HISTORY: added dates to early history Mostly thanks to this archived web page for urlget: https://web.archive.org/web/19980216125115/http://www.inf.ufrgs.br/~sagula/urlget.html - httpauth: make multi-request auth work with custom port When doing HTTP authentication and a port number set with CURLOPT_PORT, the code would previously have the URL's port number override as if it had been a redirect to an absolute URL. Added test 1568 to verify. Reported-by: UrsusArctos on github Fixes #6397 Closes #6400 - [Emil Engler brought this change] language: s/behaviour/behavior/g We currently use both spellings the british "behaviour" and the american "behavior". However "behavior" is more used in the project so I think it's worth dropping the british name. Closes #6395 - cmdline-opts/retry.d: mention response code 429 as well Reported-by: Cherish98 Bug: https://curl.se/mail/archive-2020-12/0018.html - docs/HYPER.md: mention outstanding issues To make it more obvious to users what doesn't work (yet) Closes #6389 - COPYING/configure: bump copyright year range - c-hyper: add timecondition to the request Test 77-78 Closes #6391 - c-hyper: make Digest and NTLM work Test 64, 65, 67, 68, 69, 70, 72 Closes #6390 - examples/curlgtk.c: fix the copyright year range ... and make private functions static. - [Olaf Hering brought this change] docs/examples: adjust prototypes for CURLOPT_READFUNCTION The type of the buffer in curl_read_callback is 'char *', not 'void *'. Signed-off-by: Olaf Hering <olaf@aepfle.de> Closes #6392 - examples: fix more empty expression statement has no effect Follow-up to 26e46617b9 - cleanup: fix two empty expression statement has no effect Follow-up to 26e46617b9 - configure: set -Wextra-semi-stmt for clang with --enable-debug To have it properly complain on empty statements with no effect. Ref: #6376 Closes #6378 - tests/unit: fix empty statements with no effect ... by making macros use "do {} while(0)" - [Paul Groke brought this change] dns: extend CURLOPT_RESOLVE syntax for adding non-permanent entries Extend the syntax of CURLOPT_RESOLVE strings: allow using a '+' prefix (similar to the existing '-' prefix for removing entries) to add DNS cache entries that will time out just like entries that are added by libcurl itself. Append " (non-permanent)" to info log message in case a non-permanent entry is added. Adjust relevant comments to reflect the new behavior. Adjust documentation. Extend unit1607 to test the new functionality. Closes #6294 - schannel: fix "empty expression statement has no effect" Bug: https://github.com/curl/curl/commit/8ab78f720ae478d533e30b202baec4b451741579#commitcomment-45445950 Reported-by: Gisle Vanem Closes #6381 - [Denis Laxalde brought this change] docs: remove redundant "better" in --fail help Closes #6385 - [Kevin Ushey brought this change] curl.1: fix typo microsft -> microsoft Closes #6380 - [XhmikosR brought this change] misc: assorted typo fixes Closes #6375 - RELEASE-NOTES: synced - tool_operate: avoid NULL dereference of first_arg Follow-up to 6a5e020d4d2b04a Identified by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28999 Closes #6377 - misc: fix "warning: empty expression statement has no effect" Turned several macros into do-while(0) style to allow their use to work find with semicolon. Bug: https://github.com/curl/curl/commit/08e8455dddc5e48e58a12ade3815c01ae3da3b64#commitcomment-45433279 Follow-up to 08e8455dddc5e4 Reported-by: Gisle Vanem Closes #6376 - KNOWN_BUGS: 6.10 curl never completes Negotiate over HTTP Closes #5235 Closes #6370 - writeout: fix NULL dereference for "this url" Detected by torture test 1029 Follow-up to 7a90ddf88f5a Closes #6374 - failf: remove newline from formatting strings ... as failf adds one itself. Also: add an assert() to failf() that triggers on a newline in the format string! Closes #6365 - [XhmikosR brought this change] CI: fix warning with the latest versions `git checkout HEAD^2` is no longer needed Closes #6369 - INSTALL: update the list known OSes and CPU archs curl has run on Closes #6366 - [Cherish98 brought this change] curl: fix handling of -q option The match of the "-q" option (short for "--disable") should: a) allow concatenation with other single-letters; and b) be case-sensitive, lest confusing with "-Q" ("--quote") Closes #6364 - tests/badsymbols.pl: ignore stand-alone single hash lines Bug: https://curl.se/mail/lib-2020-12/0084.html Reported-by: Dennis Clarke Assisted-by: Jay Satiro Closes #6355 - curl_easy_pause.3: add multiplexed pause effects and generally refresh and update. Remove details for ancient versions. Reviewed-by: Jay Satiro Closes #6360 Jay Satiro (22 Dec 2020) - curl_easy_pause.3: fix man page reference Follow-up to ac9a724 from earlier today. Ref: https://github.com/curl/curl/pull/6359 Daniel Stenberg (22 Dec 2020) - EXPERIMENTAL: add the Hyper backend to the list ... of current experimental features in curl. - speedcheck: exclude paused transfers Paused transfers should not be stopped due to slow speed even when CURLOPT_LOW_SPEED_LIMIT is set. Additionally, the slow speed timer is now reset when the transfer is unpaused - as otherwise it would easily just trigger immediately after unpausing. Reported-by: Harry Sintonen Fixes #6358 Closes #6359 - h2: do not wait for RECV on paused transfers ... as the socket might be readable all the time when paused and thus causing a busy-loop. Reported-by: Harry Sintonen Reviewed-by: Jay Satiro Fixes #6356 Closes #6357 - RELEASE-NOTES: synced - cmdline-opts/gen.pl: return hard on errors ... as the warnings tend to go unnoticed otherwise! Closes #6354 - examples/libtest: add .checksrc to dist ... so that (auto)builds from tarballs also get the correct instructions. Fixes #6176 Closes #6353 - test: verify new --write-out variables Extended test 1029 and added 1188 - test970: adapted to the new internal order of variables - curl: add variables to --write-out In particular, these ones can help a user to create its own error message when one or transfers fail. writeout: add 'onerror', 'url', 'urlnum', 'exitcode', 'errormsg' onerror - lets a user only show the rest on non-zero exit codes url - the input URL used for this transfer urlnum - the numerical URL counter (0 indexed) for this transfer exitcode - the numerical exit code for the transfer errormsg - obvious Reported-by: Earnestly on github Fixes #6199 Closes #6207 - [Matthias Gatto brought this change] tests: add very simple AWS HTTP v4 Signature test Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com> - [Matthias Gatto brought this change] docs: add AWS HTTP v4 Signature - [Matthias Gatto brought this change] tool: add AWS HTTP v4 Signature support Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com> - [Matthias Gatto brought this change] http: Make the call to v4 signature This patch allow to call the v4 signature introduce in previous commit Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com> - [Matthias Gatto brought this change] http: introduce AWS HTTP v4 Signature It is a security process for HTTP. It doesn't seems to be standard, but it is used by some cloud providers. Aws: https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html Outscale: https://wiki.outscale.net/display/EN/Creating+a+Canonical+Request GCP (I didn't test that this code work with GCP though): https://cloud.google.com/storage/docs/access-control/signing-urls-manually most of the code is in lib/http_v4_signature.c Information require by the algorithm: - The URL - Current time - some prefix that are append to some of the signature parameters. The data extracted from the URL are: the URI, the region, the host and the API type example: https://api.eu-west-2.outscale.com/api/latest/ReadNets ~~~ ~~~~~~~~ ~~~~~~~~~~~~~~~~~~~ ^ ^ ^ / \ URI API type region Small description of the algorithm: - make canonical header using content type, the host, and the date - hash the post data - make canonical_request using custom request, the URI, the get data, the canonical header, the signed header and post data hash - hash canonical_request - make str_to_sign using one of the prefix pass in parameter, the date, the credential scope and the canonical_request hash - compute hmac from date, using secret key as key. - compute hmac from region, using above hmac as key - compute hmac from api_type, using above hmac as key - compute hmac from request_type, using above hmac as key - compute hmac from str_to_sign using above hmac as key - create Authorization header using above hmac, prefix pass in parameter, the date, and above hash Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com> Closes #5703 - [Matthias Gatto brought this change] http: add hmac support for sha256 It seems current hmac implementation use md5 for the hash, V4 signature require sha256, so I've added the needed struct in this commit. I've added the functions that do the hmac in v4 signature file as a static function ,in the next patch of the serie, because it's used only by this file. Signed-off-by: Matthias Gatto <matthias.gatto@outscale.com> - [Cristian Rodríguez brought this change] connect: on linux, enable reporting of all ICMP errors on UDP sockets The linux kernel does not report all ICMP errors back to userspace due to historical reasons. IP*_RECVERR sockopt must be turned on to have the correct behaviour which is to pass all ICMP errors to userspace. See https://bugzilla.kernel.org/show_bug.cgi?id=202355 Closes #6341 - curl: add --create-file-mode [mode] This option sets the (octal) mode to use for the remote file when one is created, using the SFTP, SCP or FILE protocols. When not set, the default is 0644. Closes #6244 - c-hyper: fix compiler warnings Identified by clang on windows. Reported-by: Gisle Vanem Bug: 58974d25d8173aec154e593ed9d866da566c9811 Closes #6351 - KNOWN_BUGS: Remote recursive folder creation with SFTP Closes #5204 Jay Satiro (20 Dec 2020) - badsymbols.pl: Add verbose mode -v Use -v as the first option to enable verbose mode which will show source input, extracted symbol and line info. For example: Source: ./../include/curl/typecheck-gcc.h Symbol: curlcheck_socket_info(info) Line #423: #define curlcheck_socket_info(info) \ Ref: https://curl.se/mail/lib-2020-12/0084.html Closes https://github.com/curl/curl/pull/6349 - KNOWN_BUGS: Secure Transport disabling hostname validation also disables SNI That behavior is a limitation of Apple's Secure Transport. Reported-by: Cory Benfield Reported-by: Ian Spence Confirmed-by: Nick Zitzmann Ref: https://github.com/curl/curl/issues/998 Closes https://github.com/curl/curl/issues/6347 Closes https://github.com/curl/curl/pull/6348 Daniel Stenberg (18 Dec 2020) - TODO: alt-svc should fallback if alt-svc doesn't work Closes #4908 - travis: restrict the openssl3 job to only run https and ftps tests ... as it runs too long otherwise and the other tests are verified in other builds anyway. Closes #6345 - build: repair http disabled but mqtt enabled build ... as the mqtt code reuses the "method" originally used for HTTP. Closes #6344 - [Jon Wilkes brought this change] cookie: avoid the C1001 internal compiler error with MSVC 14 Fixes #6112 Closes #6135 - RELEASE-NOTES: synced - mqtt: handle POST/PUBLISH without a set POSTFIELDSIZE Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28735 Added test 1916 and 1917 to verify. Closes #6338 - travis: add CI job for Hyper build - tests: updated tests for Hyper - lib: introduce c-hyper for using Hyper ... as an alternative HTTP backend within libcurl. - tool_setopt: provide helper output in debug builds ... for when setopt() returns error. - setopt: adjust to Hyper and disabled HTTP builds - rtsp: disable if Hyper is used - getinfo: build with disabled HTTP support - version: include hyper version - docs: add HYPER.md - configure: add --with-hyper As the first (optional) HTTP backend alternative instead of native Close #6110 - test1522: add debug tracing I used this to track down some issues and I figured I could just as well keep this extra logging in here for future needs. Closes #6331 - http: show the request as headers even when split-sending When the initial request isn't possible to send in its entirety, the remainder of request would be delivered to the debug callback as data and would wrongly be counted internally as body-bytes sent. Extended test 1295 to verify. Closes #6328 - multi: when erroring in TOOFAST state, act as for PERFORM When failing in TOOFAST, the multi_done() wasn't called so the same cleanup and handling wasn't done like when it fails in PERFORM, which in the case of FTP could mean that the control connection wouldn't be marked as "dead" for the CURLE_ABORTED_BY_CALLBACK case. Which caused ftp_disconnect() to use it to send "QUIT", which could end up waiting for a response a long time before giving up! Reported-by: Tomas Berger Fixes #6333 Closes #6337 - cmake: enable gophers correctly in curl-config Closes #6336 - test1198/9: add two mqtt publish tests without payload lengths Closes #6335 - tests/mqttd: extract the client id from the correct offset Closes #6334 - TODO: Prevent terminal injection when writing to terminal Closes #6150 - Revert "CI/github: work-around for brew breakage on macOS" This reverts commit 4cbb17a2cbbbe6337142d39479e21c3990b9c22f. ... as the work-around now causes failures. Closes #6332 - examples: remove superfluous asterisk uses ... for function pointers. Breaks in ancient compilers. - RELEASE-NOTES: synced - test1272: fix line ending Follow-up to f24784f9143 - URL-SYNTAX: add gophers details - test1272: test gophers - runtests: add support for gophers, gopher over TLS - [parazyd brought this change] gopher: Implement secure gopher protocol. This commit introduces a "gophers" handler inside the gopher protocol if USE_SSL is defined. This protocol is no different than the usual gopher prococol, with the added TLS encapsulation upon connecting. The protocol has been adopted in the gopher community, and many people have enabled TLS in their gopher daemons like geomyidae(8), and clients, like clic(1) and hurl(1). I have not implemented test units for this protocol because my knowledge of Perl is sub-par. However, for someone more knowledgeable it might be fairly trivial, because the same test that tests the plain gopher protocol can be used for "gophers" just by adding a TLS listener. Signed-off-by: parazyd <parazyd@dyne.org> Closes #6208 - TODO: Package curl for Windows in a signed installer Closes #5424 - mqtt: deal with 0 byte reads correctly OSS-Fuzz found it Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=28676 Closes #6327 - BUG-BOUNTY: minor language update ... and remove the wording about entries from before 2019 as the "within 12 months" is still there and covers that. Closes #6318 - tooĺ_writeout: fix the -w time output units Fix regression from commit fc813f80e1bcac (#6248) that changed the unit to microseconds instead of seconds with fractions Reported-by: 不确定 Fixes #6321 Closes #6322 - quiche: remove fprintf() leftover Jay Satiro (14 Dec 2020) - KNOWN_BUGS: SHA-256 digest not supported in Windows SSPI builds Closes https://github.com/curl/curl/issues/6302 - digest_sspi: Show InitializeSecurityContext errors in verbose mode The error is shown with infof rather than failf so that the user will see the extended error message information only in verbose mode, and will still see the standard CURLE_AUTH_ERROR message. For example: --- * schannel: InitializeSecurityContext failed: SEC_E_QOP_NOT_SUPPORTED (0x8009030A) - The per-message Quality of Protection is not supported by the security package * multi_done * Connection #1 to host 127.0.0.1 left intact curl: (94) An authentication function returned an error --- Ref: https://github.com/curl/curl/issues/6302 Closes https://github.com/curl/curl/pull/6315 Daniel Stenberg (13 Dec 2020) - URL-SYNTAX: add default port numbers and IDNA details Closes #6316 - URL-SYNTAX: mention how FILE:// access can access network on windows Closes #6314 Jay Satiro (12 Dec 2020) - URL-SYNTAX: Document default SMTP port 25 Note that ports 25 and 587 are common ports for smtp, the former being the default. Closes https://github.com/curl/curl/pull/6310 Daniel Stenberg (12 Dec 2020) - CURLOPT_URL.3: remove scheme specific details ... that are now found in URL-SYNTAX.md Closes #6307 Dan Fandrich (12 Dec 2020) - docs: Fix some typos [skip ci] Daniel Stenberg (12 Dec 2020) - URL-SYNTAX: mention all supported schemes Closes #6311 - [Douglas R. Reno brought this change] URL-SYNTAX.md: minor language improvements Closes #6308 - docs/URL-SYNTAX: the URL syntax curl accepts and works with Closes #6285 - [0xflotus brought this change] docs: enable syntax highlighting in several docs files ... for better readability Closes #6286 - test1564/1565: require the 'wakeup' feature to run Fixes #6299 Fixes #6300 Closes #6301 - runtests: add 'wakeup' as a feature - tests/server/disabled: add "wakeup" To allow the test suite to know if wakeup support is disabled in the build. - lib1564/5: verify that curl_multi_wakeup returns OK - tests: make --libcurl tests only test FTP options if ftp enabled Adjust six --libcurl tests to only check the FTP option if FTP is actually present in the build. Fixes #6303 Closes #6305 - runtests.pl: fix "uninitialized value" warning follow-up to e12825c642a88774 - runtests: add support for %if [feature] conditions ... to make tests run differently or expect different results depending on what features that are present or not in curl. Bonus: initial minor 'Hyper' awareness but nothing is using that yet Closes #6304 - [Jon Rumsey brought this change] OS400: update ccsidcurl.c Add 'struct' to cast and declaration of cfcdata to fix compilation error. Fixes #6292 Closes #6297 - ngtcp2: make it build it current master again Closes #6296 - [Cristian Rodríguez brought this change] connect: defer port selection until connect() time If supported, defer port selection until connect() time if --interface is given and source port is 0. Reproducer: * start fast webserver on port 80 * starve system of ephemeral ports $ sysctl net.ipv4.ip_local_port_range="60990 60999" * start a curl/libcurl "crawler" $curl --keepalive --parallel --parallel-immediate --head --interface 127.0.0.2 "http://127.0.0.[1-254]/file[001-002].txt" current result: (possible some successful data) curl: (45) bind failed with errno 98: Address already in use result after patch: (complete success or few connections failing, higlhy depending on load) Fail only when all the possible 4-tuple combinations are exhausted, which is impossible to do when port is selected at bind() time becuse the kernel does not know if socket will be listen()'ed on or connect'ed yet. Closes #6295 - [Hans-Christian Noren Egtvedt brought this change] connect: zero variable on stack to silence valgrind complaint Valgrind will complain that ssrem buffer usage if not explicit initialized, hence initialize it to zero. This completes the change intially started in commit 2c0d7212151 ('ftp: retry getpeername for FTP with TCP_FASTOPEN') where the ssloc buffer has a similar memset to zero. Signed-off-by: Hans-Christian Noren Egtvedt <hegtvedt@cisco.com> Closes #6289 - RELEASE-NOTES: synced start over on the next release cycle Version 7.74.0 (9 Dec 2020) Daniel Stenberg (9 Dec 2020) - RELEASE-NOTES: synced for 7.74.0 Jay Satiro (7 Dec 2020) - [Jacob Hoffman-Andrews brought this change] urldata: restore comment on ssl_connect_data.use This comment was originally on the `use` field, but was separated from its field in 62a2534. Closes https://github.com/curl/curl/pull/6287 Daniel Stenberg (7 Dec 2020) - VERSIONS: refreshed We always use the patch number these days: all releases are "major.minor.patch" - [Jakub Zakrzewski brought this change] cmake: don't use reserved target name 'test' CMake up to 3.10 always reserves this name Fixes #6257 Closes #6258 - openssl: make the OCSP verification verify the certificate id CVE-2020-8286 Reported by anonymous Bug: https://curl.se/docs/CVE-2020-8286.html - ftp: make wc_statemach loop instead of recurse CVE-2020-8285 Fixes #6255 Bug: https://curl.se/docs/CVE-2020-8285.html Reported-by: xnynx on github - ftp: CURLOPT_FTP_SKIP_PASV_IP by default The command line tool also independently sets --ftp-skip-pasv-ip by default. Ten test cases updated to adapt the modified --libcurl output. Bug: https://curl.se/docs/CVE-2020-8284.html CVE-2020-8284 Reported-by: Varnavas Papaioannou - urlapi: don't accept blank port number field without scheme ... as it makes the URL parser accept "very-long-hostname://" as a valid host name and we don't want that. The parser now only accepts a blank (no digits) after the colon if the URL starts with a scheme. Reported-by: d4d on hackerone Closes #6283 - Revert "multi: implement wait using winsock events" This reverts commit d2a7d7c185f98df8f3e585e5620cbc0482e45fac. This commit also reverts the subsequent follow-ups to that commit, which were all done within windows #ifdefs that are removed in this change. Marc helped me verify this. Fixes #6146 Closes #6281 - [Klaus Crusius brought this change] ftp: retry getpeername for FTP with TCP_FASTOPEN In the case of TFO, the remote host name is not resolved at the connetion time. For FTP that has lead to missing hostname for the secondary connection. Therefore the name resolution is done at the time, when FTP requires it. Fixes #6252 Closes #6265 Closes #6282 - [Thomas Danielsson brought this change] scripts/completion.pl: parse all opts For tab-completion it may be preferable to include all the available options. Closes #6280 - RELEASE-NOTES: synced - openssl: use OPENSSL_init_ssl() with >= 1.1.0 Reported-by: Kovalkov Dmitrii and Per Nilsson Fixes #6254 Fixes #6256 Closes #6260 - SECURITY-PROCESS: disclose on hackerone Once a vulnerability has been published, the hackerone issue should be disclosed. For tranparency. Closes #6275 Marc Hoersken (3 Dec 2020) - tests/util.py: fix compatibility with Python 2 Backporting the Python 3 implementation of setStream to ClosingFileHandler as a fallback within Python 2. Reported-by: Jay Satiro Fixes #6259 Closes #6270 Daniel Gustafsson (3 Dec 2020) - docs: fix typos and markup in ETag manpage sections Reported-by: emanruse on github Fixes #6273 Daniel Stenberg (2 Dec 2020) - quiche: close the connection Reported-by: Junho Choi Fixes #6213 Closes #6217 Jay Satiro (2 Dec 2020) - ngtcp2: Fix build error due to symbol name change - NGTCP2_CRYPTO_LEVEL_APP -> NGTCP2_CRYPTO_LEVEL_APPLICATION ngtcp2/ngtcp2@76232e9 changed the name. ngtcp2 master is required to build curl with http3 support. Closes https://github.com/curl/curl/pull/6271 Daniel Stenberg (1 Dec 2020) - [Klaus Crusius brought this change] cmake: check for linux/tcp.h The HAVE_LINUX_TCP_H define was not set by cmake. Closes #6252 - NEW-PROTOCOL: document what needs to be done to add one Closes #6263 - splay: rename Curl_splayremovebyaddr to Curl_splayremove ... and remove the old unused proto for the old Curl_splayremove version. Closes #6269 - openssl: free mem_buf in error path To fix a memory-leak. Closes #6267 - openssl: remove #if 0 leftover Follow-up to 4c9768565ec3a9 (from Sep 2008) Closes #6268 - ntlm: avoid malloc(0) on zero length user and domain ... and simplify the too-long checks somewhat. Detected by OSS-Fuzz Closes #6264 - RELEASE-NOTES: synced Marc Hoersken (28 Nov 2020) - tests/server/tftpd.c: close upload file in case of abort Commit c353207 removed the closing right after do_tftp which covered the case of abort. This handles that case. Reviewed-by: Jay Satiro Reviewed-by: Daniel Stenberg Follow up to #6209 Closes #6234 Daniel Stenberg (26 Nov 2020) - [Daiki Ueno brought this change] ngtcp2: use the minimal version of QUIC supported by ngtcp2 Closes #6250 - [Daiki Ueno brought this change] ngtcp2: advertise h3 ALPN unconditionally Closes #6250 - [Daiki Ueno brought this change] vquic/ngtcp2.h: define local_addr as sockaddr_storage This field needs to be wide enough to hold sockaddr_in6 when connecting via IPv6. Otherwise, ngtcp2_conn_read_pkt will drop the packets because of the address mismatch: I00000022 [...] con ignore packet from unknown path We can safely assume that struct sockaddr_storage is available, as it is used in the public interface of ngtcp2. Closes #6250 - socks: check for DNS entries with the right port number The resolve call is done with the right port number, but the subsequent check used the wrong one, which then could find a previous resolve which would return and leave the fresh resolve "incomplete" and leaking memory. Fixes #6247 Closes #6253 - curl_setup: USE_RESOLVE_ON_IPS is for Apple native resolver use ... so don't define it when instructed to use c-ares! - test506: make it not run in c-ares builds As the asynch nature of it may trigger events in another order. A c-ares upgrade made it break. Reported-by: Marc Hörsken Fixes #6247 - runtests: make 'c-ares' a "feature" to depend on ... also added to the docs. - tool_writeout: use off_t getinfo-types instead of doubles Commit 3b80d3ca46b12e52342 (June 2017) introduced getinfo replacement variables that use curl_off_t instead of doubles. Switch the --write-out function over to use them. Closes #6248 - [Emil Engler brought this change] file: avoid duplicated code sequence file_disconnect() is identical with file_do() except the function header but as the arguments are unused anyway so why not just return file_do() directly! Reviewed-by: Daniel Stenberg Closes #6249 - [Rikard Falkeborn brought this change] infof/failf calls: fix format specifiers Update a few format specifiers to match what is being printed. Closes #6241 - docs/INTERNALS: remove reference to Curl_sendf() The function has been removed from common usage. Also removed comment in gopher.c that still referenced it. Reported-by: Rikard Falkeborn Fixes #6242 Closes #6243 - [Rikard Falkeborn brought this change] examples: update .gitignore Add files that are generated by 'make examples' and remove some that have been renamed. The commits that renamed the programs are e9625c5bc6c046a (imap.c and simplesmtp.c were renamed to imap-fetch.c and smtp-send.c) and ad39e7ec01e7 (pop3slist.c and pop3s.c were renamed to pop3-list.c and pop3-ssl.c). Closes #6240 - asyn: use 'struct thread_data *' instead of 'void *' To reduce use of types that can't be checked at compile time. Also removes several typecasts. ... and rename the struct field from 'os_specific' to 'tdata'. Closes #6239 Reviewed-by: Jay Satiro Viktor Szakats (23 Nov 2020) - Makefile.m32: add support for UNICODE builds It requires the linker to support the `-municode` option. This is available in more recent mingw-w64 releases. Ref: https://gcc.gnu.org/onlinedocs/gcc/x86-Windows-Options.html Ref: https://stackoverflow.com/questions/3571250/wwinmain-unicode-and-mingw/11706847#11706847 Reviewed-by: Jay Satiro Reviewed-by: Marcel Raad Closes #6228 Daniel Stenberg (23 Nov 2020) - urldata: remove 'void *protop' and create the union 'p' ... to avoid the use of 'void *' for the protocol specific structs done per transfer. Closes #6238 - winbuild: remove docs from Makefiles and refer to README.md Reduce risk for conflicting docs and makes it to a single place to fix and polish. add these missing options to the readme: ENABLE_OPENSSL_AUTO_LOAD_CONFIG and ENABLE_UNICODE clarify ENABLE_SCHANNEL default varies Fixes #6216 Closes #6227 Co-Authored-by: Jay Satiro - [Daiki Ueno brought this change] http3: use the master branch of GnuTLS for testing Closes #6235 - KNOWN_BUGS: curl with wolfSSL lacks support for renegotiation Closes #5839 - KNOWN_BUGS: wakeup socket disconnect causes havoc Closes #6132 Closes #6133 - RELEASE-NOTES: synced - [Oliver Urbann brought this change] curl: add compatibility for Amiga and GCC 6.5 Changes are mainly reordering and adding of includes required to compile with a more recent version of GCC. Closes #6220 Marc Hoersken (20 Nov 2020) - tests/server/tftpd.c: close upload file right after transfer Make sure uploaded file is no longer locked after the transfer while waiting for the final ACK to be handled. Assisted-by: Daniel Stenberg Bug: #6058 Closes #6209 - CI/cirrus: simplify logic for disabled tests The OpenSSH server instance for the testsuite cannot be started on FreeBSD, therefore the SFTP and SCP tests are disabled right away from the beginning. The previous OS version specific logic for SKIP_TESTS is no longer needed/used and can therefore be removed. Reviewed-by: Daniel Stenberg Follow up to #6211 Closes #6229 Daniel Gustafsson (20 Nov 2020) - mailmap: Daniel Hwang Add Daniel Hwang to the mailmap to cover the alternative spelling Daniel Lee Hwang which was used in one commit. Closes #6230 Reviewed-by: Daniel Stenberg <daniel@haxx.se> - openssl: guard against OOM on context creation EVP_MD_CTX_create will allocate memory for the context and returns NULL in case the allocation fails. Make sure to catch any allocation failures and exit early if so. In passing, also move to EVP_DigestInit rather than EVP_DigestInit_ex as the latter is intended for ENGINE selection which we don't do. Closes #6224 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Reviewed-by: Emil Engler <me@emilengler.com> Daniel Stenberg (19 Nov 2020) - [Vincent Torri brought this change] cmake: use libcurl.rc in all Windows builds Reviewed-by: Marcel Raad Closes #6215 - [Cristian Morales Vega brought this change] cmake: make CURL_ZLIB a tri-state variable By differentiating between ON and AUTO it can make a missing zlib library a hard error when CURL_ZLIB=ON is used. Reviewed-by: Jakub Zakrzewski Closes #6221 Fixes #6173 - quiche: remove 'static' from local buffer For thread-safety Closes #6223 - KNOWN_BUGS: cmake: libspsl is not supported Closes #6214 - KNOWN_BUGS: cmake autodetects cert paths when cross-compiling Closes #6178 - KNOWN_BUGS: cmake build doesn't fail if zlib not found Closes #6173 - KNOWN_BUGS: cmake libcurl.pc uses absolute library paths Closes #6169 - KNOWN_BUGS: cmake: generated .pc file contains strange entries Closes #6167 - KNOWN_BUGS: cmake uses -lpthread instead of Threads::Threads Closes #6166 - KNOWN_BUGS: cmake build in Linux links libcurl to libdl Closes #6165 - KNOWN_BUGS: make a new section for cmake topics Closes #6219 - [Emil Engler brought this change] cirrus: build with FreeBSD 12.2 in CirrusCI Closes #6211 Marc Hoersken (14 Nov 2020) - tests/*server.py: close log file after each log line Make sure the log file is not locked once a test has finished and align with the behavior of our logmsg. Rename curl_test_data.py to be a general util.py. Format and sort Python imports with isort/VSCode. Bug: #6058 Closes #6206 Daniel Stenberg (13 Nov 2020) - CURLOPT_HSTS.3: document the file format Closes #6205 - RELEASE-NOTES: synced - release-notes.pl: detect #[number] better for Ref: etc - curl: only warn not fail, if not finding the home dir ... as there's no good reason to error out completely. Reported-by: Andreas Fischer Fixes #6200 Closes #6201 - httpput-postfields.c: new example doing PUT with POSTFIELDS Proposed-by: Jeroen Ooms Ref: #6186 Closes #6188 - [Tobias Hieta brought this change] cmake: correctly handle linker flags for static libs curl CMake was setting the the EXE flags for static libraries which made the /manifest:no flag ended up when linking the static library, which is not a valid flag for lib.exe or llvm-lib.exe and caused llvm-lib to exit with an error. The better way to handle this is to make sure that we pass the correct linker flags to CMAKE_STATIC_LINKER_FLAGS instead. Reviewed-by: Jakub Zakrzewski Closes #6195 - [Tobias Hieta brought this change] cmake: don't pass -fvisibility=hidden to clang-cl on Windows When using clang-cl on windows -fvisibility=hidden is not an known argument. Instead it behaves exactly like MSVC in this case. So let's make sure we take that path. In CMake clang-cl sets both CMAKE_C_COMPILER_ID=clang and MSVC get's defined since clang-cl is basically a MSVC emulator. So guarding like we do in this patch seems logical. Reviewed-by: Jakub Zakrzewski Closes #6194 - http_proxy: use enum with state names for 'keepon' To make the code clearer, change the 'keepon' from an int to an enum with better state names. Reported-by: Niranjan Hasabnis Bug: https://curl.se/mail/lib-2020-11/0026.html Closes #6193 - curl_easy_escape: limit output string length to 3 * max input ... instead of the limiting it to just the max input size. As every input byte can be expanded to 3 output bytes, this could limit the input string to 2.66 MB instead of the intended 8 MB. Reported-by: Marc Schlatter Closes #6192 - docs: document the 8MB input string limit for curl_easy_escape and curl_easy_setopt() The limit is there to catch mistakes and abuse. It is meant to be large enough to allow virtually all "fine" use cases. Reported-by: Marc Schlatter Fixes #6190 Closes #6191 - mqttd: fclose test file when done Reported-by: Marc Hörsken Reviewed-by: Jay Satiro Bug: #6058 Closes #6189 - RELEASE-NOTES: synced - THANKS-filter: ignore autobuild links - Revert "libcurl.pc: make it relocatable" This reverts commit 3862c37b6373a55ca704171d45ba5ee91dec2c9f. That fix should either be done differently or with an option. Reported-by: asavah on github Fixes #6157 Closes #6183 - examples/httpput: remove use of CURLOPT_PUT It is deprecated and unnecessary since it already sets CURLOPT_UPLOAD. Reported-by: Jeroen Ooms Fixes #6186 Closes #6187 - Curl_pgrsStartNow: init speed limit time stamps at start By setting the speed limit time stamps unconditionally at transfer start, we can start off a transfer without speed limits and yet allow them to get set during transfer and have an effect. Reported-by: Kael1117 on github Fixes #6162 Closes #6184 - ngtcp2: adapt to recent nghttp3 updates 'reset_stream' was added to the nghttp3_conn_callbacks struct Closes #6185 - configure: pass -pthread to Libs.private for pkg-config Reported-by: Cristian Morales Vega Fixes #6168 Closes #6181 - altsvc: minimize variable scope and avoid "DEAD_STORE" Closes #6182 - FAQ: remove "Why is there a HTTP/1.1 in my HTTP/2 request?" This hasn't been the case for a while now, remove. - FAQ: refresh "Why do I get "certificate verify failed" Add more details, remove references to ancient curl version. - test493: verify --hsts upgrade and that %{url_effective} reflects that Closes #6175 - url: make sure an HSTS upgrade updates URL and scheme correctly Closes #6175 - tool_operate: set HSTS with CURLOPT_HSTS to pass on filename Closes #6175 - hsts: remove debug code leftovers Closes #6175 - FAQ: refreshed - remove a few ancient questions - add configure with static libs question - updated wording in several places - lowercased curl Closes #6177 Daniel Gustafsson (5 Nov 2020) - examples: fix comment syntax Commit ac0a88fd2 accidentally added a stray character outside of the comment which broke compilation. Fix by removing. Reported-by: autobuild https://curl.se/dev/log.cgi?id=20201105084306-12742 - hsts: Remove pointless call to free in errorpath The line variable will always be NULL in the error path, so remove the free call since it's pointless. Closes #6170 Reviewed-by: Daniel Stenberg <daniel@haxx.se> - docs: Fix various typos in documentation Closes #6171 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Daniel Stenberg (5 Nov 2020) - copyright: fix year ranges Follow-up from 4d2f8006777 - HISTORY: the new domain - curl.se: new home Closes #6172 - KNOWN_BUGS: FTPS with Schannel times out file list operation Reported-by: bobmitchell1956 on github Closes #5284 - KNOWN_BUGS: SMB tests fail with Python 2 Reported-by: Jay Satiro Closes #5983 - KNOWN_BUGS: LDAPS with NSS is slow Reported-by: nosajsnikta on github Closes #5874 Sergei Nikulov (4 Nov 2020) - travis: use ninja-build for CMake builds Added package ninja-build to environment Use ninja to speed up CMake builds Closes #6077 Daniel Stenberg (4 Nov 2020) - [Harry Sintonen brought this change] rtsp: error out on empty Session ID, unified the code - [Harry Sintonen brought this change] rtsp: fixed the RTST Session ID mismatch in test 570 Closes #6161 - [Harry Sintonen brought this change] rtsp: fixed Session ID comparison to refuse prefix Closes #6161 - RELEASE-NOTES: synced (forgot to update the list of contributors) - RELEASE-NOTES: synced - curlver: bumped to 7.74.0 - hsts: add read/write callbacks - read/write callback options - man pages for the 4 new setopts - test 1915 verifies the callbacks Closes #5896 - hsts: add support for Strict-Transport-Security - enable in the build (configure) - header parsing - host name lookup - unit tests for the above - CI build - CURL_VERSION_HSTS bit - curl_version_info support - curl -V output - curl-config --features - CURLOPT_HSTS_CTRL - man page for CURLOPT_HSTS_CTRL - curl --hsts (sets CURLOPT_HSTS_CTRL and works with --libcurl) - man page for --hsts - save cache to disk - load cache from disk - CURLOPT_HSTS - man page for CURLOPT_HSTS - added docs/HSTS.md - fixed --version docs - adjusted curl_easy_duphandle Closes #5896 - [Sergei Nikulov brought this change] CI/tests: enable test target on TravisCI for CMake builds Added test-nonflaky target to CMake builds Disabled test 1139 because the cmake build doesn't create docs/curl.1 Closes #6074 - tool_debug_cb: do not assume zero-terminated data Follow-up to d70a5b5a0f5e3 - sendf: move the verbose-check into Curl_debug Saves us from having the same check done everywhere. Closes #6159 - travis: use valgrind when running tests for debug builds Except the non-x86 and sanitizer builds Closes #6154 - header.d: fix syntax mistake follow-up from 1144886f38fd0 - [Harry Sintonen brought this change] gnutls: fix memory leaks (certfields memory wasn't released) Closes #6153 - tests: add missing global_init/cleanup calls Without the cleanup call in these test files, the mbedTLS backend leaks memory. Closes #6156 - tool_operate: --retry for HTTP 408 responses too This was inadvertently dropped from the code when the parallel support was added. Regression since b88940850 (7.66.0) Reviewed-by: Jay Satiro Closes #6155 - http: pass correct header size to debug callback for chunked post ... when the chunked framing was added, the size of the "body part" of the data was calculated wrongly so the debug callback would get told a header chunk a few bytes too big that would also contain the first few bytes of the request body. Reported-by: Dirk Wetter Ref: #6144 Closes #6147 - header.d: mention the "Transfer-Encoding: chunked" handling Ref: #6144 Closes #6148 - acinclude: detect manually set minimum macos/ipod version ... even if set in the CC or IPHONEOS/MACOSX_DEPLOYMENT_TARGET variables. Reported-by: hamstergene on github Fixes #6138 Closes #6140 Jay Satiro (29 Oct 2020) - tests: fix some http/2 tests for older versions of nghttpx - Add regex that strips http/2 server header name to those http/2 tests that don't already have it. - Improve that regex in all http/2 tests. Tests 358 and 359 were failing for me before this change on a system that uses an older version of nghttpx which includes its version number in the server header. Closes https://github.com/curl/curl/pull/6139 Daniel Stenberg (30 Oct 2020) - RELEASE-NOTES: synced - [Cristian Morales Vega brought this change] configure: use pkgconfig to find openSSL when cross-compiling This reverts 736a40fec (November 2004), which doesn't explain why it was done. Closes #6145 - tool_operate: bail out proper on errors for parallel setup ... otherwise for example trying to upload a missing file just causes a loop. Reported-by: BrumBrum on hackerone Closes #6141 - [Sergei Nikulov brought this change] CMake: make BUILD_TESTING dependent option CMake will now handle BUILD_TESTING depending on PERL_FOUND and CURL_DISABLE_TESTING Ref: #6036 Closes #6072 - libssh2: fix transport over HTTPS proxy The fix in #6021 was not enough. This fix makes sure SCP/SFTP content can also be transfered over a HTTPS proxy. Fixes #6113 Closes #6128 - curl.1: add an "OUTPUT" section at the top of the manpage Explain the basic concepts behind curl output. Inspired by #6124 Closes #6134 - mailmap: set Viktor Szakats's email - runtests: show keywords when no tests ran To help out future debugging, runtests now outputs the list of keywords when it fails because no tests ran. Ref: #6120 Closes #6126 Jay Satiro (26 Oct 2020) - CURLOPT_DNS_USE_GLOBAL_CACHE.3: fix typo Reported-by: Rui LIU Closes https://github.com/curl/curl/issues/6131 - range.d: fix typo Follow-up to 15ae039 from earlier today. Daniel Stenberg (26 Oct 2020) - CI/github: work-around for brew breakage on macOS ... and make it use OpenSSL 1.1 properly Fixes #6130 Closes #6129 - [José Joaquín Atria brought this change] range.d: clarify that curl will not parse multipart responses Closes #6127 Fixes #6124 - RELEASE-NOTES: synced - [Baruch Siach brought this change] libssh2: fix build with disabled proxy support Build breaks because the http_proxy field is missing: vssh/libssh2.c:3119:10: error: 'struct connectdata' has no member named 'http_proxy' Regression from #6021, shipped in curl 7.73.0 Closes #6125 - alt-svc: enable by default Remove CURLALTSVC_IMMEDIATELY, which was never implemented/supported. alt-svc support in curl is no longer considered experimental Closes #5868 - CI/appveyor: remove (unused) runtests.pl -b option - [Emil Engler brought this change] tool_help: make "output" description less confusing Currently the description of "output" is misleading when comparing it "verbose". Closes #6118 - CI/appveyor: disable test 571 in two cmake builds ... they're simply too flaky there. Closes #6119 - cmake: set the unicode feature in curl-config on Windows ... if built that way. To make it match curl -V output. Reviewed-by: Marcel Raad Closes #6117 - libssh2: require version 1.0 or later ... and simplify the code accordingly. libssh2 version 1.0 was released in April 2009. Closes #6116 - KNOWN_BUGS: mention the individual cmake issues ... to make them easier to refer to and address separately and one-by-one. - CMake: store IDN2 information in curl_config.h This allows the build to enable IDN properly and it makes test 1014 happier. Ref: #6074 Closes #6108 - CMake: call the feature unixsockets without dash ... so that curl-config gets correct and makes test 1014 happy! Ref: #6074 Closes #6108 - CI/travis: add brotli and zstd to the libssh2 build ... to make sure such tests are run with valgrind. Suppress the zstd valgrind warnings we get with version 1.3.3 on Ubuntu 18.04 (for debug and non-debug builds). Closes #6105 - runtests: revert the mistaken edit of $CURL Regression from c4693adc62 - RELEASE-NOTES: synced - curl_url_set.3: fix typo in the RETURN VALUE section Reported-by: Basuke Suzuki Fixes #6102 Jay Satiro (17 Oct 2020) - [Daniel Stenberg brought this change] packages/OS400: make the source code-style compliant ... and make sure 'make checksrc' in the root dir also verifies the packages/OS400 sources. Closes https://github.com/curl/curl/pull/6085 - os400: Sync libcurl API options This fixes the OS400 build and also an incorrect entry for CURLINFO_APPCONNECT_TIME_T where it was treated as CURLINFO_STARTTRANSFER_TIME_T. Reported-by: Jon Rumsey Fixes https://github.com/curl/curl/issues/6083 Closes https://github.com/curl/curl/pull/6084 Daniel Stenberg (16 Oct 2020) - CURLOPT_NOBODY.3: fix typo Reported-by: Basuke Suzuki Fixes #6097 Marc Hoersken (16 Oct 2020) - CI/azure: improve on flakiness by avoiding libtool wrappers Install curl binaries into MinGW bin folder and use that for the tests in order to avoid libtool wrapper binaries. The libtool wrapper binaries (not scripts) on Windows seem to be one of the possible causes for the following issues: 1. Process output can be lost in the wrapper process chain. 2. Killing the wrapper process does not kill the actual one. Derived from #5904 Closes #6049 Daniel Stenberg (16 Oct 2020) - CURLOPT_URL.3: clarify SCP/SFTP URLs are for uploads as well - [Zenju brought this change] CURLOPT_TCP_NODELAY.3: fix comment in example code Closes #6096 - openssl: acknowledge SRP disabling in configure properly Follow-up to 68a513247409 Use a new separate define that is the combination of both HAVE_OPENSSL_SRP and USE_TLS_SRP: USE_OPENSSL_SRP Bug: https://curl.haxx.se/mail/lib-2020-10/0037.html Closes #6094 Viktor Szakats (16 Oct 2020) - http3: fix two build errors, silence warnings * fix two build errors due to mismatch between function declarations and their definitions * silence two mismatched signs warnings via casts Approved-by: Daniel Stenberg Closes #6093 - Makefile.m32: add support for HTTP/3 via ngtcp2+nghttp3 Approved-by: Daniel Stenberg Closes #6092 Daniel Stenberg (16 Oct 2020) - tool_operate: fix compiler warning when --libcurl is disabled Closes #6095 - checksrc: warn on empty line before open brace ... and fix a few occurances Closes #6088 - urlapi: URL encode a '+' in the query part ... when asked to with CURLU_URLENCODE. Extended test 1560 to verify. Reported-by: Dietmar Hauser Fixes #6086 Closes #6087 - [Cristian Morales Vega brought this change] libcurl.pc: make it relocatable It supposes when people specify the libdir/includedir they do it to change where under prefix/exec_prefix it should be, not to make it independent of prefix/exec_prefix. Closes #6061 - runtests: return error if no tests ran ... and make TESTFAIL stand out a little better by adding newlines before and after. Reported-by: Marc Hörsken Issue: #6052 Closes #6053 - docs/FEATURE: convert to markdown ... and clean it up a bit. Closes #6067 - [Philipp Klaus Krause brought this change] strerror: use 'const' as the string should never be modified Closes #6068 - [Jay Satiro brought this change] connect: repair build without ipv6 availability Assisted-by: Daniel Stenberg Reported-by: Tom G. Christensen Fixes https://github.com/curl/curl/issues/6069 Closes https://github.com/curl/curl/pull/6071 - RELEASE-NOTES: synced Started over for the journey to next release. - src/tool_filetime: disable -Wformat on mingw for this file With gcc 10 on mingw we otherwise get this warning: error: ISO C does not support the 'I' printf flag [-Werror=format=] Fixes #6079 Closes #6082 - test122[12]: remove these two tests ... and remove the objnames scripts they tested. They're not used for anything anymore so testing them serves no purpose! Reported-by: Marc Hörsken Fixes #6080 Closes #6081 Version 7.73.0 (14 Oct 2020) Daniel Stenberg (14 Oct 2020) - RELEASE-NOTES: synced for 7.73.0 - THANKS: from 7.73.0 and .mailmap fixes - mailmap: fixups of some contributors - projects/build-wolfssl.bat: fix the copyright year range Marc Hoersken (14 Oct 2020) - [Sergei Nikulov brought this change] CI/tests: fix invocation of tests for CMake builds Update appveyor.yml to set env variable TFLAGS and run tests Remove curly braces due to CMake error (${TFLAGS} -> $TFLAGS) Move testdeps build to build step (per review comments) Reviewed-by: Marc Hörsken Closes #6066 Fixes #6052 - tests/server/util.c: fix support for Windows Unicode builds Detected via #6066 Closes #6070 Daniel Stenberg (13 Oct 2020) - [Jay Satiro brought this change] strerror: Revert to local codepage for Windows error string - Change get_winapi_error() to return the error string in the local codepage instead of UTF-8 encoding. Two weeks ago bed5f84 fixed get_winapi_error() to work on xbox, but it also changed the error string's encoding from local codepage to UTF-8. We return the local codepage version of the error string because if it is output to the user's terminal it will likely be with functions which expect the local codepage (eg fprintf, failf, infof). This is essentially a partial revert of bed5f84. The support for xbox remains but the error string is reverted back to local codepage. Ref: https://github.com/curl/curl/pull/6005 Reviewed-by: Marcel Raad Closes #6065 Marc Hoersken (13 Oct 2020) - CI/tests: use verification curl for test reporting APIs Avoid using our own, potentially installed, curl for the test reporting APIs in case it is broken. Reviewed-by: Daniel Stenberg Preparation for #6049 Closes #6063 Viktor Szakats (12 Oct 2020) - windows: fix comparison of mismatched types warning clang 10, mingw-w64: ``` vtls/openssl.c:2917:33: warning: comparison of integers of different signs: 'DWORD' (aka 'unsigned long') and 'HRESULT' (aka 'long') [-Wsign-compare] if(GetLastError() != CRYPT_E_NOT_FOUND) ~~~~~~~~~~~~~~ ^ ~~~~~~~~~~~~~~~~~ ``` Approved-by: Daniel Stenberg Closes #6062 Daniel Stenberg (11 Oct 2020) - [Viktor Szakats brought this change] src/Makefile.m32: fix undefined curlx_dyn_* errors by linking `lib/dynbuf.c` when building a static curl binary. Previously this source file was only included when building a dynamic curl binary. This was likely possibly because no functions from the `src/Makefile.inc` / `CURLX_CFILES` sources were actually required for a curl tool build. This has recently changed with the introduction of `curlx_dyn_*()` memory functions and their use by the tool sources. Closes #6060 - HISTORY: curl verifies SSL certs by default since version 7.10 Marc Hoersken (8 Oct 2020) - runtests.pl: use $LIBDIR variable instead of hardcoded path Reviewed-by: Daniel Stenberg Closes #6051 Daniel Stenberg (7 Oct 2020) - checksrc: detect // comments on column 0 Spotted while working on #6045 Closes #6048 - [Frederik Wedel-Heinen brought this change] mbedtls: add missing header when defining MBEDTLS_DEBUG Closes #6045 - curl: make sure setopt CURLOPT_IPRESOLVE passes on a long Previously, it would pass on a define (int) which could make libcurl read junk as a value - which prevented the CURLOPT_IPRESOLVE option to "take". This could then make test 2100 do two DoH requests instead of one! Fixes #6042 Closes #6043 - RELEASE-NOTES: synced - scripts/release-notes.pl: don't "embed" $ in format string for printf() ... since they might contain %-codes that mess up the output! Jay Satiro (5 Oct 2020) - [M.R.T brought this change] build-wolfssl: fix build with Visual Studio 2019 Closes https://github.com/curl/curl/pull/6033 Daniel Stenberg (4 Oct 2020) - runtests: add %repeat[]% for test files ... and use this new keywords in all the test files larger than 50K to reduce their sizes and make them a lot easier to read and understand. Closes #6040 - [Emil Engler brought this change] --help: move two options from the misc category The cmdline opts delegation and suppress-connect-headers fit better into auth and proxy rather than misc. Follow-up to aa8777f63febc Closes #6038 - [Samanta Navarro brought this change] docs/opts: fix typos in two manual pages Closes #6039 - ldap: reduce the amount of #ifdefs needed Closes #6035 - runtests: provide curl's version string as %VERSION for tests ... so that we can check HTTP requests for User-Agent: curl/%VERSION Update 600+ test cases accordingly. Closes #6037 - checksrc: warn on space after exclamation mark Closes #6034 - test1465: verify --libcurl with binary POST data - runtests: allow generating a binary sequence from hex - tool_setopt: escape binary data to hex, not octal - curl: make --libcurl show binary posts correctly Reported-by: Stephan Mühlstrasser Fixes #6031 Closes #6032 Jay Satiro (1 Oct 2020) - strerror: fix null deref on winapi out-of-memory Follow-up to bed5f84 from several days ago. Ref: https://github.com/curl/curl/pull/6005 Daniel Stenberg (1 Oct 2020) - [Kamil Dudka brought this change] vtls: deduplicate some DISABLE_PROXY ifdefs ... in the code of gtls, nss, and openssl Closes #5735 - RELEASE-NOTES: synced - [Emil Engler brought this change] TODO: Add OpenBSD libtool notice See #5862 Closes #6030 - tests/unit/README: convert to markdown ... and add to dist! Closes #6028 - tests/README: convert to markdown Closes #6028 - include/README: convert to markdown Closes #6028 - examples/README: convert to markdown Closes #6028 - configure: don't say HTTPS-proxy is enabled when disabled! Reported-by: Kamil Dudka Reviewed-by: Kamil Dudka Bug: https://github.com/curl/curl/pull/5735#issuecomment-701376388 Closes #6029 Daniel Gustafsson (30 Sep 2020) - src: Consistently spell whitespace without whitespace Whitespace is spelled without a space between white and space, so make sure to consistently spell it that way across the codebase. Closes #6023 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Reviewed-by: Emil Engler <me@emilengler.com> - MANUAL: update examples to resolve without redirects www.netscape.com is redirecting to a cookie consent form on Aol, and cool.haxx.se isn't responding to FTP anymore. Replace with examples that resolves in case users try out the commands when reading the manual. Closes #6024 Reviewed-by: Daniel Stenberg <daniel@haxx.se> Reviewed-by: Emil Engler <me@emilengler.com> Daniel Stenberg (30 Sep 2020) - HISTORY: add some 2020 events - sectransp: make it build with --disable-proxy Follow-up from #5466 and f3d501dc678d80 Reported-by: Javier Navarro Fixes #6025 Closes #6026 - ECH: renamed from ESNI in docs and configure Encrypted Client Hello (ECH) is the current name. Closes #6022 - configure: use "no" instead of "disabled" for the end summary ... for consistency but also to make them more distinctly stand out next to the "enabled" lines. - TODO: SSH over HTTPS proxy with more backends ... as right now only the libssh2 backend supports it. - libssh2: handle the SSH protocols done over HTTPS proxy Reported-by: Robin Douine Fixes #4295 Closes #6021 - [Emil Engler brought this change] memdebug: remove 9 year old unused debug function There used to be a way to have memdebug fill allocated memory. 9 years later this has no value there (valgrind and ASAN etc are way better). If people need to know about it they can have a look at VCS logs. Closes #5973 - sendf: move Curl_sendf to dict.c and make it static ... as the only remaining user of that function. Also fix gopher.c to instead use Curl_write() Closes #6020 - ROADMAP: updates and cleanups Fix the HSTS PR Remove DoT, thread-safe init and hard-coded localhost. I feel very little interest for these with users so I downgrade them to plain "TODO" entries again. - schannel: return CURLE_PEER_FAILED_VERIFICATION for untrusted root This matches what is returned in other TLS backends in the same situation. Reviewed-by: Jay Satiro Reviewed-by: Emil Engler Follow-up to 5a3efb1 Reported-by: iammrtau on github Fixes #6003 Closes #6018 - RELEASE-NOTES: synced - ftp: make a 552 response return CURLE_REMOTE_DISK_FULL Added test 348 to verify. Added a 'STOR' command to the test FTP server to enable test 348. Documented the command in FILEFORMAT.md Reported-by: Duncan Wilcox Fixes #6016 Closes #6017 - pause: only trigger a reread if the unpause sticks As an unpause might itself get paused again and then triggering another reread doesn't help. Follow-up from e040146f22608fd9 (shipped since 7.69.1) Bug: https://curl.haxx.se/mail/lib-2020-09/0081.html Patch-by: Kunal Chandarana Fixes #5988 Closes #6013 - test163[12]: require http to be built-in to run ... as speaking over an HTTPS proxy implies http! Closes #6014 - ngtcp2: adapt to new NGTCP2_PROTO_VER_MAX define Closes #6012 - [Javier Blazquez brought this change] strerror: honor Unicode API choice on Windows Closes #6005 - imap: make imap_send use dynbuf for the send buffer management Reuses the buffer and thereby reduces number of mallocs over a transfer. Closes #6010 - Curl_send: return error when pre_receive_plain can't malloc ... will probably trigger some false DEAD CODE positives on non-windows code analyzers for the conditional code. Closes #6011 - ftp: separate FTPS from FTP over "HTTPS proxy" When using HTTPS proxy, SSL is used but not in the view of the FTP protocol handler itself so separate the connection's use of SSL from the FTP control connection's sue. Reported-by: Mingtao Yang Fixes #5523 Closes #6006 Dan Fandrich (23 Sep 2020) - tests/data: Fix some mismatched XML tags in test cases This allows these test files to pass xmllint. Daniel Stenberg (23 Sep 2020) - pingpong: use a dynbuf for the *_pp_sendf() function ... reuses the same dynamic buffer instead of doing repeated malloc/free cycles. Test case 100 (FTP dir list PASV) does 7 fewer memory allocation calls after this change in my test setup (132 => 125), curl 7.72.0 needed 140 calls for this. Test case 103 makes 9 less allocations now (130). Down from 149 in 7.72.0. Closes #6004 - dynbuf: add Curl_dyn_vaddf Closes #6004 - dynbuf: make *addf() not require extra mallocs ... by introducing a printf() function that appends directly into a dynbuf: Curl_dyn_vprintf(). This avoids the mandatory extra malloc so if the buffer is already big enough it can just printf directly into it. Since this less-malloc version requires tthe use of a library internal printf function, we only provide this version when building libcurl and not for the dynbuf code that is used when building the curl tool. Closes #5998 - KNOWN_BUGS: Unable to use PKCS12 certificate with Secure Transport Closes #5403 - pingpong: remove a malloc per Curl_pp_vsendf call This typically makes 7-9 fewer mallocs per FTP transfer. Closes #5997 - symbian: drop support The OS is deprecated. I see no traces of anyone having actually built curl for Symbian after 2012. The public headers are unmodified. Closes #5989 - RELEASE-NOTES: synced - curl_krb5.h: rename from krb5.h Follow-up from f4873ebd0be32cf Turns out some older openssl installations go bananas otherwise. Reported-by: Tom van der Woerdt Fixes #5995 Closes #5996 - test1297: verify GOT_NOTHING with http proxy tunnel - http_proxy: do not count proxy headers in the header bytecount ... as that counter is subsequently used to detect if nothing was returned from the peer. This made curl return CURLE_OK when it should have returned CURLE_GOT_NOTHING. Fixes #5992 Reported-by: Tom van der Woerdt Closes #5994 - setopt: return CURLE_BAD_FUNCTION_ARGUMENT on bad argument Fixed two return code mixups. CURLE_UNKNOWN_OPTION is saved for when the option is, yeah, not known. Clarified this in the setopt man page too. Closes #5993 - krb5: merged security.c and krb specific FTP functions in here These two files were always tightly connected and it was hard to understand what went into which. This also allows us to make the ftpsend() function static (moved from ftp.c). Removed security.c Renamed curl_sec.h to krb5.h Closes #5987 - Curl_handler: add 'family' to each protocol Makes get_protocol_family() faster and it moves the knowledge about the "families" to each protocol handler, where it belongs. Closes #5986 - parsedate: tune the date to epoch conversion By avoiding an unnecessary error check and the temp use of the tm struct, the time2epoch conversion function gets a little bit faster. When repeating test 517, the updated version is perhaps 1% faster (on one particular build on one particular architecture). Closes #5985 - cmake: remove scary warning Remove the text saying "the curl cmake build system is poorly maintained. Be aware" ... not because anything changed just now, but to encourage users to use it and subsequently improve it. Closes #5984 - docs/MQTT: remove outdated paaragraphs - docs/MQTT: not experimental anymore Follow-up to e37e4468688d8f - docs/RESOURCES: remove This document is not maintained and rather than trying to refresh it, let's kill it. A more up-to-date document with relevant RFCs is this page on the curl website: https://curl.haxx.se/rfc/ Closes #5980 - docs/TheArtOfHttpScripting: convert to markdown Makes it easier to browse on github etc. Offers (better) links. It should be noted that this document is already mostly outdated and "Everything curl" at https://ec.haxx.se/ is a better resource and tutorial. Closes #5981 - BUGS: convert document to markdown Closes #5979 - --help: strdup the category ... since it is converted and the original pointer is freed on Windows unicode handling. Follow-up to aa8777f63febc Fixes #5977 Closes #5978 Reported-by: xwxbug on github - CHECKSRC: document two missing warnings - RELEASE-NOTES: synced - ftp: avoid risk of reading uninitialized integers If the received PASV response doesn't match the expected pattern, we could end up reading uninitialized integers for IP address and port number. Issue pointed out by muse.dev Closes #5972 - [Quentin Balland brought this change] easy_reset: clear retry counter Closes #5975 Fixes #5974 - ftp: get rid of the PPSENDF macro The use of such a macro hides some of what's actually going on to the reader and is generally disapproved of in the project. Closes #5971 - man pages: switch to https://example.com URLs Since HTTPS is "the new normal", this update changes a lot of man page examples to use https://example.com instead of the previous "http://..." Closes #5969 - github: remove the duplicate "Security vulnerability" entry ... since github adds an entry automatically by itself. Closes #5970 - [Emil Engler brought this change] github: use new issue template feature This helps us to avoid getting feature requests as well as security bugs reported into the issue tracker. Closes #5936 - [Emil Engler brought this change] urlapi: use more Curl_safefree Closes #5968 Marc Hoersken (17 Sep 2020) - multi: align WinSock mask variables in Curl_multi_wait Also skip pre-checking sockets to set timeout_ms to 0 after the first socket has been detected to be ready. Reviewed-by: rcombs on github Reviewed-by: Daniel Stenberg Follow up to #5886 - multi: reuse WinSock events variable in Curl_multi_wait Since the struct is quite large (1 long and 10 ints) we declare it once at the beginning of the function instead of multiple times inside loops to avoid stack movements. Reviewed-by: Viktor Szakats Reviewed-by: Daniel Stenberg Closes #5886 Daniel Stenberg (16 Sep 2020) - TODO: dynamically decide to use socketpair Suggested-by: Anders Bakken Closes #4829 - TODO: add PR reference for native IDN support on macOS As there was work started on this that never got completed. Closes #5371 - tool_help.h: update copyright year range Follow-up from aa8777f63febca - CI/azure: disable test 571 in the msys2 builds It's just too flaky there Reviewed-by: Marc Hoersken Closes #5954 - tool_writeout: protect fputs() from NULL When the code was changed to do fputs() instead of fprintf() it got sensitive for NULL pointers; add checks for that. Follow-up from 0c1e767e83ec66 Closes #5963 - test3015: verify stdout "as text" Follow-up from 0c1e767e83e to please win32 tests Closes #5962 - travis: use libressl v3.1.4 instead of master ... as their git master seems too fragile to use (and 3.2.1 which is the latest has a build failure). Closes #5964 - tests/FILEFORMAT: document type=shell for <command> - tests/FILEFORMAT: document nonewline support for <file> The one in <client>, that creates files. Follow-up from b83947c8df7 - [anio brought this change] tool_writeout: add new writeout variable, %{num_headers} This variable gives the number of headers. Closes #5947 - tool_urlglob: fix compiler warning "unreachable code" (On Windows builds.) Follow-up to 70a3b003d9 - [Gergely Nagy brought this change] vtls: deduplicate client certificates in ssl_config_data Closes #5629 - ftp: a 550 response to SIZE returns CURLE_REMOTE_FILE_NOT_FOUND This is primarily interesting for cases where CURLOPT_NOBODY is set as previously curl would not return an error for this case. MDTM getting 550 now also returns this error (it returned CURLE_FTP_COULDNT_RETR_FILE before) in order to unify return codes for missing files across protocols and specific FTP commands. libcurl already returns error on a 550 as a MDTM response (when CURLOPT_FILETIME is set). If CURLOPT_NOBODY is not set, an error would happen subsequently anyway since the RETR command would fail. Add test 1913 and 1914 to verify. Updated several tests accordingly due to the updated SIZE behavior. Reported-by: Tomas Berger Fixes #5953 Closes #5957 - curl: make checkpasswd use dynbuf Closes #5952 - curl: make glob_match_url use dynbuf Closes #5952 - curl: make file2memory use dynbuf Closes #5952 - curl: make file2string use dynbuf Closes #5952 - [Antarpreet Singh brought this change] imap: set cselect_bits to CURL_CSELECT_IN initially ... when continuing a transfer from a FETCH response. When the size of the file was small enough that the entirety of the transfer happens in a single go and schannel buffers holds the entire data. However, it wasn't completely read in Curl_pp_readresp since a line break was found before that could happen. So, by the time we are in imap_state_fetch_resp - there's data in buffers that needs to be read via Curl_read but nothing to read from the socket. After we setup a transfer (Curl_setup_transfer), curl just waits on the socket state to change - which doesn't happen since no new data ever comes. Closes #5961 - RELEASE-NOTES: synced - test434: test -K use in a single line without newline Closes #5946 - runtests: allow creating files without newlines Closes #5946 - curl: use curlx_dynbuf for realloc when loading config files ... fixes an integer overflow at the same time. Reported-by: ihsinme on github Assisted-by: Jay Satiro Closes #5946 - dynbuf: provide curlx_ names for reuse by the curl tool Closes #5946 - dynbuf: make sure Curl_dyn_tail() zero terminates Closes #5959 - tests: add test1912 to the dist Follow-up to 70984ce1be4cab6c - docs/LICENSE-MIXING: remove This document is not maintained and I feel that it doesn't provide much value to users anymore (if it ever did). Closes #5955 - [Laramie Leavitt brought this change] http: consolidate nghttp2_session_mem_recv() call paths Previously there were several locations that called nghttp2_session_mem_recv and handled responses slightly differently. Those have been converted to call the existing h2_process_pending_input() function. Moved the end-of-session check to h2_process_pending_input() since the only place the end-of-session state can change is after nghttp2 processes additional input frames. This will likely fix the fuzzing error. While I don't have a root cause the out-of-bounds read seems like a use after free, so moving the nghttp2_session_check_request_allowed() call to a location with a guaranteed nghttp2 session seems reasonable. Also updated a few nghttp2 callsites to include error messages and added a few additional error checks. Closes #5648 - HISTORY: mention alt-svc added in 2019 ... and make 1996 the first year subtitle - base64: also build for pop3 and imap Follow-up to the fix in 20417a13fb8f83 Reported-by: Michael Olbrich Fixes #5937 Closes #5948 - base64: enable in build with SMTP The oauth2 support is used with SMTP and it uses base64 functions. Reported-by: Michael Olbrich Fixes #5937 Closes #5938 - curl_mime_headers.3: fix the example's use of curl_slist_append Reported-by: sofaboss on github Fixes #5942 Closes #5943 - lib583: fix enum mixup grrr the previous follow-up to 17fcdf6a31 was wrong - libtest: fix build errors Follow-up from 17fcdf6a310d4c8076 - lib: fix -Wassign-enum warnings configure --enable-debug now enables -Wassign-enum with clang, identifying several enum "abuses" also fixed. Reported-by: Gisle Vanem Bug: https://github.com/curl/curl/commit/879007f8118771f4896334731aaca5850a154675#commitcomment-42087553 Closes #5929 - RELEASE-NOTES: synced - [Diven Qi brought this change] url: use blank credentials when using proxy w/o username and password Fixes proxy regression brought in commit ad829b21ae (7.71.0) Fixed #5911 Closes #5914 - travis: add a build using libressl (from git master) The v3.2.1 tag (latest release atm) results in a broken build. Closes #5932 - configure: let --enable-debug set -Wenum-conversion with gcc >= 10 Unfortunately, this option is not detecting the same issues as clang's -Wassign-enum flag, but should still be useful to detect future mistakes. Closes #5930 - openssl: consider ALERT_CERTIFICATE_EXPIRED a failed verification If the error reason from the lib is SSL_R_SSLV3_ALERT_CERTIFICATE_EXPIRED, libcurl will return CURLE_PEER_FAILED_VERIFICATION and not CURLE_SSL_CONNECT_ERROR. This unifies the libcurl return code and makes libressl run test 313 (CRL testing) fine. Closes #5934 - FAQ: refreshed some very old language - cmake: make HTTP_ONLY also disable MQTT ... and alphasort the order of disabling protocols to make it easier to browse. Closes #5931 - libtest: remove lib1541 leftovers Caused automake errors. Follow-up to 8ca54a03ea08a - tests/libtests: remove test 1900 and 2033 We already remove the test files, now remove the libtest codes as well. Follow-up to e50a877df74 Marc Hoersken (7 Sep 2020) - CI/azure: add test number to title for display in analytics To ease identification of tests the test number is added to the test case title in order to have it on the Azure DevOps Analytics pages and reports which currently do not show it. Bump test case revision to make Azure DevOps update titles. Closes #5927 Daniel Stenberg (6 Sep 2020) - altsvc: clone setting in curl_easy_duphandle The cache content is not duplicated, like other caches, but the setting and specified file name are. Test 1908 is extended to verify this somewhat. Since the duplicated handle gets the same file name, the test unfortunately overwrites the same file twice (with different contents) which makes it hard to check automatically. Closes #5923 - test1541: remove since it is a known bug A shared connection cache is not thread-safe is a known issue. Stop testing this until we believe this issue is addressed. Reduces occasional test failures we don't care about. The test code in lib1541.c is left in git to allow us to restore it when we get to fix this. Closes #5922 - tests: remove pipelining tests Remove the tests 530, 584, 1900, 1901, 1902, 1903 and 2033. They were previously disabled. The Pipelining code was removed from curl in commit 2f44e94efb3df8e, April 2019. Closes #5921 - curl: retry delays in parallel mode no longer sleeps blocking The previous sleep for retries would block all other concurrent transfers. Starting now, the retry will instead be properly marked to not get restarted until after the delay time but other transfers can still continue in the mean time. Closes #5917 - curl:parallel_transfers: make sure retry readds the transfer Reported-by: htasta on github Fixes #5905 Closes #5917 - build: drop support for building with Watcom These files are not maintained, they seem to have no users, Watcom compilers look like not having users nor releases anymore. Closes #5918 - winbuild/rundebug.cmd: remove Seems to have been added by mistake? Not included in dists. Closes #5919 - curl: in retry output don't call all problems "transient" ... because when --retry-all-errors is used, the error isn't necessarily transient at all. Closes #5916 - easygetopt: pass a valid enum to avoid compiler warning "integer constant not in range of enumerated type 'CURLoption'" Reported-by: Gisle Vanem Bug: https://github.com/curl/curl/commit/6ebe63fac23f38df911edc348e8ccc72280f9434#commitcomment-42042843 Closes #5915 - [Emil Engler brought this change] tests: Add tests for new --help This commit is a part of "--help me if you can" Closes #5680 - [Emil Engler brought this change] tool: update --help with categories This commit is a part of "--help me if you can" Closes #5680 - [Emil Engler brought this change] docs: add categories to all cmdline opts Adapted gen.pl with 'listcats' This commit is a part of "--help me if you can" Closes #5680 - RELEASE-NOTES: synced - [ihsinme brought this change] connect.c: remove superfluous 'else' in Curl_getconnectinfo Closes #5912 - [Samuel Marks brought this change] CMake: remove explicit `CMAKE_ANSI_CFLAGS` This variable was removed from cmake in commit https://gitlab.kitware.com/cmake/cmake/commit/5a834b0bb0bc288. A later CMake commit removes the variable from the tests, claiming that it was removed in CMake 2.6 Reviewed-By: Peter Wu Closes #5439 - [cbe brought this change] libssh2: pass on the error from ssh_force_knownhost_key_type Closes #5909 - scripts/delta: add diffstat summary ... and make output more table-like - [Martin Bašti brought this change] http_proxy: do not crash with HTTPS_PROXY and NO_PROXY set ... in case NO_PROXY takes an effect Without this patch, the following command crashes: $ GIT_CURL_VERBOSE=1 NO_PROXY=github.com HTTPS_PROXY=https://example.com \ git clone https://github.com/curl/curl.git Minimal libcurl-based reproducer: #include <curl/curl.h> int main() { CURL *curl = curl_easy_init(); if(curl) { CURLcode ret; curl_easy_setopt(curl, CURLOPT_URL, "https://github.com/"); curl_easy_setopt(curl, CURLOPT_PROXY, "example.com"); /* set the proxy type */ curl_easy_setopt(curl, CURLOPT_PROXYTYPE, CURLPROXY_HTTPS); curl_easy_setopt(curl, CURLOPT_NOPROXY, "github.com"); curl_easy_setopt(curl, CURLOPT_VERBOSE, 1L); ret = curl_easy_perform(curl); curl_easy_cleanup(curl); return ret; } return -1; } Assisted-by: Kamil Dudka Bug: https://bugzilla.redhat.com/1873327 Closes #5902 - travis: add a CI job with openssl3 (from git master) Closes #5908 - openssl: avoid error conditions when importing native CA The code section that is OpenSSL 3+ specific now uses the same logic as is used in the version < 3 section. It caused a compiler error without it. Closes #5907 - setopt: avoid curl_ on local variable Closes #5906 - mqtt.c: avoid curl_ prefix on local variable Closes #5906 - wildcard: strip "curl_" prefix from private symbols Closes #5906 - vtls: make it 'struct Curl_ssl_session' Use uppercase C for internal symbols. Closes #5906 - curl_threads: make it 'struct Curl_actual_call' Internal names should not be prefixed "curl_" Closes #5906 - schannel: make it 'struct Curl_schannel*' As internal global names should use captical C. Closes #5906 - hash: make it 'struct Curl_hash' As internal global names should use captical C. Closes #5906 - llist: make it "struct Curl_llist" As internal global names should use captical C. Closes #5906 Marc Hoersken (2 Sep 2020) - telnet.c: depend on static requirement of WinSock version 2 Drop dynamic loading of ws2_32.dll and instead rely on the imported version which is now required to be at least 2.2. Reviewed-by: Marcel Raad Reviewed-by: Jay Satiro Reviewed-by: Daniel Stenberg Reviewed-by: Viktor Szakats Closes #5854 - win32: drop support for WinSock version 1, require version 2 IPv6, telnet and now also the multi API require WinSock version 2 which is available starting with Windows 95. Therefore we think it is time to drop support for version 1. Reviewed-by: Marcel Raad Reviewed-by: Jay Satiro Reviewed-by: Daniel Stenberg Reviewed-by: Viktor Szakats Follow up to #5634 Closes #5854 - select: align poll emulation to return all relevant events The poll emulation via select already consumes POLLRDNORM, POLLWRNORM and POLLRDBAND as input events. Therefore it should also return them as output events if signaled. Also fix indentation in input event handling block. Assisted-by: Jay Satiro Reviewed-by: Daniel Stenberg Replaces #5852 Closes #5883 - CI/azure: MQTT is now enabled by default Reviewed-by: Daniel Stenberg Follow up to #5858 Closes #5903 Daniel Stenberg (2 Sep 2020) - copyright.pl: ignore buildconf - test971: show test mismatches "inline" - lib/Makefile.am: bump VERSIONINFO due to new functions ... we're generally bad at this, but we are adding new functions for this release. Closes #5899 - optiontable: use DEBUGBUILD Follow-up to commit 6e18568ba38 (#5877) - cmdline-opts/gen.pl: generate nicer "See Also" in curl.1 If there are more than two items in the list, use commas for all but the last separator which is set to 'and'. Reads better. Closes #5898 - curl.1: add see also no-progress-meter on two spots Ref: #5894 Closes #5897 - RELEASE-NOTES: synced - mqtt: enable by default No longer considered experimental. Closes #5858 - [Michael Baentsch brought this change] tls: add CURLOPT_SSL_EC_CURVES and --curves Closes #5892 - url: remove funny embedded comments in Curl_disonnect calls - [Chris Paulson-Ellis brought this change] conn: check for connection being dead before reuse Prevents incorrect reuse of an HTTP connection that has been prematurely shutdown() by the server. Partial revert of 755083d00deb16 Fixes #5884 Closes #5893 Marc Hoersken (29 Aug 2020) - buildconf: exec autoreconf to avoid additional process Also make buildconf exit with the return code of autoreconf. Reviewed-by: Daniel Stenberg Follow up to #5853 Closes #5890 - CI/azure: no longer ignore results of test 1013 Follow up to #5771 Closes #5889 - docs: add description about CI platforms to CONTRIBUTE.md Reviewed-by: Daniel Stenberg Reviewed-by: Marcel Raad Reviewed-by: Jay Satiro Closes #5882 Daniel Stenberg (29 Aug 2020) - tests/getpart: use MIME::Base64 instead of home-cooked Since we already use the base64 package since a while back, we can just as well switch to that here too. It also happens to use the exact same function name, which otherwise causes a run-time warning. Reported-by: Marc Hörsken Fixes #5885 Closes #5887 Marcel Raad (29 Aug 2020) - ntlm: fix condition for curl_ntlm_core usage `USE_WINDOWS_SSPI` without `USE_WIN32_CRYPTO` but with any other DES backend is fine, but was excluded before. This also fixes test 1013 as the condition for SMB support in configure.ac didn't match the condition in the source code. Now it does. Fixes https://github.com/curl/curl/issues/1262 Closes https://github.com/curl/curl/pull/5771 - AppVeyor: switch 64-bit Schannel Debug CMake builds to Unicode The Schannel builds are the most useful to verify as they make the most use of the Windows API. Classic MinGW doesn't support Unicode at all, only MinGW-w64 and MSVC do. Closes https://github.com/curl/curl/pull/5843 - CMake: add option to enable Unicode on Windows As already existing for winbuild. Closes https://github.com/curl/curl/pull/5843 Marc Hoersken (29 Aug 2020) - select: simplify return code handling for poll and select poll and select already return -1 on error according to POSIX, so there is no need to perform a <0 to -1 conversion in code. Also we can just use one check with <= 0 on the return code. Assisted-by: Daniel Stenberg Reviewed-by: Jay Satiro Replaces #5852 Closes #5880 Daniel Stenberg (28 Aug 2020) - RELEASE-NOTES: synced - [Jeroen Ooms brought this change] tests: add test1912 with typechecks Validates that gcc-typecheck macros match the new option type API. Closes #5873 - easyoptions: provide debug function when DEBUGBUILD ... not CURLDEBUG as they're not always set in conjunction. Follow-up to 6ebe63fac23f38df Fixes #5877 Closes #5878 Marc Hoersken (28 Aug 2020) - sockfilt: handle FD_CLOSE winsock event on write socket Learn from the way Cygwin handles and maps the WinSock events to simulate correct and complete poll and select behaviour according to Richard W. Stevens Network Programming book. Follow up to #5867 Closes #5879 - multi: handle connection state winsock events Learn from the way Cygwin handles and maps the WinSock events to simulate correct and complete poll and select behaviour according to Richard W. Stevens Network Programming book. Reviewed-by: Jay Satiro Reviewed-by: Marcel Raad Follow up to #5634 Closes #5867 Daniel Stenberg (28 Aug 2020) - Curl_pgrsTime - return new time to avoid timeout integer overflow Setting a timeout to INT_MAX could cause an immediate error to get returned as timeout because of an overflow when different values of 'now' were used. This is primarily fixed by having Curl_pgrsTime() return the "now" when TIMER_STARTSINGLE is set so that the parent function will continue using that time. Reported-by: Ionuț-Francisc Oancea Fixes #5583 Closes #5847 - TLS: fix SRP detection by using the proper #ifdefs USE_TLS_SRP will be true if *any* selected TLS backend can use SRP HAVE_OPENSSL_SRP is defined when OpenSSL can use it HAVE_GNUTLS_SRP is defined when GnuTLS can use it Clarify in the curl_verison_info docs that CURL_VERSION_TLSAUTH_SRP is set if at least one of the supported backends offers SRP. Reported-by: Stefan Strogin Fixes #5865 Closes #5870 - [Dan Kenigsberg brought this change] docs: SSLCERTS: fix English syntax Signed-off-by: Dan Kenigsberg <danken@redhat.com> Closes #5876 - [Alessandro Ghedini brought this change] docs: non-existing macros in man pages As reported by man(1) when invoked as: man --warnings -E UTF-8 -l -Tutf8 -Z <file> >/dev/null Closes #5846 - [Alessandro Ghedini brought this change] curl.1: fix typo invokved -> invoked Closes #5846 - buildconf: invoke 'autoreconf -fi' instead The custom script isn't necessary anymore - but remains for simplicity and just invokes autoreconf. Closes #5853 - [Emil Engler brought this change] lib: make Curl_gethostname accept a const pointer The address of that variable never gets changed, only the data in it so why not make it a "char * const"? Closes #5866 - docs/libcurl: update "Added in" version for curl_easy_option* Follow-up to 6ebe63fac23f38 - scripts: improve the "get latest curl release tag" logic ... by insiting on it matching "^curl-". - configure: added --disable-get-easy-options To allow disabling of the curl_easy_option APIs in a build. Closes #5365 - options: API for meta-data about easy options const struct curl_easyoption *curl_easy_option_by_name(const char *name); const struct curl_easyoption *curl_easy_option_by_id (CURLoption id); const struct curl_easyoption * curl_easy_option_next(const struct curl_easyoption *prev); The purpose is to provide detailed enough information to allow for example libcurl bindings to get option information at run-time about what easy options that exist and what arguments they expect. Assisted-by: Jeroen Ooms Closes #5365 - [Eric Curtin brought this change] HTTP/3: update to OpenSSL_1_1_1g-quic-draft-29 Closes #5871 - RELEASE-NOTES: synced Jay Satiro (26 Aug 2020) - openssl: Fix wincrypt symbols conflict with BoringSSL OpenSSL undefines the conflicting symbols but BoringSSL does not so we must do it ourselves. Reported-by: Samuel Tranchet Assisted-by: Javier Blazquez Ref: https://bugs.chromium.org/p/boringssl/issues/detail?id=371 Ref: https://github.com/openssl/openssl/blob/OpenSSL_1_1_1g/include/openssl/ossl_typ.h#L66-L73 Fixes https://github.com/curl/curl/issues/5669 Closes https://github.com/curl/curl/pull/5857 Daniel Stenberg (26 Aug 2020) - socketpair: allow CURL_DISABLE_SOCKETPAIR ... to completely disable the use of socketpair Closes #5850 - curl_get_line: build only if cookies or alt-svc are enabled Closes #5851 - [fullincome brought this change] schannel: fix memory leak when using get_cert_location The get_cert_location function allocates memory only on success. Previously get_cert_location was able to allocate memory and return error. It wasn't obvious and in this case the memory wasn't released. Fixes #5855 Closes #5860 - [Emil Engler brought this change] git: ignore libtests in 3XXX area Currently the file tests/libtest/lib3010 is not getting ignored by git. This fixes it by adding the 3XXX area to the according .gitignore file. Closes #5859 - [Emil Engler brought this change] doh: add error message for DOH_DNS_NAME_TOO_LONG When this error code was introduced in b6a53fff6c1d07e8a9, it was forgotten to be added in the errors array and doh_strerror function. Closes #5863 - ngtcp2: adapt to the new pkt_info arguments Guidance-by: Tatsuhiro Tsujikawa Closes #5864 - winbuild/README.md: make <options> visible Follow-up to be753add31c2d8c - winbuild: convert the instruction text to README.md Closes #5861 - lib1560: verify "redirect" to double-slash leading URL Closes #5849 Marc Hoersken (25 Aug 2020) - multi: expand pre-check for socket readiness Check readiness of all sockets before waiting on them to avoid locking in case the one-time event FD_WRITE was already consumed by a previous wait operation. More information about WinSock network events: https://docs.microsoft.com/en-us/windows/win32/api/ winsock2/nf-winsock2-wsaeventselect#return-value Closes #5634 - [rcombs brought this change] multi: implement wait using winsock events This avoids using a pair of TCP ports to provide wakeup functionality for every multi instance on Windows, where socketpair() is emulated using a TCP socket on loopback which could in turn lead to socket resource exhaustion. A previous version of this patch failed to account for how in WinSock, FD_WRITE is set only once when writing becomes possible and not again until after a send has failed due to the buffer filling. This contrasts to how FD_READ and FD_OOB continue to be set until the conditions they refer to no longer apply. This meant that if a user wrote some data to a socket, but not enough data to completely fill its send buffer, then waited on that socket to become writable, we'd erroneously stall until their configured timeout rather than returning immediately. This version of the patch addresses that issue by checking each socket we're waiting on to become writable with select() before the wait, and zeroing the timeout if it's already writable. Assisted-by: Marc Hörsken Reviewed-by: Marcel Raad Reviewed-by: Daniel Stenberg Tested-by: Gergely Nagy Tested-by: Rasmus Melchior Jacobsen Tested-by: Tomas Berger Replaces #5397 Reverts #5632 Closes #5634 - select: reduce duplication of Curl_poll in Curl_socket_check Change Curl_socket_check to use select-fallback in Curl_poll instead of implementing it in Curl_socket_check and Curl_poll. Reviewed-by: Daniel Stenberg Reviewed-by: Jay Satiro Replaces #5262 and #5492 Closes #5707 - select: fix poll-based check not detecting connect failure This commit changes Curl_socket_check to use POLLPRI to check for connect failure on the write socket, because POLLPRI maps to fds_err. This is in line with select(2). The select-based socket check correctly checks for connect failures by adding the write socket also to fds_err. The poll-based implementation (which internally can itself fallback to select again) did not previously check for connect failure by using POLLPRI with the write socket. See the follow up commit to this for more information. This commit makes sure connect failures can be detected and handled if HAVE_POLL_FINE is defined, eg. on msys2-devel. Reviewed-by: Daniel Stenberg Reviewed-by: Jay Satiro Replaces #5509 Prepares #5707 - select.h: make socket validation macros test for INVALID_SOCKET With Winsock the valid range is [0..INVALID_SOCKET-1] according to https://docs.microsoft.com/en-us/windows/win32/winsock/socket-data-type-2 Reviewed-by: Jay Satiro Reviewed-by: Marcel Raad Reviewed-by: Daniel Stenberg Closes #5760 Daniel Stenberg (24 Aug 2020) - docs: --output-dir is added in 7.73.0, nothing else Follow-up to 5620d2cc78c0 - curl: add --output-dir Works with --create-dirs and with -J Add test 3008, 3009, 3011, 3012 and 3013 to verify. Closes #5637 - configure: fix pkg-config detecting wolfssl When amending the include path with "/wolfssl", this now properly strips off all whitespace from the path variable! Previously this would lead to pkg-config builds creating bad command lines. Closes #5848 - [Michael Musset brought this change] sftp: add the option CURLKHSTAT_FINE_REPLACE Replace the old fingerprint of the host with a new. Closes #5685 - RELEASE-NOTES: synced The next release is now to become 7.73.0 - checksrc: verify do-while and spaces between the braces Updated mprintf.c to comply Closes #5845 - curl: support XDG_CONFIG_HOME to find .curlrc Added test433 to verify. Updated documentation. Reviewed-by: Jay Satiro Suggested-by: Eli Schwartz Fixes #5829 Closes #5837 - etag: save and use the full received contents ... which makes it support weak tags and non-standard etags too! Added test case 347 to verify blank incoming ETag: Fixes #5610 Closes #5833 - setopt: if the buffer exists, refuse the new BUFFERSIZE The buffer only exists during transfer and then we shouldn't change the size (the setopt is not documented to work then). Reported-by: Harry Sintonen Closes #5842 - [COFFEETALES brought this change] sftp: add new quote commands 'atime' and 'mtime' Closes #5810 - CURLE_PROXY: new error code Failures clearly returned from a (SOCKS) proxy now causes this return code. Previously the situation was not very clear as what would be returned and when. In addition: when this error code is returned, an application can use CURLINFO_PROXY_ERROR to query libcurl for the detailed error, which then returns a value from the new 'CURLproxycode' enum. Closes #5770 - runtests: make cleardir() erase dot files too Because test cases might use dot files. Closes #5838 - KNOWN_BUGS: 'no_proxy' string-matches IPv6 numerical addreses Also: the current behavior is now documented in the curl.1 and CURLOPT_NOPROXY.3 man pages. Reported-by: Andrew Barnes Closes #5745 Closes #5841 Viktor Szakats (22 Aug 2020) - Makefile.m32: add ability to override zstd libs [ci skip] Similarly to brotli, where this was already possible. E.g. it allows to link zstd statically to libcurl.dll. Ref: https://github.com/curl/curl-for-win/issues/12 Ref: https://github.com/curl/curl-for-win/commit/d9b266afd2e5d3f5604483010ef62340b5918c89 Closes https://github.com/curl/curl/pull/5840 Daniel Stenberg (21 Aug 2020) - runtests: avoid 'fail to start' repeated messages in attempt loops Closes #5834 - runtests: clear pid variables when failing to start a server ... as otherwise the parent doesn't detect the failure and believe it actually worked to start. Reported-by: Christian Weisgerber Bug: https://curl.haxx.se/mail/lib-2020-08/0018.html Closes #5834 - TODO: Virtual external sockets Closes #5835 - [Don J Olmstead brought this change] dist: add missing CMake Find modules to the distribution Closes #5836 - RELEASE-NOTES: synced ... and version bumped to 7.72.1 - tls: provide the CApath verbose log on its own line ... not newline separated from the previous line. This makes it output asterisk prefixed properly like other verbose putput! Reported-by: jmdavitt on github Fixes #5826 Closes #5827 Version 7.72.0 (19 Aug 2020) Daniel Stenberg (19 Aug 2020) - RELEASE-NOTES: synced The curl 7.72.0 release - THANKS: add names from curl 7.72.0 release Jay Satiro (18 Aug 2020) - KNOWN_BUGS: Schannel TLS 1.2 handshake bug in old Windows versions Reported-by: plujon@users.noreply.github.com Closes https://github.com/curl/curl/issues/5488 Daniel Stenberg (17 Aug 2020) - Curl_easy: remember last connection by id, not by pointer CVE-2020-8231 Bug: https://curl.haxx.se/docs/CVE-2020-8231.html Reported-by: Marc Aldorasi Closes #5824 - examples/rtsp.c: correct the copyright year - RELEASE-PROCEDURE.md: add more future release dates - [H3RSKO brought this change] docs: change "web site" to "website" According to wikipedia: While "web site" was the original spelling, this variant has become rarely used, and "website" has become the standard spelling Closes #5822 - [Bevan Weiss brought this change] CMake: don't complain about missing nroff The curl_nroff_check() was always being called, and complaining if *NROFF wasn't found, even when not making the manual. Only check for nroff (and complain) if actually making the manual Closes #5817 - [Brian Inglis brought this change] libtest/Makefile.am: add -no-undefined for libstubgss for Cygwin copy the LDFLAGS approach for adding same option with `libhostname` in `libtest/Makefile.am`: - init `libstubgss_la_LDFLAGS_EXTRA` variable, - add option to variable inside conditional, - use variable in `libstubgss_la_LDFLAGS` Fixes #5819 Closes #5820 - docs: clarify MAX_SEND/RECV_SPEED functionality ... in particular what happens if the maximum speed limit is set to a value that's smaller than the transfer buffer size in use. Reported-by: Tomas Berger Fixes #5788 Closes #5813 - test1140: compare stdout To make problems more immediately obvious when tests fail. Closes #5814 - asyn-ares: correct some bad comments Closes #5812 - [Emil Engler brought this change] docs: Add video link to docs/CONTRIBUTE.md Closes #5811 - curl-config: ignore REQUIRE_LIB_DEPS in --libs output Fixes a curl-config issue on cygwin by making sure REQUIRE_LIB_DEPS is not considered for the --libs output. Reported-by: ramsay-jones on github Assisted-by: Brian Inglis and Ken Brown Fixes #5793 Closes #5808 - copyright: update/correct the year range on a few files - scripts/copyright.pl: ignore .muse files - [Emil Engler brought this change] multi: Remove 10-year old out-commented code The code hasn't been touched since 2010-08-18 Closes #5805 - KNOWN_BUGS: A shared connection cache is not thread-safe Closes #4915 Closes #5802 - CONTRIBUTE: extend git commit message description In particular how the first line works. Closes #5803 - RELEASE-NOTES: synced - [Stefan Yohansson brought this change] transfer: move retrycount from connect struct to easy handle This flag was applied to the connection struct that is released on retry. These changes move the retry counter into Curl_easy struct that lives across retries and retains the new connection. Reported-by: Cherish98 on github Fixes #5794 Closes #5800 - libssh2: s/ssherr/sftperr/ The debug output used ssherr instead of sftperr which not only outputs the wrong error code but also casues a warning on Windows. Follow-up to 7370b4e39f1 Reported-by: Gisle Vanem Bug: https://github.com/curl/curl/commit/7370b4e39f1390e701f5b68d910c619151daf72b#r41334700 Closes #5799 - ftp: don't do ssl_shutdown instead of ssl_close The shutdown function is for downgrading a connection from TLS to plain, and this is not requested here. Have ssl_close reset the TLS connection state. This partially reverts commit f002c850d98d Reported-by: Rasmus Melchior Jacobsen Reported-by: Denis Goleshchikhin Fixes #5797 Marc Hoersken (9 Aug 2020) - CI/azure: fix test outcome values and use latest API version This makes sure that tests ignored or skipped are not shown just in the category "Other", but with their correct state. Closes #5796 - CI/azure: show runtime stats to investigate slowness Also avoid naming conflict of TFLAGS env and tflags variables. Closes #5776 Daniel Stenberg (8 Aug 2020) - TLS naming: fix more Winssl and Darwinssl leftovers The CMake option is now called CMAKE_USE_SCHANNEL The winbuild flag is USE_SCHANNEL The CI jobs and build scripts only use the new names and the new name options Tests now require 'Schannel' (when necessary) Closes #5795 - smtp_parse_address: handle blank input string properly Closes #5792 - runtests: run the DICT server on a random port number Removed support for -b (base port number) Closes #5783 - RELEASE-NOTES: synced - runtests: move the TELNET server to a dynamic port Rename the port variable to TELNETPORT to better match the existing pattern. Closes #5785 - ngtcp2: adapt to error code rename Closes #5786 - runtests: move the smbserver to use a dynamic port number Closes #5782 - runtests: run the http2 tests on a random port number Closes #5779 - gtls: survive not being able to get name/issuer Closes #5778 - runtests: move the gnutls-serv tests to a dynamic port Affects test 320, 321, 322 and 324. Closes #5778 - runtests: support dynamicly base64 encoded sections in tests This allows us to make test cases to use base64 at run-time and still use and verify information determined at run-time, such as the IMAP test server's port number in test 842. This change makes 12 tests run again that basically never ran since we moved to dynamic port numbers. ftpserver.pl is adjusted to load test instructions and test number from the preprocessed test file. FILEFORMAT.md now documents the new base64 encoding syntax. Reported-by: Marcel Raad Fixes #5761 Closes #5775 - curl.1: add a few missing valid exit codes 93 - 96 can be returned as well. Closes #5777 - TODO: Use multiple parallel transfers for a single download Closes #5774 - TODO: Set the modification date on an uploaded file Closes #5768 - [Thomas M. DuBuisson brought this change] CI: Add muse CI config Closes #5772 - [Thomas M. DuBuisson brought this change] travis/script.sh: fix use of `-n' with unquoted envvar Shellcheck tells us "-n doesn't work with unquoted arguments. quote or use [[ ]]." And testing shows: ``` docker run --rm -it ubuntu bash root@fe85ce156856:/# [ -n $DOES_NOT_EXIST ] && echo "I ran" I ran root@fe85ce156856:/# [ -n "$DOES_NOT_EXIST" ] && echo "I ran" root@fe85ce156856:/# ``` Closes #5773 - h2: repair trailer handling The previous h2 trailer fix in 54a2b63 was wrong and caused a regression: it cannot deal with trailers immediately when read since they may be read off the connection by the wrong 'data' owner. This change reverts the logic back to gathering all trailers into a single buffer, like before 54a2b63. Reported-by: Tadej Vengust Fixes #5663 Closes #5769 Viktor Szakats (3 Aug 2020) - windows: disable Unix Sockets for old mingw Classic mingw and 10y+ old versions of mingw-w64 don't ship with Windows headers having the typedef necessary for Unix Sockets support, so try detecting these environments to disable this feature. Ref: https://sourceforge.net/p/mingw-w64/mingw-w64/ci/cf6afc57179a5910621215f8f4037d406892072c/ Reviewed-by: Daniel Stenberg Fixes #5674 Closes #5758 Marcel Raad (3 Aug 2020) - test1908: treat file as text Fixes the line endings on Windows. Closes https://github.com/curl/curl/pull/5767 - TrackMemory tests: ignore realloc and free in getenv.c These are only called for WIN32. Closes https://github.com/curl/curl/pull/5767 Daniel Stenberg (3 Aug 2020) - tests/FILEFORMAT.md: mention %HTTP2PORT - RELEASE-NOTES: synced - tlsv1.3.d. only for TLS-using connections ... and rephrase that "not all" TLS backends support it. Closes #5764 - tls-max.d: this option is only for TLS-using connections Ref: #5763 Closes #5764 Marcel Raad (2 Aug 2020) - [Cameron Cawley brought this change] tool_doswin: Simplify Windows version detection Closes https://github.com/curl/curl/pull/5754 - [Cameron Cawley brought this change] win32: Add Curl_verify_windows_version() to curlx Closes https://github.com/curl/curl/pull/5754 - runtests.pl: treat LibreSSL and BoringSSL as OpenSSL This makes the tests that require the OpenSSL feature also run for those two compatible libraries. Closes https://github.com/curl/curl/pull/5762 Daniel Stenberg (1 Aug 2020) - multi: Condition 'extrawait' is always true Reported by Codacy. Reviewed-by: Marcel Raad Closes #5759 Marcel Raad (1 Aug 2020) - openssl: fix build with LibreSSL < 2.9.1 `SSL_CTX_add0_chain_cert` and `SSL_CTX_clear_chain_certs` were introduced in LibreSSL 2.9.1 [0]. [0] https://github.com/libressl-portable/openbsd/commit/0db809ee178457c8170abfae3931d7bd13abf3ef Closes https://github.com/curl/curl/pull/5757 Daniel Stenberg (1 Aug 2020) - [Marc Aldorasi brought this change] multi_remove_handle: close unused connect-only connections Previously any connect-only connections in a multi handle would be kept alive until the multi handle was closed. Since these connections cannot be re-used, they can be marked for closure when the associated easy handle is removed from the multi handle. Closes #5749 - checksrc: invoke script with -D to find .checksrc proper Without the -D command line option, checksrc.pl won't know which directory to load the ".checksrc" file from when building out of the source tree. Reported-by: Marcel Raad Fixes #5715 Closes #5755 - [Carlo Marcelo Arenas Belón brought this change] buildconf: retire ares buildconf invocation no longer needed after 4259d2df7dd95637a4b1e3fb174fe5e5aef81069 - [Carlo Marcelo Arenas Belón brought this change] buildconf: excempt defunct reference to ACLOCAL_FLAGS retired with 09f278121e815028adb24d228d8092fc6cb022aa but kept around as the name is generic enough that it might be in use and relied upon from the environment. - [Carlo Marcelo Arenas Belón brought this change] buildconf: avoid array concatenation in die() reported as error SC2145[1] by shellcheck, but not expected to cause any behavioural differences otherwise. [1] https://github.com/koalaman/shellcheck/wiki/SC2145 Closes #5701 - travis: add ppc64le and s390x builds Closes #5752 Marc Hoersken (31 Jul 2020) - connect: remove redundant message about connect failure Reviewed-by: Daniel Stenberg Closes #5708 - tests/sshserver.pl: fix compatibility with OpenSSH for Windows Follow up to #5721 - CI/azure: install libssh2 for use with msys2-based builds This enables building and running the SFTP tests. Unfortunately OpenSSH for Windows does not support SCP (yet). Reviewed-by: Daniel Stenberg Closes #5721 - CI/azure: increase Windows job timeout once again Avoid aborted jobs due to performance issues on Azure DevOps. Reviewed-by: Daniel Stenberg Reviewed-by: Jay Satiro Closes #5738 Jay Satiro (30 Jul 2020) - TODO: Schannel: 'Add option to allow abrupt server closure' We should offer an option to allow abrupt server closures (server closes SSL transfer without sending a known termination point such as length of transfer or close_notify alert). Abrupt server closures are usually because of misconfigured or very old servers. Closes https://github.com/curl/curl/issues/4427 - url: fix CURLU and location following Prior to this change if the user set a URL handle (CURLOPT_CURLU) it was incorrectly used for the location follow, resulting in infinite requests to the original location. Reported-by: sspiri@users.noreply.github.com Fixes https://github.com/curl/curl/issues/5709 Closes https://github.com/curl/curl/pull/5713 Daniel Stenberg (30 Jul 2020) - RELEASE-NOTES: synced - [divinity76 brought this change] docs: add date of 7.20 to CURLM_CALL_MULTI_PERFORM mentions it helps make it obvious that most developers don't have to care about the CURLM_CALL_MULTI_PERFORM value (last release using it is nearly 11 years old, November 4 2009) Closes #5744 Jay Satiro (29 Jul 2020) - tool_cb_wrt: fix outfile mode flags for Windows - Use S_IREAD and S_IWRITE mode permission flags to create the file on Windows instead of S_IRUSR, S_IWUSR, etc. Windows only accepts a combination of S_IREAD and S_IWRITE. It does not acknowledge other combinations, for which it may generate an assertion. This is a follow-up to 81b4e99 from yesterday, which improved the existing file check with -J. Ref: https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/open-wopen#remarks Ref: https://github.com/curl/curl/pull/5731 Closes https://github.com/curl/curl/pull/5742 Daniel Stenberg (28 Jul 2020) - checksrc: ban gmtime/localtime They're not thread-safe so they should not be used in libcurl code. Explictly enabled when deemed necessary and in examples and tests Reviewed-by: Nicolas Sterchele Closes #5732