curl 를 이용하여 잦은 https 통신으로 데이터 연동작업을 할 경우 메모리가 순차적으로 증가하는 모습을 볼 수 있습니다.

curl-memory-leak

slab 메모리지 중 dentry cache 의 경우는 access() 시스템 콜을 통해 생성되기 때문에 curl 로 많은 양의 https 호출이 발생하면

access() 시스템 콜이 발생하여 dentry cache 역시 증가합니다.

2014년 말 패치된 nss-softoken 라이브러리에서는 sdb_init() 함수 내부에서 NSS_SDB_USE_CACHE 환경 변수가 설정되어

sdb_measureAccess() 함수를 호출하지 않도록 되어 있습니다.

패치된 버전은 다음과 같습니다.

nss-patch-version-check

1. Install or update check version nss-softokn, libcurl, libcurl-devel and openssl from base repo

# rpm -qa

grep nss
nss-softokn-3.16.2.3-14.2.el7_2.x86_64
nss-3.21.0-9.el7_2.x86_64

# rpm -qa

grep libcurl
libcurl-7.29.0-25.el7.centos.x86_64
libcurl-devel-7.29.0-25.el7.centos.x86_64

# rpm -qa

grep openssl
openssl-libs-1.0.1e-51.el7_2.5.x86_64
openssl-1.0.1e-51.el7_2.5.x86_64
openssl-devel-1.0.1e-51.el7_2.5.x86_64

2. Save the following example.c  into a file and compile with:

# gcc -o example example.c -lcurl

#include <stdio.h>
#include <curl/curl.h>

int leak(void){
CURL *curl;
CURLcode res;

curl_global_init(CURL_GLOBAL_DEFAULT);

curl = curl_easy_init();
if(curl) {
curl_easy_setopt(curl, CURLOPT_URL, “https://example.com/”);
res = curl_easy_perform(curl);

if(res != CURLE_OK)
fprintf(stderr, “curl_easy_perform() failed: %s\n”,
curl_easy_strerror(res));

curl_easy_cleanup(curl);
}

//this should clear the ssl shared memory, but instead it just leaks!
curl_global_cleanup();

return 0;
}

int main(void){
while(1){
leak();
}
}

3. run

# nohup ./example &

(stop with pkill example)

4. run top and check memory change

top – 15:51:45 up 3 days, 1:23, 3 users, load average: 0.25, 0.32, 0.32
Tasks: 114 total, 1 running, 113 sleeping, 0 stopped, 0 zombie
%Cpu(s): 11.4 us, 4.0 sy, 0.0 ni, 84.6 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
KiB Mem : 3882704 total, 1993232 free, 192176 used, 1697296 buff/cache
KiB Swap: 2621436 total, 2621436 free, 0 used. 3368812 avail Mem

 

5. how to fix

# echo “export NSS_SDB_USE_CACHE=YES” >> /etc/profile.d/plura.sh

# source /etc/profile.d/plura.sh

6. verify

# echo $NSS_SDB_USE_CACHE

YES

7. References

  • https://bugs.centos.org/view.php?id=9391
  • https://brunch.co.kr/@alden/28
  • https://www.splyt.com/blog/2014-05-16-optimizing-aws-nss-softoken
  • https://rhn.redhat.com/errata/RHBA-2014-1378.html