Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with Locust creating too many file descriptors #222

Open
4 of 5 tasks
JadKHaddad opened this issue Oct 30, 2021 · 5 comments
Open
4 of 5 tasks

Issue with Locust creating too many file descriptors #222

JadKHaddad opened this issue Oct 30, 2021 · 5 comments
Labels
enhancement New feature or request question Further information is requested support

Comments

@JadKHaddad
Copy link

My actions before raising this issue

Expected Behaviour

Increase the limit of open files in an openfaas function

Current Behaviour

Limit is stuck at 1024

Are you a GitHub Sponsor (Yes/No?)

No

Check at: https://github.com/sponsors/openfaas

  • Yes
  • No

List All Possible Solutions and Workarounds

Increasing the Limit on the whole system.
Building the image with higher limits.

Which Solution Do You Recommend?

Set an Enviroment variable in the yaml file to increase open files limit for a function?

Steps to Reproduce (for bugs)

Context

I am trying to run locust tests with openfaas. Locust tests tend to open alot of files while running.

Your Environment

  • FaaS-CLI version ( Full output from: faas-cli version ):

CLI:
commit: 72816d486cf76c3089b915dfb0b66b85cf096634
version: 0.13.13

Gateway
uri: http://127.0.0.1:8080
version: 0.21.0
sha: 539f0a2c946ee6096ef2b6ab26f81295eed49d54

Provider
name: faasd
orchestration: containerd
version: 0.13.0
sha: 12ada59

  • Docker version docker version (e.g. Docker 17.0.05 ):
    20.10.7
  • Are you using OpenFaaS on Kubernetes or faasd?
    faasd
  • Operating System and version (e.g. Linux, Windows, MacOS):
    Linux
    No LSB modules are available.
    Distributor ID: Ubuntu
    Description: Ubuntu 20.04.3 LTS
    Release: 20.04
    Codename: focal
  • Code example or link to GitHub repo or gist to reproduce problem:
    Here is a simple python flask function example just to check the ulimits:
import subprocess

def handle(req):
	p = subprocess.Popen(req,shell=True, stdout=subprocess.PIPE)
	out, err = p.communicate()
	return "out: " + str(out) + " | err: " + str(err)

when I run it with "ulimit -n 2000" for example, I get an error (not permitted), but when I run it with a number lower than 1024 it works. For example "ulimit -n 100; ulimit -n", output is : 100, and it works. But I want to make the limits higher.

Next steps

You may join Slack for community support.

@kevin-lindsay-1
Copy link
Sponsor

Where do you think this limit is coming from?

Do you know if this limit is a default set somewhere, or is a default in Go or Kubernetes, or another runtime environment?

@JadKHaddad
Copy link
Author

I think this limit is set by containerd and I found no way to set it higher

@alexellis alexellis transferred this issue from openfaas/faas Nov 8, 2021
@alexellis
Copy link
Member

/set title: Issue with Locust creating too many file descriptors

@derek derek bot changed the title Open files limits for an Openfaas-function Issue with Locust creating too many file descriptors Nov 8, 2021
@alexellis
Copy link
Member

/add label: support,question,enhancement

@derek derek bot added enhancement New feature or request question Further information is requested support labels Nov 8, 2021
@alexellis
Copy link
Member

I expect a ulimit of 1024 is quite standard, and may even be too generous for what a function typically is meant to do.

Apparently it can be configured, but I am not sure this is something that makes sense as part of the function spec or as an annotation.

containerd/containerd#3150

Perhaps you'd like to follow our patching guide to building faasd and hack on the source code to insert whatever ulimit you prefer for all your containers?

https://github.com/openfaas/faasd/blob/master/docs/PATCHES.md

Here is the container spec, where I imagine you can override the ulimit value with whatever you want: https://github.com/openfaas/faasd/blob/master/pkg/provider/handlers/deploy.go#L159

Alex

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested support
Projects
None yet
Development

No branches or pull requests

3 participants