Unexpected setuid call by non-sudo events contain no details

[manojdeshmukh45]

In the falco alerts "Unexpected setuid call by non-sudo events contain no details" is comming all

00:40:51.862247143: Notice Unexpected setuid call by non-sudo, non-root program (container_name=host user= user_loginuid=-1 cur_uid=4294967295 parent= command= pid=1224253 uid= container_id=host image=) k8s.ns= k8s.pod= container=host

Environment

• Falco version:
  {"default_driver_version":"4.0.0+driver","driver_api_version":"3.0.0","driver_schema_version":"2.0.0","engine_version":"16","falco_version":"0.34.1","libs_version":"0.10.4","plugin_api_version":"2.0.0"}
• System info:
  Falco version: 0.34.1 (x86_64)
• Cloud provider or hardware configuration: GCP
• OS: NAME="Debian GNU/Linux"
• Kernel:
  Linux 5.10.176+
• Installation method:

[Andreagit97]

thank you for reporting this! do you have a repro to trigger this rule with all <NA> fields?
Please note that since you are on the host all containers and k8s data are <NA> and this is fine.

[cristiklein]

Hello,

We bumped into this issue quite a few times and decided to make a deep-dive.

As an example, we get the following log lines from some staging Kubernetes cluster:

09:06:37.105020032: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=sh proc_exepath=/bin/sh parent=xargs command=sh -c redis-cli -p 6379 info replication \| grep master_sync_in_progress:1 \| tr -d "\r" \| tr -d "\n"  terminal=0 exe_flags=<NA> k8s.ns=redis-system k8s.pod=rfr-redis-cluster-2 container=73897ca9fa41)
09:06:37.105848477: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=tr proc_exepath=/usr/bin/tr parent=sh command=tr -d \n terminal=0 exe_flags=<NA> k8s.ns=redis-system k8s.pod=rfr-redis-cluster-2 container=73897ca9fa41)

What's really weird is that it almost looks like all coreutils trigger this event. 😄

We looked into the container image and the source of the events seems to be the fact that busybox does setuid, even if not needed. (See busybox code.)

This can also be verified as follows:

1. Spin up an Ubuntu 22.04.3 LTS machine. Your regular development laptop works just fine.

   $ cat /etc/lsb-release 
   DISTRIB_ID=Ubuntu
   DISTRIB_RELEASE=22.04
   DISTRIB_CODENAME=jammy
   DISTRIB_DESCRIPTION="Ubuntu 22.04.3 LTS"

2. Make sure you have Docker working:

   $ docker --version
   Docker version 24.0.5, build ced0996

3. Run the setuids.bt tool from bpftrace:

   $ docker run -ti -v /usr/src:/usr/src:ro        -v /lib/modules/:/lib/modules:ro        -v /sys/kernel/debug/:/sys/kernel/debug:rw        --net=host --pid=host --privileged        quay.io/iovisor/bpftrace:v0.18.1        setuids.bt
   Attaching 7 probes...
   Tracing setuid(2) family syscalls. Hit Ctrl-C to end.
   TIME     PID    COMM             UID    SYSCALL   ARGS (RET)
   10:06:57 3023954 ping             1000   setuid    uid=1000 (0)
   [...]

4. Ignore all the setuid's happening on your system. 😄

5. While you're at it, install Falco as describe here. Pick Modern eBPF driver. You can check that Falco is running as follows:

   $ journalctl --unit falco-modern-bpf.service
   sep 01 11:50:15 minitruee systemd[1]: Started Falco: Container Native Runtime Security with modern e>
   sep 01 11:50:16 minitruee falco[3007632]: Falco version: 0.35.1 (x86_64)
   sep 01 11:50:16 minitruee falco[3007632]: Falco initialized with configuration file: /etc/falco/falc>
   sep 01 11:50:16 minitruee falco[3007632]: Loading rules from file /etc/falco/falco_rules.yaml
   sep 01 11:50:16 minitruee falco[3007632]: Loading rules from file /etc/falco/falco_rules.local.yaml
   sep 01 11:50:16 minitruee falco[3007632]: The chosen syscall buffer dimension is: 8388608 bytes (8 M>
   sep 01 11:50:16 minitruee falco[3007632]: Starting health webserver with threadiness 8, listening on>
   sep 01 11:50:16 minitruee falco[3007632]: Loaded event sources: syscall
   sep 01 11:50:16 minitruee falco[3007632]: Enabled event sources: syscall
   sep 01 11:50:16 minitruee falco[3007632]: Opening 'syscall' source with modern BPF probe.
   sep 01 11:50:16 minitruee falco[3007632]: One ring buffer every '2' CPUs.
   [...]

6. In another terminal, run a busybox image:

   $ docker run --rm --user 1000 busybox cat
   [exists immediately]

7. Observe the output of setuids.bt:

   10:08:13 3026177 cat              1000   setuid    uid=1000 (0)
   

Mkey, so I guess busybox likes to setuid to itself. Doesn't feel like a big security threat. What does Falco say?

I tried both the Kmod and Modern eBPF driver. The busybox command above generates the following warning:

sep 01 12:08:13 minitruee falco[3007632]: 12:08:13.458396596: Notice Unexpected setuid call by non-sudo, non-root program (user=<NA> user_loginuid=-1 cur_uid=1000 parent=<NA> command=cat pid=3026177 uid=<NA> container_id=3401b97a71fd image=<NA>)

Judging by the timestamp, both Falco and setuids.bt report the same event (my laptop is in UTC+2, hence the 2 hours discrepancy).

However, Falco does not seem to report args.uid correctly. I tried both the Modern eBPF and the Kmod driver, and they both behave the same. If Falco reported args.uid correctly, it would be trivial to ignore events like args.uid == cur_uid.

Any clues where to dig further on why Falco reports args.uid incorrectly?

[Andreagit97]

@cristiklein thank you very much for the very detailed investigation!

Mkey, so I guess busybox likes to setuid to itself. Doesn't feel like a big security threat. What does Falco say?

Yeah it seems weird, we need to investigate more on this, thanks for the pointer!

However, Falco does not seem to report args.uid correctly. I tried both the Modern eBPF and the Kmod driver, and they both behave the same. If Falco reported args.uid correctly, it would be trivial to ignore events like args.uid == cur_uid.
Any clues where to dig further on why Falco reports args.uid incorrectly?

Thanks to your repro probably we will discover why, it could take some time since we are in the middle of a release but for sure is something we want to investigate! Thank you again for all the precious data!

[cristiklein]

Thanks @Andreagit97 for your quick response.

I discovered one more thing that will hopefully help narrow down the culprit.

If I change the rule's output and add:

args=%evt.args uid_via_index=%evt.arg[0] info=%evt.info uid_via_rawargs=%evt.rawarg.uid

Then Falco's output becomes:

14:30:28.999973619: Notice Unexpected setuid call by non-sudo, non-root program (user= user_loginuid=-1 cur_uid=1000 parent= command=cat pid=3085201 uid= container_id=d9f922a096f2 image= args=uid=1000() uid_via_index= info=uid=1000() uid_via_rawargs=)

Interestingly, it seems like uid=1000 is correctly captured in evt.args and evt.info, but is somehow not transferred into evt.arg. 🤷

Good luck with the release! Looking forward to it.

[manojbamboobox]

Though I added the above to output, i'm getting blank.

[cristiklein]

Though I added the above to output, i'm getting blank.

What do you mean? I can clearly see uid=<NA> and info=uid=0 in your screenshot above. Seems like your offending process is running with uid 0, which is available to Falco in the lower layers, but not properly propagated into the event args.

[manojdeshmukh45]

What to do for this issue. Im getting 0.

…

On Wed, 18 Oct, 2023, 4:49 pm Cristian Klein, ***@***.***> wrote: Though I added the above to output, i'm getting blank. [image: image] <https://user-images.githubusercontent.com/138666148/276120551-f25152c3-eb09-4173-84f1-0806ca985d20.png> What do you mean? I can clearly see uid=<NA> and info=uid=0 in your screenshot above. Seems like your offending process is running with uid 0, which is available to Falco in the lower layers, but not properly propagated into the event args. — Reply to this email directly, view it on GitHub <#1630>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIRWHSEKLC5T466UAT5UL53X763KZAVCNFSM6AAAAAA3CNKZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRYGIZTMOJRGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[cristiklein]

What version of Falco are you using? (I used Falco 0.35.1.)

[manojdeshmukh45]

{"default_driver_version":"6.0.1+driver","driver_api_version":"5.0.0","driver_schema_version":"2.0.0","engine_version":"26","falco_version":"0.36.0","libs_version":"0.13.1","plugin_api_version":"3.1.0"}

…

On Wed, Oct 18, 2023 at 7:08 PM Cristian Klein ***@***.***> wrote: What version of Falco are you using? (I used Falco 0.35.1.) — Reply to this email directly, view it on GitHub <#1630>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIRWHSE7WC55JTSATOMPDGDX77LVZAVCNFSM6AAAAAA3CNKZD2VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRYGQ4DCMZSGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- Thanks & Regards Manoj Deshmukh Cybersecurity Analyst CEH Practical v12 | CompTIA CySA+

[cristiklein]

I re-ran my test with Falco 0.36.1. In the meantime, the "Unexpected setuid call" rule was moved to falco-incubating_rules.yaml, so I had to do the following first:

$ sudo wget https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml -O /etc/falco/rules.d/falco-incubating_rules.yaml

# Falco 0.36.1 seem to not like the `required_engine_version` key.
$ sudo sed -i -e 's@- required_engine_version@#- required_engine_version@' /etc/falco/rules.d/falco-incubating_rules.yaml

# Also print %evt.args
sudo sed -i -e 's@output: Unexpected setuid.*$@& args=%evt.args@' /etc/falco/rules.d/falco-incubating_rules.yaml

Then I ran a busybox image:

$ docker run --rm --user 1000 busybox cat
[exits immediately]

Falco output the following:

okt 19 08:54:39 minitruee falco[1211656]: 08:54:39.141114250: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=cat proc_exepath=/bin/cat parent=containerd-shim command=cat terminal=0 exe_flags=<NA> container_id=556312d7d5d0 container_name=<NA>) args=uid=1000(<NA>)

As you can see %evt.args contains the correct uid, but `%evt.arg.uid% does not.

So it seems like there is a bug in the logic that populates %evt.arg.

What do you think?

[FedeDP]

@cristiklein can you try using evt.rawarg.uid instead?

[FedeDP]

So, i just tried to repro this with sinsp-example from libs master, that is a small executable on top of libsinsp, and it seems i receive events correctly:

sudo ./libsinsp/examples/sinsp-example -m -f "evt.type=setuid and evt.dir=>" -o "%evt.type %evt.args arg=%evt.arg.uid user:%user.name raw=%evt.rawarg.uid"
setuid uid=0(root)  arg=root user:root raw=
setuid uid=1000(federico)  arg=federico user:root raw=
setuid uid=1000(federico)  arg=federico user:federico raw=

This was captured while doing in another terminal: docker run --rm --user 1000 busybox cat

Notes:

• rawarg is empty :/
• if i move the empty rawarg before anything else, it won't print anything after it:

 sudo ./libsinsp/examples/sinsp-example -m -f "evt.type=setuid and evt.dir=>" -o "raw=%evt.rawarg.uid %evt.type %evt.args arg=%evt.arg.uid user:%user.name"
raw=
raw=
raw=

/cc @jasondellaluce for these 2 issues, perhaps you got some ideas?

[cristiklein]

@cristiklein can you try using evt.rawarg.uid instead?

Seems like evt.rawarg.uid is also empty:

cklein@minitruee:~$ sudo wget https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml -O /etc/falco/rules.d/falco-incubating_rules.yaml
--2023-10-19 17:27:48--  https://raw.githubusercontent.com/falcosecurity/rules/main/rules/falco-incubating_rules.yaml
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.110.133, 185.199.109.133, 185.199.108.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 64418 (63K) [text/plain]
Saving to: ‘/etc/falco/rules.d/falco-incubating_rules.yaml’

/etc/falco/rules.d/falco- 100%[==================================>]  62,91K  --.-KB/s    in 0,01s   

2023-10-19 17:27:48 (5,57 MB/s) - ‘/etc/falco/rules.d/falco-incubating_rules.yaml’ saved [64418/64418]

cklein@minitruee:~$ sudo sed -i -e 's@- required_engine_version@#- required_engine_version@' /etc/falco/rules.d/falco-incubating_rules.yaml
cklein@minitruee:~$ sudo sed -i -e 's@output: Unexpected setuid.*$@& args=%evt.args rawarg_uid=%evt.rawarg.uid@' /etc/falco/rules.d/falco-incubating_rules.yaml
cklein@minitruee:~$ cat /etc/falco/rules.d/falco-incubating_rules.yaml | grep rawarg
  output: Unexpected setuid call by non-sudo, non-root program (arg_uid=%evt.arg.uid evt_type=%evt.type user=%user.name user_uid=%user.uid user_loginuid=%user.loginuid process=%proc.name proc_exepath=%proc.exepath parent=%proc.pname command=%proc.cmdline terminal=%proc.tty exe_flags=%evt.arg.flags %container.info) args=%evt.args rawarg_uid=%evt.rawarg.uid
cklein@minitruee:~$ docker run --rm --user 1000 busybox sleep 10

And the output of Falco becomes:

okt 19 17:30:04 minitruee falco[1286358]: 17:30:04.278000854: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid=<NA> evt_type=setuid user=<NA> user_uid=1000 user_loginuid=-1 process=sleep proc_exepath=/bin/sleep parent=containerd-shim command=sleep 10 terminal=0 exe_flags=<NA> container_id=4ed258f5a0fa container_name=<NA>) args=uid=1000(<NA>)  rawarg_uid=<NA>

[FedeDP]

#1428 will fix the rawarg issue ;)

Also, about:

As you can see %evt.args contains the correct uid, but `%evt.arg.uid% does not.

%evt.arg.uid is resolved to the user name matching the uid, that in this case is "NA":

okt 19 08:54:39 minitruee falco[1211656]: 08:54:39.141114250: Notice Unexpected setuid call by non-sudo, non-root program (arg_uid= evt_type=setuid user= user_uid=1000 user_loginuid=-1 process=cat proc_exepath=/bin/cat parent=containerd-shim command=cat terminal=0 exe_flags= container_id=556312d7d5d0 container_name=) args=uid=1000()

Instead, rawarg.uid should print the number without further conversion. But rawarg was broken for non-strings params :)
Hopefully my patch make it to a Falco 0.36.2 patch release!

[poiana]

Issues go stale after 90d of inactivity.

Mark the issue as fresh with /remove-lifecycle stale.

Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Provide feedback via https://github.com/falcosecurity/community.

/lifecycle stale

[Andreagit97]

/remove-lifecycle stale

[Andreagit97]

it's on my plate, I'm investigating between one thing and another, and hope to come up with a solution soon!

[Andreagit97]

uhm ok, the issue should be clear now, we need to understand how to tackle it!

Running docker run --rm --user 1000 busybox cat means that you are running the processes inside the container as the user 1000 on the host. What we do in the code, and please correct me if I'm wrong @FedeDP is to try to search the user 1000 inside the container but of course, we won't find it because the user 1000 is the one outside the container, and for this reason we cannot resolve the name...

evt.arg.uid returns <NA> because it tries to resolve the username under the hood, so it correctly contains 1000 but before returning it to Falco we try to convert it to the user_name, we don't find it and we return <NA>. The reason why evt.rawarg.uid return 1000 is that for this field we don't perform the resolution

[Andreagit97]

I will move it to https://github.com/falcosecurity/libs since this seems a libsinsp issue

[Andreagit97]

How to reproduce it with sinsp-example

1. Build sinsp-example in full mode, not minimal! Minimal build doesn't know about containers so it tries to fallback on the host and by luck it resolves it correctly!
2. Run sinsp example

sudo ./libsinsp/examples/sinsp-example \
    -o "user_id=%evt.rawarg.uid user_name=%evt.arg.uid" \
    -f "evt.type in (setuid) and evt.dir=> and container.id!=host" --modern_bpf

3. Run

docker run --rm --user 1000 busybox cat

4. You should see the following

user_id=1000 user_name=<NA>
user_id=1000 user_name=<NA>

[FedeDP]

Yep, IMHO this is not a bug: if user 1000 is not existing in the container, it is fine to return NA username, right?
At the same time, if you enforce an existing (in the container) user, everything works fine.

Edit: we might add a fallback lookup for the uid/gid on the host when the container has no such uid/gid perhaps.
I am not sure whether that makes sense though.

[Andreagit97]

Yep, IMHO this is not a bug: if user 1000 is not existing in the container, it is fine to return NA username, right?

Uhm IMO Falco should try to give the right user since it runs on the host and has the possibility to see all the users...

Edit: we might add a fallback lookup for the uid/gid on the host when the container has no such uid/gid perhaps.

This could be an idea, I am not sure if there are other cases when the user is not defined inside the container but it doesn't correspond to any user on the host, I don't think so 🤔 So the fallback might be ok! @deepskyblue86 WDYT?

[FedeDP]

But please note:

docker run --rm --user 1000 busybox whoami
whoami: unknown uid 1000

I still think NA is the proper answer here.

[Andreagit97]

yeah, docker knows nothing about user 1000 because is a user outside the container, but Falco as an external tool has the knowledge to understand who is the real user...the downside of leaving all as it is is that all the processes inside the containers will have user.name = <NA> :/

[FedeDP]

docker knows nothing about user 1000 because is a user outside the container,

I am not sure what --user does, but i guess it just sets setuid for the process inside the container;
see:

docker run --rm --user federico busybox whoami
docker: Error response from daemon: unable to find user federico: no matching entries in passwd file.

the downside of leaving all as it is is that all the processes inside the containers will have user.name =

It's not a downside, it is exactly what is happening. Saying that, for example, docker run --rm --user 1000 busybox cat runs as federico inside the container is simply not true.

Found this on stackoverflow:

Is it safe to use $ docker run -u or USER variable in Dockerfile with a nonexistent user?
Yes. The two things that are important are (a) whether or not the numeric user ID is 0 and (b) whether the numeric user ID matches the ownership of files in the container or mounted volumes. If the numeric user ID doesn't match then the user won't be able to overwrite files.
...
There's no requirement to "create the user" before you do, and it's not an especially good practice to build an image that has a specific host user ID built in.

[FedeDP]

/milestone 0.17.0

[FedeDP]

Is anybody still experiencing issues on Falco 0.37?
Moving to milestone 0.18.0, but ready to close this one if the issue is gone.
/milestone 0.18.0

[Andreagit97]

The decision of closing or not this is on us... we identified the issue, if we decide that we are fine with the <NA> since the container cannot see the external user, i think we can close it

[FedeDP]

I think so, as i explained earlier it would be wrong to state the opposite since if the user is not existent in the container, it is to all effects NA to it.
/milestone 0.17.0
/close

[poiana]

@FedeDP: The provided milestone is not valid for this repository. Milestones in this repository: [0.18.0, TBD, next-driver]

Use /milestone clear to clear the milestone.

In response to this:

I think so, as i explained earlier it would be wrong to state the opposite since if the user is not existent in the container, it is to all effects NA to it.
/milestone 0.17.0
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[poiana]

@FedeDP: Closing this issue.

In response to this:

I think so, as i explained earlier it would be wrong to state the opposite since if the user is not existent in the container, it is to all effects NA to it.
/milestone 0.17.0
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.